Statistical Computing with APL
Vector 1:2, page 113
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 113
The programming language APL has been around since the mid sixties. It has been slow in gaining widespread acceptance, especially in the UK, being thought of as the province of ‘mathematicians, statisticians and nutcases’. This is because it is quite unlike the other major languages, with a lot of funny symbols (mainly Greek) which make the code very concise and mathematical-looking. This tends to put people off at first sight and means that, if you want to stay sane, you need special terminals. In operation it tends to be greedy in CPU time, making the user unpopular with the computer manager, but the days of cheap independent processors are doing away with that problem.
In the last three years or so the use of APL has started to grow rapidly in this country, with the membership and activities of the British APL Association expanding fast. Several small, lively companies specialising in APL hardware and software are active. And funnily enough the vast majority of users are not mathematicians or statisticians, many of whom have never heard of it, but accountants, management services departments and industrial and commercial research people. The essential reason is that it is cost effective; they want answers today or tomorrow, not in three months time. They want the flexibility to make changes and try out new ideas quickly, not packages that have a set routine and take ages to alter. It is typically much faster to develop programs (sets of functions) in APL than in other languages. Today it is people-time that is expensive, not machine-time. An added bonus is the elegance and logical simplicity of the language which makes it fun to work with, in a way that slogging through a hundred lines of FORTRAN can never be. And if you do write a package in APL, it can easily be arranged that the end user doesn’t have to see those funny symbols and need only use English language.
Language of Statistics¶
In my view, of all the computer languages currently available, APL is the natural language of statistics. I would urge all statisticians, who have not already done so, to spend two or three days giving it a serious try, say with a copy of Gilman and Rose (1); then they can make up their own minds. There are many reasons for this view of which some are as follows:—
a) It is flexible, interactive and very powerful. Often only a few minutes are required between having an idea and getting the results back, making it easy to experiment or try out different forms of analysis depending on the way the data looks. Data is easily manipulated.
b) There are several built-in functions which are very useful to statisticians. For example !R is factorial R, if R is an integer, or, more generally, the gamma function Γ(R + 1). The dyadic form of the same symbol used as R!N produces the combinatorial (NR).
A random sample without replacement of R objects from a finite population of data stored in a vector X can be taken simply by
which assigns the sample values to a new vector Y (what, no dimension statement?). What are the properties of the sample? You might fancy something a bit unusual like the mean of all the sample values excluding the two largest and two smallest (assuming there are more than four); that can be found simply by
Yes I know it looks queer, but it is really very easy when you know how and only takes about a minute to think of it, type it and get the answer back. How long would it take in FORTRAN or BASIC? In the above expression
gets the data into numerical order,
drops two items off each end and the rest of the statement calculates the mean
tells you how many objects there are and
adds them up. Note that APL goes from right to left.
c) Like Glim, GENSTAT or MINITAB the language is array oriented in data structure and computation, but easier to use and more versatile. For example ! 2 3 4 yields the result 2 6 24 being the factorials of each value in the vector 2 3 4. A nice example
calculates all the probabilities p(0), p(1) … p(N) for Binomial distribution Bin (N,P).
d) The language is modular in structure with the natural modules being user defined functions. Each function has a name and optional right and left arguments. For example, one of mine, NORMTAIL X calculates the upper tail probability of the unit normal distribution corresponding to each element of any array X you supply, e.g. NORMTAIL -1.96 0 2.576 elicits the immediate response
It is very easy to make use of functions you have previously written for another purpose. Even better, with a group of people working on the same system a significant increase in productivity is possible by using functions others have already written, or perhaps will write for you during the course of the afternoon if problems are freely discussed among the group.
e) Matrix algebra is very simple. Consider the following matrix expression
If the matrices on the right already exist, then S can be computed by
Apart from a slight change of notation, this has the great advantage that it is transparently obvious that it represents the above matrix expression, and you get an immediate answer.
computes the inverse of X. There is also a more general dyadic use of this symbol. Thus if Y is a vector and X a matrix of predictor variables, then
supplies the vector of regression coefficients obtained by least squares fit.
then finds the residuals and
calculates the residual sum of squares.
Why APL?¶
There are a number of good non-APL statistical packages widely available which have been refined over many years. Some are better than others, but most have a restricted range of what you can do. If you have a lot of routine analyses to do you will probably be quite happy with three or four, say SAS, CLUSTAN and GLIM, depending on your field of application. However, with APL you can soon build up a package of standard functions and leave yourself free to tie them together how you like, or slot in new functions as required; in other words leaving yourself in total command of the situation, not restricted to what the package writer will allow you to do. If you want ready made APL software, there are good published functions in Anscombe (2) and some in the periodicals VECTOR and APL Quote Quad. Packages such as STATGRAPHICS (from the USA) and various specialist ones from the Applied Statistics Research Unit at the University of Kent, from Swansea University, and elsewhere, are starting to become available.
Thus you have the ability to use other people’s packages or functions as well as your own, tie them up together with the use of very powerful raw APL (as indicated in the previous section) in a flexible mode of operation.
Similar advantages occur in using APL for teaching statistics. In Swansea, apart from service teaching, all undergraduate statistics courses now assume the student has a basic fluency in APL and the ability to use collections of functions relevant to each course, e.g. time series functions, multivariate functions etc. Students begin at the terminal the first week of the first year. They can be eased gently into computing in easy stages, yet are very quickly able to do significant calculations, using the machine like a super calculator. Within five weeks students cope quite happily with all the examples of the previous section, except matrix algebra and regression which they meet at the end of the first year. Students are able to analyse much more data and the calculations can be organised so that the process of carrying them out enhances the student’s understanding of the theory. With most packages it is a question of ‘numbers in, numbers out’ with little idea of what went on between. We find that students take to APL very readily, being much easier to learn than BASIC. For further details see Hawkes (3).
Possible disadvantages have been mentioned before. You need the equipment and the interpreter, but these are now quite cheap. Secondly, APL can be a bit slow in machine-time on large scale number crunching (not always by any means). The primitive functions are very fast, but if you write loops which are executed many times (fortunately this is often not necessary) the interpretive overheads can be quite high. Compilers have been developed, but are not widely available. However, one or two versions of APL are said to be semi-compiled and should offer a considerable improvement in performance. But if you value people-time more highly this may not be at all important. APL*PLUS have just introduced a limited compiling facility, while IBM have the production of an APL compiler as one of their research priorities.
Which APL?¶
Fortunately, unlike BASIC, there has for some time been a high degree of standardisation between different interpreters. I call this ‘standard’ APL. In the last two years or so there have been introduced at least four ‘second generation’ APL interpreters which enhance and enrich the language significantly. To some extent each has developed in its own way, but there are encouraging signs that the suppliers are talking to each other, so that a new standard may emerge. All embrace the idea of ‘nested arrays’ implemented in various ways. These allow a flexible and convenient data structure allowing numeric and string arrays of various shapes to be mixed together in a tree-like structure, together with powerful functions and operators with which they can be manipulated. IBM’s APL2 and I.P. Sharp’s APL use complex numbers as the basic field for arithmetic; other interpreters are likely to follow this lead in the future. The above interpreters run on IBM mainframes, and an I.P. Sharp version recently became available for the IBM PC micro, allowing the possibility of access to their worldwide information services. There are also a number of standard APL’s for the PC including IBM and APL*PLUS. The latter is due also to be released on the DEC Rainbow in August 1984.
Dyadic Systems, a British company, have a second generation interpreter called DYALOG. This is written in ‘C’, runs under the UNIX operating system and incorporates a lot of UNIX facilities within the language. This makes it particularly easy to interface with other processes which run under UNIX. For example you could write a special routine in the very efficient compiled language ‘C’ which could be called directly from APL and the results returned to APL. A statistical package written in FORTRAN, such as SPSS, might be called in a similar way. APL is very good at glueing things together in this way, especially in a multi-tasking environment like UNIX. APL*PLUS also has a second generation interpreter which runs on IBM mainframes and has recently produced a version written in ‘C’. Like DYALOG this gives it great portability over a wide range of machines run under UNIX. As it is said to be semi-compiled, it should run quite fast. Both of these interpreters should now be available on a wide variety of machines including VAX, Perkin Elmer, Hewlett Packard, PERQ and many based on the currently popular MC68000 16-bit chip such as Cadmus, Bleasdale, Fortune, Wicat, Cifer and Torch. A Cadmus running DYALOG or APL*PLUS appears to be a particularly powerful systam at a reasonable price.
Standard APL’s, in addition to those already mentioned for the IBM PC and Rainbow, include VAX APL by Digital and APL68000 which runs on Sages and on the Spectrum (produced by MicroAPL, not Sinclair!). These are proven, reliable tools at the minimicro level. Standard interpreters are also available on PRIME and on most mainframes. For 8-bit micros a good standard APL is VIZ::APL, which runs under CP/M. Some 8-bit APL’s tend to lack the more useful features such as matrix inversion, so look carefully at the specification.
Earlier this year there was the announcement of Acornsoft APL, developed by Inner Product Ltd., which will run on a BBC micro fitted with the new Z80B second processor and a suitable 80-track disk drive. This incorporates ready access to the excellent BBC colour graphics. As it is not likely to be in the shops before August 1984, I am unable to comment on performance. The specification sounds good, and if it comes up to expectation, it should provide a cheap APL machine that will find its way into many schools, colleges and homes. Furthermore, I understand that a good sixteen bit British APL micro is likely to be on the market at under £1,000 within the next year. Then the growth in APL usage should be really fast.
Finally, if you can afford at least £60,000, try The APL Machine being developed by Analogic. It is a small array processing machine for an array processing language which can execute APL with impressive speed.
(This paper first appeared in the Professional Statistician)
References¶
- L. GILMAN and A.J. ROSE (1984), APL an interactive approach, 3rd ed., Wiley.
- F. ANSCOMBE (1981), Computing in statistical science through APL, Springer.
- A.G HAWKES (1983), The role of APL in statistics teaching, APL Business technology 1983, UK APL User Group, 123-132.