ASL: The Basic Statistics Volume

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 45

Introduction¶

To give structure to the ASL library it is divided into shelves. Whilst most shelves have specific application the Bottom Shelf is a general shelf which will contain functions to be used throughout the library. Each shelf may be divided into a series of volumes.

The Bottom Shelf will eventually consist of the four volumes: basic statistics, numerical analysis, data management and utilities. ASL volumes are not packages but a set of well tested and established APL routines. Packages may, of course, be constructed from the routines. Those wishing to provide commercial packages based on ASL will be allowed to do so through a licensing agreement with the British APL Association.

In this article we will review the Basic Statistics volume including a brief outline of the material contained within the shelf. Since the design process for this volume differs from those previously described (the Graphics and Regression Shelves) I have added some comments on design and standards. (The Basic Statistics volume may possibly form the blueprint for the development of subsequent shelves.)

A list of the functions in the Basic Statistics volume was given in the first edition of Random Vector. For those who desire a more formal account Norman Thomson has produced a Basic Statistics specification document based on APL2 which discusses in more detail aspects of the volume.

Contents of the Volume¶

At the first ASL Workshop, held at Gregynog, Wales, it was decided that the Basic Statistics should cover the material required to teach first year undergraduate or A Level students Statistics. This would then be enhanced as the library grew with the inclusion of functions common to more than one shelf.

The statistics currently within the volume can be divided into the following areas:

• Descriptive and Summary Statistics
• Distribution functions
• Significance tests
• ANOVA
• Time Series
• Regression and Random Number Generators

It was realised though there would be need for some mathematical and utility functions. The current version consists of about 120 functions, though some are variants with particular attributes.

The majority of functions unsurprisingly belong to the first set, Descriptive and Summary Statistics. This includes functions to calculate means, variances, standard deviations and similar quantities as well as simple line printer barcharts. Not included are functions which need better graphic output than simple line graphics such as box-whisker, pie charts etc. These are contained within the Graphics Shelf.

As far as distributions are concerned we have been conservative providing only functions relating to Binomial, Poisson, Hypergeometric, Chi Squared, Normal, F and t. Where appropriate the functions provided yield the density, quantile and tail probability value. More specialised functions will appear either in subsequent basic statistics volumes or in a separate Distribution shelf.

To be controversial we have included a number of test statistics based on traditional or frequentist approach to Statistics. Again it is hoped eventually to provide appropriate Bayesian approaches to inferential problems. The tests provided are both parametric and non-parametric, for example TSTAT yields the value of t-test statistic and MANNWHIT provides a Mann-Whitney test statistic value.

In more specialised areas we have restricted ourselves to very basic functions, for example the bivariate measures are limited to correlation. In Time Series, functions such as moving averages, seasonal adjustment and exponential weighted moving averages are given. Similarly restrictions are imposed on ANOVA with one-way and two-way functions provided. Again it is expected that other shelves will provide more adequate cover of these areas.

Whilst the Regression shelf provides a powerful suite of programs for regression, it was still felt appropriate to have basic regression functions in the Bottom Shelf. Hence the Bottom Shelf provides functions to obtain parameter estimates, predictions, residuals and ANOVA tables for linear (multiple), simple linear and polynomial regression.

Again random number generators have been provided for a basic set of distributions and should cover those generally used. It is expected that a wider range of alternative routines will be provided in a Random Number shelf.

Development Process¶

Early in the evolution of ASL it was felt that the best way to proceed would be to establish working parties which would review material which had been produced or collated by a single author. When the working party was satisfied the material would be released to a larger group for testing and comment.

In the context of Bottom Shelf this approach has worked relatively well. Obviously there is a need to ensure that the working party does not continually till the same ground and that it attempts to maintain consistency over its decisions. A feature of software developed by voluntary labour is that timescales tend to be rather elastic. However even within the Basic Statistics volume the implementation time has not been unduly long and where delays have occurred it has been to ensure better software has been produced.

The ASL programmer, Tuks Gillespie, acted as collator and designer for the material in the Basic Statistics volume. The original source material on which she based her work was Norman Thomson’s Bottom Shelf: Basic Statistics [1] and University College of Swansea’s first year statistics course material based on APL. The working party which reviewed the software was Alan Hawkes, Norman Thomson, Alan Sykes, Alan Meyer, Mike Day, and Jake Ansell. The group meet regularly over about eighteen months. The group which tested the software is too large to name fully, however Maurice Jordan and Tony O’Hagan deserve special mention given the considerable effort they put in to testing and commenting on the material. Maurice Jordan adapted his APL testbed functions for basic statistics and subjected the volume to rigorous inspection.

Standards¶

Within such a major undertaking it is clear that there is a need for standards. Considerable attention has been paid to developing standards which will last throughout the likely extended lifetime of the project. There have been two major inputs: comments taken from APL programmers (particularly Adrian Smith), and those standards which have emerged through development of the software.

A more detailed article might eventually be written about the standards ASL has evolved. At the present time I will concentrate on those that directly affect presentation of the function. A major step in writing an APL function is naming it. Two aspects initially have to be taken into account. Firstly the name must identify as clearly as possible the function’s use and secondly it must not clash with other names.

Even in the Basic Statistics volume the working party has devoted time ensuring appropriate names are used. Names will usually conform to the following pattern <root><inflection><serial no.>, where the root is the type of function, and the inflection modifies the root to identify a function performing a specific task. Where alternative algorithms are available to implement this task, the serial number identifies the particular algorithm used. (Users would select the algorithm they preferred.) CHIQUANT1 is an example of this convention.

To avoid the problem of name clashes a Names Guardian has been established which will record name usage and decide on the new name(s) if there are any clashes.

Within a function (where plausible) variable names are used which imply type or structure of variable. For example for binomial functions N_P is the variable name used for the parameters.

Every ASL function will be fully commented and where appropriate will be annotated. The comments statements start with a two letter code and a list of those used are given below:

FN

Purpose/objective

RE

Result

SS

Shelf Source

RA

Right Argument

LA

Left argument

AL

Algorithm detail

This, of course, allows for automatic documentation, particularly helpful in ensuring consistency of production.

Reference¶

[1] Norman Thomson, [1990], ‘Bottom Shelf: Basic Statistics’.