ASL: A Basic Statistics Volume Tutorial
Vector 8:2, page 37
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 37
The Basic Statistics tutorial gave a demonstration of some of the Bottom Shelf functions applied to two sets of data: the monthly American Stock Exchange Market Value Index (closing levels) for 1986, 1987 and 1988 and UK TV Franchise Data on 16 UK Regional TV Stations taken from ‘The Guardian’ April 1991.
First, consider the American Stock Exchange closing levels for the period 1986 to 1988, a period which includes the October 1987 crash. The first stage in any statistical analysis is to look at the data:
M
243.87 257.35 270.03 268.97 282.6 284.2 261.56 273.85 260.69
265.59 264.81 263.27 300.47 321.76 332.66 325.19 326.39 338.13
358.03 361.35 356.45 260.36 242.39 260.35 269.1 288.46 296.43
303.14 294.19 309.25 306.18 294.8 301.63 300.95 294.36 306.01
The data itself is a simple numeric vector, and as you would expect we have a series of functions to compute some basic statistics from the data such as mean, variance, minimum value and maximum value:
MEAN M FIND THE MEAN
292.9
VAR M AND THE VARIANCE
1015
MIN M EVEN 'MIN' AND 'MAX'!
242.4
MAX M
361.4
We might also require a feel for the shape of the distribution of data. Hence we need frequencies:
FREQTAB M A FREQUENCY TABLE
240 2
250 1
260 9 ... 310 0
270 2 320 3
280 3 330 2
290 4 340 0
300 7 ... 350 2
A visual representation is an improvement on this, and hence we have a simple barchart of data using the frequencies:
BARCHART FREQTAB M ⍝ AND A VERY SIMPLE BARCHART OF THE
TABLE. EACH ⎕ REPRESENTS 1 UNITS
SCALE IS 1
240.00|⎕⎕
250.00|⎕
260.00|⎕⎕⎕⎕⎕⎕⎕⎕⎕
270.00|⎕⎕
280.00|⎕⎕⎕
290.00|⎕⎕⎕⎕
300.00|⎕⎕⎕⎕⎕⎕⎕
310.00|
320.00|⎕⎕⎕
330.00|⎕⎕
340.00|
350.00|⎕⎕
Significance Tests¶
Given the stock market collapse, we might be interested to know whether 1988’s figures are the same as 1986’s. If they are, has the market recovered?
If we are willing to assume that the data is normal and independent then we can use a parametric test. We first test the data to see if the variances are the same with a variance test. This is based on the ratio of the two variances:
The value has to be compared to the F Distribution to see if it is significant. Given the value is less than one, we can take the reciprocal test is against F on 11,11 degrees of freedom.
The probability is fortunately in the central region of the distribution and hence we can assume the data have similar variances. We can now carry out the two sample t test.
The value obtained should be compared to the t distribution on 22 degrees of freedom.
The result indicates that the populations are different, since the probability is in the tail of the distribution. It may be that one is doubtful about the normal assumption? Therefore a non-parametric test might be more sensible. The Mann-Whitney test is an appropriate choice.
This value could be compared to the appropriate table but if the analyst is willing to take the normalised value, he could compare with the t distribution as an approximation. The sample is relatively large.
This produces the normalised value of the test statistic. The probability value would be
Fortunately a similar result as before. We might consider comparing the three years assuming the values are random observations from the years, although this is a little doubtful. The one way ANOVA test can be used to test whether the three years differ.
J←(12 12 12/1 2 3) ONEWAY M
J
2 14651.35035 7325.675175 11.58166525
33 20873.27475 632.5234773 0
35 35524.6251 0 0
We have assigned the result to J so that we can test the variance ratio against the F distribution.
A low probability, implying difference, as might be expected.
Time Series¶
The above analysis was possibly not wholly appropriate since we know the data is a time series. Hence it would be sensible to treat it as such. We can then obtain the Moving Average to see whether there is a trend.
12 MOVA M
266.4 271.1 276.5 281.7 286.4 290 294.5 302.6 309.9 317.8 317.4
315.5 315.3 312.7 309.9 306.9 305 302.4 300 295.6 290.1 285.5
288.9 293.2 297
This produces a moving average whose values fall between the observations since the period 12 is even. It is more usual to produce the Centred Moving Average for even data.
12 CENTMOVA M
268.8 273.8 279.1 284 288.2 292.3 298.6 306.2 313.9 317.6 316.5
315.4 314 311.3 308.4 306 303.7 301.2 297.8 292.9 287.8 287.2
291.1 295.1
With this it is possible to calculate the seasonal effects and hence remove them from the data. The seasonally adjusted moving average can be obtained for both an additive model for seasonal effects as well as multiplicative. In the case of the data we might feel that as the trend increases, so seasonal effects increase and so the multiplicative model is appropriate. Hence we use:
12 SEASAM M
254.4 251.9 257.4 257.7 275.7 267.1 246.6 252.5 248.5 296.3
307.3 297.2 313.4 314.9 317.1 311.6 318.5 317.8 337.5 333.2
339.8 290.5 281.3 293.9 280.7 282.3 282.6 290.5 287 290.7 288.6
271.8 287.6 335.7 341.6 345.4
Now consider the other dataset - 1991 UK TV franchise data, held as a three column numeric matrix with the columns representing, successively turnover, viewing households, and staff numbers. This will be rather more topical for UK readers in view of the current TV franchise changes being considered.
The descriptive statistics functions can be applied to arrays, so we have:
DATAF
12.2 262 140
26 483 290
250 2870 1100
143.5 2000 700
220 2500 800
44 635 307
5.1 50 97
250 4900 910
86.5 2.2E4 399
347 5300 1550
314 4100 1000
125 1600 711
201.1 3000 1137
63 3000 380
110 1400 600
20 442 208
MEAN DATAF
138.6 3409 645.6
VAR DATAF
1.257E4 2.726E7 1.743E5
Linear Regression¶
In this data we might be interested to see if there is a relationship between Turnover and Household and the number of employees. Obviously as prospective franchise buyers we wish to predict Turnover.
Can we predict Turnover from Households? The model would be a linear regression:
T = b0 + b1 H
The model is fitted using a set of routines REGLINEST and REGLINRESID etc., but we can use REGLINREP to get a brief report. We store the results again, to test against the F distribution.
FP←⍎,REGLINREP DATAF[;1 2]
REGLINREP DATAF[;1 2]
EST REGRN. COEFF(S). 126.51 3.5443E¯3
EST STAN. ERROR(S). 34.508 5.6596E¯3
EST ERROR VARIANCE. 13097
EST COEFF. OF DETERMINATION 0.02725
ANALYSIS OF VARIANCE:-
SOURCE SS DF MS F-RATIO
REGRESSION 5136.6 1 5136.6 0.39219
RESIDUAL 1.8336E5 14 13097
TOTAL 1.885E5 15 12567
The result is a little disappointing, apparently Household on its own does not account for a significant amount of variation in Turnover.
Indicating that the model is not significant.
Multiple Regression¶
Is prediction of Turnover better if we add Staff Numbers to the model? The model is now a multiple regression:
T = b0 + b1H + b2S
We have a set of regression functions which enable us to do this (REGEST, REGRESID etc). Again we can use the reporting function REGREP.
FP←⍎,REGREP DATAF,1
REGREP DATAF,1
EST. REGRN. COEFF(S). 1.1239E¯3 0.25404 ¯29.243
EST. STAN. ERROR(S). 1.8005E¯3 0.022514 17.587
EST. ERROR VARIANCE. 1306.7
EST. COEFF. OF DETERMINATION 0.90988
ANALYSIS OF VARIANCE:-
SOURCE SS DF MS F-RATIO
REGRESSION 1.7151E5 2 85755 65.628
RESIDUAL 16987 13 1306.7
TOTAL 1.885E5 15 12567
We have assumed a mean, hence the column of 1 in the matrix.
The result is significant. Fitting both variables seems to account for a large proportion of the variability in Turnover.
Polynomial Regression¶
Instead of adding Staff Numbers, we could have tried using the polynomial model,
T = b0 + b1H + b2H*2
A set of Polynomial Regression functions exist in basic Statistics - REGPOLYEST etc. Using REGPOLYREP to fit a quadratic, we get:
FP←⍎,2 REGPOLYREP DATAF[;1 2]
2 REGPOLYREP DATAF[;1 2]
EST. REGRN. COEFF(S). ¯4.0766 0.078944 ¯3.4013E¯6
EST. STAN. ERROR(S). 22.096 9.7859E¯3 4.2767E¯7
EST. ERROR VARIANCE. 2404.7
EST. COEFF. OF DETERMINATION 0.83416
ANALYSIS OF VARIANCE:-
SOURCE SS DF MS F-RATIO
REGRESSION 1.5724E5 2 78618 32.694
RESIDUAL 31261 13 2404.7
TOTAL 1.885E5 15 12567
and testing against F, we get
which is significant. The model explains a large proportion of the variability.