Some J Verbs for Orthogonal Factorial Experiments
Vector 9:3, page 117
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 117
Abstract¶
Verbs are developed in J for analysis-of-variance calculations for factorial experiments with an arbitrary number of factors and with either a complete model with all main effects and interactions or a model with an arbitrary selection of main effects and interactions.
Analysis-of-Variance Computations¶
The main computational problem in the analysis of variance is the partition of the total variation of a dependent variable as measured by the sum of squares of deviations from the arithmetic mean into a number of orthogonal components for each of several main effects and their interactions. If the data are considered to be arranged in a rectangular array with an axis for each factor and one for replications, then a first step in the partitioning process is the calculation of some, or possibly all, of the marginal totals of various subarrays. For example, for a one-factor design the data may be arranged in a table, and the four marginal totals consist of the row totals, the column totals, the grand total, and the items of the array. In general, for an n-dimensional rectangular array there are \(2^n\) marginal totals. For each set of marginal totals a weighted sum of squares is found by squaring each item, summing and dividing by the number of items occurring in each total. From these weighted sums of squares the required analysis-of-variance table may be found relatively simply.
To illustrate the computational procedure to be used in this paper we shall consider a three-factor design with the factors designated by A, B and C so that the data may be considered arranged in a three-dimensional array with any one axis representing replications. Then let \(\mathbf{T}_{\mathrm{a}}\) represent the marginal totals obtained by summing over the second and third axes, \(\mathbf{T}_{\mathrm{ab}}\) the margins obtained by summing over the third axis, and so on. To each set of these marginal totals — there are eight in a three-dimensional array — there corresponds a weighted sum of squares, and, for example, \(\mathbf{S}_{\mathrm{ab}}\) represents that one obtained by squaring each item of \(\mathbf{T}_{\mathrm{ab}}\), summing and dividing by the number of items along the third axis.
Then the sum of squares associated with the main effect for factor A may be shown to be equal to
where \(\mathrm{S}\) is the weighted sum of squares obtained by summing all of the observations and dividing by their total number. Similar expressions hold for the sums of squares for the B and C main effects, \(\mathrm{SS}_{\mathrm{b}}\) and \(\mathrm{SS}_{\mathrm{c}}\), respectively. The sum of squares for the AB interaction is equal to
with similar expressions for \(\mathrm{SS}_{\mathrm{ac}}\) and \(\mathrm{SS}_{\mathrm{bc}}\). The sum of squares for the ABC interaction is
Expressions for the sum of squares for any main effect or interaction term for an arbitrary design may be written down in a similar manner.
The relationship between the subscripts for any sum of squares and the corresponding weighted sums of squares required for its calculation should be noted. For example, for the ABC interaction of the last paragraph the subscripts for the weighted sums of squares may be represented by {abc}, {ab}, {ac}, {bc}, {a}, {b}, {c} and {} which is the power set of the set {abc} of subscripts designating the interaction. Also the sign associated with each weighted sum of squares depends on the number of factors appearing in its subscript, in this example a positive sign if the number of subscripts is odd and a negative sign if the number is zero or even. The required sum of squares is the sum of these signed weighted sums of squares.
The above discussion suggests the following sequence of calculations for finding the sum of squares for an arbitrary main effect or interaction:
- Compute the weighted sums of squares for each member of the power set of the specified subscripts for the main effect or interaction with unspecified positions in the subscripts indicating axes to be summed over.
- Group the weighted sums of squares according to the number of factors appearing in the subscripts and in ascending or descending order of number of factors, and find the sum of the terms in each group.
- Compute the alternating sum of the sums found in the previous step. This gives the required sum of squares apart from the sign which may be accounted for by taking the absolute value.
The development of the verbs in J will be presented in three parts: definition of some preliminary verbs required in the main calculations, verbs for the main calculations discussed in the last paragraph, and a final verb which uses these verbs to assemble the analysis-of-variance table. All of the verbs and the sample data used are given in a listing of the script file in Figure 2.
Some Preliminary Verbs¶
First we shall define the truth table verb
which gives all boolean arguments of the truth table whose size is specified by its argument, so that, for example, tt 2 is:
and tt 3 is
+-----+-----+-----+-----+-----+-----+-----+-----+
|0 0 0|0 0 1|0 1 0|0 1 1|1 0 0|1 0 1|1 1 0|1 1 1|
+-----+-----+-----+-----+-----+-----+-----+-----+
We shall also define the sorted truth table verb
which sorts the items in ascending order of the number of 1s each contains, and stt 3 is:
+-----+-----+-----+-----+-----+-----+-----+-----+
|0 0 0|1 0 0|0 1 0|0 0 1|1 1 0|1 0 1|0 1 1|1 1 1|
+-----+-----+-----+-----+-----+-----+-----+-----+
The main effects and interactions will be labelled in the analysis-of-variance table by appropriate upper case letters but represented internally in binary form, so that, for example, in the three-factor design discussed in the last section the AB interaction will be represented externally as AB and internally as 1 1 0. To simplify the conversion between representations we shall define the verbs
where
and
and, for example, tag 1 1 0 is AB and 'AB' numtag TestData, where TestData is a three-dimensional array to be used later in the paper, is 1 1 0. We shall also define the verb
which gives all the labels for a factorial design of any size, and, for example, alltags 3 is
Finally we shall define the verb
where
has been suggested by Kenneth Iverson and is given in McIntyre (1991), to find the power set of the subscripts corresponding to a main effect or interaction. As an example, ps 1 1 0 has the value
which indicates that the calculation of the sum of squares \(\mathrm{SS}_{\mathrm{ab}}\) requires the weighted sums of squares \(\mathrm{S}\), \(\mathrm{S}_{\mathrm{a}}\), \(\mathrm{S}_{\mathrm{b}}\) and \(\mathrm{SS}_{\mathrm{ab}}\).
Basic Calculations¶
The calculation of any set of marginal totals in an arbitrary array may be accomplished by first transposing the array so that the axes to be summed over occur first and then summing over these axes. We shall specify the axes by a boolean list of length equal to the rank of the array with 0s indicating the axes being summed over. For example, consider summing over the first, third and fourth axes of an array a, where $a is 2 3 4 5, say, so that the boolean list specifying the summation is 0 1 0 0. Since /:0 1 0 0 is equal to 0 2 3 1, the transposition may be accomplished by the expression 0 2 3 1|:a which gives an array of shape 2 4 5 3.
Application of the verb +/ three times, the number of 0s in the boolean list specifying the marginal totals, to this array will give the desired marginal totals. These calculations may be accomplished simply by the explicit verb
where the right argument gives the array and the left argument specifies the marginal totals required. Then the above example is given by 0 1 0 0 T a. The weighted sum of squares is given by
where the arguments are the same as for T. As an example of the use of the above verbs consider the three-by-four table x given by
Then 0 0 T x is the sum 63 of the items of x, 1 0 T x is the row sums 22 24 17, 0 1 T x is the column sums 15 16 15 17, and 1 1 T x is the array x. The corresponding values of the weighted sums of squares given by S are 330.75, 337.25, 331.667 and 383, respectively.
A verb for calculating the sum of squares for a main effect or interaction may now be defined with a little thought as
and, for example, 'A' ss x is 6.5 and 'AB' ss x is 44.8333. If we define the verb
for the degrees of freedom for any term, then the verb
will give a three-item list of degrees of freedom, sum of squares and mean square for any one term. For example, 'A' term x is 2 6.5 3.25 and 'AB' term x is 6 44.8333 7.47222.
An arbitrary number of terms may be calculated by the verb
and ('A';,'AB') allterms x is the table
The Analysis-of-variance Verb AOV¶
It is a relatively simple matter to use these verbs to define an explicit verb AOV that will assemble and display an analysis-of-variance table for an arbitrary model and rectangular array of data.
Figure 1 gives a listing of AOV and two examples of its use. We note that the right argument gives the array and the left argument specifies the model. A monadic use of the function with the data as the single argument gives an analysis of variance for a full model with all main effects and interactions of all orders. The result of the verb is the analysis-of-variance table giving an appropriately labelled row with the degrees of freedom, sum of squares and mean square for each term, an error term if there is one, and a total. The numeric part of the table is also given as the global pronoun AOVtable.
Keith Smillie
Department of Computing Science
University of Alberta
Edmonton, Alberta T6G 2H1
References¶
[1] Bennett, C. A. and N. L. Franklin, 1954. Statistical Analysis in Chemistry and the Chemical Industry. John Wiley and Sons, Inc., New York.
[2] McIntyre, D. B., 1991. "Mastering J". APL91 Conference Proceedings, Palo Alto, California, pp. 264 - 273.
AOV¶
+--------------------+-+-----------------------------------------------+
|(alltags #$y.) AOV y.|:|AOVtable=. x. allterms y. |
| | |Labels=. 8{."1 >x. |
| | |dfTotal=. <:*/$y. |
| | |ssTotal=. (+/*:,y.) - '' ss y. |
| | |dfError=. dfTotal - +/0{"1 AOVtable |
| | |$.=. > (dfError=0) { (<6 7 8 9 10 11),<9 10 11|
| | |ssError=. ssTotal - +/1{"1 AOVtable |
| | |AOVtable=. AOVtable,ssError(],[,%)dfError |
| | |Labels=. Labels,'Error' |
| | |r=. Labels,"1 (5 12.5 12.5)":AOVtable |
| | |AOVtable=. AOVtable,TotalRow=. dfTotal,ssTotal |
| | |r, 'Total ',5 12.5":TotalRow |
+--------------------+-+-----------------------------------------------+
TestData
25 7 21 4 10 16
5 21 4 25 7 6
3 6 16 18 20 18
19 17 16 2 15 6
13 23 8 25 19 19
20 14 23 9 10 18
27 26 19 24 13 9
19 13 20 18 16 12
$TestData
2 4 6
AOV TestData
A 1 252.08333 252.08333
B 3 69.16667 23.05556
C 5 74.66667 14.93333
AB 3 10.41667 3.47222
AC 5 51.91667 10.38333
BC 15 309.83333 20.65556
ABC 15 1495.58333 99.70556
Total 47 2263.66667
('A';'B';'AB';'C') AOV TestData
A 1 252.08333 252.08333
B 3 69.16667 23.05556
AB 3 10.41667 3.47222
C 5 74.66667 14.93333
Error 35 1857.33333 53.06667
Total 47 2263.66667
AOVtable
1 252.083 252.083
3 69.1667 23.0556
3 10.4167 3.47222
5 74.6667 14.9333
35 1857.33 53.0667
47 2263.67 0
Figure 1. The verb AOV.
tt=. ,@{@(]@#&(<0 1))
stt=. <"1@|."1@(/: +/"1)@(>@tt)
alphabet=. 'ABCDEF'
tag=. ]#({.&alphabet)@#@]
numtag=. (({.&alphabet)@(#@$@])) e. [
alltags=. (tag &. >)@(1&}.)@stt
expand=. /:@\:@[{#@[{.]
ps=. <"1@expand"1 >@stt@(+/)
T=. '' : '+/^:(+/-.x.)(/:x.)|:y.'
S=. (+/@((*:@,)@T)) % (*/@(-.@[ # $@]))
ss=. | @ (-/) @ (>@ps@([numtag]) (+/"1@[ +//. S"1 _) ])
df=. */@<:@(numtag # $@])
term=. ss (] , [ , %) df
allterms=. (>@[) term"1 _ ]
s0=. '(alltags #$y.) AOV y.'
t0=. 'AOVtable=. x. allterms y.'
t1=. 'Labels=. 8{."1 >x.'
t2=. 'dfTotal=. <:*/$y.'
t3=. 'ssTotal=. (+/*:,y.) - '''' ss y.'
t4=. 'dfError=. dfTotal - +/0{"1 AOVtable'
t5=. '$.=. > (dfError=0) { (<6 7 8 9 10 11),<9 10 11'
t6=. 'ssError=. ssTotal - +/1{"1 AOVtable'
t7=. 'AOVtable=. AOVtable,ssError(],[,%)dfError'
t8=. 'Labels=. Labels,''Error'''
t9=. 'r=. Labels,"1 (5 12.5 12.5)":AOVtable'
t10=. 'AOVtable=: AOVtable,TotalRow=. dfTotal,ssTotal'
t11=. 'r, ''Total '',5 12.5":TotalRow'
t=. t0;t1;t2;t3;t4;t5;t6;t7;t8;t9;t10;t11
AOV=. s0 : t
x=. 3 4 $ 4 7 5 6 9 4 3 8 2 5 7 3
TestData=. 25 7 21 4 10 16 5 21 4 25 7 6
TestData=. TestData, 3 6 16 18 20 18 19 17 16 2 15 6
TestData=. TestData, 13 23 8 25 19 19 20 14 23 9 10 18
TestData=. TestData, 27 26 19 24 13 9 19 13 20 18 16 12
TestData=. 2 4 6 $ TestData
Figure 2. Script file.