A Word (or Two) from our Spenser
Vector 13:2, page 135
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 135
My friend Bill McLean from across the western seas (I refer of course to the Clyde, not the Atlantic) recently drew my attention to a script from John Baker in Kingston, Ontario (one small watery step westwards!) which obtains word lists and concordances using J.
I have adapted this in order to demonstrate how simply this can be done using J. Files which scholars might apply these techniques to in earnest would be of considerable size. For illustrative purposes a demonstration file is printed below which contains a text fragment from Spenser:
… to lose good days that might be better spent;
to waste long nights in pensive discontent;
to speed today, to be put back to-morrow;
to feed on hope, to pine with fear and sorrow;
to fret thy soul with crosses and with cares;
to eat thy heart through comfortless despairs …
Here is the concordancing verb set:
pc=: '.,;'':!?()/-#',10 13{a. NB. punctuation characters plus crlf
index=.e.~#i.@#@] NB. returns y.-indices of chars in x.
prepare =: 3 : 0 NB. makes punctuation-free form of text
y.=.' '(pc&index y.)}y. NB. replace punctuation with blanks
y.=.(] #~ [: -. ' '&E.)' ',y. NB. remove multiple blanks
y. }.~ - ' ' = {: y. NB. remove trailing blanks
)
wordlist =: prepare@(1!:1)@< NB. reads and prepares text file
words =. /:~ @(];._1) NB. words as matrix in alphabetical order
types =. (~:#])@words NB. words with duplicates removed
freqs =. ((#;.1)i.@#)@ ~:@words NB. word frequencies in order of occurrence
fwords =. \:@freqs{types NB. words in order of frequency
tab=. fwords;,.@ \:~ @freqs NB. word frequency table (monadic)
ktab=. ([{.fwords@]);[{.,.@\:~@freqs@] NB. first x. words in tab in
Here are some comments on interesting features of J which it highlights:
words . . the parentheses contain an example of the conjunction "cut" which joins a verb (in this case, simply ]), to a code which says where to find the cut delimiter (1=head, 2=tail), and whose sign says whether to include the delimiter if positive, or exclude it if negative. To the left of @, /: with both(~) produces an alphabetical ordering.
types . . uses "nub sieve" (~:) with copy (#) to remove duplicates.
freqs . . "cut" with another verb (#); delimiter is 0, that is the first item in the result of i., so this cut counts numbers between zeros (including the 0s themselves).
fwords . implied bracketing is (\:@freqs){types, i.e grade down of frequencies is applied to types.
tab . . . . grade down is applied also to freqs.
ktab . . . selects the x. most frequently occurring words.
The functions of the various verbs are best illustrated by a simple example:
x=.'to be, or .... not to be!'
]x1=.prepare x
to be or not to be
words x1 types x1 fwords x1 tab x1 3 ktab x1
be be be ┌───┬─┐ ┌───┬─┐
be not to │be │2│ │be │2│
not or not │to │2│ │to │2│
or to or │not│1│ │not│1│
to freqs x1 │or │1│ └───┴─┘
to 2 1 1 2 └───┴─┘
Now go to the Spenser file, and obtain a count of the most frequently occurring words: