The IML Matrix Language: A Belated Introduction
Vector 14:1, page 78
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 78
This is an introduction to yet another array-processing language. Why even bother? Because it is one that might be acceptable to Information Systems bureaucrats, and this could affect your work situation, perhaps your career.
Let me explain by using my own experience as an example. My own business is creating and selling international demographic and household income projections, mainly to car makers. The main systems are written in APL, though my new county/province system is in J. In order to have more predictable income to pay for my children’s extravagant lifestyles at university, I also create demographic systems for a government agency: this is where the trouble starts.
Information Systems (IS) bureaucrats are fairly tolerant of researchers in my group ‘doing their own thing’ with respect to personal productivity software. However, software relating to agency missions is expected to be written in languages that can, in principle, be supported by IS staff. This is reasonable. If I did all my work in APL and got hit by the proverbial truck, no one could carry on what I had been doing. (Actually, if I were in charge of my unit, I would start doing everything in J — mostly because it costs less than the major APLs — and then cross-train a couple of people to get the required redundancy.) At any rate, I began by writing ‘prototype’ systems in APL and using the results for some agency products, but eventually the law was laid down and I had to commit myself to rewriting the systems in something acceptable.
I’ll skip over a nasty experience with Visual Basic. Eventually, it was decided that I should try writing in the Interactive Matrix Language (IML) component of the SAS database/statistics system; SAS is widely used, and about five people in the research group use it heavily. There is even a State SAS users’ group that attracts 30 or so people to its monthly meetings. Clearly, SAS is not a trivial factor in the data analysis world.
I have just completed a system written in IML. My bosses should be happy because I am using SAS. I am not totally unhappy, because I didn’t have to do it in Visual Basic. But the ironic thing is that, even though I ‘did it in SAS’, I’m the only one in the research section who understands IML. If that truck hits me, the agency will still be up the creek!
The point of this introduction is that there are acceptable array-language fall-back options for APL programmers. As will be explained below, IML is a mere shadow of APL. But it can be learned quickly, and perhaps you can buy some time to convince management that J, say, is just like IML but a lot more powerful, etc., etc.
IML Overview¶
IML dates back to the late 1970s, not long after SAS Institute set up shop. In 1979, I briefly used an earlier version (known as Matrix Language, I believe) to write a simple population projection system. That was before I had heard of APL. In fact, I tried APL because it didn’t need all those loop statements — just like Matrix Language! Actually, SAS documentation admits that IML is actually based on APL. This means that it is not hard for an APL programmer to do useful work in IML within a few days; the obstacle is that a certain amount of basic SAS needs to be learned first.
SAS itself allows users to access data from a variety of formats from a variety of computer platforms. My SAS-fan co-workers, who have to dig into pretty large databases, think this is great because the same code they write on their PC can be used to process data on a VAX or an IBM mainframe. Furthermore, SAS Institute takes great pains to keep core language features consistent and compatible over time. Non-core features such as windowing environments are kept up to speed with host systems; for example, the Windows 95 version has the Microsoft approval seal, just like Dyadic’s APL.
IML is sort of an oddball in the SAS scheme. It can work with SAS data sets (the ‘data set’ — a grouping of fields and records — is the core of SAS), yet it can also deal with outside files directly and do other tasks that can be handled in SAS. In other words, it is semi-independent: a sort of APL, I suppose.
I don’t pretend to be an IML expert. I’ve only worked with it a month or so and I had no experienced user or SAS Institute tech rep to consult. Therefore, some of the methods I used are likely to be sub-optimal or even frowned upon. My only other excuse is that I was under deadline pressure and had to get something working. Bottom line: take what I say with a little salt, though the gist of it is probably sound.
The first thing you have to do is get data into IML. This can be done at the keyboard, but if you want to do back-of-the-envelope calculating, fire up APL or J instead. For more serious work, you can either read files directly using IML or else read the files with SAS and then pass the data to IML. I used the latter approach. Why? According to my IML manual (published in 1990), IML can read comma-delimited, but not tab-delimited files. SAS for sure can read either, and over the past few years I’ve been standardizing on tab-delimited files for communication with the non-APL/J world. (Specifically, I use the *.txt format that can be created by Excel. I often key in data in Excel and save the results both in spreadsheet and text format — the latter can easily be used by J with its clipunfmt function.)
The way my system works — it sounds a little screwy, but nothing else I tried seemed to do — is to have a SAS ‘Main’ program $include (this is using the SAS Macro Language — yet another tool) both SAS data read-in programs and IML scripts. The $include causes the called program to execute. The data programs cause files to be opened and SAS data sets to be created. The other programs are a little odd: they invoke IML, define an IML module (in IML jargon, a program is called a module) and then store that module in a temporary file before exiting IML. When all this has been done, the main program calls IML, loads all the modules from the temporary files, and then runs the main IML module which, in turn, can run other modules.
To move data from SAS data sets to IML takes only a few lines of code, and it takes even fewer to transform IML data objects into SAS data sets. You can get fancier, plucking or placing subsets on either side of the wall, but I haven’t needed to do that yet. In any case, we are now in IML and we have our data to chew on. What is IML really like?
IML Up Close¶
The M in IML pretty much tells you the whole story: it relentlessly deals with matrices. As nearly as I can tell, a one-number IML variable is a one-row, one-column matrix. The corollary is that you are stuck in two dimensions. What might be a three- or four-dimension array in APL has to be flattened into two dimensions, and this means that the programmer has to do a lot of careful indexing work. Clearly, you are in good shape to the extent that your data objects have fixed coordinates — you aren’t doing a lot of reshaping, even though reshaping is permitted.
Here is an example from my demographic system. In APL and J, I use three- or four-dimensional arrays deal with population data. The dimensions are sex, age, time, and (sometimes) areas: the system being discussed uses the first three. Therefore, the design decision is whether to create a single in-memory matrix for data caching or make two matrices. The two-matrix option would have an age-time matrix for males and another for females. The one-matrix alternative has males by age as rows, followed by females by age as more rows, with time as columns. Because I tend to dislike variable clutter, I opted for the one-matrix alternative, even though it slightly complicates data indexing. I call the matrix poply.
Now let’s say I’m in the midst of the process of moving a population from one year to the next. I want to extract the start-year population and do something with it, so I write
SAS is case-insensitive, and names cannot be more than eight characters long. shape is an IML function that does about what the APL rho and the J $ do. Note that all the action is specified within the parentheses. The first-position item indicates the data being used. Here, it is the column of poply called yrcnt, which is a year counter; square brackets enclose matrix position information which are in row, column order with a comma separating the specifications. The next two items — the numbers 2 and 86 — are the row and column dimensions of the new matrix: in this case, two sexes and 86 ages (0, 1, 2, … , 84, 85+). The final semicolon is IML’s end-of-statement marker. So IML creates a 2-row, 86-column matrix and places the 172 numbers from a column of poply into it.
Now I need to age the population. This entails adding the two highest age groups and moving all the rest one notch older. My IML code does it this way:
Since I am creating a new variable, the first item in the shape expression is 0; I’m filling the matrix with zeros for starters. Then I drop in the data from the first 84 columns of popt1, but beginning at the second column of agedpop. As you might have guessed, the colon serves as an indexer — the number to the left is the start position, the number to the right is the end position. The final line adds the last two columns in popt1 and places the result in the last column of agedpop. Note the difference in indexing: columns 85 and 86 are designated by curly brackets. The plus sign to the left of the comma indicates addition within rows. Also note that all this is surrounded by parentheses. This was done because I needed to rotate the result (now one row by two columns) into two rows by one column — the rotation operator is the front-quote mark to the right, just before the semicolon.
I ought to mention that IML does not use the right-to-left precedence scheme used in APL and J; it basically follows conventional precedence practice. This caused me some trouble because IML is just enough like APL to let me drop my guard. Worse, the calculations executed, but some results were odd. These had to be detected and the cause corrected.
Another irksome IML shortcoming is its string handling. Here is code I used to take a column of years, convert the numbers to characters, and place the letter ‘Y’ in front of each in order to create a set of labels. I hope I simply overlooked some documentation on the easy way to do this; at any rate, this illustrates what you might have to deal with:
use Script var{endyear};
read all var _num_ into years;
mxyrs=nrow(years);
tmp1=cshape('Y',mxyrs,1,1);
tmp2=char(years,4);
tmp2=cshape(tmp2,mxyrs,4,1);
tmp3=tmp1||tmp2;
yearcols=rowcat(tmp3);
The first two lines bring a column of data (from the forecast ‘script’ file) into IML from the SAS data set Script. Next, I use the IML function nrow to get the number of rows in the fetched column. tmp1 is a 1-column matrix of ‘Y’s; note that, since I am using character data, I must use function cshape rather than shape to create the matrix. The following line uses char to transform the years into characters with a maximum of four. Now things get ‘interesting’. I cshape the tmp2 matrix into a matrix of mxyrs rows and four columns of one character width! (IML, unlike APL, does not seem to like treating character data as a big slab; you deal with columns that contain a specified number of characters each.) Next, I use the horizontal concatenation operator, ||, to mate the ‘Y’s matrix to the years matrix; now our matrix has five columns. The final step collapses the five columns into a single column of five-character data using the function rowcat. I really hope all of the above is a huge mistake — on my part, not SAS’s.
I think this gives you the flavour of IML. You manipulate matrices using either operators or keyword functions as appropriate. Compared to APL and J, IML seems crippled, but it is still a lot nicer than, say, Visual Basic for number handling. I will soon be writing a J version of the system to validate results and will be able to make comments on execution speed once both are running.