APL and J: Working Together, Yet Separately

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 67

Other articles have described how APL can access J to have certain specialised tasks performed—for instance, doing extended-precision calculating [1]. Here, I present an example of how APL and J can be used in different aspects or phases of the same project.

The general idea is to exploit some strengths of each language while minimising exposure to weaknesses. This is a cautious strategy for the APLer who wishes to enter (tentatively) the world of J without leaving the safety of having APL remain the basis for mission-critical or enterprise-critical tasks.

Background¶

It is assumed the reader is acquainted with APL and its operation under Windows in the guise of either Dyalog APL/W or APL2000’s APL+WIN. A cursory knowledge of J is also assumed, though ignorance of J should not be a great impediment for understanding what follows: examples of J code are not presented.

Let me start by sketching how I have dealt with APL in the broad sense over the past 15 years. Some (most?) of you might find my approach strange, but I hope you will see how my use of J fits into the scheme.

My behaviour pattern—especially in the last 10 years—is to flit from APL vendor to APL vendor. I started with STSC’s APL*Plus for the IBM PC (version 2.6 at first, then a long while with 3.0). Eventually, I switched to APL*Plus II. Favourable reviews in Vector, plus a good price, led me to get APL2/PC, though it turned out to be fragile. Three years ago, I felt I needed to run systems under Windows, so I got APL*Plus III (and Iverson Software’s Windows version of APL, but it was too slow for serious data production). I bought Dyalog APL/W (version 8 for Windows 95) about a year ago, and keep up to date in APL+WIN as well. I have been using J for about two years.

All this thrashing around is meaningful to this report because it seems like almost every time I get a new APL, I rewrite my main demographic forecasting system. (I do demographic, household, and income forecasts, largely for automobile industry clients.) I find it cleaner to go to a different APL when it is time for a major re-write. Perhaps this lessens the chance of using old, patched-up code, even though key algorithms are ported from system to system.

I should explain that I have never used APL workspaces for anything but throwaway systems. I have each APL function residing in a separate file, and my systems call in slugs of code according to directory paths fed to file-reading routines. No doubt this imposes some inconveniences, but I think it makes for safer, cleaner systems. Porting programs from one vendor’s APL to another’s is a matter of using an AV index conversion routine. Obviously, this is only for data-processing code. User interfaces have to be written from scratch in the new system.

As this is written, I have nearly completed a new system in APL/W—the previous system was in APL+WIN. The new system is largely batch-oriented in that it will project data for a large number of countries in one swoop, whereas the old system had each nation as a separate operator task. This change meant a switch to using large, ASCII tables (in *.txt format) containing key parameters or control data such as annual births by nation. And it so happens I find J handier than APL when dealing with *.txt files.

Thanks to the newly perceived need to deal with ASCII files and the availability of an APL-like language that easily works with such data, I find myself using both APL and J for the same general project.

Where APL is Used¶

I suggested above that caution can be a watchword to many APLers. As Adrian Smith forcefully stated in his APL 91 presentation [2], it is prudent not to be locked into one APL. You should assume that you will switch interpreters or vendors in the future, and plan your software systems and coding practices accordingly. His Causeway graphical interface software, for instance, grew from a desire to avoid being tied to one vendor for interface reasons.

I agree with his philosophy, but must admit I lack the discipline to carry it out rigorously. When I wrote my core systems in APL2/PC, I took to using the ‘Squad’ index in order to avoid writing a lot of executable strings containing ‘;;;’ blocs sized to fit arrays of variable rank. The Squad was fine, aside from the fact that Squad-expressions tended to be rather long and perhaps a little hard to read. Most other APLs lack the Squad, so when I moved from APL2, I wound up writing a Squad cover function, and of course it was peppered with ‘;;;’s.

Safety in numbers¶

To return to the point, a really prudent APLer will not (yet) abandon APL for J. I know of five APLs for personal computers, while there is only one J vendor. The ‘safety in numbers’ concept, then, is one reason for sticking with APL for critical systems. Yes, J can do some tricks APL cannot, but this does not outweigh the risk of betting everything on one vendor. Nor does any present J speed advantage: if the speed difference were an order of magnitude—or even half that—then a switch might be worth it for speed-critical tasks.

Fully-featured interface¶

Another APL advantage, at least for now, has to do with the Windows interface. Both APL/W and APL+WIN have more fully-featured interfaces than J. For example, J does not offer MDI (scaleable, overlapping forms such as you find in Microsoft Word or Microsoft Excel). Nor, as this is written, does J have its own grid-object or spreadsheet-interface object. Yes, J can be a DLL server for C++ or Visual Basic (as can some APLs). But dual-language programming involves a learning curve and potential complexities that can be avoided for in-house systems such as those I normally write. To put it differently, it is fast and convenient to write the entire system, including the Windows part, in an APL environment, especially if the interface really needs features J’s Windows Driver lacks.

Where J is used¶

Well, if APL is so wonderful, why use J at all? Recall that I mentioned J was handy for dealing with *.txt files. It also has other features that make it useful as an adjunct to APL. Of course, APL can do most things J can do, including the work described below. But using J can be considered yet another prudence ploy. Should J really ‘take off’ or should APL ‘hit the wall’, the prudent system builder will have experience writing in J as well as a set of useful J programs!

I like examples that are concrete rather than theoretical, so let me describe how I use J in conjunction with my core APL-based systems.

Defining a uniform baseline¶

My original international demographic systems dealt with less than two dozen nations, most of them of the advanced industrial sort with lots of detailed data. Today, I need to project each country in the world with population of a million or more, and might have to include even smaller nations in the future. From the start, I had to deal with the problem of data variation from country to country, but the move to ever-smaller nations (often with sketchier and sketchier data) has forced me to keep revising the software system. The problem is this: I want to offer uniform and fairly detailed data to clients, but the source data are anything but uniform, and often are not very detailed. I have a further need to keep estimated (historical, non-forecast period) data close to actual values—plugging in actual data directly, if possible. And I have to estimate or fill in details that are absent. What this boils down to is that I must ‘calibrate’ data from many nations so that a single, baseline set emerges. The baseline set is uniform, and that is what the forecast software is designed to use.

I presently use J in the calibration phase. APL programs do the actual calibration computing. But much of the computing is based on input from a number of tables, and it is J, along with Microsoft Excel, that deals with these tables.

Dealing with text files from Excel¶

The tables are created and, for the most part, maintained using Excel; J performs critical ancillary tasks that will be described. Excel is for data entry. The spreadsheet tables are saved as tab-delimited text files (Excel’s *.txt format noted above). Most tables contain a single time-series such as total births per year. Thus, the table will consist of perhaps 230 rows (one per country) and as many columns as there are years in the time-series. Table topics include total population, total births, total deaths, household totals from censuses, Gross Domestic Product (GDP), Consumer Price Indexes (CPI), and monetary exchange ratios.

In some cases, I have more than one table for the same subject: I have a table of 1 July population totals from the United Nations, and another table with official population estimates or population register tallies and the associated dates. I standardise my populations as of 1 January. Official populations are assumed to take precedence over the more comprehensive UN data (even though many UN data are from official sources). So, I need to blend the tables so that I wind up with population totals for 1 January. J routines are used to do this blending, and the resulting table—a key control for both calibration and the projection run—is saved to file.

There are several operational benefits from this practice. For one thing, data from separate sources can remain separate: this makes it easier to update material or to check for data entry errors. Secondly, the blending program takes care of all the reference-date checking and needed interpolation, leaving the source tables with pure content for the tasks just mentioned. Thirdly, once a change is posted to one of the source tables, it takes only a couple of minutes to fire up J and create an updated blended table.

Where J feels more at home¶

What follows is purely subjective on my part, though it might be based on technical realities. To me, J seems to feel crisper, more at ease dealing with *.txt data than APL. ASCII files are read into J using fread, and the content is boxed Excel cell by Excel cell using clipunfmt. These are standard J utility programs. Once boxed, the data are generally simple to deal with in further processing. In the case cited in the previous paragraph, the blended table is boxed and then the utility clipfmt transforms it into a tab-delimited string with end-of-line characters (in other words, *.txt format!) before it gets written to file using fwrite.

One reason why APL might seem a shade clunkier is that I wrote my own format/unformat routines because, through ignorance or oversight, I do not know of professionally-written routines that do the deeds well. Still, once the data are boxed, to me J still feels ‘more at home’ than does APL with nests when it chews on table data.

My J forecast system-related scripts are nested under a top-level directory labelled DataManagement. This is coequal with the top directory for the APL forecast system itself. Besides table-blending, J is used to splice and unsplice historical and projected control numbers, to rummage database directories to inventory files by nation and data subject, and to inspect certain kinds of data in the database directory.

This last task involves more elaborate Windows work than simply selecting from lists: graphs are shown. One inspection routine lets me select a nation, then a data subject, and finally a file containing that data for a single observation. An example would be the population of France by sex and age group from the 1982 census. Once the file has been read, I can query the data source, any comments I might have entered (such as if there were people of unreported age), the age groups represented in the data, and a couple other items of potential interest. In the case of population by age-sex, I can have J draw a population pyramid—a standard demographic graph which is simply back-to-back bar graphs with the age dimension running vertically.

Finally, I have J do certain off-line tasks such as prepare estimates of age-sex-specific household headship rates using a very simple Fuzzy procedure.

Comments¶

From the above, the division of labour I have imposed on APL and J should be fairly clear. J works with Excel-generated files and Windows directories (or ‘folders’ in current parlance). So far as APL-created files are concerned, J is used only for inspection of content.

APL creates, modifies, and adds data to certain classes of files that contain only ASCII characters, but are crafted so that the file content is expressed in strand notation for easy conversion to a nested array. (J can read these files and, with a little effort, box the content.) APL also reads *.txt files and converts the content into nested arrays. These files are data controls of the type described above as well as ‘forecast script’ files. Let me emphasise that APL’s role is simply to read and convert these files to nested arrays—it is not used to modify them, even though it is capable of doing so.

If, in the future, I switch APLs again, the J-based software will be completely unaffected, so long as system revision does not change the content of control data files, scripts, and so forth. Actually, a system rewrite usually means that some of these will be changed, some will be abandoned, and new files will be added. Still, the net gain ought to be that less new APL code would be needed than if the entire system, including Windows features, were changed from APL/W back to APL+WIN.

J is also being used in my sub-national forecasting system, currently in prototype form. I am using J here so that I will have code that parallels my main system code just in case I am compelled to abandon APL; the sub-national system can be revised into a nation-level system fairly easily. However, given J’s still-limited Windows programming capabilities, I am finding myself in a situation complementary to that described above. I really do need MDI and grid objects for certain data-support programs involving county or provincial numbers, so I will need APL to support J rather than having J support APL! Either that or turn to dual-language programming.

Conclusion¶

Like may APLers, I was initially hostile to J. Truth be told, I am still not very fond if it, though I respect it more than I did. It is ugly to look at, and might well be too large. Documentation and learning tools are improving, but still sub-par, on average.

On the other hand, J seems to be a good data processing language. It is handier than APL in that I can call a J script from Windows Explorer using Notepad and make changes without having to go through the bother of firing up J: to fix APL code, you must be in APL. And, as described above, it is especially handy for working with spreadsheet-based tables and other ASCII text data.

If, having read my example, you can see where something like this might work for you, I heartily encourage you to try it. You don’t have to stray too far from APL to do so; for instance, I don’t get into particularly J-ish business such as writing tacit functions. But be warned that J is not simply APL without the symbols. In spite of claims that it is just a dialect of APL, I find it bears about the same relationship to APL as German does to Swedish.

References¶

1. Chris Burke, The J DLL and Extended Integers, Vector, Vol. 13, No. 1, pp. 149-53, July 1996.
2. Adrian Smith, Supply-Chain Management at Rowntree: Critical Success Factors for APL, APL Quote Quad, Vol. 21, No. 4, pp. 301-306, August 1991.