Hackers’ Corner: Fast Filing with ∆pack

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 125

Health Warning¶

What follows is highly specific to APL*PLUS/PC; it won’t even do you any good under PLUS II, which doesn’t have the IPSA-style package utilities provided.

Motivation¶

For many years, my APL file system has been built around two almost trivial routines called DUMP and GET. These simple take a named list of APL objects (variables or functions) and sling them on file. The only subtlety is that GET will optionally pull back variables under a different name if you ask it to:

RC←'MYFILE' DUMP '∆CODE,∆DESCR,any∆fun'
RC←GET 'MYFILE'                   ⍝ Simply retrieves the whole file
RC←'MYFILE' GET '∆DESCR'          ⍝ Selective retrieve
RC←'MYFILE' GET 'CHECK←∆CODE'     ⍝ ... with a change of name

This simple approach has the virtue of almost total portability (it even worked under VSPC using sequential text files), but when implemented on APL*PLUS/PC using the obvious strategy of one object per component (co-incidentally the format turns out to be compatible with Statgraphics files, which is quite useful) it is horribly slow for collections of lots of small objects.

      ⍴NL
112 12
      RC←'SCRATCH' DUMP NL   ⍝ 11.2 sec to dump 20480 bytes
      RC←GET 'SCRATCH'       ⍝ 1.3 sec to retrieve whole file

I had toyed with a variety of strategies for improving matters, and the best one I had come up with was to ravel like objects end to end (recording ranks and shapes in a header table) and then write out a maximum of four components. This speeds the writing process up quite a lot, but of course files are read more than they are written, and the slight reduction in reading speed brought about by all that indexing and reshaping was enough to put me off. So … there matters rested until STSC quietly included a bunch of ASMFNS to emulate the Sharp Package concepts.

Essentially, all they do is take any list of objects (in ⎕NL format) and return a magic integer vector with all the objects hidden in it. The inverse process just takes the vector and disperses it into the workspace. Now you may have noticed that the APL*PLUS/PC native file functions are absurdly fast, and that it is quite legal to write out an integer vector at two bytes per element … so without further ado:

    ∇ rC←fI PDUMP vL;⍙⍙pk;p;⎕ELX
[1]       ⍝ Package objects in vL and write as native file
[2]       ⍝ Note blocked write ... a good idea for big objects.
[3]        →(2=⍴⍴vL)↑Mat ⋄ vL←(~p←vL∊',/;: ')/vL←,vL,'/'
[4]        vL←(⍴p)⍴(,p←p∘.≥⍳⌈/0,p←(p≠0)/p←p-1+0,¯1↓p←p/⍳⍴p)\vL
[5]   Mat:⍙⍙pk←∆pack vL
[6]       ⍝ Now create/tie file, clear down and swipe out the data
[7]   Tie:⎕ELX←'fI ⎕NCREATE ¯2 ⋄ →Go'
[8]        fI ⎕NTIE ¯2 ⋄ 0 ⎕NRESIZE ¯2
[9]    Go:⎕ELX←'⎕DM' ⋄ ⍙⍙pk←,⍙⍙pk
[10] Loop:((8000⌊⍴⍙⍙pk)↑⍙⍙pk)⎕NAPPEND ¯2
[11]       →(0≠⍴⍙⍙pk←8000↓⍙⍙pk)↑Loop
[12] Done:⎕NUNTIE ¯2 ⋄ rC←0
    ∇

    ∇ rC←PLOAD fI;⎕ELX;lEN;oFF;iN;⍙⍙pk
[1]       ⍝ RE-LOAD PACKED FILE CONSTITUENT VARIABLES.
[2]       ⍝ Note oddity in shape: vectors are acceptable >32768!!
[3]        rC←6 ⋄ ⎕ELX←'→0'
[4]        ⎕NUNTIE ⎕NNUMS ⋄ fI ⎕NTIE ¯2
[5]        lEN←⌊0.5×⎕NSIZE ¯2 ⋄ oFF←0
[6]        ⍙⍙pk←lEN⍴0
[7]        rC←99 ⋄ ⎕ELX←'⎕ERROR ''UTILITY '',⎕DM'
[8]       ⍝Now drop in file contents in 4000-byte segments ...
[9]  Loop:iN←⎕NREAD ¯2 163 ,(4000⌊lEN-oFF),oFF×2
[10]       ⍙⍙pk[oFF+⍳4000⌊lEN-oFF]←iN
[11]       →(lEN≤4000+oFF)↑Done
[12]       oFF←oFF+4000 ⋄ →Loop
[13] Done:⎕NUNTIE ¯2
[14]      ⍝ Split em out ......
[15]       →(32768≤⍴⍙⍙pk)↑Def ⋄ ⍙⍙pk←(1,lEN)⍴⍙⍙pk
[16]  Def:∆pdef ⍙⍙pk ⋄ rC←0
    ∇

      ⍴NL
112 12                          ⍝ Same data as used above (honest)
      RC←'JUNK.TMP' PDUMP NL    ⍝ 0.45 sec to dump 10342 bytes
      RC←PLOAD 'JUNK.TMP'       ⍝ 0.44 sec to retrieve whole file

Time this yourself … I reckon it beats the socks off component files. Oddly, it also saves rather a lot of disk space.