Faster File Input in TSO
Vector 4:1, page 99
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 99
After talking to various people at the February BAA meeting it is obvious that the following notes need circulation to a wider audience. One of the biggest problems with APL is the large amount of CPU time required to read a single record from a QSAM file into the workspace through auxiliary processor 111. This paper sets out to find a way of speeding this process up and at the same time make use of APL’s array handling facilities.
The standard method of file processing in FORTRAN, COBOL and the like is as follows:
Read a single record
Process it
Repeat until end of file
which works fine because these languages are scalar oriented in that they can only work on one value at a time and they are compiled so that looping code can run very fast. In APL we prefer to save our values until we have either got all of the file into the workspace or, if the file is too big, as much of it as possible to reduce the looping process to the minimum. If we use AP111 as described in the APL2 System Services Reference manual we get:
Read a single record
Read another single record
. . . etc until we can’t get any more into the workspace
Process what we’ve got
Repeat until end of file
which still contains a loop that is executed once for each record in the file.
When I first wrote code like this and ran it, I seemed to be waiting for ages for the function to finish running so I decided to time it. I discovered that in the following program the loop takes about 3 milliseconds to execute and any other processing included within this loop took much less time than the file read operation.
∇TEST;I;R
...
⍝ Share variable R with AP111
...
[n-1] I←1
[n] LABEL:X←R
[n+1] →(1000≥I←I+1)/LABEL
...
∇
When I tested this function on files with different logical record lengths I discovered something that seemed strange to me at the time and is the key to what follows: it doesn’t matter how long the record is; it always takes AP111 about 3 milliseconds to read it. (The first test was of the average of 1000 reads which could have concealed wide variations, but timing individual record read operations showed that the variations were minor – 3 milliseconds is reasonably constant).
Having discovered that it takes just as long to read 10 bytes into the workspace as it does to read 10K bytes, my next thought was ‘How do I read multiple records at a time into the workspace?’ The obvious way is to make the logical record length of the data the size of the space available for processing. Although this is great in theory, it is impossible in practice for various reasons, viz:
- In MVS, the largest block size (and hence the largest record size for fixed length records) is 32600 bytes.
- You must have enough sharespace allocated to hold the record when it is being read from the file and passed to APL. This comes out of your available region size, effectively reducing the size of your active workspace.
- The owner of the data might object to your reformatting of his/her dataset and the DP department might object to your request for a couple of hundred megabytes to duplicate a dataset in your desired format.
We can get around these problems by using the Great British Compromise. If we override the values in the dataset control block (DSCB) then we do not need to reformat or copy the data. However, we cannot change the block size so we might not read as much data at a time as we could. We could consider various options:
- Specify the logical record length equal to the block size. Objections:
- with fixed length records this will usually give a wrong-length record error on the last block.
- with variable length records you must make the LRECL 4 bytes less than the blocksize to allow for the block-length prefix.
- Our first thought for getting round these objections was to specify the record format as VB (varying-length records, blocked). This doesn’t work with fixed length record files (you get an 0C4 abend) because the system is looking for a block-length prefix and of course it doesn’t find one.
- The final chance is to specify the record format as U (undefined). This works, for both fixed and varying-length records, although you have to know the type of records in the file in advance since processing is different for fixed length records and varying length records. However you can write a generalised program or a set of programs to do the work for you and if your system has the APL DDI and APL DSI commands everything can be automated.
In order to override the record format on the DSCB you use one of the following sequences of commands via AP100:
In JCL you code
For TSO/E you can issue the following command
Otherwise for TSO you will have to use the following sequence
If the dataset you are trying to read was set up with blocksize 0 then you will also have to specify the blocksize. You can do this even if the blocksize is not defined as zero, as long as the new blocksize is greater than or equal to the old blocksize. This is because the value you specify tells AP111 how much buffer space to use, and extra buffer space is immaterial even though it is never used. Although TSO/E should accept the keyword combination SHR BLKSIZE(value) on the ALLOC command, in fact this does not allow ALLOC to complete properly. Thus in TSO/E you code:
FREE DD(attrname)
ATTR attrname BLKSIZE(32600) RECFM(U)
ALLOC DD(ddname) DS(dsname) SHR USING(attrname)
If you haven’t got TSO/E you use the same code as before, replacing the ATTR command with the new one just shown. Note that we specify the maximum block size that MVS can handle here; this means that our code will work for any dataset, although we might have an awful lot of unused buffer space. Of course, if you have it, you could use the APL DSI command to determine the DSCB values and use those. APL DSI is also useful for determining the record format as mentioned above, so that you can deal with the data appropriately – see later on for amplification of this.
Of course, once you have got the data into the workspace it is up to you to process it in whatever way is appropriate for your application.
It should be emphasised that this method of deceiving the system into believing that you are reading a file with different attributes from those declared in the data control block is not peculiar to APL but is a feature of the operating system. It is a feature that IBM are not exactly over-generous with references to in their manuals. I have looked through several of them and I have only come up with the following references, for those of you who want to develop this line of thought.
PL/1 Optimising Compiler Programmer’s Guide, Chapter 4 (Data Sets and Files), pp 117-118
OS/VS2 MVS JCL pp 203-215 (The DD statement, DCB parameter)
Another Useful Idea¶
Try the following. You have a dataset with the LRECL=80, BLKSIZE=6400, RECFM=FB. Allocate it in such a way as your program thinks it has LRECL=6400 (not difficult; override the DCB parameters in a similar manner to that shown above). Then open the dataset using AP210. What do you think you can do? . . That’s right, you can read individual blocks in random order by specifying their relative record numbers in the control variable, and, what’s more, you can update them in place. This is especially useful if your dataset is big (with a few hundred thousand records, for example) and you have only one or two amendments to make. Of course if you know you are going to want to do this you will pad out the last block of the dataset with blanks to the full 6400 bytes, won’t you (because you can’t access it if you don’t).