C Source Code Modification using APL
Vector 13:4, page 89
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 89
Introduction¶
Modification of C source code using APL is not unheard of. In this case, the problem was to update a C header file using a C MAP file.
MAPFILE.MAP contains a symbol table with variable names and their respective addresses. The header file HFILE.H contains #define pre-processor commands for variable names and respective addresses. New addresses in MAPFILE.MAP had to replace the corresponding old addresses in HFILE.H.
Easy (you think)! Read the files into 2 variables, then write a couple of one liners to find occurrences of the variable names of MAPFILE.MAP in HFILE.H, and replace the respective addresses.
But you only have APL*PLUS PC which only uses 640K. Still easy!
Ahh, but the files have lots of duplicated stuff, and are vectors containing carriage returns — slightly harder! When you process the huge variables containing the files using your one liners, you get the dreaded WS FULL. Optimising memory still gives WS FULL. Not so easy after all!!!
The answer lies in solving the problem using a more conventional programming approach, as you would using a language such as C. Basically, you have to process the files line by line. Granted, this means using loops, hence slower execution in APL, but you’ll have bags of memory to spare for a decent front end. Also, few lines of code are required.
The File Structure¶
The symbol table map file looks like this:
[listed by asc name] [listed by asc address]
[address] [var name] [address] [var name]
00811321 _ER1 00811321 _ER1
0081322f _stack 00812532 _UDIV30
00812532 _UDIV30 0081322f _stack
The header file looks like this:
As can be seen, the map file has everything duplicated. Both files are one long vector.
How the System Works¶
The processing method is very straightforward.
LOOP:
- Read a line from MAPFILE.MAP.
- Extract the variable address and name.
- Search all lines in HFILE.H for the variable name extracted.
- Replace original respective HFILE address with the new MAPFILE one.
→LOOP
This is actually a little simplified because, as you’ll see later, this isn’t very fast, and there are enhancements to make it 5 times faster.
Reading a Line¶
You can’t simply read a "line" from a native file as you would using a text editor. A line (usually) contains text followed by a carriage return. You have to find the end-of-line character, ⎕TCNL, (Return Key). Let’s assume that the maximum length of a line is no longer than 95 characters. Reading 100 characters would guarantee reading a whole line. From the 100 characters, find the first occurrence of ⎕TCNL, then take all the characters before it and, hey presto, you have a line of text. Then you would repeat the process starting from the position of the previously located ⎕TCNL to get the next line. The function GETLINE does precisely this, and you can specify the file tie number, the length of the line to read, and where to start reading.
Extracting the Variables from MAPFILE.MAP¶
Remember that MAPFILE.MAP contains two columns of duplicated variable names and addresses, so we only need bother with one of them, the first one. When a line is read, we need to extract the first address and name. Fortunately, all addresses in MAPFILE.MAP are 8 characters long, so taking 8 characters is easy. Then dropping the address (and a trailing space) leaves the variable name followed by at least one space. So, find that space and take everything before it. The READVARS function does this and returns a string with an address catenated to a variable name.
Replacing Old Addresses with New Ones¶
Armed with the address name, all we have to do is use ⎕SS to search for an occurrence of that variable name in the header HFILE.H, using the GETLINE function to retrieve one line at a time from HFILE.H. Note that this time, the lines are shorter, so only 80 characters need to be read to guarantee a whole line read. If a line does contain a matching variable name, find the position in the line of the old address, and replace it with the new address. Then simply use ⎕NREPLACE to overwrite the old line with the new one, hence ending up with an updated variable address. See the SYSTEM function, lines 25 to 27.
The SYSTEM function is basically a simple loop which reads new variable addresses and replaces the old ones. It calculates the position of the next line from the length of the line currently returned (+2 to compensate for ⎕IO=0 and to skip the ⎕TCNL).
More Speed¶
Originally, when the above method was implemented using a MAPFILE.MAP of 35k, and a 70k HFILE.H, the updating of all addresses took over 5 minutes, and this was using a VX motherboard with a P120 CPU!
Obviously, this is too slow. There are ways to speed it up using various search methods, however a quick solution is to only search the HFILE for lines which have not been updated. Basically, once a variable address has been updated, make sure that line is not in the search in future. A simple way of achieving this is to store the positions of ALL the lines in HFILE.H in a variable (called LINES). Lines 7 to 11 in the SYSTEM function do this. Use this list to read one line at a time. Then, when you update a line in HFILE.H, simply remove that line position from the list of lines in the LINES variable. The pre-searching of line positions is a small time overhead to pay for the overall decrease in processing time, which turns out to be just over 1 minute, hence giving a 400% improvement in speed.
Summary¶
Although this is not the fastest method for file processing, it requires very little memory, which is a precious resource if you’re using APL*PLUS PC to process large files. You can speed things up by refining the methods, but you will usually have to trade speed in order to gain memory. Of course, you may be wondering why bother doing it this way? Why not simply use an APL interpreter that supports huge workspaces? Well, this is simply an exercise to show that there are ways around the limitations in APL*PLUS PC, methods which require a slightly different approach to the usual "one liner" APL solution.
Function Listings¶
∇ SYSTEM;CT;LINES;MPOS;HPOS;ADR;VAR;VARS;LINE
[1] ⎕IO←0
[2] 'HFILE.H' ⎕NTIE ¯99
[3] 'MAPFILE.MAP' ⎕NTIE ¯100
[4] MPOS←0 ⋄ LINE←' '
[5]
[6] ⍝ Find all lines in header, storing index for speed later.
[7] LINES←0 ⋄ HPOS←0
[8] LR0:LINE←¯99 GETLINE(80,HPOS)
[9] →(0=⍴LINE)/LOOP1
[10] HPOS←HPOS+2+⍴LINE
[11] LINES←LINES,HPOS ⋄ →LR0
[12]
[13] ⍝ Main processing loop
[14] LOOP1:LINE←¯100 GETLINE(100,MPOS) ⍝ Get line from map file.
[15] →(0=⍴LINE)/END
[16] MPOS←MPOS+2+⍴LINE ⍝ Increment position in map file.
[17] VARS←READVARS LINE ⋄ ⍝ Extract variable from map file.
[18] ADR←8↑VARS ⋄ VAR←8↓VARS
[19]
[20] ⍝ Loop to replace old variable addresses with new.
[21] CT←0 ⋄ HPOS←LINES[CT]
[22] LR1:LINE←¯99 GETLINE(80,HPOS) ⍝ Get line from header file.
[23] →(0=⍴LINE)/LOOP1
[24] ⍝ If line contains matching variable, update address.
[25] ⍎(~((8↓(((8↓LINE)⍳' ')+8)↑LINE)≡VAR))/'HPOS←LINES[CT←CT+1] ⋄ →LR1'
[26] LINE[((((LINE ⎕SS '0x')/⍳⍴LINE)+2)+⍳8)]←ADR
[27] LINE ⎕NREPLACE(¯99,HPOS)
[28] ⍝ Remove the line number updated, to skip in future.
[29] LINES←(HPOS≠LINES)/LINES
[30] →LOOP1
[31] END:⎕NUNTIE ¯99 ¯100 ⋄ ⎕IO←1
∇
∇ R←file GETLINE params;L
[1] ⍝ Read x chars from last pos. If index error, read max possible
[2] ⍝ Return line read up to first encountered carriage return.
[3] ⍝ SYSTEM fn will work out last pos from length of line returned.
[4] ⍝ params = x chars, last pos
[5] ⍝ eg. ¯100 GETLINE 100 0 = read NTIE ¯100, 100 chars, start at 0.
[6] ⍝ Choose realistic value for params[1] ie larger than longest line.
[7]
[8] →(((⎕NSIZE file)-(1↓params))≥(1↑params))/NORMAL
[9]
[10] ⍝ Do line below if you can't read x chars near end of file
[11] L←⎕NREAD(file,82,((⎕NSIZE file)-(1↑params)),(1↑params)) ⋄ →LCONT
[12]
[13] NORMAL:L←⎕NREAD(file,82,params)
[14] LCONT:L←(1↑((L=⎕TCNL)/⍳⍴L))↑L
[15] R←L
∇
∇ R←READVARS line;adr;var
[1] ⍝ Read the first address in the line with corresponding variable.
[2] ⍝ R = 8 char address catenated to arb. length variable name.
[3] adr←8↑line
[4] var←9↓line ⋄ var←(var⍳' ')↑var
[5] ⍎('_'=(1↑var))/'var←1↓var'
[6] R←adr,var
∇