Tools, Part 2. Basics too.

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 122

In the following article I use terms specific to our trade. You won’t find them in the dictionary but I assume the reader is familiar with words such as ‘monad’, ‘global’ (as a noun) and ‘default’ (verb). I use the terms ‘nested’ (as in ‘nested array’), ‘enclosed’ or ‘boxed’ interchangeably. I also use quotes and angle brackets to help determine the type of the object I am referring to. ‘Quotes’ denote a variable or workspace and <angle brackets> refer to a function/operator or file. Often, the context is sufficient to remove ambiguities. Italicized words have a special meaning and often precede their definition. Optional items are shown within [brackets].

Introduction¶

When I started programming I confronted problems that needed deep investigation. Very often I was the given code and little information. I needed to search code for clues. The tools provided were crude and limited. This is when I started thinking about having my own function to search other functions for a given string.

This is the second of a series of articles on tools in APL.

A Function to Search Code¶

This function is one of the first of my tools. It was first written for APL/PC but was quickly ported to other platforms. It allows looking at code and reporting where specific strings are found. I wrote it when there was no such thing in the public domain or what was available was of limited use. It was written when nested arrays did not exist but was eventually adapted for them and the newer types (packages, namespaces, overlays…).

SEARCH utility¶

Keywords: STRING LOOKUP, STRING SEARCH, ARGUMENT, DELIMITER

This function will show where a series of strings occur in APL objects found in a workspace or in a file.

The evolution of software¶

This is another good example of code that evolved from simple to complex utility, something we should all have seen by now (think M$).

An interesting thing, here, is the way the function works, on top of what it does. Let’s see what it does first. I’ll describe it just like it was built. I’ll show how the prototype became the program.

The idea is simple: you take the visual representation of a function and do a search a la ⎕SS on it. For each hit you perform an action like showing the name of the function where the string is found. For example, assuming that functions <fn1> and <plus> exist in the workspace, doing

fss 'A+B'
fn1 plus

Would do it. Function <fss> uses ⎕VR to look into each function in the workspace and reports where the string given as right argument is found.

This is not enough. An improvement is to show the line where the string is found:

fss 'A+B'
fn1[3]  X←A+B=X
fn1[7]  R←A+B+C
plus[2]  Z←A+B

Better yet, use the left argument to specify functions to be searched and show the exact location of the string:

'fn1' fss 'A+B'
   ∇ fn1 (2 occurrences)
[3] X←A+B=X
      ∧
[7] R←A+B+C
      ∧

So far nothing special. <fss> is about 10 statements including comments (☺) A small improvement is to allow multiple strings to be searched. With nested strings we could do ∨/strings⍷¨code but I originally didn’t have that luxury. A delimited string does the trick (the first character is the delimiter):

'fn1 show' fss '/A+B/X'
   ∇ fn1 (4 occurrences)
[3] X←A+B=X
    ∧ ∧   ∧
[7] R←A+B+C
      ∧
   ∇ show (6 occurrences)

[2] 'X=',⍕XX←A+B ⍝ XX is the answer
     ∧    ∧∧ ∧      ∧∧

So far so good. There were a few problems like searching for '[1]' which would report a hit for every function. This meant I had to recognize the decorators and mask them out. I found it simpler to use the ravel of ⎕CR of the function instead. Later, when ⍷ appeared it proved correct.

A new problem was that I was now looking at larger and larger workspaces. There were some specific cases that I needed covered:

• the ability to mask out comments
• the ability to mask out text
• the ability to mask out code
• the ability to perform syntactic searches

The first three cases are all related: on a line of code the comments follow the first lamp character (⍝) not in text. We must know where the text is first. Text is everything between 2 quotes, quotes included, not in comments. Code is everything else¹. So, the expressions

T←T∨≠\T←line='''' ⍝ strings and quotes
Comments←∨\T<line='⍝'

tell us where the comments are.

Text←T>Comments     ⍝ this where the text really is
Body←Text⍱Comments  ⍝ this is where the code (the body)
                      of the function is.

This code also works on matrices like ⎕CR.

The syntactic search is a bit more complicated but not terribly. It requires knowing where the characters forming a name are. More on that later.

With this in hand we can implement screening of the code. We need to specify what we want. We need a third argument.

So many arguments, so little room¶

Supplying a third argument to specify the behaviour of the function was impossible in the days when the first version was written. Sophisticated argument parsing was ruled out. The simplest way options could be specified was using single letters combinations. It was easier to incorporate them to the right argument, which was already a string. All that was needed to do was tack it on at either end. I opted for the beginning. The syntax was going to be:

[namelist] OSS  '/[options]/str1/.../strn'

The first character of the right argument would still be used as a delimiter and the options would be a series of characters, between the first two delimiters, one per option.

I changed the name to OSS for Object String Search when the function accepted character variables to be searched as well. At that point only matrices were searched but I extended the search to character scalars and vectors as well. To properly identify the lines where the hits where found in variables containing line delimiters (CRs) I decided to first transform them into matrices. I needed a function to transform such variables. I used a "classic" VTOM function².

The options were now like this: c would exclude comments, t would exclude text and b would exclude the code (body). Thus, to look for string ‘xyz’ only in the code of function <ABC> I would enter

'ABC' OSS '/tc/xyz' ⍝ mask out text and comments

By default only all the functions in the workspace were to be analysed. If everything had to be looked into, it had to be specified using (⎕nl 2 3) as left argument.

More requirements¶

As time went by further enhancements were made. The first one was the ability to consider executable strings as code. Strings are considered executable if they follow ⍎ or a trap assignment. This wasn’t fool proof but worked well most of the time³. The code that modifies the text mask for a given line is

text←text>(1,1↓code∧line='⋄') POS code∧(line='⍎')∨'⎕ELX'⍷line

<POS> is a "Partitioned Or Scan" function.

The code for selecting this option was set to ‘a’ (APL mode).

Case insensitive¶

This one was easy. All that was needed was a function to turn all alphabets into a single one before doing the search. For APLs like APL+Win and SHARP APL an <UPPER> function would do but later, when Dyalog APL was included, the function would also have to include any character used in identifier formation like those of the third (CAPS underlined) alphabet.

The code for selecting this option was set to ‘i’.

Search numeric¶

To include numeric variables in the search I added the ‘n’ option. Since the search string is itself character <OSS> does a format (⍕) of the variable before the search. This is not perfect⁴ but works well most of the time.

Syntactic searches¶

One of the interesting things to do is to be able to look for identifiers in code. If we need to know where the single letter variable ‘a’ is being used in the code, looking for the single letter ‘a’ won’t help much, even if we screen out comments and text. Syntactic search is really what we want here. Pushed to the extreme, regular expression would be ideal but its implementation would require much more code. A simple way to provide next-to syntactic search is to eliminate all hits preceded or followed by a character used in making up an identifier. This assumes that we are looking for a specific name. Looking for ‘+ab’ or ‘2x’ is meaningless (syntactically) and would not work. Again, knowing how the program works is useful. Looking for ‘a+b’ is acceptable.

If regular expressions are too much trouble, a small improvement consists in allowing searches to "start with" or "end with" a specific string. This is what I call left syntactic search and right syntactic search. This is easy to implement. To look for all names starting with ‘ABC’ we can use

OSS '/l/ABC'

To find all names ending in ‘xyz’ we can do

OSS '/r/xyz'

To find an exact match of ‘a’, use

OSS '/lr/a'

The code for selecting these options was set to ‘l’ for left syntactic search and ‘r’ for right syntactic search.

Local names, global names¶

After large workspaces started to appear programmers couldn’t wait to fill them up with 1000s of objects. In one case I needed to clean an application and find out where some global variables were being used to determine if they needed to be localized. I found myself rummaging thru code looking for the elusive variable ‘x’. To better find it I needed to exclude the functions where ‘x’ was localized. I came up with the ‘exclude if a hit is found in the first line’ option. This is not real "localization" exclusion but it’s close enough. To complete the exercise I added the ability to screen out code NOT found in the first line.

Obviously, this is mostly useful with functions or function representations.

I picked ‘;’ as the code for "locals" screening option and ‘⎕’ for the "globals" screening option.

Looking into files¶

It appeared fairly quickly that scanning code in the workspace was insufficient. Many applications store code on file, each component of an APL file containing code or data. For APL+Win users, the code for functions comes in the form of either matrices (⎕CR) or CR delimited vectors (⎕VR). For other APLs such as SHARP APL, other data types are also available that PACK objects together. The initial version of OSS would skip components containing these type of objects, just as it would skip them in the workspace, and wasn’t therefore very useful. The later versions, however, allowed scanning inside packages (or ⎕OR of namespaces for Dyalog APL, or overlays for APLX).

To scan files I overloaded the function. If the left argument was a list of numbers I determined they would mean FILE TIE NO, START CPT, END CPT, STEP with appropriate defaults for short arguments.

The code for selecting the option to look into PACKED objects recursively was set to ‘↓’.

To look for ‘⎕SIZE’ in file tied to 9 from component 11 until the end, into the object if packed, I used

9 11 OSS '/↓/⎕SIZE'

The result¶

OSS originally only displayed the hits. It didn’t return anything. There came a time when I was only interested if a match was found, or rather, the number of matches found in each object scanned. I made <OSS> do just that. The result would be an N (number of objects searched) by S (number of strings looked for). If 4 functions were searched for 7 different strings I would return a 4x7 table of integers.

The option for returning the number of hits was set to ‘=.’ (or ‘←’).

Patterns¶

Regular expressions are not recognized but there is a way to tell OSS to use an external function to do so. Should some code be available it should be assembled under the cover function <OSSPATMATCH>, which should take a pattern as left argument and the string to look for as right argument. It should return a boolean indicating where the matches are found. OSS will make a call to that function instead of ⍷ if the option ‘p’ is supplied.

This option was introduced for the Y2K projects I was involved with. It proved invaluable when coupled with functions that could recognize regular expressions.

How the function works¶

With those specs in hand let’s have a look at its implementation.

A stand-alone function¶

To be useful this function has to be brought in alone. No one wants to have to remember to also copy a slew of utility functions with a utility every time it’s needed and have to erase them all when finished with it. The need to encapsulate the whole thing into a single function implies that utility functions like <POS> and <UPPER> be defined locally. The same for the <VTOM> function used to make character vectors into matrices. Furthermore, <VTOM> itself can be used in defining other functions. For example, to define the POS function I used

⎕FX MTOV '* r←p POS s;t * s←s∨p ⋄ r←≠\s\t≠¯1↓0,t←s/p'

The fact that ⎕FX is a standard system function means any APL can accept this statement.

Name conflicts¶

With all the names in use in the function it is now obvious I was going to run into a name to scan that I was already using. And I did. I resorted to using ‘⍙+number’ names for variables and a mixture of upper and lowercase letters for functions. It wouldn’t guarantee I’d never run into problems but it would minimize the odds. For example, function <POS> became <pOs>.

• determine what kind of search is required
• build 5 functions that will be needed:

MTOV, POS and UPPER, which are always the same. MASK, which in its simplest form returns the scalar 1; in its complex form it returns a boolean of the same shape as its argument showing where hits are valid. For example, in the case where the comments and text are to be excluded from the search the function is

     ∇m←MASK m;c;t;b;a
[1]   b←c⍱t←t>c←∨\(m∊'⍝')>t←t∨≠\t←m∊'''' ⋄ m←~t∨c
     ∇

Finally, SEARCH is the function that loops on each string (remember there’s no nested strings here) to look for and builds a boolean using various glocals⁵. It takes into account the "appear in first line" case. It calls MASK to screen out unwanted portions.

… for each object to scan:

• get the matrix representation of the object
• look for matches using SEARCH; if any, display the hits

Restrictions¶

Quotes. You have to account for the APL syntax. For example, to look for the string "I can’t" means you have to write something like

OSS '//I can''''t' ⍝ 4 quotes for 2

GUI objects are not searched.

Size does matter. Larger objects may make OSS fail.

OSS with all these options remains a fairly simple piece of code. It spends some time determining the options, building the local functions before looping tightly on each name to scan. To limit the number of symbols (name conflicts) I reused variables and removed labels. The final version takes less than 6K⁶.

Wait, there’s more¶

Replacing strings¶

This entire search is fine but what if you want to replace a string by another? I ran into that too. And created another function to deal with it.

One of the reasons for creating a separate function is the lookup method.

In OSS, the search is simple: we use ⍷. In OSR (for Object Search and Replace), OSS’s counterpart, the search is orderly, non-overlapping. This makes the searching code quite different. Furthermore, OSS looks into matrices, making reporting of lines easier. In OSR, there is no string shown and a different algorithm, using CR delimited strings, is used to find, mask and replace.

Some of the options don’t apply in OSR. For example, the "return a result" option is not used.

Why do this again?¶

Again, when I started this project there was nothing like this on the market. Even today, APL+Win offers a user command ]LOC or ]WSLOC and Dyalog APL has library functions that can be put in ⎕SE but none offers all these features at once. Furthermore, because I go from one APL to the other I like my tools to be consistent and made them so. So whether I use SHARP APL, Dyalog APL, APL+Win or APLX, I know how to use and expect the same result.

I have been using these functions for quite some time now. They’ve saved the day a few times.

You can find this code for all the above APLs at www.milinta.com.

Appendix¶

These are the codes accepted by OSS and OSR and their meaning