Tools, Part 1. Basics

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 106

In the following article I use terms specific to our trade. You won’t find them in the dictionary but I assume the reader is familiar with words such as ‘monad’, ‘global’ (as a noun) and ‘default’ (verb). I also use quotes and angle brackets to help determine the type of the object I am referring to. ‘Quotes’ denote a variable or workspace and <angle brackets> refer to a function/operator or file. Often, the context is sufficient to remove ambiguities. Italicized words have a special meaning and often precede their definition.

Introduction¶

Programmers get used to some tools, often their own tools. Most APL vendors come with tools, which are not usually compatible with other APLs.

I have been working with different APLs and I have had a need for cross platform tools. Rewriting them for each APL is cumbersome. When you make modifications to a version on one of the APLs you have to keep track of it in order to change the other versions in the other APLs. I eventually put everything under one platform and made the change there, distributing the changes across the various APLs. This will be the subject of another article, if time permits…

This is the first of a series of articles on tools for APL.

A function to parse command lines¶

This function is the first of my tools to have been ported across platforms. That’s because I use it almost everywhere. It’s one of my favourite functions. It allows parsing a string following specific rules. I wrote it when I started using user commands¹. I was tired of having to write up to 20 lines before doing anything useful every time I had to write a user command. It also allowed standardizing my code by using a uniform syntax for all my user commands and all my code. Eventually it was incorporated in my code everywhere else.

Now, the way software if written is not always by careful analysis of the problem. More than often it is the result of experimentation. Some code is written to study the problem. The prototype becomes the program and is refined until it reaches, in an asymptotical fashion, the desired result… Until the requirements change again…!

This case is no different. I wrote the program specifically for user commands in APL/PC a while back when nested arrays weren’t implemented. Over time it proved useful for other things and I adapted the code accordingly. It has doubled in size since I first wrote it. Typical.

In the following text I give the user commands case as an example but remember it could apply to any other program that takes a string to be parsed as argument. Here it is:

PARSE utility.

Keywords: USER COMMAND, ARGUMENT, TOKEN, SWITCH, DELIMITER

This function will return the parsed tokens of a string argument given to a user command. It will also set a series of globals, one per allowable command switch.

To understand what this means we must first know that the user command processor does not perform any parsing of its own. The processor will only take whatever is to the right of the command name and supply it to the APL function performing the command. For example, assuming the user command ‘bingo’ has been defined. If we enter

]bingo a/b=x O41

the processor will start the program <cmdbingo> with the argument ‘ a/b=x O41’, all spaces included.

That’s the way it works. It is general and allows us to do whatever we want. It is our problem to make some sense of the string given as argument. We can adopt a convention. Consider DOS or Linux: commands in these systems follow a loose convention using switches². Take the Linux command line ‘cp’, which copies a file to another:

cp filea fileb -R -v

Here ‘filea fileb’ is the argument list and ‘R’ and ‘v’ are switches. There are other acceptable switches to this command but they are not used nor shown here. In this case the argument list is made of 2 tokens: filea and fileb. Here, the character identifying a switch is the hyphen (-). DOS uses ‘/’ as the switch delimiter.

Another system closer to APL is Logos, a SHARP APL product for the mainframe. Commands in Logos use a similar convention but much tighter. The switch delimiter in Logos is the ‘+’.

A switch alters the behaviour of the command. A token is what the command needs to do its job.³

Conventions¶

We can use our own convention. In this case the convention I use is

• All commands take 0 or more tokens and accept 0 or more switches.
• All tokens come first (all switches come last).
• Tokens are separated by one or more spaces. A delimiter identifies and precedes a switch.
• Switches may be absent, present and may take a value (a string).
• Switches and their values end at the beginning of the next switch or at the last non-blank character.
• Switches can be entered in any order.
• Tokens and switch values may be surrounded by quotes to include spaces and switch delimiters.
• Switches, once prefixed, must produce a valid variable name (case sensitive).

Example:

]MYCOMMAND   1ST-ARG   2ND-ARG   MEBBE-3RD /SWITCH1 /SWITCH2=abc

Here ‘1ST-ARG’, ‘2ND-ARG’ and ‘MEBBE-3RD’ are the tokens. They are separated by spaces.

‘SWITCH1’ and ‘SWITCH2’ are switches that modify the behaviour of ‘MYCOMMAND’. ‘SWITCH1’ is present and ‘SWITCH2’ is not only present but has the value "abc". There may be other acceptable switches that have not been specified. Here, the character starting a switch, its delimiter, is the slash (/).

The function¶

The APL <⍙PARSE> function will return (in this case) the string ‘*1ST-ARG*2ND-ARG*MEBBE-3RD’ and set the global variables ‘⍙SWITCH1’ to 1 and ‘⍙SWITCH2’ to ‘abc’. The star character (*) is an example. In practice the function should use a token delimiter that cannot be entered from the keyboard such as ⎕TCDEL.

Example. Here’s the function associated to the user command ‘MYCOMMAND’:

[0] CMDMYCOMMAND ARG;TOK;⍙SWITCH1; ⍙SWITCH2; ⍙SW3 ;⍙SW4 ;DELIMITER
[1] ⍝ THIS COMMAND WILL SHOW WHAT IS PARSED BY <⍙PARSE>
[2]  TOK←'/SWITCH1 /SWITCH2= /SW3 /SW4[=]' ⍙PARSE ARG
[3]  (⍕+/TOK=DELIMITER←1↑TOK),' TOKENS ENTERED'
[4]  'VALUE OF SWITCH1: ',⍕⍙SWITCH1
[5]  'VALUE OF SWITCH2: ',⍕⍙SWITCH2
[6]  'VALUE OF SW3: ',⍕⍙SW3
[7]  'VALUE OF SW4: ',⍕⍙SW4

The user command associated with this function will only accept 4 different switches: SWITCH1, SWITCH2, SW3 and SW4. The syntax of the command is specified on line 2. It is the left argument to <⍙PARSE>.

Here SWITCH1 and SW3 can only appear without any value assigned to them. If /SWITCH1 appears in the command line (which is passed as argument to this function) then variable ⍙SWITCH1 will contain the boolean scalar value 1. If not, it will contain the boolean scalar value 0.

Same thing for /SW3 and ⍙SW3.

/SWITCH2 can only appear with a value assigned to it, that is followed by an ‘=’ sign and a non-empty string. If it appears in ‘ARG’ (the command line), variable ⍙SWITCH2 will contain the assigned string, that is the string to the right of the ‘=’ sign, otherwise it will contain 0.

/SW4 is ambiguous; if it appears in the command line without a value then ⍙SW4 will be set to 1. If it is assigned (with the ‘=’ character) then ⍙SW4 will be set to the string next to it and if NOT present on the command line ⍙SW4 will be assigned 0. The ‘[=]’ meaning ‘assignment not compulsory’.

These are the rules of this particular command. If someone enters

]MYCOMMAND   now I know /SW4   /SWITCH2=Oh! Really?

then we’ll have ⍙SWITCH1≡0, ⍙SWITCH2≡‘Oh! Really?’, ⍙SW3≡0, ⍙SW4≡1 and TOK will contain an 11 character string whose 3 tokens (‘now’, ‘I’ and ‘know’) are each preceded by the same character⁴.

Problems with tokens and switches arise when they must contain spaces or switch delimiters. Using quotes around the tokens or the switch values will circumvent this problem. <⍙PARSE> will recognize empty strings and strings with quotes which must be doubled. Here’s another example:

]xxx OK ''  's/he is'  'I''M HERE' /S1='''' /DT='3/12/27'

Here the command ‘xxx’ has 4 tokens: ‘OK’, an empty string, ‘s/he is’ (with ‘/’) and a string with embedded spaces and (doubled) quotes. Switch S1’s value is a single quote and DT’s value contains ‘/’s.

The syntax specification¶

The left argument specifies how the right argument to <⍙PARSE> must be parsed. It contains

1. the switch delimiter
2. all the acceptable switches and their application and possibly domain
3. the character used to separate the tokens (defaulted)

1. <⍙PARSE> uses the first character of the left argument as switch delimiter. In other words if ‘/a /b=’ is the left argument then ‘/’ is the delimiter. If ‘+a +b=’ is the left argument then it is ‘+’.

2. Each allowed switch is described. It starts with the delimiter then the name. If it may take a value it is followed immediately by ‘[=]’. If it requires a value it is followed by ‘=’. Moreover, if only specific strings are accepted they can be listed here. For example if only the cities of PARIS, LONDON and ROME are acceptable then

'/city= LONDON  PARIS  ROME' ⍙PARSE arg

would do it. Note that this is not a foolproof test. The verification relies loosely on ⍷.⁵

If only a set of characters is acceptable then ‘∊’ should be used instead of ‘=’. As in

'/code∊XYZ' ⍙PARSE arg

In this case any combination of the letters X, Y and Z will be accepted (minimum 1 character).

The complete specification is

3. The character used to separate the argument’s tokens.

By default tokens are returned with surrounding quotes stripped (same for switch values) and separated by ⎕TCDEL. This can be changed. The left argument to <⍙PARSE> may contain a special empty switch name to specify this character. As in (there are no other switches specified in this example).

'/=*' ⍙PARSE 'a b dsa'

which specifies the ‘*’ as the token delimiter and will return

*a*b*dsa

Why ⍙? Why globals?¶

When I first wrote <⍙PARSE> it was for user commands. ‘⍙’ is the character used throughout the user command processor to limit name clashing. I was following the convention in use.

Also, because APL/PC (for which this was written for) wasn’t using nested arrays I could not return arguments and switches together. The natural thing to do was to set globals. The scheme I used was to set global names with the same name as switches proceeded by, you guessed it, ‘⍙’. So switch X was going to be stored in variable ⍙X⁶.

Now, the first thing my first user wanted to do was to change this prefix so I made my first (quick) modification: if there exist a local variable named ‘⍙’ in the immediately calling environment to <⍙PARSE> I use its contents as prefix. See here; ‘⍙’ is assigned ‘ΔΔ’, the prefix to use:

[0] CMDMYCOMMAND ARG;TOK;ΔΔSWITCH1;ΔΔSWITCH2;ΔΔSW3 ;ΔΔSW4 ;⍙
[1] ⍝ THIS COMMAND WILL SHOW WHAT IS PARSED BY <⍙PARSE>; '⍙' must be local
[2]  ⍙←'ΔΔ' ⋄ TOK←'/SWITCH1 /SWITCH2= /SW3 /SW4[=]' ⍙PARSE ARG
[3]  (⍕+/TOK=1↑TOK),' ARGUMENTS ENTERED'
[4]  'VALUE OF SWITCH1: ',⍕ΔΔSWITCH1
[5]  'VALUE OF SWITCH2: ',⍕ΔΔSWITCH2
[6]  'VALUE OF SW3: ',⍕ΔΔSW3
[7]  'VALUE OF SW4: ',⍕ΔΔSW4

This is ugly, I concede. I did with what I had.

Exceptions¶

If <⍙PARSE> detects something wrong it will signal an exception. Under the user command processor the error is resignalled outside (unless you’ve disabled that "feature").

Validation¶

Obviously once these variables are set you may need to validate them. You need to verify that if, for example, switch ‘DATE’ is expecting a date, that it was, indeed, given a date. Or that /AGE=xx was given a single valid positive integer. And so on. Since <⍙PARSE> always returns values as character strings you must at least ⎕FI them when you really want numbers. This is pretty much beyond the scope of <⍙PARSE>. It does only provide string and character set verification. And that is already beyond the scope of parsing.

Default¶

You may also want to default some values. For example, if /FILE isn’t set you may want to default it to ‘MYFILE’. Or if /LEVEL wasn’t specified you may want to default it to the integer 3. In the first case you’d have to do something like

⍎(0∊⍙FILE)/ '⍙FILE←''MYFILE'''

and in the second case

⍎(0∊⍙LEVEL)/'⍙LEVEL←3' ⋄ ⍙LEVEL←⎕FI⍕⍙LEVEL

the ⎕FI⍕ is to cover the case where /LEVEL was indeed set to a string representing a number.

This default value thing is also annoying. It would be nice to have something like

⍙FILE←⍙FILE ⍙DEFAULT 'MYFILE'   or
⍙LEVEL←⍙LEVEL ⍙DEFAULT 3

So I created such a function. It is smart enough to convert the switch value (the left argument) to numeric if it "sees" that the default is also numeric.

Where do I find these functions?¶

<⍙PARSE> and <⍙DEFAULT> are both part of my user command files. You can find the file <DANBCMDS.SF> for APL+Win and for SAX⁷ at http://www.milinta.com. They also exist in the workspace ‘utils’ under the section of your favourite APL at the same location.

How do I use them with ucmds?¶

If you’ve used user commands before you know that you must regroup all the items a user command needs under the name GRPCMD… The functions <⍙PARSE> and <⍙DEFAULT>, if needed, must be specified there. They must, of course, also exist in your user command file. For example, if command ABC requires functions <sub1> and <⍙PARSE> you would do, to make it effective:

GRPCMDABC←'CMDABC sub1  ⍙PARSE'
]usave .GRPCMDABC

User commands already have <⍙GETOPT>. What’s wrong with it?¶

Nothing. It’s different. It handles simple cases as well as <⍙PARSE>. <⍙PARSE> is standalone and sets globals. It can be used anywhere. <⍙GETOPT> uses globals and subroutines and returns a result. It can only be used with user commands. It’s smaller but does less. It does not handle unknown switches, for example.

Features:¶

• standalone, no subroutine or globals needed
• the token delimiter may be changed using the null switch
• no need to enter the full name of a switch; the minimum to distinguish it from the others is sufficient
• unspecified switches are refused
• the prefix for the variables to contain the switch values may be changed
• the switch delimiter can be chosen
• minimal string validation can be performed by specifying the acceptable strings or characters
• a separate default function returns a specific value if a switch hasn’t been specified

An example¶

This user command runs under APL/II. It translates a string (the argument to the command) into an argument suitable for ⎕SOUND to play the string in MORSE code. The string can be made out of several words (tokens), each returned preceded by a space (/= ). It accepts 3 switches: FREQ to specify the frequency to use (default 1024), PER to specify the duration of a dot and boolean TEST to also perform a test using ⎕sound:

    ∇ CMDMORSE  STR;F;LD;LP;LS;⎕IO;LIST;⍙FREQ;⍙PER;⍙TEST
[1]  ⍝ USE ⎕SOUND TO PLAY STR IN MORSE CODE
[2]   STR←1↓'/= /TEST/FREQ=/PER=' ⍙PARSE STR
[3]   F←⍙FREQ ⍙DEFAULT 1024
[4]   LP←⍙PER ⍙DEFAULT 50
[5]   LS←⌊0.5×LD←⌊2.5×LP ⍝ LENGTH OF SPACE,DASH,DOT (PERIOD)
[6]   STR←UPPERCASE STR ⋄ LIST←Morse[;⎕io←1]
[7]   F←(3 2 ⍴F,LP,F,LD,0,LS)[,3,[1.1]'.-'⍳(1↓,Morse[LIST⍳STR;])~' ';]
[8]   ⎕SOUND ⍙TEST/⍙RESULT←F
    ∇

It’s a bit overkill for the task at hand but it gives an idea on how useful it can be (imagine a fn with 12 switches!)

By contrast lines [2,3,8] would be, using <⍙GETOPT>

[2]   STR←⍙SQUEEZE (¯1+STR⍳'/')↑copy←STR
[3]   F←1024 ⋄ →(2=⍴⍴t←copy ⍙GETOPT '/FREQ')/⎕lc+1 ⋄ F←⎕FI t
[8]   ⎕SOUND (''≡copy ⍙GETOPR '/TEST')/⍙RESULT←F

Wish list⁸¶

I would like another pseudo switch in the left argument to specify the prefix. It might work by using /⍙= as in /⍙=ΔΔ (for "use ΔΔ as prefix"). See:

'/⍙=ΔΔ /SWITCH1 /SWITCH2= /SW3 /SW4[=]' ⍙PARSE ARG

Another thing I’d like the function to do is return command argument(s) followed by switches as a nested array. This would allow performing multiple assignments to random variables and avoiding the ‘globals’ specification. Something like:

(tokens sw1 sw2)← '/⍙=← /sw1 /sw2=' ⍙parse cmdarg

The ability to ignore the case of switches might also be interesting. Switch /sa and /SA would be considered the line. This would be in line with the APL+Win version of user commands.

Limitations of this particular version¶

A switch may not be the exact beginning of another switch. For example, /SW and /SWITCH may not coexist, but /SW1 and /SWITCH may. It doesn’t also verify that your prefix is valid. If you set it to something that produces an invalid variable name it will stop ungracefully.

Note¶

This is not a foolproof function. It’s a tool to help saving time parsing an argument line. You must do your own checking from there.

Finally, I wish STSC (and subsequent owners) had made something like this available right at the beginning. I think it would have made user commands easier to adopt …