How STSC’s ⎕MF can help in testing APL workspaces

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 135

The use of STSC’s monitor function (⎕MF) in the area of optimizations has been well documented in the past, e.g. Dave Ziemann’s excellent article "Efficiency in APL" in Vector Vol.4 No.4, but I think ⎕MF has a potentially more useful role to play in testing.

What ⎕MF does is to record the amount of cpu time spent in each line of code executed in each function for which monitoring has been turned on. It also (and from my point of view, more importantly) records the number of times each line is entered.

I use ⎕MF during testing to check that every line of code has been executed. Note, this does not show that every path through the code has been tested.

At its simplest, )XLOAD the WS to be tested, turn ⎕MF on for all functions, run the WS using your test data, save a text matrix of the results of ⎕MF alongside the ⎕CR of each function being monitored to file. Check the normal test results. Print the ⎕MF results file and check that every line of every function has been executed, and if some lines have not, modify the test data and repeat the process.

Sounds simple, and the idea is. However there are some problems.

First of all, not all the functions may be in the test workspace at the start and end of testing (dynamically loaded or fixed functions). The most difficult area in this respect is with function overlay. You may need to alter the overlay procedure to do the ⎕MF recording automatically for you and then merge the results from all the overlays.

Secondly locked and compiled functions can’t be tested in this way. Testing of these function needs to be done in their unlocked/pre-compiled state.

Thirdly, the resolution of ⎕MF is only 1 line, not an APL statement. So ⎕MF cannot show that all statements in lines with multiple, conditionally executed statements (e.g. lines with "DOIF") have been executed. (This is one reason I do not like "DOIF" and lines with statement separators.)

Fourthly, it is VERY difficult to create test data/input that executes every part of the system.

What ⎕MF does show you more often than not, is that your testing is incomplete.

I recently had to test an archiving system being used at a large site. This had a great deal of error trapping designed to be as fail safe as possible. I used ⎕MF to great effect in proving that every line of code had been tested. I then repeated the testing of the error trapped function, with a generated error being put on each line in turn. ⎕MF was indispensible in showing what lines had been tested, and the path followed during error trapping.

The benefits of using ⎕MF during testing are thus:

• Confidence that every line of code has been tested;
• Confidence that your test data contains at least a minimum subset;
• It supplies a simple discipline and challenge to testing;
• It provides a measurable first stage testing completion target.

This last point may be the most important from your manager’s point of view.

If I may mis-quote the words of the famous beer commercial: "⎕MF shows the parts that other testing has not reached."

Happy testing!