Is APL a Team Sport?

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 67

Abstract¶

The article discusses ways in which APL programming teams can be managed to complete large, time-sensitive projects using parallel development. All aspects of the team effort are considered including design considerations, tool-kit use and staffing and organisation. The concepts described were used with success during the author’s five-year consulting engagement with a large retailer.

Do We Really Need a Team?¶

Many APLers view APL as so powerful that only one or two of us are needed to do any project, no matter how large or complex. However, I think that a combination of complexity, lack of project definition and extreme time pressure can make a project impossible without a team. Let me give you some background on the business situation so you can understand my experience with APL teams.

The projects I had to do were all merchandise planning systems with a four to six month development window. They all started life with a one or two page management directive requiring a new system or major enhancement in very general terms. The only clear requirement was the deadline. The main objective was to make it as painless as possible for company management in hundreds of stores, with over forty categories of merchandise, to plan sales, inventory, profits or losses.

When I started, I was hired to get a creaky ADRS (A Departmental Reporting System) based annual planning system through one last planning cycle while all of the previous programmers were being transferred to other areas. The PL/1 development of a replacement system was running late. I got through it with a lot of help from the folks waiting to be transferred and a lot of overtime. The job was just too big for me alone, or even for two people. Possibly Superman could have done it without help, but he wasn’t available. We had to have a team.

The second effort was a major redevelopment with a six-person team. The PL/1 development was going to be even later, so the current system had to be used yet again one more time. Our success in this effort led management to cancel the PL/1 project. Instead, we would develop the added functionality in the old system. Once that was completed, we were given a lot more assignments. The APL team got as big as 21 people doing multiple projects. Personally, I ended up with a five-year extension of a two-month contract. As we did more and more, I saw a pattern to our successes, which I describe below.

Design Considerations¶

The first step to each of our projects was to get the project defined. "What’s it gotta do?" is the first question to be answered. Once the desired behaviour was described, the APL team had to answer "How ya gonna make it work?"

I found that the best way to get the users to describe what they wanted was to work first on what they wanted to see. We could mock up menus and menu paths first. Then we would design the interactive screens and the calculations that supported them. The next step was the reports and the calculations needed for them. Once we had what the user wanted to see, we could then go through each screen and report to ask where the data would come from. The data sources usually would lead us to reading files from other systems. Finally, we would ask what data had to be sent to other systems.

Once there was a fairly complete definition of what the system should do, we had to talk about deadlines. Using the menu paths, we would define which options needed to be available immediately and which could wait. For example, reading last year’s actual sales had to be ready the day the system was installed, but exporting the plan data to accounting systems could wait. Because of the time pressure, development effort had to focus on the immediate needs, with development of delayed features while the users were using the initial product.

Specification Changes¶

Planning systems seem to suffer from a lack of planning, possibly for the same reason that a shoemaker’s children lack shoes. Specification changes during development were frequent. I found that the best way to handle them was first to ask often about future directions, and second to discourage changes enough to make sure proposed changes were worth fighting for.

The users were reluctant to reveal what new features might be required until the planning methodology was approved at the highest levels. However, over time I was able to get them to reveal ‘hypothetical’ directions so that I could allow for them in the design. In exchange, I would provide ‘ball park’ estimates on how hard it would be to make the ‘hypothetical’ changes. The team also kept their mouths shut about the information.

Discouraging last minute changes requires a light touch. I always mentioned our motto, "We’re in a hurry!". I did this so often that the users would quote it to me before they asked for a change. I would then tell them how much the change would cost, particularly in terms of time. The easy stuff I would do quickly. The hard stuff I would try to negotiate into something functionally adequate that was easier to program. In particular, I would try to keep changes within the capabilities of the existing tool-kit.

Technical Design¶

The primary design consideration is to maximise parallel development. This means that dependencies between functions have to be minimised by design. The key design concept to minimise dependencies is the hub for data flows. Just as airlines use a hub airport to move their passengers, the APL team should use pre-defined global workspace variables as a hub for all data flow. Restrict all data movement to movement into or out of the hub variables. In summary, to go to hell you have to change planes in Atlanta.

I mean all data movement goes through the hub variables: file I/O, screen entry and display, reports and even data imports and exports. Make as few exceptions as possible. In effect, the hub variables and the utility functions that get and set the hub allow object-oriented programming.

Let me illustrate by describing the data flow in a typical interactive screen. The screen driver program uses file I/O functions to fill the hub variables. It calls common calculation functions which use and set hub variables. Toolkit functions get data from the hub variables for special calculations using local variables. The screen driver then puts the results back into the hub using tool-kit functions. A screen-handling utility displays hub contents, and updates the hub from screen entries.

Benefits of Hub Concept¶

Using a hub maximises parallel development by making each piece of a project independent of every other piece. Each feature depends only on the hub, and is specified in terms of its interaction with the hub. Screen development, for example, can begin before there are any files because the screen driver program interacts with the hub. In the actual projects, screen development often started before there were any test files.

Using a hub also improves reliability. Testing can be done on each piece of code because each is independent. Results are verified in observed effects on the hub variables. Once verified, more code is re-usable because it works on the hub in the same way whenever it’s used. For example, a gross profit calculation using and setting the hub can be called from both screens and reports because they all use the hub.

Tool Kit Construction¶

My basic philosophy on tool-kit functions is that they have to be easy-to-use black boxes. It has to be easy to learn how to use them or they won’t get used. Over half of the people I worked with were short-term consultants hired specifically for the four to six month project window. They needed to become productive as fast as possible, so the tools had to be easy to learn. (Remember our motto.)

I started with the functions provided by ADRS. The hub started as the global variable structure of ADRS and the functions ADRS used for the structure. I gradually replaced the file system, the screen handlers and the reporting system by liberally borrowing from whatever was lying around. Improvements were based on customer requests for new features or persistent performance bottlenecks. While the specifics are probably obsolete, an example is worth discussing to show how the evolution was eased by having the hub to work with.

File I/O in ADRS, with VSAPL under MVS/XA, was slow because there was far more data than ADRS was ever designed for. In addition, allowing for multiple users had been accomplished by splitting the data into multiple small files which users could lock exclusively for update. All the files were read in a loop for combined reports. Starting with the functions in library 2 VAPLFILE using AP121, I replaced the multiple ADRS files with component pseudo-files in VSAM clusters. File locking was based on exclusive control of QSAM "shadow" files, one "shadow" for each component pseudo-file.

At the highest level, the new file functions set and used the hub variables, just like the ones they replaced. The only differences were in behaviour outside the hub. In the old file structure, multiple small files were segregated using hub values. In the new file structure, pieces of the component files were locked by using the same hub values stored on the file. None of the application code other than the file function calls changed at all.

The new arrangement changed clock time for reports from 40 minutes to three minutes. The file functions were designed to make a switch to real component files relatively painless. When we finally did switch to a commercial component file server, no code above the file functions changed at all. The hub variables were set just as before. True component files made the file-read time for the big reports about 15 to 30 seconds.

The advantages of using a tool-kit are many. Reliability is enhanced because the code is debugged thoroughly then re-used a lot. Flexibility in addressing performance problems comes from modular replacement, like the file I/O evolution described above. Application maintenance is also easier because everybody knows what the tools do and can follow the code by following the tool use. Coding speed is increased because you only invent the wheel once. Training is simplified because initially people only have to learn what the tools do and can leave how they do it for later.

Staffing and Organisation¶

Once you have more than a few people, staffing and organisation become almost as important as purely technical considerations. You have to interview, train and structure a team of notoriously quirky APLers. You have to maintain discipline and morale. You also have to handle a lot more external politics because a team is much more visible than the usual 2-3 APL people in the back closet.

The first step is interviewing new prospects. The most important thing here is to be yourself. You want the prospect to get a good idea of what working with you will be like so he can decide whether to take the job or not. Start by explaining the job. It’s a good idea to have several people gang up on the applicant so that the applicant gets a more complete picture of who he’ll work with and what he’ll be doing. It also gives several people in your shop an idea of whether or not they want to work with the applicant. I always gave a little test, asking for an explanation of a few lines of APL. This gave me a surprisingly accurate view of how well the prospect knew APL. However, I found that a real burning desire to work in an APL shop was probably more important than specific knowledge. I found that good training expanded our choices.

I did most of the training personally. I did this to show the newcomer that training is important. My involvement allowed me to evaluate potential assignments for the trainees, but more importantly, allowed the trainee to get used to working with me. Remember that training is both social and technical. On their first day, everybody is a trainee even if they have lots of technical experience. They don’t know your application and they don’t know how to work with you.

At the start of training, I had everybody work through APL: An Introduction by Howard Peelle. This served as a review, since many of them had not been working in APL recently. While the book may be out of print, any similar introduction would do. The idea is to get everybody up to speed in APL without embarrassing anybody. Working at their own pace, trainees took from 2 days to 2 weeks to get through the book and some selected problems. I never made a big deal about how long it took. I did mention often that once through the book, everyone was starting more or less even in terms of his exposure.

It is important to give realistic training problems so that the trainee knows you are not wasting his time. The realism should include incomplete specification of the problems. You can do this in a fairly humorous fashion. I always used to tell trainees that I was going to simulate our typical users in specifying problems, so they had to ask me for clarification if they needed it. Then the first time I caught them assuming something I hadn’t told them, I would get very wide-eyed and ask, "Whatever gave you that idea?". I did this even if the assumption was OK, just so they would learn to ask.

One good training exercise is to have trainees write new tool-kit functions you always wanted or re-write functions that are too slow. If the results are good enough, and they usually are, you can install the new code. For example, I needed a menu generator which would take a character matrix and put it on the screen, making every underscore character an entry field. My boss thought this would be too difficult to be practical. (We were using GDDM.) I had two trainees do it to my specification, then installed it. It was a big boost for their morale to see their "graduation exercise" used by everybody on the team. It also demonstrated feasibility without taking any experienced people off time-critical projects.

Team Structure¶

When you are structuring an APL team, you have to remember that 1 man-month of APL code is a LOT of functionality. APL teams tend to have more design and management work per coder than programming teams using other, less productive, languages. I found that I needed a foreman for every 3–5 people. Foremen came in two varieties, an architect/designer who didn’t code much and a general contractor who coded about half-time. The architect handled most of the user-contact during development and designed like mad trying to keep ahead of the coding. The general contractor answered most programmers’ detailed questions during coding, contacting the architect or the users on points that had yet to be decided.

I found that people tended to develop functional, rather than project, specialisation. Some were really good at reports, some were super screen specialists and some were great at getting data from other systems. People would switch from project to project often, but would usually do the same area on each project.

Functional specialisation seemed to be popular with the team. It also allowed rapid redeployment from one project to the next, which was usually a scheduling requirement. Team members understood their functional area completely and found it easy to apply their knowledge to new projects. The methods in each project tended to converge toward a de facto standard. As the same people worked on similar problem areas across multiple projects, they found it easier to use familiar tools in new situations.

Discipline and Morale¶

APLers are special characters… a sense of humour is required. For example, if everybody is writing his own tools instead of looking what’s available first, you can’t just order everyone to look before writing. You instead suggest that the Committee of Public Safety and Programming Practice, fresh from its efforts in the recent (1789) French Revolution, is now in control of approving new tool-kit functions. The penalty for submitting a duplicate function could be a pain, given their history. Well, at least it worked for me.

It’s best to explain why something has to be done, and not just what has to be done. Then you can ask for advice, which you should do whenever you have time for it. You have to be approachable, particularly if you’re the architect/designer. The specification is not going to be complete unless questions are asked.

Everybody has to be approachable on past mistakes. If you have a problem with others finding your mistakes, remember our motto. We were in a hurry when you messed up.

Some things will go right, some wrong. When they go right, broadcast compliments loudly. When they go wrong, let the punishment fit the crime. Remember, you want solved problems not fear and trembling. For example, when I got beeped at 3 a.m. about a bug in the overnight job, I didn’t yell at the programmer. I just walked into his office the next morning and put the pager on his desk. I quietly told him the beeper was his to carry until he got a good night’s sleep with no interruptions. Two nights later the code worked perfectly, and he gave me the beeper back.

Politics¶

Most programmers I’ve met fear and loathe politics, probably because they don’t really understand how politics works. For some reason, I’ve always enjoyed politics and have managed to do fairly well at playing them. The key thing to remember in the politics of programming is that programming exists to provide valuable information with the least possible problems. To find out what’s valuable, you have to talk to management and your users. You also have to minimise their problems in dealing with you and your system.

APL team political problems come up dealing with three types of people outside the team. The most immediate, if not most important, is the team’s non-APL boss, who can barely spell APL before he gets stuck with a very strange collection of subordinates. The second group includes all the poor desperate users of your systems. They are stuck with a new or vastly altered system with advance billing that reminds them of the last three unnatural systems disasters they lived through. The last type, the most important type of all, is the client / systems administrator who was selected by senior management to explain their grand idea to the ignorant slobs in the systems department. Each type has special needs, which I will describe based on my results of trying a lot of things that didn’t work before stumbling on something that did.

Your Non-APL Boss¶

This individual probably has a background in data processing management and is used to a normal pace of development. The speed with which you do things will pleasantly surprise him. How you do it will make him uncomfortable. Always tell him what you are doing, describing the features as the users will view them. Avoid telling him how these things are coded in APL. Use as little jargon as possible so you can be sure he understands what you tell him and never gets the impression you are trying to baffle him with fancy-sounding baloney. Especially avoid APL jargon, since he is unlikely ever to have heard it and will begin to wonder if you speak English (or the local equivalent).

While there are many things about APL that might require special treatment from the systems bureaucracy, never ask to be a special case unless there is NO WAY you can avoid it. Remember that special cases are special trouble for your manager. Save your special requests for things you will die without.

Your Poor Desperate Users¶

These people may never have seen a system that they liked using. They are used to rude responses from data processing every time they find a bug in the code and may have given up on reporting them. They will love you if you admit your mistakes immediately. You should encourage error reporting. Offer to fix the first guy’s data for free. You have to test anyway, so you might as well use his data. Follow up after you put out the fire and ask for suggestions. Your quick response will contrast starkly with what users have seen before and you will see a big change in their attitude.

Using this approach I quickly became very popular. Users became almost deferential, saying things like, "I know you’re really busy, but I think I may have found something that doesn’t work right." Secretaries would say things like, "I KNOW he’ll talk to you, please wait while I find him." I found out that many of the managers I talked to routinely were notorious for not taking phone calls from mere peons, like my boss’s boss.

Your Client / Systems Administrator¶

[Author’s note to the Pronoun Gender Police: female pronouns are used in this section because the majority of the people I dealt with in this position were women. If you are sensitive about pronouns which reflect reality, I apologise.]

You are wondering how she got this job? She was drafted! She probably had a succession of increasingly important jobs with managerial responsibility in the user area. Now her success depends on you, and you look weird to her. She doesn’t know what to expect and the lack of control scares her. The trick here is to be reassuring without over-promising. Never make anything look or sound easy. Tell her doing the job on time will be tricky, a LOT of hard work, but you have been successful before and expect to be this time as well.

It is better to be early than late, so be careful when making time estimates that there is some room for unexpected problems. In scheduling, remember the specifications are going to change and make allowances. She has to publish your completion schedule and explain to her management when the schedule slips, so it is very important to make the schedule realistic.

If you solicit rumours of changes in requirements, you can provide her with rough guesses about how they might change the schedule. Then she can use the estimates to force her management to decide if the changes are worth the delay. No matter how they decide, everyone involved will feel more comfortable having exchanged the information.

Encourage her to be as loud about your successes as she is about your problems. Constantly remind her, your motto is "We’re in a hurry!".