APL - a Client-Server Language?

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 114

It is becoming increasingly difficult to sell a product if the magic words Client/Server are not clearly visible on the package. As a result, the term has become broadened by marketing departments to mean everything from dumb terminal emulation performed by a PC, through remote SQL queries, to applications using the most sophisticated peer-to-peer networked messaging systems or Object Request Brokers.

What’s New?¶

Is there any reason why the pragmatic APL user, intent on solving real problems (sorry, I mean pursuing real opportunities), should take notice? After all, many APL programmers were writing Client/Server applications back in 1980. Page 97 of the original SHARP APL Reference Manual by Paul Berry (copyright I.P. Sharp Associates, 1979), contains a complete coding example (almost an entire page) of a multi-user server mechanism based on an "object-oriented" communications mechanism called Shared Variables (if I may allow myself the liberty of widening the term object-oriented a bit).

The data processing community has succeeded in reinventing Shared Variables in a number of guises, some of which are now supported by a critical mass of applications and development tools. The driving force behind most of the market interest in these mechanisms is that they make it practical to split an application into independent Client and Server modules. Even when both modules run on the same machine, this strategy can simplify the architecture of complex systems. When client and server can run on two different machines, it becomes possible to "downsize" applications and move the presentation layer, plus some computation to workstations, which have superior price/performance ratios when compared to mainframes. In the current economic climate, this is the immediately attractive aspect of client/server computing — but in the long term, the ability to "upsize" applications is also important. When the usage of an application increases, or data must be made available to other parts of an organisation, it is often necessary to upgrade from a local or LAN-based database to a more secure or robust multi-user server environment.

Client/Server is just what APL needs¶

Modern workstation operating systems are perfect environments for APL. APL’s hunger for CPU and memory resources can easily be satisfied by modern workstations; APL’s performance is often better on a workstation than on a shared mainframe. By succumbing to Client/Server, you can finally make the manager of data processing happy and remove your APL code from his or her mainframe — and still maintain access to all that corporate data!

The evolution of tools for software development has resulted in a number of extremely useful tools and powerful paradigms for organisation (OO), distribution (SQL and Client/Server) and presentation (GUI). APL retains most of its competitive edge with respect to general-purpose manipulation and analysis, but this advantage is of little value if we cannot deal effectively with the other aspects of application development and delivery.

The APL vendors have come a long way in providing the mechanisms that we need to attach ourselves to standard tools available under workstation operating systems. Support for Dynamic Link Libraries (DLL), Dynamic Data Exchange (DDE) and the Clipboard have made it possible for us to plug straight into thousands of tools and applications written by people who had never heard of APL.

We need to extend this capability across the network using Client/Server techniques. The immediate requirement is to enable APL as a client, so that we can make use of all the servers that are available to users and developers using other tools than APL. But we should not forget that the unique capabilities of APL also make APL interesting as a Server — especially if we can just make APL applications available in the same way as any other service such as a database, ticket sales, or mail server — without anyone having to know that we used APL to write it!

Client/Server Models¶

We all know what Client/Server means, so rather than bore you with repetitions, let us take a look at how Client/Server products are marketed to the non-APL community (the so-called Information Technology Marketplace). The Client/Server product vendor will tell you that his or her product supports n distinct Client/Server models.

This means that the physical split between client and server can be inserted in n different locations in the simplified application model below:

End User
User Interface
Functionality
Data Access
Data

The split can be inserted between two layers or in the middle of a layer. Four of the most common models are described in a little more detail below:

Distributed Presentation¶

The X-Windows system has become relatively popular in the Unix environment. With X-Windows, the split is inside the user interface layer. Some presentation code is embedded in the application on the host (X-Windows terminology calls this end the Client, just to confuse you), and the rest is in the X-Windows Server which runs at the user’s end and provides presentation services to the application. Although this mechanism can be used to offload an important part of the presentation from the host CPU, a good deal remains, and the network load is rather heavy. The main advantage is that administration and coding is still as simple as it was on the mainframe, since the entire application is still running on a single machine.

Remote Database¶

With the current state of the art, Client/Server tends to mean the use of a remote SQL database. The split is between the functionality and data access layers. Despite the fact that this is the most common form of Client/Server computing, and that it is still rapidly growing, it is not the final solution. The problem is that SQL is an awful language for describing transactions. As the name implies, it was not designed for that. It is a query language. Performing all but the most trivial updates using SQL over the network is painful, inefficient and will always be a fragile mechanism.

Remote Transaction¶

The split is inserted in the functionality layer. Client and Server communicate using transactions that correspond to logical events in the application. The conversation might consist of messages like "Debit the amount X from account Y", or "Give me the latest forecast for currency X". In the first example, the steps required to implement the transaction might be:

1. Check whether account Y exist
2. Check credit limit
3. Register transaction
4. Update balance

Using Remote Database, each step is likely to require at least one network transaction. With Remote Transaction, all the logic is embedded in a portion of the application running on the server, which is much closer to the database in terms of turnaround time, bandwidth, and reliability. This is potentially the most complex solution in terms of coding, but with good modular coding techniques (and that includes object orientation), it is the most flexible solution.

Distributed Database¶

Some database systems are able to distribute the data itself, so that different parts of it reside on different computers. The management of a relational database that is split in this manner is complex. No matter how sophisticated your mechanism for ensuring the data integrity (often referred to as Two Phase Commit mechanisms) is, you will have to deal with worst-case scenarios such as restoring a distributed database from backups of different machines. To be safe, you will have to write code that is tolerant of small failures of integrity.

Next Year’s Model¶

The state of Client/Server is evolving in a direction where it is becoming easier to implement Remote Transaction systems. The first step in this direction is the implementation of added functionality in many commercially available SQL servers. You can now use stored procedures, or triggers, which means that the server has a simple procedural language which makes it possible to perform complex updates in a single call. Unfortunately there has been no standardisation of these new features, so applications that use them are not portable between different SQL databases.

The architecture of the future may be something like CORBA — the Common Object Request Broker Architecture, which enables dedicated servers to reside on an intelligent network, and allows applications to issue high-level calls on the network without knowing where in the network any particular service resides. A lot of work needs to be done to make this practical; for example, vendors must agree on a cross-network security system which makes it possible for you to log into the network once, and then use facilities under multiple host operating systems using the privileges assigned to you by a common security system — but there is little doubt that these systems will become commonplace during the next 5 to 10 years.

What is Available Now?¶

Let us assume that your goal is to develop applications that make use of the cheap memory and processing power available on workstations, to provide your users with a nice, easy-to-use graphical user interface, which can be integrated seamlessly with other applications on the desktop. You should buy a workstation running Macintosh System 7, Unix, OS/2, or Microsoft Windows, and one of the excellent new APL interpreters which provide access to all local services and applications available in the selected environment.

Access to Remote Databases¶

The remote service you are most likely to need is an SQL database. This access will be available to you in at least four different ways:

1. A Proprietary API Extender from database vendor
2. An API Converter, such as the Q+E Library
3. ODBC: Open Database Connectivity (or ODAPI)
4. A Generic Middleware product such as SequeLink

Application Programme Interfaces for SQL databases tend to have almost identical functionality. Typical calls are:

How do the four delivery mechanisms differ?

1. Proprietary API Extenders

   All databases make their API available in the host environment. AP127 for APL2 or SHARP APL, and the Oracle Interface for Dyalog APL under AIX are examples of products which hook APL interpreters up to database managers running on the same machine as APL itself.

   Almost all database vendors provide an extension of the database API, which is available to programmers on workstations. Examples are SQL/NET from Oracle, INFORMIX-NET from Informix, and the DOS/Windows Application Enabler for OS/2 Database Manager. These API extenders simply make the same API which is available to programmers on the host available on a remote system.

   The problem with these proprietary extenders is that they are all different, despite the fact that the functionality they provide is almost identical. Each API will use calls with slightly different parameters. If you use one of these tools, you must rewrite your application if you need to use another database.

   The database vendors often provide modules which make the API easily accessible from within selected development tools; either their own, or tools from vendors with whom they have strategic partnerships. As long as you stick to these, and your network provides the (often limited) list of protocols which the API Extender supports, you are OK.

2. API Converters

   The "API confusion" caused by proprietary API extenders has been the driving force behind the development of products such as the Q+E Library. The Q+E product provides a single API, which bolts on to a large variety of proprietary API extenders. Q+E converts your calls and your data to the formats required by each proprietary API. The DB+ product from TMT-Team is an interface from APL to Q+E, which in turn provides access to all database servers supported by Q+E.

   Note that you still have to have a proprietary API Extender present in order to use these converters. Simultaneous access to multiple services may be complicated, since all required API extenders must be loaded simultaneously; they may have different requirements with respect to network protocols, etc.

3. ODBC

   Instead of writing more API Converters, Microsoft provided a specification for a common API called Open Database Connectivity (ODBC), in the hope that database and middleware vendors would write compliant ODBC drivers, and that development tool vendors will deliver tools which can use these drivers. Microsoft has declared that it will provide ODBC drivers for its own database managers and that development tools such as Visual Basic and Microsoft Access will be able to use ODBC. The only piece of code provided directly by Microsoft is the ODBC Driver Manager, a switching station which loads ODBC drivers when applications request connections to them, and passes ODBC calls to the drivers after performing a small amount of validation.

   It is important to note that Microsoft and Borland primarily see tools like Access and Paradox (respectively) as development tools rather than database servers. Support for access to remote databases using ODBC or ODAPI — Borland and IBM’s alternative standard API (the best thing about standards is that there are so many of them!) was provided at an early stage, but there were no ODBC or ODAPI interfaces giving access to data stored in Access and Paradox until recently.

   The database vendors are scrambling to provide ODBC-compliant versions of their proprietary API extenders, and ODBC is rapidly reaching critical mass. This will eventually do away with the need for products like the Q+E Database Library, and Pioneer Software, the company behind this product, is moving on to the business of providing ODBC drivers on behalf of the database vendors. Providing a fully functional ODBC driver is no simple task, since the full ODBC specification is far richer than most database APIs. In fact, very few drivers support the full API, and one of the most irritating aspects of using ODBC is that you need to inquire about the level of functionality provided by a particular driver, and possibly adopt alternative strategies depending on the answer you get back. The result is that quite a bit of API Confusion remains, and one of the points made by recent Q+E advertising material is that they provide a "consistent set of ODBC drivers" (!?).

   Examples of additional features of ODBC not usually found in other APIs are calls to translate SQL statements between ODBC SQL and the native SQL supported by the server; scroll backwards; do cellwise or columnwise fetches; or even retrieve columns that were not specified in earlier fetches. Many of these ODBC calls make sense when dealing with a local database such as Access or dBase, but are very hard to implement efficiently for a remote database.

   The ODBC Support provided by tools such as Visual Basic or Excel tends to be at a much higher level than the ODBC API itself. For example, Visual Basic provides the concept of a Dynaset, which is created by naming a Table or View. The Visual Basic ODBC interface copies data into local storage, and you can make it easy for the user to browse or modify the data with a minimum of programming effort. However, it is often hard to control the resources used to implement this functionality, and solutions which run extremely fast when tested using a dBase driver with almost unlimited bandwidth may have totally unacceptable performance when used across the network. It is important to note that this behaviour is not a function of ODBC itself, but rather of the high-level access to it that is built into some development tools.

   Using proprietary ODBC drivers, or ODBC converters that bolt on to a traditional API Extender, you still have the same problems with respect to compatibility of the underlying (network) products. Drivers from different vendors will not always peacefully inhabit a Windows system side by side. To solve this problem, you need a generic middleware solution.

4. Generic Middleware

   Generic middleware products solve the compatibility problem by providing their own transport layer. You do not buy any communications software from your database vendor — the middleware server component connects straight to the database API on your host, performs API conversion at the host end, and uses whatever network you have available to transfer data to the middleware client component running on a workstation. This means that you only have a single product running on your workstation, even if you are using several networks to connect to a number of database systems on a variety of host platforms.

   The middleware vendors are also capable of providing an ODBC driver under Windows. A single ODBC driver can thus provide access to multiple services without risk of conflicts. The fact that the middleware product is in control of both ends of the network conversation means that it can optimise the use of the network to provide better performance than a lot of proprietary ODBC drivers, which are often bolted on to API Extenders which were not designed with ODBC in mind. As an additional benefit, you get a completely consistent ODBC implementation across all supported databases.

   The networked versions of AP127 available from IBM and Soliton have a "generic" middleware architecture, but these vendors provide access to a small number of clients, network protocols, host operating systems, and database managers. A product like SequeLink supports just about all databases running under OS/2, Unix, VMS, AS/400, VM, and MVS, over NetBIOS, SPX/IPX, more than 20 "flavours" of TCP/IP, AppleTalk, SNA and DecNET, from Macintoshes, Windows, OS/2, Unix and VMS clients.

   The SQAPL Client product from Insight Systems is an interface to SequeLink. Because identical SequeLink APIs are available on so many platforms, SQAPL can provide access to all services accessible through SequeLink from APL*PLUS II under Windows and Unix, APL2 under OS/2 and Unix, and Dyalog APL under Windows, OS/2 and Unix — providing exactly the same API in all environments.

Access to Non-SQL Services¶

At the moment, SQL and Client/Server are more or less synonymous — but I do not believe that this will continue to be true. SQL is a query language, and will continue to be used for that purpose — but SQL is not a particularly elegant or effective language for expressing updates or transactions. We need other mechanisms to initiate transactions, or to make Remote Procedure Calls (RPCs).

Vendors of Unix systems often provide RPC kits, which make it possible to compile an application consisting of a client and a server part. It is very hard to provide flexible systems in this manner, because the two parts are so closely interlinked. Changes require recompilation and testing of both ends of the application. If communication problems cause the partners to get "out of synch", it may be hard to return the system to a known state.

Vendors of Transaction Monitors such as IBM’s Customer Information Control System (CICS) are beginning to provide API extenders in the same way as the SQL database vendors. This makes it possible to code and test transactions on the host, and use the same transactions from host applications and Client/Server applications running on workstations.

Advanced communications protocols such as TCP/IP and IBM’s APPC (LU6.2) also provide mechanisms by which two consenting applications can exchange data over the network, but this approach still requires a bit too much network expertise for the comfort of the average APL programmer — and of course the solutions will be dependent on the particular network protocol being used.

CORBA¶

A working group with origins in the Object-Oriented community has defined CORBA — the Common Object Request Broker Architecture. This architecture will extend the development environments of the future (it is assumed that these will be Object-Oriented to some degree), so that they can send messages to objects, without worrying about whether the objects exist nearby, or somewhere on the network. CORBA needs to be built into the networks and integrated with security facilities of all networks and hosts involved, and the required technology is still under development in the laboratories. Once we have it, the network will more or less disappear from sight, and it will be as easy to extract data from a server on another planet as it is to open a spreadsheet. Let’s hope we all have fibre-optic telephone cabling by then!

While we wait for CORBA¶

For the time being, Generic Middleware is as close as we can get. Products such as SequeLink provide "open" servers, which you can use to encapsulate any existing host application. By customising an open server, you can call your favourite C program, CICS or OS/400 (RPG) transactions under MVS and OS/400 — or even APL! The SQAPL Server product from Insight Systems makes APL2/6000, APL*PLUS II/UNX and Dyalog APL/X available as SequeLink Servers under Unix. You can execute APL function calls from any SequeLink Client — not just the SQAPL Clients. You can make APL function calls from Excel under Windows, OS/2 or the Mac — from Omnis 7, WingZ, Visual Basic, Smalltalk, and so forth — using exactly the same API as you would to make SQL requests.

A Glimpse of The Future¶

We are beginning to get a glimpse of a future in which the users’ workstation provides a single entry point to any number of local, departmental, corporate, national, or even global services — many of which will be running on dedicated server machines located somewhere on a widely branched network, but all seeming to be an integrated part of the users’ desktop.

This future is not far off — and as always, APL is a step ahead. With the SQAPL Server, APL became one of the first general-purpose languages to have a dynamic function call mechanism bolted on to a generic middleware product. So far, the only language available has been SQL, plus a few operating system command shells implemented for the VAX and Unix. The APL Server is available for three of the most popular APL interpreters available under Unix.

APL will be a First Class Citizen of the new world order. From APL, we can not only call local and remote services, but we can provide services to local applications via standard calling mechanisms, from the Clipboard through DDE to DLLs, and to remote applications via Middleware. When CORBA arrives, you can be sure that APL will be there to meet it!

Acknowledgements¶

Many thanks to Kim Andreasen, Gitte Christensen and Martin Gfeller for helpful comments and suggestions.