Suggested Standards for a Component File System at a Commercial APL Site

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 105

Need for File Standards¶

At any commercial APL site using a multi-user component file system (such as the ones offered by STSC, I.P.SHARP or Dyalog) there is a need for a set of minimum standards and guide-lines for files. It can be argued that if all the files adhere to the standards, user and programmer productivity may be increased along with other benefits. This document will attempt to set out such standards and guide-lines, and the reasoning behind them.

Areas Covered by the Standards¶

Naming Convention. The standards relating to APL file names are constrained by the File System Software and existing applications. However, some guide-lines can be given, along with do’s and dont’s.

Naming conventions for new applications systems will be suggested, as they often simplify the users’ task of choosing the right file.

Component use. The most important part of this set of standards will relate to the use of standard components across all systems. This gives all files a similar "feel".

Use of standard components will allow the development of utility software (and "meta-systems") relating to APL files, which in turn should have benefits in the areas of maintenance, housekeeping and personal interchange (that is end users changing jobs in the company).

Access security. By standardising the access security of APL files (which itself can be implemented with the use of utility software) benefits relating to maintenance, housekeeping, and personal interchanges can again be achieved.

Naming Convention¶

Purpose of a file name. The purpose of an APL file name is to uniquely identify that file within the APL file system. The file name however may also be used to convey additional information to the user of the system (such as the year the file relates to).

Conflicts between making the file name unique, the limited format of the file name imposed by the system, and making the name as meaningful as possible, is mitigated by the use of a naming convention. Typically, the information a file name can be expected to convey includes:

• Owner. This information is often (but not always) contained in an account or library number, being the first part of the file name.
• Application system. The system the file belongs to is often encoded into the file name via a standard prefix or suffix.
• Type. The type of data (e.g. documentation, tables, temporary storage, variable data, back-up or archive data etc.).
• Date. The date the data in the file relates to in systems that run in cycles.
• Version. The version of the run producing the file, if run more than once.
• Release. The release version of the software if the files are not compatible across all versions.
• Status. The status of the data (e.g. Test or Real, Checked or Unchecked etc.)

By adopting the use of standard components, then some of the above information may better be conveyed within the file, via standard utilities.

Do’s and Dont’s. (Some of these may apply in some environments but not others, e.g. under CMS but not under UNIX.) File names should contain only (uppercase?) alphanumeric characters. Use of "delta" or any underlined character is discouraged. These non-ASCII characters may cause problems with some hardware (e.g. non-APL keyboards/printers). Remember that file names may well have to be manipulated by the end user outside the APL environment.

System prefix. If file directories are not available, do use a 2 or 3 character prefix to identify the system to which it belongs.

Date. If the date is needed, do use a simple format that can easily be read e.g. YYMMDD.

Type. To convey the type of data in a file the following suffixes (extensions) are suggested: DOC for formatted documents; TEXT or TXT for ASCII text files; TEMP or TMP for temporary files; TEST or TST for test data; DATA or DAT for the main production data; CODE or BIN or HEX for machine code and other "unreadable" files.

Note. After a run has been completed, no "temporary" files should be significant, and users should be free to delete any such files found "lying about".

A copy of the name of the file should NOT normally be held within the body of the file, unless all utilities to "copy" and "rename" files know of this convention and can take appropriate action.

Components¶

With the widespread use of standard (reserved) components within an installation, it becomes possible to write "system utility software" to manipulate files. This should ease the maintenance and housekeeping workload. Also it means that standard tasks needed in most systems (e.g. copying, access control, archiving etc.) need not be built afresh for every system.

Reserved components. Most importantly, all files should be numbered from 1 upwards. Components must NEVER be dropped from the front end. Reserved components should occupy a block at the front of the file and a suggested set up is defined below.

Component 1. File type identifier. This component should contain either a simple text vector, or a nested vector with a simple text vector as its first element. This text vector should identify the file application type. The meaning of the remaining elements (if any) should be defined in component 2. Suggested uses however might include sub-identifiers, version number, index offset, etc.

Component 2. Text description of the file. The second component should contain a description of the file in an easily readable format (e.g. a text matrix). This component should define exactly what is contained in the reserved and other components. The index method (if any) should be explained.

Component 3. Top level index component. The exact format of this component will be defined to the software by the "file type" in component 1, and to people in component 2. The main point of this reserved component is to prevent "partial updates" occurring by providing a single component to be updated which "fixes" any changes made to the file.

Components 4 to x (where x is defined by information in component 1). Other system level information, such as free component lists, secondary indexes, audit trail, etc., and SPARE COMPONENTS. Every system’s file layout should have some "slack" or spare capacity in the form of unused spare components. A few empty components in every file take up very little space, but they can be extremely useful when the system undergoes major modifications.

Indices. With all but the simplest data files, during updates file data integrity may be lost due to a system failure causing a partial update. One method of preventing partial updates is to never replace data in-situ during an update, but to write the changed components out to new components (multiple writes) and then REPLACE the index pointing to the old data by the updated index pointing to the new data, in a single disk write. If the system fails before the index is replaced, the file holds an intact copy of the data as it was BEFORE the update started. If the system fails after the index has been replaced, the file is again intact but also updated.

When the file contains more than one sort of data (e.g. control information as well as program data) it may be useful to have a separate index for each data type, with a single high level "master-index".

Index offset. The "index offset" allows an extra level of flexibility in a file design. The idea is that every index on the file holds component pointers rather than absolute component numbers. The "index offset" is used to convert between the two.

With this extra level, files can easily be expanded to allow for extra reserved components, merely by creating a copy of the files with the extra component inserted after the existing reserved components before the data pointed to by the index(indices) and changing the index offset value.

Access Security¶

Note. The DATA PROTECTION ACT may have implications in this area. These suggested standards DO NOT cover this at all. However, these standards may provide pointers to problems raised by the Act. In-house file standards manuals should go into this area at a more appropriate level.

In some environments such as UNIX, access is controlled by the operating system, rather than the APL file system. Most of what follows may be not applicable in such environments.

As far as security is concerned, there are two types of file, Private and Shared. In most systems the default security provided by the system for private files is sufficient.

For APL file systems that employ "access matrices" the type of problems that one encounters with shared files include:

• The use of "Generic user IDs" (non-specific logon accounts used by a group of users);
• The difficulty of maintaining complex access matrices with the addition of new users or movement of users within the company;
• Utility software unable to handle password protected files.

Types of access - "WHAT". It is suggested that there should be 5 pre-defined levels of "shared" file access controlled by passwords known to the system. These levels are:

NONE

No access allowed to the file.

READID

Allowed to read the 1st component (plus owner, date, time and file size information) only.

READ

Allows reading of the data, but not changing the data on the file.

WRITE

Allows reading and writing the data. May not change the access matrix.

STEWARD

Complete unrestricted access.

Groups of users - "WHO". It is often desirable to set the access level for a group of users. This group may well be dynamic during the life of the file. This may be most easily achieved by applications "owning" their own files. In so doing, the application decides who can do what to one of its files via passwords in the access matrix known only to the system. Such systems must provide access to all the housekeeping utilities needed (e.g. archive, copy, rename, back-up, delete, etc.).

The application can then maintain (in a relational data-base?) lists of file owned, along with users’ access authority to different classes of files.

File utilities - "HOW". A single set of file utilities should be developed, through which ALL file accesses are made to shared files within application. These utilities would be common across all applications within a site. Once written, the application programmer’s task would be much simplified, and if the application platform changed (e.g. from CMS mainframe to DOS PC) only these utilities would need to be rewritten for the new platform.

Other Notes and Guidelines¶

Always "share" tie a file if possible. Tied files reduce system restartability but may increase efficiency. Don’t "HOLD" files for longer than is strictly necessary.

Within a system use "system-wide" cover functions to access the files via lower level utility functions.

System passwords. The availability of a system password, which the system CALCULATES from the file name, via a secret algorithm, can be extremely useful, even if this password only allows "READID" access.

Summary¶

This section will attempt to highlight the pros and cons of implementing these standards. Please note that the benefits and disadvantages are not nearly as "black and white" as shown here.

Benefits:

• Use of standards reduces training times of new users to systems, when those users are already familiar with other systems using the same standards and utilities.
• Reduced development, testing and documentation time for new systems, since standard utilities can be used in multiple systems.
• Easier housekeeping and other maintenance. (Standard utilities can read files from many applications.) Rather than changing possibly hundreds of file access matrices when a member of staff changes job, only a single data-base entry may need to be changed.
• Better integration of systems. Use of "Umbrella systems".
• Tighter control of security. Compliance with the "Data Protection Act" may be easier to achieve and prove.

Disadvantages:

Strict adherence to the standards (working to rule) may be counter-productive. Standards may stifle the "best" solution and create inefficiencies.

Imposing new standards on old systems costs money, (rewriting, retraining etc.) which may never be recovered in the life of the system.

"Standard files" may take up more disk space and be less efficient than "optimised" non-standard files.

New packages acquired from "external" sources will not follow these (or your) standards and thus cause difficulty.