Report: BAA Anniversary K day
Vector 20:4, page 40
Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 40
Acknowledgements¶
Arthur Whitney (and his considerable entourage) all worked really hard to make this a success. The Royal Society provided the venue, and followed it up with an excellent meal afterwards (in slightly premature celebration of 20 years of Vector). Everyone who came had a good time, and felt that we should do more of this sort of thing. Ray promptly organised an APL Moot in a shed near Stansted, and that went off well too. It is reported next.
The KX Users Group Meeting¶
Stephen Taylor & I were privileged to be invited along to the KX User’s Meeting which occupied the whole day before the BAA workshop. The morning was taken up by a superb exposition of K4 (the latest incarnation of the K language) and attendees were provided with a temporary copy to try out during the meeting. It expired exactly 24 hours later, which gives you an idea how protective KX are feeling about their new baby.
The afternoon was occupied with a series of user-presentations. All I can say about these was that the content was seriously impressive, and that the volume of data routinely handled by Kdb (current release) can leave an audience stunned. Any more, and the corporate lawyers would be after us, so back to the relatively safe ground of K4 and Arthur Whitney.
Arthur on K, Kdb, K4 and Kdb+¶
My notes are a mixed bag, taken initially from Arthur’s talk to the Users Group, and consolidated with material gleaned from the BAA workshop the day after. Here are a few opening quotes:
“The best programming language is the one where the program is the shortest”
“The only programs that have a chance of being correct are the short programs”
“Words instead of symbols is fine - orthographic terseness is a bonus”
“If you find something in the system that could be shorter - let me know!”
So the goal of K4 and Kdb+ is very simple - to be the world’s best programming language coupled to the world’s best database. From the language side we have concepts like ‘Atom’ and ‘List’ and ‘Dictionary’ and from the database we have ‘Select’ and ‘Delete’, ‘Update’, Insert’ and ‘Join’. How to put them together in a way which makes sense?
From A+ to K to K4¶
Arthur’s attitude to A+ fascinates me - it was a massive success at Morgan Stanley “ … we did everything in A+. Trading systems, LP, computer-aided instruction, fixed income etc. All in A+ …” and yet “ … I didn’t like A+ ’cos its a lousy language!”. Which is very interesting, as I still think A+ is one of the best APLs around. However I also like quirky buildings with bits missing (like Ely cathedral) in preference to anything mathematically pure and technically ‘correct’ so maybe I just enjoy A+ because it’s a mass of pragmatism and compromise?
Anyway, onward from K to K4, and some ideas where Arthur is headed next.
His first objective (as always) is speed, which is why the immensely successful Kdb is being redone as native C code, rather than its current implementation as a library of K functions. This I can immediately relate to, and I am certain it will be an instant winner with his Kdb user-base. I am less clear about the migration to K4 particularly as “it is about 90% compatible with K3, but that is enough to kill any script”. Some of the key features are:
- the language has about 40 datatypes, 40 monads and 40 dyads, so roughly 1600 × 40 method signatures. Probably a few thousand are commonly used. The sourcecode is currently 9k bytes, so the average method length is around ½ a byte (K gets smaller with every build, but this has to be a record).
- K4 is either 32-bit or 64-bit, and includes TCP/IP support (native) and a nice HTTP server as a script. Arthur showed some of the K4 source for this, but may have wished he hadn’t - it takes terseness to new extremes and at least some of the audience were old-fashioned enough to appreciate the value of white-space in programs!
- K4 is headed to having tables as native data-types. This requires raw language support for many more data-types (e.g. 000011b is a boolean vector) including .Net types like DateTime. In fact the K4 list of types maps almost exactly to the .Net Common Language Runtime, with some extra little structures added.
The move to 64-bit addressing has taken much longer than anyone expected. “In 1992 we thought we couldn’t live another day without 64bit. Now (eventually) people are hitting WS FULL so the move to 64bit is going to happen.” The target hardware has 32-64 Gbytes of main memory, as mapped files become the bottleneck otherwise. Arthur put the addressibility limit at about 2 × physical memory. “Better to get WS FULL than hit the pagefile”.
Dictionaries and Tables¶
This (assuming it all works) is a major foray for functional programming into the world of relational tables. It was not clear from the presentation how much of this is still experimental, but there are clearly lots of edge-effects that can really spoil your day when you turn loose the APL primitives into a database.
Health-warning - the syntax could change and my notes were sparse, so take this as a flavour, not as a commitment from anyone to do it exactly like this!
This gives us an array which is addressible by key (nothing new about that) but what is new in K4 is that we can do arithmetic with it:
So far, so obvious. Now let’s make another one:
The challenge is how to ‘add’ two relations where the keys don’t match up.
“This stuff is kind of neat …” and implements at the primitive level stuff which many of us have spent many happy hours coding around. Usually at the point where databases and APL meet up. What bothers me is that Arthur has opened so many new Pandora’s boxes here in one go that he may never quite achieve the simple, logical consistency he craves. For example:
d & e
`a `b `c ! 2 3 4 min is obvious
d | e
`a `b `c ! 3 3 4 max is obvious
d,e catenation needs a bit of thought
As soon as you hit catenation, you are forced to choose between the ‘set’ paradigm, the ‘multiset’ paradigm and the ‘list’.
- a set like (2 3) is a simple unordered collection of unique values
- a multiset allows repeats but does not care about ordering (2 2 3 = 2 3 2)
- a list allows repeats and preserves ordering.
Sets are useless for arithmetic, multisets are useless for answering questions like ‘where did it change?’ so there are plenty of good answers to this one. Arthur’s choice was to treat it as an override “try e - if e fails use d” which I would never have guessed at as a possibility!
What about:
If you allow indexing by keys, you need a null to represent the ‘undefined’ state. OK, suddenly we have nulls so it gets tempting to allow:
… and so it goes. If you allow dictionaries on both axes, you end up with a cross-tab, and (of course) Arthur has made sure the arithmetic primitives all work on cross-tabs too.
Summary¶
Kdb+ re-implements Kdb and offers speed and almost infinite memory. It looks to have a ready market, and it is hard to see how it can fail. K4 takes K to a new level of abstraction and technical brilliance. I just hope Arthur can finish it!