Googling with APL+Win

by William R. Parke

Transcribed from the printed issue; not yet reviewed. Read it in the PDF of the issue, page 75

Introduction¶

The Google Web API allows developers to incorporate customized access to Google’s Internet database in their applications. It also provides a simple and useful example for learning how to use Extensible Markup Language (XML). The purpose of this article is to describe the essential steps in accessing the Google API from APL+Win, including techniques for handling XML documents and transmitting them over the Internet, and some comments on developing Google API search applications.

In presenting this information, it is assumed that the reader has some familiarity with XML notation and the APL+Win ⎕wi system function, including creating ActiveX objects, using object notation, and setting and referencing object properties.

The Google Web API¶

Google publishes an Application Programming Interface (API) that allows access to their Internet database with three search methods:

doGoogleSearch

the standard Google query operation

doGetCachedPage

returns Google’s cached version of a Web page

doSpellingSuggestion

returns a suggested spelling for words and names in Google’s database

(this is what you see when Google says “Did you mean: …” at the top of the page when you mis-spell a search term)

Each method’s request and response are sent to and returned from Google.com as SOAP messages which are specially-conforming XML documents. The XML elements in these messages identify the parameters of the search request and the named data values comprising the search results. Since these are simply text strings, they can be constructed or parsed easily in APL.

Four processes are needed to implement the Google API:

Obtain a Google user-authorization key and developer toolkit

The first step in using the Google API is to download the developer’s toolkit and obtain a free user-authorization key from Google. The key must be sent with each search request and entitles the user to 1,000 requests per day. Go to Google.com and follow the links to Services & Tools | Google Web APIs (as of this writing still www.google.com/apis/).

It is helpful to look at the APIs_Reference.html document and the \soap-samples\*.xml request and response examples in the Google Web API toolkit. XML is essentially a method of labelling data, enclosing values in pairs of name-tags of the form <name>value</name>, and all the element names and hierarchies needed for the Google API are presented in these examples.

Opening the soap-sample XML files in a Web browser is a good way to view them, since the browser will display the tree hierarchy that structures the XML document. For the SOAP request messages, the outer elements, <SOAP-ENV:Envelope> and <SOAP-ENV:Body> are the same in all methods. The elements of consequence are contained in <ns1:doGoogleSearch>, <ns1:doGetCachedPage> and <ns1:doSpellingSuggestion>. In the response messages, the <return> element contains Google’s results for each method.
Implement methods for handling XML

XML can be manipulated easily in APL, either with an XML parser object or with traditional APL character-vector methods. For simplicity, all the methods described in this article use Microsoft’s MSXML DOMDocument object which should be available on any Windows-based system with Internet Explorer 4 or above installed. MSXML3 currently is the version installed with Internet Explorer. You may prefer working with MSXML4 which has some improvements and can be obtained online from Microsoft. If you decide to install it, I recommend getting the version with the optional SDK which you’ll find helpful once you get hooked on XML. However, MSXML version 3 has all the functionality you need for this project.
Connect your application to the Internet

To connect to Google.com from your application, you can use an MSXML HTTP object, as shown in the examples below, or APL+Win’s ⎕ni sockets. AplWebServices or other 3rd-party socket managers also can be used to handle the connections.
Manage the search results

What do you want to do with collections of search results from Google? There are many possible uses for API-managed searches, but the principle benefit is the ability to archive and work with the data over time. Some comments on creating search applications will be offered below.

Gentlemen, Start Your (Search) Engines¶

It is convenient to prepare XML templates for each Google API search method, either as variables in the workspace or as external files, by taking the examples that come with the developer toolkit and removing the content (values) from the search-parameter elements. The templates then can be loaded into the MSXML object and “filled-in” with the values specific to each search request. To illustrate, let’s start with the simplest API method, doSpellingSuggestion.

Open the file doSpellingSuggestion.xml from the API toolkit and read it into a character vector in APL (or copy and paste from a text editor), then remove everything between the <key xsi:type="xsd:string"> and </key> tags and the <phrase xsi:type="xsd:string"> and </phrase> tags. Save the changes to a reusable (global) variable or to a file.

Next, create an MSXML DOMDocument object:

'xmlobj' ⎕wi 'Create' 'MSXML2.DOMDocument'

This will default to version 3 even if version 4 is installed. If you have MSXML4 installed and want to use it, create 'MSXML2.DOMDocument.4.0' instead to force version 4.

Now load the XML document into the MSXML object:

⍝ string variable:
'xmlobj' ⎕wi 'loadXML' xml_doSpellingSuggestion
⍝ or file name:
'xmlobj' ⎕wi 'load' 'c:\googleapi\soap-samples\doSpellingSuggestion.xml'

On success, the result of the method-call is 1, otherwise the document must contain an XML validation error. As an aside, you may find it useful to place the following statements in a function or user-command to help track down these errors, especially if you are generating XML from code:

'Error: ', 'xmlobj' ⎕wi 'parseError.reason'
'Line : ', 'xmlobj' ⎕wi 'parseError.line'
'Char : ', 'xmlobj' ⎕wi 'parseError.linepos'

MSXML object methods are used to set the content of the XML document elements specific to a particular search request. In MSXML syntax, an XML element is referenced as a “node” in the tree hierarchy of the parsed XML document, and the MSXML method selectSingleNode is used to specify an XML element or “node” to set or reference. The argument to selectSingleNode is an XPath expression, which in its simplest form is an XML element name, most often prefixed with ‘//’ to reference it relative to the document root. (There are many other ways to identify a specific node. You can learn more about XPath in the MSXML SDK or other resources. Just be aware that XML is case-sensitive, so you must specify element names exactly as they appear in the examples.) The result of the selectSingleNode method is an object representing that element’s node in the document tree, and the node object’s text property contains the element’s value or content.

In the case of the doSpellingSuggestion SOAP message, only two parameters are required: the Google user-authorization key, and the word, name or phrase to check for spelling. For the example below, I have created a string variable google_user_key that contains my user authorization key. I use this in ⎕wi object notation to set the text property of the document node object representing the key element. By doing this, I am setting the content of the XML <key> element in the document.

'xmlobj' ⎕wi 'selectSingleNode("//key").text' google_user_key
'xmlobj' ⎕wi 'selectSingleNode("//phrase").text' 'apl sftwar'

Now if you reference the node’s text property, the value is shown to be what was just set:

      'xmlobj' ⎕wi 'selectSingleNode("//key").text'
EGJk8grQ1jL5LHqs

      'xmlobj' ⎕wi 'selectSingleNode("//phrase").text'
apl sftwar

(In general use, you would want to wrap such methods in APL functions which validate the existence of the specified node object before attempting to set or reference it.)

You can can inspect the completed XML document string by referencing the XML object’s xml property:

document ← 'xmlobj' ⎕wi 'xxml'

Note that this is a read-only property. If you manually edit the string variable outside the MSXML DOM object, you must use the loadXML method to place it back into the object, as shown above.

Once the SOAP request is set, it can be sent to Google, in this example by using an MSXML ServerHTTP object (if you have MSXML4 installed, you can create 'MSXML2.ServerXMLHTTP.4.0' instead).

'httpobj' ⎕wi 'Create' 'MSXML2.ServerXMLHTTP'
⍝                     method url                                  async
'httpobj' ⎕wi 'open' 'POST' 'http://api.google.com/search/beta2' 0
⍝ send the property of the XML DOM which contains the SOAP request string
'httpobj' ⎕wi 'send' ('xmlobj' ⎕wi 'xxml')

The third argument to open, 0 (zero), turns off asynchronous mode, causing the XMLHTTP object’s send method to wait until it connects and receives a response. When the send method returns, you can check its status property, then retrieve Google’s SOAP response and handle it with the XML DOM object or APL code. For doSpellingSuggestion, the response is a simple string, or nothing if Google has no suggestion:

:if 200 = 'httpobj' ⎕wi 'status'
    ⍝ transfer the returned SOAP message to the XML DOM object
    :if 1 = 'xmlobj' ⎕wi 'loadXML' ('httpobj' ⎕wi 'responseXML.xxml')
        ⍝ extract the response from the <return> element
        spelling ← 'xmlobj' ⎕wi 'selectSingleNode("//return").text'
    :endif
:endif

That’s all there is to the process. The doGetCachedPage request differs only in that it sends a URL instead of a phrase, and Google returns a base-64 encoded string which must be converted to HTML text. Following the example above:

'xmlobj' ⎕wi 'loadXML' xml_doGetCachedPage
'xmlobj' ⎕wi 'selectSingleNode("//key").text' google_user_key
'xmlobj' ⎕wi 'selectSingleNode("//url").text' 'www.apl2000.com/new.htm'
 ... XMLHTTP steps not shown as they are exactly the same as above ...
html ← fromBase64 'xmlobj' ⎕wi 'selectSingleNode("//return").text'

doGoogleSearch¶

doGoogleSearch is more complicated. It takes ten request parameters and returns ten general data elements describing the search results plus ten more elements for each matching result item. To use this method, load the XML object with a doGoogleSearch template and set the search parameters:

'xmlobj' ⎕wi 'loadXML' xml_doGoogleSearch
'xmlobj' ⎕wi 'selectSingleNode("//key").text'        google_user_key
'xmlobj' ⎕wi 'selectSingleNode("//q").text'          'apl'
'xmlobj' ⎕wi 'selectSingleNode("//start").text'      '0'
'xmlobj' ⎕wi 'selectSingleNode("//maxResults").text' '10'
'xmlobj' ⎕wi 'selectSingleNode("//filter").text'     'true'
'xmlobj' ⎕wi 'selectSingleNode("//safeSearch").text' 'true'
'xmlobj' ⎕wi 'selectSingleNode("//restrict").text'   'linux.(countryFR|countryBE)'
'xmlobj' ⎕wi 'selectSingleNode("//lr").text'         'lang_fr|lang_de'
(<ie> and <oe> are required but ignored by Google, so just send defaults)

Note that the search elements may be set in any order, and numeric values must be specified in character format. The query string element <q> may include various qualifiers such as double-quotes, exclude (-), and several filtering terms such as intitle:, site:, daterange:, etc. In addition, Google accepts country, language and topic restrictions to further refine the results. The APIs_Reference.html document in the API toolkit explains all these terms and how to use them to modify searches.

After sending the request to Google, the response data are retrieved from the child elements contained in the <return> element. You can reference them in any order, or skip ones you’re not interested in. Here is the complete list:

q ← 'xmlobj' ⎕wi 'selectSingleNode("//searchQuery").text'
r ← 'xmlobj' ⎕wi 'selectSingleNode("//estimatedTotalResultsCount").text'
x ← 'xmlobj' ⎕wi 'selectSingleNode("//estimateIsExact").text'
t ← 'xmlobj' ⎕wi 'selectSingleNode("//searchTime").text'
s ← 'xmlobj' ⎕wi 'selectSingleNode("//startIndex").text'
e ← 'xmlobj' ⎕wi 'selectSingleNode("//endIndex").text'
f ← 'xmlobj' ⎕wi 'selectSingleNode("//documentFiltering").text'
p ← 'xmlobj' ⎕wi 'selectSingleNode("//searchTips").text'
c ← 'xmlobj' ⎕wi 'selectSingleNode("//searchComments").text'

One additional element, <directoryCategories>, must be handled differently, since it may contain zero or more DirectoryCategory arrays as child elements. You can determine how many arrays are present by checking the arrayType attribute which looks like this: ns2:arrayType="ns1:DirectoryCategory[2]"

n ← 'xmlobj' ⎕wi 'selectSingleNode("//directoryCategories").getAttribute("ns2:arrayType")'
n ← ((~⎕io)+n⍳'[')↓n
n ← ⎕fi ((-⎕io)+n⍳']')↑n

However, it is easier to simply loop on an MSXML selectNodes object which references all matching XML elements as a group. In this instance, it is useful to create additional temporary named objects for use with ⎕wi:

:if 0 ≠ 'xmlobj' ⎕wi 'selectNodes("directoryCategories/item")'
    'xmlobj' ⎕wi 'selectNodes("directoryCategories/item") > snodes'
    :for i :in (-⎕io)+⍳'snodes' ⎕wi 'length'
        'snodes' ⎕wi 'Item(',(⍕i),') > inode'
        encoding ← 'inode' ⎕wi 'selectSingleNode("specialEncoding").text'
        viewname ← 'inode' ⎕wi 'selectSingleNode("fullViewableName").text'
        ⍝ ... do something here with each data pair ...
    :endfor
:endif

doGoogleSearchResponse may contain up to ten <resultElement> items representing the “hits” or Web pages matching the search query. To extract their contents, run a loop using the MSXML selectNodes method and named objects as described above.

:if 0 ≠ 'xmlobj' ⎕wi 'selectNodes("//return/resultElements/item")'
    'xmlobj' ⎕wi 'selectNodes("//return/resultElements/item") > snodes'
    :for i :in (-⎕io)+⍳'snodes' ⎕wi 'length'
        'snodes' ⎕wi 'Item(',(⍕i),') > inode'
        title    ← 'inode' ⎕wi 'selectSingleNode("title").text'
        url      ← 'inode' ⎕wi 'selectSingleNode("URL").text'
        size     ← 'inode' ⎕wi 'selectSingleNode("cachedSize").text'
        host     ← 'inode' ⎕wi 'selectSingleNode("hostName").text'
        snippet  ← 'inode' ⎕wi 'selectSingleNode("snippet").text'
        rip      ← 'inode' ⎕wi
            'selectSingleNode("relatedInformationPresent").text'
        dirtitle ← 'inode' ⎕wi 'selectSingleNode("directoryTitle").text'
        summary  ← 'inode' ⎕wi 'selectSingleNode("summary").text'
        encoding ← 'inode' ⎕wi
            'selectSingleNode("directoryCategory/specialEncoding").text'
        viewname ← 'inode' ⎕wi
            'selectSingleNode("directoryCategory/fullViewableName").text'
        ⍝ ... do something here with each result set ...
    :endfor
:endif

Another way to do this is to reference a selectNodes object for each data item of interest:

urls ← ⍬
:if 0 ≠ 'xmlobj' ⎕wi 'selectNodes("//URL")'
    'xmlobj' ⎕wi 'selectNodes("//URL") > snodes'
    :for i :in (-⎕io)+⍳'snodes' ⎕wi 'length'
        urls ← urls, ⊂ 'snodes' ⎕wi 'Item(',(⍕i),').text'
    :endfor
:endif

sizes ← ⍬
:if 0 ≠ 'xmlobj' ⎕wi 'selectNodes("//cachedSize")'
     'xmlobj' ⎕wi 'selectNodes("//cachedSize") > snodes'
     ⍝ as above ...
...

There are other ways to work with MSXML objects, depending on the needs of your application, but for the Google API, these methods should be sufficient.

In Search Of…¶

Missing from these examples is what to do with the search data obtained. The results could be displayed as an HTML page similar to Google’s own browser format by using an XML Stylesheet Transform (XSLT) or generating HTML directly from APL code. Of more value would be to organize and save the data so that topics of interest could be researched offline and tracked for changes over time. You might like to study changes in page rankings or compare newer and older versions of selected Web pages. Any search query you type in more than once would be a good candidate for an API-managed search and archival system, even more convenient if special search terms or filters are used. But no matter how sophisticated your search techniques, still a fair number of the results Google returns often are not applicable to your needs. The ability to build, save and update custom result databases could be quite useful and timesaving if you didn’t have to sift through pages of unwanted items each time you revisited a topic.

Beyond the Google API, the techniques for working with XML documents are interesting and useful by themselves. Once you start, you see uses for XML everywhere. It is ideal for configuration files, hierarchical data, and any situation where portability of named values is desired. As XML becomes the standard for data exchange, it should be well worth the time and effort to learn the rudiments of this format. Try it out with the Google API.