XML Reference Guide

Long before Yahoo! unveiled Yahoo! Web Services, developers were accessing the information in Google's database of 8 billion web pages through a web services interface.

Part of Google's offering is fairly standard. Using the API, you can perform the same searches you can perform through the web interface, except that instead of a web page, you get back an XML document of results, which you can then analyze or display however you like. The web services API actually uses the exact same queries as the web site, including modifiers such as site: and intext:. (We'll look more closely at the actual searches in a future guide section.)

Searches are done using SOAP messages. For example, if I were searching for conspiracy theories, I might send a SOAP message such as:

<?xml version='1.0' encoding='UTF-8'?>

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
      xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" 
      xmlns:xsd="http://www.w3.org/1999/XMLSchema">
  <SOAP-ENV:Body>
    <ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch" 
         SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
      <key xsi:type="xsd:string">00000000000000000000000000000000</key>
      <q xsi:type="xsd:string">government conspiracies</q>

      <start xsi:type="xsd:int">0</start>
      <maxResults xsi:type="xsd:int">10</maxResults>
      <filter xsi:type="xsd:boolean">true</filter>
      <restrict xsi:type="xsd:string"></restrict>

      <safeSearch xsi:type="xsd:boolean">false</safeSearch>
      <lr xsi:type="xsd:string"></lr>
      <ie xsi:type="xsd:string">latin1</ie>
      <oe xsi:type="xsd:string">latin1</oe>

    </ns1:doGoogleSearch>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

The response would also be a SOAP message, such as:

<?xml version='1.0' encoding='UTF-8'?>

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
         xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" 
         xmlns:xsd="http://www.w3.org/1999/XMLSchema">

  <SOAP-ENV:Body>
    <ns1:doGoogleSearchResponse xmlns:ns1="urn:GoogleSearch" 
             SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
      <return xsi:type="ns1:GoogleSearchResult">
        <documentFiltering xsi:type="xsd:boolean">false</documentFiltering>
        <estimatedTotalResultsCount xsi:type="xsd:int">718000</estimatedTotalResultsCount>

        <directoryCategories xmlns:ns2="http://schemas.xmlsoap.org/soap/encoding/" 
                xsi:type="ns2:Array" ns2:arrayType="ns1:DirectoryCategory[0]"></directoryCategories>
        <searchTime xsi:type="xsd:double">0.194871</searchTime>
        <resultElements xmlns:ns3="http://schemas.xmlsoap.org/soap/encoding/"
                  xsi:type="ns3:Array" ns3:arrayType="ns1:ResultElement[3]">
          <item xsi:type="ns1:ResultElement">
            <cachedSize xsi:type="xsd:string">40k</cachedSize>

            <hostName xsi:type="xsd:string"></hostName>
            <snippet xsi:type="xsd:string"> <b>Conspiracy</b> related news and discusion. Main topics include <b>government</b> cover-ups, freemasons, illuminati, aliens, ufo's, secret societies, politics, ...</b></snippet>

            <directoryCategory xsi:type="ns1:DirectoryCategory">
              <specialEncoding xsi:type="xsd:string"></specialEncoding>
              <fullViewableName xsi:type="xsd:string"></fullViewableName>
            </directoryCategory>
            <relatedInformationPresent xsi:type="xsd:boolean">true</relatedInformationPresent>

            <directoryTitle xsi:type="xsd:string"></directoryTitle>
            <summary xsi:type="xsd:string"></summary>
            <URL xsi:type="xsd:string">http://www.abovetopsecret.com/</URL>
            <title xsi:type="xsd:string"><b>AboveTopSecret.com - The Internet's Most Popular Conspiracy ...</b></title>

          </item>
          <item xsi:type="ns1:ResultElement">
            <cachedSize xsi:type="xsd:string">100k</cachedSize>
            <hostName xsi:type="xsd:string"></hostName>
            <snippet xsi:type="xsd:string">Alternative news and commentary on corporate/<b>government</b> fraud and criminal <b>conspiracy</b>. Features "Behind the Scenes in the Beltway," a column by <b>government</b> ...</snippet>

            <directoryCategory xsi:type="ns1:DirectoryCategory">
              <specialEncoding xsi:type="xsd:string"></specialEncoding>
              <fullViewableName xsi:type="xsd:string"></fullViewableName>
            </directoryCategory>
            <relatedInformationPresent xsi:type="xsd:boolean">true</relatedInformationPresent>

            <directoryTitle xsi:type="xsd:string"></directoryTitle>
            <summary xsi:type="xsd:string"></summary>
            <URL xsi:type="xsd:string">http://www.almartinraw.com/</URL>
            <title xsi:type="xsd:string">Al Martin Raw: Criminal Government Conspiracy</title>

          </item>
...
        <endIndex xsi:type="xsd:int">10</endIndex>
        <searchTips xsi:type="xsd:string"></searchTips>
        <searchComments xsi:type="xsd:string"></searchComments>
        <startIndex xsi:type="xsd:int">1</startIndex>

        <estimateIsExact xsi:type="xsd:boolean">false</estimateIsExact>
        <searchQuery xsi:type="xsd:string">government conspiracies</searchQuery>
      </return>
    </ns1:doGoogleSearchResponse>
  </SOAP-ENV:Body>

</SOAP-ENV:Envelope>

Depending on what I wanted to do with the data, I could either extract specific pieces of information (such as the estimated number of results) or I could extract the payload of the SOAP message, the contents of the doGoogleSearchResponse element, and use XSLT to display it on a web page.

The actual formats of these messages are detailed in a WSDL file, so if your programming environment supports complex types -- notice that the results come back as nested XML documents rather than simple types -- you can use the WSDL file to generate a client.

Not that you would necessarily need to. The Google API Software Developer's Kit includes sample code for both Java and .NET, including a Java library that makes requesting and working with results fairly straightforward.

Google actually provides a web services interface to three of its offerings. The first, and most obvious, is search, where you can submit a query and receive a list of pages that satisfy that query. The second is Google's cache database, where you can submit a URL and get back the contents of the page the last time Google indexed it. The third is Google's spell-check, which takes a word (or an attempt at spelling a word) and gives back a list of suggested spellings.

Any one (or more) of these searches can be integrated into your own applications. For example, you could write an application that does a spell-check on user-supplied text, or one that enables you to determine whether your web page is in the top ten results for a particular keyword.

Not that there aren't restrictions of course. The Google Web Service API has been "experimental" and "in beta" since its release several years ago, which means that Google can justify limiting the functionality of the service. For example, you, as a developer, can only request 1000 results per day with your application, no matter how many people are visiting it. (Although I should point out that some developers get around this problem by requiring users to supply their own license key, which Google provides free for tracking purposes.) You are also limited to 10 results at a time, and only the first 1000 results are available for any given search. Commercial applications are permissible with written approval from Google, but are still limited to 1000 requests per day.

Next, a look at actually using the API.