Improve Your Search Engine Ranking with AJAX

Date: Jul 28, 2006

Navigational elements embedded in your web pages can deteriorate your search engine ranking and reduce the responsiveness of your web site. Ivan Pepelnjak describes how you can use AJAX to solve both problems.

Many well-designed web sites contain significant amounts of navigational information together with the actual content. The HTML markup used for navigation can impact your search engine ranking as well as increase the page download time experienced by your visitors. In this article, you'll see how you can use AJAX to create more focused, faster-loading web pages.

Separate Navigation from Content

Let's start with an example. Consider the article you're reading right now—it has the following items:

A header with predefined links to various parts of Informit.
A line below the header positioning this article within the content structure of Informit. (This is the only navigational element actually related to the content.)
A sidebar on the right with links to similar or popular articles and promotions.
A footer, yet again with predefined links that never change.

This mixture can affect the way search engines index your data:

Because search engines can't distinguish content from navigation text, they index everything they find on your page. Since the keywords from the content are mixed with unrelated navigational keywords, the relevancy of the content is reduced. Some search engines claim that they can discover repetitive text on crawled pages (for example, fixed headers and footers) and remove them. Don't depend on that possibility, though; even if they do it, it's not always reliable.
Outgoing links on web pages affect the page rank of your content. While this might increase the page rank of popular articles because many pages link to them, the content to which all pages link (for example, Privacy policy) usually gets the highest page rank—which isn't always what you've intended. On one of my web sites, the highest-ranking page was the form used to send messages to the web administrator—not something you want the users to find first on Google.

The navigational elements added to a web page can also impact users with low-speed Internet access, more so if the web page doesn't use positioned DIV elements (which are used by Informit) but rather uses tables. In this case, the whole table has to be loaded in some web browsers before it's displayed to the user.

Traditionally, web designers have tried to separate navigation from content with framesets or by building the whole navigational structure with extensive JavaScript code. Both approaches have drawbacks, so it's no wonder that a large number of major web sites avoid framesets.

Techniques used within the AJAX framework give you an alternate solution to this problem:

Each web page contains only the placeholders for the navigational elements and the actual content.
The navigational elements are loaded after the web page has loaded through inline frames (IFRAME) or by using the XmlHttpRequest object.
The content of the navigational elements is then merged with the web page content, resulting in a clean page with no embedded frames.

Before redesigning your web pages with this approach, you need to consider the following issues:

Search engines will see only the initial web page. You have to make sure that the page contains links to related pages or to a sitemap. The navigation line above the article header (breadcrumbs) and the links within the Article information section of the page on Informit serve this purpose very nicely.
Visitors who have disabled JavaScript in their browsers will have the same view of the page as the search engines. You have to ensure that their limited view will not fatally impair the functionality of your web site.
You might want to retain some static content on your page. For example, the Informit logo and the copyright information would have to be shown to all visitors at all times.

When you've decided which parts of your navigational structure should be attached with the page and which parts should be separated from it, you're ready for the next step.

Design Your Web Page

The first step in separation of content and navigation is the creation of placeholders on the web page where the navigational elements will be inserted. For each contiguous area of navigational elements, you should create a separate DIV element with a unique id, so that you can identify it later in your JavaScript code. To prevent excessive flicker during the page composition, the DIV elements above or to the right of the content should be sized close to the actual size of your navigational elements so the content won't shift when you replace them with the desired HTML code. The most versatile means of achieving this goal is to insert an empty, properly sized DIV element within the placeholder.

In the case of Informit, the page structure is already very well designed, with the DIV elements already in place, as shown in Figure 1.

Figure 1 Page structure of an Informit article.

You would only need to remove the content from the navigational DIV elements and insert an empty box where the header will appear, resulting in a web page structure similar to this example. (For the sake of simplicity, we'll conveniently forget the requirement to have logo and copyright information embedded in every page.) Here's the code:

<div id="header">
  <div style="height: 100px; width: 100%"></div>
</div>
<div id="contentArticle">
  <div id="firstCol">
  ... article content ....
  </div>
  <div id="secondCol" ></div>
</div>
<div id="footer"></div>

The navigational elements that have been removed from the web pages have to be re-created as independent pages. You should use static HTML files for static content (this will allow the content to be cached regardless of the web server you use) and create a server-side script that will display the dynamic elements based on the web page loading them. For Informit, each web page has a unique article identifier (the p= parameter in the URL), so you would need to create a server-side script that would accept the article identifier and create the right-hand column. In most cases, you can reuse server-side code that creates inline navigational elements.

After the web pages have been redesigned, you're ready to implement the AJAX portion of this solution. As always, you can work with inline frames (IFRAME elements) or use an XmlHttpRequest object.

Inline Frames

You should use inline frames if you're concerned about browser compatibility. Some older browsers support IFRAME elements, but not the XmlHttpRequest object. There are also a few other reasons to use this approach:

The content being loaded into an IFRAME is displayed in the browser during the loading process, giving the end user a visual indication of progress.
Page caching always works with the content loaded into an IFRAME. Some versions of Opera are not very good at handling cached responses with the XmlHttpRequest object.

To implement this solution, insert an empty IFRAME into each DIV container and add a short JavaScript statement after each IFRAME, as follows:

<div id="header">
  <div style="height: 100px; width: 100%"></div>
  <iframe id="header_iframe" style="height: 0px;"></iframe>
  <script>loadIframe("header","/navigation/header.html")</script>
</div>

The id of the IFRAME should be equal to the id of the placeholder suffixed by _iframe. The loadIframe function takes two arguments: the id of the placeholder and the URL to load into it.

The loadIframe function that starts the load process is very simple:

function loadIframe(id,url) {
  try {
    var iframeObj = document.getElementById(id+"_iframe");
    iframeObj.src = url ;
  } catch (err) {
    alert("cannot load "+url+" into "+id) ;
  }
}

However, there's no mechanism to notify the requesting page that the desired content has been loaded into the placeholder IFRAME. The loaded content thus has to notify the parent page (via a JavaScript call) that it's ready to be used. The best moment to do this is after the page load has finished. The BODY tag in the IFRAME content should thus contain an onLoad event:

<body onload="contentLoaded('header')" style="margin: 0px 0px;
padding: 0px 0px">

The contentLoaded function executed in the context of the IFRAME will extract the HTML content of the body and pass it to a function executing in the context of the parent page, which will use it to populate the placeholder:

contentLoaded, executed in the IFRAME context:

function contentLoaded(parentID) {
  var myContent = document.body.innerHTML ;
  parent.copyContent(parentID,myContent);
}

copyContent, executed in the context of the parent web page:

function copyContent(id,content) {
  try {
    var placeholder = document.getElementById(id) ;
    placeholder.innerHTML = content;
  } catch (err) {
    alert("Cannot copy HTML content into "+id);
  }
}

A careful reader should wonder by now about the reasons for this complexity—wouldn't it be simpler to load the navigational content in the IFRAME elements? As it turns out, there are a few caveats with that simplistic approach:

IFRAME has fixed height and width. If the content exceeds one or the other, it gets cropped or the IFRAME gets "decorated" with the scroll bars. HTML markup copied into a DIV element in the parent page is always perfectly sized.
When left within an IFRAME, the links (A elements) in the navigation content would load new pages within the IFRAME (unless you append target="_parent" to each link).
JavaScript event handlers attached to navigation elements would work within the context of IFRAME if left there. If you move the navigational content to the main page, the event handlers get access to functions and variables defined within the main page.

Working with `XmlHttpRequest`

If your audience primarily uses later versions of Internet Explorer or Gecko-based browsers (Mozilla, Firefox, Netscape 7), you could decide to use the XmlHttpRequest object to download additional content into your web pages. The initial step is very similar to the one described earlier. For each placeholder, you need a JavaScript function call to start the load process:

<div id="header">
  <div style="height: 100px; width: 100%"></div>
  <script>loadContent("header","/navigation/header.html")</script>
</div>

The loadContent function is radically different, however: It creates a new XmlHttpRequest object, assigns an event handler to it, and starts the asynchronous loading process:

function loadContent(id,url) {
  try {
    var rq = new XMLHttpRequest() ;
    rq.open("GET", url, true);
    rq.onreadystatechange =  function() { contentLoaded(rq,url,id) }
    rq.send(null);
  } catch (err) {
    alert("cannot load "+url+" into "+id) ;
  }
}

The callback function contentLoaded checks the readiness of the XmlHttpRequest object and the completion status (if the request has completed), and extracts the HTML markup from the response. The easiest way to extract the HTML code (unless you use XHTML, in which case you can use the XML DOM interface) is to use string-handling functions to find the text between the <body> and </body> tags:

function contentLoaded(rq,url,id) {
  try {
    if (rq.readyState != 4) { return; }
    if (rq.status != 200) { alert("failed to load "+url); return; }

    var txt = rq.responseText ;

// find the start of the <body> tag
    var startBodyTag = txt.indexOf("<body")

// find the end of the <body> tag, skipping any attributes
    var endOfStartTag = txt.indexOf(">",startBodyTag+1)

// find the </body> tag, copy until end of text if missing
    var endBodyTag = txt.indexOf("</body")
    if (endBodyTag == -1) { endBodyTag = txt.length ; }

// extract the actual content
    var bodyContent = txt.substring(endOfStartTag+1,endBodyTag)

    if (bodyContent) {
      var placeholder = document.getElementById(id) ;
      placeholder.innerHTML = bodyContent;
    }
  } catch (err) {
    alert("cannot load "+url+" into "+id) ;
  }
}

Compared to the IFRAME-based method described earlier, using the XmlHttpRequest object has the following benefits:

The code is cleaner and doesn't rely on context switches between pages.
The XmlHttpRequest object gives you the ability to detect and process errors through its readyState and status properties. The only indication that something went wrong with an IFRAME-loaded content is the lack of the callback call.
You can implement parallel load of the content elements (as shown in this section) or sequence the load requests to minimize bandwidth utilization.

Summary

In this article, you've learned how you can separate the content of your web page from the navigational elements surrounding it. The separation results in a more focused presentation of your content to search engines, as well as reduced loading time for users with low-speed Internet access, since the actual content is shown to the user before the navigational elements are downloaded.

When redesigning your web page to take advantage of this solution, keep in mind that some basic form of navigation has to remain on the page in order to allow search engines and users who have decided to disable JavaScript to navigate between the pages on your site.