This chapter is from the book
This chapter is from the book
This chapter is from the book
HTTP and REST
Now that you have your instances proxied and ready to accept requests, it's time to think about how to accept requests. In general, it's a bad practice to reinvent the wheel when you can just use another protocol that's already been established and well tested. There are entire books on using HTTP and REST to build your own SaaS, but this section provides the basic details.
Although you can use HTTP in many ways, including SOAP, the simplest of all these is Representational State Transfer (REST), which was officially defined in 2000 by Roy Fielding in a doctoral dissertation "Architectural Styles and the Design of Network-based Software Architectures" (http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm). It uses HTTP as a communication medium and is designed around the fundamental idea that HTTP already defines how to handle method names, authentication, and many other things needed when working with these types of communications. HTTP is split into two different sections: the header and the body (not to be confused with the HTML <head> and <body> tags), each of which is fully used by REST.
This book uses REST and XML for most of the examples, but this is not the only option and may not even suite your specific needs. For example, SOAP is still quite popular for many people because of how well it integrates with Java. It also makes it easy for other developers to integrate with your APIs if you provide them with a Web Service Definition Language (WSDL) that describes exactly how a system should use your API. The important point here is that the HTTP protocol is highly supported across systems and is one of the easiest to use in many applications because much of the lower-level details, such as authentication, are already taken care of.
The Header
The HTTP header describes exactly who the message is designed for, and what method the user is instantiating on the recipient end. REST uses this header for multiple purposes. HTTP method names can be used to define the method called and the arguments (path) which is sent to that method. The HTTP header also includes a name, which can be used to differentiate between applications running on the same port. This shouldn't be used for anything other than differentiating between applications because it's actually the DNS name and shouldn't be used for anything other than a representation of the server's address.
The method name and path are both passed into the application. Typically you want to use the path to define the module, package, or object to use to call your function. The method name is typically used to determine what function to call on that module, package, or object. Lastly, the path also contains additional arguments after the question mark (?) that usually are passed in as arguments to your function. Now take a look at a typical HTTP request:
GET /module_name/id_argument?param1=value1¶m2=value2
In this example, most applications would call module_name.get(id_argument,param1=value1,param2=value2) or module_name.get(id_argument,{param1=value1,param2=value2}). By using this simple mapping mechanism, you're decoupling your interface (the web API) from your actual code, and you won't actually need to call your methods from the website. This helps greatly when writing unit tests.
Many libraries out there can handle mapping a URI path to this code, so you should try to find something that matches your needs instead of creating your own. Although REST and RESTful interfaces are defined as using only four methods, most proxies and other systems, including every modern web browser, support adding your own custom methods. Although many REST developers may frown on it, in general it does work, and when a simple CRUD interface isn't enough, it's much better than overloading an existing function to suit multiple needs. The following sections reference some of the most common HTTP headers and how you can use them in a REST API.
If-Match
The most interesting header that you can provide for is the If-Match header. This header can be used on any method to indicate that the request should be performed only if the conditions in the header represent the current object. This header can be exceptionally useful when you operate with databases that are eventually consistent, but in general, because your requests can be made in rapid succession, it's a good idea to allow for this so that they don't overwrite each other. One possible solution to this is to provide for a version number or memento on each object or resource that can then be used to ensure that the user knows what the value was before it replaces it.
In some situations, it may be good to require this field and not accept the special * case for anyone other than an administrative user. If you require this field to be sent and you receive a request that doesn't have it, you should respond with an error code of 412 (Precondition Failed) and give the users all they need to know to fill in this header properly. If the conditions in this header do not match, you must send back a 412 (Precondition Failed) response. This header is typically most used when performing PUT operations because those operations override what's in the database with new values, and you don't know if someone else may have already overridden what you thought was there.
If-Modified-Since
The If-Modified-Since header is exceptionally useful when you want the client to contain copies of the data so that they can query locally. In general, this is part of a caching system used by most browsers or other clients to ensure that you don't have to send back all the data if it hasn't been changed. The If-Modified-Since header takes an HTTP-date, which must be in GMT, and should return a 304 (Not Modified) response with no content.
If-Unmodified-Since
If you don't have an easy way to generate a memento or version ID for your objects, you can also allow for an If-Unmodified-Since header. This header takes a simple HTTP date, formatted in GMT, which is the date the resource was last retrieved by the client. This puts a lot of trust in the client, however, to indicate the proper date. It's generally best to use the If-Modified header instead, unless you have no other choice.
Accept
The Accept header is perhaps the most underestimated header in the entire arsenal. It can be used not only to handle what type of response to give (JSON, XML, and so on), but also to handle what API version you're dealing with. If you need to support multiple versions of your API, you can support this by attaching it to the content type. This can be done by extending the standard content types to include the API version number:
Accept: text/xml+app-1.0
This enables you to specify not only a revision number (in this case, 1.0) and content type, but also the name of the application so that you can ensure the request came from a client that knew who it was talking to. Traditionally, this header will be used to send either HTML, XML, JSON, or some other format representing the resource or collection being returned.
Authorization
The Authorization header can be used just like a standard HTTP authentication request, encoding both the password and the username in a base64 encoded string, or it can optionally be used to pass in an authentication token that eventually expires. Authentication types vary greatly, so it's up to you to pick the right version for your application. The easiest method is by using the basic HTTP authentication, but then you are sending the username and password in every single request, so you must ensure that you're using SSL if you're concerned about security.
In contrast, if you choose to use a token or signing-based authentication method, the user has to sign the request based on some predetermined key shared between the client and server. In this event, you can hash the entire request in a short string that validates that the request did indeed come from the client. You also need to make sure to send the username or some other unique identifier in this header, but because it's not sending a reversible hash of the password, it's relatively safe to send over standard HTTP. We won't go into too much depth here about methods of hashing because there are a wide variety of hashing methods all well-documented online.
The Body
The body of any REST call is typically either XML or JSON. In some situations, it's also possible to send both, depending on the Accept header. This process is fairly well documented and can be used to not only define what type of response to return, but also what version of the protocol the client is using. The body of any request to the system should be in the same format as the response body.
In my applications, I typically use XML only because there are some powerful tools, such as XSLT, that can be used as middleware for authorization purposes. Many clients, however, like the idea of using JSON because most languages serialize and deserialize this quite well. REST doesn't specifically require one form of representation over the other and even enables for the clients to choose which type they want, so this is up to you as the application developer to decide what to support.
Methods
REST has two distinct, important definitions that you need to understand before continuing. A collection is a group of objects; in your case this usually is synonymous with either a class in object terms, or a table in database terms. A resource is a specific instantiation of a collection, which can be thought of as an instance in object terms or a row in a table in database terms. This book also uses the term property to define a single property or attribute on an instance in object terms, or a cell in database terms. Although you can indeed create your own methods, in general you can probably fit most of your needs into one of the method calls listed next.
GET
The GET method is the center of all requests for information. Just like a standard webpage, applications can use the URL in two parts; everything before the first question mark (?) is used as the resource to access, and everything after that is used as query parameters on that resource. The URL pattern can be seen here:
/collection_name/resource_id/property_name?query
The resource_id property_name and query in the preceding example are all optional, and the query can be applied to any level of the tree. Additionally, this tree could expand exponentially downward if the property is considered a reference. Now take a simple example of a request on a web blog to get all the comments of a post specified by POST-ID submitted in 2010. This query could look like this:
/posts/POST-ID/comments?submitted=2010%
The preceding example queries for the posts collection for a specific post identified as POST-ID. It then asks for just the property named comments and filters specifically for items with the property submitted that matches 2010%.
Responses to this method call can result in a redirection if the resource is actually located at a different path. This can be achieved by sending a proper redirect response code and a Location header that points to the actual resource.
A GET on the root of the collection should return a list of all resources under that path. It can also have the optional ?query on the end to limit these results. It's also a good idea to implement some sort of paging system so that you can return all the results instead of having to limit because of HTTP timeouts. In general, it's never a good idea to have a request that takes longer then a few seconds to return on average because most clients will assume this is an error. In general, most proxy systems will time out any connection after a few minutes, so if your result takes longer than a minute, it's time to implement paging.
If you use XML as your communication medium, think about implementing some sort of ATOM style next links. These simple tags give you a cursor to the next page of results, so you can store a memento or token of your query and allow your application to pick up where it left off. This token can then be passed in via a query parameter to the same URL that was used in the original request. In general, your next link should be the full URL to the next page of results. By doing this, you leave yourself open to the largest range of possibilities for implementing your paging system, including having the ability to perform caching on the next page of results, so you can actually start building it before the client even asks for it.
If you use Amazon Web Services and SimpleDB, it's generally a good idea to use the next_token provided by a SimpleDB query as your memento. You also need to provide enough information in the next link to build the entire original query, so just using the URL that was originally passed and adding the next_token to the end of the query is generally a good idea. Of course if this is a continuation, you have to replace the original next_token with the new one.
Performing a GET operation on the root path (/) should return a simple index that states all the available resource types and the URLs to those resource types. This machine-readable code should be simple enough that documentation is not required for new developers to build a client to your system. This methodology enables you to change around the URLs of the base resources without modifying your client code, and it enables you build a highly adaptable client that may adapt to new resources easily without making any modifications.
PUT
The HTTP definition of a PUT is replace, so if you use this on the base URL of a resource collection, you're actually requesting that everything you don't pass in is deleted, anything new is created, and any existing resources passed in are modified. Because PUT is logically equivalent to a SET operation, this is typically not allowed on a collection.
A PUT on a resource should update or create the record with a specific ID. The section after the collection name is considered the ID of the resource, and if a GET is performed after the PUT, it should return the resource that was just created or updated. PUT is intended as an entire replacement, but in general, if you don't pass in an attribute, that is assumed to be "don't change," whereas if you pass in an empty value for the attribute, you are actually requesting it to be removed or set to blank.
A PUT on a property should change just that specific property for the specified resource. This can be incredibly useful when you're putting files to a resource because those typically won't serialize very well into XML without lots of base64 conversions.
Most PUT requests should return either a 201 Created with the object that was just created, which may have been modified due to server application logic, or a 204 No Content if there are no changes to the original object request. If you operate with a database that has eventual consistency, it may also be a good idea to instead return a 202 Accepted request to indicate that the client should try to fetch the resource at a later time. You should also return an estimate of how long it will be before this object is created.
POST
A POST operation on a collection is defined as a creation request. This is the user requesting to add a new resource to the collection without specifying a specific ID. In general, you want to either return a redirection code and a Location header, or at the least the ID of the object you just created (if not the whole object serialized in your representation).
A POST operation on a resource should actually create a subobject of that resource; although, this is often not used. Traditionally, browsers don't handle changing form actions to PUT, so a POST is typically treated as a special form of a PUT that takes form-encoded values.
A POST operation on a property is typically used only for uploading files but could also be used as appending a value to a list. In general, this could be considered an append operation, but if your property is a single value, it's probably safe to assume that the client wanted this to be a PUT, not a POST.
DELETE
A DELETE operation on a collection is used to drop the entire collection, so you probably don't want to allow this. If you do allow it, this request should be treated as a request to remove every single resource in the collection.
A DELETE operation on a specific resource is simply a request to remove that specific resource. If there are errors in this request, the client should be presented with a message explaining why the request failed. The resulting error code should explain if the request can be issued again, or if the user is required to perform another operation before reissuing the request. The most typical error message back from this request is a 409 Conflict, which indicates that another resource is referencing this resource, and the server is refusing to cascade the delete request. A DELETE may also return a 202 Accepted response if the database has eventual consistency.
A DELETE operation on a specific property is identical to a PUT with an empty value for that property. It can be used to delete just a single property from a resource instead of having to send a PUT request. This can also be used as a differentiation between setting a value to blank and removing the value entirely. In programming terms, this the difference between an empty string and None or Null.
HEAD
A HEAD request on any URL should return the exact same headers as a standard GET request but shouldn't send back the body. This is typically used to get a count of the number of results in a response without retrieving the actual results. In my applications, I use this to send an additional header X-Results, which contains the number of results that would have been retrieved.
OPTIONS
An OPTIONS request on any URL returns the methods allowed to be performed on this URL. This can return just the headers with the additional Accept header, but it can also return a serialized version of them in the body of the results that describes what each method actually does. This response should be customized for the specific user that made the request, so if the user is not allowed to perform DELETE operations on the given resource for any reason, that option should not be returned. This allows the client to specifically hide options that the users aren't allowed to perform so that they don't get error responses.