Performance and Load-Testing of Axis with Various Web Services Styles

Date: Jul 16, 2004

Rajal Shah and Naresh Apte evaluate the performance and load-testing characteristics of various styles of web services with Axis.

Before continuing with our discussion of web services architecture and framework in this series, it's important to evaluate performance and load-testing characteristics of web services. The service-oriented architecture also introduces dependency of various software modules, including its performance and ability to handle transactions under load. This article presents the results of our tests performed for RPC-style, doc-literal, and attachment-style web services using the Apache Axis web engine.

Introduction

We ran a complete load-test exercise to determine the Axis web service engine's performance under heavy payloads and under simultaneous transactions for RPC-style and document-literal services, and services with attachments. Here are the overall results:

RPC-style web services performed very poorly, with many failed transactions and response times of several seconds for even a moderate-sized (50KB) payload, even with very few simultaneous users.
Document-literal web services and web services with attachments performed much better and could actually handle a production environment load with an impressive small roundtrip overhead (500 milliseconds) at the web services infrastructure level.
Our results indicated no transaction failures with document-literal web services, even with a high payload (200KB per transaction) and many concurrent users (500 users).
Clearly, RPC-style web services are not fit for production, even with moderate-sized payload exchanges. Document-literal and attachments-style web services are a sure winner given their performance and reliability.

Setup

Dual-processor Linux machines with Tomcat servers were used to test the web services infrastructure overhead. The time measured is the total roundtrip time taken from the client making the request through the web services client stub until it received a response. (The web services business module was stubbed out to avoid including business module processing time.) The client application and the web service were hosted on the same machine to reduce network delay.

Control Parameters

We varied two parameters to determine the performance impact and stability of the system:

Payload size. This measurement is the total amount of bytes exchanged in each transaction. We categorized into three different sizes:

Low payload (2KB per transaction)
Medium payload (50KB per transaction)
High payload (200KB per transaction)

Simultaneous users accessing the system. This measurement provides insight into scalability of the system.

Exit Strategy To Stop a Test

The tests were set up to keep the payload constant for all users. We ramped up the simultaneous user base until we started seeing transaction errors or the CPU usage hit 25–30% during our testing process.

These are the graphs and metrics relevant to our discussion (we'll get to these shortly):

RPC-style web service under moderate load
Document-literal web service under light load
Document-literal web service under medium load
Document-literal web service under heavy load

NOTE

Due to a setup error, our initial startup time didn't reflect the true web services overhead and thus needed to be discarded. Ignore the initial spike on the left side of each graph.

Performance Summary for RPC-Style Web Service Exchange

Payload size: Medium (50KB)

Figure 1 shows response time versus number of virtual users for this test.

Figure 1 Performance summary for RPC-style web service exchange with a medium payload.

The following table shows the results of each measurement.

Measurement	Result
Total transactions passed	57
Total transactions failed	40
Average response time	11.224 seconds

Summary of Results

RPC-style services work very well for low payloads (graph not shown), but soon reached scalability issues with even a moderate payload of 50KB per transaction. Ignoring the startup error reading, we still see roundtrip time between 6–10 seconds for just 8–15 simultaneous users (not even concurrent users). This incredible overhead is even poorer if we observe the failed transactions that the client application received: 40%. Clearly these were unacceptable numbers for an infrastructure layer, which adds on to the business module processing time. No further testing was done for high payloads with RPC-style services.

We've analyzed the reasons for this poor performance at the end of all our test results, later in this article.

Performance Summary for Document-literal Web Service with Low Payload

Payload size: Low (2KB)
Ramp up: 25 users per minute
Iteration pacing: 120 seconds
Total test duration: 45 minutes

Figure 2 shows response time versus number of virtual users for this test.

Figure 2 Performance summary for document-literal web service with low payload.

The following table shows the results of each measurement.

Measurement	Result
Number of users at 25% CPU utilization	700
Total transactions failed	0
Average response time	0.228 seconds

Like RPC-style services, document-literal web services perform very well under small payloads. The roundtrip infrastructure overhead is about 200 milliseconds. We could ramp up the simultaneous users all the way up to 900 users and still kept CPU utilization under 25%. Most importantly, we had no failed transactions. RPC-style services graphs were very similar with an even smaller overhead of about 150 milliseconds for the transactions.

Performance Summary for Document-literal Web Service with Medium Payload

Payload size: Medium (50KB)
Ramp up: 25 users per minute
Iteration pacing: 120 seconds
Total test duration: 35 minutes

Figure 3 shows response time versus number of virtual users for this test.

Figure 3 Performance summary for document-literal web service with medium payload.

The following table shows the results of each measurement.

Measurement	Result
Number of users at 25% CPU utilization	650
Total transactions failed	0
Average response time	0.412 seconds

Refreshingly, performance numbers for a document-literal web services infrastructure layer with a moderate payload were far superior to what we observed with RPC-style web services. Transaction response time was about 400 milliseconds, up from the 200 milliseconds we saw with low-payload transactions. The most encouraging observation with this test was that there were no failed transactions despite ramping up simultaneous user base to 600 users before we hit the 25% CPU utilization mark.

This test emphatically states the superiority of document-literal web services over its non-production-capable RPC-style counterpart. With these encouraging results, we decided to get similar numbers for a heavy payload with document-literal services.

Performance Summary for Document-Literal Web Service with High Payload

Payload size: High (200KB)
Ramp up: 25 users per minute
Iteration pacing: 120 seconds
Total test duration: 27 minutes

Figure 4 shows response time versus number of virtual users for this test.

Figure 4 Performance summary for document-literal web service with high payload.

The following table shows the results of each measurement.

Measurement	Result
Number of users at 29.5% CPU utilization	500
Total transactions failed	0
Average response time	0.752 seconds

We see encouraging numbers again. With a heavy payload of 200KB per transaction, we could ramp up the simultaneous users to 500 before hitting 30% CPU utilization. The average roundtrip overhead was noted around 700 milliseconds—still less than a second, under a very heavy load of simultaneous users handling a large payload. Again, the most impressive number was that of no failures in any transactions.

Analysis

We've identified three aspects that affect web services engine performance (specifically Axis) and give some insight into why document-literal web services provide such excellent results.

These three areas seem to be the major sources of performance bottlenecks in the web services infrastructure layers:

Java reflection
DOM objects
Parsers

We'll discuss each area in detail.

Java Reflection

For some Axis developers and users, Java reflection could have a serious impact on the performance of the engine. Java reflection is a technique that allows the web services engine to receive the SOAP message and use generic serializers to convert them to Java objects (RPC-style service). This is a dynamic conversion done at runtime on every request—without any knowledge about the incoming message or the object signatures. Reflection is an expensive operation; you should weigh the convenience of letting Axis do the mapping for you transparently. In certain conditions—for example, for large amounts of data—it may be optimal to process data directly or use custom serializers to directly map the incoming XML into your objects.

Another option is to process the data directly in XML (document-literal), avoiding this mapping entirely and gaining significant performance. With document-literal services, even if you don't process the data directly in XML but convert it to Java objects first, because you have knowledge of the incoming message and your object signatures at development time you don't have to rely on Reflection to do the mapping.

Using a SAX parser and populating your data objects during the XML parsing performs significantly better than having Axis performing a dynamic binding for you at runtime. In our tests for document-literal services, we didn't process the XML messages exchanged, as that now falls under the domain of the business module with an XML interface. Avoid the Java reflection to bind data to objects was the major contributor to the excellent performance of document-literal web services.

DOM Objects

Creating DOM objects generates a large memory footprint. When Axis does the dynamic binding of the SOAP message to the Java objects, it has to create the entire DOM of the message in memory. There might be better ways to process incoming messages—using SAX processors, pull processors, attachment techniques, etc.—to avoid this inflated memory requirement. These alternate strategies can increase the transaction load and the payload size that your service/infrastructure can handle.

Document-literal web services don't have to create DOM objects. They can parse all incoming messages using SAX parsers and validation turned on in the same step. Also, if the business module manipulates data directly without converting to Java objects, we skip the step to serialize the message into data objects, giving an additional performance boost. No DOM creation in document-literal web services reduces the constraint for system resources, thus gaining the advantage of processing successful transactions for a large user base without any errors.

Parsers

Certain web services engines such as GLUE use custom XML parsers instead of Xerces. They claim to get significant performance gains through this setup. Considering that SOAP is XML-based, it could reduce web services engine overhead significantly. For Axis, this may just be the normal evolutionary path to resolve this issue as the parser implementations improve.

Implementation Issues with Axis Document-Literal Messaging

Attachments are yet another way to a document-literal style of exchange. Web services with attachments are published as a way to send binary files. We can use this technique to send XML documents. Even though we haven't discussed attachments so far, this style of web service has the same benefits as document-literal services. The only difference is that the attachments are received outside the SOAP message body, whereas in document-literal services they're inside the envelope. Now that your business module does the validation checking itself, it doesn't matter whether you use document-literal messaging in web services as part of the SOAP body or outside it.

The problem with attachments is that they may not yet be fully compatible with all web services engines. Hence, we don't recommend this style as of now. However, there is one major flaw in the Axis document-literal web services that might compel you to consider web services with attachments. The current implementation of document-literal web services with Axis always returns the DOM structure of the incoming message directly to the business module. There's no way to retrieve the message as a string and bypass the DOM creation.

Why is this important? If your document is large, the DOM structure uses up all the memory—with our project, we constantly get OutOfMemoryExceptions on both the client and the server with a relatively small set of data due to the DOM generation in RPC-style services and also with document-literal services (were we to not use our alternate solution). We believe that Axis should provide an option to not convert to DOM but instead receive the raw data for document-literal messaging. In the absence of this capability, for our tests we embedded the entire XML inside a CDATA element and transmitted that over the wire. Axis returned the entire CDATA element as one string, which was our unprocessed XML message. We could then pass the message "as is" to the business module. Alternatively, we could use web services with attachments to transmit the XML documents, if we could manage the incompatibility issues that come with this style.

Another suggestion would be to use an RPC-style service in which your business module has just one String parameter that takes in an XML document.

Performance numbers with all these hacks were very similar to those of document-literal web services, and hence not illustrated in this article.

Conclusion

Document-literal services seemed to perform admirably. Under heavy payloads and with many simultaneous users accessing the web service, the infrastructure still processed all transactions successfully, keeping the roundtrip infrastructure overhead well under a second. Document-literal web services are thus determined to be deployable to production. Web services with attachments had near identical numbers to those of document-literal web services, due to their nearly identical processing flow.

RPC-style services may be convenient for quick prototypes, but clearly are not production-ready and cannot be deployed by an enterprise for any real enterprise web services.