Java Reference Guide

Load Testing

Last updated Jul 7, 2006.

The first step in application testing is unit testing: test the functionality (and performance) of individual units, or components, of your application. The purpose of this testing is to ensure that when you integrate your components to a working application, it is built from working parts.

After your components are integrated into a working application, the application is tested for functionality, followed by a test for performance. In order to accurately test your application for performance, you need to subject it to some amount of load, and as it turns out, the type of load it is subjected to is of great importance. The reason being that the load test will identify problems with the application and the configuration of the environment only for the load that it generates: generate the wrong load and you will not necessarily find problems that your users will experience. The answer to this dilemma must be divided into two categories of applications:

1. Existing Applications
2. New Applications

Existing Applications

When testing an existing application, (for example, the next version of an application already in production), existing end-user behavior can be analyzed to construct load tests. In a web-based application there are two approaches to doing this: analyzing access logs, or using an end-user experience monitor. If you have divided functionality in your application such that the URLs themselves define what functionality is executed, then you can determine what your end-users are doing and in what proportions by looking at access log entries. If, on the other hand, URLs do not clearly differentiate application functionality (such as posting an XML document to a controller servlet in which the contents of the XML dictate the functionality to execute), you can add an end-user experience monitor into your environment which allows you to further examine the contents of those URLs. An end-user experience monitor is a piece of hardware that sits in various places of your network and sniffs network traffic as it passes between machines; from this sniffing it can provide much deeper information about what each request is doing. Regardless of where you gather your usage patterns, try to generate load scripts to simulate at least the top 80% of your user requests, and in the proper proportion (for example, if a user logs in once and generates 50 reports, the balance between logging in and generating reports should be 1 to 50.)

New Applications

If you are testing a new application, your job is far more difficult because you can only "guess" at what you expect your users to do. I purposely put the word "guess" in quotes because while we are guessing, the activity is really a very educated guess. I define this using the following three activities:

• Estimate
• Validate
• Reflect

Estimate

The first step is to estimate what you expect your end-users will do. Defining explicit use cases and working with your application business owner are the keys to being successful here. The use cases define a basis to test with ,and your application business owner should be able to guide you to an estimation of the balance between use cases. Construct your load tests according to this analysis, tune your application and environment, and then deploy your application to production.

Validate

With your application in production, the next activity is to validate whether or not your estimations were accurate. I usually do this in two phases: a sanity check and a full validation. The sanity check is a quick comparison of user activity a couple weeks into the deployment to see if your estimations were grossly off. It may take your users a month or even a quarter to fall into a groove in how they use your application on a day-to-day basis. Therefore, whenever I feel that usage patterns have stabilized, I perform the aforementioned analysis for an existing application (log file analysis or end-user experience monitor analysis) to validate the accuracy of my estimations. This may lead to a new load testing profile and subsequent tuning efforts.

Reflect

The final step is to reflect on the accuracy of your estimations and identify where you were wrong and try to determine why. As your organization matures and this process is repeated, your estimations should become more and more accurate. But without this post-mortem analysis, you will continue to make the same mistakes over and over again.

With the proper load scripts defined, you are ready to start tuning your application and environment. Load testing can cause an application that behaves well in an isolated test to perform horribly. Consider creating temporary objects to satisfy each request: to satisfy a request you construct a 200 kilobyte business object and pass that business object to a business controller. When the process is complete, the business object is deleted (or more specifically it is de-referenced in memory and now eligible for garbage collection.) This behavior is not harmful to your system under light load, but consider 500 users instigating the same functionality every second: now instead of orphaning 200 kilobytes in memory you are orphaning 100 megabytes in memory every second. This causes garbage collection to run more frequently, and some objects may become prematurely tenured, causing major garbage collection to run more frequently and hence greatly impacting the end-user experience. And this behavior can only appear under load.

Load testing is therefore very important to perform against your environment, but to do so you need the right set of tools. There are a hoard of load testers on the market, some commercial and some open source. The top commercial load testers are:

• Mercury LoadRunner
• Segue (now Borland) SilkPerformer

These are highly respected advanced tools, but they also come at a premium. If you have a budget for such a tool, these vendors are the way to go, but if you are budget conscience there are some open source alternatives that very well may suit your needs. These include:

• Apache JMeter
• The Grinder
• OpenSTA

There are several others in the industry, but these are the ones that I see most frequently (with the most frequent being JMeter). For the purposes of this series we'll focus on JMeter because of its widespread acceptance as well as its mature feature set and expandability.