Java

Custom Thread Pools

Last updated Aug 26, 2005.

One neglected aspect of performance tuning from which I have been able to leverage extreme value is the creation of custom thread pools. The typical manifestation of thread pool tuning is to follow the wait-based tuning methodology to determine the optimal value for the application server thread pool to minimize waiting. The potential drawback to this line of reasoning is that certain pieces of application functionality may be neglected. The change in thought process revolves around subdividing the utilization of application components by the components themselves.

Through the deep analysis of usage patterns and use cases, you can determine application functionality that takes precedence to other functionality. For example, if you are running an online store, analysis of your access logs may reveal that the majority of users browse your catalog, which consumes resources away from the personalization component to your application. While it is obvious that your catalog needs the majority of the usage; however, if the personalization component allows customers to register for product releases and notifications (hence driving future sales), then you do not necessarily want that functionality hindered by catalog browsers.

The answer is to allocate a subset of threads to the personalization component and the majority of threads to the catalog browser. In this way, when the catalog has utilized all of its threads, there are still dedicated threads to service the personalization component.

The end result is that the catalog may slow down slightly, but the personalization will continue to work even while the application is under stress.

Looking to application use cases, the application business owner my show favoriability to the checkout functionality over the browsing functionality. It is important that customers are able to browse the contents of your store, but it is more important that they are actually able to buy their selected items. Therefore, we can allocate threads specifically to the e-commerce checkout functionality, which no other application process can touch. This helps ensure that, while the site is under stress, users can still check out.

As a final example of the benefit of using custom thread pools, as an application monitoring vendor, my company deploys components to the application server that reveal its internal performance. To facilitate reliable communication with the server, we require a custom thread pool dedicated to our components. Why? If the application server is under stress and hence performing poorly then that is the time period we really need to monitor. If the poorly-performing application is consuming all of the threads that our components need to run, our monitoring tool cannot communicate with them — and hence cannot point to the root cause of the performance problem. On a handful of occasions, our consultants have forgotten the step of adding a custom thread pool. During the first load test that the customer runs, our tools fail to gather information. Fortunately, that has happened prior to our consultants leaving the customer site, but it really emphasizes the need for custom thread pools.

Application Server Implementation

Now that you have bought off on the need for custom thread pools, there are two mechanisms to implementing it across application server vendors:

Built-in support
Multiple application server instances

In the former case, such as in the case of BEA's WebLogic, the application server natively supports custom thread pools. You create a custom queue into which the application server drops its requests, and then assign a thread pool to service that queue. WebLogic allows you to assign application components or specific Servlets, JSPs, and EJBs to a specific execution queue and size its thread pool accordingly. This is one of the features that I most like about WebLogic and this capability has served me well in the past.

If, on the other hand, the application server does not natively support custom thread pools, you can work around the limitation by running multiple instances of your application servers, each hosting a logical component with its own resources. This is somewhat of a maintaince pain, but when you do it a few times, it is quite effective and allows you not only to control threads, but the entire environment (including heap, database connection pools, etc.). I have been at a customer site where they had 22 WebSphere instances running on one box, clustered to another box with 22 WebSphere instances. And these were "beefy" boxes! At first I thought it was going to be a nightmare to configure and tune, and to some extent it is more difficult; but as you perform a capacity assessment on a single application, you get an idea of its capabilities and how it uses resources. Based on the size of the computer system, you can scale down your application system resource utilization to fit into the shared environment and allocate resources appropriately. I found this to be a very effective strategy that yielded top performance.

High-level Theory

You've seen the benefits of using custom thread pools and the technical implementation. I'd like to finish up this essay by sharing general guidelines about how to identify the candidate components that should have their own thread pools:

Look to Your Use Cases

Your architecture should dictate the most important functionality of your application, and that should be available!

Access Logs and End-User Experience Monitors

While use cases define the required functionality (and performance) of your application as well as expected usage, end users cannot be controlled. Your end-users will do things their own way, in a way that is either optimal to their needs, or totally random and unexplainable!

The only way to understand what they are doing is to carefully observe and record their behavior. There are several ways to do this, but the two least expensive (in terms of performance) are analyzing access logs and integrating an end-user experience monitor into your environment. The access log is free because your application server is probably recording this information anyway and your analysis is offline. It provides a fairly representative summary of your users' interaction with your environment, but it is limited to only reporting information captured in the access log. In the general case, this is sufficient, but if you need more information you need an end-user experience monitor.

An end user experience monitor is a physical device that sits in your environment and sniffs all network traffic between your users and your web servers. The reports that it generates are extremely detailed and can be customized to capture the specifics of your application that are above and beyond what an access log can capture. Furthermore, the sniffing technology can run at almost 0% overhead. On the flip side, these are not free and you will need to buy one.

The key observations that you should be drawing from this information are the allocations of application utilization compared to your use case. By correlating actual behavior with expected behavior, you learn how to test better, and you can make educated decisions about key functionality that should run in custom thread pools.

Follow the Money

Any time a customer is going to give you money through your application, it should be real easy, don't you think?

Management

All management functionality should be partitioned into its own custom thread pools. If your application starts behaving poorly, and you need to adjust it, you need the resources to get into the application and adjust it. Most application servers provide a management console that runs with its own dedicated resources to facilitate this functionality, but if you are deploying your own management functionality (e.g. through custom managed beans and a managed bean application) then you need to ensure that it is more available than your application when under stress!

Monitoring

Any time you want to be notified of exceptional circumstances, you need to be able to identify them. This means that monitoring functionality, along with management functionality, needs to take precedence to application functionality. Without implementing a monitoring solution you are in effect delegating that role to your end users. Which, of course, is not a good idea.

Summary

Most companies dedicate time upfront to application architecture and design, but few follow through with a formal deployment methodology. The architecture of the application deployment is just as important as the architecture of the application itself. Even if your application is written perfectly, the application server won't give it the resources that it needs, so it will perform poorly.

In a subsequent article, I will present my formal Java EE Deployment Methodology that includes custom thread pools and so much more. This section was included to get you thinking about the importance of deployment strategies and provide you with some low-hanging fruit that you can use to realize real performance gains today!