Java

J2EE Performance Tuning

Last updated Feb 4, 2005.

If you are a regular reader of the Java Reference Guide, you have undoubtedly heard me ramble about starting a J2EE Performance Tuning service at my company, Quest Software. If you are not a regular, hopefully this article will entice you to keep coming back for more.

To establish some context: I spent several years as an enterprise architect and later as a J2EE architect. Three and one half years ago, I began to design software for monitoring J2EE enterprise environments, specifically focused on J2EE performance. After spending almost three years designing performance monitoring and tuning software, I was tasked with starting a J2EE performance tuning service. Now, I would be thrown back out to real world situations, equipped with the tools I designed to solve customer problems. By now, I have spent nine months in the field, so I felt it a good time to start sharing with you the most common problems that I have encountered.

Before diving into detailed problems, let me identify the four sources of performance problems in J2EE applications:

By identifying these four facets of performance problems, we can better formulate a plan to analyze the root-cause(s) of performance problems, and then develop a resolution plan.

To summarize these four performance facets: the problem can be in the application itself, the configuration of the environment in which the application is running, or on something upon which the application depends. If it is none of the above, then the environment has reached its capacity and it's time to add hardware. Note that throwing hardware at a problem is the last choice, not the first!

Although I cannot give you the tools to solve your problems through this series of articles (though many commercial vendors, such as my employer, can), it will help to review the common pitfalls that I describe below. They may be good starting points at improving the performance of your J2EE environment.

Environment: Application Server (Overview)

It is my basic tenet that, from an application server configuration-only perspective, you can reach within 80% of your ideal configuration by addressing four key tuning metrics:

While there are some glaring errors that can be made elsewhere in your configuration that will nullify my previous statement, in general the configuration of these components yields the "biggest bang for the buck."

Heap

By and large, the biggest problem I've seen in my adventures is a misconfiguration of the heap. The problems come in one of two flavors:

  1. a lack for formal education about how to properly configure a heap, or
  2. documented, but yet relatively unknown JVM behavior.

I do not fault anyone for a lack of formal education on heap configuration. After all, I hope that after reading this article you are ready to dive in and tune your heap. It is an advanced topic, but one of the utmost importance.

NOTE

Before delving deep into heap configurations, note that in this article I look at Sun's implementation of the JVM, so this discussion may benefit anyone running any application using Sun's JVM. If you use BEA's JRockit or IBM's JVM, tune back in later for the idiosyncracies of their implementations.

The Sun JVM divides the heap into two primary areas:

The Young Generation is further subdivided into three areas:

The heap is designed so that objects that are created and destroyed quickly are clean up easily. Objects that live for a long time are difficult to cleam up, but that is done much less frequently. The following steps describe how the garbage collector moves objects around and eventually frees them:

  1. Objects are created in Eden
  2. When Eden becomes full, live objects are copied to the first survivor space; dead objects are then discarded
  3. When the first survivor space is full, then live objects are copied to the second survivor space; dead objects are discarded
  4. When the second survivor space is full then all of its live objects are tenured, or retired, to the old generation; dead objects are discarded
  5. When the young generation is full and there is not room to copy objects to the old generation, then the garbage collector walks through every object in the old generation and discards objects that are dead. During this time, nothing running in the JVM can do anything; this is referred to as a "stop-the-world" collection

Objects move from Eden to S1 and to S2 prior to being retired to the old generation, giving them plenty of time to be destroyed, right? If the heap is configured properly, the answer to this question is a resounding "yes," but if the heap is left in its default configuration then sadly the answer is "no."

Here is the meat of this article (I thought I would point it out in case you got bored): the reason that the default heap is not configured properly is because the young generation is too small and the survivor spaces can be measured in kilobytes, not in megabytes. Objects are created and retired to the old generation well before they can be collected.

Here is what I have found to be the best initial configuration for the heap, based off the total size of the heap:

Putting this into command line arguments that you can send to your JVM, consider a 1GB heap (1024MB):

–Xms1024m 
–Xmx1024m 
-XX:NewSize=500m
-XX:MaxNewSize=500m
-XX:SurvivorRatio=6

The following breaks down what each option means:

My recommendation is to use these settings, then load test your heap and observe the heap's performance. If the heap repeatedly climbs quickly and falls, there is still too much major garbage collection occurring. In that case, you need to increase the size of your survivor spaces (decrease the XX:SurvivorRatio) value.

Your goal is for the heap to grow up to a critical mass (this could take a couple minutes or a couple hours), and then to see the heap executing relatively frequent minor collections.

Permanent Generation

Now that you are comfortable with the ideas of a young and old generation, here is another curve ball: have you ever seen java.lang.OutOfMemoryError's appear in your log files while you heap appears to be under-utilized? The heap maintains another area that it uses to load classes into; note these are the actual class file, not the class instances.

The JVM uses the class files that are loaded into the permanent generation in order to create class instances that are placed into the heap. If the permanent generation is sized too small, it must unload class files to make room for new ones before it can create new instances. This is a performance hit at runtime, while the heap is unloading classes.

On the other hand, 1.3.x versions of the JVM do not unload classes from its permanent generation. If you are running a 1.3.x variation, you need to seriously examine this setting. By default, 1.4.x versions of the JVM unload classes, but you can pass them an option, billed as a "performance enhancement," that stops this behavior: -noclassgc. As I previously mentioned, a performance hit is incurred when classes are unloaded, but rather than use this option, the best bet is to size the permanent generation appropriately. The default value is 4MB, which for enterprise applications is typically too small.

While you are counting your classes, don't overlook your JavaServer Pages. Recall how application servers process JSPs:

  1. The JSP is translated to a Servlet source code file
  2. That Servlet source code file is compiled into a class file
  3. That class file is loaded into the permanent generation space in the JVM memory
  4. An instance of that Servlet is created and placed in the heap (in Eden)

So for each JSP you have, you will need to account for that as an additional class file that will occupy permanent generation memory space.

Your best bet is to avoid the whole problem by sizing the permanent generation yourself. Depending on how many classes you have, you'll want to increase this value to at least 32MB and up to 128MB. For a company that had literally thousands of JSP files, I increased that value to 256MB, but that was an extreme case. The following parameters modify the permanent generation size:

-XX:PermSize=128m
-XX:MaxPermSize=128m

Note that we fix the size of the permanent generation by setting the minimum and maximum values to be the same.

Summary

In this series, I am relating information I've learned from my hands-on experience tuning J2EE enterprise applications. Thus far, we have looked at the most common problem: misconfigured heaps. I hope that this information better equips you to understand and handle problems as they occur.

InformIT Articles and Sample Chapters

"J2EE Performance Tuning". In this article I define what is meant by performance tuning and how we measure it. It is an overview of a J2EE Performance Tuning methodology.

"J2EE Performance Tuning, Part 2". I define the methodology used for tuning J2EE systems in detail. The article further includes a description of the type of information that you need when tuning and how to acquire it.

"J2EE Performance Tuning, Part 3". I dive deep into the architecture of application servers. By identifying all of the moving pieces, we become better equipped to troubleshoot performance problems.

Online Resources

In a Sun Developer article you can learn more about garbage collection, specifically related to the 1.4.x JVM.

Take a look at jvmstat if you would like to start diving in to look at the heap and its performance in real-time.