- Assessment
- Measurement
- Interpretation and Analysis
- Identifying Bottlenecks
- Tuning or Upgrading
This chapter is from the book
This chapter is from the book
This chapter is from the book
3.5 Tuning or Upgrading
Once the bottleneck is identified, there are two choices. Either the amount of the resource can be increased, or the demand for the resource can be reduced. Tuning techniques will be discussed along with the particular bottlenecks. However, there are some general tips that can be applied to tuning bottlenecks:
-
Determine whether the data indicate a particular bottleneck.
-
Devise a simple test for measuring results.
-
Do not tune randomly.
-
Use heuristics and logic in choosing what to tune.
-
Look for simple causes.
-
Develop an action plan.
-
Change only one thing at a time.
-
Prioritize the goals.
-
Know when to stop tuning.
3.5.1 Determine Whether There Is a Particular Bottleneck
This is the first tip to apply. Using the multiple characteristics of a bottleneck, determine if the data indicate a particular bottleneck. There is no use in tuning disk I/O on a system that is CPU-bound.
3.5.2 Devise a Simple Test for Measuring Results
When something in the operating system or application is tuned, it is advisable to quantitatively measure whether performance has improved or degraded. The test should be simple, short, and repeatable, so that it can be performed as each tuning step is taken.
3.5.3 Do Not Tune Randomly
There was once a system administrator who always tuned the size of the buffer cache, no matter what the bottleneck. Tuning the wrong thing can make the situation worse rather than better. Use knowledge of how the kernel works to help decide the cause of the bottleneck. Always try to visualize what the data should look like and compare it to the actual data before assuming that a cause has been found.
3.5.4 Use Heuristics and Logic
Use heuristics to select the most likely cause of the bottleneck. Experience can point to a particular bottleneck rather quickly. Watching the access lights on the disk drives can quickly indicate which disk drives may be saturated even before you look at the metrics. Use the qualitative measures as well as the quantitative metrics. Logically work through the possible causes and potential tuning solutions. Experience in tuning systems will tell which tuning alternatives offer the largest probability of success.
3.5.5 Look for Simple Causes
By looking for simple causes, easier and less expensive solutions can be tried. This is the K. I. S. S. principle: Keep it Simple, Stupid! Simple causes tend to occur more frequently, and simple solutions should be tried first. For example, tuning a system parameter is easier and less costly than modifying the design of an application. Either tactic might result in performance improvements, but one is clearly a better solution than the other.
3.5.6 Develop an Action Plan
Write up an ordered, prioritized list of measurements and tuning actions. The list can sometimes be turned over to other people for execution. Include contingencies in case some of the steps do not result in improvement. Analyze the results after performing each step of the plan.
3.5.7 Change Only One Thing at a Time!
Otherwise, the cause of the bottleneck cannot be found. If you attempt to tune multiple things at once, the benefits from one type of tuning can be counteracted by the degradation caused by a different type of tuning.
3.5.8 Prioritize Goals
Often, you may have multiple goals when developing a tuning solution. Prioritizing the goals can solve the problem caused by different goals requiring different solutions. It is usually possible to tune for response time or for throughput, but not both. In a transaction-processing environment, one should tune for the most frequent type of transaction, or the most important.
3.5.9 Understand the Limits of the Application's Architecture
Some applications cannot fully utilize a system. An example is a single-threaded process flow designed with no parallelism. Adding a CPU to such an application may not increase performance.
Instrumenting an application (adding code to measure specific events or code paths) to provide performance metrics can also create this problem if the instrumentation is not created with parallelism in mind. For example, incrementing a single global counter may be a choke point, as in the following code design:
while not done {
do transaction work
lock
increment global counter
unlock
}
3.5.10 Know When to Stop
Finally, knowing when to stop tuning is very important. Setting the performance expectations in advance is the easiest way to establish completion. Getting the user to agree to success criteria in advance is important for deciding not only when to stop tuning, but when you can declare success. Baselines can be used to help establish that performance has returned to a normal level. The "eternal hope syndrome" will not occur if the performance expectations are met through tuning.
From Bob's Consulting Log— I spent several days at a customer site improving performance by a factor of 50%. Once the customer saw the extent of the improvement, she insisted I do additional work to improve things even more. Soon it became clear that we had already maxed out the performance improvement and then wasted several more days trying to eke out just a little bit extra.
It is not useful to spend a lot of time on gaining 1% improvement in performance. Optimizing a part of the application that is executed only once and consumes less than 5% of the total execution time is just not worth it.