- Objectives
- 25.1 Problem Prevention Best Practices
- 25.2 Problem Determination Methodology
- 25.3 Working with IBM WebSphere Support
- 25.4 Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
25.1 Problem Prevention Best Practices
25.1.1 Testing Best Practices
One of the best ways to avoid and flush out problems prior to releasing a new application into the production environment is to properly test it. To most, the concept of testing is an obvious prerequisite to putting an application into the production environment. However, it is important to note that the key is not just "testing," but "proper testing". To paraphrase a famous statement, it is not whether you test, but how you test. A large percentage of the problems that occur in production environments can be prevented if the application is properly tested. It is the goal of this section to describe a test methodology that, when followed, will result in a significantly reduced risk of production outages.
25.1.1.1 Properly Scaled Tests
Prior to testing an application, it is important to ensure the environment in which it is tested is appropriate for that application. An appropriate test environment for an application is one that is a scale model of its production environment, including all components from Web server to database. Having an exact replica of the production environment for testing may not always be economically feasible; however, it is highly recommended that the test environment be an accurate model of the production environment.
What Does It Mean to Have a Scale Model of Production in Test?
Prior to discussing the benefits of having a scale model of the production environment in test, it is important to clearly articulate what we mean when using this phrase. It is broader than simply having a fraction of the production computing power in the test environment. As an example, imagine a production environment consisting of four machines, each with four processors. Two possible test configurations may be
One 4-processor machine
Four 1-processor machines
While it is true that both of these proposed test environments have one-quarter the processing power of the production environment, both leave out complexity. The first configuration, one 4-processor machine, disregards the complexity of having the application distributed across multiple systems. The second option, four 1-processor machines, ignores the impact of having multiple processors per machine. Although the number of processors per machine does not directly affect the administrative configuration of the WebSphere environment, it does impact the ability to troubleshoot the application. More specifically, this configuration increases the difficulty of finding synchronization bugs.
Most synchronization problems are only found on multiple processor machines where true multithreading and parallelization occur. Thus, not testing on multiprocessor servers can lead to deploying an application, fraught with synchronization problems, into the production environment. A more correct scale would be to have four 2-processor machines or, at a minimum, two 2-processor machines.
Another key component to having a scale model of the production environment in test is to ensure that the entire path length of the application is tested with no parts skipped, ensuring that all tests are functionally and systematically complete. This includes having all database connections, Web servers, proxy servers, messaging servers, firewalls, and so on, configured and running in the test environment as they will in production. Failing to test the entire environment may result in deployment or run time failures.
Benefits of a Scale Model
The most obvious reason for testing an application is for correctness. It is important to verify that the application actually works as designed, but this is only one of the reasons for testing. Another commonly overlooked reason for testing is to allow you to become familiar with the dynamics of the application, its attributes in steady state, startup, and under load in a safe environment. This is important because it will allow you to quickly identify and isolate anomalies when the application enters the production environment. If the test environment is not to scale of the production environment, then the dynamics observed will change as the application moves to production, thus crippling your problem determination ability and putting the company in a dangerous situation.
Maximizing performance tuning efforts is yet another reason for having a scale model of the production environment in test. Performance is one of the key characteristics of any application, often the second most important next to correctness. For an application to achieve maximum performance, it must be tuned. Performance tuning is a highly iterative process dependent on many parameters set both within an application server and the wider environment. If the test environment is not to scale of the production environment, much of the hard work that went into tuning the application and its environment is wasted because the parameters that achieved the best results cannot be directly applied in production. A concrete example of a parameter that has a large effect on an application server's performance, thus an application's performance, is the Java Virtual Machine's heap size. For example, if, by tuning, it is discovered that a maximum heap size of 256 megabytes achieves the best performance in the test environment and the test environment is one-half scale of production, then this number can be doubled in production with a higher degree of confidence than if the test environment were not to scale of production.
25.1.1.2 Isolated Test Environment
To ensure the accuracy of any test, it is important to mandate that the test environment be isolated from other environments and activities. This is necessary to ensure that the observed behaviors are the result of the application or services with which the application is interacting and not some external event. An isolated test environment consists of machines dedicated to running WebSphere, an isolated subnet, dedicated database, and any other external systems and resources. One of the most frustrating situations when attempting to debug a problem is being unable to isolate its source, only to later find out that the problem occurred because of some external situation.
25.1.1.3 Test Scenarios
To ensure that an application is ready to be deployed in the production environment, care must be taken to ensure that the scenarios under which it is tested are realistic. Realistic test scenarios consist of two parts.
The first step in creating realistic test scenarios is to ensure that the scripts used for testing the application are realistic examples of what a user might do in production. General usage scenarios are usually known prior to developing an application in the architecture phase; however, like most things, enterprise applications are not always used the way their architects envisioned. Instead, people use the application in the way that feels most intuitive to them. Therefore, it is best to get test cases from the users themselves. There are several ways to gather this information, which vary depending on what is being deployed into production.
If the application is an update to an existing application already in production, then real-world usage can be tracked on the current production version of the application, either by logging or with an external monitoring tool capable of capturing transactions.
If the application is completely new, with functions never before available to its users, then the task of gathering real-world scenarios becomes more difficult. One option is to use IBM's Rational Unified process or the Extreme Programming paradigm at http://www.extremeprogramming.org/, in which the end users are constantly evaluating the application as it is being developed. Another option is to verify the application's general functionality via some architected use cases then release the application to a small alpha or beta test group to gather more user oriented test cases.
GOTCHA
A commonly forgotten step when creating test scripts is to validate the returned results. By default, many of the stress-testing applications only check for a HTTP 200 response, which is insufficient when testing J2EE applications. If the results are not verified, the application could be displaying incomplete Web pages or, worse yet, the data for a different user.
Once the test cases have been created, the next step is to run them against the application in the test environment. As it is important to gather realistic test scenarios, it is also important to run a diverse mixture of these scenarios while testing the application. The mixture should be a representative set of your user population. For instance, if there are six different user types expected in the production environment and the majority (50 percent) are expected to be of one type (type A) while the remaining 50 percent will be equally distributed among the five remaining user types, then you will want to run the tests with a similar distribution. In this example 50 percent of the virtual users would be executing test A while the remaining 50 percent would be distributed evenly among the other test cases.
GOTCHA
It is important to capture and test all possible user types even if there is one user type that is only used once in a great while. An example would be an administrator performing some weekly or monthly functions. Though this administrator may be less than 1 percent of the user population, it is important to understand the implications of that user's actions.
TIP
While monitoring the performance of the application and its dependent parts, also monitor the statistics of the testing machine. Never let the stress testing software or hardware be the bottleneck in a test environment.
Test Types
After the test scenarios have been created, the next step is to execute them in various test types. The first tests that should be run are those that verify the correctness of the application. There are several others tests that go beyond just testing functionality of the application which should also be run. Three of the most important tests are performance, stress, and endurance. Describing the detailed steps required to execute each of these tests is beyond the scope of this book, so we will give a brief overview of each and pointers to more in-depth resources as appropriate.
The performance test is probably the most well-known test type. In a performance test, as the name suggests, the goal is to optimize the performance tuning parameters to maximize the overall performance of the applications being tested. The process for performance tuning WebSphere Application Server is described in great detail in chapter 22 of this book and in the whitepaper, WebSphere 4.0 Performance Tuning Methodology found at http://www.ibm.com/software/webservers/appserv/doc/v40/ws_40_tuning.pdf. The paper was written for the 4.0 version of WebSphere Application Server; however, the logic and methodology detailed is applicable to every WebSphere release, including the latest, V5.1.
One of the main themes of this chapter, thus far, has been the importance of having a scale model of your production environment in test. A key component of this is having an accurate model of user workload to ensure that the planned capacity is enough to handle the estimated user load. Stress testing is designed to apply user workload, above and beyond what is expected, in an attempt to find the breaking point of the application and its environment. This type of test has several benefits. First, it will help you better understand the characteristics of the application under high load and what piece of the application is likely to fail. If a potential weak link can be identified, procedures can be put into place to deal with the problem, allowing you to be proactive instead of reactive. The second benefit is that application bugs, which might not surface under normal load, will get flushed out. One of the most prominent types of these bugs are those caused by synchronization (or lack of it). Flushing out synchronization bugs is a function of time, load, and parallelism. The larger these components are, the greater the possibility of finding a synchronization problem. Stress testing increases the load part of this equation.
The last type of test we discuss is the endurance test. The length of the endurance test is what differentiates it from all other tests. Most tests last several minutes or, at most, an hour, but in the endurance test the application is run under heavy load, as expected in production, for many hours or even days. The purpose of this is to uncover problems that may appear after extended usage. There are many problems that fall into this category, including session problems, intermittent failures, and the aforementioned synchronization bugs.
25.1.2 Change Control Best Practices
The production environment should be a strictly controlled environment. Failure to adhere to a stringent change control policy can result in inexplicable problems in the production environment. This section describes some best practices associated with change control with the production environment.
25.1.2.1 Restricting Administrative Privileges
The first step in ensuring proper change control is to limit the number of people who have administrative privileges on the production machines. Oftentimes we have encountered production systems on which many people in the organization have administrative privileges, leading to a myriad of problems. This is especially problematic if the production machines are shared across many departments within the organization. In most cases, each department will have their own set of priorities, and, if the administrative privileges are not restricted to one or a select few administrators, people can unwittingly make changes that impact other applications within the environment.
25.1.2.2 Access History
Even with proper restriction on administrative privileges, problems can occur because of administrative errors. When they happen it is useful to have a log detailing who made the last change, at what time, and from which remote machine. This is not to place blame but rather to identify what actions caused the problem and to be able to inquire as to why that particular action was performed. Once the problematic action is identified, an alternative procedure can be created to avoid future outages.
Another reason to keep an access history is to identify hackers. By following the potential hacker's actions, you can see what techniques he/she is using to access your site, and you may be able to identify their intentions if they should break in.
25.1.2.3 Backing Up Your Configuration
Prior to making any configuration change, it is a good idea to back up the current, working configuration. WebSphere provides a utility, backupConfig, to aid this process. BackupConfig is command line utility (found in the <WASND_ROOT>/bin and <WAS_ROOT>/bin directories) that creates a backup of the installation's configuration files, including application and server data. BackupConfig can be run by executing the provided executable. Be aware that its default behavior is to stop all WebSphere processes prior to performing the backup. This behavior can be overridden by supplying the switch –nostop. The complete command would be backupConfig –nostop.
TIP
When using WebSphere in a distributed environment, backupConfig only needs to be run on the Deployment Manager.
To restore from a previously saved configuration, use the restoreConfig command. Like backupConfig, restoreConfig's default behavior is to stop the WebSphere processes prior to restoring the configuration. Again, the –nostop command line option can be used to override the default behavior.
TIP
Unfortunately, we are all human and it is possible to forget to back up the configuration prior to making a change. Creating cron jobs, a scheduled execution of a task, is one way to overcome this human limitation. By creating a daily or weekly cron job that runs the backupConfig command, you can ensure that you have a recovery point even if you forget to manually run the backupConfig command.
Beyond backing up the configuration, it is also a good idea to backup the entire WebSphere installation directory immediately after installation, before and after any major changes, such as adding a node to a cell. This is useful in the event an administrator accidentally deletes a critical file, such as startupServer.bat or java.exe. Mistakes such as these are not uncommon, especially when working from the command line and dealing with many different directories at once. Regardless of whether this has happened in your organization before, it is best to protect against these types of mistakes and create regular backups.
25.1.2.4 Maintain a Log History
Usually a problem is discovered only after several occurrences and perhaps not until an external complaint is logged. By keeping a log history, you can uncover how long this problem has been occurring and potentially find a pattern in the events that caused the problem to occur.
25.1.2.5 Documented Procedures
It is easy to forget a step when performing a complex operation such as installing WebSphere or upgrading an application. The best way to avoid such errors is to document the steps for these complex operations. The document should consist of clear and detailed steps with screen shots that show expected results to avoid possible confusions and misinterpretations. Every administrator should have a hard copy of these documents and follow them to the letter.
Execution of the procedures should involve, at the very least, two people. The first administrator executes the steps and the other verifies them.
TIP
Automating the documented procedures is an advanced technique that, when implemented properly, eliminates all potential user error. The risk is that there is no human control on the process. For more information on automated scripting, refer to chapter 20, Automated WebSphere Administration.
25.1.3 WebSphere Best Practices
IBM provides numerous white papers and articles about best practices and design patterns for WebSphere. The WebSphere Developer Domain Web site is a great place to find many useful WebSphere articles, including best practices-specific papers. A Web site that hosts an extensive collection of best practices for administering the WebSphere Application Server Web site is also available at http://www.software.ibm.com/wsdd/zones/bp. IBM best practices are reviewed and updated for new versions of WebSphere. It is recommended you stay informed and follow the best practices published by IBM.
25.1.3.1 Application Best Practices
Application performance and scalability is heavily influenced by the design of the application, database, and other resources. The importance of following application best practices is underestimated in many cases. A good application design for performance and scalability follows certain fundamental best practices and avoids common mistakes. Following the WebSphere application best practices is a good starting point for designing applications with great performance and scalability. Performance, in particular, suffers from poorly designed and implemented applications.
25.1.3.2 Code Reviews
Code reviews are an extremely effective way to find code bugs and, thus, prevent problems, but many times they are not fully completed due to project deadlines or other time constraints. Code reviews are performed by programmers who are not the authors of the code that is investigated. In many cases code reviews are as effective as or even more effective than testing.
Code reviews include simple things like checking for naming conventions, code indenting or adding Javadoc comments, and more complex tasks like code optimization or logic checking. The tasks in the first category make your code easier to maintain and reuse, while tasks in the second category check the correctness and improve the performance of your application. Code optimization like caching data, efficient use of database connections, or eliminating unnecessary object instantiation can significantly increase performance.
The good news is that you can use Java code analyzers to automate much of the code review process. Most Java code analyzers detect things like unused or duplicate imports, unused private and local variables, missing Javadoc comments, violation of naming conventions, and empty "catch" blocks or "if" statements. You will still have to check for logic errors and code optimization, but most of these simple (but time-consuming) code checking tasks are automated.
TIP
You can integrate Java code analyzers with Ant. In this way you can check your code any time you run a unit test or complete a build.
25.1.4 WebSphere Fix Packs and Interim Fixes
In many cases, the solution to your problem may be as simple as applying a fix pack or interim fix. For this reason, we recommend that you stay up to date with fix packs for your version of WebSphere and check for interim fixes when problems occur. Although applying a fix pack requires careful planning, this operation pays off in most cases by fixing or preventing problems. It is recommended that you also keep current supporting products such as DB2, and so on.
TIP
Fix packs should be tested in nonproduction environment first. Occasionally features in a fix pack may not be compatible with existing applications.
Interim fixes are WebSphere code fixes created for known individual problems and should be applied when you have a critical problem without a valid workaround. Interim fixes are individually tested and are integrated with the next WebSphere fix pack. Make sure that you check the WebSphere Application Server fix packs and interim fixes Web site http://www.ibm.com/software/webservers/appserv/was/support/ when a problem occurs in order to determine if this is a documented problem. Each released version of WebSphere details the defects that were fixed in its accompanied release notes. You can also create a "My support" profile to receive weekly e-mail notifications about IBM product updates. For WebSphere, go to http://www-306.ibm.com/software/webservers/appserv/was/support/ and select My Support from the right menu.