Home > Articles

  • Print
  • + Share This
This chapter is from the book

25.1 Problem Prevention Best Practices

25.1.1 Testing Best Practices

One of the best ways to avoid and flush out problems prior to releasing a new application into the production environment is to properly test it. To most, the concept of testing is an obvious prerequisite to putting an application into the production environment. However, it is important to note that the key is not just "testing," but "proper testing". To paraphrase a famous statement, it is not whether you test, but how you test. A large percentage of the problems that occur in production environments can be prevented if the application is properly tested. It is the goal of this section to describe a test methodology that, when followed, will result in a significantly reduced risk of production outages. Properly Scaled Tests

Prior to testing an application, it is important to ensure the environment in which it is tested is appropriate for that application. An appropriate test environment for an application is one that is a scale model of its production environment, including all components from Web server to database. Having an exact replica of the production environment for testing may not always be economically feasible; however, it is highly recommended that the test environment be an accurate model of the production environment.

What Does It Mean to Have a Scale Model of Production in Test?

Prior to discussing the benefits of having a scale model of the production environment in test, it is important to clearly articulate what we mean when using this phrase. It is broader than simply having a fraction of the production computing power in the test environment. As an example, imagine a production environment consisting of four machines, each with four processors. Two possible test configurations may be

  1. One 4-processor machine

  2. Four 1-processor machines

While it is true that both of these proposed test environments have one-quarter the processing power of the production environment, both leave out complexity. The first configuration, one 4-processor machine, disregards the complexity of having the application distributed across multiple systems. The second option, four 1-processor machines, ignores the impact of having multiple processors per machine. Although the number of processors per machine does not directly affect the administrative configuration of the WebSphere environment, it does impact the ability to troubleshoot the application. More specifically, this configuration increases the difficulty of finding synchronization bugs.

Most synchronization problems are only found on multiple processor machines where true multithreading and parallelization occur. Thus, not testing on multiprocessor servers can lead to deploying an application, fraught with synchronization problems, into the production environment. A more correct scale would be to have four 2-processor machines or, at a minimum, two 2-processor machines.

Another key component to having a scale model of the production environment in test is to ensure that the entire path length of the application is tested with no parts skipped, ensuring that all tests are functionally and systematically complete. This includes having all database connections, Web servers, proxy servers, messaging servers, firewalls, and so on, configured and running in the test environment as they will in production. Failing to test the entire environment may result in deployment or run time failures.

Benefits of a Scale Model

The most obvious reason for testing an application is for correctness. It is important to verify that the application actually works as designed, but this is only one of the reasons for testing. Another commonly overlooked reason for testing is to allow you to become familiar with the dynamics of the application, its attributes in steady state, startup, and under load in a safe environment. This is important because it will allow you to quickly identify and isolate anomalies when the application enters the production environment. If the test environment is not to scale of the production environment, then the dynamics observed will change as the application moves to production, thus crippling your problem determination ability and putting the company in a dangerous situation.

Maximizing performance tuning efforts is yet another reason for having a scale model of the production environment in test. Performance is one of the key characteristics of any application, often the second most important next to correctness. For an application to achieve maximum performance, it must be tuned. Performance tuning is a highly iterative process dependent on many parameters set both within an application server and the wider environment. If the test environment is not to scale of the production environment, much of the hard work that went into tuning the application and its environment is wasted because the parameters that achieved the best results cannot be directly applied in production. A concrete example of a parameter that has a large effect on an application server's performance, thus an application's performance, is the Java Virtual Machine's heap size. For example, if, by tuning, it is discovered that a maximum heap size of 256 megabytes achieves the best performance in the test environment and the test environment is one-half scale of production, then this number can be doubled in production with a higher degree of confidence than if the test environment were not to scale of production. Isolated Test Environment

To ensure the accuracy of any test, it is important to mandate that the test environment be isolated from other environments and activities. This is necessary to ensure that the observed behaviors are the result of the application or services with which the application is interacting and not some external event. An isolated test environment consists of machines dedicated to running WebSphere, an isolated subnet, dedicated database, and any other external systems and resources. One of the most frustrating situations when attempting to debug a problem is being unable to isolate its source, only to later find out that the problem occurred because of some external situation. Test Scenarios

To ensure that an application is ready to be deployed in the production environment, care must be taken to ensure that the scenarios under which it is tested are realistic. Realistic test scenarios consist of two parts.

The first step in creating realistic test scenarios is to ensure that the scripts used for testing the application are realistic examples of what a user might do in production. General usage scenarios are usually known prior to developing an application in the architecture phase; however, like most things, enterprise applications are not always used the way their architects envisioned. Instead, people use the application in the way that feels most intuitive to them. Therefore, it is best to get test cases from the users themselves. There are several ways to gather this information, which vary depending on what is being deployed into production.

If the application is an update to an existing application already in production, then real-world usage can be tracked on the current production version of the application, either by logging or with an external monitoring tool capable of capturing transactions.

If the application is completely new, with functions never before available to its users, then the task of gathering real-world scenarios becomes more difficult. One option is to use IBM's Rational Unified process or the Extreme Programming paradigm at http://www.extremeprogramming.org/, in which the end users are constantly evaluating the application as it is being developed. Another option is to verify the application's general functionality via some architected use cases then release the application to a small alpha or beta test group to gather more user oriented test cases.


A commonly forgotten step when creating test scripts is to validate the returned results. By default, many of the stress-testing applications only check for a HTTP 200 response, which is insufficient when testing J2EE applications. If the results are not verified, the application could be displaying incomplete Web pages or, worse yet, the data for a different user.

Once the test cases have been created, the next step is to run them against the application in the test environment. As it is important to gather realistic test scenarios, it is also important to run a diverse mixture of these scenarios while testing the application. The mixture should be a representative set of your user population. For instance, if there are six different user types expected in the production environment and the majority (50 percent) are expected to be of one type (type A) while the remaining 50 percent will be equally distributed among the five remaining user types, then you will want to run the tests with a similar distribution. In this example 50 percent of the virtual users would be executing test A while the remaining 50 percent would be distributed evenly among the other test cases.


It is important to capture and test all possible user types even if there is one user type that is only used once in a great while. An example would be an administrator performing some weekly or monthly functions. Though this administrator may be less than 1 percent of the user population, it is important to understand the implications of that user's actions.


While monitoring the performance of the application and its dependent parts, also monitor the statistics of the testing machine. Never let the stress testing software or hardware be the bottleneck in a test environment.

Test Types

After the test scenarios have been created, the next step is to execute them in various test types. The first tests that should be run are those that verify the correctness of the application. There are several others tests that go beyond just testing functionality of the application which should also be run. Three of the most important tests are performance, stress, and endurance. Describing the detailed steps required to execute each of these tests is beyond the scope of this book, so we will give a brief overview of each and pointers to more in-depth resources as appropriate.

The performance test is probably the most well-known test type. In a performance test, as the name suggests, the goal is to optimize the performance tuning parameters to maximize the overall performance of the applications being tested. The process for performance tuning WebSphere Application Server is described in great detail in chapter 22 of this book and in the whitepaper, WebSphere 4.0 Performance Tuning Methodology found at http://www.ibm.com/software/webservers/appserv/doc/v40/ws_40_tuning.pdf. The paper was written for the 4.0 version of WebSphere Application Server; however, the logic and methodology detailed is applicable to every WebSphere release, including the latest, V5.1.

One of the main themes of this chapter, thus far, has been the importance of having a scale model of your production environment in test. A key component of this is having an accurate model of user workload to ensure that the planned capacity is enough to handle the estimated user load. Stress testing is designed to apply user workload, above and beyond what is expected, in an attempt to find the breaking point of the application and its environment. This type of test has several benefits. First, it will help you better understand the characteristics of the application under high load and what piece of the application is likely to fail. If a potential weak link can be identified, procedures can be put into place to deal with the problem, allowing you to be proactive instead of reactive. The second benefit is that application bugs, which might not surface under normal load, will get flushed out. One of the most prominent types of these bugs are those caused by synchronization (or lack of it). Flushing out synchronization bugs is a function of time, load, and parallelism. The larger these components are, the greater the possibility of finding a synchronization problem. Stress testing increases the load part of this equation.

The last type of test we discuss is the endurance test. The length of the endurance test is what differentiates it from all other tests. Most tests last several minutes or, at most, an hour, but in the endurance test the application is run under heavy load, as expected in production, for many hours or even days. The purpose of this is to uncover problems that may appear after extended usage. There are many problems that fall into this category, including session problems, intermittent failures, and the aforementioned synchronization bugs.

25.1.2 Change Control Best Practices

The production environment should be a strictly controlled environment. Failure to adhere to a stringent change control policy can result in inexplicable problems in the production environment. This section describes some best practices associated with change control with the production environment. Restricting Administrative Privileges

The first step in ensuring proper change control is to limit the number of people who have administrative privileges on the production machines. Oftentimes we have encountered production systems on which many people in the organization have administrative privileges, leading to a myriad of problems. This is especially problematic if the production machines are shared across many departments within the organization. In most cases, each department will have their own set of priorities, and, if the administrative privileges are not restricted to one or a select few administrators, people can unwittingly make changes that impact other applications within the environment. Access History

Even with proper restriction on administrative privileges, problems can occur because of administrative errors. When they happen it is useful to have a log detailing who made the last change, at what time, and from which remote machine. This is not to place blame but rather to identify what actions caused the problem and to be able to inquire as to why that particular action was performed. Once the problematic action is identified, an alternative procedure can be created to avoid future outages.

Another reason to keep an access history is to identify hackers. By following the potential hacker's actions, you can see what techniques he/she is using to access your site, and you may be able to identify their intentions if they should break in. Backing Up Your Configuration

Prior to making any configuration change, it is a good idea to back up the current, working configuration. WebSphere provides a utility, backupConfig, to aid this process. BackupConfig is command line utility (found in the <WASND_ROOT>/bin and <WAS_ROOT>/bin directories) that creates a backup of the installation's configuration files, including application and server data. BackupConfig can be run by executing the provided executable. Be aware that its default behavior is to stop all WebSphere processes prior to performing the backup. This behavior can be overridden by supplying the switch –nostop. The complete command would be backupConfig –nostop.


When using WebSphere in a distributed environment, backupConfig only needs to be run on the Deployment Manager.

To restore from a previously saved configuration, use the restoreConfig command. Like backupConfig, restoreConfig's default behavior is to stop the WebSphere processes prior to restoring the configuration. Again, the –nostop command line option can be used to override the default behavior.


Unfortunately, we are all human and it is possible to forget to back up the configuration prior to making a change. Creating cron jobs, a scheduled execution of a task, is one way to overcome this human limitation. By creating a daily or weekly cron job that runs the backupConfig command, you can ensure that you have a recovery point even if you forget to manually run the backupConfig command.

Beyond backing up the configuration, it is also a good idea to backup the entire WebSphere installation directory immediately after installation, before and after any major changes, such as adding a node to a cell. This is useful in the event an administrator accidentally deletes a critical file, such as startupServer.bat or java.exe. Mistakes such as these are not uncommon, especially when working from the command line and dealing with many different directories at once. Regardless of whether this has happened in your organization before, it is best to protect against these types of mistakes and create regular backups. Maintain a Log History

Usually a problem is discovered only after several occurrences and perhaps not until an external complaint is logged. By keeping a log history, you can uncover how long this problem has been occurring and potentially find a pattern in the events that caused the problem to occur. Documented Procedures

It is easy to forget a step when performing a complex operation such as installing WebSphere or upgrading an application. The best way to avoid such errors is to document the steps for these complex operations. The document should consist of clear and detailed steps with screen shots that show expected results to avoid possible confusions and misinterpretations. Every administrator should have a hard copy of these documents and follow them to the letter.

Execution of the procedures should involve, at the very least, two people. The first administrator executes the steps and the other verifies them.


Automating the documented procedures is an advanced technique that, when implemented properly, eliminates all potential user error. The risk is that there is no human control on the process. For more information on automated scripting, refer to chapter 20, Automated WebSphere Administration.

25.1.3 WebSphere Best Practices

IBM provides numerous white papers and articles about best practices and design patterns for WebSphere. The WebSphere Developer Domain Web site is a great place to find many useful WebSphere articles, including best practices-specific papers. A Web site that hosts an extensive collection of best practices for administering the WebSphere Application Server Web site is also available at http://www.software.ibm.com/wsdd/zones/bp. IBM best practices are reviewed and updated for new versions of WebSphere. It is recommended you stay informed and follow the best practices published by IBM. Application Best Practices

Application performance and scalability is heavily influenced by the design of the application, database, and other resources. The importance of following application best practices is underestimated in many cases. A good application design for performance and scalability follows certain fundamental best practices and avoids common mistakes. Following the WebSphere application best practices is a good starting point for designing applications with great performance and scalability. Performance, in particular, suffers from poorly designed and implemented applications. Code Reviews

Code reviews are an extremely effective way to find code bugs and, thus, prevent problems, but many times they are not fully completed due to project deadlines or other time constraints. Code reviews are performed by programmers who are not the authors of the code that is investigated. In many cases code reviews are as effective as or even more effective than testing.

Code reviews include simple things like checking for naming conventions, code indenting or adding Javadoc comments, and more complex tasks like code optimization or logic checking. The tasks in the first category make your code easier to maintain and reuse, while tasks in the second category check the correctness and improve the performance of your application. Code optimization like caching data, efficient use of database connections, or eliminating unnecessary object instantiation can significantly increase performance.

The good news is that you can use Java code analyzers to automate much of the code review process. Most Java code analyzers detect things like unused or duplicate imports, unused private and local variables, missing Javadoc comments, violation of naming conventions, and empty "catch" blocks or "if" statements. You will still have to check for logic errors and code optimization, but most of these simple (but time-consuming) code checking tasks are automated.


You can integrate Java code analyzers with Ant. In this way you can check your code any time you run a unit test or complete a build.

25.1.4 WebSphere Fix Packs and Interim Fixes

In many cases, the solution to your problem may be as simple as applying a fix pack or interim fix. For this reason, we recommend that you stay up to date with fix packs for your version of WebSphere and check for interim fixes when problems occur. Although applying a fix pack requires careful planning, this operation pays off in most cases by fixing or preventing problems. It is recommended that you also keep current supporting products such as DB2, and so on.


Fix packs should be tested in nonproduction environment first. Occasionally features in a fix pack may not be compatible with existing applications.

Interim fixes are WebSphere code fixes created for known individual problems and should be applied when you have a critical problem without a valid workaround. Interim fixes are individually tested and are integrated with the next WebSphere fix pack. Make sure that you check the WebSphere Application Server fix packs and interim fixes Web site http://www.ibm.com/software/webservers/appserv/was/support/ when a problem occurs in order to determine if this is a documented problem. Each released version of WebSphere details the defects that were fixed in its accompanied release notes. You can also create a "My support" profile to receive weekly e-mail notifications about IBM product updates. For WebSphere, go to http://www-306.ibm.com/software/webservers/appserv/was/support/ and select My Support from the right menu.

  • + Share This
  • 🔖 Save To Your Account

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information

To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.


Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.


If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information

Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.


This site is not directed to children under the age of 13.


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information

If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information

Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents

California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure

Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact

Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice

We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020