- Item 13: Define Roles and Responsibilities
- Item 14: Require a Mixture of Testing Skills, Subject-Matter Expertise, and Experience
- Item 15: Evaluate the Tester's Effectiveness
- Test engineer self-evaluation
This chapter is from the book
This chapter is from the book
This chapter is from the book
Item 15: Evaluate the Tester's Effectiveness3
Maintaining an effective test program requires that the implementation of its elements, such as test strategy, test environment, and test-team make-up, be continuously evaluated, and improved as needed. Test managers are responsible for ensuring that the testing program is being implemented as planned and that specific tasks are being executed as expected. To accomplish this, they must track, monitor, and evaluate the implementation of the test program, so it can be modified as needed.
At the core of test-program execution are the test engineers. The ability of testers to properly design, document, and execute effective tests, accurately interpret the results, document any defects, and track them to closure is critical to the effectiveness of the testing effort. A test manager may plan the perfect testing process and select the ideal strategy, but if the test-team members do not effectively execute the testing process (for example, participating effectively in requirements inspections and design walk-throughs) and complete all strategic testing tasks as assigned (such as executing specific test procedures), important defects may be discovered too late in the development life cycle, resulting in increased costs. Worse, defects may be completely overlooked, and make their way into production software.
A tester's effectiveness can also make a big difference in relationships with other project groups. A tester who frequently finds bogus errors, or reports "user errors" when the application works as expected but the tester misunderstands the requirement, or (worst of all) often overlooks critical defects loses credibility with other team members and groups, and can tarnish the reputation of an entire test program.
Evaluating a tester's effectiveness is a difficult and often subjective task. Besides the typical elements in any employee's performance, such as attendance, attentiveness, attitude, and motivation, there are specific testing-related measures against which a tester can be evaluated. For example, all testers must be detail oriented and possess analytical skills, independent of whether they are technical testers, subject-matter experts, security testers, or usability testers.
The evaluation process starts with recruitment. The first step is to hire a tester with the skills required for the roles and responsibilities assigned to each position. (See Item 13 for a discussion on roles, responsibilities, and skills.)
In the case where a testing team is "inherited" rather than hired for the project, evaluation is more complicated. In such a case it is necessary for the manager to become familiar with the various testers' backgrounds, so the team members can be tasked and evaluated based on their experience, expertise, and backgrounds. It may become necessary to reassign some team members to other roles as their abilities become better known.
A test engineer's performance cannot be evaluated unless there are specified roles and responsibilities, tasks, schedules, and standards. The test manager must, first and foremost, state clearly what is expected of the test engineer, and by when.
Following is a typical list of expectations that must be communicated to testers.
Observe standards and procedures. The test engineer must be aware of standards and procedures to be followed, and processes must be communicated. Standards and procedures are discussed in Item 21.
Keep schedules. Testers must be aware of the test schedule, including when test plans, test designs, test procedures, scripts, and other testing products must be delivered. In addition, the delivery schedule of software components to testing should be known by all testers.
Meet goals and perform assigned tasks. Tasks must be documented and communicated, and deadlines must be scheduled, for each tester. The test manager and the test engineer must agree on the assigned tasks.
Meet budgets. For testers evaluating testing tools or other technology that must be purchased, the available budget must be communicated so the tester can work within that range and avoid wasting time evaluating products that are too expensive.
Expectations and assignments differ depending on the task at hand and the skill set of the tester. Different types of tests, test approaches, techniques, and outcomes may be expected.
Once expectations are set, the test manager can start comparing the work of the test team against the established goals, tasks, and schedules to measure effectiveness of implementation. Following is a list of points to consider when evaluating a tester's effectiveness.
Subject-matter expert vs. technical expert. The expertise expected from a subject-matter expert is related to the domain of the application, while a technical tester is concerned with the technical issues of the application.
When a technical tester functions as an automater, automated test procedures should be evaluated based on defined standards that must be followed by the test engineers. For example, the supervisor might ask: Did the engineer create maintainable, modular, reusable automated scripts, or do the scripts have to be modified with each new system build? Did the tester follow best practices, such as making sure the test database was baselined and could be restored when the automated scripts need to be rerun? If the tester is developing custom test scripts or a test harness, the tester will be evaluated on some of the same criteria as a developer, including readability and reliability of the code.
A tester who specializes in the use of automated tools, yet does not understand the intricacies of the application's functionality and underlying concepts, will usually be ineffective. Automated scripts based only on high-level knowledge of the application will often find less-important defects. It is important that the automater understand the application's functionality in order to be an effective member of the testing team.
Another area for evaluation is technical ability and adaptability. Is the test engineer capable of picking up new tools and becoming familiar with their capabilities? Testers should be trained regarding the various capabilities of a testing tool, if they are not already thoroughly familiar with them.
Experienced vs. novice tester. As previously mentioned, the skill level of the tester must be taken into account. For example, novice testers may overlook some errors, or not realize they are defects. It is important to assign novice testers to lower-risk testing areas.
Inexperienced testers are not alone in overlooking defects. Experienced testers may ignore some classes of defects based on past experience ("the product has always done that") or the presence of work-arounds. Appropriately or not, testers may become "acclimated" to familiar errors, and may not report defects that seem unimportant to them but may be unacceptable to end users.
Functional vs. nonfunctional testing. A tester's understanding of the various testing techniques available (see Chapter 5) and knowledge of which technique is most effective for the task at hand should be evaluated. If the tester doesn't understand the various techniques and applies a technique inappropriately, test designs, test cases, and test procedures will be adversely affected.
Functional testing can additionally be based on a review of the test procedures. Typically, testers are assigned to test procedures for testing specific areas of functionality based on assigned requirements. Test procedure walk-throughs and inspections should be conducted that include the requirements, testing, and development teams. During the walk-through, it should be verified that all teams agree on the behavior of the application.
The following questions should be considered during an evaluation of functional test procedures:
How completely are the test-procedure steps mapped to the requirements steps? Is traceability complete?
Are the test input, steps, and output (expected result) correct?
Are major testing steps omitted in the functional flow of the test procedure?
Has an analytical thought process been applied to produce effective test scenarios?
Have the test-procedure creation standards been followed?
How many revisions have been required as a result of misunderstanding or miscommunication before the test procedures could be considered effective and complete?
Have effective testing techniques been used to derive the appropriate set of test cases?
During a test-procedure walk-through, the "depth" or thoroughness of the test procedure should be verified. In other words, what does the test procedure test? Does it test functionality only at a high level, or does it really dig deep down into the underlying functionality of the application?
To some extent, this is related to the depth of the requirement steps. For example, a functional requirement might state, "The system should allow for adding records of type A." A high-level test procedure establishes that the record can be added through the GUI. A more-effective test procedure also includes steps that test the areas of the application affected when this record is added. For instance, a SQL statement might verify that the record appears correctly in the database tables. Additional steps could verify the record type. There are numerous other testing steps to be considered, such as verifying the system's behavior when adding multiple records of type A—whether duplicates are allowed, for example.
If test procedures are at a very high level, it is important to confirm that the requirements are at the appropriate level and pertinent details are not missing. If there is a detail in the requirement that is missing in the test procedure, the test engineer might need coaching on how to write effective test procedures. Or, it could be that the engineer did not adequately understand the requirement.
Different criteria apply to evaluating functional testing than to nonfunctional testing. For example, nonfunctional tests must be designed and documented in a different manner than functional test procedures.
Testing phase. Different tasks are to be performed by the tester depending on the testing phase (alpha test, beta test, system test, acceptance test, and so on).
During system testing, the tester is responsible for all testing tasks described in this book, including the development and execution of test procedures, tracking defects to closure, and so on. Other testing phases may be less comprehensive.
During alpha testing, for example, a tester might be tasked with simply recreating and documenting defects reported by members of a separate "alpha testing team," which is usually the company's independent testing (Independent Verification and Validation, or IV&V) team.
During beta testing, a tester might be tasked with documenting the beta-test procedures to be executed, in addition to recreating and documenting defects found by other beta testers. (Customers are often recruited to become beta testers.)
Phase of the development life cycle. As mentioned throughout this book, testers should be involved from the beginning of the life cycle. Evaluation of tester performance should be appropriate to each phase. For example, during the requirements phase, the tester can be evaluated based on defect-prevention efforts, such as identification of testability issues or requirements inconsistencies.
While a tester's evaluation can be subjective, many variables related to the phase of testing must be considered, rather than jumping to the first seemingly obvious conclusion. For example, when evaluating the test engineer during the requirements phase, it is important to consider the quality of the requirements themselves. If the requirements are poorly written, even an average tester can find many defects. However, if the requirements are well laid out and their quality is above average, only an exceptional tester is likely to find the most subtle defects.
Following of instructions and attention to detail. It is important to consider how well a test engineer follows instructions and pays attention to detail. Reliability and follow-through must be monitored. If test procedures must be updated and executed to ensure a quality product, the test manager must be confident that the test engineers will carry out this task. If tests have to be automated, the test manager should be confident that progress is being made.
Weekly status meetings where engineers report on their progress are useful to track and measure progress. In the final stages of a testing phase, these meetings may be held daily.
Types of defects, defect ratio, and defect documentation. The types of defects found by the engineer must be considered during the evaluation. When using this metric to evaluate a tester's effectiveness, some factors to keep in mind include the skill level of the tester, the types of tests being performed, the testing phase being conducted, and the complexity and the maturity of the application under test. Finding defects depends not only upon the skill of the tester, but also on the skill of the developer who wrote, debugged, and unit tested the code, and on the walk-through and inspection teams that reviewed the requirements, design, and code. Ideally, they will have corrected most defects before formal testing.
An additional factor to evaluate in this context is whether the test engineer finds errors that are complex and domain related, or only cosmetic. Cosmetic defects, such as missing window text or control placement, are relatively easy to detect and become high priority during usability testing, whereas more complicated problems relating to data or cause-effect relationships between elements in the application are more difficult to detect, require a better understanding of the application, and become high priority during functional testing. On the other hand, cosmetic-defect fixes, since they are most visible, may have a more immediate effect on customer happiness.
The test manager must consider the area for which the tester is responsible. The tester responsible for a specific area where the most defects are discovered in production should not necessarily be assumed to have performed poorly. If the tester's area is very complex and error-prone and the product was released in a hurry, failure to catch some defects may be understandable.
The types of defects discovered in production also matter. If they could have been discovered by a basic test within the existing test-procedure suite, and if there was plenty of time to execute the test procedure, this would be a major oversight by the tester responsible for this area. However, before passing judgment, some additional questions should be considered:
Was the test procedure supposed to be executed manually? The manual tester may have become tired of executing the same test procedures over and over, and after many trials concluded it should be safe not to execute the tests because that part of the application has always worked in the past.
Was the software delivered under pressure of a deadline that could not be changed even though it ruled out a full test cycle? Releases should not be allowed without having met the release criteria, time pressures notwithstanding.
Was this test automated? Did the automated script miss testing the step containing the error? In such a case, the automated scripts must be reevaluated.
Was the defect discovered using some combination of functional steps that are rarely executed? This type of defect is more understandable.
Additionally, it may be necessary to review the test goals, risks of the project, and assumptions made when the test effort started. If it had been decided not to conduct a specific type of test because of time constraints or low risk, then the tester should not be held responsible. This risk should have been taken with full knowledge of the possibility of problems.
Effectiveness can also be evaluated by examining how a defect is documented. Is there enough detail in the documented defect for a developer to be able to recreate the problem, or do developers have a difficult time recreating one specific tester's defects? Standards must be in place that document precisely what information is required in defect documentation, and the defect tracking life cycle must be well communicated and understood. All testers must follow these standards. (For a discussion of the defect tracking life cycle, see Item 50.)
For each issue uncovered during evaluation of a tester, the cause of the issue should be determined and a solution should be sought. Each issue must be evaluated with care before a judgment regarding the tester's capability is made. After careful evaluation of the entire situation, and after additional coaching has been provided where called for, it will be possible to evaluate how detail oriented, analytical, and effective this tester is. If it is determined that the tester lacks attention to detail or analytical skills or there are communication issues, that tester's performance may need to be closely monitored and reviewed, and there may be a need for additional instruction and training, or other appropriate steps need to be taken.
Testers' effectiveness must be constantly evaluated to ensure the success of the testing program.