Four Classic Problems with Scripted Testing

• By Michael Kelly
• Jun 2, 2006

Like this article? We recommend
Software Testing, 2nd Edition

On a recent project, we were instructed by the test manager to create hundreds of scripted test cases. Our intention was to execute these tests on the first good build we received from the development staff. We spent months developing the test scripts and planning for their execution. We created the test cases, entered them in our test management tool, traced them to requirements, and reviewed them with the business and other testers.

In spite of months of planning and hard work, I noticed some problems on our first day of test execution.

Problem 1: Almost None of the Test Scripts Were Correct

The expected results didn’t match the actual requirements once the day of testing came around. Given the number of test cases and the number of changing requirements, it was impossible to keep them up to date. I won’t say all that time was wasted, because it did get us looking at the requirements, and it did give us something close to resembling an expected result. But it didn’t do what it was intended to do: It didn’t accurately capture the steps needed to execute a test, or serve as an effective oracle for expected results. That is, it didn’t reduce the need for brain-engaged testing.