Switching from Ad Hoc to Framework
On a recent project that tested a web service, we had a wonderful opportunity to run a large amount of production data through the web service before we released it to production. This was after most of our functional and performance testing was complete, but we had some suspicions that, given the nature of the service and the data, there might be some data abnormalities in production that we hadn’t anticipated.
I designed a one-time test, using Ruby to read in, line by line, an ASCII file dump of the production data and submit the data using the web service. I looked only for the most basic of application error codes and spectacular failures such as Java exceptions. We were quite surprised to find that this testing did in fact reveal many problems with our data of which we were unaware. If fact, we were so happy with the results of this test that we made it a standard part of our testing going forward.
What I had originally written in around two hours was now to be used by multiple people, for multiple data sets, on multiple services. At that point, it made sense to make some changes. I paired with another tester and we rewrote the script to be more modular. We added a command-line interface and options for running in different environments; we put more comments in the code, and we added the script (and test data) to source control. All the things that I didn’t want to do when I thought it was a one-time test, now made sense going forward.