6. Pilot-Based Tests
Problem
How can you test the framework sufficiently and reliably?
Context
You are in the process of coding the framework. Tests are necessary to make sure that the framework works correctly and reliably.
Forces
Testing is an important aspect of quality assurance. Testing is particularly important when you build a framework, since bugs would quickly manifest in all applications that use the framework.
However, testing a framework is difficult [Fayad+1999]. A framework alone is just an abstract architecture, not something that can be executed. In order to test the framework, you need a sample application that uses the framework and so acts as a test driver.
Moreover, when you test software, you need test cases with sufficient coverage. Using just one application as a test driver is probably insufficient, because this one application might not use all the features the framework offers.
In addition, it can be difficult to find realistic test scenarios when there are no precursor applications that you could use.
Solution
Set up tests based on the pilot applications.
These tests can take different forms:
- Identify core components from the pilot applications that call the framework, and then use these components as test drivers for the framework.
- Find typical use cases from the pilot applications and maintain them as a test suite.
- Shape these test cases into regression tests that you can run before every release of a new version of your framework.
However, you can’t rely on the pilot applications alone. You also have to include some exotic scenarios in your test suite, ones that take your framework to its limits and that can detect more unexpected bugs.
In addition, you need test cases that test the time, performance, and stability of your framework under load.
Examples
The Data Access Layer Framework
The two-dimensional versioning of application data had to be tested with real-world examples. The first time the framework team performed such realistic tests was in a two-week workshop together with the team that developed the health insurance system. In this workshop, the framework team learned some subtle details about two-dimensional versioning that they had not yet implemented. The framework team was therefore able to do some fine-tuning at a relatively early stage. Overall, these tests were very successful, because the few changes that were necessary could quickly be made. These tests would not have been possible without the pilot users—the health insurance application team.
Moreover, the realistic examples that were used in the workshop represented typical use cases for the health insurance system. The framework team used these scenarios as a test suite for a series of future framework versions.
The customer system acted as the second pilot application. The framework team occasionally tested together with the customer system team, since this reduced the necessary testing effort for both teams. Everybody was able to fix problems very quickly, which was equally good for both teams.
The Web Portal Framework
Tests of the Web portal framework were always performed using the pilot applications. It was extremely convenient to have two applications at hand that provided realistic test cases. New versions of the framework were only released after they had been approved by test teams that had checked whether the integration of the life insurance system and the customer system worked smoothly.
In addition to this, performance tests were carried out from time to time to check the portal’s time performance.
Discussion
Testing is an activity where the Pilot Applications (4) are particularly helpful. Finding real-world test scenarios would otherwise be very difficult.
Joining efforts with the users for testing is a particular kind of Framework User Involvement (7) from which both the framework developers and users can benefit. The framework developers receive valuable test scenarios, while the framework users can run tests with the framework developers readily available for immediate bug fixing, if necessary.