- 3.1 xUnit Test Frameworks
- 3.2 In-Browser Test Frameworks
- 3.3 Headless Testing Frameworks
- 3.4 One Test Runner to Rule Them All
- 3.5 Summary
In this chapter we will take a look at "the tools of the trade"—essential and useful tools to support a test-driven workflow. The most important tool is of course the testing framework, and after an overview of available frameworks, we will spend some time setting up and running JsTestDriver, the testing framework used for most of this book's example code. In addition to a testing framework, this chapter looks at tools such as coverage reports and continuous integration.
3.1 xUnit Test Frameworks
In Chapter 1, Automated Testing, we coined xUnit as the term used to describe testing frameworks that lean on the design of Java's JUnit and Smalltalk's SUnit, originally designed by Kent Beck. The xUnit family of test frameworks is still the most prevalent way of writing automated tests for code, even though the past few years have seen a rise in usage for so-called behavior-driven development (or BDD) testing frameworks.
3.1.1 Behavior-Driven Development
Behavior-driven development, or BDD, is closely related to TDD. As discussed in Chapter 2, The Test-Driven Development Process, TDD is not about testing, but rather about design and process. However, due to the terminology used to describe the process, a lot of developers never evolve beyond the point where they simply write unit tests to verify their code, and thus never experience many of the advantages associated with using tests as a design tool. BDD seeks to ease this realization by focusing on an improved vocabulary. In fact, vocabulary is perhaps the most important aspect of BDD, because it also tries to normalize the vocabulary used by programmers, business developers, testers, and others involved in the development of a system when discussing problems, requirements, and solutions.
Another "double D" is Acceptance Test-Driven Development. In acceptance TDD, development starts by writing automated tests for high level features, based on acceptance tests defined in conjunction with the client. The goal is to pass the acceptance tests. To get there, we can identify smaller parts and proceed with "regular" TDD. In BDD this process is usually centered around user stories, which describe interaction with the system using a vocabulary familiar to everyone involved in the project. BDD frameworks such as Cucumber allow for user stories to be used as executable tests, meaning that acceptance tests can be written together with the client, increasing the chance of delivering the product the client had originally envisioned.
3.1.2 Continuous Integration
3.1.3 Asynchronous Tests
3.1.4 Features of xUnit Test Frameworks
Chapter 1, Automated Testing, already introduced us to the basic features of the xUnit test frameworks: Given a set of test methods, the framework provides a test runner that can run them and report back the results. To ease the creation of shared test fixtures, test cases can employ the setUp and tearDown functions, which are run before and after (respectively) each individual test in a test case. Additionally, the test framework provides a set of assertions that can be used to verify the state of the system being tested. So far we have only used the assert method which accepts any value and throws an exception when the value is falsy. Most frameworks provide more assertions that help make tests more expressive. Perhaps the most common assertion is a version of assertEqual, used to compare actual results against expected values.
When evaluating test frameworks, we should assess the framework's test runner, its assertions, and its dependencies.
18.104.22.168 The Test Runner
A related concern is the test report. Clear fail/success status is vital to the test-driven development process, and clear feedback with details when tests fail or have errors is needed to easily handle them as they occur. Ideally, the test runner should produce test results that are easily integrated with continuous integration software.
Additionally, some sort of plugin architecture for the test runner can enable us to gather metrics from testing, or otherwise allow us to extend the runner to improve the workflow. An example of such a plugin is the test coverage report. A coverage report shows how well the test suite covers the system by measuring how many lines in production code are executed by tests. Note that 100% coverage does not imply that every thinkable test is written, but rather that the test suite executes each and every line of production code. Even with 100% coverage, certain sets of input can still break the code—it cannot guarantee the absence of, e.g., missing error handling. Coverage reports are useful to find code that is not being exercised by tests.
A rich set of assertions can really boost the expressiveness of tests. Given that a good unit test clearly states its intent, this is a massive boon. It's a lot easier to spot what a test is targeting if it compares two values with assertEqual(expected, actual) rather than with assert(expected == actual). Although assert is all we really need to get the job done, more specific assertions make test code easier to read, easier to maintain, and easier to debug.
Ideally, a testing framework should have as few dependencies as possible. More dependencies increase the chance of the mechanics of the framework not working in some browser (typically older ones). The worst kind of dependency for a testing framework is an obtrusive library that tampers with the global scope. The original version of JsUnitTest, the testing framework built for and used by the Prototype.js library, depended on Prototype.js itself, which not only adds a number of global properties but also augments a host of global constructors and objects. In practice, using it to test code that was not developed with Prototype.js would prove a futile exercise for two reasons:
- Too easy to accidentally rely on Prototype.js through the testing framework (yielding green tests for code that would fail in production, where Prototype.js would not be available)
- Too high a risk for collisions in the global scope (e.g., the MooTools library adds many of the same global properties)