Home > Articles > Programming

Driving Your Development with Acceptance Tests: An Interview with Markus Gärtner

Acceptance test-driven development (ATTD) sure sounds nice. But what does it look like in practice? How do developers write the ‘plumbing’ code to automate the tests, and how can they help testers and customers to write test cases? To find answers, Matt Heusser interviewed Markus Gärtner, author of ATDD by Example: A Practical Guide to Acceptance Test-Driven Development.
Like this article? We recommend

One of Europe's premier software consultants, he is also an instructor at the Miagi-Do School of Software Testing, will give his first international keynote address at Agile-Testing Days 2012 in Berlin, is just finishing up the tour for his first book…and now we have him here on InformIT. Meet Markus Gärtner, software test/developer and trainer for it-agile GmbH, and an advocate for acceptance test-driven development (ATDD), a development method in which customers specify their requirements by example—so precisely that developers can write tests before creating the production code.

With ATDD, when the code is complete, the programmers can demonstrate it's complete by running the tests. They also know "the code is done, because the automated tests run."

Markus' recommendations are a little more nuanced—he also recommends exploratory testing, for example—as he explains in detail in his book ATDD by Example: A Practical Guide to Acceptance Test-Driven Development, which shows what ATDD is really like, with specific code examples.

I asked Markus to tell us what it was like to transition to ATDD, share any hints that could help us avoid bumps in the road, describe what ATDD looks like in practice, and explain what he saw coming as the next great advancement in software quality.

Matt Heusser: When we talk about things like automated tests, I get a little worried. What is our goal here? Is it to eliminate manual tests? What does ATDD do for us?

Markus Gärtner: Like any tool, automation should support our activities as much as possible. There are certain activities where automation doesn't help at all, and there are other activities where automation brings us great relief. Since reading Harry Collins' book Tacit and Explicit Knowledge, [1] I like to compare test automation to the process of baking bread. Certain actions all lead to the same results when we execute them; Collins calls these mimeomorphic actions. When baking bread, kneading the dough is a mimeomorphic action. You can automate such actions, but you must not forget the decisions surrounding those actions that are not repeatable and will need an engaged, thinking human. When it comes to bread-baking automation, such actions might include coming up with a recipe for the bread, selecting the ingredients in the supermarket, deciding how thin to slice the bread once it's been baked, and choosing whether to cover a slice with peanut butter or strawberry jam. Collins calls these polimorphic actions—that is, actions that can lead to varying results when repeated.

Thinking about that with our eyes focused on test automation, software development teams have to make the right tradeoffs—which particular types of tests to automate, and which to leave as tests that an engaged human brain executes. Testing quadrants (see Figure 1) are a tool to help you come up with the right balance between the different types of tests. Michael Bolton's distinction between testing and checking [2] goes in this direction as well. But the bigger lesson you can learn from the testing quadrants is actually that you need to involve testers, programmers, and business experts when coming up with a balanced strategy for your efforts in the course of the iteration.

Regarding ATDD, another goal is involved. One of the main goals for ATDD is to "trawl for requirements," as Mike Cohn puts it. [3] This approach, when applied well, helps the team to come up with tested requirements, and it helps us to find the right things to build. We'll still need to explore these functions with an engaged human brain at the point when these testable requirements indicate that we might be ready to ship the product.

Matt: Tell me more about how ATDD works in practice. Could you explain what happens—the life of a story, maybe? Perhaps you could use an example from a real client company?

Markus: ATDD doesn't need to start with a story right away. More often than not, it starts even before the story—sometimes it starts with a poor story, and sometimes there is already a quite okay story to get started. While working on the acceptance criteria for an existing story, the team often identifies another story derived from the current story, or they find out that the story needs to be cut, or it isn't ready for the iteration. I remember endless client workshops where we entered with a product backlog, only to find out that we didn't have a clue about what to build at all, and afterwards we ended up with a completely new backlog.

At least three different roles in building the thing must get together to reach a common understanding of the story. The three roles include a minimum of one tester, one programmer, and one business expert (like the product owner in Scrum). They discuss the story and derive acceptance criteria for it based on their discussion. Ron Jeffries' three Cs for user stories refer to card, conversation, and confirmation. [4] The confirmation part is the interesting thing here, and that's derived from the conversation.

When the "three amigos," as George Dinwiddie calls them, [5] arrive at a basic set of acceptance criteria, before the story enters an iteration, testers will work on refining these examples. Usually there will be too few examples for the team to be able to start working on it. With their knowledge about boundaries, edge cases, and critical thinking skills, testers help to flesh out the story to the degree that the team needs. This still happens before work starts on the story. Sometimes this goes on parallel to the iteration itself, but then it might prove a risk. One client I knew had the problem that programmers and testers weren't talking enough. The programmers were pretty surprised when the examples had grown, and they hadn't yet thought about many edge cases. At first, this can be very unexpected. Over time, I expect teams to get rid of these sort of surprises, and start on the common understanding even earlier.

In the iteration, programmers start to program the story and extend the system with the new functionality. Testers and programmer also work on automating the examples as automated checks. There they'll stay as close as possible to the original discussion and the acceptance criteria they defined. They'll work together when necessary—that does not mean that testers need to program. I once worked with a client that had educated nurses as their testers. It was pretty clear that none of the nurses would ever learn how to program. But the team did have some testers who were inclined toward programming, and they were eager to learn how to automate the examples, with help from the programming team. In the end, they became the automation toolsmiths. This was a good way to solve that problem as a whole development team.

Finally, the story makes it into the final product. The team decides which of the automated examples to keep, and they might show the results of the automated tests to their business representative. One client of mine had invited customers from another country, and I presented our examples to them in a one-hour meeting. Afterward I had the impression that the customer actually wanted to spend money to buy our suite of automated tests. I found out later that they had troubles similar to ours with understanding the system, and they didn't have a suite of automated tests that could help them to understand the dependencies for the various decisions involved in building the next features.

Matt: It sounds great in theory, and I have a long-term client that has seen a great deal of success with ATDD. Yet, for every success, there seem to be several failures. Why do you think that is—and what can we do about it?

Markus: Back in 2010, I was taking a closer look at the books on this topic. Even with the focus on outside-in development, ATDD wasn't very new at that time; I traced back books on it to at least 2005. One thing that puzzled me, though, was the number of books that tried to explain the approach by focusing on its principles. Through Alistair Cockburn, I had a concept of the Shu Ha Ri model for learning. On the Shu level, someone who is new to a particular skill doesn't learn by applying principles. Normally that's a Ha-level understanding, reached through years of practice. Instead, a Shu-level learner needs clear advice and a working set of things to do to get started. None of the books I explored described these things clearly—or I found the discussions to be too outdated.

That's when I decided to close the gap. I had read Kent Beck's Test-Driven Development by Example, [6] which consists of three parts: Parts 1 and 2 show a working example of how to apply the approach, and part 3 describes some of the principles behind it. I found this structure compelling, so I asked whether Kent thought writing about it was a good idea, and whether he would support me. A year and a half later, I submitted the final revision following review comments to Addison-Wesley, applying the structure to ATDD.

As it stands, ATDD by Example: A Practical Guide to Acceptance Test-Driven Development gives two different ways of working with ATDD. One approach is side-by-side development, with testers and programmers working in parallel; the other approach describes an outside-in development technique that derives the application domain model from the acceptance criteria. With these two sketches, Shu-level learners can get started in their own companies, over time reaching a set of principles that work for them. I also provide some guidance to trade off different factors in the third part of the book. The whole mix helps to avoid the failures you mentioned.

Matt: What do you think are some of the common failure modes in ATDD? Is it in the initial adoption? Do companies get things wrong? Or is it more long-term issues? I know many companies in which the test suite started out great, but quickly became slow and clunky.

Markus: From what I have seen at clients' offices, I could conclude that neither testers nor programmers pay much attention to the nature of test automation. In the end, test automation is software development. To me, that means rigorously applying techniques such as SOLID design principles, TDD, and refactoring to my test automation code. I strive toward taking more care with my automation code than with the production code; I want to keep that steady loop of quick feedback as vital as possible. Also, I become pretty annoyed when it takes longer to express what I want to test than it would take to write down the code. This is usually a sign of obfuscated design, a problem with my automation code, or both.

When the client's problem is failed test automation, I often find long test methods. Writing and then copying-and-pasting long test methods are the worst things I've seen, and they're usually sort of a starting point for test automation—on whatever level—to fail. Next is coming up with test helpers, but (despite their weird conditional logic) not unit-testing the helpers on their own. Ultimately, we want to reuse them, but reusing a component comes at a price—and that price is transferring the knowledge of that component to the next person. I once worked on a project where we learned this fact the hard way over the course of roughly two years. The solution was to start from scratch, but most projects can't do that. We were lucky enough to be able to do so, and within 18 weeks we replaced a failed test automation approach that had "grown" for about a year. Of course, we were able to put a year of learning into it as well. Two years later, the components we grew and unit-tested were still in widespread use in that company.

For me, it boils down to keeping an eye on what you do, reflecting on your worst problems, and fixing them. If you don't have such an empirical process in place, you'll probably run down that rabbit hole too fast, unable to back up. In places where test automation fails, I suspect that something else isn't working right—maybe retrospectives not coming up with proper actions to cope with the biggest problem, a disassembled team, or management incentives that are counter-productive. I try to keep an eye on those things as well.

Matt: These acceptance tests—do you recommend that they drive the entire application, including the graphical user interface ("end-to-end" tests), or should they get behind the GUI to test the business logic? Or is it some of both? How do companies decide?

Markus: In my experience, a single answer won't address all the problems. In software development, this shouldn't be too surprising, since we face tradeoff decisions in our daily work. Tradeoffs might involve binding a unit test too tightly to your code, so that you can't change the code without changing the unit test. Or tying the code to a particular framework that will help you come up with the data bindings for your database, but will become a nightmare to test automatically. And then there's the "vi versus emacs" discussion, but don't get me started on that topic.

I recommend taking into account what your codebase currently does. How hard is it to automate checks for your domain logic? If there's no direct entry point, you might have to automate tests on the GUI level only. I once worked with a client that had three different tiers of servers. The middle tier wasn't well known, but there were several interfaces between the GUI and the middle-tier server. While refactoring the GUI to fit a model-view-presenter pattern, we were able to discover and rebuild the middle-tier API, thereby enabling more tests to go behind the GUI.

Usually I prefer to run as many tests as possible behind the GUI. GUIs come with unpleasant behavior such as asynchronous thread-handling, where automation has to wait, and it becomes slow and brittle pretty quickly. Whenever possible, I strive to automate behind the GUI level; once the business logic is automated, I can decide how to close the gap between the GUI and the business logic. That could mean running fewer tests manually, or automating fewer slow tests on the GUI level, but not investing too much money and time into it. In the end, we have to maintain all that stuff that we automated, right?

Matt: Part of the theme I'm picking up here is that testers and product owners can learn some programming aspects. Maybe they learn a very powerful domain-specific language; maybe they learn a scripting language. Do you find many non-technical people willing to learn these tools? Do you encounter much resistance? If so, what do you do about it? You told us earlier about a group of educated nurses; can you give us a few more examples of how companies deal with this problem?

Markus: Well, the fact that I have seen business experts use something like Cucumber, and nurses learn some table structure, doesn't mean that they were capable of doing good automation. One crucial ingredient to me is the ability to program and find abstractions. More often than not, this means putting programmers, testers, and business experts in touch with each other. I think this is a magic ingredient, somehow, and maybe we should focus more on that. Behavior-driven development (BDD) tries to do that, but neglects the thought that you cannot automate every test.

Regarding resistance, I think it doesn't pay off to have customers write examples alone, but many companies try to do that. "Oh, we just have the business expert sit alone in the corner, write some examples in a tool she doesn't really know," and then they find out three months later that their approach isn't working. The crucial ingredient to a working approach—whatever name you pick—is to have programmers, testers, and business experts work together on the examples. I provide an example for that in Chapter 1 of my book, during the workshop, where a lot of stuff happens between the lines. While at clients' offices, I try to grasp the emotions of the situation and help people see things from different angles, and not run too far into technical details where that's unnecessary. Most of the resistance comes from boring collaboration where people don't listen to each other, or they don't respect the different focuses and try to understand the other parties at the table.

Even then, there are surprises. I saw a team adopt ATDD within one day. I entered the team room to give a workshop on ATDD; by the end of the day, this team had pulled off a story from their backlog, came up with examples for it, and tried out three different automation tools for it. One smaller group even managed not only to automate the examples, but to implement the whole story—all within one day. This team was pretty far ahead when it comes to technical excellence. That's why they were able to work in such an advanced manner. At other locations I have seen test automation run by business experts alone. We helped with getting automation started there, and half a year later they had tripled the number of examples without any further help. I also have helped a company getting started with ATDD and test automation at behind-the-GUI level. The approach was introduced very slowly. It finally took off once this particular team decided to refactor the middle layer of their system and could see the benefits of a safety net of automated checks for their system in that period of time.

These are just three examples. I think there are infinite other ways to discover out there.

Matt: If companies want to adopt a more examples-driven way of working, where would you suggest they start?

Markus: My short suggestion would be to start wherever they are currently. I actually find it hard not to start with wherever companies are. Things are the way they are because they got that way, as Jerry Weinberg taught me. That said, I don't see value in completely changing everything in a company. Instead I try to see the context of the team. Who do they need to work with? What are the particular skills in the team? What would a "plan zero" for this particular group look like? Usually that means that I fit the elements of workshops, backlog grooming, examples, and test automation onto the team, and they make their own experiences. Over time, they should be able to reflect on their approach and tune it to fit their particular needs.

Of course, this approach fails in companies that don't develop the necessary self-reflection skills or that put obstacles in the way of team learning and responsibility. But I think this goes for any new approach in software development—and that's probably why Agile approaches embrace retrospectives on a short timescale such as every two to four weeks. As Elisabeth Hendrickson taught me, "Empirical evidence trumps speculation." Iterations deliver the empirical evidence that teams need to make adaptations ("pivots," to take a more recent buzzword from the Lean Startup folks) and make the hard decisions after collecting some experience. Every successful team I've seen experiments and tries to learn from it. Where people stopped doing that, I wouldn't give too much hope to getting started with anything new.

But I digress from your original question. Start wherever you are, learn what works for you, and then make adaptations as needed. If you want to use a basic set of practices to get started with a more examples-driven way of working, start with specification workshops, and invest some of your slack to come up with an automation approach that works for you. Then start to collect experience with your picks. That's it. No magic necessary.

Matt: Thank you for participating. Where can we go for more?

Markus: Shameless plug: People might want to visit my blog for news and additions on ATDD by Example: A Practical Guide to Acceptance Test-Driven Development. I am also working on some GitHub repositories that can help people take a closer look into more code to come up with a tool decision more easily. I've started to run Testautomation Coderetreats in Germany. And, of course, there are lots of articles to read, workshops to attend, and conferences where I'll be speaking.

References

[1] Harry Collins, Tacit and Explicit Knowledge. University of Chicago Press, 2010.

[2] Michael Bolton, "Acceptance Tests: Let's Change the Title, Too," 2010.

[3] Mike Cohn, User Stories Applied: For Agile Software Development. Addison-Wesley, 2004.

[4] Ron Jeffries, Extreme Programming Adventures in C#. Microsoft Press, 2004.

[5] George Dinwiddie, "The Three Amigos: All for One and One for All." StickyMinds.com, Nov. 29, 2011.

[6] Kent Beck, Test-Driven Development by Example. Addison-Wesley, 2002.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020