Home > Articles > Programming

The Realities of Software Testing

Unfortunately, in the real world you will never see a project perfectly follow any of the development models. You will never be given a thoroughly detailed specification that perfectly meets the customer’s needs and you will never have enough time to do all the testing you need to do. It just doesn’t happen. This chapter will help you understand that software testing doesn't alway go perfectly, and help you prepare for that eventuality.
This chapter is from the book

This chapter is from the book

In This Chapter

  • Testing Axioms
  • Software Testing Terms and Definitions

In Chapter 1, "Software Testing Background," and Chapter 2, "The Software Development Process," you learned about the basics of software testing and the software development process. The information presented in these chapters offered a very high-level and arguably idealistic view of how software projects might be run. Unfortunately, in the real world you will never see a project perfectly follow any of the development models. You will never be given a thoroughly detailed specification that perfectly meets the customer’s needs and you will never have enough time to do all the testing you need to do. It just doesn’t happen. But, to be an effective software tester, you need to understand what the ideal process is so that you have something to aim for.

The goal of this chapter is to temper that idealism with a reality check from a software tester’s perspective. It will help you see that, in practice, trade-offs and concessions must be made throughout the development cycle. Many of those trade-offs are directly related to the software test effort. The bugs you find and the problems you prevent all significantly affect the project. After reading this chapter, you’ll have a much clearer picture of the roles, the impact, and the responsibilities that software testing has and you’ll hopefully appreciate the behind-the-scenes decisions that must be made to create a software product.

The highlights of this chapter include

  • Why software can never be perfect
  • Why software testing isn’t just a technical problem
  • The terms commonly used by software testers

Testing Axioms

This first section of this chapter is a list of axioms, or truisms. Think of them as the "rules of the road" or the "facts of life" for software testing and software development. Each of them is a little tidbit of knowledge that helps put some aspect of the overall process into perspective.

It’s Impossible to Test a Program Completely

As a new tester, you might believe that you can approach a piece of software, fully test it, find all the bugs, and assure that the software is perfect. Unfortunately, this isn’t possible, even with the simplest programs, due to four key reasons:

  • The number of possible inputs is very large.
  • The number of possible outputs is very large.
  • The number of paths through the software is very large.
  • The software specification is subjective. You might say that a bug is in the eye of the beholder.

Multiply all these "very large" possibilities together and you get a set of test conditions that’s too large to attempt. If you don’t believe it, consider the example shown in Figure 3.1, the Microsoft Windows Calculator.

Figure 3.1

Figure 3.1 Even a simple program such as the Windows Calculator is too complex to completely test.

Assume that you are assigned to test the Windows Calculator. You decide to start with addition. You try 1+0=. You get an answer of 1. That’s correct. Then you try 1+1=. You get 2. How far do you go? The calculator accepts a 32-digit number, so you must try all the possibilities up to

1+99999999999999999999999999999999=

Once you complete that series, you can move on to 2+0=, 2+1=, 2+2=, and so on. Eventually you’ll get to

99999999999999999999999999999999+99999999999999999999999999999999=

Next you should try all the decimal values: 1.0+0.1, 1.0+0.2, and so on.

Once you verify that regular numbers sum properly, you need to attempt illegal inputs to assure that they’re properly handled. Remember, you’re not limited to clicking the numbers onscreen—you can press keys on your computer keyboard, too. Good values to try might be 1+a, z+1, 1a1+2b2,.... There are literally billions upon billions of these.

Edited inputs must also be tested. The Windows Calculator allows the Backspace and Delete keys, so you should try them. 1<backspace>2+2 should equal 4. Everything you’ve tested so far must be retested by pressing the Backspace key for each entry, for each two entries, and so on.

If you or your heirs manage to complete all these cases, you can then move on to adding three numbers, then four numbers,....

There are so many possible entries that you could never complete them, even if you used a supercomputer to feed in the numbers. And that’s only for addition. You still have subtraction, multiplication, division, square root, percentage, and inverse to cover.

The point of this example is to demonstrate that it’s impossible to completely test a program, even software as simple as a calculator. If you decide to eliminate any of the test conditions because you feel they’re redundant or unnecessary, or just to save time, you’ve decided not to test the program completely.

Software Testing Is a Risk-Based Exercise

If you decide not to test every possible test scenario, you’ve chosen to take on risk. In the calculator example, what if you choose not to test that 1024+1024=2048? It’s possible the programmer accidentally left in a bug for that situation. If you don’t test it, a customer will eventually enter it, and he or she will discover the bug. It’ll be a costly bug, too, since it wasn’t found until the software was in the customer’s hands.

This may all sound pretty scary. You can’t test everything, and if you don’t, you will likely miss bugs. The product has to be released, so you will need to stop testing, but if you stop too soon, there will still be areas untested. What do you do?

One key concept that software testers need to learn is how to reduce the huge domain of possible tests into a manageable set, and how to make wise risk-based decisions on what’s important to test and what’s not.

Figure 3.2 shows the relationship between the amount of testing performed and the number of bugs found. If you attempt to test everything, the costs go up dramatically and the number of missed bugs declines to the point that it’s no longer cost effective to continue. If you cut the testing short or make poor decisions of what to test, the costs are low but you’ll miss a lot of bugs. The goal is to hit that optimal amount of testing so that you don’t test too much or too little.

Figure 3.2

Figure 3.2 Every software project has an optimal test effort.

You will learn how to design and select test scenarios that minimize risk and optimize your testing in Chapters 4 through 7.

Testing Can’t Show That Bugs Don’t Exist

Think about this for a moment. You’re an exterminator charged with examining a house for bugs. You inspect the house and find evidence of bugs—maybe live bugs, dead bugs, or nests. You can safely say that the house has bugs.

You visit another house. This time you find no evidence of bugs. You look in all the obvious places and see no signs of an infestation. Maybe you find a few dead bugs or old nests but you see nothing that tells you that live bugs exist. Can you absolutely, positively state that the house is bug free? Nope. All you can conclude is that in your search you didn’t find any live bugs. Unless you completely dismantled the house down to the foundation, you can’t be sure that you didn’t simply just miss them.

Software testing works exactly as the exterminator does. It can show that bugs exist, but it can’t show that bugs don’t exist. You can perform your tests, find and report bugs, but at no point can you guarantee that there are no longer any bugs to find. You can only continue your testing and possibly find more.

The More Bugs You Find, the More Bugs There Are

There are even more similarities between real bugs and software bugs. Both types tend to come in groups. If you see one, odds are there will be more nearby.

Frequently, a tester will go for long spells without finding a bug. He’ll then find one bug, then quickly another and another. There are several reasons for this:

  • Programmers have bad days. Like all of us, programmers can have off days. Code written one day may be perfect; code written another may be sloppy. One bug can be a tell-tale sign that there are more nearby.
  • Programmers often make the same mistake. Everyone has habits. A programmer who is prone to a certain error will often repeat it.
  • Some bugs are really just the tip of the iceberg. Very often the software’s design or architecture has a fundamental problem. A tester will find several bugs that at first may seem unrelated but eventually are discovered to have one primary serious cause.

It’s important to note that the inverse of this "bugs follow bugs" idea is true, as well. If you fail to find bugs no matter how hard you try, it may very well be that the feature you’re testing was cleanly written and that there are indeed few if any bugs to be found.

The Pesticide Paradox

In 1990, Boris Beizer, in his book Software Testing Techniques, Second Edition, coined the term pesticide paradox to describe the phenomenon that the more you test software, the more immune it becomes to your tests. The same thing happens to insects with pesticides (see Figure 3.3). If you keep applying the same pesticide, the insects eventually build up resistance and the pesticide no longer works.

Figure 3.3

Figure 3.3 Software undergoing the same repetitive tests eventually builds up resistance to them.

Remember the spiral model of software development described in Chapter 2? The test process repeats each time around the loop. With each iteration, the software testers receive the software for testing and run their tests. Eventually, after several passes, all the bugs that those tests would find are exposed. Continuing to run them won’t reveal anything new.

To overcome the pesticide paradox, software testers must continually write new and different tests to exercise different parts of the program and find more bugs.

Not All the Bugs You Find Will Be Fixed

One of the sad realities of software testing is that even after all your hard work, not every bug you find will be fixed. Now, don’t be disappointed—this doesn’t mean that you’ve failed in achieving your goal as a software tester, nor does it mean that you or your team will release a poor quality product. It does mean, however, that you’ll need to rely on a couple of those traits of a software tester listed in Chapter 1—exercising good judgment and knowing when perfection isn’t reasonably attainable. You and your team will need to make trade-offs, risk-based decisions for each and every bug, deciding which ones will be fixed and which ones won’t.

There are several reasons why you might choose not to fix a bug:

  • There’s not enough time. In every project there are always too many software features, too few people to code and test them, and not enough room left in the schedule to finish. If you’re working on a tax preparation program, April 15 isn’t going to move—you must have your software ready in time.
  • It’s really not a bug. Maybe you’ve heard the phrase, "It’s not a bug, it’s a feature!" It’s not uncommon for misunderstandings, test errors, or spec changes to result in would-be bugs being dismissed as features.
  • It’s too risky to fix. Unfortunately, this is all too often true. Software can be fragile, intertwined, and sometimes like spaghetti. You might make a bug fix that causes other bugs to appear. Under the pressure to release a product under a tight schedule, it might be too risky to change the software. It may be better to leave in the known bug to avoid the risk of creating new, unknown ones.
  • It’s just not worth it. This may sound harsh, but it’s reality. Bugs that would occur infrequently or bugs that appear in little-used features may be dismissed. Bugs that have work-arounds, ways that a user can prevent or avoid the bug, are often not fixed. It all comes down to a business decision based on risk.

The decision-making process usually involves the software testers, the project managers, and the programmers. Each carries a unique perspective on the bugs and has his own information and opinions as to why they should or shouldn’t be fixed. In Chapter 19, "Reporting What You Find," you’ll learn more about reporting bugs and getting your voice heard.

When a Bug’s a Bug Is Difficult to Say

If there’s a problem in the software but no one ever discovers it—not programmers, not testers, and not even a single customer—is it a bug?

Get a group of software testers in a room and ask them this question. You’ll be in for a lively discussion. Everyone has their own opinion and can be pretty vocal about it. The problem is that there’s no definitive answer. The answer is based on what you and your development team decide works best for you.

For the purposes of this book, refer back to the rules to define a bug from Chapter 1:

  1. The software doesn’t do something that the product specification says it should do.
  2. The software does something that the product specification says it shouldn’t do.
  3. The software does something that the product specification doesn’t mention.
  4. The software doesn’t do something that the product specification doesn’t mention but should.
  5. The software is difficult to understand, hard to use, slow, or—in the software tester’s eyes—will be viewed by the end user as just plain not right.

Following these rules helps clarify the dilemma by making a bug a bug only if it’s observed. To claim that the software does or doesn’t do "something" implies that the software was run and that "something" or the lack of "something" was witnessed. Since you can’t report on what you didn’t see, you can’t claim that a bug exists if you didn’t see it.

Here’s another way to think of it. It’s not uncommon for two people to have completely different opinions on the quality of a software product. One may say that the program is incredibly buggy and the other may say that it’s perfect. How can both be right? The answer is that one has used the product in a way that reveals lots of bugs. The other hasn’t.

If this is as clear as mud, don’t worry. Discuss it with your peers in software testing and find out what they think. Listen to others’ opinions, test their ideas, and form your own definition. Remember the old question, "If a tree falls in the forest and there’s no one there to hear it, does it make a sound?"

Product Specifications Are Never Final

Software developers have a problem. The industry is moving so fast that last year’s cutting-edge products are obsolete this year. At the same time, software is getting larger and gaining more features and complexity, resulting in longer and longer development schedules. These two opposing forces result in conflict, and the result is a constantly changing product specification.

There’s no other way to respond to the rapid changes. Assume that your product had a locked-down, final, absolutely-can’t-change-it product spec. You’re halfway through the planned two-year development cycle, and your main competitor releases a product very similar to yours but with several desirable features that your product doesn’t have. Do you continue with your spec as is and release an inferior product in another year? Or, does your team regroup, rethink the product’s features, rewrite the product spec, and work on a revised product? In most cases, wise business dictates the latter.

As a software tester, you must assume that the spec will change. Features will be added that you didn’t plan to test. Features will be changed or even deleted that you had already tested and reported bugs on. It will happen. You’ll learn techniques for being flexible in your test planning and test execution in the remainder of this book.

Software Testers Aren’t the Most Popular Members of a Project Team

Remember the goal of a software tester?

  • The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.

Your job is to inspect and critique your peer’s work, find problems with it, and publicize what you’ve found. Ouch! You won’t win a popularity contest doing this job.

Here are a couple of tips to keep the peace with your fellow teammates:

  • Find bugs early. That’s your job, of course, but work hard at doing this. It’s much less of an impact and much more appreciated if you find a serious bug three months before, rather than one day before, a product’s scheduled release.
  • Temper your enthusiasm. Okay, you really love your job. You get really excited when you find a terrible bug. But, if you bounce into a programmer’s cubicle with a huge grin on your face and tell her that you just found the nastiest bug of your career and it’s in her code, she won’t be happy.
  • Don’t just report bad news. If you find a piece of code surprisingly bug free, tell the world. Pop into a programmer’s cubicle occasionally just to chat. If all you ever do is report bad news, people will see you coming and will run and hide.

Software Testing Is a Disciplined Technical Profession

It used to be that software testing was an afterthought. Software products were small and not very complicated. The number of people with computers using software was limited. And, the few programmers on a project team could take turns debugging each others’ code. Bugs weren’t that much of a problem. The ones that did occur were easily fixed without much cost or disruption. If software testers were used, they were frequently untrained and brought into the project late to do some "ad-hoc banging on the code to see what they might find." Times have changed.

Look at the software help-wanted ads and you’ll see numerous listings for software testers. The software industry has progressed to the point where professional software testers are mandatory. It’s now too costly to build bad software.

To be fair, not every company is on board yet. Many computer game and small-time software companies still use a fairly loose development model—usually big-bang or code-and-fix. But much more software is now developed with a disciplined approach that has software testers as core, vital members of their staff.

This is great news if you’re interested in software testing. It can now be a career choice—a job that requires training and discipline, and allows for advancement.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020