Home > Articles > Programming

This chapter is from the book

The Recipe

You can follow many different approaches to testing. You can test the rough functionality first and then selectively refine it. You can select a functional, architectural, or design section and dive into it thoroughly. There is value in taking a more ad hoc or random approach akin to manual exploratory testing. Yet another approach guides your testing by metrics, such as defect densities, complexity, or criticality.

This section details an approach, visualized in Figure 3-1, that I have found useful in driving toward high coverage, independent of how you select the code or functionality to test. The approach favors deep over broad testing. It works well when taking a test-driven approach to new code, but also applies well when reproducing bugs, enhancing existing code, or bringing existing code under test.

Figure 3-1

Figure 3-1: Visualization of the testing recipe

Test the “Happy Path”

The “happy path” of code or functionality is the primary purpose, the main reason the software exists. If you composed a single sentence to describe what the software does, you would describe the happy path.

Testing the happy path lays the foundation on which the rest of your tests are built. It establishes the context in which all of the variations add further value. In the sense of tests as documentation, it expresses the purpose in an executable form that can capture regressions and evolve with the functionality.

The happy path may require several tests to fully verify depending on the scope of the test. Start with the one or two characteristic purposes for the initial tests. For a unit test, a single test should capture it. For a full stack system test, it may require a suite of tests, but breaking that suite down into functional areas should make the task at hand manageable.

Test the Alternate Paths

Once you have established that the primary functionality works as intended, you can tackle the useful variations of normal behavior. For example, if the primary functionality was to save a file, special accommodations for network file systems might be a good alternate path. At a unit-test level, you might make sure that an event-processing loop functions properly when no events are queued.

At this point, your coverage targets should guide your thoroughness. If you are doing coverage-driven unit testing, then you will want to test exhaustively. You will most likely direct yourself by a sense of functional coverage for full stack system tests. Subsystem integration tests will strive for a more local definition of completeness.

Test the Error Paths

Many people stop before testing the error handling of their software. Unfortunately, much of the perception of software quality is forged not by whether the software fails, because it eventually will, but by how it handles those failures. The world is full of unexpected occurrences. Even if your software runs on stand-alone, hardened computers, it will eventually fail. Power fluctuations, magnetic interference, and component failure are just a few of the many things that can happen, often in a cascading chain.

Error-handling verification ensures that your response to the deviant variations in your environment are deliberate rather than accidental. Deliberate error handling should give your user the experience you desire and hopefully the one they desire as well.

Many organizations skip or skimp on error-path testing because of the difficulties involved. Generally, inducing errors in larger scopes is harder than in smaller ones. Simulating network errors at a class level is much easier than for a full stack application. Making error handling an architectural concern with clearly defined guidelines for how components should participate in that framework outlines the intent to verify at the lower levels, allowing you to focus on the correctness of the local behaviors.

Test the Data Permutations

Data5 drives almost all software. When testing user interfaces and public APIs, boundary and validation conditions significantly impact the security and stability of your software. At more programmatic levels, various forms of data-controlled behaviors can comprise non-trivial portions of the functionality. Even in statically typed languages like Java and C#, higher levels of abstraction in your system design naturally decrease the effectiveness of code coverage as a guide for complete testing. Dynamic languages and features like reflection-based execution compound the challenge.

Boundary Conditions

One of the more common forms of data variations in software behavior arises from boundary conditions. Boundary conditions occur for a wide range of reasons. Your happy and alternate path tests verify the behavior within normal input values, but may not test all input values. Boundary condition tests verify how the software behaves

  • At the edges of the normal inputs to detect problems like off-byone errors
  • At the edges of the abnormal inputs also to detect off-by-one errors
  • Using anticipated variations of abnormal inputs for concerns like security
  • Using specifically dysfunctional abnormal inputs such as divide-by-zero errors or using inputs that trigger contextually determined limits such as numerical accuracy or representation ranges

You may have tested some boundary conditions when testing error paths. However, looking at the variations from the perspective of boundary conditions can highlight omissions in error-handling logic and drive more thorough test coverage.

Natural or pragmatic value and resource constraints provide a rich vein of boundary conditions. Natural limits occur when using values with a naturally finite set of states. True/false and yes/no are the most trivial of these. Menu picks that ask the user to choose from a limited number of options also provide contextually natural constraints. Pragmatic limits like field lengths yield a rich source of boundary conditions, especially when you manipulate or append to the input data internal to the software. At the resource-constrained or extreme end of the spectrum, you can test limits like memory and file size.

Numerical and mathematical variations can be thought of as natural or pragmatic but have a broad yet specialized enough affinity to deserve their own treatment and attention. Division-by-zero errors are perhaps the most common mathematical issues in programming, requiring attention regardless of representation format or size. Value limits due to discrete representations continue to factor into consideration, as the migration to wider representations is balanced by the inevitable increase in data volumes. Precision presents a more complicated set of conditions to test, as accuracy issues affect both the code being tested and the test code.

Standards- and convention-based formats yield structured and predictable, yet sometimes complex, patterns from which to derive boundary conditions, particularly as they evolve. For example, the syntactic rules of the Domain Name System (DNS)6 are relatively simple. However, you can find opportunities for startling variations even within this simplicity. Security concerns drive people to attempt to validate domains. Those who choose not to validate them through lookup, regardless of whether for good or bad reasons, must make assumptions about the rules of domain names that go beyond the syntactic conventions. I have seen code that assumes that all top-level domains (TLDs) must be two or three characters in length, as was true for most of the original set of TLDs. This ignores the originally allocated single-letter domains used for administrative purposes and does not automatically account for the longer TLDs that have been and will be added, such as .name and .info. Expansion of the DNS syntax to allow non-European character sets adds another wrinkle to validation.

More ad hoc or unstructured sources provide some of the most challenging inputs to predict. Any free-form text field has numerous considerations to validate. The simplest may involve restrictions on or stripping of white space or selection from a limited character set. The more complex can include evaluating inputs to detect SQL injection or cross-site scripting attacks and natural language processing for semantic content.

Data-Driven Execution

Guiding tests by code coverage, particularly at the unit level, works well to test behavioral variations that derive from code structure. However, many constructs provide significant behavioral variations without explicit branches in the code. The so-called Fundamental Theorem of Software Engineering7 says, “We can solve any problem by introducing an extra level of indirection.”

A common data-driven scenario arises when processing command-line or some remote-invocation interfaces in which a dispatcher uses an Abstract Factory to generate Command pattern [DP] objects for execution, as shown in Listing 3-2. The function of the CommandFactory and each of the available Command implementations should be tested in their own right, but the CommandDispatcher integrates the behaviors to create a larger set of behaviors that cannot be identified through static analysis or evaluated for coverage.

Listing 3-2: A dispatcher using an Abstract Factory in a data-driven way to create Command pattern objects to do the work

class CommandDispatcher {
  private CommandFactory commandFactory;

  public void dispatch(String commandName) {
    Command command =
      commandFactory.createCommand(commandName);
    command.execute();
  }
}

When testing these constructs at the unit level, we should verify the correctness of the dispatch mechanism. Ideally, the definition of the dispatch targets is dynamic or separate in a manner conducive to independent testing. We should test each of the dispatch targets independently.

For tests at a larger scope, like system or integration tests, we must test each of the dynamic variations to ensure thorough testing of the software. A dispatch mechanism that works generically at the unit level typically has a well-defined and finite set of possibilities when integrated into a component or system.

Run-Time and Dynamic Binding

Most languages that run in a virtual machine and/or are dynamically bound like scripting languages have a feature called reflection. Reflection provides the ability to inspect the program’s namespace at runtime to discover or verify the existence of elements like classes, functions, methods, variables, attributes, return types, and parameters and, where applicable, invoke them.

The ability to access or invoke arbitrary symbols resembles a built-in form of data-driven execution based on data maintained by the runtime system but with a higher degree of capability and flexibility than most applications will create on their own. The power of reflection has led many teams to discourage or outright ban it from their applications to avoid some justifiably distasteful uses. In languages like Java (Listing 3-3) or Perl, this will not inhibit most applications excessively. Languages like Smalltalk and JavaScript (Listing 3-4) suffer without the use of these features. Even if your team avoids writing reflection-based code, many frameworks, like Java Spring and Quartz, use reflection extensively to enable configuration-based application assembly and dependency injection.

Listing 3-3: Basic dynamic invocation in Java using reflection, omitting error handling and exceptions

class Invoker {
  public static void invokeVoidMethodNoArgs(String className,
      String methodName) {
    Class clazz = Class.forName(className);
    Object object = clazz.newInstance();
    Method method = class.getMethod(methodName, null);
    method.invoke(object, null);
  }
}

Listing 3-4: Basic dynamic invocation in JavaScript

function invokeNoArgsNoReturn(object, func) {
  if (object[func] && typeof object[func] === "function") {
    object[func]();
  }
}

Even less capable languages for reflection, such as C and C++, can exhibit some of the dynamic-binding properties of reflection-able language through POSIX dynamic library APIs like dlopen(3) as shown in Listing 3-5. This API gives the application the ability to load a shared library dynamically and to invoke functions within it, all by specifying the library and function names as strings under the constraint that the invocation signature is known.

Listing 3-5: Runtime binding with the POSIX dynamic library API in C without error handling

#include <dlfcn.h>

int main(int argc, char **argv)
{
  void *lib;
  void (*func)(void);
  lib = dlopen(argv[0], RTLD_LAZY);
  func = dlsym(lib, argv[1]);
  (*func)();
  dlclose(lib);
  return 0;
}

Just as in data-driven execution, tests need to verify that the mechanism for the dynamic invocation works at the unit level and that the assembled pieces work together at the higher levels.

Test the Defects

No matter how much you test your code, there will be defects. If your team does its job well, you will find all of your significant defects before you get to production. Regardless of when and by whom the defect is found, writing a test that duplicates the defect and then passes after the fix helps you know that you have fixed the defect and ensures that the defect remains fixed over time.

I prefer to test each defect, at least at the unit level. Every defect, including defects that can be broadly described as integration or interaction problems, trace to one or more defects in a unit of code. Perhaps the caller passes the wrong parameters or invokes functionality in the wrong order. Perhaps the callee does the wrong thing with the arguments or returns the wrong format or value. Maybe the synchronization is handled in a way that allows occasional race conditions. All of these and more can be duplicated and fixed at the unit level.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020