Home > Articles > Programming

Automated Software Testing: An Example of a Working Solution

Elfriede Dustin, author of Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality, discusses the challenges involved in developing a custom automated software testing (AST) framework and provides some tips on how to address such challenges.
Like this article? We recommend

Most automated testing tool vendors—even open-source automated testing tool efforts—claim to have the "silver bullet" automated testing solution; yet the current crop of automated software testing (AST) tools and solutions is still riddled with numerous challenges.

Because vendor-provided tools and open source solutions alone didn't meet the needs of our project for the U.S. Department of Defense (DoD), the Innovative Defense Technologies (IDT) team developed a custom automation framework for the DoD's real-time, mission-critical systems. Our eventual solution used a mixture of available open source tools and other tools developed in-house.

This article summarizes my experience as part of the team that developed the Automated Test and Re-Test (ATRT) tool, now in use throughout Navy programs. Within this article you'll find automated testing hints that can be useful nuggets as part of any automated testing effort.

System Requirements for the AST Solution

IDT started by developing a clear statement of requirements and objectives for the automation, to ensure that our solution would take the right direction. After gaining a thorough understanding of the unique DoD automated testing challenge, we came up with seven major requirements for an automated software testing solution used on a typical DoD system under test (SUT):

  1. Cannot be intrusive to the SUT
  2. Must be OS-independent (compatible with Windows, Linux, Solaris, etc.)
  3. Must be GUI-independent (should be able to handle GUI technologies written in Motif, C#, etc., and handle any type of third-party non-custom GUI control)
  4. Must be able to handle GUI-centric and "backend" automation (handle test operations on SUTs with the GUI as well as operations on the SUT's backend, such as various messages and other data being sent)
  5. Must be able to handle a networked multicomputer environment (multiple servers and multiple monitors/displays interconnected to form one SUT)
  6. Non-developers should be able to use the tool (testers in the DoD labs were subject matter experts, but not necessarily software developers who could use an automated testing tool efficiently)
  7. Must support an automated requirements traceability matrix (RTM)

Our diligent research determined that while various automated testing tools on the market met one or more of these requirements, no existing tool met all seven requirements in one solution.

Challenge Solutions

This section describes how we tackled each of the seven automation requirements one by one and how we solved each challenge.

Requirement 1: Not Intrusive to SUT

Originally, we set out to reuse any applicable vendor-provided automated testing tool as part of our automated testing framework. Before knowing the detailed requirements, I thought we'd be using tools such as IBM's Rational Functional Tester, HP's QuickTest Pro and WinRunner, or SmartBear's TestComplete.

However, given that our first requirement was that the AST could not be intrusive to the SUT, we had to cross these tools off our list of potentials. While some partially met the requirement, none met it 100%, meaning that portions of the tool actually had to be installed on the SUT. Our customer wouldn't be happy with the tool being even partially intrusive.

Many of the SUTs we tested used virtual network computing (VNC) as part of their system configuration. We decided that an automated testing solution that can plug into the VNC server already installed on the SUT would allow our framework to meet the requirement of not being intrusive to the SUT. As part of our ATRT framework, we developed a VNC client that would allow connecting to the SUT's VNC server.

Requirement 2: Operating System Independence

We also had to be sure that the tool we selected as part of our ATRT framework wasn't dependent on a specific operating system (OS). Our client uses every type of OS imaginable, but initially we wanted to set out our proof-of-concept on Windows and Linux. Since various VNC versions exist for most operating systems, we were able to meet this requirement through our VNC solution.

Requirement 3: GUI Independence

Our solution needed to be able to handle any type of graphical user interface (GUI) technology written in any language—Motif, C#, and so on—and handle any type of third-party non-custom GUI control.

Many of the current vendor-provided AST tools depend on specific GUI technology. That means that if proprietary programming languages or third-party controls are used in the SUT GUI, the automated testing tool often isn't compatible, which presents automated testing problems. Our ATRT solution works by using VNC to interact with all the GUI elements of the SUT as images, independent of the GUI technology used.

Requirement 4: Tests Multiple Interface and Protocol Types

Our solution needed to be able to handle test operations on systems through their GUI but also should be able to handle operations on the system's backend (not through a GUI), such as various messages and other data being sent.

While our ATRT/VNC-based solution met the previously described GUI automation requirements well, our customer also needed to be able to test various backend (non-GUI) interfaces using various protocols. We couldn't find a vendor-provided solution that would support this requirement out of the box, so we had to develop an in-house solution for this backend "interface" testing.

This task was a more involved undertaking because we needed to understand the protocol requirements, which sometimes were proprietary, plus the makeup of the backend system. Our biggest challenge was the message data. Each SUT used a different protocol—including Transmission Control Protocol/Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Common Object Request Broker Architecture (CORBA), proprietary, and more—and all used different message data formats. We developed an approach whereby all protocols and data could be tested via our ATRT framework, and thus our ATRT development effort was born.

Eclipse

We decided to use the open source Eclipse development environment. We chose the Eclipse rich client platform because ATRT could simply expand on the base environment to allow for configurability/extensibility via a powerful plug-in framework, which allowed us to integrate various plug-ins into that framework. Various new feature sets and tools can be integrated, and plug-ins can be added, removed, or modified easily through the Eclipse framework. (See the next section for more on plug-ins.)

Eclipse provides design elements that have to be considered for the various features required when developing and adding to an automated testing framework. Eclipse is an open source community whose projects are focused on building an open development platform comprising extensible frameworks and tools for building, deploying, and managing software across the lifecycle. A large and vibrant ecosystem of major technology vendors, innovative startups, universities, research institutions, and individuals extend, complement, and support the Eclipse platform. Eclipse is much more than just a Java integrated development environment (IDE).

The exciting thing about Eclipse is that many people are using Eclipse in many ways. The common thread is that they are building innovative, industrial-strength software and want to use great tools and frameworks to make their job easier.

Plug-ins

Eclipse comes bundled with a plug-in development environment (PDE), which itself is a plug-in that streamlines plug-in development. Because the Eclipse framework is written in Java, the plug-in code ideally is written in Java as well. Other languages can be used in the plug-in development, but that approach would limit the multiplatform support you get with Java.

We created an Eclipse extension point that defines an interface for our framework, giving the framework user a method to interact with an external tool in a standard way. Support for various protocols was developed, and ATRT can now test message data sent over TCP/IP, UDP, CORBA, proprietary, and many other protocols.

After zeroing in on the most efficient development environment, we needed a way to develop test-message data. Although we could reuse samples of "live" message data (that is, message data that was actually used in the SUTs), we wouldn't get the data coverage we wanted. We needed a tool that would allow us to measure test-data coverage so we could cover a broad range of test data. We initially evaluated FireEye (now called Advanced Combinatorial Testing System [ACTS]), developed by the U.S. National Institute of Standards and Technology (NIST) for an orthogonal-array approach of data generation, but at the time it didn't allow for the complexity of data interactions and relations required. We determined that MATLAB was the solution for our complex test data generation.

Requirement 5: Handles Networked Multicomputer Environments

While the commercial world mainly seems to focus on web-based applications and web testing, the DoD's systems consist of networked computers—multiple servers, systems of systems, multiple monitors and displays interconnected to form one SUT. With its VNC use, ATRT handles such interconnected mesh of environments very well. For example, it handles distributed and concurrent testing over a network: Automated tests can be executed concurrently over the network for the test case where various GUI- or message-based outputs are dependent on each other over a network or have to run in parallel. As long as a VNC server is installed on any of the networked computers, ATRT (which now also serves as a VNC client) can connect to them.

Requirement 6: Scriptless Automated Test Development for Non-Developers

One of our main requirements was that non-developers must be able to use the ATRT framework. Testers in the DoD labs were subject matter experts, but not necessarily software developers who could use an automated testing tool efficiently. Non-developers generally don't want to be bothered with developing automated testing scripts; they want to be able to use the tool with the simple click of a button. We therefore developed a model-based approach to automated testing. Our ATRT solution allows testers to drag the desired action onto a "canvas" to develop their automated tests. No scripting is involved. The tester can click an icon that matches any mouse feature (left-click, right-click, center-click, mouse down, etc.); via that icon click, ATRT generates the respective code behind the scenes.

This model-driven automated testing capability allows test flows to be generated via a simple select-and-drag model-driven interface. We wanted to provide a solution whereby testers could model a test before the SUT becomes available, and they would be able to see a visual representation of their tests.

GUI capture and test modeling is described here, but the tool also allows for non-GUI message-based test-case modeling by non-developers.

In summary, we met the requirement that non-developers be able to use this framework. ATRT provides the tester with every GUI action a mouse can perform, in addition to various other keywords. Our experience with customers using ATRT for the first time is that it provides a very intuitive user interface. Interestingly, we find that some of the best test automators are younger testers who have grown up playing video games, coincidentally developing eye-hand coordination. Some of the testers even find working with ATRT to be fun, which alleviates the monotony of manually testing the same features repeatedly with each new release.

Requirement 7: Must Support an Automated Requirements Traceability Matrix

We needed a framework that could allow for test management and automated requirements traceability. We developed ATRT to handle all related test-management activities, including documenting a test case in the ATRT manager, along with a hierarchical breakdown of test cases into test steps and substeps. Additionally, to meet this final requirement, ATRT allows users to import their requirements and map test cases to requirements. From this, we allow for RTM reporting. We integrated the open source Business and Intelligence Reporting Tools (BIRT), which is an Eclipse-based reporting system.

Automated Testing Solution Applied: Return on Investment

We successfully applied our ATRT solution to various DoD programs and have shown return on investment (ROI). We developed a complex ROI matrix that allows us to measure things like these as accurately as possible:

  • Nonrecurring costs such as original setup and connection in the labs and required workarounds
  • Cost and time to create and maintain the automated vs. manual test cases
  • Learning curve

ROI includes time savings; for example, on one program for a 48-hour endurance test, we reduced the number of required testers from 5 to 1; those extra testers could now focus on new features. It also includes increased testing coverage; that is, our backend message-based testing allows a 100-fold increase in test-scenario testing and analyzing—testing that would be time-prohibitive if performed manually.

Although we showed ROI on all programs, our biggest challenge for bringing automated testing, ATRT, or any such type of framework into the DoD labs was often the testers themselves, who have done manual testing for years and often feel that their jobs are in jeopardy. After demonstrating ATRT on their test systems in labs, more often than not we hear the comment, "Why don't you just put a sign on these computers that says 'You have been replaced!'?"

Well, that's not quite true. Our main goal is not to replace testers but to make them more efficient. Our goal with bringing automated testing to the DoD systems (old and new) is to increase the speed of the testing certification process, which is currently mainly a manual process, with hundreds of thousands of dollars spent on inefficient testing processes. Often the mantra we hear is "This is how we have been doing it; don't rock the boat." ATRT is a new and more efficient way to implement testing on systems—it's not a silver bullet. We've shown initial savings, but we'll continue to enhance ATRT to be even more effective. Each day, our entire development and testing department focuses on developing a better automated testing solution. With each application to different systems, we learn more and add features accordingly.

One of our additional challenges is testing of ATRT. We need to make sure that ATRT behaves flawlessly. If the ATRT solution has problems, how can we claim to have an automated testing solution that will save you time and make you more efficient? We have a rigorous ATRT development process in place that includes software development using continuous integration, with code checked in daily to the source control tool (such as Subversion), nightly builds (using tools such as Hudson/Jenkins), and automated unit tests (using tools such as NUnit).

Automated unit and integration testing is required in order to add continuously to the automated ATRT unit and integration testing framework. The more interesting testing part here is that we use ATRT to functionally test ATRT—a technique known as "dog-fooding" or "eating your own dog-food." Along with the JUnit tests, we have automated functional ATRT tests in place that test all of the ATRT requirements in a nightly build. We've created a subset of automated tests of results reporting, and a build is not considered complete until it passes all automated unit and automated functional tests satisfactorily.

Conclusion

I'm excited to work at a company that's funded by the government to develop an automated testing solution. It allows me to see the testing challenges that testers face and to help create and provide automated solutions to these unique challenges while continuously evolving in effectiveness. It also made me realize that there's no need to reinvent the wheel: A successful automated testing framework can be built by reusing open source and in-house developed code. Just make sure that the manual testers don't feel threatened by a solution that will make them 10 times as effective, and be sure to understand the limitations of the currently available vendor-provided tools. While we have made great strides toward solving the automated testing challenge, there is still much more to be done, and better solutions are yet to come.

InformIT Promotional Mailings & Special Offers

I would like to receive exclusive offers and hear about products from InformIT and its family of brands. I can unsubscribe at any time.

Overview


Pearson Education, Inc., 221 River Street, Hoboken, New Jersey 07030, (Pearson) presents this site to provide information about products and services that can be purchased through this site.

This privacy notice provides an overview of our commitment to privacy and describes how we collect, protect, use and share personal information collected through this site. Please note that other Pearson websites and online products and services have their own separate privacy policies.

Collection and Use of Information


To conduct business and deliver products and services, Pearson collects and uses personal information in several ways in connection with this site, including:

Questions and Inquiries

For inquiries and questions, we collect the inquiry or question, together with name, contact details (email address, phone number and mailing address) and any other additional information voluntarily submitted to us through a Contact Us form or an email. We use this information to address the inquiry and respond to the question.

Online Store

For orders and purchases placed through our online store on this site, we collect order details, name, institution name and address (if applicable), email address, phone number, shipping and billing addresses, credit/debit card information, shipping options and any instructions. We use this information to complete transactions, fulfill orders, communicate with individuals placing orders or visiting the online store, and for related purposes.

Surveys

Pearson may offer opportunities to provide feedback or participate in surveys, including surveys evaluating Pearson products, services or sites. Participation is voluntary. Pearson collects information requested in the survey questions and uses the information to evaluate, support, maintain and improve products, services or sites, develop new products and services, conduct educational research and for other purposes specified in the survey.

Contests and Drawings

Occasionally, we may sponsor a contest or drawing. Participation is optional. Pearson collects name, contact information and other information specified on the entry form for the contest or drawing to conduct the contest or drawing. Pearson may collect additional personal information from the winners of a contest or drawing in order to award the prize and for tax reporting purposes, as required by law.

Newsletters

If you have elected to receive email newsletters or promotional mailings and special offers but want to unsubscribe, simply email information@informit.com.

Service Announcements

On rare occasions it is necessary to send out a strictly service related announcement. For instance, if our service is temporarily suspended for maintenance we might send users an email. Generally, users may not opt-out of these communications, though they can deactivate their account information. However, these communications are not promotional in nature.

Customer Service

We communicate with users on a regular basis to provide requested services and in regard to issues relating to their account we reply via email or phone in accordance with the users' wishes when a user submits their information through our Contact Us form.

Other Collection and Use of Information


Application and System Logs

Pearson automatically collects log data to help ensure the delivery, availability and security of this site. Log data may include technical information about how a user or visitor connected to this site, such as browser type, type of computer/device, operating system, internet service provider and IP address. We use this information for support purposes and to monitor the health of the site, identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents and appropriately scale computing resources.

Web Analytics

Pearson may use third party web trend analytical services, including Google Analytics, to collect visitor information, such as IP addresses, browser types, referring pages, pages visited and time spent on a particular site. While these analytical services collect and report information on an anonymous basis, they may use cookies to gather web trend information. The information gathered may enable Pearson (but not the third party web trend services) to link information with application and system log data. Pearson uses this information for system administration and to identify problems, improve service, detect unauthorized access and fraudulent activity, prevent and respond to security incidents, appropriately scale computing resources and otherwise support and deliver this site and its services.

Cookies and Related Technologies

This site uses cookies and similar technologies to personalize content, measure traffic patterns, control security, track use and access of information on this site, and provide interest-based messages and advertising. Users can manage and block the use of cookies through their browser. Disabling or blocking certain cookies may limit the functionality of this site.

Do Not Track

This site currently does not respond to Do Not Track signals.

Security


Pearson uses appropriate physical, administrative and technical security measures to protect personal information from unauthorized access, use and disclosure.

Children


This site is not directed to children under the age of 13.

Marketing


Pearson may send or direct marketing communications to users, provided that

  • Pearson will not use personal information collected or processed as a K-12 school service provider for the purpose of directed or targeted advertising.
  • Such marketing is consistent with applicable law and Pearson's legal obligations.
  • Pearson will not knowingly direct or send marketing communications to an individual who has expressed a preference not to receive marketing.
  • Where required by applicable law, express or implied consent to marketing exists and has not been withdrawn.

Pearson may provide personal information to a third party service provider on a restricted basis to provide marketing solely on behalf of Pearson or an affiliate or customer for whom Pearson is a service provider. Marketing preferences may be changed at any time.

Correcting/Updating Personal Information


If a user's personally identifiable information changes (such as your postal address or email address), we provide a way to correct or update that user's personal data provided to us. This can be done on the Account page. If a user no longer desires our service and desires to delete his or her account, please contact us at customer-service@informit.com and we will process the deletion of a user's account.

Choice/Opt-out


Users can always make an informed choice as to whether they should proceed with certain services offered by InformIT. If you choose to remove yourself from our mailing list(s) simply visit the following page and uncheck any communication you no longer want to receive: www.informit.com/u.aspx.

Sale of Personal Information


Pearson does not rent or sell personal information in exchange for any payment of money.

While Pearson does not sell personal information, as defined in Nevada law, Nevada residents may email a request for no sale of their personal information to NevadaDesignatedRequest@pearson.com.

Supplemental Privacy Statement for California Residents


California residents should read our Supplemental privacy statement for California residents in conjunction with this Privacy Notice. The Supplemental privacy statement for California residents explains Pearson's commitment to comply with California law and applies to personal information of California residents collected in connection with this site and the Services.

Sharing and Disclosure


Pearson may disclose personal information, as follows:

  • As required by law.
  • With the consent of the individual (or their parent, if the individual is a minor)
  • In response to a subpoena, court order or legal process, to the extent permitted or required by law
  • To protect the security and safety of individuals, data, assets and systems, consistent with applicable law
  • In connection the sale, joint venture or other transfer of some or all of its company or assets, subject to the provisions of this Privacy Notice
  • To investigate or address actual or suspected fraud or other illegal activities
  • To exercise its legal rights, including enforcement of the Terms of Use for this site or another contract
  • To affiliated Pearson companies and other companies and organizations who perform work for Pearson and are obligated to protect the privacy of personal information consistent with this Privacy Notice
  • To a school, organization, company or government agency, where Pearson collects or processes the personal information in a school setting or on behalf of such organization, company or government agency.

Links


This web site contains links to other sites. Please be aware that we are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of each and every web site that collects Personal Information. This privacy statement applies solely to information collected by this web site.

Requests and Contact


Please contact us about this Privacy Notice or if you have any requests or questions relating to the privacy of your personal information.

Changes to this Privacy Notice


We may revise this Privacy Notice through an updated posting. We will identify the effective date of the revision in the posting. Often, updates are made to provide greater clarity or to comply with changes in regulatory requirements. If the updates involve material changes to the collection, protection, use or disclosure of Personal Information, Pearson will provide notice of the change through a conspicuous notice on this site or other appropriate way. Continued use of the site after the effective date of a posted revision evidences acceptance. Please contact us if you have questions or concerns about the Privacy Notice or any objection to any revisions.

Last Update: November 17, 2020