This chapter is from the book
This chapter is from the book
This chapter is from the book
Item 12: Set Up a Tinderbox
Tinderbox is a tool that was developed initially at Netscape, but that is now open source software maintained by the Mozilla project. Tinderbox is designed to manage the complexity one encounters when developing software, especially in terms of large-scale cross-platform software that involves a widely distributed team of developers. Tinderbox is particularly useful in cross-platform projects, as you will see. Coupled with a system known as bonsai, the goals of Tinderbox are fairly simple:
- Communicate any and all changes made over time to the source code repository to the entire development team, in a centralized location, as soon as the changes have been made.
- Communicate the overall health of the repository by continually pulling and building the source code on each supported platform. For each pull and build cycle, a pass/fail status is reported to a centralized location. This allows developers to determine when they should update their local trees to avoid pulling source code that will not build (or run) correctly.
- Combining the above, Tinderbox can be used to assign accountability of the health of the tree to specific individuals and/or changes to the repository. Knowing this information helps get problems solved as quickly and accurately as possible.
Basically, Tinderbox is a group of machines that continually pull and build the contents of a CVS repository (see Item 13), and a server that retrieves and reports the status of these builds on a Web server that everyone in the organization can monitor. Tinderbox is currently supported as three versions. Version 1, perhaps the most widely used, was developed by Netscape/Mozilla, and is still in use by mozilla.org. Tinderbox 2.0 is a rewrite of version 1, providing essentially the same feature set. The goal of Tinderbox 2.0 was to essentially clean up the implementation of version 1. Both Tinderbox 1 and Tinderbox 2 are available from mozilla.org. Tinderbox 3 is a more recent version, available as a tarball from John Keiser, an ex-Netscape developer. Tinderbox 3 adds a number of desirable features, and strives to make Tinderbox easier to set up and administer.
Figure 3-7 illustrates the Web page displayed by a Tinderbox server. (You can access a large number of live Tinderboxen by visiting http://Tinderbox.mozilla.org.)
)
Figure 3-7 Tinderbox
The Tinderbox in Figure 3-7 illustrates the state of Mozilla's Seamonkey reporting the health of some of the "port" platforms that are defined by the Mozilla project. (Seamonkey was the code name used by Netscape/Mozilla during the development of the Mozilla browser suite. A port is a platform that is not considered to be tier-1 by Mozilla.)
The use of Tinderbox is pervasive in the software development community. Not only is it used by mozilla.org, but by other open source projects (for example, OSDL) and in commercial development (AOL, for example). Tinderbox is particularly well-suited to cross-platform development, as we discuss later.
The Tinderbox Web page consists of a table viewed as a series of mutually exclusive columns that are organized from left to right. In the first column (Build Time), each row contains a timestamp that can be used to identify the time associated with events that are represented in the remaining columns of the table. The second column identifies each check-in made by developers; the time of these check-ins can easily be determined by looking at the corresponding row in the Build Time column. The remaining columns each represent a specific platform that is being reported on by the Tinderbox. (An organization may have several Tinderboxen, each reporting a specific group of builds. You can see an example of this by visiting http://tinderbox.mozilla.org/showbuilds.cgi.)
Any given column represents a build machine, and a platform, and contains a series of colored boxes. Green boxes indicate a successful build of the repository for that platform on that machine. Conversely, a red box indicates a failed build, and a yellow box indicates a build that is currently in the process of being produced. Furthermore, the lower edge of any of these boxes represents the start of a pull and build cycle, and the upper edge represents the time of completion. The time corresponding to both of these events, for a particular colored box, can be determined by looking at the timestamp at the same row in column one as the upper or lower edge of that box. For example, the uppermost failed build in Figure 3-7 (Linux balsa Dep (static), represented by column four of the table) was started at about 13:07, and failed about five minutes later, at about 13:12.
Let's take a closer look at this failed build, and see what we can infer about it. It is clear that the check-in by sspitzer at 13:09:41 did not result in the failure of the build for two reasons. First of all, the Linux balsa Dep build was already burning prior to sspitzer's check-in. (See the red box in the same column that completed around 12:00, and also notice how the lower edge of the uppermost red box is lower than the entry for sspitzer's check-in.) Another piece of evidence that sspitzer is not the cause of the problem is that each of the other Linux platform builds are green. (Generally, one finds that Linux builds of Mozilla are generally are either all red at the same time, or all green.) Finally, and perhaps most important, we can see that the build was previously red at noon (12:00), and had not gone through a green cycle since then. (Gray portions of the Tinderbox indicate no build was in progress, or the progress of a build was not reported to the Tinderbox server.)
For largely the same reasons cited previously for sspitzer, we can also infer that mozilla.mano is not to blame for the redness of the build, either. The tree was already red prior to his or her check-in, and the other Linux builds were not affected.
Let's say I am not entirely sure that sspitzer is not to blame, and want to take a closer look at what the exact cause of the broken build might be. There are several other facilities provided by Tinderbox that you can use to drill down for further information. The L (or L1) link inside of the red box can be used to obtain a log of the build; this log contains compiler and perhaps linker output that should identify what caused the build to break. Clicking the L1 link gives the result shown in Figure 3-8.
)
Figure 3-8 L1 link output
Clicking View Brief Log results in Figure 3-9, which indicates a problem building libmozjs.so, the shared library that contains the implementation of the Mozilla Spidermonkey JavaScript engine (which is used in Trixul; see Chapter 9, "Developing a Cross-Platform GUI Toolkit in C++").
)
Figure 3-9 View brief log output
The other major source of input that I would want to consider is an understanding of what portion of the tree was impacted by sspitzer. If the check-in he made does not correlate to the error messages displayed in the log (Figure 3-9), I can eliminate him from the "blame" list and focus my search elsewhere. I can do this in two ways. First, by clicking the C link in the red box; this will display a list of checkins that were made prior to the start of the build. It is these checkins that likely would be the cause of any state change in the build (that is, going from red to green, or from green to red, which is not the case here since the tree was already red at the 12:00 hour). Figure 3-10 shows the result of clicking the C link.
)
Figure 3-10 List of checkins
This result clearly confirms that sspitzer is not on the blame list—and it also shows that the problem is not related to a check-in by mozilla.mano, because this change was made to a Cascading Style Sheets (CSS) source file that would have no impact on the JavaScript engine.
The final technique to eliminate sspitzer from the blame list (short of sending him an e-mail and asking him whether his check-in caused the build failure) would be to click the sspitzer link in column one and see what changes he made. Doing so gives us Figure 3-11.
)
Figure 3-11 List of checkins made by sspitzer
Once again, these are changes to Extensible Markup Language (XML) markup and JavaScript sources that have no bearing whatsoever on the stability of the JavaScript engine, the failure of which was clearly the cause of the red tree.
In terms of cross-platform development, it is best not to rely on a Tinderbox to find all of your portability problems. At Netscape, developers were required to ensure that any changes they were considering for checkin built and executed cleanly on each tier-1 platform, before submitting the changes to the repository (see Item 4). However, you can never be sure that all developers will follow this rule all the time, without exception. Because of this, having a Tinderbox monitor the health of the repository is an especially good idea. It isolates the problems in terms of change made, time of the change, and developer who made the change, focusing the area of investigation to a small fraction of the overall possibilities.
At Netscape, Tinderbox was a way of life (as it remains for Mozilla and many other development projects). The state of the tree was closely monitored, and acted as the focal point of development. If the tree was red, you could not checkin to the tree until it turned green. After you checked in your changes, you were added to a group called "the hook," and as a member of the hook, you were required to watch the Tinderbox and ensure that the your changes built cleanly (that is, were green) for each platform affected. (Obviously, if the change was only made to, say, Mac-specific code, you were only obligated to see the Mac builds go green.) If the tree affected by your changes was green when you checked in, and then it went red in the first build that followed, then, as a member of the hook, you were required to help identify and fix the problem.
In addition to the hook, a sheriff was assigned by Netscape (or Mozilla) to monitor the overall state of the Tinderbox monitoring the tier-1 platforms, and to ensure that the builds all remained green. We eventually rotated the responsibility of sheriffing inside of Netscape among the various development teams, one day the responsibility would fall to members of the mail-news team, on another, it was the responsibility of the IM team, and so on. Should a problem arise, the sheriff had the power to close the tree to all check-ins (this was communicated by a simple message at the top of the Tinderbox page), which was done in an attempt to aid those trying to isolate the cause of problems. The sheriff also had the authority to contact anyone on the hook, usually by e-mail, but by phone if necessary, should a diagnosis of the problem indicate that the person being contacted was to blame for the tree going red. As sheriff, I recall numerous times calling people by phone who had left the tree burning and had gone home for the night. Not everyone was happy about the policy, but it did cause people to be more careful about their work.
In a nutshell, Tinderbox plays an important role in cross-platform development because it forces developers to confront issues that affect the portability of code being committed to the repository. Although a responsible developer will try to determine the impact of changes on other platforms before landing code in the tree, this is not always done. Tinderbox acts as a backstop, ensuring that nothing slips through the cracks. And when problems are detected, either because of a red tree or a failed QA test the next day, Tinderbox can be used as an aid in determining which changes had been made that might be the cause of the problem.
Getting a green build on all the platforms is, of course, not the end of the story. A green build does not ensure that cross-platform feature parity is met for the platform, for example. A developer could implement a cross-platform feature and check in a full implementation for, say, Mac OS X and only stub implementations for Linux and Windows, and Tinderbox will not make this fact evident. What Tinderbox does do a good job of is ensuring that code shared among platforms builds everywhere, and that no one platform can hijack the stability of the tree (that is, leave the repository in a state such that it builds cleanly for only a subset of the supported platforms). As such, Tinderbox helps ensure that when platforms are worked on at the same time, the cross-platform train is moving ahead at the same pace for all platforms involved. This in turn helps ensure that the organization will be able to release a product to the market for all the supported platforms at about the same point in time, which is something that we strived for at Netscape. Remember, however, that Tinderbox is just an aid—only by testing the resulting builds carefully can you confirm that a product or feature has achieved cross-platform parity.