Lesson 3: Fault Tolerance Deserves Investment
Like everyone else, NASCAR teams are vulnerable to hard disk failures, power outages, electrical surges, and associated system failures. Maybe even more so. Smoke, vibration from 750-horsepower motors, heat from unimpeded sunlight, rain, and shocks from drops and falls all contribute to the demanding environment in which systems are put to the test more than 35 weeks a year.
Redundancy planning, while never an enjoyable task, is viewed as at least a necessary chore by most NASCAR teams. While Fuller says his Roush Racing team doesn’t have an official fault-tolerance plan in place, he notes that the operation utilizes UPS devices in the team transport and pit lane, and carries multiple laptops to each event. "If one goes down, we steal the main one from the truck."
Most teams operate three or four Windows XP–powered PCs per car fielded. Data is typically stored on each system, so unlike a server that must be backed up (because a sole server constitutes a single point of failure), each workstation essentially serves as a built-in backup.
In addition to carting backup race cars to the track, many teams also carry backup cellular Internet PC cards, backup radios, spare batteries, and extra Pi Research modules. Further, almost every electronic device is plugged into a surge-protecting uninterruptible power supply.
NASCAR, the official sanctioning body for the race series, sends backup laptops to and operates redundant servers at each race. The official scoring system, which relies on a series of wireless broadcasting units placed around the track, is connected to gas-powered generators. A separate fiber optic network serves as a backup.
NASCAR officials are also trained in resorting to manual systems, should electrical power fail. Even Goodyear, which is responsible for distributing and collecting all race tires distributed at the track, boasts a manual system as backup for its electronic RFID program, should computer or networking failures occur.
Most race teams, not to be outdone, possess low-tech backup plans that include the most rudimentary of alternatives: paper-based systems. Engineers’ scientific calculators appear in abundance. If all else fails, these techies have no trouble resorting to old-school methods.
"I always bring my trusty three-ring notebooks," admits Clay Andrews Racings’ Wilburn. "With handwritten notes we at least have a hard copy. We rely upon computers and their efficiency, but we still have hard copy."
He’s not alone.
Brewco’s Esterline adds, "The computer’s a tool to make things easier. Sometimes people get in trouble placing too much trust in computers, though. You can’t just sit back and let computers do it all; you still need to rely upon personal experience and knowledge."
At the end of the day, all event data is returned to the team shop, where it’s subsequently copied to the organization’s server, which itself is backed up every day. Information gained this week is later put to use at similar tracks next month or at the same venue later this season or again next year.
As with sharing data effectively and properly securing sensitive data, businesses should work to emulate several common NASCAR fault-tolerance strategies:
- A system running critical applications or storing important data should have a backup system preconfigured and ready for immediate deployment in the event that the primary system fails.
- All critical systems should be connected to surge-protecting uninterruptible power supplies. While a seeming no-brainer, this simple preventive measure is often overlooked, particularly in small businesses.
- Achieving fault tolerance doesn’t require that funds be invested in a mirror system that’s unused until a problem arises; active systems can store one another’s data and provide online redundancy.
- Organizations should strive to understand their most fundamental business activities and implement paper-based backup systems where possible. Technologies should serve to add efficiencies where appropriate—not serve as the sole method of fulfilling a task or activity.