Building Safer Technology: A Field Guide to Failing Well

Description

Failure is inevitable. Failing well is a skill.

The era of "move fast and break things" is over. In a world where software and AI are woven into everything from power grids to the devices in our homes, failure is no longer a minor inconvenienceit's a critical business risk that can cost lives, paralyze economies, and erode trust. Building Safer Technology delivers a practical field guide to safety engineering, a practice dedicated to designing systems that are not only innovative but also fundamentally resilient.

Go beyond theory and learn to implement safety practices directly into your workflows and organizational structure. This book provides concrete tools, mental models, and communication strategies for individuals and team leaders to build products that can withstand real-world chaos. You'll learn how to prevent failures and also how to manage them gracefully when they inevitably occur, preserving trust and turning crises into powerful learning opportunities.

You will learn how to:

• Learn to apply the Safety Engineering Methodology, a four-step process to understand, analyze, and mitigate risk in any sociotechnical system.
• Identify and analyze potential failure modes in any product, from AI models to infrastructure, using real-world case studies like the Boeing 737 MAX and Zillows home-pricing algorithm.
• Develop robust mitigation plans and resiliency strategies that transform potential disasters into manageable, predictable events.
• Establish effective incident response processes that protect users, contain harm, and strengthen your systems after a crisis.
• Integrate safety into your organization's culture with proven techniques for measurement, accountability, and persuasive communication.
• Utilize ready-to-use templates for resiliency plans, risk strategies, and checklists for brainstorming failures.

Stop reacting to disasters and start building systems that are designed to fail well.

Sample Content

Sample Pages

Download the sample pages (includes Chapter 1)

Table of Contents

Ch. 1 The Rationale, Principles, and Methodology of Safety Engineering.. . . . . 1

The Four Principles of Safety Engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

From Principles to Practice: Safety Engineering Methodology. . . . . . . . . . . . . . . 13

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Ch. 2 Applied Examples.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Example 1: AI-Assisted Bank Loans.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Example 2: VioGen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Example 3: Social Media Comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Ch. 3 Understanding Failure Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Common Ways Technology Fails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

From Failure to Impact.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

How Failures Happen.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Understanding the Impacts of Failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Common Types of Harm.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Understand the Risks.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Ch. 4 Responding to Failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Methods of Mitigation.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Repeated Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Summary: Choosing a Mitigation Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Ch. 5 Managing Incidents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

The Role of Response Within Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

Preparing to Act.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Building the Foundation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Your Response Team.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Getting Back to Normal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Always Prepared: Next Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Ch. 6 Communicating Effectively.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Building Collaboration.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Choosing Priorities.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

Communicating Plans Effectively.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Persuasive Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Ch. 7 Putting Safety into Practice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Measurement.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Getting People to Do Safety Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

Checking Everyone's Work.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

Ch. 8 Organizations and Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Safety Team Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

Organizational Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

Hiring and Onboarding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

Performance Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Culture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

Leading a Safety Organization.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

Ch. 9 Safety Engineering Illustrated: Additional Worked Examples. . . . . . . . . 201

Zillow's AI Home-Pricing Failure.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

Data Deletion Systems.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

Wimbledon's Automated Line-Calling Glitch.. . . . . . . . . . . . . . . . . . . . . . . . . . . 211

The COMPAS Algorithm.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

Boeing 737 MAX Groundings.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

Blackout Cascade: Puerto Rico's Power Distribution System. . . . . . . . . . . . . . . . 227

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236

Ch. 10 Conclusion: This Book Is an Example of Itself. . . . . . . . . . . . . . . . . . . . . . . 239

Mitigating the Risk: Safety Engineering.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245

A. Resiliency Plan Template.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

B. Risk Strategy Template. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

C. Checklists for Brainstorming Failures.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

D. Sample Resiliency Plan: AI-Assisted Bank Loans. . . . . . . . . . . . . . . . . . . . 275

Index.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Updates

Submit Errata