Barriers to Scrum Adoption
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Robert Asprin's novel Phule's Company [1] tells a remarkable story of two competing units crossing an obstacle course. One is an elite infantry unit, and the other is a bunch of overweight, out-of-shape civilians in a support unit. Yet it's the second group, the weekend warriors, who win the race and are the heroes of the story.
The first team runs through the course with full combat gear, and earns a respectable time. The second team actually uses the gear in their packs. Instead of going under the barbed wire, they have an advance team cut the wires. Instead of going over the wall, the advance team uses demolitions to blow it up. Once the advance team destroys the obstacles, the rest of the company can just walk on through, winning the race easily.
This is an article about opposition to Scrum, which every organization seems to experience when switching from established practices and policies. However natural Scrum may be, navigating the barriers to it can still be challenging and demanding, and sometimes a failure.
The good news is that I'm not going to write about how to navigate the obstacles to Scrum. This article is about being the second team.
Just What Is Scrum, Anyway?
The first and most common barrier I find is that Scrum is a single word; it's a symbol. What Scrum means to your engineering director when he reads an article in a trade magazine could be very different from what the marketing vice president hears when talking to a friend in a different company, and that will probably differ from what the developer envisions when he reads the latest book, or what the newly hired project manager (who came from a consulting company) thinks Scrum is.
The problem with isn't finding Scrum advice—it's choosing which advice to accept!
This conflict of advice sets up the organization for failure. In most cases, a good third of the company has no idea what's going on, while another third has the wrong idea. Unless the company does something to prevent them at this stage, all the classic dysfunctional behaviors will crop up: technical staff avoiding commitments and using weasel words; management cajoling, manipulating, and using scare tactics; protectionism of function and codebase; command-and-control thinking; and all the rest.
Now compare all that behavior to the classic definition of Scrum, published over 20 years ago in Wicked Problems, Righteous Solutions: [2]
Suppose you have a software development project to do. For each traditional phase, you can draw from a pool of experienced people. Rather than have several designers do the design phase and have several coders do the construction phase, etc., you form a team by carefully selecting one person from each pool. During a team meeting, you will tell them that they have each been carefully chosen to do a project that is very important to the company, country, organization, or whatever. This unsettles them somewhat. You then give them a description of the problem to be solved, the figures for how much it cost in time and money to do similar projects, and what the performance figures for those systems are. Then, after you have gotten them used to the idea that they are special, having been specifically chosen to do an important job, you further unsettle the team by saying that their job is to produce the system in, say, half the time and money and it must have twice the performance of other systems. Next, you say that how they do it is their business. Your business is to support them in getting resources. Then, you leave them alone.
You stand by to give them advice if you are asked. You get their reports, which come regularly but not as often nor as voluminously as the waterfall model. But, mostly you wait. In something like the appointed time, out pops the system with the performance and cost figures you want.
Sounds like a fairy tale, doesn't it?
Modern Scrum adds a few wrinkles; for example, the idea of a product owner (a single person who can decide what must be built) and the idea of iterating on the system (bringing it to production quality, if not shipping, every few weeks). We call this increment a sprint.
Even with this brief description, you can get the general idea: The project team is one team, consisting of every person needed to get the work done, working full-time, meeting daily to remove obstacles, and doing what needs to be done in order to get the product out the door. In addition, the schedule belongs to the team. They own it, they maintain it, and all commitments require the agreement of the project team.
Contrast Scrum to a command-and-control approach, and things are radically different. Instead of a committee of stakeholders, you get the one voice of a single customer; instead of a multi-year plan, the team adds value incrementally, adjusting the plan.
This change is a big deal, and you need the whole company to be on the same page about it. In addition, the big deal implies a reorganization, which threatens people. We need to approach the conversion in a way that represents the least amount of threat, the least loss of face, and the most mutual understanding.
One way to present conversion in an unthreatening way is to take a second look at Scrum training. Instead of training only Scrum Masters, consider a two- or three-day course that is "an introduction to a new way of thinking about product development." Also consider sending the entire organization to the course. Likewise, you may want to look at a more gradual Scrum adoption method, giving people a choice to decide when and how to change the way they work.
I'll cover gradual Scrum adoption later. First, let's talk about project managers and Scrum Masters.
Project Managers as Scrum Masters
Prior to a Scrum adoption, the typical IT group is organized functionally. In other words, every job has its own team, leading to an applications programming team, a data warehouse team, a team of analysts, and so on. Of course, to get anything done, you need to coordinate cross-functionally, something that line managers are not particularly equipped to do, so the organization creates a project management team.
Now think about how Scrum teams are organized: as independent, self-directed, multidisciplinary work teams. When you look at project managers in a certain way, they look like a step in that direction. It's almost as if you could cut out the line managers, take the existing ("dotted line") project teams, convert the project managers to Scrum Masters, call that the new organization, and be done.
Notice the "almost" and "as if."
The roles of traditional project manager and Scrum Master are different. Wildly different. Fundamentally different. Not only are the roles different, but they make fundamentally different assumptions about how the world works and how problems are solved.
Project managers gather requirements, gather estimates, coordinate between teams, monitor conformance to plan, and either adjust the plan or tweak reality. That isn't what Scrum Masters do at all. Wikipedia describes the Scrum Master in this way:
Scrum is facilitated by a ScrumMaster, also written as Scrum Master, who is accountable for removing impediments to the ability of the team to deliver the sprint goal/deliverables. The ScrumMaster is not the team leader but acts as a buffer between the team and any distracting influences. The ScrumMaster ensures that the Scrum process is used as intended. The ScrumMaster is the enforcer of rules. A key part of the ScrumMaster's role is to protect the team and keep them focused on the tasks at hand. The role has also been referred to as servant-leader to reinforce these dual perspectives.
Notice the subtle differences here: The Scrum Master doesn't build the plan, create any sort of chart, or serve as any sort of middle-person between the customer and the project team. Instead, all of those traditional project-management roles are assumed by the technical team itself. That approach frees the Scrum Master to enforce the shared rules of how the team will operate, removes obstacles from team performance, and actively works to protect the team from drag and impediments to progress. (In Phule's Company, the Scrum Master would be in the advance party.)
Now What Do the Managers Do?
In an entirely project-based organization, the role for line supervisors is unclear. On one hand, the organization will likely still want someone creating standards by practice (such as "tester" or "programmer"), approving vacation requests, and performing evaluations. On the other hand, there's no obvious need for such a role—all of these activities feel like part-time jobs. Since the vast majority of those tasks—if not all—can be done by the team, who needs a manager?
Suddenly, the people with the most experience, the most clout, the most personal power in the organization have their prestige—or even their jobs—threatened. But people who rise to be managers are good at playing politics. If anyone can throw sand in the works, the managers can.
My advice here matches what I hinted earlier: Don't attempt a "flip the switch" massive Scrum adoption. Instead, peel off one multidisciplinary project team at a time, and staff them with volunteers. This technique allows the people who actually want to do Scrum to do it, while those who fear Scrum can avoid it. The line managers also get time to decide what to pursue, whether that's moving into a development team, becoming a Scrum Master, transferring out of the technology group, or perhaps finding a job at a command-and-control organization that suits their temperament.
Once your organization has one high-functioning Scrum team, you might call that task "done." After all, it was a single skunkworks team that built the IBM PC (and saved the company), while the rest of Big Blue was cranking out big iron. Alternatively, you might continue slowly spooling up additional Scrum teams for new projects, leaving the old structure in place to deal with maintenance issues.
Eventually, the traditional structure will shrink to the point that they're dealing with maintenance, ongoing operations, and support work—sometimes called "MOOSE" work. By that point, the managers will probably have had some turnover, so you may be able to reorganize the MOOSE teams into a smaller department with fewer managers.
No Single Product Owner
Another of the key roles in Scrum is the product owner (PO); the "single wringable neck" who decides what to build. Under Scrum, the product owner decides what features and bug fixes will be done in what order. The PO also balances the requests of customers, sales, technical support, senior management, and all other stakeholders on the project.
The product owner doesn't deal in requirements or talk about conforming to plan. Instead, the PO simply owns the product. The PO's job is simply to steer to the best possible outcome. It's wonderful.
Yet organization after organization chooses to "go to Scrum" with no product owner defined. Perhaps the Scrum Master becomes a sort of product owner, at least in taking responsibility for mapping requests from "the business" onto cards or documents. The problem is that no one decides what to work on right now, or what can be cut. The implicit assumption is that the team must do everything—even when the director of technical support and the VP of sales disagree on what "everything" means.
If your team is going to move to Scrum, these questions are critical:
- Who decides when we ship?
- Who decides in what order the team will work on features and fixes?
- When new requirements emerge, who decides whether those new requirements should go into the next sprint?
Under Scrum, not only does the PO decide whether the work should be done, and when, but the earliest the new work can be done is in a new sprint. Beyond the occasional emergency, massive changes of work to be done within a sprint is considered against the rules.
Without a single person to answer those three critical questions, the organization is likely to be at the mercy of a committee, or a dozen conflicting voices. In order to satisfy one demand, the team has to reject 10 or 11 others, and that's not exactly a recipe for success.
It's possible to have a committee speak as one voice, if you have a process in place to answer those three critical questions. But success is much more likely with a single product owner—although selecting the product owner tends to be rife with politics and personalities. (Project managers also struggle as product owners, because they're used to representing the owners of the project, rather than being owners.)
One thing I know: Speaking as one voice about product tradeoffs is crucial to your Scrum success. Bank on it.
Dirty Systems and Long Test Cycles
In an ideal Scrum, a sprint takes from two weeks to two months. Yet many of us have worked on integrations, upgrades, and other major projects where the test cycle alone took two months. Or eight months. Or twelve. How can the team possibly compress a shippable increment into two months if the testing phase alone takes eight?
This nut can be especially hard to crack, because it contains a social challenge as well as a technical one:
- Social challenge. The social challenge lies in fighting established thinking, in moving from "test at the end" to "test as we go." Teammates likely need to rethink how they interact with each other. They'll also need to restructure the work—perhaps not shipping every sprint, but at least performing some demonstration or proof of concept.
- Technical challenge. The technical challenge is in building infrastructure, test tools, and computing environments to make this new strategy possible; perhaps by redesigning the system as components, each with a clean "seam" that can be tested in isolation.
One way to make this process less painful is to go with the gradual adoption strategy I recommended earlier, combined with selecting a project that's exclusively new development. New projects can't have long test cycles because there just isn't much code yet. Likewise, new projects give the team the opportunity to experiment with new test approaches, including "test as you go," test automation, and exploratory approaches that may be able to cover more ground in less time. It's also possible that programmers can learn new defensive coding techniques to minimize the bugs that slip out of development, thus decreasing the number of fix/retest cycles and shrinking overall test time.
Choosing the wrong first project might just mean taking six months to deliver not-much in new systems. Don't make that mistake.
Magical Thinking
The final problem I want to address here is magical thinking, a sort of childish approach to problems in which we believe that if we just hope enough, the problem will go away.
Magical thinking happens everywhere in organizations, but in this instance I mean something specific: Trying to cram ten pounds of work into a five-pound sack.
This often happens on technical teams facing an important deadline: "If we just work hard enough, somehow we'll be okay." It also happens to customers, managers, and others who believe that inserting extra requirements into a project plan or other document will magically make the team accomplish more work. Organizations that suffer from magical thinking are constantly surprised by some "unpredictable" event or external force that causes all the projects to be late.
Ironically, the fix for magical thinking is something I've already discussed—a single product owner, making decisions based on evidence, not wishful thinking.
If your organization suffers from magical thinking, take that problem into account when planning the first project. It may be hard to do, it may cause pain, but if your Scrum conversion is going to be effective, your organization has to grow up.
Epilogue: Flipping the Switch
When a command-and-control organization switches to Scrum, most people win, but often a small, powerful group stands to lose—or at least faces the fear of loss. That fear of loss—loss of control, loss of authority and power—can cause resistance and sabotage. I suggest three ways to address this fear:
- Anticipate this kind of resistance. Expect it and count it as part of the cost of the change.
- Organize the transition so that those people don't fear loss.
- Show judgment and make tough choices in the face of sabotage.
Concrete barriers to Scrum adoption are inevitable. Test cycles won't magically shrink without effort. Most of the challenges in Scrum adoption aren't technical, but social. Because Scrum makes ineffectiveness obvious and control organic, some people will fight tooth and nail to stop the effort.
Expect that response, be prepared for it, but by no means should you take those obstacles seriously. Carry some grenades in your kit, blow up the obstacles, and move on.
References
[1] Robert Asprin, Phule's Company. Ace Books: New York, NY. 1990.
[2] Peter DeGrace and Leslie Hulet Stahl, Wicked Problems, Righteous Solutions: A Catalogue of Modern Software Engineering Paradigms. Prentice Hall: Englewood Cliffs, NJ. 1990.