The Challenge of SRE Transformation

This chapter is from the book

Establishing SRE Foundations: A Step-by-Step Guide to Introducing Site Reliability Engineering in Software Delivery Organizations

Learn More Buy

This chapter is from the book

This chapter is from the book 

Establishing SRE Foundations: A Step-by-Step Guide to Introducing Site Reliability Engineering in Software Delivery Organizations

Learn More Buy

2.3 Ownership Using SRE

What does it mean to have partial ownership of production operations using SRE? This question needs to be answered specifically for each party in the product delivery organization.

2.3.1 Product Development

In product development, the benefits of partially owning production operations are rooted in the insights of how the system behaves in production under real user, data, and infrastructure load. The most effective way to continuously learn about a system in production is to observe it in production. This is done using on-call rotations. Traditionally, product operations would go on call for services in production. This way, production insights do not go directly to product development. It follows that product development needs to get involved in on-call rotations for their services in production. Each development team owns some services. For exactly those services, the respective developers would need to be involved in on-call rotations to gain insights from operating the services under real conditions. These insights lead to the following improvements in product development and operations.

• Developers with product implementation knowledge conduct product failure investigations.

• The number of steps in the chain between a production issue occurring and a person with the best knowledge to fix it can be exactly one. The issue can go directly to the developer who implemented the service and can fix it the fastest, provided the alerting is targeted well and there is an agreement with the product owner to fix production issues immediately. The developer can take the learnings from the failure itself, the failure analysis, and the fix into the new-feature development process, supporting infrastructure and debugging tools. This should lead to the product being more operable in the future with less time required to operate it. In turn, it should lead to more time for feature development.

• Developers get to experience the quality of the product in the real world by testing it at production sites. Internal testing is rarely as intensive as the strain a system undergoes in production. Seeing real-world scenarios informs the development of automated test suites and contributes greatly to closing the gap between the internal testing and production scenarios. Thus, confidence is increased in deploying the product to production once the test results from the internal test suites are green. This should lead to fewer failures in production due to scenarios that were untested internally. In turn, it should require less time to fix production issues, leading to more time for feature development.

• Developers gain the knowledge necessary to operate and troubleshoot the product. This informs the development process, among other things, leading to better tools for operations. In turn, it leads to less time spent on troubleshooting production issues, which frees up more time for feature development.

• Developers use the knowledge from product operations in the development of new features. For example, scalability and performance requirements can be learned, and this can often lead to architecture changes. Although making such changes requires a significant amount of work, it is necessary to implement resilience in accordance with the load profiles seen in production. Only then can the system’s operational burden (also known as toil) reduce, freeing time for more feature development.

• Developers gain a better understanding of the kind of testing and tooling necessary to deliver a product that works well. Test scenarios, test levels, test runs, and test environments need to be designed in such a way that all the testing activities combined address important scenarios the system encounters in production. To be sure, production itself can be one of the test environments with tests running there 24/7. Gaining insight into product operations in production can greatly inform the entire test management process. This should lead to test suites and test runs being more focused on the scenarios taking place in production, which can reduce the amount of time spent on tests that are not effective in catching bugs encountered in production, as well as the rework and maintenance of such tests. Streamlining test management may lead to more time available for feature development.

• Developers have the incentive to implement reliability features and tools for a great product operations experience. This is because if the developers go on call, they actually want to spend as little time as possible dealing with production issues. In this context, they have full control of the situation. It is in their power to implement the product with production operations in mind. Doing this leads to spending as little time as possible on production issues and maximizing the time spent on feature development. This benefits customers and product management alike. Customers also do not want to deal with product failures in production. Rather, they want existing features to work in production and new features added to the product quickly. The product management team, driven by customer requests, wants product development to work on new features.

• Developers with experience in product operations are more highly valued in the industry. Going on call directly contributes to learning the skills necessary to command higher wages in the marketplace.

The idea of going on call for the developers gives rise to a plethora of questions, such as the following.

• Do the developers always need to go on call for their services? No.

• Could the developers go on call only during business hours? Yes.

• Can the on-call responsibility be shared with product operations? Yes.

• What is the best setup for a given organization? It depends.

• Would a development team setup need to be adapted to enable on call? Yes.

• Can developers in a team perform the on-call duties on rotation? Yes.

• Can focused feature development still be done despite going on call? Yes.

• How do you achieve it? It depends.

• Can developers stay developers if they go on call? Yes. They will become better developers. Their skills will be more highly valued in the job market.

These and other related questions will be explored in the book in due course. It is not necessary to answer them in-depth here. For now, I will only outline the scope of the challenge posed by the SRE transformation. What is important to understand at this point is that product development needs to go on call to one extent or another depending on the organizational setup chosen for the organization’s SRE implementation.

Without developers going on call to some extent, the benefits of collective ownership of production operations cannot be realized by product development. Feature development is difficult to improve from an operational standpoint without a live feedback loop between the production and development teams. An outage profile in production cannot be sustainably influenced if the feature development team is not well informed using the live feedback loop from production experienced by those who implement the features. In other words, without developers going on call to some extent, things in product development remain the same by and large as far as operational concerns go.

This key insight is illustrated on the left-hand side of Figure 2.2.

Figure 2.2 Collective ownership of product operations using SRE

2.3.2 Product Operations

The product operations discipline is depicted on the right-hand side of Figure 2.2. With developers going on call, the product operations team would need to provide support to enable the developers to do operations.

What kind of support would the developers need? They may never have done operations before, in which case this will be unfamiliar turf for them. Are there trainings for this? Does the operations team provide some onboarding? What does “good” look like in product operations? Is there any documentation available? These are the questions that come to mind for developers when confronted with going on call for the first time.

The entire body of knowledge about product operations is with the operations team. But what kind of knowledge is that? Mostly it is about taking the product as a black box, putting it into the production environment, activating monitoring of IT resources, and alerting on some threshold violations. Developers can learn and understand this. With their insider knowledge of the product, they will also be able to find many more scenarios that can be monitored and alerted upon. The developers’ knowledge about the architecture, implementation, configuration, and deployment of the product is an invaluable resource for improving monitoring of the product in production. But how can they utilize that knowledge to improve product operations? How can they bridge the gap between development and operations as suggested by the term DevOps?

Let us look more closely at what the developers know. They know how specific routines that contribute to the fulfillment of user requests are implemented. They know the paths the user requests go all the way from the user interface to the deepest service in the service network, and from there to the infrastructure. If the product exposes APIs to customers, the developers also know the paths the API requests take from the API gateway through the network of services all the way down to the infrastructure. Moreover, the developers know which services they implemented versus those implemented by the company and third parties. They know which third-party services are difficult to integrate with, where the domain model of the third-party services is overly complicated and cluttered, where the third-party services are slow occasionally, and where there is simply sporadic behavior that can be explained. The developers also know all this for the internal services of the company, which are the services they depend upon.

Their knowledge does not stop there. The developers and architects know the strengths and weaknesses of their architecture. They know where the architecture limitations lead to performance and scalability issues. They know the circumstances where the performance and scalability issues are likely to exhibit and probably impact the customer. They know the architectural debt in the system and which part of it is planned to be paid off in the near future. They know of any major architectural refactorings that must take place, which are not planned due to the size of the effort involved.

The developers’ knowledge goes much further. They know about the infrastructure limitations the product is running on. They know how each service can impact the others; for example, they know what will happen if a particular service in the service network eats up the lion’s share of memory in a given area of the infrastructure. They might know some parameters of the container clusters the services are running in and anticipate issues that might occur based on the changing data and user load profiles.

There is yet more to the developers’ knowledge. They may know the way the services are deployed: Which infrastructure parameters are set by the deployment infrastructure, and which ones are set in the service at the deployment time, startup time, or runtime. They know which services are deployed independently, which ones use a shared deployment pipeline, and which ones are deployed manually for the time being. They may know the tests running on the deployment pipelines, the quality of those tests, and whether the test results can be trusted. They may know the test management process for a service, the test levels available, the test infrastructure, and any test gaps that exist.

Additionally, the developers might be aware of security implications in the architecture and implementation. Which security vulnerabilities are taken care of? Which are mitigated? Which are known but are not currently taken care of ? Which bugs from penetration testing were not yet fixed?

Finally, the developers know the most painful product areas from a development point of view. What area is the most difficult to integrate with? To test? To speed up? To debug?

This amount of knowledge is staggering to the operations engineers. How do they take all this knowledge from the developers and apply it to product operations? Can it be done with some tool support? What kind of role would automation play here? Does it all sit between the ears of the software developers and cannot be easily repurposed to improve product operations? How can it support the developers effectively?

In other words, the developers know the car engine from the inside. But how do you help them use that knowledge to improve how the car operates?

To approach these questions, we need to turn our attention to how developers make known to the outside world what is going on with the system on the inside. This is done using logging. During development, developers decide what to log and under what circumstances. This way, once the product runs in production, log entries are generated that contain logging information. The log entries stored in, for example, log files or other storage systems can then be analyzed to understand what was going on in the system at runtime. This is the basic process of how developers make known to the outside world what is going on inside a system at runtime. The process is sophisticatedly supported by tools providing all sorts of runtime instrumentation out of the box. That is, the developers’ knowledge about the product can be encoded in logs that can be analyzed outside the system.

The next question to ask is what should be logged to improve product operations? Let us imagine, these questions would be answered.

Once that question is answered, we would consider how to log relevant information in a uniform way. What should be the log format? Which log format would lend itself to automated log processing? Would several log formats be required for different operational aspects; for example, one log format for calculating service availability and another for calculating service latency? What about asynchronous operations—how do you log those? Where do you store the logs? Should the logs be stored in regional data centers or centrally? How long should the logs be stored? Let us imagine, also these questions would be answered.

With the answers to these questions, we would next consider how to detect abnormal situations. What should be considered broken availability? What should be considered broken latency? What should be considered insufficient throughput? Which aspects beyond availability, latency, and throughput are important to consider? Let us imagine, these questions would be answered too.

Next, we would want to know how to alert in abnormal situations. Should alerts be generated as soon as the abnormal situation has been detected, or a bit later? Should the alerts be sampled? How do you avoid alert fatigue, in which those who receive the alerts become overwhelmed with too many alerts and stop reacting to them? How do you strike a good balance between alerting people so often that it causes alert fatigue and so rarely that it causes incidents to go unnoticed? What kind of information needs to be included in the alert? Even if these questions would also be answered, there would be more.

The next questions would be about whom to alert—specifically, which developers receive the alert? How do you alert developers in such a way that they do not get distracted from their feature development work? How do you alert developers in such a way that they will actually react to the alerts, provided the alerting does not lead to alert fatigue? Can any developer in general be alerted? What kind of knowledge would a developer need to have to be able to react to alerts within a reasonable time frame and with reasonable effort?

The list of questions can go on. What it shows is that a comprehensive framework that would enable developers to conduct product operations is required. But what is a framework? According to Wikipedia, “a software framework is an abstraction in which software providing generic functionality can be selectively changed by additional user-written code, thus providing application-specific software.”² So, what is needed in the context of an operational framework is some generic functionality that can selectively be changed. In the context of SRE, a framework like that can be referred to as SRE infrastructure. It needs to provide generic functionality supporting the use cases exemplarily outlined previously, implemented within an SRE context. The generic functionality needs to be selectively changeable to adapt the infrastructure to a specific use within the overall set of SRE activities.

At the time of this writing, some off-the-shelf tool support for the SRE infrastructure exists, but it is not comprehensive enough to eliminate the need for custom development of missing pieces. Therefore, in all likelihood, building an SRE infrastructure is going to require some custom software development combined with ready-to-use off-the-shelf tools. This means product operations would need to learn to do software development.

The challenge for product operations is lack of experience providing frameworks that enable others to do operations work. The product operations has always conducted operations work in a hands-on manner using existing tools. What is required from product operations now is the enablement of product development to perform service operations. The enablement is done using the envisioned SRE infrastructure. The SRE infrastructure needs to be built using first-class software development techniques.

This is in line with SRE and the words of Benjamin Treynor Sloss: “SRE is what happens when you ask a software engineer to design an operations team.” Following this, it should be no surprise that enabling the product development team to do operations work requires the software development team to build a suitable SRE infrastructure. Building frameworks is common in software development. Using frameworks is familiar to software developers. Neither of these will be familiar turf for operations engineers from the product operations discipline.

The following now unfolds as a challenge in SRE transformation.

• Software developers need to learn how to do product operations work by going on call.

• Operations engineers need to learn how to enable software developers to do operations work by developing the SRE infrastructure as a framework.

This is illustrated in Figure 2.3. The two arrows resemble the moves from fencing. It might sound ironic, but this is exactly what needs to happen during SRE transformation.

Figure 2.3 Key SRE transformation challenge

Neither of those arrows is easy to achieve. However, as evidenced by the growing number of software delivery organizations around the world, it is entirely possible, and will be explored at length in this book.

Figure 2.3 shows what it truly means and takes to implement DevOps. It is about developers doing operations work and operations engineers doing development work. It goes to the heart of both long-standing disciplines, product development and product operations, and shakes their fundamental responsibilities. Truly implementing DevOps takes far more than just achieving good collaboration between product development and product operations.

The difficulties are especially great in traditional software delivery organizations. A development organization that has never done operations and an operations organization that has never enabled others to do operations lack the very foundations on which SRE can be established. Developers do not understand why they should be doing operations. Operations engineers do not provide frameworks to enable developers to do operations. Managers do not promote the endeavor, let alone fund it.

Despite these difficulties, it is well worth the effort to embark on an SRE transformation. The kind of DevOps implementation that can be achieved using SRE is where developers maximize their feature development time while having evidence that the product works well for customers in production. Without SRE, developers maximize their feature development time, ignoring production.

Further, in a DevOps implementation using SRE, operations engineers scale well by providing the SRE infrastructure to the developers, which enables them to do production operations. Without SRE, the operations engineers are the bottlenecks. They do production operations purely by themselves, regardless of the product quality and insider knowledge available about the product.

2.3.3 Product Management

Having clarified what collective ownership of production operations means for product development and product operations, it is time for such a clarification in the context of product management. What does the product management team need to do to partially own product operations?

Traditionally, product management is pretty far away from product operations. As discussed in Section 2.2, Collective Ownership, product management’s benefit in getting involved in production operations is to reduce customer escalations. How on earth can product management reduce customer escalations if everything the customers escalate about is a technically broken product? Product owners are not technical experts. They neither implemented nor deployed the product.

To approach this, let us explore what leads to customer escalations. Before a customer gets to the point of picking up the phone and calling customer support to complain, a series of events take place. The customer works with the product and notices something annoying. It might be a sluggish display of data; an inconvenient way of accomplishing a task in too many back- and-forth steps; an action taken, like a button click, that does not result in the action actually performed; or a downright crash with accompanying data loss. Whatever the reason, it is directly linked to the customer having lost so much time or money that they call customer support to release their anger and get help.

Now, could the technical experts—namely, the product development or product operations team—have noticed anything wrong with the product and fixed it earlier? Are product development and product operations set up for such incident detection and resolution? Again, this is technical, so what does it have to do with product management?

Let us dive deeper. Imagine that product development and product operations want to set up incident detection and resolution to detect and fix abnormal situations before customers escalate. How would they go about doing this?

As you saw in Section 2.3.1, Product Development, the developers have an enormous amount of knowledge about all sorts of technical aspects regarding the product. The operations engineers have vast experience with customer escalations. They remember a lot of past escalations by heart. They can cluster them. They know by means of anticipation the weak areas of the product that are going to be escalated about soon because product development has not started fixing them. Overall, this is a good mix of knowledge that is brought to the table by product development and product operations. The product development team brings knowledge of technical implementation while the product operations team brings knowledge of the actual issues from production. Taken together, this knowledge can be used to create an incident detection and resolution process rooted in technical implementation and past customer escalations. This is great. It would be a huge leap from ad hoc, unsystematic incident response. It would reduce customer escalations.

The goal, however, is to aim higher. The goal is to create an incident response and resolution process that for every existing and new feature would detect abnormal situations early enough for product development to fix and then to deploy the fixes before customers escalate. This would be a real benefit to product management. To emphasize, the process should work for every existing and new feature, not just for features known to product operations based on the experience of past customer escalations. Also, to emphasize, the developers would allocate their time in such a way that they fix detected issues and deploy the fixes to production before the customers get angry enough to escalate. This means the developers would not just work on the feature backlog prioritized by the product owners. The other prioritization driver would be the product reliability issues detected by the incident response process.

With that, the contribution of product management to the collective ownership of product operations is starting to emerge.

1. Product owners would need to contribute user journey knowledge to the incident detection process. Impaired and broken user journeys should be at the core of incident detection. Which user journeys are the most important ones to detect incidents with? What are the most important steps within a given user journey that must work for the user journey to still make sense? Conversely, which steps of a user journey could fail, and how badly, without rendering the entire user journey broken? Overall, the incident detection process is as good as the defined incidents it can detect. To define detectable incidents well, the user journey knowledge of the product owners, the implementation knowledge of the developers, and the operations knowledge of the operations engineers need to be combined.

2. Product owners would need to understand and agree to the importance of setting up a backlog management procedure in which developers can flexibly allocate time to fix production issues as they are detected by the incident detection process. Traditionally, the product owners prioritize the backlog of user stories, and they want developers to focus on the backlog. To reduce customer escalations, the product owners would want the developers to take immediate action on the issues reported by incident detection.

This now makes sense to the product owners. They were part of and shaped the incident detection definition. They know what the incident detection is going to detect. It is going to detect real broken user journeys and not merely some technical deviations. Now it is easier for the product owners to accept the engineering time being spent on incident resolution. Why? Because spending that time directly contributes to the reduction of customer escalations. If developers do not fix the incidents in production within a reasonable time frame, the customers will still escalate despite the right incidents being detected early enough.

That is, to reduce customer escalations, the following criteria need to be fulfilled.

• The incident detection detects broken and impaired user journeys as defined together by the operations engineers, developers, and product owners.

• The developers prioritize fixing broken and impaired user journeys as they are detected without having to negotiate with product owners every time about the engineering time allocation.

• The developers fix the broken and impaired user journeys in production within a specified time frame before customers get angry and frustrated enough to escalate.

This process is shown in Figure 2.4.

Figure 2.4 Process for reducing customer escalations

The top left of Figure 2.4 shows the incident detection definition process. It takes as input the implementation knowledge by developers, the operations knowledge by operations engineers, and the user journey knowledge by product owners. The outcome of the incident detection definition process is an understanding of the incidents to detect in production. It is about detection of unhealthy patterns in

• The user journeys from an operational criticality perspective

• The critical service dependencies fulfilling the user journeys

• The critical infrastructure components and their scaling, fulfilling the user journeys

With the definition of what to look for in production to detect incidents, incident detection can be switched on. The production environment in the middle of Figure 2.4 can now run under monitoring for fulfillment of real user journeys, instead of having monitoring for fulfillment of technical parameters. Monitoring for fulfillment of real user journeys is more targeted to reduce customer escalations, which is product management’s benefit in terms of getting involved in product operations.

Once the incident detection process has detected some incidents, it will put them into a backlog. This is shown on the right-hand side of Figure 2.4. The incident backlog exists side by side with the user story backlog. The user story backlog is prioritized by the product owner. The incident backlog also needs prioritization. This prioritization needs to be done as a just-in-time process while the incidents get detected. It also needs to be done very quickly. Lengthy negotiations between the operations engineers, developers, and product owners cannot take place to efficiently prioritize the incident backlog. This means prior agreements among the three parties need to be reached. A good place for timely incident prioritization agreements is the incident detection definition process itself. As part of the process, not only are the incidents defined but also their relative priority can be agreed. These agreements should enable all the people on call, and especially the developers, to make autonomous incident prioritization decisions for a majority of the incidents.

Because the incidents from the incident backlog need to be worked on in a just-in-time manner, the developers working on the incidents cannot work on the user stories from the user story backlog at the same time. Also, just-in-time switching between the incident backlog and the user story backlog leads to a great context switching overhead. Not only is this inefficient, it also places a significant mental burden on the developers. To counter this, there are strategies to setting up development teams in such a way that the ongoing on-call work from the incident backlog and the focused user story work from the user story backlog are well balanced. These strategies will be explored later.

The bottom of Figure 2.4 shows the incident processing timeline. It begins with an incident, shown on the far left. At the time of the incident, the potential for customer escalation is very low. It grows to 100% over time, which is the point to avoid. The goal is to fix the incident before the customer gets angry and frustrated enough to call customer support with an escalation.

After the incident started, it can be detected by incident detection. The next step on the timeline is the point in time when the developer starts working on the fix. Once the issue has been fixed, it needs to be deployed to production. Once it is deployed, the fix needs to be monitored to ensure that the incident has truly been resolved. The goal is to perform the fix deployment and associated monitoring, confirming the incident resolution before the red line of customer escalation.

The incidents that have started may go unnoticed, or be noticed too late or too early by incident detection. Conversely, the incidents may represent false positives. This happens when an incident is reported that does not lead to deterioration in the user experience. All these cases need to serve as input for adjusting the incident detection definition. This is an important part of the overall process. It enables the incident definition adjustments to be done regularly based on the real feedback loop from production. The feedback loop is data driven. This enables the three parties—product operations, product development, and product management—to decide on the incident definitions in an opinion-neutral, data-driven way.

In the context of SRE, such an incident detection and response process is set up using specific mechanisms and terms, such as service level indicators (SLIs), service level objectives (SLOs), and error budget policies. Soon, an exploration of these concepts will begin. Before this exploration, let us summarize the benefits and costs of the collective production operations ownership using SRE.

2.3.4 Benefits and Costs

The analysis in the previous chapter showed what it would take for product operations, product development, and product management to truly work together as a team using SRE methodology. It showed the deep integration among the three parties necessary to implement DevOps using SRE. It takes much more than only a good collaboration among the three parties. Table 2.2 juxtaposes the benefits and costs.

Table 2.2 Benefits and Costs of Collective Ownership of Production Operations Using SRE

Now that the benefits and costs of the common ownership of production operations using SRE are clear, let us take a look at the overall picture of what SRE is trying to achieve (Figure 2.5).

Figure 2.5 Collective ownership of production operations using SRE

SRE aligns the product delivery organization on operational concerns under its flag. Product development contributes to production operations by going on call to get firsthand experience with how the product meets customer demands in production. This experience is fed into the new feature and infrastructure development. The result is a maximization of feature development time while ensuring that the product meets customer demands in production.

Product operations makes a contribution by enabling developers to do production operations themselves. This is done through development of the SRE infrastructure as a framework to be used by the developers.

Product management contributes by making data-driven prioritization decisions about the most important user journeys for which the incidents need to be detected. Further, the contribution is in agreements with the autonomous incident backlog prioritization by the people on call. Additionally, the contribution is to be aligned with the data-driven prioritization decisions about reliability enablers to be included in the user story backlog.

The SRE transformation does not come for free. An investment in time, money, and effort is required to align the product delivery organization on operational concerns using SRE. That is why the executives need to also get involved. The executives can contribute twofold to the SRE transformation. First, they need to support the topic. This can be done in all-hands meetings, smaller conversations, and one-on-one discussions. Executive communication regarding SRE needs to clarify to everyone in the organization that the topic has executive support. This goes a long way toward creating alignment behind SRE at every level of the product delivery organization.

Moreover, the SRE transformation requires some slight investments in tooling and infrastructure. Although they are small scale, these investments need to be done in a timely manner so as not to impede the speed of transformation. In a large enterprise, it could take a significant amount of time to place orders due to supplier selection and data protection processes. Still, it is worth the effort. Voting with one’s wallet is a good way for executives to underpin the verbal message of endorsing the SRE transformation.