GoKawiilIn yesterday’s post about the recent GitHub outage, there was a detail in the writeup that I didn’t say anything about: the autoscaling policy on the service with the saturated Istio sidecar.
Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I suspect readers of this blog are familiar with what autoscaling is and how it works, but here’s a brief summary in case you aren’t. The amount of compute and memory resources that a service requires depends on the load that’s placed on that service. The relevant source of load here is external requests against the service, also known as traffic. The volume of traffic varies over time. For example, for a company like GitHub, my guess is that they more traffic during working hours than evening and weekends.
Given that load changes dynamically, and that the compute and memory resources a service need is a function of load, there are two general strategies. One strategy is to provision your service for peak load. The other strategy is to dynamically adjust the resources allocated to your service, based on its current load; that’s called autoscaling.
If you want your service to use autoscaling, you need to define an autoscaling policy. In particular, you need to pick which metrics you want to use that represent load, and then you need to specify how resources should be added or removed based on how that metric changes.
CPU utilization is a common metric used for autoscaling. But note that a service can become saturated even if CPU is low. For example, imagine a scenario where you use thread-per-request with a threadpool, and the latency of your downstream requests increase, and all of the threads in the pool end up blocked. Here the service is saturated, and you’d benefit from spinning up new pods, but CPU is actually low, because the threads are blocked waiting on I/O (this happened to Slack back in 2021). Now, you can add additional rules to your autoscaling policy to handle such cases (which is what Slack did, where they rapidly scaled up based on number of threads). Or you can scale based on incoming request volume instead of CPU, if your service isn’t CPU-bound.
Based on the GitHub writeup, it sounds like the autoscaling policy for the impacted service used load metrics that only took into account load on the service itself, and not on the Istio sidecar.
In general, each service behaves differently under load, which means that every autoscaling policy is effectively bespoke. This means that a team that owns a service is not only responsible for the business logic, but also for an operational control system with custom parameters, that can really only be checked via load testing. (Are you doing load testing on all your services?) The service owners are also almost certainly not autoscaling experts. And so it’s not surprising to me that a misconfigured autoscaling policy was a contributor here.
But, while I think it’s worth discussing the particular defect with this policy, since it’s good for people to be aware of the risks of autoscaling, I also think it’s too easy to fixate on it to the exclusion of other factors involved in this incident. This is what David Woods refers to as the component substitution fallacy – the idea that the way to improve reliability is to focus efforts on identifying and fixing the defective components.
While, yes, you should identify and fix the defects uncovered by an incident, you should also recognize that:
... continue reading