GoKawiilI was recently trying to validate some performance improvements related to lld at $DAYJOB and it was a little frustrating to see the improvements in our benchmarks but not in the live-production dashboards.
Having come from a background working on web-services, I was used to looking at individual time-series dashboards, sometimes over a few percentiles, and I was expecing to see some noticeable change but the data seemed too noisy to make any conclusions.
Turns out a colleague had also faced similar issues when trying to evaluate build-speed improvements. There are lots of variables that can affect the build: cold-cache, incremental, local, remote, etc. and the build times can vary wildly depending on the state of the system and the workload. She ended up leveraging a cummulative distribution function (CDF) to visualize the data and it was a revelation to me.
This led me to explore a few other different ways to visualize data, in addition to the CDF, and how a single image or statistic is often not enough to tell the whole story. This post will walk through a single synthetic dataset and show how different visualizations can tell different stories about the same data. The goal is to convince you to look at your data and not just summarize it with a single number.
Everything below comes from one synthetic dataset with a fixed seed. The full script can be found in this gist. It is a single file with a nix-shell shebang, so you can reproduce every figure exactly as long as you are using nix.
Note I leveraged AI to help generate the data and charts in this post for the story. If that bugs you, sorry. 🤷
§The rollout that “made it worse”
Here is the setup: we operate a typical web-service and we rolled out a new caching tier over a week, hoping to cut request latency.
The change is fully deployed, and the latency dashboard that plots the mean looks like this:
Mean latency went up, from 112 ms to 122 ms. ☹️
... continue reading