Systems Report: Diagnosing a Linux Performance Regression
Automattic · January 14, 2025
From time to time, our systems engineers write up a case study detailing a notable moment on the infrastructure front lines. This month’s comes from Ale Crismani and Joshua Coughlan, systems wranglers who work on WordPress VIP.
At Automattic, we use Kubernetes to orchestrate the infrastructure running WordPress VIP applications. We have firewall rules that ensure an application cannot connect to resources that are dedicated to other applications, and we monitor those firewall rules in real time.
During routine maintenance of our servers, we noticed that our firewall monitoring had started failing. Our ensuing investigation uncovered a regression in the Linux kernel ipset module that resulted in some operations running up to 1,000 times slower. Read on to learn how we went from failure to fix.
The first symptom
As mentioned in the introduction, we have monitoring on our Kubernetes hosts to ensure that they conform to our security policies.
One of our monitoring scripts checks if the host has the correct IPs assigned to it, if the file system has been tampered with, if firewall rules are the ones we expect, and if too much traffic is getting rejected/dropped by them. It usually runs in about 2 seconds:
time ./security-checks.sh
ALL OK
... continue reading