GoKawiilJune 30, 2026
How I tracked down a tiny Go lifecycle bug in Kubernetes kubelet which leaked a context on every startPodSync.
A few weeks ago, I started getting alerts from a tiny Kubernetes cluster: a single-node Kubernetes test cluster, not running any of our production workload. I’d recently upgraded this cluster to v1.36, hosted on DigitalOcean’s managed DOKS service. My frugality paid unexpected dividends: this tiny (ahem, cheap!) 2 GiB RAM node had so much memory pressure that it quickly revealed a deeper issue in Kubernetes 1.36, which would have taken longer to show up if memory were abundant.
Investigating the alerts revealed that Pods were being restarted, but kubectl top pods didn’t show any unusually-large pods. The applications running on the node weren’t experiencing memory growth and were nowhere near their memory limits.
Peeling back the Kubernetes facade, I opened up a root shell on the node itself, and a short htop and M later, quickly discovered that the kubelet process itself had grown and was growing!
A quick systemctl restart kubelet on the node made the cluster happy again, but the underlying leak was still there, and would come back soon unless I determined the origin of the leak.
Dumping kubelet’s heap
Kubernetes is written in Go, and kubelet is a core component of how Kubernetes works: it runs on every node, and is responsible for keeping that node’s containers in sync with the desired cluster state.
Go’s pprof package lets you capture a heap memory profile from a running process, which I saved to a file:
kubectl get --raw "/api/v1/nodes/ ${ NODE } /proxy/debug/pprof/heap?debug=0" > "kubelet_pprof_heap.pb.gz"
... continue reading