GoKawiilHunting Down a Go Runtime Bug on 32-bit Embedded Systems
Our daily work usually revolves around Linux and security topics, deep down in the software stack. Still, more often than you might think, we end up debugging applications which live much higher up. Sometimes such a problem has its roots in the Linux kernel, sometimes elsewhere. In this blog post we show how we found and fixed a bug inside the Go runtime.
Recently a customer reported that an application written in Go crashes from time to time on one of their embedded Linux systems.
Introduction
The crash was always the same fatal error with the following signature:
runtime: netpoll: eventfd ready for 5 fatal error: runtime: netpoll: eventfd ready for something unexpected ...stack trace...
At first we assumed that the application itself was buggy and needed fixing. But after inspecting the error more closely, it looked much more like an internal assumption in Go’s netpoll mechanism no longer holds.
The error message comes from netpoll() in src/runtime/netpoll_epoll.go:
if ev . Events != linux . EPOLLIN { println ( "runtime: netpoll: eventfd ready for" , ev . Events ) throw ( "runtime: netpoll: eventfd ready for something unexpected" ) }
In this code path, the netpoll code expects EPOLLIN to be the only firing event, but it got something else. In our case it got 5 , which is EPOLLIN|EPOLLOUT . Why would epoll suddenly report more than EPOLLIN if the code asked only for EPOLLIN ?
... continue reading