GoKawiilI’ve left the Microsoft Debugger Platform team twice, and each time I’ve started writing my own debugger. I must really like debuggers or something. This time, I have two reasons for writing a new debugger. The first is because I want to learn Rust better, and writing something I already understand pretty well seems like a good way to learn. The second reason is to make it easier for people to learn how a debugger works. Using Rust also helps here because there are lots of crates that can take care of things like symbols and disassembly, and it will let us focus on the core ideas involved in writing a debugger.
I’m a Rust novice, and if you’re looking to learn Rust better, I’d suggest to start with https://www.rust-lang.org/learn. I’m not going to try to explain too much about the Rust code, but the main concepts and APIs will apply across language. You should be able to follow along even without much knowledge of Rust. Those of you who read this and know Rust better than me, please feel free to create issues in the GitHub repo. Or even better, make a pull request! The code for this first part is in a branch called “part1”, and you can see the full code here. If you want to follow along, I suggest cloning the whole repo locally and viewing it in a good IDE like VS Code with the rust-analyzer plugin.
What is a debugger?
First, what is a debugger? There are lots of tools that are “debugging tools” but when most people hear “debugger” they usually think of a tool that can analyze either a running system or a static snapshot (such as a core file, crash dump, or VM snapshot)1. Most debuggers like GDB, LLDB, Visual Studio, and WinDbg can do both of these things. We’ll start by creating a debugger that supports live usermode debugging on Windows. The core concepts will apply to nearly any OS, with the main differences being the OS APIs for debugging a process and the terminology that is used for certain concepts.
In Windows, there are just a few APIs that provide the necessary functionality for implementing a debugger. Most of these are described on MSDN in the Debugging Functions page. Note that these are the APIs that are used by a debugger such as Visual Studio or WinDbg. The core engine of WinDbg is called DbgEng, and has its own set of APIs that can be accessed through the IDebugClient interface and related APIs. These higher level APIs and are very powerful, with built in stack unwinding, symbol analysis, flow control, disassembly, scripting, and lots of other capabilities. If you want to automate some debugging task, this is a better API to use. But that’s not what we’re going to do in this post series. We are going to implement a debugger from scratch using the most basic APIs provided by the OS.
There are a number of other functions we’ll need to interact with a running process, but these APIs are not debugging-specific. We’ll cover those as we encounter them.
Basic structure
At the center of a live debugging session is an event loop. A debugger registers for debug events from a target process by “attaching” to it. For each event that occurs, the OS will freeze the target process and notify the debugger with information about that debug event. The debugger then has the opportunity to examine or manipulate the state of the target process. The debugger then continues from the debug event.
There’s a ton of complexity embedded in here, but most of that complexity is in examining and manipulating the state of the target process. We’ll start by creating a debugger that attaches to a process, monitors the debug events, and continues from them to allow the process to execute normally. That part is relatively straightforward, and will give us a basis for other functionality that can be built on top of this.
Attaching to a process
... continue reading