Profiling without Source code – how I diagnosed Trackmania stuttering
A very common side effect of working as a programmer is the constant frustration of not having source code access to all the software you use. Bugs, problems or missing features in your own work can be frustrating enough — you know you’ll have to address all those issues at some point. But it’s even worse when you experience an issue and don’t have the option to solve it.
A recent example of this for me was playing the game Trackmania (2020) by Ubisoft Nadeo. If you’re not familiar with the game or any of its predecessors, just think of it as a racing game with wacky tracks and absurdly fast cars, which requires very precise and fast inputs.
While I enjoy the game very much, one issue kept cropping up: The game would stutter randomly but repeatedly, with the framerate dropping from around 130 fps to 10 or even 5 fps for short periods of time. At first, I didn’t worry too much about what I assumed were shader compilation stutters, but they persisted, even when I played the same track dozens of times. This would be bad enough in almost any game, but in a quick-reaction game like Trackmania, a spike of just 100 ms could cause the car to travel more than 40 metres between two frames. Getting a gold medal on certain tracks was hard enough, but with those stutters it bordered on the impossible. So, I was willing to put in the effort to figure out what was going on.
Sadly, I’m not a programmer working on Trackmania or have any access to it’s source files, my best shot at fixing the issues was that anyone online had had the same issue and fixed it. Although I found a bunch of passionate players describing similar issues, the proposed solutions were standard recommendations like: “Updating the drivers”, “Verifying steam files”, “Disabling VSync”… we all know the list.
At this point I started wondering if I could figure out the problem on my own? I don’t have a lot of experience with reverse engineering, but performance optimization is my passion. I started up the game alongside Superluminal to profile (you can read more about the program in my optimization toolbox article).
Reading the Matrix
The profiling itself went as smoothly as usual, and the initial results were as expected. One thread seems to handle most of the work: the ‘main thread’ of the game, plus others that handle tasks such as video playback. We will come back to this later. Just by looking at the thread utilization (the red/green diagram), we can already see the repeating pattern of the central update function.
This type of pattern recognition is an important first step. After all, we don’t know anything about the functions besides their usage pattern. Not only do we have no access to the source code, but we also can’t see any function names without access to debug symbols. Superluminal simply displays their memory addresses.
Fortunately, Trackmania is not the only application executing code here. Like most software, it also calls functions in other libraries, such as DirectX and SDL. Debug symbols are available for many of these libraries (Superluminal detects these automatically). In this example, the game is waiting for DirectX 11:
Overlays
After getting a sense for the overall structure of the code being run here, we can start looking for those stutter issues. For this, I select the function that I assume is the main tick function, and jump to it’s longest/slowest call. I noticed two different libraries in the callstack here. Even without function names these library names can be helpful.
In my case those library names were overlay64.dll and GameOverlayRenderer.dll. I found both dlls on my PC, one of them in the folder “Ubisoft Game Launcher” and the other in “Steam“. Yep, the game interacts with not only with one overlay but both the Steam Overlay and the Ubisoft Connect Overlay, even though I am not exactly sure how and why. On a second look I doubted that they were responsible for the lags, since they only called other OS functions for frame presentation and some Input-related functions that I would expect to be called anyway.
Nevertheless, a lead was a lead and disabling both overlays via their respective launchers was a quick potential fix to test. Sadly those indeed weren’t it, the stutters were still there.
Interestingly, the Steam Overlay still appeared in the profiling even after disabling it – maybe the Steam Input API is also handled via the same dll? Whatever the case, it seemed too unlikely that an overlay that was active in almost all games that I played would only act up in Trackmania.
Sidequest: .webm!?
The next interesting discovery wasn’t to be found on the game thread but in a thread that Trackmania had named WebmDecoder. As it was a separate thread and didn’t seem to block the main thread at any point, it couldn’t be directly responsible for my stutter issues.
However, this didn’t stop me from being curious. Why was Trackmania decoding videos while I was playing? Even if it wasn’t causing the stutters, it seemed eerie that the game was spending CPU time on decoding videos while I was trying to diagnose performance issues. What video files could the game be decoding during gameplay?
Naturally the next steps were finding .webm files that the game accessed. For this I downloaded Process Explorer, a program that, besides other features, also listed all active file handles of a given program.
In my tests the game used a single video file, a 40-seconds-shot of a busy race track.
I didn’t recognize this shot until I started the game up for my next round of tests:
Yes, that’s the main menu. Why would the decoding thread do anything during gameplay? Was it decoding the video constantly in the background? The thread activity didn’t seem to change between the main menu and gameplay, so this seemed like a sound theory. However, testing the theory turned out to be difficult. Deleting the file just led to the game automatically re-downloading it on start-up. The logical counter to this? Disabling the Wi-Fi and starting the game offline. The result was a main menu showing a fallback video which I could not find anywhere on my machine -was it embedded in the .exe file or an archive?
Replacing the video with a different .webm file didn’t work either. It might actually use a checksum to verify that the video is the expected one. I’d love to look into it further, but I’d already spent far too much time on this side discovery, which probably wouldn’t solve my original problem. For now, I have decided to return to my analysis of the main thread.
Another library? – The real issue
I continued my search for any readable information or API calls until I finally stumbled upon this function call:
The function name was unreadable again. The interesting part was the .dll that this function was part of. Open Planet is a community-driven scripting platform for the game that adds plugin support, among other things. The short function calls to it didn’t really match the long spikes I was looking at. However, it is in the nature of a scripting platform to affect the behaviour of the actual game code and therefore potentially cause frame spikes indirectly.
To test this I uninstalled OpenPlanet for my next rounds of measurements. Even before reading the profiling results, I knew that I found my culprit – the gameplay was suddenly so smooth, that I even set a new track record in my short test.
This is where things got interesting: After reinstalling Open Planet, the stutters did not reappear. This was likely because the stutters were caused by one of the installed plugins; a smooth installation without this plugin worked fine again.
So how did I not detect this earlier? When investigating performance problems, wouldn’t an optional plugin be the first thing you would suspect? The reason for this is actually an Open Planet feature designed to address performance issues. It’s a warning system that triggers notifications when a specific plugin causes a stutter (in this screenshot you can see it trigger during the game’s startup).
Similar systems are often used by programs with many plugins, for example Visual Studio:
However, if game code is called indirectly by a plugin the caused CPU time can’t be always attributed back to the plugin, which I assume is what happened here. In theory you could dive even deeper at this point, since the plugins themselves are open source – but for me personally being able to play stutter free again is enough, and I still have multiple gold medals to get 🙂
In case you enjoy reading articles about game performance and optimization, you can follow me on Mastodon, Bluesky or LinkedIn. I publish new articles about game programming, Unreal and game development in general about every month.