GoKawiilIn the prequel, we introduced Neptune, a Direct3D protocol forwarding layer for VirtIO. Neptune allowed us to serialize Direct3D API calls across the hypervisor boundary and this allowed us to run Wine games on a Linux guest with a Linux host faster than with DXVK directly in the guest. Admittedly, the payoff there was not that exciting but it laid the groundwork for our real goal: modern graphics acceleration for Windows guests. We have now achieved this by building a brand new Windows driver called Triton which along with Neptune brings full DirectX 11 support to QEMU virtual machines. Crash Bandicoot Trilogy (x64) running on Windows 11 ARM64 virtualized on macOS through QEMU
What is Triton?
You might be wondering: if Neptune can serialize Direct3D API calls and Windows uses Direct3D, then aren’t we already done? If Direct3D works in Wine, it should also work in Windows, right? After all, what is Wine, if not a Windows emulator? The short answer is: you sort of can. The Neptune Mesa drivers build a d3d11.dll and dxgi.dll which fully implements the Direct3D API set and so if you just put those files next to the game’s executable, it should load them instead of Windows’ own drivers and you can get some games to run that way. This is also the approach done by previous attempts which used DXVK→Vulkan→Venus to run Direct3D locally inside an application. There are a few disadvantages to this approach. First and most importantly, you cannot get good performance because the window compositor (DWM) “sees” your frame as an image so it needs to use CPU blitting to copy the GPU image buffer to the correct window location. You might be able to do some tricks for full-screen applications to scanout natively but you will never get a smooth desktop experience. Second, because d3d11.dll and dxgi.dll are core components of Windows, you cannot replace the system files themselves and expect Windows to still work. Even if you do manage to get it to work, you will not be able to play many games with anti-cheats which specifically detect this kind of modification. That is why it is only possible to get the DLL to load on a per-application basis (and compatibility varies). Which brings us to the last point: needing to copy files to every application you want working graphics acceleration is not a user friendly experience. The correct approach is not to implement the DirectX APIs but to implement the DirectX DDIs (Device Driver Interface). If you want a full play by play of the entire development process, check out this companion post.
DDI
User mode Application Direct3D 11 (d3d11.dll) User-mode driver (DDI) DXGI (dxgi.dll) Kernel mode Kernel-mode driver Hardware / Virtualization
In Windows, the application communicates with the system Direct3D and DXGI libraries. The d3d11.dll (as well as older versions) do the complicated work of state tracking and send a more sanitized stream of commands to the user-mode driver (UMD) which implements the DDI. The application also talks to dxgi.dll to initialize the graphics adapters, set up the swapchain, etc. The UMD also goes through the DXGI to talk with the kernel-mode driver (KMD). The KMD is implemented by the graphics vendor (us) to drive the actual hardware (or in our case the virtual hardware). For Wine, we implemented a custom d3d11.dll and dxgi.dll to intercept the API calls and now for Windows, we need to instead implement the UMD and KMD. So that’s the challenge: implement the UMD with the DirectX DDI interface and also set up a private interface with the KMD which communicates with the VirtIO device. Lucky for us, the second part has already been solved. Both anonymix007 and arehnman had independently been working on a KMD for Venus (Vulkan). Since Vulkan is a completely independent graphics API, it does not need to implement the DirectX DDI and its UMD is similar to the “replace d3d11.dll ” approach in that it talks directly with the KMD to drive commands to QEMU. Since Neptune is modelled after Venus, the high level kernel interfaces (for DMA, command buffers, etc) is very similar and the interface between UMD and KMD is exactly the same. Ultimately, we chose to use anonymix007’s branch as the base because their implementation had more features working on the KMD side. That leaves us with the hard part. We have to implement the DDI for DirectX 11. When you are designing a new system, it is always wise to understand how others who have come before have solved similar problems. Unfortunately, there are not many open source DDI implementations to draw inspiration from. Windows graphics drivers is a very niche subject and most of the experts work in one of the handful of graphics hardware vendors. This is one of the reasons that QEMU has never got far with Windows GPU acceleration. Fortunately for us, there are two working open source implementation that we can learn from. First, Mesa has a DirectX 10 UMD. If you did not read the last article, the short version is that Mesa implements OpenGL for Linux. Mesa performs state tracking for OpenGL and emits Gallium API calls. The Mesa DirectX 10 UMD is an alternative to OpenGL that emits the same Gallium API calls. Then the Gallium backend driver (AMD, Intel, VirGL, etc) converts them into native graphics driver APIs. The upstream Mesa only supports the software rasterization backend for DirectX 10 but there was some recent work to get it working with VirGL. Unfortunately, the macOS virglrenderer lacks support for many of the features that this UMD requires so it is not a viable way to get graphics acceleration for macOS hosts. However, the integration with the Mesa codebase provides us with a clean example for integrating Triton. VirtualBox has the only working open source DirectX 11 UMD. However, this driver cannot really be “adopted” for our use. The way their driver works is that they translate the DDI calls into an intermediate bytecode and then on the host side, they interpret the bytecode into DirectX API calls. While it would be easy (with AI help) to just port this bytecode emitter and interpreter into QEMU, we decided against it for a couple of reasons. First, we think this conversion of DDI into a bytecode and then lifting that bytecode back to DirectX API can result in bugs that limit compatibility with games. Indeed, reading forum threads online, it seems like many games do not run in VirtualBox for this reason. If there is a missing feature or bug in the translation engine, it would require a lot of active maintenance effort to fix and we do not want to depend on Oracle for that. Second, there is a licence incompatibility between VirtualBox’s GPLv3 and virglrenderer’s MIT License or QEMU’s LGPLv2. VirtualBox’s code can’t be integrated but we did learn some valuable insight from it. Their list of which DDI prototypes were implemented and which ones returned error is used as the minimal requirements for a working implementation. This information isn’t readily available from MSDN documentation and trying to implement every single prototype would be massive in scope. Their DXBC signature algorithm is also helpful to understand because Microsoft does not publish it anywhere. Since we didn’t want to take the VirtualBox approach of using an intermediate transport format for DDI calls, we can do something better. If you imagine d3d11.dll as a component that roughly transforms DirectX API calls into UMD DDI calls, then what we want our UMD to do is to transform the DDI call back to DirectX API calls. Why is this useful? Because then we can use our tested and working Neptune protocol without having to invent a new transport for serializing DDI calls. On the host side, we do not have to do any extra work to execute those calls. VirtualBox needs an emitter and transport layer on the guest as well as an interpreter and dispatcher on the host. Each step adds latency and the chance to introduce errors and incompatibility. We still need an emitter and transport on the guest but on the host side we do not need an interpreter because the deserialized Neptune commands ARE DirectX 11 API calls and can be dispatched without any additional parsing. One less transform step means less opportunity for mistakes. Another advantage of a DDI→API transform is that most DDI calls in D3D11 have an API equivalent meaning that the transform is as simple as mapping some API handles to device handles and sometimes doing a lookup for API→DDI enum differences. The biggest win however, also turns out to be the most complicated part of the story, which is the DXBC shader code.
DXBC
DXBC (DirectX Byte Code) is the IR code that Microsoft’s shader compiler (FXC) emits. Specifically, it is the older (pre-DirectX 12) format and is compiled from HLSL, the shader language that DirectX uses. Since Triton acts as a reverse transform from DDI to API, it does not need to disassemble and convert this shader bytecode. This is a huge win for us in terms of complexity and compatibility. Unfortunately, it is not as simple as passing the bytecode unmodified to the host.
Application authors HLSL shader source HLSL source FXC HLSL → DXBC shader compiler emits DXContainer header + parts Header magic, version, part table SHDR DXBC bytecode ISGN input signature (metadata) OSGN output signature (metadata) … other metadata parts ID3D11Device::CreateVertexShader passes the entire container d3d11.dll Direct3D 11 runtime pfnCreateVertexShader passes the SHDR part only UMD (Triton) user-mode driver
The compiler (FXC) emits the DXBC bytecode along with other metadata. d3d11.dll expects to see this metadata and consumes it. When the DDI is called, only the bytecode is passed. That means for us to make a valid “inverse transform” back to the API call, we need to re-construct all that metadata by interpreting the bytecode. In the end we still pass through the bytecode unmodified but since we don’t see the original DXContainer file, we have to synthesize fields that the host DirectX renderer expects. This was a lot of trial and error that the AI assistant handled but it is the weakest and most error prone part of our implementation.
... continue reading