GoKawiil3D Gaussian splatting is a technique that answers this question: given a dataset of pictures of a scene, how can I reconstruct it in 3D? This is achieved using a machine learning algorithm that, for many different camera angles, does the following: render the scene, compare it to the picture taken at the same camera angle, and update the scene to reduce the difference between the rendered image and the ground truth. However, unlike traditional 3D renderers, 3DGS does not use triangles as primitives, but objects called Gaussian splats, making the rendering algorithm unique to 3DGS.
In this post, I aim to show how 3D Gaussian splatting works by building a simplified renderer from scratch, in ~1000 lines of code. The main motivation is to build intuition on the 3DGS math. It is recommended to understand the basics of linear algebra, probability theory and computer graphics.
The renderer is written in C++ and OpenGL. All code is available on GitHub, but I tried to make this tutorial general enough that you can reproduce it using any graphics engine (WebGPU, Metal, DirectX…). At the end of the tutorial, we’ll be able to render a Gaussian scene like this, in real time:
I’ve also included interactive WebGPU visualizations that you can navigate with WASD and the mouse.
This article covers only rendering, not training. However, there are technical decisions in the 3DGS renderer Kerbl et al. 2023 that are tightly coupled to the training pipeline (differentiability, positive semi-definite covariance preservation) that we’ll highlight.
Loading a 3DGS scene#
First, we need a scene to render. At the time of this article, good 3DGS scenes are still hard to find, but thankfully Supersplat recently made it possible to download splats from their website, so this is what we’ll be doing. We’ll use this plate of tomatoes scene because it’s fairly lightweight (only ~200k splats), but feel free to use any scene you like. Just be aware that the renderer we’re building is not really optimized for large scenes.
Then, we’re going to load the Gaussian splat scene we downloaded from Supersplat. This lets us check that the scene is oriented correctly, and more importantly, gives us a first look at what a splat actually is. The typical format is .ply , for which I’ve included a custom loader in ply_loader.h . The loader parses the .ply file into an array of GaussianSplat objects, wrapped in a Scene :
constexpr int SH_COUNT = 16 ; constexpr int SH_CHANNEL_COUNT = 3 ; constexpr int SH_FLOAT_COUNT = SH_COUNT * SH_CHANNEL_COUNT ; struct GaussianSplat { glm :: vec3 centroid = glm :: vec3 ( 0.0f ); float opacity = 0.0f ; std :: array < float , SH_FLOAT_COUNT > sphericalHarmonics = {}; std :: array < float , 3 > scale = { 0.0f , 0.0f , 0.0f }; std :: array < float , 4 > rotation = { 1.0f , 0.0f , 0.0f , 0.0f }; }; struct Scene { std :: vector < GaussianSplat > splats ; };
Let’s break this down:
... continue reading