GoKawiilGit packfiles were designed for mmap and local disk, so pulling one object out of a bucket means guessing at a byte range. I wrote a new format instead.
It sure seems that a bunch of companies are trying to ship a git product of some kind as of late. Wonder why that is.
Either way, I’m building a Git server backed by object storage as an open-source project. It sounded simple enough to start: Git looks like a filesystem, so let’s use a filesystem as a translation layer on top of object storage to make Git speak object storage. This model worked… ok, I guess? But it didn’t work for real-world size repositories, so I needed a different approach. Git stores everything in Objects, so why not store those as objects in Tigris?
Turns out Git packfiles and how they intersected with my (admittedly somewhat terrible) filesystem shim were the main reason why it was slow. I ended up having to invent my own packfile format with a columnar store that’s object storage native. This is the fruit of all of my performance analysis, metrics annotations, and more Texas-style distributed systems work than you can make your k8s cluster shake sticks at.
This approach worked surprisingly well for production-sized repositories, so I’m sticking with this new Packfile format for now. It seems the least obtrusive change to make Git Objects feel like object storage Objects, without any client side changes.
When you make a commit, Git stores the changes you make as objects inside the .git (I’ll call this “dotgit” so I don’t have to write as many backticks) folder.
Imagine Git as two things: a sea of objects and named references to individual objects. Each object is a content-addressed and compressed file. Here’s an example from a tiny git repository:
$ mkdir ~/tmp/gitexample
$ git init && git branch -m main
$ echo "Hello, blog!" >> hello.txt
... continue reading