If you’re anxious about the size of your binary, there’s a lot of useful advice on the internet to help you reduce it. In my experience, though, people are reticent to discuss their static libraries. If they’re mentioned at all, you’ll be told not to worry about their size: dead code will be optimized away when linking the final binary, and the final binary size is what matters.
But that advice didn’t help me, because I wanted to distribute a static library and the size was causing me problems. Specifically, I had a Rust library that I wanted to make available to Go developers. Both Rust and Go can interoperate with C, so I compiled the Rust code into a C-compatible library and made a little Go wrapper package for it. Like most pre-compiled C libraries, I can distribute it either as a static or a dynamic library. Now Go developers are accustomed to static linking, which produces self-contained binaries that are refreshingly easy to deploy. Bundling a pre-compiled static library with our Go package allows Go developers to just go get https://github.com/nickel-lang/go-nickel and get to work. Dynamic libraries, on the other hand, require runtime dependencies, linker paths, and installation instructions.
So I really wanted to go the static route, even if it came with a slight size penalty. How large of a penalty are we talking about, anyway?
❯ ls -sh target/release/ 132M libnickel_lang.a 15M libnickel_lang.so
😳 Ok, that’s too much. Even if I were morally satisfied with 132MB of library, it’s way beyond GitHub’s 50MB file size limit. (Honestly, even the 15M shared library seems large to me; we haven’t put much effort into optimizing code size yet.)
The compilation process in a nutshell
Back in the day, your compiler or assembler would turn each source file into an “object” file containing the compiled code. In order to allow for source files to call functions defined in other source files, each object file could announce the list of functions that it defines, and the list of functions that it very much hopes someone else will define. Then you’d run the linker, a program that takes all those object files and mashes them together into a binary, matching up the hoped-for functions with actual function definitions or yelling “undefined symbol” if it can’t. Modern compiled languages tweak this pipeline a little: Rust produces an object file per crate instead of one per source file. But the basics haven’t changed much.
A static library is nothing but a bundle of object files, wrapped in an ancient and never-quite-standardized archive format. No linker is involved in the creation of a static library: it will be used eventually to link the static library into a binary. The unfortunate consequence is that a static library contains a lot of information that we don’t want. For a start, it contains all the code of all our dependencies even if much of that code is unused. If you compiled your code with support for link-time optimization (LTO), it contains another copy (in the form of LLVM bitcode — more on that later) of all our code and the code of all our dependencies. And then because it has so much redundant code, it contains a bunch of metadata (section headers) to make it easier for the linker to remove that redundant code later. The underlying reason for all this is that extra fluff in object files isn’t usually considered a problem: it’s removed when linking the final binary (or shared library), and that’s all that most people care about.
Re-linking with ld
I wrote above that a linker takes a bunch of object files and mashes them together into a binary. Like everything in the previous section, this was an oversimplification: if you pass the --relocatable flag to your linker, it will mash your object files together but write out the result as an object file instead of a binary. If you also pass the --gc-sections flag, it will remove unused code while doing so.
... continue reading