TL;DR We explore the internal structure of Rust's Vec , expecting to find simple ptr , len , and capacity fields. Instead, we uncover a series of nested types, each adding a layer of safety and abstraction on top of a raw pointer ( *const u8 ). This deep dive reveals that Vec is built from components like NonNull , RawVec and others (which we explain layer by layer) to form the safe, user-friendly API that we love.
A conspiracy!
As I was reading the Rust API documentation for std::vec::Vec, something interesting caught my eye in the Vec struct definition.
pub struct Vec < T , A = Global > where A : Allocator , { }
I am looking at you { /* private fields */ } ! "What are you trying to hide from me?" I thought. Was I being sucked into a grand conspiracy? The documentation gives no hints as to what these fields are other than giving us a visual representation of the data structure:
ptr len capacity +--------+--------+--------+ | 0x0123 | 2 | 4 | +--------+--------+--------+ | v Heap +--------+--------+--------+--------+ | 'a' | 'b' | uninit | uninit | +--------+--------+--------+--------+
Surely we will find that our Vec has three fields ptr len and capacity . But to be sure we have to go straight to the source. Are you ready to come with me down the rabbit hole and see if we can uncover an age-old mystery?
The many faces of Vec
Diving into the struct definition of Vec in std::vec this is what we find:
pub struct Vec < T , A : Allocator = Global > { buf : RawVec < T , A > , len : usize , }
NOTE We will be ignoring the Allocator type entirely. This is a topic worth of its own article.
Yay, we have len ! Okay... that was easy. Now we only need ptr and capacity . We might be home very early, right?
No, not really!
"What is this misterious RawVec ?" you rightly ask yourself and where the hell is the ptr and the capacity ? Well, let's follow the breadcrumbs!
If we type RawVec into the search field of the Rust API documentation we find... nothing!?
I knew it! They really are trying to hide something from us!
Okay, okay... stay calm, don't stress it! Let's take a deep breath and look at the source code:
pub ( crate ) struct RawVec < T , A : Allocator = Global > { inner : RawVecInner < A > , _marker : PhantomData < T > , }
Ah, so that's why we can't find it in the documentation, it is only public within its crate pub(crate) and not accessible from outside. Good, one mistery solved but what in the world is the RawVecInner type now and what is PhantomData 1!? How deep does this rabbit hole go?
Looking at RawVecInner we get a clearer picture:
struct RawVecInner < A : Allocator = Global > { ptr : Unique < u8 > , cap : Cap , alloc : A , }
Haha, no, we don't... Well at least somewhat. We finally found our lost ptr and cap acity! But both of them are defined by new types. We're three layers deep now, with no end in sight. But we've come this far we're not stopping now, are we?
NOTE Cap is just a type which manages its min and max bounds so we won't go deep into this one. But you can check it out here and here
So what is Unique ?
pub struct Unique < T : PointeeSized > { pointer : NonNull < T > , _marker : PhantomData < T > , }
No surprises here, just another wrapper Type NonNull .
pub struct NonNull < T : PointeeSized > { pointer : * const T , }
Wait?! Are we done? I think we are! Hallelujah, we now have a broad overview of the whole Vec stack! Let's try to unravel it's secrets, shall we?
Understanding vecs layers
Our journey looked something like this:
Vec holds a...
holds a... RawVec which holds a...
which holds a... RawVecInner which holds a...
which holds a... Unique which holds a...
which holds a... NonNull which holds a...
which holds a... *const u8 (a raw pointer)
Phew, a lot of abstractions. But what does this tell us? To understand why the engineering team behind the standard library chose to go this route we first need to learn each layers purpose. We will start at the bottom and climb our way up until we reach the top of our Vec !
Silhouette of layers of mountains during dawn: Photo by Simon Rizzi on Pexels
At the camp
We start at the very bottom, our foundation is our constant basecamp which defines our entire structure: *const u8.
This is the most primitive way to refer to a location in memory and is also called a raw pointer. It is just plain memory address, a number. The official documentation tells us it's a risky tool since it can be null, dangle, or be unaligned. It doesn't have any lifetime information, so the compiler can't know if the data it points to is still valid.
Using it requires stepping into an unsafe block, telling the compiler, "I know what I'm doing." It's the necessary starting point because, to manage memory, you must first be able to talk about memory addresses directly.
GOOD TO KNOW A good question was asked in the Reddit thread for this post where a user (u/Anaxamander57) asked 'why is the pointer u8?' since pointers in Rust are based on usize. The answer is: Yes, the value of a pointer (the adress) is based on usize but the pointer points to a single byte (u8) in memory (the starting point of the data). In fact these are 'type erased' bytes (thank you u/Hy-o-pye). Why type-erased? Because at this level we do not care about the type of the data. It is just a growable buffer of memory.
The first step in adding security in our ascension is to make sure we have all our necessary tools at hand! So we check our tool belt and see if it is NonNull.
A simple but incredibly important wrapper that now secures our *const u8 and provides one crucial guarantee: the pointer is never null. Null pointers are the source of countless bugs and crashes in other languages (the infamous "billion-dollar mistake"). By encoding the non-null guarantee directly into the type system, Rust can eliminate this entire class of errors.
This also unlocks a fantastic compiler optimization. Since a NonNull can never be null, the compiler knows it can use the 0 address to represent the None variant when it sees an Option> . This means Option> takes up the exact same amount of space as a regular raw pointer! It's the first step in building our safe abstraction which is also a zero-cost one at that.
A unique path uncovers
Continuing our way up we come across a seemingly Unique path (at least the sign is telling us so)... It looks very dangerous because it is sooooo slim. Only one Person can safely travel this path at a time!
Unique is a crucial (internal) building block that sits between raw pointers and safe abstractions like RawVecInner . It builds on NonNull by adding a powerfull and very needed concept: a signal of ownership.
It is not a guard that enforces ownership rules but more of a sign held by a type like RawVecInner to tell the compiler two things:
"I am the sole owner of the data this sign points to."
"I am responsible for Drop ing that data when I go out of scope."
What's interesting here is that you can create a Unique pointer and another mutable reference pointing to the same data and the compiler won't complain!
MIND BLOWN! I know, right?! So why is that?
The creation of Unique is safe because you cannot cause undefined behaviour just by creating it. The safety boundary is drawn at a different place, namely while accessing the data. The safety concern arises when we want to use the value the pointer points to.
When you want to use the pointer to read or write memory ( as_mut() or as_ref() ), you must enter an unsafe block. And from now on you are on your own :). Wild West baby!
You as the implementor need to make the following promise: "I know what I am doing and I guarantee that no other pointer is being used to access this memory."
By upholding this promise (with a little sweat and work) you get compile time guarantees in return (like borrow checking and data race prevention). It's a zero cost contract that makes Rust's memory safety possible.
Eh, can I have some help here?
After the engraving (why would we do this in the middle of a climb?!) we are out of breath and need help from the top. In order to send us down a rope, the helpful people at the top need to first ask for one.
RawVec is a "low-level utility for managing a buffer of memory." It's the component that actually talks to the memory allocator by handling the allocation and asking the allocator for a block of memory on the heap, growing it when needed (usually by doubling the capacity), and, crucially, deallocating it when the Vec is dropped.
As we have learned already, this one only knows about the capacity (the total allocated space). It does not track how many elements are actually initialized and in use. This deliberate separation of concerns makes RawVec a perfect, reusable building block. Other collection types in the standard library that also need a growable buffer of memory, like VecDeque, can reuse this component instead of reinventing the wheel.
GOOD TO KNOW Remember that RawVecInner type we saw? That's a clever compile-time optimization. By splitting the logic, the part that isn't generic over T (RawVecInner) doesn't get duplicated for every Vec you create, which helps speed up compilation.
Reaching the top
At last, we arrive at the Vec, time to plant some flags!
This is the one that brings everything together. It holds the RawVec (which manages the memory) and adds the final piece of information: len .
This acts as the public API that hides all the unsafe complexity we've just learned about. The Vec knows how many elements are initialized (len) within the allocated block (capacity). It's responsibility is to ensure that you can only access initialized elements. This gives us the safe methods we love to use every day: push , pop , insert , indexing, and so on.
Each level builds upon the last, adding new guarantees and responsibilities, until we have a completely safe, efficient, and powerful data structure.
An ordinary Vec
Seems like my life remains somewhat uneventful. There is no big conspiracy and no hidden truth to be found... just very good engineering.
While we were digging deep to learn what Vec 's private fields are, we uncovered something very interesting: good API design. We saw how Rust's engineers built one of its most common types from the ground up, starting with an unsafe pointer and carefully wrapping it, layer by layer, until it's perfectly safe and ergonomic.
Every layer is crucial for Vec 's contract to be fulfilled. It’s a testament to the power of abstraction and separation of concerns. The "conspiracy" was about managing complexity, allowing each piece of the puzzle to do one thing and do it well. And in doing so, it provides reusable types that power other parts of the standard library.
Maybe next time, when you push an element to a vector, you'll know about the stack of abstractions working under the hood to make it all happen safely and efficiently. And that, in itself, is a pretty cool secret to have uncovered.
Conclusion
Uncovering Vec 's inner structure helped me a lot to understand a little more what happens behind the scene in the standard library.
Unfortunately we only touched on a fraction of the complex structure that forms Vec (but I hope enough, to have a better understanding). Explaining every little bit in detail would be too much for this post, so maybe there will be more Under the hood articles which explain Rust's inner life even better since obvious questions already arise while reading this article: "How does NonNull guarantee its promise?" or "How does RawVec manage memory?"