GoKawiilEvery time you validate data with Pydantic v2, the data-validation library most Python apps reach for, a Rust extension does the work. Its core, pydantic-core, is built with PyO3, the same toolchain we'll use here.
This post builds that same kind of bridge, small enough to read in one sitting: a JSON parser written in Rust, exposed to Python, so you can import it like any other package. The last step, turning the Rust result into Python objects, is the one to understand before you port anything: for a parser like this, it can cost more than the parsing itself.
The four steps from Rust to import
Getting Rust code into Python takes four steps:
Write a normal Rust module. Annotate it with PyO3 macros. Let maturin compile and install it. Import the result.
#[pyfunction] and #[pymodule] are the two Rust macros that do the wiring. A Rust attribute macro is close to a Python decorator: it rewrites the function it sits on, here adding the glue that lets Python call it and handles the type conversions and reference counting at the boundary.
Maturin then compiles the crate to a shared library (.so, .dylib, .dll) and drops it into your virtual environment, so import just works. I walk through this whole setup, from cargo new to the first import, in How to run Rust in Python with PyO3 and Maturin.
That first tutorial returns a single number. This one picks up where it left off, because the interesting part starts once you return a structure instead of a scalar.
The parser produces a Rust value first
The structure this parser returns is a JSON tree, and it's the running example for the rest of this post. In our Python to Rust cohort, students spend six weeks writing a JSON parser from scratch in Rust, a hand-rolled tokenizer and recursive-descent parser with no serde, then expose it to Python through PyO3. Josh's version beat CPython's C json module on real-world fixtures; Jochen's ran up to 3.5x faster than the Python version.
... continue reading