typed-arrow provides a strongly typed, fully compile-time way to declare Arrow schemas in Rust. It maps Rust types directly to arrow-rs typed builders/arrays and arrow_schema::DataType — without any runtime DataType switching — enabling zero runtime cost, monomorphized column construction and ergonomic ORM-like APIs.
Why compile-time Arrow?
Performance: monomorphized builders/arrays with zero dynamic dispatch; avoids runtime DataType matching.
matching. Safety: column types, names, and nullability live in the type system; mismatches fail at compile time.
Interop: uses arrow-array / arrow-schema types directly; no bespoke runtime layer to learn.
Quick Start
use typed_arrow :: { prelude :: * , schema :: SchemaMeta } ; use typed_arrow :: { Dictionary , TimestampTz , Millisecond , Utc , List } ; # [ derive ( typed_arrow :: Record ) ] struct Address { city : String , zip : Option < i32 > } # [ derive ( typed_arrow :: Record ) ] struct Person { id : i64 , # [ record ( nested ) ] address : Option < Address > , tags : Option < List < Option < i32 > > > , // List column with nullable items code : Option < Dictionary < i32 , String > > , // Dictionary joined : TimestampTz < Millisecond , Utc > , // Timestamp(ms) with timezone (UTC) } fn main ( ) { // Build from owned rows let rows = vec ! [ Person { id : 1 , address : Some ( Address { city : "NYC" . into ( ) , zip : None } ) , tags : Some ( List ( vec! [ Some ( 1 ) , None , Some ( 3 ) ] ) ) , code : Some ( Dictionary ( "gold" . into ( ) , std :: marker :: PhantomData ) ) , joined : TimestampTz :: < Millisecond , Utc > ( 1_700_000_000_000 , std :: marker :: PhantomData , ) , } , Person { id : 2 , address : None , tags : None , code : None , joined : TimestampTz :: < Millisecond , Utc > ( 1_700_000_100_000 , std :: marker :: PhantomData , ) , } , ] ; let mut b = < Person as BuildRows > :: new_builders ( rows . len ( ) ) ; b . append_rows ( rows ) ; let arrays = b . finish ( ) ; // Compile-time schema + RecordBatch let batch = arrays . into_record_batch ( ) ; assert_eq ! ( batch . schema ( ) . fields ( ) . len ( ) , < Person as Record > :: LEN ) ; println ! ( "rows={}, field0={}" , batch . num_rows ( ) , batch . schema ( ) . field ( 0 ) . name ( ) ) ; }
Add to your Cargo.toml (derives enabled by default):
[ dependencies ] typed-arrow = { version = " 0.x " }
When working in this repository/workspace:
[ dependencies ] typed-arrow = { path = " . " }
Examples
Run the included examples to see end-to-end usage:
01_primitives — derive Record , inspect DataType , build primitives
— derive , inspect , build primitives 02_lists — List and List>
— and 03_dictionary — Dictionary
— 04_timestamps — Timestamp units
— units 04b_timestamps_tz — TimestampTz with Utc and custom markers
— with and custom markers 05_structs — nested structs → StructArray
— nested structs → 06_rows_flat — row-based building for flat records
— row-based building for flat records 07_rows_nested — row-based building with #[record(nested)]
— row-based building with 08_record_batch — compile-time schema + RecordBatch
— compile-time schema + 09_duration_interval — Duration and Interval types
— Duration and Interval types 10_union — Dense Union as a Record column (with attributes)
— Dense Union as a Record column (with attributes) 11_map — Map (incl. Option values) + as a Record column
Run:
cargo run --example 08_record_batch
Core Concepts
Record : implemented by the derive macro for structs with named fields.
: implemented by the derive macro for structs with named fields. ColAt : per-column associated items Rust , ColumnBuilder , ColumnArray , NULLABLE , NAME , and data_type() .
: per-column associated items , , , , , and . ArrowBinding : compile-time mapping from a Rust value type to its Arrow builder, array, and DataType .
: compile-time mapping from a Rust value type to its Arrow builder, array, and . BuildRows : derive generates Builders and Arrays with append_row(s) and finish .
: derive generates and with and . SchemaMeta : derive provides fields() and schema() ; arrays structs provide into_record_batch() .
: derive provides and ; arrays structs provide . AppendStruct and StructMeta : enable nested struct fields and StructArray building.
Metadata (Compile-time)
Schema-level: annotate with #[schema_metadata(k = "owner", v = "data")] .
. Field-level: annotate with #[metadata(k = "pii", v = "email")] .
. You can repeat attributes to add multiple pairs; later duplicates win.
Nested Type Wrappers
Struct fields: use #[record(nested)] (or #[nested] ) on a struct-typed field to treat it as an Arrow Struct . Make the parent struct nullable with Option ; child nullability is independent.
(or ) on a struct-typed field to treat it as an Arrow . Make the parent struct nullable with ; child nullability is independent. Lists: List (items non-null) and List> (items nullable). Use Option> for list-level nulls.
(items non-null) and (items nullable). Use for list-level nulls. LargeList: LargeList and LargeList> for 64-bit offsets; wrap with Option<_> for column nulls.
and for 64-bit offsets; wrap with for column nulls. FixedSizeList: FixedSizeList (items non-null) and FixedSizeListNullable (items nullable). Wrap with Option<_> for list-level nulls.
(items non-null) and (items nullable). Wrap with for list-level nulls. Map: Map where keys are non-null; use Map> to allow nullable values. Column nullability via Option> . SORTED sets keys_sorted in the Arrow DataType .
where keys are non-null; use to allow nullable values. Column nullability via . sets in the Arrow . OrderedMap: OrderedMap uses BTreeMap and declares keys_sorted = true .
uses and declares . Dictionary: Dictionary with integral keys K ∈ { i8, i16, i32, i64, u8, u16, u32, u64 } and values: String / LargeUtf8 (Utf8/LargeUtf8) Vec / LargeBinary (Binary/LargeBinary) [u8; N] (FixedSizeBinary) primitives i* , u* , f32 , f64 Column nullability via Option> .
with integral keys and values: Timestamps: Timestamp (unit-only) and TimestampTz (unit + timezone). Units: Second , Millisecond , Microsecond , Nanosecond . Use Utc or define your own Z: TimeZoneSpec .
(unit-only) and (unit + timezone). Units: , , , . Use or define your own . Decimals: Decimal128 and Decimal256 (precision P , scale S as const generics).
and (precision , scale as const generics). Unions: #[derive(Union)] for enums with #[union(mode = "dense"|"sparse")] , per-variant #[union(tag = N)] , #[union(field = "name")] , and optional null carrier #[union(null)] or container-level null_variant = "Var" .
Arrow DataType Coverage
Supported (arrow-rs v56):
Primitives: Int8/16/32/64, UInt8/16/32/64, Float16/32/64, Boolean
Strings/Binary: Utf8, LargeUtf8, Binary, LargeBinary, FixedSizeBinary (via [u8; N] )
) Temporal: Timestamp (with/without TZ; s/ms/us/ns), Date32/64, Time32(s/ms), Time64(us/ns), Duration(s/ms/us/ns), Interval(YearMonth/DayTime/MonthDayNano)
Decimal: Decimal128, Decimal256 (const generic precision/scale)
Nested: List (including nullable items), LargeList, FixedSizeList (nullable/non-null items) Struct, Map (Vec<(K,V)>; use Option for nullable values), OrderedMap (BTreeMap) with keys_sorted = true Union: Dense and Sparse (via #[derive(Union)] on enums) Dictionary: keys = all integral types; values = Utf8 (String), LargeUtf8, Binary (Vec), LargeBinary, FixedSizeBinary ( [u8; N] ), primitives (i*, u*, f32, f64)
Missing: