When a few variables belong together, we put them in a struct. Programmers do this automatically without thinking about it much.
And most of the time it's the right choice.
Structs are simple, fast, and predictable. But once in a while they break down. This is the story of one of those cases.
How it started
One of our customers reported a strange performance problem. A new use case processed about the same amount of data as their existing pipelines, but it ran much slower.
That was unusual. Our engine normally keeps up with the data customers send. So we felt compelled to take a closer look.
How we use structs
In Feldera, users define input data as SQL tables and output data as SQL views. We compile the SQL in between into a Rust program that incrementally evaluates the query.
Each row of a table becomes a Rust struct.
Here is an (anonymized) excerpt from the workload that triggered the slowdown:
... continue reading