Where This Story Begins
In 2022, we had three different query interfaces. Logs had a custom search syntax with no autocomplete. Traces only had predefined filters - no query builder at all. Metrics had a raw PromQL input box where you'd paste queries from somewhere else and hope they worked.
Each system spoke a different language. An engineer debugging a production issue had to context-switch not just between data types, but between entirely different mental models of how to query data.
When we built v3 in 2022, we thought we were solving this. We created a unified query builder - essentially a UI abstraction over SQL. Count, group by, filter, limit. It worked well enough to carry us from 2022 to 2024.
But we were building with the wrong assumptions.
The v3/v4 Design Flaw That Took Two Years to Understand
We designed v3 around traces and metrics. In these data types, you rarely need complex boolean logic. A simple AND between conditions is usually enough.
But logs are different. When you're searching logs during an incident, you need expressions like:
( node_name contains 'management' OR pod_name contains 'test' ) AND NOT ( status_code >= 500 )
v3 couldn't do this. No OR support. No complex boolean expressions. No parentheses for precedence.
... continue reading