GoKawiilDuckDB v2.0: Your Database Deserves a Better Parser
Daniël ten Wolde | 16 min
TL;DR: DuckDB v2.0 replaces its PostgreSQL-derived SQL parser with a PEG-based parser that is easier to evolve and can be extended at runtime.
At DuckDB, one of our goals is to make working with a database system as easy as possible. Users interact with the system through the widely understood Structured Query Language (SQL). Previous blog posts have covered DuckDB’s friendly SQL, including GROUP BY ALL and column selection using SELECT * EXCLUDE (...) . Before DuckDB can execute a query using these features, however, it first has to determine whether its syntax is valid. That is the job of the parser, and in DuckDB v2.0 we are completely replacing it without you noticing.
At a high level, DuckDB processes a SQL query through the following stages:
In this blog, we focus on the tokenizer, parser, and transformer:
Tokenizer: This is the first step and is responsible for splitting up the raw input string into tokens. These can be of various categories, for example: KEYWORD , NUMBER , or IDENTIFIER . It is also where comments, in SQL denoted with either -- or /* */ , are recognized and skipped.
, , or . It is also where comments, in SQL denoted with either or , are recognized and skipped. Parser: The parser determines whether these tokens follow DuckDB's grammar and produces a ParseResult tree.
tree. Transformer: Converts the generic parse results into DuckDB’s internal abstract syntax tree (AST), forming structures such as SQLStatement , TableRef , and ParsedExpression . The resulting AST is passed on to the binder.
The parser determines whether a query is syntactically valid, while the binder determines whether the tables, columns, and functions it refers to actually exist.
... continue reading