At the core of every DE engine there’s some form of persistent durable execution log. You can think of this a bit like the write-ahead log of a database. It captures the intent to execute a given flow step, which makes it possible to retry that step should it fail, using the same parameter values. Once successfully executed, a step’s result will also be recorded in the log, so that it can be replayed from there if needed, without having to actually re-execute the step itself.
DE logs come in two flavours largely speaking; one is in the form of an external state store which is accessed via some sort of SDK. Example frameworks taking this approach include Temporal, Restate, Resonate, and Inngest. The other option is to persist DE state in the local database of a given application or (micro)service. One solution in this category is DBOS, which implements DE on top of Postgres.
To keep things simple, I went with the local database model for Persistasaurus, using SQLite for storing the execution log. But as we’ll see later on, depending on your specific use case, SQLite actually might also be a great choice for a production scenario, for instance when building a self-contained agentic system.
The structure of the execution log table in SQLite is straight-forward. It contains one entry for each durable execution step:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 CREATE TABLE IF NOT EXISTS execution_log ( flowId TEXT NOT NULL , (1) step INTEGER NOT NULL , (2) timestamp INTEGER NOT NULL , (3) class_name TEXT NOT NULL , (4) method_name TEXT NOT NULL , (5) delay INTEGER , (6) status TEXT (7) CHECK ( status IN ( 'PENDING' , 'WAITING_FOR_SIGNAL' , 'COMPLETE' ) ) NOT NULL , attempts INTEGER NOT NULL DEFAULT 1 , (8) parameters BLOB , (9) return_value BLOB , (10) PRIMARY KEY ( flowId , step ) )
1 The UUID of the flow 2 The sequence number of the step within the flow, in the order of execution 3 The timestamp of first running this step 4 The name of the class defining the step method 5 The name of the step method (currently ignoring overloaded methods for this PoC) 6 For delayed steps, the delay in milli-seconds 7 The current status of the step 8 A counter for keeping track of how many times the step has been tried 9 The serialized form of the step’s input parameters, if any 10 The serialized form of the step’s result, if any
This log table stores all information needed to capture execution intent and persist results. More details on the notion of delays and signals follow further down.
When running a flow, the engine needs to know when a given step gets executed so it can be logged. One common way for doing so is via explicit API calls into the engine, e.g. like so with DBOS Transact:
1 2 3 4 5 @Workflow public void workflow () { DBOS . runStep (() -> stepOne (), "stepOne" ); DBOS . runStep (() -> stepTwo (), "stepTwo" ); }
This works, but tightly couples workflows to the DE engine’s API. For Persistaurus I aimed to avoid this dependency as much as possible. Instead, the idea is to transparently intercept the invocations of all step methods and track them in the execution log, allowing for a very concise flow expression, without any API dependencies:
... continue reading