5 ms·

Building a Durable Execution Engine with SQLite

178 points by ingve 11mo ago

fiddlerwoaroof 11mo ago
Every several years people reinvent serializable continuations
- andersmurphy 11mo ago
  Haha so true. Shame image based programming never really caught on. Janet lang lets you serialize coroutines which is fun. Make this sort of stuff trivial.
- gunnarmorling 11mo ago
  Yupp, making that same point in the post :) > You could think of [Durable Execution] as a persistent implementation of the memoization pattern, or a persistent form of continuations.
- smitty1e 11mo ago
  Is this reinvention somehow "transactional" in nature?
- rileymichael 11mo ago
  unfortunately they've never really taken off so folks reach for explicit state machines instead. there have been a handful of options on the jvm over the years (e.g. quasar, kilim) but they're all abandoned now, the loom continuation API is internal with no hint of it becoming public, kotlin's aren't serializable and the issue is inactive (https://github.com/Kotlin/kotlinx.coroutines/issues/76 https://github.com/Kotlin/kotlinx.coroutines/issues/76), etc.. such a shame
websiteapi 11mo ago
there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here? you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.
- tptacek 11mo ago
  We build what is effectively a durable execution "engine" for our orchestrator (ours is backed by boltdb and not SQLite, which I objected to, correctly). The steps in our workflows build running virtual machines and include things like allocating addresses, loading BPF programs, preparing root filesystems, and registering services. Short answer: we need to be able to redeploy and bounce the orchestrator without worrying about what stage each running VM on our platform is in. JP, the dev that built this out for us, talks a bit about the design rationale (search for "Cadence") here: https://fly.io/blog/the-exit-interview-jp/ https://fly.io/blog/the-exit-interview-jp/ The library itself is open: https://github.com/superfly/fsm https://github.com/superfly/fsm
- ryeats 11mo ago
  As you say it can be done but it's an anti-pattern to use a message queue as a database which is essentially what you are doing for these kinds of long running tasks. The reason is that their are a lot of state your likely going to want to status as a task runs and persist and checkpoint yes you can carefully string together a series of database calls chained with message transactions so you don't lose something when an issue happens but then you also need bespoke logic to restart or retry each step and it can turn into a bit of a mess.
  - deleted 11mo ago
    [deleted]
- snicker7 11mo ago
  Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as: - Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag) You really do need a database for this.
  - yyx 11mo ago
    Sounds like a Celery with SQLAlchemy backend.
the_mitsuhiko 11mo ago
I think this is great. We should see more simple solution to this problem. I recently started doing something very similar on Postgres [1] and I'm greatly enjoying using it. I think the total solution I ended up with is under 3000 lines of code for both the SQL and the TypeScript SDK combined, and it's much easier to use and to operate than many of the solutions on the market today. [1]: https://github.com/earendil-works/absurd https://github.com/earendil-works/absurd
- gunnarmorling 11mo ago
  Ah, that's awesome. Definitely need to take a closer look at your implementation.
- nileshtrivedi 11mo ago
  This looks useful. Hope would you say it compares with pgqueuer?
  - the_mitsuhiko 11mo ago
    I think pgqueuer is like pgmq which I used a lot. pgmq is just the queue part, absurd does the state storage. I wrote some more about why it exists here: https://lucumr.pocoo.org/2025/11/3/absurd-workflows/ https://lucumr.pocoo.org/2025/11/3/absurd-workflows/
Brian-Watkins 11mo ago
[dead]
qianli_cs 11mo ago
I really enjoyed this post and love seeing more lightweight approaches! The deep dive on tradeoffs between different durable-execution approaches was great. For me, the most interesting part is that Persistasaurus (cool name btw) use of bytecode generation via ByteBuddy is a clever way to improve DX: it can transparently intercept step functions and capture execution state without requiring explicit API calls. (Disclosure: I work on DBOS [1]) The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX. [1]: https://github.com/dbos-inc https://github.com/dbos-inc
- kodablah 11mo ago
  > The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX. There is value in such a wrapper/call at invocation time instead of using the proxy pattern. Specifically, it makes it very clear to both the code author and code reader that this is not a normal method invocation. This is important because it is very common to perform normal method invocations and the caller needs to author code knowing the difference. Java developers, perhaps more than most, likely prefer such invocation explicitness over a JVM agent doing byte code manip. There is also another reason for preferring a wrapped-like approach - providing options. If you need to provide options (say timeout info) from the call site, it is hard to do if your call is limited to the signature of the implementation and options will have to be provided in a different place.
  - gunnarmorling 11mo ago
    I'm still swinging back and forth which approach I ultimately prefer. As stated in the post, I like how the proxy approach largely avoids any API dependency. I'd also argue that Java developers actually are very familiar with this kind of implicit enrichment of behaviors and execution semantics (e.g. transaction management is weaved into applications that way in Spring or Quarkus applications). But there's also limits to this in regards to flexibility. For example, if you wanted to delay a method for a dynamically determined period of time, rather than for a fixed time, the annotation-based approach would fall short.
roughly 11mo ago
One thing that needs to be emphasized with “durable execution” engines is they don’t actually get you out of having to handle errors, rollbacks, etc. Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? You failed while updating the inventory system - did the product get marked out or not? All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.
- hedgehog 11mo ago
  In my one encounter with one of these systems it induced new code and tooling complexity, orders of magnitude performance overhead for most operations, and made dev and debug workflows much slower. All for... an occasional convenience far outweighed by the overall drag of using it. There are probably other environments where something like this makes sense but I can't figure out what they are.
  - throwaway894345 11mo ago
    > All for... an occasional convenience far outweighed by the overall drag of using it If you have any long-running operation that could be interrupted mid-run by any network fluke (or the termination of the VM running your program, or your program being OOMed, or some issue with some third party service that your app talks to, etc), and you don’t want to restart the whole thing from scratch, you could benefit from these systems. The alternative is having engineers manually try to repair the state and restart execution in just the right place and that scales very badly. I have an application that needs to stand up a bunch of cloud infrastructure (a “workspace” in which users can do research) on the press of a button, and I want to make sure that the right infrastructure exists even if some deployment attempt is interrupted or if the upstream definition of a workspace changes. Every month there are dozens of network flukes or 5XX errors from remote endpoints that would otherwise leave these workspaces in a broken state and in need of manual repair. Instead, the system heals itself whenever the fault clears and I basically never have to look at the system (I periodically check the error logs, however, to confirm that the system is actually recovering from faults—I worry that the system has caught fire and there’s actually some bug in the alerting system that is keeping things quiet).
whinvik 11mo ago
Sorry for the off-topic but I have been lately seeing a lot of hype around durable execution. I still cannot figure out how this is any different than launching a workflow in something like Airflow. Is the novel thing here that it can be done using the same DB you already have running?
adamzwasserman 11mo ago
Reminds me of IBM TPF (Transaction Processing Facility) - the system that powered airline reservations for decades. TPF used per-transaction logging with restart/recovery semantics at massive scale. You could literally unplug the power mid-transaction, plug it back in, and resume exactly where you left off. The embedded database approach here is interesting though - low latency, no network calls, perfect for single-agent workflows. TPF assumed massive concurrent load across distributed terminals. Different problems, similar durability patterns.
9642370096647 11mo ago
This is irrelevant for local LLMs where a seed value may be specified and generation is totally deterministic. This only helps with online LLMs.
throwaway290 11mo ago
> A workflow engine running a BPMN job Does anyone really do this?
- gunnarmorling 11mo ago
  Sure, systems like Activiti or Camunda come to mind.
  - throwaway290 11mo ago
    my question is does anyone (here on HN is implied) really do this? in production for some serious purpose?
    - gunnarmorling 11mo ago
      Camunda (not affiliated btw.) crossed 100M ARR last year. So yes, this apparently is being used. Whether these users hang out here on HN or not, that I don't know.
    - tauchunfall 11mo ago
      I worked on a product that used BPMN where users could define processes. The company I worked for used Java for decades already. Clients of the product were banks. The people I worked with were not specifically HN audience. Rather in the Java bubble in Germany-Austria-Switzerland which is also surprisingly a small world. If BPMN is not really needed, then I would also not use it nowadays. It increases complexity, and who knows if it makes project communication better at all. Update: On the Camunda website there are 60 case-studies of customers/clients using BPMN, https://camunda.com/case-studies/ https://camunda.com/case-studies/. One of them has the teaser: "The 10th largest US Bank created an omnichannel onboarding platform that handles 12m process instances per year across 100 workflows". Now I have something to read for this Sunday evening.
      - throwaway290 11mo ago
        Thanks. I guess I'm in my bubble...
RagnarD 11mo ago
Serious question: How does "Durable Execution" differ from "Atomic Transaction"? At most, it seems that DE refers to more concrete details around implementing Atomic Transactions.
- athrowaway3z 11mo ago
  Maybe its about tools for inspecting the trace of a running atomic transaction? Not really sure because "Lately, there has been a lot of excitement around Durable Execution (DE) engines" is also news to me, and AFAICT it seems to be yet-another-reinvention CS is famous for.
- gunnarmorling 11mo ago
  It depends on what exactly you mean by "Atomic Transaction". If you are referring to database transactions, then DE is not that. But DE helps you implement potentially long-running multi-step business transactions, a.k.a. workflows, sometimes spanning over days or weeks. Oftentimes, these transactions are atomic, but don't satisfy the isolation requirement of ACID transactions, i.e. partial outcomes of in-flight flows may be visible to external observers.

178 points by ingve 11mo ago

fiddlerwoaroof 11mo ago

Every several years people reinvent serializable continuations

andersmurphy 11mo ago
Haha so true. Shame image based programming never really caught on. Janet lang lets you serialize coroutines which is fun. Make this sort of stuff trivial.
gunnarmorling 11mo ago
Yupp, making that same point in the post :) > You could think of [Durable Execution] as a persistent implementation of the memoization pattern, or a persistent form of continuations.
smitty1e 11mo ago
Is this reinvention somehow "transactional" in nature?
rileymichael 11mo ago
unfortunately they've never really taken off so folks reach for explicit state machines instead. there have been a handful of options on the jvm over the years (e.g. quasar, kilim) but they're all abandoned now, the loom continuation API is internal with no hint of it becoming public, kotlin's aren't serializable and the issue is inactive (https://github.com/Kotlin/kotlinx.coroutines/issues/76 https://github.com/Kotlin/kotlinx.coroutines/issues/76), etc.. such a shame

websiteapi 11mo ago

there's a lot of hype around durable execution these days. why do that instead of regular use of queues? is it the dev ergonomics that's cool here? you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.

tptacek 11mo ago
We build what is effectively a durable execution "engine" for our orchestrator (ours is backed by boltdb and not SQLite, which I objected to, correctly). The steps in our workflows build running virtual machines and include things like allocating addresses, loading BPF programs, preparing root filesystems, and registering services. Short answer: we need to be able to redeploy and bounce the orchestrator without worrying about what stage each running VM on our platform is in. JP, the dev that built this out for us, talks a bit about the design rationale (search for "Cadence") here: https://fly.io/blog/the-exit-interview-jp/ https://fly.io/blog/the-exit-interview-jp/ The library itself is open: https://github.com/superfly/fsm https://github.com/superfly/fsm
ryeats 11mo ago
As you say it can be done but it's an anti-pattern to use a message queue as a database which is essentially what you are doing for these kinds of long running tasks. The reason is that their are a lot of state your likely going to want to status as a task runs and persist and checkpoint yes you can carefully string together a series of database calls chained with message transactions so you don't lose something when an issue happens but then you also need bespoke logic to restart or retry each step and it can turn into a bit of a mess.
- deleted 11mo ago
  [deleted]
snicker7 11mo ago
Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as: - Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag) You really do need a database for this.
- yyx 11mo ago
  Sounds like a Celery with SQLAlchemy backend.

the_mitsuhiko 11mo ago

I think this is great. We should see more simple solution to this problem. I recently started doing something very similar on Postgres [1] and I'm greatly enjoying using it. I think the total solution I ended up with is under 3000 lines of code for both the SQL and the TypeScript SDK combined, and it's much easier to use and to operate than many of the solutions on the market today. [1]: https://github.com/earendil-works/absurd https://github.com/earendil-works/absurd

gunnarmorling 11mo ago
Ah, that's awesome. Definitely need to take a closer look at your implementation.
nileshtrivedi 11mo ago
This looks useful. Hope would you say it compares with pgqueuer?
- the_mitsuhiko 11mo ago
  I think pgqueuer is like pgmq which I used a lot. pgmq is just the queue part, absurd does the state storage. I wrote some more about why it exists here: https://lucumr.pocoo.org/2025/11/3/absurd-workflows/ https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

Brian-Watkins 11mo ago

[dead]

qianli_cs 11mo ago

I really enjoyed this post and love seeing more lightweight approaches! The deep dive on tradeoffs between different durable-execution approaches was great. For me, the most interesting part is that Persistasaurus (cool name btw) use of bytecode generation via ByteBuddy is a clever way to improve DX: it can transparently intercept step functions and capture execution state without requiring explicit API calls. (Disclosure: I work on DBOS [1]) The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX. [1]: https://github.com/dbos-inc https://github.com/dbos-inc

kodablah 11mo ago
> The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX. There is value in such a wrapper/call at invocation time instead of using the proxy pattern. Specifically, it makes it very clear to both the code author and code reader that this is not a normal method invocation. This is important because it is very common to perform normal method invocations and the caller needs to author code knowing the difference. Java developers, perhaps more than most, likely prefer such invocation explicitness over a JVM agent doing byte code manip. There is also another reason for preferring a wrapped-like approach - providing options. If you need to provide options (say timeout info) from the call site, it is hard to do if your call is limited to the signature of the implementation and options will have to be provided in a different place.
- gunnarmorling 11mo ago
  I'm still swinging back and forth which approach I ultimately prefer. As stated in the post, I like how the proxy approach largely avoids any API dependency. I'd also argue that Java developers actually are very familiar with this kind of implicit enrichment of behaviors and execution semantics (e.g. transaction management is weaved into applications that way in Spring or Quarkus applications). But there's also limits to this in regards to flexibility. For example, if you wanted to delay a method for a dynamically determined period of time, rather than for a fixed time, the annotation-based approach would fall short.

roughly 11mo ago

One thing that needs to be emphasized with “durable execution” engines is they don’t actually get you out of having to handle errors, rollbacks, etc. Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? You failed while updating the inventory system - did the product get marked out or not? All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.

hedgehog 11mo ago
In my one encounter with one of these systems it induced new code and tooling complexity, orders of magnitude performance overhead for most operations, and made dev and debug workflows much slower. All for... an occasional convenience far outweighed by the overall drag of using it. There are probably other environments where something like this makes sense but I can't figure out what they are.
- throwaway894345 11mo ago
  > All for... an occasional convenience far outweighed by the overall drag of using it If you have any long-running operation that could be interrupted mid-run by any network fluke (or the termination of the VM running your program, or your program being OOMed, or some issue with some third party service that your app talks to, etc), and you don’t want to restart the whole thing from scratch, you could benefit from these systems. The alternative is having engineers manually try to repair the state and restart execution in just the right place and that scales very badly. I have an application that needs to stand up a bunch of cloud infrastructure (a “workspace” in which users can do research) on the press of a button, and I want to make sure that the right infrastructure exists even if some deployment attempt is interrupted or if the upstream definition of a workspace changes. Every month there are dozens of network flukes or 5XX errors from remote endpoints that would otherwise leave these workspaces in a broken state and in need of manual repair. Instead, the system heals itself whenever the fault clears and I basically never have to look at the system (I periodically check the error logs, however, to confirm that the system is actually recovering from faults—I worry that the system has caught fire and there’s actually some bug in the alerting system that is keeping things quiet).

whinvik 11mo ago

Sorry for the off-topic but I have been lately seeing a lot of hype around durable execution. I still cannot figure out how this is any different than launching a workflow in something like Airflow. Is the novel thing here that it can be done using the same DB you already have running?

adamzwasserman 11mo ago

Reminds me of IBM TPF (Transaction Processing Facility) - the system that powered airline reservations for decades. TPF used per-transaction logging with restart/recovery semantics at massive scale. You could literally unplug the power mid-transaction, plug it back in, and resume exactly where you left off. The embedded database approach here is interesting though - low latency, no network calls, perfect for single-agent workflows. TPF assumed massive concurrent load across distributed terminals. Different problems, similar durability patterns.

9642370096647 11mo ago

This is irrelevant for local LLMs where a seed value may be specified and generation is totally deterministic. This only helps with online LLMs.

throwaway290 11mo ago

> A workflow engine running a BPMN job Does anyone really do this?

gunnarmorling 11mo ago
Sure, systems like Activiti or Camunda come to mind.
- throwaway290 11mo ago
  my question is does anyone (here on HN is implied) really do this? in production for some serious purpose?
  - gunnarmorling 11mo ago
    Camunda (not affiliated btw.) crossed 100M ARR last year. So yes, this apparently is being used. Whether these users hang out here on HN or not, that I don't know.
  - tauchunfall 11mo ago
    I worked on a product that used BPMN where users could define processes. The company I worked for used Java for decades already. Clients of the product were banks. The people I worked with were not specifically HN audience. Rather in the Java bubble in Germany-Austria-Switzerland which is also surprisingly a small world. If BPMN is not really needed, then I would also not use it nowadays. It increases complexity, and who knows if it makes project communication better at all. Update: On the Camunda website there are 60 case-studies of customers/clients using BPMN, https://camunda.com/case-studies/ https://camunda.com/case-studies/. One of them has the teaser: "The 10th largest US Bank created an omnichannel onboarding platform that handles 12m process instances per year across 100 workflows". Now I have something to read for this Sunday evening.
    - throwaway290 11mo ago
      Thanks. I guess I'm in my bubble...

RagnarD 11mo ago

Serious question: How does "Durable Execution" differ from "Atomic Transaction"? At most, it seems that DE refers to more concrete details around implementing Atomic Transactions.

athrowaway3z 11mo ago
Maybe its about tools for inspecting the trace of a running atomic transaction? Not really sure because "Lately, there has been a lot of excitement around Durable Execution (DE) engines" is also news to me, and AFAICT it seems to be yet-another-reinvention CS is famous for.
gunnarmorling 11mo ago
It depends on what exactly you mean by "Atomic Transaction". If you are referring to database transactions, then DE is not that. But DE helps you implement potentially long-running multi-step business transactions, a.k.a. workflows, sometimes spanning over days or weeks. Oftentimes, these transactions are atomic, but don't satisfy the isolation requirement of ACID transactions, i.e. partial outcomes of in-flight flows may be visible to external observers.