Durable Workflows for Software Architects: Crash-Proof Execution and the Best FOSS Engines, Hands-On

If you have been following the automation conversation on this blog, you already know the pattern: the n8n review settled what a self-hosted automation tool looks like on one box, the Compose vs Kubernetes post drew the line for when a cluster earns its keep, and the Docker secrets post made the case that every layer only earns its keep when it matches a real threat model. What none of them answered is the question that sits underneath all of them: what does it actually take to make a workflow survive? Not survive a typo, survive a process crash, a machine restart, a deploy, a network failure, or a wait measured in days.

That property has a name: durable execution. This post is the architect version of the field guide. I cover what durable execution actually is, the use cases that genuinely need it, and an honest, hands-on comparison of the best FOSS options: Temporal, Hatchet, Conductor OSS, Apache Airflow, and BullMQ, including the footprint question that decides which one you can actually self-host.

The short version

• Durable execution means the state of a running workflow survives failure, not just the queue.
• A job queue survives the worker dying; a durable workflow engine survives the worker dying, the machine rebooting, the code deploying, and the network partitioning, and resumes at the last completed step.
• The core mechanism is event history plus deterministic replay. The engine records every step, and when a worker comes back it re-runs the workflow code against that history instead of starting over.
• Temporal is the most mature and most complete, and it is also the heaviest to run. Self-hosted it needs a server with several components plus a database.
• Hatchet is the modern lightweight challenger: a Go engine on PostgreSQL with no Redis required, durable tasks, and a small footprint.
• Conductor OSS is the JSON-workflow engine from Netflix: declarative workflows, great for orchestrating many services, heavier Java-based server.
• Apache Airflow is a scheduler with a workflow skin, built for batch data pipelines, not for long-running transactional workflows.
• BullMQ is not a workflow engine at all, it is a durable job queue for Node.js on Redis, and it has the smallest footprint of the five.
• The footprint ranking, roughly: BullMQ (Redis only), Hatchet (Postgres plus a Go binary), Conductor (Java server plus Postgres and Redis), Temporal (server cluster plus Postgres), Airflow (scheduler plus webserver plus workers plus Postgres).

What durable execution actually is

Start with the failure it is built for. You are processing an order: charge the card, reserve inventory, trigger fulfillment, send the receipt. The process dies after the charge and before the receipt. A job queue with an ack-based retry re-runs the whole thing from the top, which means the card gets charged twice unless your handlers are idempotent. A durable workflow engine instead records each completed step as an event. When the worker comes back, it replays the workflow function against that event history, skips the steps that already completed, and resumes at the one that did not.

That is the whole idea, and everything else is engineering detail around it. Temporal calls the mechanism event history and replay. Hatchet calls the same shape durable tasks with cached results replayed on retry. Conductor persists the workflow state as a graph of task executions. All three share the same core promise: execution state survives, not just the message in the queue.

The price of that promise is determinism. A replayed workflow must produce the same decisions when it re-runs, which means the workflow code cannot call the network, read the clock, or use randomness directly. Side effects, API calls, database writes, waits, go into separate steps the engine treats as external events. That discipline is why the code samples below all look the same: the workflow function is a thin, boring script, and the real work lives in activities or tasks.

The two families: engines and schedulers

It helps to split the five tools into two families, because the comparison otherwise goes in circles.

The durable execution engines: Temporal, Hatchet, and Conductor. These are built around the event-history model. They handle retries with backoff, long waits, human approval steps, and sagas, and they resume mid-flight after crashes and deploys. This is the family that matches the definition at the top of this post.

The schedulers and queues: Airflow and BullMQ. Airflow schedules DAGs of Python tasks and tracks task state in its metadata database, which gives it real durability for batch jobs, but it is not built for long-running interactive workflows. BullMQ is a queue: jobs are durable in Redis, a worker picks one up, completes it, and moves on. Chaining jobs into a workflow is on you, and a mid-workflow crash restarts from the chain you wrote, not from a recorded event history.

Neither family is better in the abstract. They answer different questions, and the use cases section is where the split shows up.

Use cases that actually need durable workflows

• Money movement. Charge, refund, payout, reconciliation. You need at-most-once side effects and resumability, and you need it audited. This is the canonical saga use case.
• Order and subscription lifecycle. Create, provision, bill, welcome email, then every renewal and dunning step for months. The workflow is a state machine that outlives any process.
• Human approval inside automation. Submit, wait for a person, resume. The wait can be hours or weeks, and the engine parks the workflow without holding compute, which the event-history model does naturally.
• Long waits that must survive redeploys. A 30 day trial expiry, a 24 hour delayed retry, a scheduled follow-up. Cron fires and forgets; a durable timer fires after the deploy that happened in between.
• Multi-service orchestration with compensation. If step 7 fails, undo steps 1 through 6 in reverse order. Doing that by hand in application code is where sagas become unmaintainable.
• AI agent and LLM pipelines. Multi-step prompts, tool calls, retries, and long-running reasoning loops where the same crash-survival argument applies.
• Cron on steroids. Same trigger, but with retries, backoff, and the guarantee that a missed run is picked up instead of silently skipped.

What does not need a durable workflow engine: a simple fire-and-forget background job, a fan-out that a message queue handles fine, a nightly batch that you are comfortable re-running. Adding Temporal to that is the platform-for-its-own-sake mistake this blog keeps warning about.

The FOSS options, one by one

Temporal

Temporal is the reference implementation of durable execution. The server is written in Go and MIT-licensed, and it is split into several services: frontend, history, matching, and worker, plus the optional visibility store. Workflows are code in Go, Java, TypeScript, Python, .NET, and Ruby, and the SDKs handle the deterministic-replay contract for you. Activities are where the real work happens, and the engine retries them with configurable policies.

// TypeScript, @temporalio/client and @temporalio/workflow
import { proxyActivities, sleep } from '@temporalio/workflow';

const { chargeCard, sendReceipt } = proxyActivities({
  retry: { maximumAttempts: 5 },
});

export async function orderWorkflow(orderId: string) {
  await chargeCard(orderId); // activity: side effects live here
  await sleep('24h');        // durable timer, survives crashes and deploys
  await sendReceipt(orderId);
}

The strengths: the most complete feature set, the largest ecosystem, the clearest documentation, and the strongest guarantees around retries, timers, signals, and child workflows. The honest cost: it is the heaviest to operate. Self-hosted Temporal means running the server services, a persistence backend (PostgreSQL, MySQL, or Cassandra, with SQLite for local dev), and optionally Elasticsearch for advanced visibility. That is real infrastructure, and this blog's Compose vs Kubernetes framing applies: Temporal is a platform, and it earns its keep when the workflows are worth a platform. Postiz moved to using Temporal, if I remember well.

Hatchet

Hatchet is the modern challenger that answers the operational weight of Temporal directly. The engine is written in Go, the project is MIT-licensed, and the v1 rewrite made PostgreSQL the single source of truth, no Redis required. Workers are separate processes in Go, Python, TypeScript, or Ruby that register tasks with the engine. The durable task model caches intermediate results, so a retry replays what already ran instead of re-running it, and durable sleep and durable events cover the waiting cases.

# Python, hatchet-sdk
from hatchet_sdk import Hatchet

hatchet = Hatchet()

@hatchet.task(retries=5)
def charge_payment(input: dict) -> dict:
    # runs at most once per input, cached and replayed on retry
    return charge_card(input["order_id"])

Where Temporal is a suite of Go services plus a database, Hatchet is a single Go binary plus PostgreSQL. That footprint difference is the whole story for a homelab or a small team. The trade-offs: a younger ecosystem, fewer SDK languages than Temporal, and less of the battle-tested, we-have-run-this-at-scale history. For greenfield self-hosted durable execution in 2026, it is the option to try first.

Conductor OSS

Conductor is the workflow engine Netflix built and later open-sourced under Apache 2.0, now maintained by the Conductor OSS project with Orkes offering the managed version. Its defining trait: workflows are JSON definitions, not code. Tasks are either system tasks (HTTP, wait, fork, join) or worker tasks that external services poll and complete. The server is a Java/Spring application that needs Redis, PostgreSQL, MySQL, or Cassandra, and optionally Elasticsearch layered on for indexing.

{
  "name": "media_pipeline",
  "version": 1,
  "tasks": [
    { "name": "download_source", "taskReferenceName": "dl", "type": "SIMPLE" },
    { "name": "transcode", "taskReferenceName": "tx", "type": "SIMPLE",
      "inputParameters": { "source": "${dl.output.path}" } },
    { "name": "publish", "taskReferenceName": "pub", "type": "SIMPLE" }
  ]
}

The strengths: declarative workflows that non-engineers can read, a proven record at Netflix scale, event-driven execution, and task workers in any language. The honest cost: JSON workflows are less flexible than code for complex logic, the JVM server plus Postgres plus Redis is a heavier footprint than Hatchet, and the ecosystem momentum has shifted to the code-first engines in recent years.

Apache Airflow

Airflow is the default answer for batch data pipelines, and it is Apache 2.0. DAGs are Python code that declare dependencies between tasks, the scheduler turns them into task instances, and the workers execute them. Airflow 3.x is current, with 3.0 going GA in 2025 and 3.3.1 released in August 2026. The architecture is scheduler, webserver, and workers, backed by a metadata database (PostgreSQL in practice), with the Celery executor optionally adding a broker such as Redis or RabbitMQ.

# dags/nightly_sync.py
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG("nightly_sync", schedule="0 2 * * *", start_date=datetime(2026, 1, 1), catchup=False) as dag:
    extract = PythonOperator(task_id="extract", python_callable=extract_data)
    load = PythonOperator(task_id="load", python_callable=load_data)
    extract >> load

The strengths: unmatched for scheduled data pipelines, huge operator ecosystem, and the most mature DAG semantics in the batch world. The honest cost: it is a scheduler, not a durable execution engine. Task state is durable in the metadata DB, but the DAG itself does not replay like a workflow, long-running interactive workflows are a poor fit, and the full stack (scheduler, webserver, workers, Postgres, optional Celery broker) is one of the heaviest on this list to run for what it gives a workflow developer.

BullMQ

BullMQ is the honest lightweight answer for the middle ground. It is a Redis-based job queue for Node.js (with Python, Rust, Elixir, and PHP ports), MIT-licensed, and it has no server of its own: it is a library. Jobs are durable in Redis, workers process them, retries and delays are built in, and repeatable jobs take cron expressions. It is the smallest footprint of the five, literally a Redis instance plus your Node app.

// Node.js
import { Queue, Worker } from 'bullmq';

const queue = new Queue('payments', { connection: { url: process.env.REDIS_URL } });
await queue.add('charge', { orderId: '123' }, { attempts: 5, backoff: { type: 'exponential' } });

new Worker('payments', async (job) => chargeCard(job.data), { connection: { url: process.env.REDIS_URL } });

What BullMQ is not: a workflow engine. It does not record an event history, it does not replay a workflow function, and chaining steps into a saga is code you write. What it is: the right tool when your need is durable background jobs with retries and a cron schedule, and the weight of Temporal would be the mistake. Its durability is exactly Redis's durability, so Redis persistence (AOF and replication) is a real operational requirement, not an option.

Hands-on: standing each one up

# Temporal: server plus Postgres, or the auto-setup image for local dev
$ docker run -d --name temporal -p 7233:7233 temporalio/auto-setup:1.24

# Hatchet: quickstart stack, Postgres is the only state store
$ docker compose up -d   # from the hatchet quickstart

# Conductor OSS: server plus Postgres plus Redis
$ docker compose up -d   # from conductor-oss

# Airflow: official docker-compose, scheduler + webserver + workers + Postgres
$ docker compose up airflow-init && docker compose up -d

# BullMQ: no server to run, just Redis plus your Node app
$ npm install bullmq

Footprint and self-hosting

Two honest notes on the footprint table. First, the ranking is about what you operate, not raw memory: a Go binary uses less RAM than a JVM, but the bigger cost is the number of moving parts you patch and back up, which is exactly the release-tracking discipline this blog applies to everything it runs. Second, BullMQ's tiny footprint is real only if you treat Redis as durable infrastructure, which means AOF enabled, replication configured, and backups tested, the same backup rules that apply to Postgres.

The honest limits

• Determinism is a real tax. Temporal-style workflows cannot call the network or read the clock inline, and new developers constantly trip on this. The SDKs enforce it at runtime, which is good, and confusing at first.
• At-most-once side effects are your job. Durable execution prevents double-runs of completed steps by replay, but a step interrupted mid-execution can still run twice. Make activities idempotent or gate them with a human approval, exactly like the Docker secrets post says about side effects.
• Event history grows. Every step is recorded, and very long workflows accumulate history, which costs storage and replay time. Temporal has history compaction, but the discipline of short workflows with clear activities is the real fix.
• Airflow and BullMQ are the wrong tools for long-running transactional workflows. Choosing them for a saga is choosing the queue because it is light, then writing the workflow engine yourself, badly.
• Conductor's JSON model is a feature and a ceiling. It is great for readable, declarative orchestration and awkward for complex branching logic that code expresses naturally.
• None of these tools make your application idempotent. They make the workflow resumable; the application still has to survive a step running twice, and that is a separate design requirement.

Which should you pick?

• Choose Temporal when the workflow is mission-critical, the team is comfortable operating a platform, and you need the deepest feature set: child workflows, signals, updates, and the largest ecosystem. Pay the operations cost knowingly.
• Choose Hatchet when you want durable execution on your own hardware with the smallest engine footprint. One Go binary, PostgreSQL, real retries and durable sleep. This is the default I would reach for on a self-hosted stack in 2026.
• Choose Conductor OSS when you want declarative JSON workflows, your team is JVM-shaped, or you are orchestrating many heterogeneous services and want the workflow definition readable by non-engineers.
• Choose Airflow when the workload is scheduled data pipelines, period. It is the right tool for batch ETL and the wrong tool for transactional workflows.
• Choose BullMQ when the need is durable background jobs with retries and cron inside an existing Node application, and adding a platform would be the mistake. It is the smallest footprint on this list and the honest answer for most web backends.
• Consider none of them when a plain queue, cron, or the n8n-style automation tool already covers the job. The release-tracking workflow and the Compose vs Kubernetes reasoning both apply: a durable workflow engine is a platform, and a platform earns its keep when the workflows are worth it.

My honest read after running a few of these: durable execution is one of the few genuinely new primitives in backend architecture in the last decade, and the FOSS options are now good enough that the decision is about footprint and fit, not capability.

Hatchet is the one I would bet on for a self-hosted small-to-medium stack, Temporal when the organization is already running a platform, Airflow and BullMQ for the batch and queue jobs they were built for, and Conductor when JSON workflows and JVM operations are the home team. Measure the moving parts, pin the versions, and test the restart-mid-workflow behavior before you trust it with money.

Official sources

• Temporal: https://temporal.io/ and https://github.com/temporalio/temporal
• Temporal changelog: https://temporal.io/changelog
• Hatchet: https://hatchet.run/ and https://github.com/hatchet-dev/hatchet
• Hatchet architecture and guarantees: https://docs.hatchet.run/v1/architecture-and-guarantees
• Conductor OSS: https://conductor-oss.org/ and https://github.com/conductor-oss/conductor
• Apache Airflow: https://airflow.apache.org/ and https://endoflife.date/apache-airflow
• BullMQ: https://bullmq.io/ and https://docs.bullmq.io/
• Our Compose vs Kubernetes post: https://systhoughts.com/posts/docker-compose-vs-kubernetes-self-hosted-apps
• Our release-tracking workflow: https://systhoughts.com/posts/tracking-software-releases-across-forges
• Our Docker secrets post: https://systhoughts.com/posts/docker-secrets-arent-really-secrets

Are you running a durable workflow engine yet, or still chaining queue jobs and hoping? Which one earned its keep on your stack, and where did the footprint bite? Drop it in the comments.

Until next time, keep your systems thoughtful.