The workflow engine described in A Workflow Is Just a Database Table is not a novel idea in isolation – polling a database for work is as old as job queues. What makes it worth examining is how the specific design decisions stack up against the tools engineers actually reach for: Temporal, Apache Airflow, Prefect, and Oban. The comparison is honest on both sides.
The Contenders
Temporal – Originally Uber’s Cadence project, now its own company. Treats workflows as durable functions that survive crashes via event sourcing. The gold standard for long-running, stateful, fault-tolerant workflows in distributed systems. Its internal architecture has a Frontend Service, History Service, Matching Service, Worker Service, and Persistence Service – each independently scalable, each requiring its own deployment.
Apache Airflow – The dominant open-source workflow orchestrator. Defines workflows as Python DAGs (directed acyclic graphs). Requires a scheduler, workers, a metadata database, and typically a message broker like Redis or Celery. Massive ecosystem. Significant operational overhead.
Prefect – A modern Airflow alternative with a “negative engineering” philosophy: remove abstractions, let developers write code. Separates observation from execution – your code runs wherever you want, Prefect watches it. Cleaner than Airflow, still requires its own server or cloud subscription.
Oban – An Elixir job processing library backed by PostgreSQL (or SQLite). Uses a polling model for job queues, supports workflow composition with dependency chains, and is purpose-built for Elixir/Phoenix apps. Closest in spirit to the design being compared.
Simplicity
Temporal requires five distinct server processes to run a cluster. A local development setup involves Docker Compose with multiple containers. Adding a new workflow means understanding activities, workflow functions, task queues, and worker registration. The learning curve is steep by design – Temporal is solving a genuinely hard distributed systems problem, and it does not hide that.
Airflow requires a scheduler, a metadata database, a worker pool, and often a message broker. DAG definitions live in Python files that must be discovered by the scheduler. Operators, sensors, hooks, connections, variables, XComs – the mental model is large before you write your first meaningful workflow.
Prefect is simpler than Airflow but still requires a Prefect server or cloud account for scheduling, observability, and state persistence. Workflows are Python functions decorated with @flow and @task. The local path is clean; production requires infrastructure.
Oban is the simplest of the established tools. It runs inside your existing Elixir application, uses your existing database, and adds one dependency. Workflow composition is supported but requires understanding how job dependencies are held and scheduled.
The two-table design adds two SQLite tables and one GenServer to an already-running Elixir application. There is no separate server to deploy, no new infrastructure to operate, no SDK to learn. The entire engine is a poll loop and a message handler. A developer can read and fully understand the implementation in an afternoon.
Winner: the two-table design, closely followed by Oban. Both eliminate infrastructure overhead. The two-table design is more explicit about its mechanics.
Efficiency
Temporal handles extremely high throughput at scale. Its architecture is designed for tens of thousands of concurrent workflow executions across a cluster. The tradeoff is infrastructure cost – you’re running five services to get there.
Airflow is not particularly efficient. The scheduler polls the metadata database frequently. Workers pick up tasks, but the overhead of the Python interpreter, the scheduler loop, and the message broker adds latency. Fine for batch data pipelines. Not designed for low-latency event-driven workflows.
Prefect is faster than Airflow for orchestration overhead. Still subject to Python’s limitations and the round-trip to the Prefect server for state management.
Oban is efficient within Elixir. PostgreSQL LISTEN/NOTIFY eliminates polling latency for new jobs. Advisory locks prevent double-processing without row-level contention. At scale, the database becomes the bottleneck, but for most applications it’s more than adequate.
The two-table design uses SQLite with a single writer (the executor). One writer eliminates all concurrency complexity. Polling latency is in the order of seconds – fine for human-scale workflows and agent-driven tasks, not appropriate for microsecond job dispatch. Each step runs in a dedicated Elixir Task (lightweight process), so step parallelism is essentially free. The absence of network round-trips (SQLite is embedded, not a server) makes individual operations fast.
The efficiency ceiling is lower than Temporal or Oban at scale. SQLite is not a distributed database. But for the workload it targets – hundreds to low thousands of concurrent workflow steps – it is more than sufficient, and the overhead per operation is minimal.
Winner: Temporal at scale, Oban for mid-range Elixir apps, the two-table design for single-server embedded use. The two-table design is efficient for its target workload and honest about its ceiling.
Flexibility
Temporal is highly flexible for distributed systems – microservices, cross-service coordination, long-running sagas. Less flexible for simple event-driven automation because the operational cost is high relative to the value for small tasks.
Airflow is deeply flexible for data pipeline DAGs. Less flexible for anything that isn’t a batch ETL process. Human-in-the-loop workflows, approval gates, and event-driven triggers are awkward to model in DAG terms.
Prefect is more flexible than Airflow – dynamic task generation, event-driven triggers, and AI workflow support (via ControlFlow) are all supported. Still Python-centric.
Oban is flexible within job processing patterns. Workflow composition handles sequential and fan-out/fan-in. Not designed for approval gates, human-in-the-loop steps, or event-driven trigger models where the triggering event is external (a button press, a notification response).
The two-table design is flexible in a different direction. Because every trigger reduces to {:step_ready, step_id}, anything can participate in a workflow without being aware it’s doing so. An alarm, a button press, a completed task, an incoming message, a webhook – they all speak the same two-message protocol. Adding a new event source requires no changes to the workflow engine.
Approval gates are a first-class pattern, not a workaround. Human-in-the-loop is as natural as any other step. The workflow definition can express any DAG – sequential, branching, parallel, conditional – because branching is just the workflow returning a different next step spec.
Winner: tied between Temporal (for distributed systems) and the two-table design (for human-in-the-loop, event-driven, embedded use). They solve different problems.
Reliability
Temporal is the benchmark for reliability. Event sourcing means every state transition is logged. Workflows survive crashes, network partitions, and worker restarts by replaying history. Exactly-once execution is a design goal. This is why companies like Uber, Netflix, and Stripe use it.
Airflow is reliable for batch workflows but has known failure modes: the scheduler is a single point of failure in basic setups, zombie tasks can accumulate, and the metadata database becomes a bottleneck under load. Reliability at scale requires careful tuning.
Prefect is reliable for the workloads it targets. Handles retries, state persistence, and crash recovery. The Prefect Cloud offering provides more reliability guarantees than self-hosted.
Oban is reliable within its scope. PostgreSQL-backed jobs survive crashes. Advisory locks prevent double-processing. Retry policies, dead letter queues, and job discarding are built in. For an Elixir application that already runs PostgreSQL, Oban’s reliability properties are excellent.
The two-table design derives its reliability from a single principle: all state is in SQLite, and one process owns it. There are no partial writes from concurrent processes. Crash recovery is a poll loop that reads current state – no reconstruction, no replay, no coordination. Steps that were running when the server crashed are caught by a watchdog and retried or failed. Steps in pending wait safely.
The honest limitation: SQLite is a single-file database on one server. There is no replication, no failover, no geographic redundancy. If the server fails, the database is unavailable until it recovers. For a single-instance deployment this is acceptable; for a high-availability requirement it is a ceiling.
Winner: Temporal for distributed high-availability requirements. The two-table design and Oban are reliable within single-server constraints – and the two-table design is arguably simpler to reason about because its failure modes are fewer.
Elegance
This is the subjective dimension, but it matters because elegant systems are easier to maintain, extend, and debug.
Temporal’s event sourcing model is intellectually elegant – workflows are deterministic functions, state is a log of events, replay is the recovery mechanism. The elegance is at the concept level. At the implementation level, it requires determinism constraints on workflow code (no random numbers, no direct I/O), a distinction between workflow functions and activity functions, and careful handling of versioning when changing a deployed workflow. The implementation complexity is real.
Airflow’s DAG model is clean in theory. In practice, the gap between the conceptual DAG and the operational reality (operators, hooks, XComs, connections, sensors) erodes the elegance quickly. It was designed for a specific use case and stretched beyond it.
Prefect is cleaner than Airflow. The decorator model is ergonomic for Python developers. The negative engineering philosophy produces less boilerplate. Still, the separation between the orchestration layer (Prefect server) and the execution layer (your code) introduces a conceptual split that requires constant attention.
Oban is elegant for what it is. A job queue backed by a database, with dependency support added cleanly. No ceremony. The Elixir idioms feel natural. The limitation is that it’s a job queue extended toward workflows, and that ancestry shows in the API.
The two-table design achieves something specific: every component knows exactly one thing, interfaces are two messages wide, and the database is always consistent because one process owns it. There are no special cases in the executor – an approval gate is not a special case, a parallel step is not a special case, an alarm trigger is not a special case. They are all steps with a ready flag, and events that set it.
This is the definition of elegance in system design: a small number of simple rules that compose to produce complex behavior. Two tables. Two messages. One owner. The entire workflow system falls out of those constraints.
Winner: the two-table design, for eliminating special cases and achieving genuine composability from minimal primitives. Temporal wins on conceptual elegance at the distributed systems level.
The Honest Summary
| Dimension | Winner | Notes |
|---|---|---|
| Simplicity | Two-table design | No infrastructure, fully readable implementation |
| Efficiency | Temporal (scale) / Two-table (embedded) | Different ceilings, different targets |
| Flexibility | Tied | Temporal for distributed, two-table for event-driven human-in-the-loop |
| Reliability | Temporal (distributed) / Two-table (single server) | Two-table is reliable within its scope |
| Elegance | Two-table design | Fewest concepts, no special cases, maximum composability |
The two-table workflow design is not a replacement for Temporal or Airflow. It does not solve distributed systems coordination across microservices. It does not handle tens of thousands of concurrent workflows across a cluster. It does not provide a cloud-hosted observability dashboard.
What it does is solve the workflow problem for a single-server deployment with embedded SQLite – completely, reliably, and with less code and fewer concepts than any of the established alternatives. For the class of problems it targets (agent workflows, human-in-the-loop approval chains, event-driven automation, scheduled multi-step tasks), it is not a compromise. It is the right tool.
The measure of a good architecture is not how many features it has. It is how few concepts are required to understand it fully, and how naturally those concepts compose when requirements grow. By that measure, two tables and two messages is a strong result.