ReLab
Distributed Workflow Reliability & Replay
Sep 2026 – Present
Kill a worker after it charges a customer, then check whether the workflow recovers without charging twice. ReLab makes that failure observable and testable across real processes.
Real worker crashes · Deterministic replay · 160+ tests
- Go workers coordinate through PostgreSQL, with each state change and its append-only event written in one transaction. A pure reducer reconstructs run state from the journal alone.
- Lease-based recovery requeues abandoned tasks. An idempotency ledger suppresses effects already recorded; delivery is at least once, and a crash between an external effect and its ledger write can still duplicate it.
- 160+ unit and integration tests cover simulated failures, alongside process-level SIGKILL tests. Five fault types exercise worker crashes, duplicate delivery, latency, HTTP errors and database disconnects.
Distributed systemsBackend & APIs
GoPostgreSQLNext.jsReactTypeScriptDockerOpenTelemetry