Backend Engineer Guide: Deterministic Finite State Machines & Saga Workflows in Microservices
When order fulfillment, KYC onboarding, or escrow payout states are mutated via scattered if status == 'PENDING': status = 'PAID' conditionals across multiple microservices, race conditions inevitably produce impossible states—such as an order transitioning from REFUNDED back to SHIPPED when a delayed webhook arrives. As a backend engineer designing financial and logistics workflows, I enforce deterministic Finite State Machines (FSM) backed by PostgreSQL optimistic concurrency control and immutable transition audit logs.
1. Explicit Transition Matrix and Guard Predicates
Every domain aggregate defines an explicit transition table mapping (current_state, event) -> next_state along with synchronous guard predicates and side-effect outbox events:
# workflows/escrow_fsm.py
from dataclasses import dataclass
from enum import StrEnum
from typing import Callable
from django.db import transaction
class EscrowState(StrEnum):
CREATED = "CREATED"
FUNDED = "FUNDED"
IN_DELIVERY = "IN_DELIVERY"
RELEASED = "RELEASED"
DISPUTED = "DISPUTED"
REFUNDED = "REFUNDED"
class EscrowEvent(StrEnum):
PAYMENT_CONFIRMED = "PAYMENT_CONFIRMED"
WORK_SUBMITTED = "WORK_SUBMITTED"
BUYER_APPROVED = "BUYER_APPROVED"
DISPUTE_OPENED = "DISPUTE_OPENED"
ARBITRATION_REFUND = "ARBITRATION_REFUND"
TRANSITIONS: dict[tuple[EscrowState, EscrowEvent], EscrowState] = {
(EscrowState.CREATED, EscrowEvent.PAYMENT_CONFIRMED): EscrowState.FUNDED,
(EscrowState.FUNDED, EscrowEvent.WORK_SUBMITTED): EscrowState.IN_DELIVERY,
(EscrowState.IN_DELIVERY, EscrowEvent.BUYER_APPROVED): EscrowState.RELEASED,
(EscrowState.FUNDED, EscrowEvent.DISPUTE_OPENED): EscrowState.DISPUTED,
(EscrowState.IN_DELIVERY, EscrowEvent.DISPUTE_OPENED): EscrowState.DISPUTED,
(EscrowState.DISPUTED, EscrowEvent.ARBITRATION_REFUND): EscrowState.REFUNDED,
}
class InvalidStateTransitionError(Exception):
pass
def apply_escrow_event(escrow_id: str, event: EscrowEvent, actor_id: str, idempotency_key: str) -> EscrowState:
with transaction.atomic():
from payments.models import EscrowOrder, EscrowTransitionLog, OutboxMessage
# Idempotency check on transition key
existing = EscrowTransitionLog.objects.filter(idempotency_key=idempotency_key).first()
if existing:
return EscrowState(existing.to_state)
order = EscrowOrder.objects.select_for_update().get(id=escrow_id)
current = EscrowState(order.state)
next_state = TRANSITIONS.get((current, event))
if next_state is None:
raise InvalidStateTransitionError(
f"Illegal transition from {current} on event {event} for escrow {escrow_id}"
)
order.state = next_state
order.version += 1
order.save(update_fields=["state", "version", "updated_at"])
EscrowTransitionLog.objects.create(
escrow_id=escrow_id,
from_state=current,
to_state=next_state,
event=event,
actor_id=actor_id,
idempotency_key=idempotency_key,
)
OutboxMessage.objects.create(
aggregate_id=escrow_id,
event_type=f"escrow.{next_state.lower()}",
payload={"escrow_id": escrow_id, "version": order.version},
)
return next_state
2. Saga Compensation for Multi-Service Workflows
When a state transition triggers downstream calls across separate microservice databases (e.g., reserving inventory, capturing payment, booking courier dispatch), every forward step must have an idempotent compensating action registered in the workflow coordinator so partial failures roll back cleanly without human DB intervention.