Backend Engineer Guide: Deterministic Finite State Machines & Saga Workflows in Microservices

When order fulfillment, KYC onboarding, or escrow payout states are mutated via scattered if status == 'PENDING': status = 'PAID' conditionals across multiple microservices, race conditions inevitably produce impossible states—such as an order transitioning from REFUNDED back to SHIPPED when a delayed webhook arrives. As a backend engineer designing financial and logistics workflows, I enforce deterministic Finite State Machines (FSM) backed by PostgreSQL optimistic concurrency control and immutable transition audit logs.

1. Explicit Transition Matrix and Guard Predicates

Every domain aggregate defines an explicit transition table mapping (current_state, event) -> next_state along with synchronous guard predicates and side-effect outbox events:

# workflows/escrow_fsm.py
from dataclasses import dataclass
from enum import StrEnum
from typing import Callable
from django.db import transaction

class EscrowState(StrEnum):
    CREATED = "CREATED"
    FUNDED = "FUNDED"
    IN_DELIVERY = "IN_DELIVERY"
    RELEASED = "RELEASED"
    DISPUTED = "DISPUTED"
    REFUNDED = "REFUNDED"

class EscrowEvent(StrEnum):
    PAYMENT_CONFIRMED = "PAYMENT_CONFIRMED"
    WORK_SUBMITTED = "WORK_SUBMITTED"
    BUYER_APPROVED = "BUYER_APPROVED"
    DISPUTE_OPENED = "DISPUTE_OPENED"
    ARBITRATION_REFUND = "ARBITRATION_REFUND"

TRANSITIONS: dict[tuple[EscrowState, EscrowEvent], EscrowState] = {
    (EscrowState.CREATED, EscrowEvent.PAYMENT_CONFIRMED): EscrowState.FUNDED,
    (EscrowState.FUNDED, EscrowEvent.WORK_SUBMITTED): EscrowState.IN_DELIVERY,
    (EscrowState.IN_DELIVERY, EscrowEvent.BUYER_APPROVED): EscrowState.RELEASED,
    (EscrowState.FUNDED, EscrowEvent.DISPUTE_OPENED): EscrowState.DISPUTED,
    (EscrowState.IN_DELIVERY, EscrowEvent.DISPUTE_OPENED): EscrowState.DISPUTED,
    (EscrowState.DISPUTED, EscrowEvent.ARBITRATION_REFUND): EscrowState.REFUNDED,
}

class InvalidStateTransitionError(Exception):
    pass

def apply_escrow_event(escrow_id: str, event: EscrowEvent, actor_id: str, idempotency_key: str) -> EscrowState:
    with transaction.atomic():
        from payments.models import EscrowOrder, EscrowTransitionLog, OutboxMessage

        # Idempotency check on transition key
        existing = EscrowTransitionLog.objects.filter(idempotency_key=idempotency_key).first()
        if existing:
            return EscrowState(existing.to_state)

        order = EscrowOrder.objects.select_for_update().get(id=escrow_id)
        current = EscrowState(order.state)
        next_state = TRANSITIONS.get((current, event))
        if next_state is None:
            raise InvalidStateTransitionError(
                f"Illegal transition from {current} on event {event} for escrow {escrow_id}"
            )

        order.state = next_state
        order.version += 1
        order.save(update_fields=["state", "version", "updated_at"])

        EscrowTransitionLog.objects.create(
            escrow_id=escrow_id,
            from_state=current,
            to_state=next_state,
            event=event,
            actor_id=actor_id,
            idempotency_key=idempotency_key,
        )
        OutboxMessage.objects.create(
            aggregate_id=escrow_id,
            event_type=f"escrow.{next_state.lower()}",
            payload={"escrow_id": escrow_id, "version": order.version},
        )
        return next_state

2. Saga Compensation for Multi-Service Workflows

When a state transition triggers downstream calls across separate microservice databases (e.g., reserving inventory, capturing payment, booking courier dispatch), every forward step must have an idempotent compensating action registered in the workflow coordinator so partial failures roll back cleanly without human DB intervention.

3. Related Distributed Systems Resources


About the Author — Hire Aldian Fazrihady (Cloud Engineer, GCP Professional Cloud Architect, GCP Professional Data Engineer, Backend, AI & Full Stack Engineer)

Aldian Fazrihady is a Top-Rated Senior Cloud Engineer, GCP Professional Cloud Architect (Google Cloud Certified), GCP Professional Data Engineer (Google Cloud Certified), Backend Engineer (Python, Django, FastAPI, Go, PostgreSQL), AI Application Engineer (LLMs, RAG, MCP, Google ADK, LangGraph), and Full Stack Engineer (React, Next.js, TypeScript) with 15+ years of production software engineering experience.

Hire Aldian Fazrihady on Upwork | Book 1-Day Programming Service ($100 USD / Rp 1.000.000) | View Live AI Portfolio | Send a Direct Message

Related Engineering Articles

Engineering Role Pillar Guides & Category Hubs