Building Fallback Routing for FSMA 204 Trace Gaps by Risk Tier

The failure this page fixes is specific: a primary KDE source times out or returns a 503 in the middle of ingesting a shipping_event, and the fallback logic behind it makes the gap worse instead of closing it. Two anti-patterns dominate production incidents. The first is an undifferentiated fallback that treats a Food Traceability List (FTL) romaine shipment exactly like a low-risk commodity — it burns the same long exponential-backoff window on both, and the FTL record blows through the FDA’s 24-hour retrieval budget while the router is still politely waiting on a dead endpoint. The second is a fallback that iterates its secondary sources in whatever order a Python dict happens to yield, so the same missing traceability_lot_code gets patched from the ERP on one run and from an IoT telemetry log on the next. Both records look “recovered,” but their provenance is non-deterministic, and at audit time you cannot prove which authority actually supplied the field.

Deterministic fallback routing removes both defects. It classifies every gap by product risk tier before it chooses a backoff budget, walks secondary sources in a fixed authority order, and tags each recovered field with the exact system that resolved it. This is the concrete, risk-tiered specialization of the exception control plane defined in the parent Fallback Routing Logic guide.

Root Cause: Why Trace Gaps Recover Non-Deterministically

Under 21 CFR Part 1, Subpart S, every Critical Tracking Event must carry a complete set of Key Data Elements, and the recovery of a missing element is only defensible if it is reproducible. Three upstream realities make naive fallback non-deterministic:

1. Unordered source resolution. When the router asks “who can supply ship_to_location?” and iterates a mapping of candidate systems, insertion order, hash seeding, or a config reload can change which system answers first. The ERP and the WMS may both hold the field with slightly different values (a corrected address versus a shipped-to-dock code), so the order is the compliance decision, not an implementation detail.
2. Uniform latency budgets. A single global retry budget ignores that an FTL commodity has near-zero tolerance for fallback latency while a non-FTL item can absorb minutes. Applying the FTL budget to everything wastes capacity; applying the non-FTL budget to everything risks the 24-hour mandate on exactly the records that matter most.
3. Silent drops on exhaustion. When every source fails, a naive router logs an error and returns, discarding the event. A discarded Critical Tracking Event is indistinguishable at audit time from an event that never happened — the worst possible outcome under a Subpart S traceback.

The fix places the authority order and the risk tier in configuration the router reads deterministically, and it replaces the silent drop with an explicit manual-review terminal state. The canonical definition of each field the router resolves against lives in the KDE Field Mapping Guide; the secondary channels it queries are the same feeds established in the Supplier Data Ingestion pipeline.

The table below is the routing contract: for each gate KDE, it fixes the ordered list of secondary authorities the router is permitted to consult and the regulatory source that makes the field mandatory. Authority order is read left to right and never varies at runtime.

Diagnosing the Gap Before Routing

Trace gaps manifest as null KDEs, mismatched temporal sequences, or broken lot_code lineage. Before routing to any fallback source, the ingestion layer must isolate the exact failure vector — retrying an unparseable record or fabricating a field both fail the audit. The diagnostic routine below performs structural validation, temporal consistency checks, and KDE completeness scoring, emitting a structured gap report that the router consumes. It cross-references incoming payloads against the mandatory field set so that a shipping_event or transformation_event missing traceability_lot_code, product_description, or location_identifier is flagged before any recovery attempt.

import datetime
import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Any

# Structured logging feeds the compliance audit trail directly.
logger = logging.getLogger("fsma204.trace_router")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(asctime)s | %(levelname)s | %(name)s | %(message)s"))
logger.addHandler(handler)


class KDEStatus(Enum):
    COMPLETE = "complete"
    PARTIAL = "partial"
    MISSING = "missing"


class RiskTier(Enum):
    FTL = "ftl"          # Food Traceability List — minimal fallback latency tolerated
    STANDARD = "standard"


@dataclass
class TraceEvent:
    event_id: str
    event_type: str
    timestamp: datetime.datetime
    kde_payload: dict[str, Any]
    source_system: str
    risk_tier: RiskTier = RiskTier.STANDARD


@dataclass
class GapReport:
    event_id: str
    missing_kdes: list[str]
    temporal_drift_ms: float
    status: KDEStatus
    risk_tier: RiskTier
    audit_notes: list[str] = field(default_factory=list)


REQUIRED_KDES: dict[str, list[str]] = {
    "shipping_event": [
        "traceability_lot_code", "ship_from_location",
        "ship_to_location", "product_description",
    ],
    "transformation_event": [
        "traceability_lot_code", "input_lot_codes",
        "output_lot_codes", "transformation_date",
    ],
    "receiving_event": [
        "traceability_lot_code", "received_from_location", "receipt_date",
    ],
}


def diagnose_trace_gap(event: TraceEvent, max_drift_ms: float = 5000.0) -> GapReport:
    required = REQUIRED_KDES.get(event.event_type, [])
    if not required:
        logger.warning("Unknown event_type '%s'. Skipping KDE validation.", event.event_type)
        return GapReport(event.event_id, [], 0.0, KDEStatus.COMPLETE, event.risk_tier)

    missing = [kde for kde in required if not event.kde_payload.get(kde)]

    # Temporal validation: large clock skew can indicate replay or misconfigured clocks,
    # both of which corrupt CTE timestamp monotonicity across a lot's event chain.
    now_utc = datetime.datetime.now(datetime.timezone.utc)
    drift = abs((now_utc - event.timestamp).total_seconds() * 1000)

    if not missing:
        status = KDEStatus.COMPLETE
    elif len(missing) < len(required):
        status = KDEStatus.PARTIAL
    else:
        status = KDEStatus.MISSING

    notes = [f"Source: {event.source_system}", f"Drift: {drift:.1f}ms", f"Tier: {event.risk_tier.value}"]
    if drift > max_drift_ms:
        notes.append(f"WARNING: temporal drift {drift:.1f}ms exceeds threshold {max_drift_ms:.1f}ms")
        logger.warning("Temporal drift exceeded | Event: %s | Drift: %.1fms", event.event_id, drift)

    report = GapReport(event.event_id, missing, drift, status, event.risk_tier, notes)
    if status != KDEStatus.COMPLETE:
        logger.info("Gap detected | Event: %s | Missing: %s | Status: %s",
                    event.event_id, missing, status.value)
    return report

Minimal Reproducible Example: The Non-Deterministic Recovery

The snippet below reproduces the core defect in isolation. A naive resolver holds each KDE’s candidate sources in a plain dict, iterates it, and takes the first non-None answer. Because two sources disagree on the value and the iteration order is not pinned to authority, the same gap resolves differently depending on which source object is queried first — and when both are down, the event is silently dropped.

def naive_resolve(missing_kde: str, sources: dict[str, dict[str, str]]) -> str | None:
    # BUG 1: iteration order is not authority order — ERP and WMS are interchangeable here.
    # BUG 2: on total failure this returns None and the caller drops the event.
    for _system, table in sources.items():
        value = table.get(missing_kde)
        if value is not None:
            return value
    return None


erp = {"ship_to_location": "GLN-0614141000012"}      # corrected dock GLN
wms = {"ship_to_location": "GLN-0614141999998"}       # stale staging GLN
resolved = naive_resolve("ship_to_location", {"wms": wms, "erp": erp})
# Returns the WMS value purely because 'wms' was inserted first — a provenance defect.
print(resolved)  # GLN-0614141999998  (should have been the ERP-authoritative value)

The record is patched, sealed, and shipped downstream carrying a lower-authority value with no marker that a higher authority was skipped. Nothing raises; the gap is “closed” incorrectly.

Fix Implementation: Authority-Ordered Routing With a Circuit Breaker

The fix pins the authority order, tiers the retry budget by risk, and replaces the silent drop with a manual-review terminal state. A lightweight circuit breaker prevents the router from hammering an already-degraded ERP or WMS endpoint during a partial outage.

Figure — KDE source-authority fallback routing:

Figure — Circuit breaker state transitions:

import time
from collections.abc import Callable
from enum import Enum
from typing import Any


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    def __init__(self, failure_threshold: int = 3, recovery_timeout: float = 60.0) -> None:
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.last_failure_time = 0.0
        self.state = CircuitState.CLOSED

    def call(self, func: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN
                logger.info("Circuit breaker transitioning to HALF_OPEN")
            else:
                raise RuntimeError("Circuit breaker OPEN. Fallback source unavailable.")
        try:
            result = func(*args, **kwargs)
            if self.state == CircuitState.HALF_OPEN:
                self.state = CircuitState.CLOSED
                self.failure_count = 0
                logger.info("Circuit breaker CLOSED after successful recovery")
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
                logger.error("Circuit breaker OPENED after %d failures", self.failure_count)
            raise


# Fixed authority order per KDE — read deterministically, never dict-iteration order.
SOURCE_AUTHORITY: dict[str, list[str]] = {
    "traceability_lot_code": ["erp", "wms"],
    "ship_to_location": ["erp", "wms", "iot"],
    "ship_from_location": ["erp", "wms", "iot"],
    "product_description": ["erp"],
    "transformation_date": ["wms", "iot"],
}

# Retry budget scales by risk tier: FTL records fail over fast to protect the 24h mandate.
RETRY_BUDGET: dict[RiskTier, int] = {RiskTier.FTL: 1, RiskTier.STANDARD: 3}

_BREAKERS: dict[str, CircuitBreaker] = {
    "erp": CircuitBreaker(failure_threshold=2, recovery_timeout=30.0),
    "wms": CircuitBreaker(failure_threshold=3, recovery_timeout=45.0),
    "iot": CircuitBreaker(failure_threshold=5, recovery_timeout=15.0),
}


def resolve_kde_fallback(
    report: GapReport,
    payload: dict[str, Any],
    clients: dict[str, Callable[[str, str], str]],
) -> dict[str, Any]:
    """Resolve missing KDEs in fixed authority order under circuit-breaker protection.

    Every patched field is provenance-tagged. When all authorities are exhausted the
    record is routed to manual review, never silently dropped.
    """
    patched = payload.copy()
    lot = payload.get("traceability_lot_code", "UNKNOWN")
    resolved = 0

    for kde in report.missing_kdes:
        for system in SOURCE_AUTHORITY.get(kde, []):
            breaker, client = _BREAKERS.get(system), clients.get(system)
            if breaker is None or client is None:
                continue
            try:
                value = breaker.call(client, kde, lot)
                patched[kde] = value
                # Provenance is inline so downstream export can never lose it.
                patched.setdefault("_kde_provenance", {})[kde] = system
                report.audit_notes.append(f"Resolved '{kde}' via {system.upper()}")
                resolved += 1
                logger.info("Patched KDE '%s' via %s | Event: %s", kde, system, report.event_id)
                break  # first authority to answer wins — order is deterministic
            except Exception as exc:
                report.audit_notes.append(f"'{kde}' via {system}: {exc}")
                logger.warning("Fallback miss | KDE: %s | Source: %s | %s", kde, system, exc)

    if resolved == len(report.missing_kdes):
        report.status = KDEStatus.COMPLETE
        report.audit_notes.append("COMPLIANCE: all KDEs resolved via fallback routing")
    elif resolved:
        report.status = KDEStatus.PARTIAL
        report.audit_notes.append(f"COMPLIANCE: {resolved}/{len(report.missing_kdes)} KDEs resolved")
    else:
        report.status = KDEStatus.MISSING
        patched["_manual_review"] = True  # explicit terminal state, never a silent drop
        report.audit_notes.append("COMPLIANCE: fallback exhausted — routed to manual review queue")

    _ = RETRY_BUDGET[report.risk_tier]  # tier-scaled budget applied by the caller's retry wrapper
    return patched

Three decisions here are load-bearing for the audit. Authority order comes from SOURCE_AUTHORITY, a fixed table, so the ERP always outranks the WMS for traceability_lot_code regardless of dict iteration — the non-determinism from the reproduction is gone. Each patched field records its resolving system under _kde_provenance, so an FDA traceback can always distinguish an ERP-verified value from an IoT-inferred one. And exhaustion sets an explicit _manual_review flag rather than returning empty, converting a silent data loss into a tracked, SLA-bound remediation item. Because secondary channels sit across the same trust boundary the parent architecture hardens through Security Boundaries for Trace Data, each client must authenticate to the same standard as the primary endpoint before the router trusts its value.

Verification Steps

Confirm the router before pointing it at live sources. Drive it with the disagreeing ERP/WMS pair from the reproduction, an all-down outage, and a clean event, then check three independent signals.

1. Log output. Authority order and provenance must be visible in the audit line:

2026-07-02T09:14:02 | INFO | fsma204.trace_router | Patched KDE 'ship_to_location' via erp | Event: ship-7742
2026-07-02T09:14:02 | INFO | fsma204.trace_router | COMPLIANCE: all KDEs resolved via fallback routing

2. Unit assertions. Encode the invariants so a regression cannot ship silently:

def test_authority_order_prefers_erp_over_wms() -> None:
    report = GapReport("ship-7742", ["ship_to_location"], 0.0, KDEStatus.PARTIAL, RiskTier.FTL)
    clients = {
        "erp": lambda kde, lot: "GLN-0614141000012",   # authoritative
        "wms": lambda kde, lot: "GLN-0614141999998",   # stale
        "iot": lambda kde, lot: "GLN-IOT",
    }
    patched = resolve_kde_fallback(report, {"traceability_lot_code": "LOT-1"}, clients)
    assert patched["ship_to_location"] == "GLN-0614141000012"
    assert patched["_kde_provenance"]["ship_to_location"] == "erp"


def test_total_exhaustion_routes_to_manual_review() -> None:
    def down(kde: str, lot: str) -> str:
        raise ConnectionError("endpoint unreachable")
    report = GapReport("ship-9001", ["product_description"], 0.0, KDEStatus.MISSING, RiskTier.STANDARD)
    patched = resolve_kde_fallback(report, {"traceability_lot_code": "LOT-2"},
                                   {"erp": down, "wms": down, "iot": down})
    assert patched["_manual_review"] is True          # never silently dropped
    assert report.status is KDEStatus.MISSING

3. SQL state check. After a real run, no recovered record may reach the ledger without a provenance tag on every patched field. This query must return zero rows:

SELECT event_id, kde_name
FROM fsma204_recovered_kde
WHERE fallback_origin IS NULL
   OR fallback_origin NOT IN ('erp', 'wms', 'iot', 'primary');

Any row returned is a field that entered the ledger without a defensible source — the exact defect authority-ordered routing exists to prevent.

Related Edge Cases to Check Next

• Conflicting values across surviving authorities. When the ERP and WMS both answer but disagree, first-wins is correct only if the authority order encodes the real trust hierarchy. Log the discarded lower-authority value alongside the winner so a reconciliation review can catch a systematically wrong ERP feed before it corrupts a lot’s lineage.
• Circuit breaker stuck open during a long outage. If the ERP breaker trips and stays OPEN past a busy window, FTL records fail straight to manual review while the endpoint is actually recovering. Tune recovery_timeout against your real MTTR and confirm the half-open probe cadence lines up with the retry discipline in API Polling Strategies.
• Replayed events re-patching an already-recovered field. A retry storm can re-deliver an event whose gap was already closed. Guard with a deterministic idempotency key so a re-patched field does not overwrite a higher-authority value from the first pass — the same dead-letter discipline covered in Error Handling Workflows applies here.

Related

• Fallback Routing Logic — the parent exception control plane this risk-tiered router specializes.
• KDE Field Mapping Guide — canonical field definitions the resolver validates recovered values against.
• Data Retention Policies — retention rules for the provenance and manual-review metadata this router emits.
• Security Boundaries for Trace Data — authenticating secondary sources across the trust boundary.
• Error Handling Workflows — upstream retry, quarantine, and dead-letter routing in the supplier pipeline.

Up: Fallback Routing Logic — this how-to builds the risk-tiered resolver that sits inside that cluster’s Validation & Normalization layer.

Why order fallback sources by authority instead of by which responds fastest?

Because the value’s correctness, not its latency, is the compliance requirement. A WMS may answer a ship_to_location query faster than the ERP, but if it holds a stale staging GLN, the fast answer is the wrong answer. Authority order encodes the trust hierarchy so the router always prefers the system of record, and the circuit breaker — not the ordering — handles a slow or dead endpoint.

Should FTL and non-FTL records use different retry budgets?

Yes. Food Traceability List commodities have near-zero tolerance for fallback latency because the 24-hour retrieval mandate applies with full force, so they run a minimal retry budget and fail over to manual review fast. Standard commodities can absorb a longer budget. The routing topology stays identical for both to keep the audit trail uniform; only the budget scales by tier.

What happens to a record when every fallback source is down?

It is flagged _manual_review and routed to an SLA-tracked queue with its full gap report and audit notes — never dropped. A discarded Critical Tracking Event is indistinguishable at audit time from an event that never occurred, so the router preserves it as an explicit, tracked remediation item instead.

Which 21 CFR Part 1 subpart governs the KDEs this router recovers?

Subpart S. The Traceability Lot Code requirement is in 1.1320, the shipping and receiving KDE lists in 1.1340, and transformation KDEs in 1.1345. Every field in the source-authority table maps to one of those sections.