Fallback Routing Logic for FSMA 204 Trace Gaps

Food traceability pipelines operating under FSMA 204 face a deterministic reality: upstream data sources will fail, KDE payloads will arrive malformed, and lot-level chain-of-custody links will fracture during peak harvest or distribution cycles. Regulatory compliance cannot pause for network latency, vendor API degradation, or intermittent EDI transmission errors. A production-grade fallback routing layer intercepts ingestion failures, preserves trace continuity, and routes incomplete or delayed records through deterministic exception paths without violating the FDA’s 24-hour record retrieval mandate. This pattern converts unpredictable supply chain noise into auditable, recoverable data streams.

Problem Statement: Where Trace Gaps Originate

Under 21 CFR Part 1, Subpart S, every Critical Tracking Event (CTE) obligates a regulated facility to capture a complete set of Key Data Elements (KDEs). The obligation is per-event and non-negotiable: a Shipping event with a null traceability_lot_code, a Transformation event whose event_timestamp arrives in a supplier’s local timezone, or a Receiving event that never reaches the ledger because a vendor API returned a 503 all produce the same regulatory outcome — a broken lot chain that cannot be reconstructed on demand. When an outbreak investigation lands and the FDA requests sortable electronic records within 24 hours, a single unrecovered gap can escalate a scoped, lot-level recall into an indefensible, brand-wide event.

The specific engineering problem this page solves is trace-gap recovery: what the ingestion pipeline does at the exact moment a primary source fails to deliver a compliant KDE payload. The naive answer — retry, then drop — is a compliance defect, because a silently dropped CTE is indistinguishable at audit time from a CTE that never happened. Fallback routing replaces that failure mode with a deterministic decision tree: attempt primary ingestion, exhaust a bounded retry budget, pivot to secondary sourcing, reconcile against the mandatory KDE set, and — only when reconciliation genuinely fails — isolate the record in a compliance-safe quarantine rather than discarding it. Every branch is logged, hashed, and provenance-tagged so the data lineage remains reconstructable.

This behavior is a specialization of the ingestion gateway defined in the parent FSMA 204 Architecture & KDE Compliance Mapping reference. Where the parent establishes the four-layer flow from event capture to FDA-ready export, this page owns the exception-handling control plane inside the Validation & Normalization layer. The full decision matrix for classifying gap severity by product risk tier and vendor reliability is expanded in Building fallback routing for trace gaps.

Routing Architecture

Resilient ingestion requires a strict, state-aware routing hierarchy. The system first validates the raw payload against the mandatory KDE schema. A structurally invalid payload never enters the retry loop — it is quarantined immediately, because retrying an unparseable record wastes the retry budget and delays the human review the record actually needs. A schema-valid payload attempts primary API ingestion under a bounded, exponentially backed-off retry budget. If the primary source remains unreachable after the budget is exhausted, the router pivots to secondary channels: EDI interchange files, supplier portal exports, or pre-staged CSV manifests sourced through the same Supplier Data Ingestion pipeline that feeds the primary path. When a secondary channel yields partial data, the system reconciles it against the canonical field definitions in the KDE Field Mapping Guide to determine whether the mandatory KDEs are now satisfiable.

Records tied to foods on the Food Traceability List (FTL) bypass long backoff windows and immediately trigger secondary sourcing, because their compliance criticality does not tolerate extended latency. Non-FTL commodities may tolerate longer backoff, but the routing topology stays identical to prevent fragmented audit trails. The routing engine must also enforce idempotency: during network flapping or retry storms, the same CTE can be resubmitted repeatedly, and a deterministic idempotency key (derived from the supplier transaction ID, lot code, and event type) ensures duplicate transmissions resolve to a single ledger record rather than inflating lot quantities.

KDE Data Contract for Routing Decisions

Every routing decision is made against a fixed data contract. Before the router evaluates fallback eligibility, the payload must be resolvable to the mandatory KDE set; the table below defines the fields the router treats as gate conditions, their expected types, the validation rule applied at the boundary, and the regulatory source that makes each field mandatory. Fields marked as internal are provenance metadata added by the router itself — they are not FDA-mandated KDEs but are retained alongside the record so the exception path is auditable.

The event_timestamp rule is the most common source of silent trace gaps: a timezone-naive value from a supplier WMS coerces to the ingestion server’s local time and shifts the recorded event by hours, breaking timestamp monotonicity across a lot’s CTE sequence. The router rejects naive timestamps outright rather than guessing an offset. The four fields above the divider are gate conditions — if any is unresolvable after secondary sourcing, the record is quarantined, never defaulted.

Production Implementation

The following implementation is a runnable fallback router built on pydantic v2 for contract enforcement, tenacity for bounded retry orchestration against the primary endpoint, and structured JSON logging for audit readiness. It routes each raw payload to exactly one of three terminal states — processed, secondary-recovered, or quarantined — and tags every non-primary record with explicit provenance.

import hashlib
import json
import logging
from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import Any

from pydantic import BaseModel, ConfigDict, Field, field_validator
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
    RetryError,
)

logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("fsma204.fallback_router")


def _audit(event: str, **fields: Any) -> None:
    """Emit one structured JSON audit line to stdout / SIEM."""
    logger.info(json.dumps({"event": event, "ts": datetime.now(timezone.utc).isoformat(), **fields}))


class CTEType(str, Enum):
    HARVESTING = "harvesting"
    COOLING = "cooling"
    INITIAL_PACKING = "initial_packing"
    SHIPPING = "shipping"
    RECEIVING = "receiving"
    TRANSFORMATION = "transformation"


class PrimarySourceError(Exception):
    """Raised when the primary ingestion endpoint is unreachable or returns 5xx."""


class KDEPayload(BaseModel):
    """Canonical, validated KDE record. Construction fails closed on any gate violation."""

    model_config = ConfigDict(extra="ignore")

    traceability_lot_code: str = Field(min_length=1)
    event_type: CTEType
    event_timestamp: datetime
    location_identifier: str = Field(min_length=1)
    product_description: str = Field(min_length=1)
    quantity_and_uom: str
    fallback_origin: str = "primary"
    audit_hash: str = ""

    @field_validator("event_timestamp")
    @classmethod
    def require_tz_aware_utc(cls, v: datetime) -> datetime:
        # Reject timezone-naive timestamps outright: coercion to server-local
        # time silently shifts the event and breaks CTE timestamp monotonicity.
        if v.tzinfo is None or v.utcoffset() is None:
            raise ValueError("event_timestamp must be timezone-aware")
        return v.astimezone(timezone.utc)

    def sealed(self) -> "KDEPayload":
        """Return a copy carrying a deterministic audit hash of its content."""
        body = self.model_dump(exclude={"audit_hash"}, mode="json")
        digest = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()[:12]
        return self.model_copy(update={"audit_hash": digest})


class QuarantineError(Exception):
    """Terminal routing outcome: the record could not be made compliant."""

    def __init__(self, reason: str, raw: dict[str, Any]) -> None:
        super().__init__(reason)
        self.reason = reason
        self.raw = raw


class FallbackRouter:
    def __init__(self, primary_client: Any, secondary_client: Any) -> None:
        self._primary = primary_client
        self._secondary = secondary_client
        self.quarantine: list[dict[str, Any]] = []

    @retry(
        retry=retry_if_exception_type(PrimarySourceError),
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=1, max=8),
        reraise=True,
    )
    def _ingest_primary(self, payload: dict[str, Any]) -> dict[str, Any]:
        """Bounded, exponentially backed-off attempt against the primary endpoint."""
        return self._primary.submit(payload)  # raises PrimarySourceError on 5xx / timeout

    def _quarantine(self, raw: dict[str, Any], reason: str) -> None:
        record = {"reason": reason, "raw": raw, "quarantined_at": datetime.now(timezone.utc).isoformat()}
        self.quarantine.append(record)
        _audit("record_quarantined", reason=reason, lot=raw.get("traceability_lot_code", "UNKNOWN"))

    def route(self, raw: dict[str, Any]) -> KDEPayload:
        """Route one raw payload to a single terminal state. Never drops silently."""
        lot = raw.get("traceability_lot_code", "UNKNOWN")

        # Gate 1: structural validation. Invalid records skip the retry budget entirely.
        try:
            KDEPayload.model_validate(raw)
        except Exception as exc:
            self._quarantine(raw, f"schema_rejection: {exc}")
            raise QuarantineError("schema_rejection", raw) from exc

        # Gate 2: primary ingestion under a bounded retry budget.
        try:
            confirmed = self._ingest_primary(raw)
            _audit("primary_ingest_success", lot=lot)
            return KDEPayload.model_validate(confirmed).sealed()
        except (RetryError, PrimarySourceError):
            _audit("primary_exhausted", lot=lot)

        # Gate 3: secondary sourcing + KDE reconciliation.
        recovered = self._secondary.fetch(lot)
        if recovered is None:
            self._quarantine(raw, "secondary_unavailable")
            raise QuarantineError("secondary_unavailable", raw)

        merged = {**raw, **recovered, "fallback_origin": "secondary_edi"}
        try:
            record = KDEPayload.model_validate(merged).sealed()
        except Exception as exc:
            # Secondary data still leaves a mandatory KDE unresolved -> isolate, don't guess.
            self._quarantine(merged, f"reconciliation_failed: {exc}")
            raise QuarantineError("reconciliation_failed", merged) from exc

        _audit("fallback_recovered", lot=lot, origin=record.fallback_origin, audit_hash=record.audit_hash)
        return record

The design decisions here are all compliance-relevant. Schema validation runs before the retry loop so unparseable payloads do not consume the retry budget. The tenacity decorator caps the primary attempt at three tries with exponential backoff (1s, 2s, 4s, capped at 8s), then reraise=True surfaces the failure to the fallback branch instead of swallowing it. Secondary data is merged over the raw payload — never the reverse — so a partial secondary record fills gaps without overwriting fields the primary source already supplied. And critically, when reconciliation still leaves a mandatory KDE missing, the router quarantines rather than synthesizing a value, because a fabricated traceability_lot_code is worse than an honest gap flagged for review.

Error Handling and Quarantine Strategy

Quarantine is a compliance-safe holding state, not a discard bin. Every quarantined record carries its original raw input, a machine-readable reason, and a timestamp, so an operator or an automated supplier-query workflow can resolve the gap and replay the record through route() once corrected. Because the raw payload is preserved verbatim, the quarantine store doubles as evidence: at audit time it demonstrates that no CTE was ever silently lost, only deferred.

The router distinguishes three quarantine reasons, each with a different remediation path. A schema_rejection means the payload is structurally wrong at the source and needs a supplier-side fix or a parser correction — this is the same failure surface handled upstream by Error Handling Workflows in the supplier ingestion pipeline. A secondary_unavailable reason means both channels are down simultaneously and the record should be replayed automatically once connectivity returns, often on the cadence set by the pipeline’s API Polling Strategies. A reconciliation_failed reason is the highest-priority signal, because it means a genuine data gap exists that neither channel can close — these records require human review and, frequently, a direct query back to the trading partner. Structured JSON audit lines let a monitoring layer alert on the ratio of reconciliation_failed events per supplier, which is an early indicator of a degrading data relationship well before it produces an FDA-visible gap.

Non-critical gaps — optional or descriptive fields that are not gate conditions — are the only fields ever filled with deterministic defaults, and each defaulted value is tagged fallback_origin: synthetic_default so downstream traceability queries can always distinguish source-verified data from system-generated placeholders. Mandatory KDEs are never defaulted.

Integration with the Parent Pillar

Fallback routing sits inside the Validation & Normalization layer of the parent architecture and hands its output to the immutable ledger exactly as the primary path does — the ledger cannot tell whether a record arrived via primary ingestion or secondary recovery, only that it is sealed with an audit_hash and carries an accurate fallback_origin. This is deliberate: the export layer that generates FDA submissions must treat every record uniformly, and the provenance tag lives inside the record rather than in a side channel that could drift out of sync.

Two integration seams matter most. First, the provenance and quarantine metadata this router produces must be retained for the full statutory window; the storage schema, cold-partition strategy, and expiration rules are defined in Data Retention Policies, and quarantine records inherit the same two-year retention as the CTEs they represent. Second, because the fallback path can pull from secondary supplier channels, it operates across the same trust boundary the parent architecture hardens through Security Boundaries for Trace Data — secondary sources must be authenticated to the same standard as the primary endpoint, or the fallback path becomes an unaudited injection point for unverified trace data.

Operational Notes

The router targets Python 3.10+ and depends on pydantic>=2.5 (for the v2 field_validator API and model_validate) and tenacity>=8.2. Both are pure-Python and add no system dependencies, which keeps the ingestion worker image small and portable across Celery workers, serverless functions, or a long-running consumer.

Configure the following as environment variables rather than hard-coded constants, so the same image can run against staging and production without a rebuild:

• FSMA_PRIMARY_MAX_ATTEMPTS — retry budget for the primary endpoint (default 3); FTL commodities should run at the low end to shorten time-to-fallback.
• FSMA_BACKOFF_MAX_SECONDS — ceiling for exponential backoff (default 8); keep the total retry window well inside your per-record SLA so the 24-hour export budget is never at risk.
• FSMA_QUARANTINE_ALERT_THRESHOLD — per-supplier reconciliation_failed rate that trips an operator alert.
• FSMA_IDEMPOTENCY_TTL_DAYS — how long the idempotency store retains keys to suppress duplicate CTE submissions during retry storms.

Emit the structured audit lines to stdout and let the platform’s log shipper forward them to a SIEM or centralized store; avoid writing audit state to local disk on ephemeral workers. When load-testing, exercise the fallback branch explicitly by injecting PrimarySourceError from a stub primary client — the branch that only runs during an outage is the branch most likely to be untested when a real outage arrives.

Common Failure Questions

Related

• Building fallback routing for trace gaps — the gap-severity decision matrix and routing thresholds by risk tier.
• KDE Field Mapping Guide — canonical field definitions the reconciliation step validates against.
• Data Retention Policies — retention and expiration rules for quarantine and provenance metadata.
• Security Boundaries for Trace Data — authenticating secondary sources across the trust boundary.
• Error Handling Workflows — upstream retry, quarantine, and dead-letter routing in the supplier pipeline.

Up: FSMA 204 Architecture & KDE Compliance Mapping