Recall Simulation & FDA 24-Hour Response Automation Architecture

When an outbreak investigation escalates, the FDA does not ask a regulated facility to describe its records — it asks for them. Under FSMA 204 (21 CFR Part 1, Subpart S), a facility that manufactures, processes, packs, or holds a food on the Food Traceability List must produce an electronic, sortable spreadsheet of the relevant traceability information within 24 hours of the request. That obligation is not a data-capture deadline; it is a query-and-export deadline. The Key Data Elements already exist in the ledger, but assembling a defensible, lot-scoped chain across millions of Critical Tracking Events under active time pressure is where unprepared programs fail. When the traceback query cannot resolve the specific lots implicated by a contamination signal, the only safe action left is an over-broad, brand-wide recall — pulling every SKU and every date code because the system cannot prove which ones are clean. That is the concrete, expensive risk this page exists to eliminate.

This page is the reference for recall readiness across the traceability program. It defines the recall-response pipeline that turns a contamination signal into an FDA-ready submission, the data contract a recall query must resolve, the production engine that walks the immutable ledger to assemble affected lots, and the failure modes that quietly break a traceback the day it matters most. It reads from the immutable KDE ledger built by the FSMA 204 architecture reference and depends on the clean, normalized inputs produced by Supplier Data Ingestion. The specialized guides it links to — Lot-Level Recall Scoping, One-Up, One-Back Chain Reconstruction, FDA 24-Hour Response Automation, and Mock Recall Drills — each expand one stage of the pipeline defined here.

Recall-Response System Architecture

A recall-response system is a read-side companion to the append-only ingestion pipeline. It never writes traceability data; it reads the immutable KDE ledger, resolves a scope, reconstructs a chain, and produces an export. Keeping the recall path strictly read-only is a deliberate architectural choice: an investigation must never mutate the evidence it is querying, and separating the read model from the write model lets the response engine be tuned for graph traversal and export throughput without competing with ingestion for write locks.

The pipeline has five stages, each with a single responsibility and an explicit contract with the stage downstream:

1. Trigger. A recall begins with a signal — an epidemiological signal the FDA is investigating, an internal positive test result, a supplier notification, or a scheduled drill. The trigger carries the minimum facts known at that moment: a product, an approximate date range, and sometimes a single confirmed Traceability Lot Code. The engine treats every trigger identically, whether it is a real outbreak or a rehearsal, so the mock-drill path exercises exactly the same code as a live event.
2. Lot-scope resolver. This stage translates a fuzzy signal into a precise, defensible set of Traceability Lot Codes. Given a confirmed lot or a product-and-window predicate, it identifies exactly which lots are implicated and — just as importantly — which are provably not. Precise scoping is what keeps a recall lot-level rather than brand-wide, and it is the single highest-leverage stage in the whole pipeline. The rules that make scoping defensible are detailed in Lot-Level Recall Scoping.
3. Chain reconstruction. For each in-scope lot, the engine walks the one-up/one-back links recorded at every Shipping and Receiving CTE to assemble the full upstream sources and downstream recipients. This is a graph traversal over the ledger: sources are followed backward toward origin, recipients forward toward retail. The traversal must terminate cleanly on cycles and missing links rather than looping or silently truncating, a discipline covered in One-Up, One-Back Chain Reconstruction.
4. Sortable-spreadsheet export. The reconstructed chain is serialized into the sortable electronic format the FDA expects — one row per CTE occurrence, columns aligned to the KDE contract, sortable by lot code, location, and timestamp. Generating that artifact deterministically, with stable column ordering and no lossy coercion, is the subject of FDA 24-Hour Response Automation.
5. FDA submission. The validated export is handed to the readiness workflow for delivery within the 24-hour window, alongside the audit-log record of exactly which query produced it.

The resolver deserves particular attention because it is where a recall’s blast radius is decided. It accepts two kinds of predicate: a confirmed seed lot, which is unambiguous, or a product-and-window descriptor, which is not. A window predicate must translate a vague “romaine shipped in mid-February” into a concrete set of lots without either under-scoping (missing an implicated lot and endangering the public) or over-scoping (sweeping in clean product and inflating the recall). Getting that translation right is a discipline in itself — the mechanics of turning a single confirmed code into a full scope are worked through in scoping a recall by traceability lot code, and the end-to-end rehearsal of the whole request is walked through in how to simulate an FDA 24-hour traceability record request.

Wrapping the whole pipeline is a mock-drill loop. A scheduled drill injects a synthetic contamination signal at the trigger, runs the full pipeline, and measures the elapsed time-to-export against the 24-hour budget — surfacing gaps before a real investigation does. The feedback path is what converts recall readiness from a claim into a measured metric. Because the drill uses the same trigger interface as a live event, there is no separate rehearsal code path that could drift out of sync with production; the only difference between a drill and a real recall is the provenance of the signal and the fact that no product is actually withdrawn.

Because the recall engine is read-only, it can be scaled and cached independently of ingestion, and it can be pointed at a point-in-time snapshot of the ledger to reproduce exactly what a query would have returned on the day of an event. That reproducibility is essential when an inspector asks not just “what is the chain now” but “what did your system know at the moment you scoped the recall.” The append-only ledger makes every historical state reconstructable, and the recall engine is the tool that reconstructs it.

Core Data Contract: KDEs a Recall Query Must Resolve

A recall query is only as defensible as the fields it can resolve for every event in the chain. The table below is the canonical data contract for the recall path — the KDEs the lot-scope resolver and chain-reconstruction traversal must read for each CTE occurrence. Two of these are structural rather than single columns in the source schema: the immediate previous source and the immediate subsequent recipient are the one-up/one-back links that make a chain a graph rather than a list. The Regulatory Source column cites the specific Subpart S provision that mandates each element, so that every scoping and traversal decision remains traceable back to the rule.

Two contract rules keep a traceback defensible. First, the one-up and one-back links must reference a Traceability Lot Code the ledger has actually seen; a Receiving event whose source lot does not exist is an orphaned link, and the traversal must record it as a gap rather than silently terminating the branch. Second, event_timestamp must be timezone-aware and monotonic within a lot’s lifecycle, because the chain’s temporal order is what distinguishes a legitimate upstream source from a downstream recipient. A non-monotonic timestamp inverts that order and can push a clean lot into the recall scope — or, worse, exclude an implicated one. The contract enforced here is the same one produced by the ingestion program’s Schema Validation Rules; the recall engine consumes it verbatim rather than re-deriving it.

Production-Grade Python: The RecallScope Engine

The following is a runnable recall-scope engine. It defines pydantic v2 models for the immutable ledger records and for a trace query, builds an in-memory one-up/one-back graph over the ledger, and walks that graph to assemble the complete set of affected lots for a given trigger. It uses Decimal for quantities, timezone-aware timestamps, structured logging as the audit trail, and explicit exceptions so that a broken chain surfaces as a diagnosable fault rather than a silent truncation. An in-memory graph is used for clarity; in production the same traversal runs against the persisted ledger, but the contract and the algorithm are identical.

from __future__ import annotations

import logging
from collections import defaultdict, deque
from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import Iterable

from pydantic import BaseModel, ConfigDict, Field, field_validator

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("recall_scope")


class CTEType(str, Enum):
    HARVESTING = "Harvesting"
    INITIAL_PACKING = "Initial Packing"
    SHIPPING = "Shipping"
    RECEIVING = "Receiving"
    TRANSFORMATION = "Transformation"


class RecallScopeError(Exception):
    """Base class for recall-scope faults."""


class OrphanedLinkError(RecallScopeError):
    """A one-up/one-back link references a lot the ledger has never seen."""


class NonMonotonicChainError(RecallScopeError):
    """Events for a lot are not in a resolvable temporal order."""


class LedgerEvent(BaseModel):
    """One immutable CTE record as read from the KDE ledger."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    traceability_lot_code: str = Field(..., min_length=3, max_length=64)
    cte_type: CTEType
    location_id: str = Field(..., pattern=r"^\d{13}$")
    event_timestamp: datetime
    quantity: Decimal = Field(..., gt=0)
    reference_document_number: str | None = None
    # one-back on Receiving, one-up on Shipping; the counterpart TLC in the link.
    linked_lot_code: str | None = Field(default=None, min_length=3, max_length=64)

    @field_validator("event_timestamp")
    @classmethod
    def require_timezone(cls, v: datetime) -> datetime:
        if v.tzinfo is None:
            raise ValueError("event_timestamp must be timezone-aware (UTC recommended)")
        return v


class TraceQuery(BaseModel):
    """The trigger that scopes a recall simulation."""

    model_config = ConfigDict(extra="forbid")

    seed_lot_code: str = Field(..., min_length=3, max_length=64)
    window_start: datetime | None = None
    window_end: datetime | None = None
    max_depth: int = Field(default=25, ge=1, le=500)

    @field_validator("window_end")
    @classmethod
    def window_ordered(cls, v: datetime | None, info) -> datetime | None:
        start = info.data.get("window_start")
        if v is not None and start is not None and v < start:
            raise ValueError("window_end must not precede window_start")
        return v


class RecallScope(BaseModel):
    """The defensible result of a traceback: the affected lots and any gaps."""

    seed_lot_code: str
    affected_lots: list[str]
    event_count: int
    gaps: list[str]


class LedgerGraph:
    """In-memory one-up/one-back graph assembled from immutable ledger events."""

    def __init__(self, events: Iterable[LedgerEvent]) -> None:
        self._events_by_lot: dict[str, list[LedgerEvent]] = defaultdict(list)
        self._upstream: dict[str, set[str]] = defaultdict(set)
        self._downstream: dict[str, set[str]] = defaultdict(set)
        self._known_lots: set[str] = set()
        self._load(events)

    def _load(self, events: Iterable[LedgerEvent]) -> None:
        for event in events:
            lot = event.traceability_lot_code
            self._known_lots.add(lot)
            self._events_by_lot[lot].append(event)
        for lot, lot_events in self._events_by_lot.items():
            ordered = sorted(lot_events, key=lambda e: e.event_timestamp)
            if any(
                a.event_timestamp == b.event_timestamp and a.cte_type == b.cte_type
                for a, b in zip(ordered, ordered[1:])
            ):
                raise NonMonotonicChainError(
                    f"lot {lot} has duplicate timestamped events of the same CTE type"
                )
            for event in lot_events:
                if event.linked_lot_code is None:
                    continue
                if event.cte_type is CTEType.RECEIVING:
                    self._upstream[lot].add(event.linked_lot_code)
                elif event.cte_type is CTEType.SHIPPING:
                    self._downstream[lot].add(event.linked_lot_code)

    def neighbors(self, lot: str) -> set[str]:
        return self._upstream[lot] | self._downstream[lot]

    def is_known(self, lot: str) -> bool:
        return lot in self._known_lots

    def event_count(self, lot: str) -> int:
        return len(self._events_by_lot.get(lot, []))


def assemble_recall_scope(graph: LedgerGraph, query: TraceQuery) -> RecallScope:
    """Walk the ledger graph from the seed lot to every connected affected lot."""
    if not graph.is_known(query.seed_lot_code):
        raise OrphanedLinkError(f"seed lot {query.seed_lot_code} is absent from the ledger")

    visited: set[str] = set()
    gaps: list[str] = []
    frontier: deque[tuple[str, int]] = deque([(query.seed_lot_code, 0)])
    total_events = 0

    while frontier:
        lot, depth = frontier.popleft()
        if lot in visited:
            continue
        visited.add(lot)
        total_events += graph.event_count(lot)
        logger.info("scoped lot %s at depth %d", lot, depth)
        if depth >= query.max_depth:
            logger.warning("max traversal depth reached at lot %s", lot)
            continue
        for neighbor in sorted(graph.neighbors(lot)):
            if not graph.is_known(neighbor):
                gaps.append(neighbor)
                logger.error("orphaned link: lot %s references unknown lot %s", lot, neighbor)
                continue
            if neighbor not in visited:
                frontier.append((neighbor, depth + 1))

    scope = RecallScope(
        seed_lot_code=query.seed_lot_code,
        affected_lots=sorted(visited),
        event_count=total_events,
        gaps=sorted(set(gaps)),
    )
    logger.info(
        "recall scope resolved: %d lots, %d events, %d gaps",
        len(scope.affected_lots),
        scope.event_count,
        len(scope.gaps),
    )
    return scope


if __name__ == "__main__":
    now = datetime(2026, 2, 20, 14, 0, tzinfo=timezone.utc)
    ledger = [
        LedgerEvent(traceability_lot_code="TLC-ROMAINE-0420", cte_type=CTEType.INITIAL_PACKING,
                    location_id="0614141000012", event_timestamp=now, quantity=Decimal("900")),
        LedgerEvent(traceability_lot_code="TLC-ROMAINE-0420", cte_type=CTEType.SHIPPING,
                    location_id="0614141000012", event_timestamp=now, quantity=Decimal("900"),
                    reference_document_number="BOL-88231", linked_lot_code="TLC-DIST-7781"),
        LedgerEvent(traceability_lot_code="TLC-DIST-7781", cte_type=CTEType.RECEIVING,
                    location_id="0614141000029", event_timestamp=now, quantity=Decimal("900"),
                    reference_document_number="BOL-88231", linked_lot_code="TLC-ROMAINE-0420"),
        LedgerEvent(traceability_lot_code="TLC-DIST-7781", cte_type=CTEType.SHIPPING,
                    location_id="0614141000029", event_timestamp=now, quantity=Decimal("450"),
                    reference_document_number="BOL-88240", linked_lot_code="TLC-RETAIL-3300"),
    ]
    graph = LedgerGraph(ledger)
    result = assemble_recall_scope(graph, TraceQuery(seed_lot_code="TLC-ROMAINE-0420"))
    print(result.model_dump())

The engine enforces the recall contract at every boundary. Ledger events are frozen pydantic models, so a traceback can never mutate the evidence it reads. Timestamps are rejected unless timezone-aware, quantities are Decimal, and unresolvable links become explicit gaps in the result rather than silent dead ends. The breadth-first traversal is depth-bounded to guarantee termination even on a cyclic graph, and the visited set makes the walk idempotent. Critically, an orphaned link is logged and recorded — never swallowed — because a gap the export omits is a lot the recall failed to scope. For production traversal against a large persisted ledger, the same algorithm is expressed over the graph library covered in reconstructing trace chains with NetworkX, which adds cycle detection and shortest-path diagnostics on top of this contract.

Integration Points Across the Traceability Program

The recall engine is a consumer, not an island. It sits at the read edge of the program and depends on the correctness of everything upstream of it.

• It reads the immutable KDE ledger. The chain-reconstruction stage queries the append-only ledger defined by the FSMA 204 architecture reference. The recall path never writes to that ledger; it reads point-in-time snapshots so a scope is reproducible. The KDE contract this page’s traversal consumes is the exact contract that page’s KDE Field Mapping Guide defines, which is why the two must stay byte-for-byte compatible on field names and enumerations.
• It depends on clean supplier ingestion. A traceback can only follow links that were captured correctly at the source. Broken one-back references are usually born upstream, when a distributor’s feed omits a source lot or mislabels a reference document. The quality of that inflow is governed by Supplier Data Ingestion and monitored by its Data Quality Monitoring workflow; a rising rate of orphaned links in a drill is often the first visible symptom of an ingestion regression.
• It feeds audit-log export. Every recall query — real or drill — emits an audit record of exactly which seed lot, window, and ledger snapshot produced the export, so the submission is defensible after the fact. That record flows into the program’s Data Retention Policies for the full regulatory retention period.
• It closes trace gaps through fallback routing. When the traversal records a gap, it is not a dead end; the missing link is routed to the reconciliation path defined in Fallback Routing Logic so a human can resolve it before the export is submitted.

Because the recall path is downstream of both the ledger and the ingestion program, its drills double as an integration test for the whole system. A mock recall that comes back clean is evidence that ingestion, the ledger, and the export contract are all in agreement.

Compliance Failure Modes

The failures below cause the overwhelming majority of recall-response incidents. Each is paired with the diagnostic that isolates it quickly during a drill — or, if you are unlucky, during a live investigation.

1. Orphaned receiving events. A Receiving CTE references a source Traceability Lot Code the ledger has never seen, usually because the matching upstream Shipping event was quarantined or never ingested. The traversal branch terminates early and the true origin is missed. Diagnostic: run a nightly continuity check that flags any Receiving event whose immediate_previous_source lot has no record, and treat the count as a recall-readiness metric rather than a warning. The engine above records these as explicit gaps.
2. Missing one-up/one-back links. A Shipping event is captured with no recipient lot, or a Receiving event with no source, so the chain has a node but no edge. The lot appears in the ledger yet connects to nothing. Diagnostic: assert that every Shipping and Receiving event carries a non-null linked_lot_code at ingestion; count link-less shipping events per supplier and alert on any nonzero value.
3. Non-monotonic timestamps. A naive or mis-zoned timestamp inverts the temporal order of a lot’s events, so an upstream source is read as a downstream recipient. This can pull a clean lot into scope or push an implicated one out. Diagnostic: reject naive timestamps at ingestion and run a per-lot monotonicity check; the engine raises NonMonotonicChainError when it detects same-CTE timestamp collisions.
4. Over-broad scoping from lot-code collisions. Two physically distinct lots share a Traceability Lot Code because an upstream system reused a code across date ranges, so the resolver merges them and scopes both. The recall balloons beyond the truly affected product. Diagnostic: enforce TLC uniqueness per assigning location and date window at ingestion, and in the resolver, disambiguate by traceability_lot_code_source before merging any two events under one code.
5. Slow full-chain queries. The traversal is correct but too slow — an unindexed link column or an unbounded depth turns a full-chain reconstruction into a query that misses the 24-hour window. Diagnostic: index the link columns, bound traversal depth as the engine does, and benchmark the export end-to-end under realistic ledger volume in every drill, not in an idle staging copy.
6. Stale snapshot reads. The recall reads a lagging replica during an active event and returns a chain that is missing the most recent shipments. Diagnostic: pin recall queries to a synchronously replicated or explicitly snapshotted view and assert the event count against the primary before export.

Operational Checklist

Use this as the deployment gate for any change that touches the recall path, and as the pre-flight for any scheduled drill.

• Recall engine is strictly read-only against the ledger; no code path writes traceability data
• KDE contract version pinned and identical to the ingestion and ledger contracts
• Every Shipping and Receiving event asserted to carry a non-null one-up/one-back link at ingestion
• Orphaned-link continuity check scheduled nightly and its count tracked as a readiness metric
• Per-lot timestamp monotonicity check enabled; naive timestamps rejected upstream
• Traceability Lot Code uniqueness enforced per assigning location and date window
• Traversal depth bounded and link columns indexed for full-chain performance
• Recall queries pinned to a synchronous or explicitly snapshotted ledger view
• Sortable-spreadsheet export validated against the current FDA submission template
• Every query emits an audit record of seed lot, window, and ledger snapshot
• Full pipeline benchmarked end-to-end against the 24-hour budget under realistic volume
• Recorded gaps routed to fallback reconciliation before any export is submitted
• Mock recall drill executed on schedule and its time-to-export reviewed against target

The FDA’s traceability rule provides the definitive regulatory baseline for the recall obligation (FSMA 204 Food Traceability Rule); aligning drill acceptance criteria with those requirements is what turns a passing rehearsal into genuine readiness.

Frequently Asked Questions

Conclusion

Recall readiness is the moment every other layer of a traceability program is tested at once. The 24-hour window is not a target to aim for; it is a hard constraint that a system either meets deterministically or fails publicly. By treating the recall path as a strictly read-only companion to the immutable ledger — resolving a precise lot-level scope, reconstructing the one-up/one-back chain, exporting a sortable spreadsheet, and rehearsing the whole loop with scheduled drills — food safety teams convert an existential risk into a measured, defensible capability. A recall that stays lot-level because the traceback proved which lots were clean is the difference between a scoped action and a brand-wide loss.

Related

• Lot-Level Recall Scoping — resolving a fuzzy signal into a precise, defensible set of Traceability Lot Codes
• One-Up, One-Back Chain Reconstruction — walking the ledger graph to assemble upstream sources and downstream recipients
• FDA 24-Hour Response Automation — generating the sortable electronic spreadsheet within the regulatory window
• Mock Recall Drills — scheduled synthetic recalls that measure time-to-export before a real event
• FSMA 204 Architecture & KDE Compliance Mapping — the immutable KDE ledger this recall path reads from
• Supplier Data Ingestion & Sync Automation — the upstream pipeline whose link quality determines whether a traceback resolves

Up: All content — this recall reference is a top-level entry point for the FSMA 204 traceability program on this site.