Mock Recall Drills & Traceability Exercises for FSMA 204

The only way to know a recall system works is to fire it before the FDA does. A traceability database can look complete on a dashboard and still collapse the first time someone asks it to produce, from a single implicated lot, the full one-up/one-back chain in a machine-sortable form within 24 hours. Missing reference documents, orphaned receiving events, quantities that will not reconcile, and lot codes that dead-end at a supplier who never sent a shipping event are invisible until you actually run the query under time pressure. A scheduled mock recall drill is the rehearsal that surfaces those defects while they are cheap to fix — days or weeks before a real outbreak turns them into a public-health and legal liability. This page belongs to the Recall Simulation & FDA 24-Hour Response reference and defines how to design, score, and continuously improve those drills so the 24-hour obligation is a measured capability rather than an untested assumption.

The triggering requirement is unambiguous. Under 21 CFR Part 1, Subpart S, when the FDA requests traceability information during an outbreak or recall, a covered facility must furnish an electronic, sortable spreadsheet of the relevant records within 24 hours (§ 1.1455). A mock recall drill treats that clause as a stopwatch: it selects a seed Traceability Lot Code, reconstructs the trace both directions, and measures whether the reconstruction is complete, reconciled, and fast enough to satisfy the mandate with margin to spare. The scoping mechanics that a drill exercises are covered in depth by the Lot-Level Recall Scoping guide, and the graph-walking it depends on lives in One-Up, One-Back Reconstruction; this page is about the exercise wrapped around them — the drill design, the metrics that turn a run into a pass-or-fail verdict, the scoring model, and the remediation loop that closes the gaps a drill exposes.

Drill Design and the Scoring Cycle

A mock recall drill is a closed loop, not a one-off report. Each cycle starts by selecting a scenario — a seed Traceability Lot Code plus the scope of product it represents — then runs a traceback against the live traceability graph, scores the result against fixed thresholds, logs every gap it finds, drives those gaps into a remediation queue, and schedules the next run. The discipline that makes the loop valuable is that it is scored: a drill either meets 100% trace completeness, full mass-balance reconciliation, and a time-to-reconstruct comfortably inside the 24-hour ceiling, or it fails and produces a concrete remediation backlog. There is no partial credit, because a partial trace is exactly what fails an FDA request.

Figure — The mock recall drill cycle:

Scenario design determines what a drill actually proves. A drill that always seeds the same well-behaved lot measures nothing after the first pass — it confirms a path you already know works. Effective scenarios rotate across the risk surface: high-velocity commodities on the Food Traceability List, lots that pass through a transformation step where quantities are combined or split, shipments that cross a co-packer or third-party logistics node, and lots sourced from suppliers whose data quality is historically weak. Each scenario names a seed Traceability Lot Code, the direction of trace to exercise, and the known-good expectations — how many nodes the chain should contain and what quantity should reconcile end to end — so the scoring step has a ground truth to compare against. Scenarios where the ground truth itself is uncertain are the most valuable, because they expose the difference between what the system reports and what actually shipped.

Traceback is the work the drill times. Starting from the seed lot, the runner walks upstream to the source (harvesting, cooling, initial packing) and downstream to the last recorded receiver, assembling every Critical Tracking Event that touches the lot. The same reconstruction an FDA 24-Hour Response Automation job performs for real is what the drill performs under a clock, which is why the drill runner and the production responder should share the same graph-walking core — a drill that exercises different code than the real responder proves nothing about the real responder.

Drill Metrics and Regulatory Basis

A drill is only as credible as the metrics it scores. Each metric below maps to a specific defect an FDA traceback would expose, and each carries a target threshold that a passing drill must meet. Trace completeness and mass-balance are held at 100% deliberately: under Subpart S there is no acceptable fraction of a lot that goes untraced, because the untraced fraction is precisely the product that stays on shelves during a recall.

Trace completeness answers the blunt question: if the FDA asks for everything about this lot, can we produce everything? A score below 100% means at least one node in the chain is missing, and the missing node is usually the interesting one — the transformation that combined lots, or the receiving event that never got its shipping counterpart. Mass-balance reconciliation is the quantity check: the total quantity that entered the traced subtree at its source must equal the quantity accounted for at its terminals, subject to documented shrink; a discrepancy means product moved without a record, which is both a compliance failure and a food-safety blind spot. Bill-of-lading reconciliation confirms that every shipment CTE points at a real reference document, because a shipping event without its reference document (§ 1.1340) cannot be corroborated against the carrier’s paperwork during an investigation. The full field-level basis for these KDEs is catalogued in the KDE Field Mapping Guide; the drill’s job is to prove those fields are present and consistent across an actual reconstructed chain rather than in the abstract.

Production Implementation: A Scored Drill Runner

The runner below executes a simulated traceback over an in-memory event graph and scores it against explicit thresholds, emitting a structured scorecard and a pass/fail verdict. It uses pydantic v2 for the event and scorecard contracts, Decimal for every quantity so reconciliation is exact, and timezone-aware ISO 8601 timestamps. In production the events iterable is populated from the traceability store rather than a literal list, but the scoring logic — and the thresholds it enforces — is identical, which is the point: the drill scores the same reconstruction the FDA responder produces.

from __future__ import annotations

import json
import logging
import time
from datetime import datetime
from decimal import Decimal
from typing import Iterable

from pydantic import BaseModel, ConfigDict, Field, field_validator

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    handlers=[logging.StreamHandler()],
)
logger = logging.getLogger("fsma204.mock_drill")

CTE_VOCABULARY = {
    "harvesting",
    "cooling",
    "initial_packing",
    "shipping",
    "receiving",
    "transformation",
}


class TraceEvent(BaseModel):
    """A single Critical Tracking Event node in the traceability graph."""

    model_config = ConfigDict(frozen=True, extra="forbid")

    tlc: str
    event_type: str
    quantity: Decimal
    unit: str
    reference_doc: str | None = None
    upstream_tlcs: tuple[str, ...] = ()
    event_timestamp: str

    @field_validator("event_type")
    @classmethod
    def known_event(cls, v: str) -> str:
        if v not in CTE_VOCABULARY:
            raise ValueError(f"unrecognized CTE type: {v}")
        return v

    @field_validator("event_timestamp")
    @classmethod
    def tz_aware_iso(cls, v: str) -> str:
        parsed = datetime.fromisoformat(v.replace("Z", "+00:00"))
        if parsed.tzinfo is None:
            raise ValueError("event_timestamp must be timezone-aware ISO 8601")
        return v


class DrillThresholds(BaseModel):
    """Pass/fail gates a scored drill must clear."""

    model_config = ConfigDict(extra="forbid")

    min_completeness_pct: Decimal = Decimal("100")
    max_reconstruct_seconds: Decimal = Decimal("14400")  # 4h working buffer
    min_mass_balance_pct: Decimal = Decimal("100")
    min_bol_reconciliation_pct: Decimal = Decimal("100")
    max_orphaned_events: int = 0


class DrillScenario(BaseModel):
    """A scripted seed lot and its known-good expectations."""

    model_config = ConfigDict(extra="forbid")

    scenario_id: str
    seed_tlc: str
    expected_node_count: int = Field(gt=0)
    expected_mass: Decimal = Field(gt=0)


class DrillScorecard(BaseModel):
    model_config = ConfigDict(extra="forbid")

    scenario_id: str
    completeness_pct: Decimal
    reconstruct_seconds: Decimal
    mass_balance_pct: Decimal
    bol_reconciliation_pct: Decimal
    orphaned_events: int
    passed: bool


class DrillRunner:
    """Executes a simulated traceback and scores it against thresholds."""

    def __init__(
        self, events: Iterable[TraceEvent], thresholds: DrillThresholds
    ) -> None:
        self._index: dict[str, TraceEvent] = {e.tlc: e for e in events}
        self._thresholds = thresholds

    def _traceback(self, seed_tlc: str) -> set[str]:
        """Breadth-first one-up walk from the seed lot across upstream links."""
        seen: set[str] = set()
        frontier: list[str] = [seed_tlc]
        while frontier:
            tlc = frontier.pop()
            if tlc in seen or tlc not in self._index:
                continue
            seen.add(tlc)
            frontier.extend(self._index[tlc].upstream_tlcs)
        return seen

    def run(self, scenario: DrillScenario) -> DrillScorecard:
        start = time.monotonic()
        reached = self._traceback(scenario.seed_tlc)
        elapsed = Decimal(str(round(time.monotonic() - start, 4)))
        reconstructed = [self._index[t] for t in reached]

        completeness = (
            Decimal(len(reached))
            / Decimal(scenario.expected_node_count)
            * Decimal(100)
        )

        source_mass = sum(
            (e.quantity for e in reconstructed if e.event_type == "harvesting"),
            Decimal(0),
        )
        mass_balance = source_mass / scenario.expected_mass * Decimal(100)

        shipment_events = [
            e for e in reconstructed if e.event_type in {"shipping", "receiving"}
        ]
        with_docs = [e for e in shipment_events if e.reference_doc]
        bol_pct = (
            Decimal(len(with_docs)) / Decimal(len(shipment_events)) * Decimal(100)
            if shipment_events
            else Decimal(100)
        )

        orphaned = sum(
            1
            for e in reconstructed
            if e.event_type == "receiving"
            and any(up not in self._index for up in e.upstream_tlcs)
        )

        passed = (
            completeness >= self._thresholds.min_completeness_pct
            and elapsed <= self._thresholds.max_reconstruct_seconds
            and mass_balance >= self._thresholds.min_mass_balance_pct
            and bol_pct >= self._thresholds.min_bol_reconciliation_pct
            and orphaned <= self._thresholds.max_orphaned_events
        )

        scorecard = DrillScorecard(
            scenario_id=scenario.scenario_id,
            completeness_pct=completeness.quantize(Decimal("0.01")),
            reconstruct_seconds=elapsed,
            mass_balance_pct=mass_balance.quantize(Decimal("0.01")),
            bol_reconciliation_pct=bol_pct.quantize(Decimal("0.01")),
            orphaned_events=orphaned,
            passed=passed,
        )
        logger.info(
            "drill scored | %s",
            json.dumps(scorecard.model_dump(), default=str),
        )
        return scorecard


if __name__ == "__main__":
    events = [
        TraceEvent(
            tlc="LOT-ROMA-0419",
            event_type="harvesting",
            quantity=Decimal("500"),
            unit="cs",
            event_timestamp="2026-04-19T06:12:00-05:00",
        ),
        TraceEvent(
            tlc="LOT-ROMA-0419-PK",
            event_type="initial_packing",
            quantity=Decimal("500"),
            unit="cs",
            reference_doc="PACK-88213",
            upstream_tlcs=("LOT-ROMA-0419",),
            event_timestamp="2026-04-19T11:40:00-05:00",
        ),
        TraceEvent(
            tlc="LOT-ROMA-0419-SH",
            event_type="shipping",
            quantity=Decimal("500"),
            unit="cs",
            reference_doc="BOL-556120",
            upstream_tlcs=("LOT-ROMA-0419-PK",),
            event_timestamp="2026-04-20T02:05:00-05:00",
        ),
        TraceEvent(
            tlc="LOT-ROMA-0419-RC",
            event_type="receiving",
            quantity=Decimal("500"),
            unit="cs",
            reference_doc="BOL-556120",
            upstream_tlcs=("LOT-ROMA-0419-SH",),
            event_timestamp="2026-04-20T15:22:00-04:00",
        ),
    ]
    scenario = DrillScenario(
        scenario_id="DRILL-2026-Q2-01",
        seed_tlc="LOT-ROMA-0419-RC",
        expected_node_count=4,
        expected_mass=Decimal("500"),
    )
    runner = DrillRunner(events, DrillThresholds())
    card = runner.run(scenario)
    print(
        f"PASS={card.passed} completeness={card.completeness_pct}% "
        f"mass_balance={card.mass_balance_pct}% "
        f"seconds={card.reconstruct_seconds}"
    )

The scoring method encodes the pass/fail contract as a single boolean conjunction, so a drill fails if any metric misses its threshold — there is no averaging that could let a strong completeness score mask an orphaned event. The _traceback walk is deliberately identical in shape to the one the real responder runs; seeding it from a receiving event and following upstream_tlcs reconstructs the one-up chain that a recall query depends on. Because every quantity is a Decimal, mass-balance reconciliation is exact rather than subject to floating-point drift, which matters when a 0.3% discrepancy is the signal that a co-packer under-reported a split lot.

From Gaps to Remediation

A drill that only produces a red or green light wastes most of its value. The point of scoring is the gap list a failing drill emits, and the remediation loop that consumes it. Every failed metric decomposes into specific, addressable defects: a completeness score below 100% names the exact Traceability Lot Codes that could not be reached; an orphaned-event count above zero names the receiving CTEs whose upstream shipping event is missing; a mass-balance shortfall points at the transformation or shipping node where quantity leaked. Each of these becomes a ticket with an owner, a root cause, and a due date, and the drill is not considered closed until a re-run of the same scenario clears the threshold it previously failed.

The most common gaps a first drill exposes are predictable. Orphaned receiving events almost always trace to a supplier who sends receiving data but never sent the corresponding shipping CTE, or sent it under a lot code that does not match — a problem whose resolution lives in Resolving orphaned receiving events in recall queries. Missing reference documents gather around manual or EDI-sourced shipments where the reference document number was dropped during ingestion. Mass-balance failures concentrate at transformation nodes where an input lot was split across multiple outputs without the split being recorded. Because these categories recur, the remediation loop should feed structural fixes back upstream — into supplier onboarding, parser configuration, and validation rules — rather than patching individual records, so the same gap does not resurface at the next drill.

Remediation only works if the loop is honest about regression. A scenario that passed last quarter can fail this quarter because a supplier changed an export format, a new co-packer was onboarded without proper lot linkage, or a schema migration silently dropped a field. Persisting each scorecard and comparing against the prior baseline turns the drill into a regression detector, not just a point-in-time snapshot — the automation for exactly that is covered in Automating Recurring Mock Recall Drills in Python with APScheduler.

Integration with the Recall Simulation Reference

A mock recall drill is the verification harness for the entire Recall Simulation & FDA 24-Hour Response reference. It does not implement scoping or reconstruction; it exercises them. When a drill selects a scenario, it invokes the same lot-scoping logic documented in Lot-Level Recall Scoping to bound the affected product, then hands the seed lot to the graph walk from One-Up, One-Back Reconstruction to assemble the chain. The scored output feeds the same export path that FDA 24-Hour Response Automation uses to produce the sortable spreadsheet, so a passing drill is direct evidence that the real responder will succeed on the same lot.

The signals a drill emits are also monitoring inputs. Trace completeness, orphaned-event counts, and mass-balance percentages per scenario are exactly the kind of leading indicators the Data Quality Monitoring layer tracks against per-supplier SLAs. A drop in a supplier’s contribution to reconstructable trace chains shows up in a drill weeks before it would surface as a live recall failure, which is what makes the drill a preventive control rather than a post-mortem one. The upstream data those metrics depend on arrives through Supplier Data Ingestion & Sync Automation, so a drill failure frequently resolves to an ingestion or onboarding fix rather than anything in the recall engine itself.

Operational Notes

Run drills on a fixed cadence and treat the schedule as a compliance artifact. A weekly automated drill against a rotating scenario, plus a quarterly full-scope drill with the compliance team in the room, is a defensible baseline; regulated facilities handling high-risk commodities should drill more often. Record every run — the scenario, the scorecard, the gaps, and the remediation outcome — in append-only storage so the drill history itself is auditable, aligning with the retention discipline in Data Retention Policies.

Recommended runtime and dependencies:

• Python 3.10+ (the runner uses str | None unions and modern generics throughout).
• pydantic ≥ 2.5 for the v2 field_validator / model_config / model_dump API. Do not mix in v1 validator.
• APScheduler ≥ 3.10 (or a cron entry) to drive recurring runs; see the automation guide.
• A durable scorecard store — SQLite for a single node, Postgres for shared history — so regression comparison and audit retention survive process restarts.

Configuration belongs in the environment, not the code. Externalize the drill cadence, the scenario catalogue, the threshold set, and the alerting target. Tune three things per facility: the scenario rotation (broaden it until every high-risk path is covered within a quarter), the time-to-reconstruct target (keep a working buffer well under the 24-hour ceiling so a slow run is still a pass with room to spare), and the regression sensitivity (how large a drop from baseline triggers an alert). Wire alerts on any failed drill to the same on-call path that a real recall would use, so the rehearsal exercises the human response as well as the code.

Frequently Asked Questions

Related

• Automating Recurring Mock Recall Drills in Python with APScheduler — schedule the drill, randomize the seed lot, persist scorecards, and alert on regressions.
• Lot-Level Recall Scoping — the scoping logic a drill scenario invokes to bound the affected product.
• One-Up, One-Back Reconstruction — the trace-chain walk the drill runner times and scores.
• FDA 24-Hour Response Automation — the live responder a passing drill validates, sharing the same reconstruction core.
• Data Quality Monitoring — tracks drill metrics as leading indicators against per-supplier SLAs.

Up: Recall Simulation & FDA 24-Hour Response — this reference owns the end-to-end simulation of an FDA traceability request, and drills are how it stays proven.