Handling Missing Optional KDEs During Mapping: Not-Applicable vs Not-Captured

The production failure this guide resolves surfaces only during a recall, when it is expensive: an optional Key Data Element that a supplier never sent is indistinguishable, in your traceability store, from one the supplier deliberately marked as not applicable. Both landed in the database as the same empty string ''. A field mapper took a missing key, a null, and a genuine blank, and flattened all three into one value on the way in. Months later a recall query asks “which lots are missing a reference document identifier we are required to have?” and cannot answer it, because the lots that legitimately have no such document and the lots whose document was silently dropped look identical.

This is a mapping-layer defect that belongs to the KDE Field Mapping Guide: the point where a supplier’s raw payload is translated into the canonical KDE record. Some KDEs are conditionally required — the rule attaches them to specific Critical Tracking Events — so “absent” is a meaningful state that has to survive mapping intact. When the mapper collapses absence into an empty string, it destroys the one distinction a compliance query later depends on: was this data not applicable, or was it not captured? Those demand opposite responses. The first is fine; the second is a gap you must chase before an FDA request arrives.

Root Cause: Three Distinct States Collapsed Into One

The defect is a modeling omission. A KDE field can be in one of three states, and naive mapping code represents only one:

1. Present with a value. The supplier sent the field and it holds real data.
2. Not captured. The supplier’s payload omitted the key entirely, or sent null. The data may exist somewhere upstream but never reached you — this is a gap.
3. Not applicable. The supplier explicitly indicated the field does not apply to this event — for example, a transformation event that has no receiving reference. This is a valid, complete answer.

Common mapper code uses payload.get("field", ""), which folds states 2 and 3 into an identical empty string. Two forces make this the path of least resistance:

• Empty-string defaults feel safe. A downstream str column is simpler if every value is a string, so mappers reach for "" as the default and never surface the absence.
• get with a fallback hides the key’s presence. dict.get("k", "") returns "" whether the key was missing or held an empty value, erasing the very signal — was the key there? — that separates not-captured from not-applicable.

Under 21 CFR Part 1, Subpart S, several KDEs are required only in defined conditions, and a records system must be able to demonstrate that every required element was in fact captured. A store that cannot separate “we have no reference document because none applies” from “we lost the reference document” cannot make that demonstration — it silently under-reports its own gaps. The engineering rule: model absence explicitly with Optional[...] = None, and represent not-applicable with a distinct sentinel, so the three states never collapse.

Figure — Three KDE states collapsed, then preserved:

The KDE States This Mapper Preserves

The optional and conditionally-required fields this guide is concerned with, and the regulatory source that makes their absence meaningful. The point of the table is the rightmost distinction: for each field, absence can be legitimate or a gap, and the mapper must keep them apart.

Minimal Reproducible Example

Two suppliers send the same shipping event. Supplier A omits the reference document because their transaction has none; Supplier B’s export dropped it through an integration bug. The naive mapper renders both identically:

def naive_map(payload: dict) -> dict:
    # get(..., "") folds "missing key" and "empty value" into one empty string.
    return {
        "traceability_lot_code": payload.get("tlc", ""),
        "reference_document_number": payload.get("ref_doc_num", ""),
    }


supplier_a = {"tlc": "0071234500017"}                       # no ref doc: not applicable
supplier_b = {"tlc": "0071234500123", "ref_doc_num": None}  # ref doc lost: not captured

print(naive_map(supplier_a))
# -> {'traceability_lot_code': '0071234500017', 'reference_document_number': ''}
print(naive_map(supplier_b))
# -> {'traceability_lot_code': '0071234500123', 'reference_document_number': ''}

Both records now carry reference_document_number == ''. Nothing raised, and both rows look complete. A later query filtering WHERE reference_document_number = '' cannot tell Supplier A (fine) from Supplier B (a captured-data gap you are obligated to close). The information needed to separate them was present at mapping time and thrown away.

Fix: Explicit Null for Not-Captured, a Sentinel for Not-Applicable

The fix distinguishes the states at the boundary where they still exist. Missing or null maps to Python None (stored as SQL NULL), meaning not captured. An explicit not-applicable marker maps to a dedicated sentinel string, meaning deliberately blank. A present value maps to itself. A pydantic v2 model makes the three states first-class and refuses to invent an empty string.

from __future__ import annotations

from enum import Enum
from typing import Any, Optional

from pydantic import BaseModel, ConfigDict, field_validator


# The sentinel is a reserved value a supplier can never legitimately send as data.
NOT_APPLICABLE = "__NOT_APPLICABLE__"

# Tokens a supplier uses to say "this field does not apply to this event".
_NA_TOKENS = {"n/a", "na", "not applicable", "none-applicable"}


def map_optional_kde(payload: dict[str, Any], key: str) -> Optional[str]:
    """Map one optional KDE, preserving the three distinct states.

    Returns:
        None                -> not captured (key absent or null); a gap to chase.
        NOT_APPLICABLE      -> supplier explicitly marked the field inapplicable.
        <the value>         -> present and populated.
    """
    if key not in payload:
        return None  # key never arrived: not captured
    raw = payload[key]
    if raw is None:
        return None  # explicit null: also not captured
    text = str(raw).strip()
    if text == "":
        # A blank string is ambiguous on its own; treat it as not captured, not N/A.
        return None
    if text.lower() in _NA_TOKENS:
        return NOT_APPLICABLE  # deliberate, valid blank
    return text


class ShippingKDE(BaseModel):
    model_config = ConfigDict(str_strip_whitespace=True)

    traceability_lot_code: str
    # Optional[...] = None makes "absent" a real, representable state — never "".
    reference_document_number: Optional[str] = None
    reference_document_type: Optional[str] = None

    @field_validator("traceability_lot_code")
    @classmethod
    def require_tlc(cls, v: str) -> str:
        if not v or not v.strip():
            raise ValueError("traceability_lot_code is required on every CTE")
        return v.strip()

    @classmethod
    def from_payload(cls, payload: dict[str, Any]) -> "ShippingKDE":
        """Build the record while preserving optional-KDE state semantics."""
        return cls(
            traceability_lot_code=payload.get("tlc", ""),
            reference_document_number=map_optional_kde(payload, "ref_doc_num"),
            reference_document_type=map_optional_kde(payload, "ref_doc_type"),
        )

Persist the sentinel and the null faithfully. In SQL the column stays nullable, NULL carries not-captured, and the sentinel string carries not-applicable — two values a query can separate cleanly:

-- The reference-document column keeps NULL and the sentinel as distinct states.
CREATE TABLE shipping_kde (
    traceability_lot_code      TEXT NOT NULL,
    reference_document_number  TEXT,         -- NULL = not captured
    reference_document_type    TEXT,
    CONSTRAINT tlc_present CHECK (length(traceability_lot_code) > 0)
);

Because the mapper never manufactures '', the two suppliers from the reproduction now diverge: Supplier A’s field becomes the sentinel (a valid answer), and Supplier B’s becomes None/NULL (a gap the next query will surface).

Verification Steps

Confirm the three states stay separate end to end — first in Python, then in the query an auditor would run:

a = ShippingKDE.from_payload({"tlc": "0071234500017", "ref_doc_num": "N/A"})
b = ShippingKDE.from_payload({"tlc": "0071234500123", "ref_doc_num": None})
c = ShippingKDE.from_payload({"tlc": "0071234500999", "ref_doc_num": "R-88"})

# Not-applicable is preserved as the sentinel, distinct from not-captured None.
assert a.reference_document_number == NOT_APPLICABLE
assert b.reference_document_number is None          # gap: must be chased
assert c.reference_document_number == "R-88"        # present value intact

# The two "blank" rows are no longer equal — the collapse is gone.
assert a.reference_document_number != b.reference_document_number
print("Three states preserved:", a.reference_document_number, b.reference_document_number)

The recall query that could not answer before now can. Gaps are exactly the NULL rows; legitimate blanks are the sentinel rows, and they never contaminate each other:

-- Genuine capture gaps only — the rows an FDA request would expose.
SELECT traceability_lot_code
FROM shipping_kde
WHERE reference_document_number IS NULL;        -- not-captured, NOT the N/A rows

-- Deliberately not-applicable rows, reported separately and correctly.
SELECT traceability_lot_code
FROM shipping_kde
WHERE reference_document_number = '__NOT_APPLICABLE__';

The two queries return disjoint sets, which is the whole point: the first is your remediation list, the second is your evidence that those blanks are intentional. Feed the not-captured count into data-quality monitoring so a supplier whose gap rate climbs is flagged before a recall, not during one.

Related Edge Cases to Check Next

• Whitespace-only and placeholder values. A supplier sends " ", "-", or "pending" in an optional field. These are not real values and not explicit N/A markers; decide deliberately whether each maps to not-captured or is rejected at the Schema Validation Rules gate, and document the choice so it is consistent across suppliers.
• Conditionally-required fields flipping on event type. transformation_input_lot is not-applicable on a shipping CTE but a genuine gap on a transformation CTE. The correct classification depends on the event type, so pair the mapper with an event-type-aware rule rather than treating optionality as a fixed property of the column.
• Round-tripping the sentinel to FDA exports. When you produce records for an FDA request, the sentinel must render as a human-meaningful “not applicable”, never leak the raw __NOT_APPLICABLE__ token, and never be confused with an empty cell. Verify the export layer translates both NULL and the sentinel into the correct presentation, coordinating with Data Retention Policies on how long each state is retained.

Frequently Asked Questions

Related

• KDE Field Mapping Guide — the parent mapping layer where supplier payloads become canonical KDE records.
• FSMA 204 Architecture & KDE Compliance Mapping — the Critical Tracking Events that make some KDEs conditionally required.
• Schema Validation Rules — the gate that decides whether placeholder and whitespace values are rejected before mapping.
• Data Quality Monitoring — where the not-captured gap rate is tracked so it surfaces before a recall.
• Data Retention Policies — how long each KDE state is retained and produced for audits.

Up: KDE Field Mapping Guide — this guide is the missing-value discipline that mapping depends on to stay auditable.

For the authoritative regulatory text, reference the FDA Food Traceability Final Rule.