Theming: Flex Mapping Drift Detection and Upstream Pin #4266

Summary

Replace the manual synchronization note in flex-token-map.yaml with a
CI-enforced consistency check across the three internal mapping artifacts, and
record a pinned x3 upstream revision with a reproducible, offline-safe drift
report. The internal check runs entirely from checked-in files and blocks CI on
any mismatch. The upstream report is a checked-in artifact regenerated by an
explicit command, so CI never requires a network call while the diff between the
recorded pin and a candidate new revision is always inspectable.

Goal

Enforce that flex-token-map.yaml, flex-from-theme.json, and
nonFluentFlexTokens are structurally consistent at all times, and make any
divergence between FURN's mapping assumptions and the pinned x3 source commit
visible through a reproducible report rather than a manual reading of upstream
diffs.

Stage

Stage 1 and Stage 2.

The internal consistency check is a Stage 1 prerequisite: it enforces
correctness of the mapping table that Default Values
Codegen will generate from and that
Dynamic Theme Building depends on for its
projection. The upstream pin and drift report are a Stage 2 hardening step
that makes the Runtime Color Utilities port
and any future YAML update reproducibly checkable.

Why it matters

• Observed. The synchronization rule is currently a manual note.
  flex-token-map.yaml
  states under rules.notes: "Keep flex-from-theme.json synchronized with
  furn-theme mappings" and "Keep nonFluentFlexTokens in ../defaultTokens.ts
  synchronized with unmapped supported destinations." No automated check
  enforces either rule.
• Observed. The YAML, the JSON, and nonFluentFlexTokens encode three
  complementary views of the same mapping. A token added to the YAML with a
  furn-theme path must appear in flex-from-theme.json; a token added without
  a furn-theme path must appear in nonFluentFlexTokens; a token whose
  furn-theme entry is removed must move to nonFluentFlexTokens and leave the
  JSON. Each invariant is invisible until flexTokensFromTheme produces a wrong
  value at runtime.
• Observed. The interaction fallback rule has additional semantics that are
  easy to violate silently. A hover or pressed destination in
  nonFluentFlexTokens is allowed only when its rest destination is also
  unmapped; if the rest has a furn-theme source the state fallback must remain
  only in defaultFlexTokens, not in nonFluentFlexTokens, to prevent a
  theme-provided rest value from inheriting a mismatched fallback interaction
  color.
• Observed. x3 upstream is alpha and explicitly instructs consumers to pin
  exact versions. The
  flex-themes README
  documents that the legacy delta variables (--lightness-hover,
  --lightness-press, --alpha-hover, --alpha-press) are deprecated and
  scheduled for removal in 1.0.0. FURN records x3 source file paths in
  flex-token-map.yaml and
  flex.types.ts
  but no upstream commit, so there is no machine-checkable record of which
  revision those references describe.
• Inferred. A codegen script that reads flex-token-map.yaml to generate
  default values (see Default Values Codegen)
  must be able to trust that the three artifacts are consistent before it runs.
  An offline CI check that fails fast on any mismatch is a cleaner gate than
  discovering the inconsistency from a bad generated output downstream.

Observed current state

• Observed. flex-token-map.yaml at
  packages/agentic/design/src/tokens/mappings/flex-token-map.yaml
  records, per Flex token destination, a generic source, an optional fluent
  source, and an optional furn-theme dot-path into the FURN Theme type. For
  color tokens the shape nests rest, hover, and pressed sub-entries; for
  non-color tokens the mapping is flat.
• Observed. flex-from-theme.json at
  packages/agentic/design/src/tokens/mappings/flex-from-theme.json
  projects 169 FlexToken destination paths (color states expanded to dotted
  paths such as color.hover.backgroundNeutralSubtle) to FURN Theme source
  paths such as colors.neutralBackground2Hover. It is consumed directly by
  flexTokensFromTheme.ts
  as a declarative lookup table.
• Observed. nonFluentFlexTokens in
  defaultTokens.ts
  holds Flex token values that have no FURN Theme source. It is merged under the
  theme-projected values in flexTokensFromTheme, providing defaults for tokens
  the active theme does not supply. Hover and pressed entries appear here only
  when their rest destination is also absent from flex-from-theme.json.
• Observed. No CI step, script, or test validates that the three artifacts
  agree. The design package's test suite covers the runtime behavior of
  flexTokensFromTheme and useThemeState, but not the structural consistency
  of the mapping table itself.
• Observed. The x3 upstream baseline used when authoring the current
  mappings is x3-design/fluent-design@d334acf5cbad813f2b7cd554da942b09a7ff8f10,
  known from the research record (see Theming Workstream
  README).
  This SHA is not recorded anywhere in the FURN repository; the only x3
  references are file paths in flex-token-map.yaml and flex.types.ts.
• Observed. The three upstream files whose content FURN's mappings depend on
  at that commit are
  dev/web/flex-themes/css/semantic.css,
  dev/web/flex-themes/css/interaction-fallback.css,
  and
  dev/web/flex-tokens/mappings.json
  (the interaction table and generic token names). Token additions, removals, or
  renames in any of these files could require changes to flex-token-map.yaml,
  flex-from-theme.json, or flex.types.ts.

Scope

Internal consistency check

• Write a script (runnable as yarn check:mappings or equivalent) and a
  corresponding Jest test that reads flex-token-map.yaml and verifies:
  1. Every destination path with a furn-theme entry in the YAML appears in
     flex-from-theme.json with the same source path, and every entry in the
     JSON corresponds to a furn-theme entry in the YAML.
  2. Every supported destination path without a furn-theme entry in the YAML
     has a value in nonFluentFlexTokens; no destination that has a furn-theme
     entry appears in nonFluentFlexTokens.
  3. The interaction fallback rule: a hover or pressed destination in
     nonFluentFlexTokens must have its rest destination also absent from
     flex-from-theme.json.
  4. Every destination path in the YAML corresponds to a valid key in the
     FlexTokens type; no entry names a path the type does not declare.
• The check reads only checked-in files and makes no network call; it must pass
  on a machine with no internet access.
• Surface a clear diagnostic per violation: which destination path failed, which
  rule, and what the two conflicting values are.
• Integrate the check into the design package's yarn test target so it blocks
  CI when any rule is violated.

Upstream pin and drift report

• Add a machine-readable pin record to the repository (for example
  packages/agentic/design/src/tokens/mappings/upstream-pin.json) recording
  the x3 commit SHA, the three upstream file paths, and the date the pin was
  last verified. Initial value: d334acf5cbad813f2b7cd554da942b09a7ff8f10.
• Write a drift report script (for example yarn gen:upstream-drift) that
  fetches the three files from the pinned commit via the GitHub REST API, then
  compares them to the checked-in snapshots, and emits a structured diff report
  covering:
  • Generic token names added or removed in mappings.json's generics and
    interaction sections.
  • CSS custom property names added or removed in semantic.css.
  • Precomputed interaction token names added, removed, or changed in
    interaction-fallback.css.
• Check in snapshots of the three upstream files at the current pin alongside
  the pin record, so the diff can be regenerated offline without network access.
• The drift report script is network-optional: when run offline it compares
  against the checked-in snapshots rather than fetching, and clearly labels
  which mode it used.
• Define an explicit pin-update workflow: update the SHA in the pin record, run
  the fetch-and-snapshot command to refresh the three checked-in copies, re-run
  the drift script to confirm zero diff at the new pin, and commit all four
  files together with a changeset entry describing what changed upstream.
• Document the update workflow in the mapping directory's authoring notes or in
  a colocated README.md.

Codegen integration

• Update
  packages/agentic/design/scripts/codegen.cts
  to run the internal consistency check as its first step and abort with a clear
  error when any violation is found, so a contributor cannot regenerate defaults
  from an inconsistent mapping table.

Out of scope

• Updating the YAML, JSON, or nonFluentFlexTokens to fix any pre-existing
  inconsistency; that work belongs to whichever task introduces or exposes the
  inconsistency.
• Generating Flex defaults from the YAML; see
  Default Values Codegen.
• Changing FlexTokens or InteractiveColorOverrides type shapes; see
  Dynamic Theme Building.
• Acting on upstream drift (deciding whether FURN's mappings should change in
  response to x3 changes); that is an editorial decision outside this task.
• Enforcing that FURN's 59 interactive token names still match x3's 118-entry
  interaction table at the type level; the type is locked for now and any
  reconciliation belongs to Dynamic Theme Building.
• Fetching or running upstream x3 build scripts; the drift check reads only the
  three static content files.

Deliverables

1. A consistency-check script colocated with the mapping scripts (or in
   packages/agentic/design/src/tokens) that implements all four rules.
2. A Jest test that imports and runs the check, failing with a diagnostic
   message per violation, integrated into the design package's yarn test
   target.
3. packages/agentic/design/src/tokens/mappings/upstream-pin.json recording
   the pinned x3 SHA d334acf5cbad813f2b7cd554da942b09a7ff8f10, the three
   upstream file paths, and the date first recorded.
4. Checked-in snapshots of the three upstream files at the pinned commit.
5. A drift report script that compares the checked-in snapshots (or a freshly
   fetched set) against the pin record and emits a structured summary.
6. Documented pin-update workflow in the mapping directory.
7. Codegen integration: the consistency check runs as the first step of
   codegen.cts and aborts on any violation.
8. Changesets for the design package and, if changed,
   @fluentui-react-native/scripts.

Acceptance criteria

• yarn test in the design package fails with a clear per-rule diagnostic
  when any of the four consistency rules is violated by a manual edit to
  flex-token-map.yaml, flex-from-theme.json, or defaultTokens.ts, and
  passes without error when the three artifacts are consistent.
• The consistency check makes no network call and passes on a machine with
  no internet access.
• upstream-pin.json is present and records commit
  d334acf5cbad813f2b7cd554da942b09a7ff8f10 plus the three upstream file
  paths.
• Snapshots of the three upstream files at the pinned commit are checked in
  and match the content of those files at that commit.
• Running the drift report script against the checked-in snapshots produces
  a zero-diff result at the current pin.
• The drift report script runs offline using the checked-in snapshots and
  labels its output accordingly.
• The pin-update workflow is documented; following it from the initial pin
  to a hypothetical later SHA produces a commit that includes a refreshed pin
  record, refreshed snapshots, a zero-diff drift result, and a changeset.
• codegen.cts aborts with a clear error message when any consistency rule
  is violated before proceeding with code generation.
• yarn build, yarn lage test, and yarn lage lint pass at the
  repository root, and changesets are present.

Dependencies and ordering

• No hard prerequisite tasks, but the consistency check should land before or
  alongside Default Values Codegen so that task
  can trust the mapping table it reads.
• The upstream pin snapshot provides a concrete conformance baseline for
  Runtime Color Utilities, which requires a
  recorded source commit for the algorithm port.
• Does not depend on Package Consolidation, but if
  the mapping files relocate as part of consolidation the check and snapshot
  paths should follow.

Risks and open decisions

• Open decision. Where to colocate the upstream snapshots. Storing three
  upstream CSS and JSON files in the FURN repository adds bytes not published by
  the package; placing them in a scripts/ or test-fixtures/ subtree avoids
  bundling them. The choice should follow whatever pattern existing codegen tests
  use for fixtures.
• Open decision. Whether the drift report should be a committed Markdown
  file or only on-demand console output. A committed report provides a visible
  audit trail but requires regeneration whenever the snapshots update; on-demand
  output is simpler. Either way the command must be documented.
• Risk. The GitHub API rate-limits unauthenticated requests. The fetch step
  should document the rate limit and use authenticated requests where a token is
  available. CI must always use the checked-in snapshots and never make a network
  call.
• Risk. A future structural change to the YAML schema (for example moving
  from flat to nested keys for non-interactive tokens) would require updating the
  check script. The check should validate against the declared schemaVersion: 1
  field so breaking schema changes produce a clear error rather than silently
  passing with no violations.

Evidence and references

• packages/agentic/design/src/tokens/mappings/flex-token-map.yaml:
  the authoring-source mapping YAML, including the manual synchronization note
  in rules.notes and the schemaVersion: 1 declaration.
• packages/agentic/design/src/tokens/mappings/flex-from-theme.json:
  the 169-entry FlexToken-destination-to-FURN-Theme-source projection table.
• packages/agentic/design/src/tokens/defaultTokens.ts:
  nonFluentFlexTokens and defaultFlexTokens; the interaction fallback rule
  is encoded in the comment above nonFluentFlexTokens.
• packages/agentic/design/src/tokens/flexTokensFromTheme.ts:
  the runtime consumer of flex-from-theme.json; merges the JSON projection
  over nonFluentFlexTokens.
• packages/agentic/design/src/tokens/flex.types.ts:
  FlexTokens, SemanticColorTokenValues, and InteractiveColorOverrides;
  the type the consistency check validates destination paths against.
• packages/agentic/design/scripts/codegen.cts:
  the codegen entry point that will run the consistency check before generating.
• x3 dev/web/flex-tokens/mappings.json at d334acf:
  the interaction table (118 entries, 59 base tokens) and generic token map;
  the primary source for addition and removal drift detection.
• x3 dev/web/flex-themes/css/semantic.css at d334acf:
  the interaction fragment definitions cited by flex-token-map.yaml's generic
  source.
• x3 dev/web/flex-themes/css/interaction-fallback.css at d334acf:
  the precomputed interaction values cited by flex.types.ts and
  flex-token-map.yaml as the authoritative fallback source.
• x3 dev/web/flex-themes/README.md at d334acf:
  the upstream alpha status, exact-pin instruction, and deprecated variable
  removal notice.
• Theming Workstream README -- Suggested Additions 3 and 7:
  the two decision-gated additions this task combines.
• Default Values Codegen: the downstream task
  that depends on a consistent mapping table and notes the manual sync risk.
• Runtime Color Utilities: the algorithm port
  that requires a recorded upstream source commit.