Skip to content

Preserve hydration summary diagnostics across failures and cache fallback - #2144

Merged
Tyrie Vella (tyrielv) merged 4 commits into
microsoft:masterfrom
tyrielv:tyrielv/hydration-summary-diagnostics
Oct 28, 2026
Merged

Tyrie Vella (tyrielv) merged 4 commits into
microsoft:masterfrom
tyrielv:tyrielv/hydration-summary-diagnostics

Conversation

@tyrielv

@tyrielv Tyrie Vella (tyrielv) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem and Context

Hydration status can report an error without telemetry that explains the result.
The duration event omits final validity and counts.
Exceptions and cancellation discard earlier observations.
Modified-path read failures can appear as collected zeros.

Cache lookup failures also lose their cause before the health command calculates a fallback.
The health completion event does not distinguish detailed health from status mode.
These gaps prevent diagnosis from telemetry alone.

Changes

  • Extend HydrationSummaryDuration with one terminal record for success, invalid counts, incomplete data, cancellation, and exceptions.
  • Preserve completed and partial count observations with explicit source states. Keep unavailable counts and unattempted durations distinct from zero.
  • Record every count-validation failure, 64-bit diagnostic sums, phase timings, index-header observations, placeholder folder states, and modified-path source outcomes.
  • Add command and calculation identifiers to the JSON payload. Record cache outcomes, parseable response counts, fallback decisions, and displayed results in health completion.
  • Keep command success/failure separate from calculation outcome and cache completeness. Retain exception identity when a health command throws.
  • Publish cache counts and diagnostic identity together. Add an optional, versioned IPC extension with cache state, calculation identity, age, completeness, and invalidation state.
  • Aggregate mount-side hydration requests in heartbeat telemetry. Keep exception stacks in local warnings and restrict structured telemetry to exception types and HRESULT values.
  • Add unit tests for partial results, source failures, cancellation, invalid counts, arithmetic overflow, cache skips, malformed responses, and legacy IPC compatibility.
  • Add unit tests that exercise source-loader orchestration, worker timeout phases, late worker completion, snapshot publication, invalidation, and heartbeat resets.
  • Add unit tests that send nested metadata through the telemetry pipe writer to a local collector. These tests assert collected zero values and unavailable values at the serialized sink.
  • Add functional tests that assert cache-served status, fallback status, and detailed health completion through the real command and mount pipe.
  • Document the diagnostic schema, IPC extension, privacy boundary, and delivery limits.

Scope and Compatibility

Count sets, percentages, disk-log replay, cache acceptance, and circuit-breaker policy stay unchanged.
The existing event names and duration fields remain.
The terminal duration record replaces generic exception telemetry and the early-exit warning.
Success and cancellation use the Informational level. Invalid, incomplete, and failed calculations use the Warning level.
Full exception text stays in local warnings instead of generic Error-level telemetry or telemetry warnings.
An unattempted phase now has a null duration instead of a synthetic zero.
Older clients ignore the IPC extension, and new clients accept legacy responses without diagnostics.

The existing cache and modified-path wait behavior stays unchanged.
A health worker can finish after the outer timeout, but it cannot write to the command's tracer or completion record.
Mount telemetry cannot observe a hook connection failure that never reaches the mount.
Collector outages, queue overflow, and process termination can still prevent telemetry delivery.

Validation limits

CI exercises the health/status cases, but no changed-line coverage report is available.
The local collector tests verify payload serialization and privacy, not production ingestion or deployed alert routing.
Production ingestion and deployed alert routing remain unverified.

…back

Record typed count availability, partial observations, validation failures, and phase outcomes. Correlate health commands with cached and fallback calculations through explicit payload identifiers and a compatible IPC extension. Keep count semantics and wait behavior unchanged.

Add unit and functional tests for the diagnostic contract, serialization, and compatibility.

Assisted-by: GPT-6.1 Sol

Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
…t tests

Preserve cached summaries and circuit-breaker counts when configuration reads fail. Preserve cancellation policy while recording cancellation origin. Separate command outcome from calculation outcome and name the heartbeat cache-state point sample explicitly.

Pin IPC state integers and add regression tests for source-loader orchestration, failed phases, worker timeouts, cache snapshots, real command paths, and collector payloads.

Assisted-by: GPT-6.1 Sol

Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Poll the structured cache state before checking disabled and available results. Filesystem background readiness does not guarantee that the hydration refresh passed its backoff. Keep the deadline bounded and retain the final completion record on failure.

Add functional test cases with zero-second and five-second cache backoff. Keep product behavior unchanged.

Assisted-by: GPT-6.1 Sol

Signed-off-by: Tyrie Vella <tyrielv@gmail.com>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI exercises the new health/status cases, but no changed-line coverage report is published, and I could not verify deployed alert routing for the new outcome fields. Please link any coverage and monitoring/alert validation, or explicitly record these limits in the PR validation notes. This is nonblocking.

Comment thread docs/hydration-status-diagnostics.md Outdated
Keep telemetry replacement history in the pull request description. State the count, cache, and circuit-breaker preservation requirement directly in the diagnostic contract.

Assisted-by: GPT-6.1 Sol

Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv

Copy link
Copy Markdown
Contributor Author

Note

🤖 Machine-drafted reply — reviewed and approved by Tyrie Vella (@tyrielv).

Added Validation limits to the PR description. No changed-line coverage report is available. Local collector tests verify serialization and privacy, not production ingestion or deployed alert routing. Those production checks remain unverified.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the documentation revision: it now describes durable diagnostic and policy-preservation contracts rather than change history. The prior finding is addressed; no new findings in the reviewed delta. The PR description explicitly records the remaining coverage and production telemetry/alert-validation limits.

@tyrielv
Tyrie Vella (tyrielv) merged commit f2f7f78 into microsoft:master Oct 28, 2026
39 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants