Skip to content

Add model and generation time to LLM gist reports - #433

Merged
spalen0 merged 2 commits into
mainfrom
feat/llm-report-model-timing
Oct 21, 2026
Merged

spalen0 merged 2 commits into
mainfrom
feat/llm-report-model-timing

Conversation

@spalen0

@spalen0 spalen0 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Gist reports show the distinct models that produced a summary, revision, passing critique, or detail, plus total LLM generation time including retries and refinement. Empty or failed responses do not add models to the header. Model names are escaped as single-line Markdown text.

Claude CLI labels use the resolved model from response metadata, including aliases such as opus. Codex labels use LLM_CODEX_MODEL when set, otherwise codex-cli, because its JSON event stream does not expose the model.

The fallback chain forwards the answering provider's token usage and resets model and usage metadata before each request. One INFO usage line records each stage's model, outcome, token counts, and total generation time, including empty summaries. DEBUG attempt logs identify the provider class before the resolved model is available.

Example header:

Chain: Mainnet (chain id 1)
Risk: MEDIUM
Model: claude-cli:claude-opus-5-5
Time: 10.0s

Validation:

  • 291 relevant tests and 21 subtests passed across the LLM, explainer, report, and gist tests.
  • Repository Ruff lint, formatting, ty, and all pre-commit hooks passed.
  • Dependency audit found no known vulnerabilities or adverse project statuses.
  • Live Claude CLI check resolved claude-cli:claude-opus-5-5 without a model override.
  • Codex review found no actionable regressions.

@spalen0
spalen0 marked this pull request as ready for review October 21, 2026 16:37
P2: Preserve answering-provider usage, clear stale fallback metadata, and exclude empty stages from gist model labels. Retain the usage guard for failed direct API calls.

P3: Remove stale attempt labels and duplicate INFO logs, escape model labels, record elapsed time for empty summaries, and simplify mocks and clock tests. Trim duplicate test coverage.

Validation: 291 relevant tests and 21 subtests pass. Repository lint, format, types, hooks, and dependency audit pass; Codex review found no actionable regressions.
@spalen0
spalen0 merged commit 846236f into main Oct 21, 2026
3 checks passed
@spalen0
spalen0 deleted the feat/llm-report-model-timing branch October 21, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant