Skip to content

chore(evals): Update model evaluations 2026-10-13 - #278

Merged
mtodor merged 1 commit into
mainfrom
chore/update-model-evaluation-2026-10-13
Oct 13, 2026
Merged

mtodor merged 1 commit into
mainfrom
chore/update-model-evaluation-2026-10-13

Conversation

@rhacs-bot

Copy link
Copy Markdown
Contributor

Automated weekly model evaluation update.

Models evaluated: gpt-5-mini
Date: 2026-10-13

This PR was automatically generated by the Model Evaluation workflow.

@rhacs-bot
rhacs-bot requested a review from janisz as a code owner October 13, 2026 06:40
@coderabbitai

coderabbitai Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 6f8ecd91-7e41-4694-8ac5-2669f5b540b9

Comment @coderabbitai help to get the list of available commands.

@codecov-commenter

codecov-commenter commented Oct 13, 2026 •

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
380 2 378 12
View the full list of 2 ❄️ flaky test(s)
::policy 1

Flake rate in main: 100.00% (Passed 0 times, Failed 178 times)

Stack Traces | 0s run time
- test violation 1
- test violation 2
- test violation 3
::policy 4

Flake rate in main: 100.00% (Passed 0 times, Failed 178 times)

Stack Traces | 0s run time
- testing multiple alert violation messages 1
- testing multiple alert violation messages 2
- testing multiple alert violation messages 3

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

Copy link
Copy Markdown

E2E Test Results

Commit: ad3708d
Workflow Run: View Details
Artifacts: Download test results & logs

=== Evaluation Summary ===

  ✗ cve-cluster-does-not-exist (assertions: 3/3)
      one or more verification steps failed
  ✓ cve-cluster-does-exist (assertions: 3/3)
  ✓ cve-cluster-list (assertions: 3/3)
  ~ cve-nonexistent (assertions: 2/3)
      - MaxToolCalls: Too many tool calls: expected <= 5, got 7
  ✓ cve-clusters-general (assertions: 3/3)
  ✓ cve-detected-clusters (assertions: 3/3)
  ✓ rhsa-not-supported (assertions: 2/2)
  ✓ cve-log4shell (assertions: 3/3)
  ✓ list-clusters (assertions: 3/3)
  ✓ cve-multiple (assertions: 3/3)
  ✓ cve-detected-workloads (assertions: 3/3)

Tasks:      10/11 passed (90.91%)
Assertions: 31/32 passed (96.88%)
Tokens:     ~51943 (estimate - excludes system prompt & cache)
MCP schemas: ~12562 (included in token total)
Agent used tokens:
  Input:  10081 tokens
  Output: 18310 tokens
Judge used tokens:
  Input:  68837 tokens
  Output: 70222 tokens

@mtodor
mtodor merged commit b0ea75b into main Oct 13, 2026
10 checks passed
@mtodor
mtodor deleted the chore/update-model-evaluation-2026-10-13 branch October 13, 2026 11:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants