Repository navigation
[bot] Merge master/71c9de7f into rel/dev - #1901
Merged
Merged
Conversation
An expected chart may state `chart_type`. The dashboard evaluator reads the type from the chart body in the response's references, because a widget names only the chart id, and publishes `dashboard_chart_types_correct` when a case states one. A title alone cannot catch a chart authored as another type than the one asked for. An unknown type is rejected before the run. JIRA: QA-29722 risk: nonprod Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The visualization evaluator now reads every bucket the extended chart types use: - `from` and `to` of a sankey or dependency wheel count as dimensions, and `size_by` of a bubble chart as a metric; - a metric a repeater writes into `columns` counts as a metric, so the chart scores alike whichever bucket the agent put it in; - a null metric holds the x position of a bubble with no x measure. Three opt-in switches on an expected chart sharpen the check: - `buckets_strict` compares bucket by bucket and publishes `assertion-vis-buckets`, so swapped heatmap rows and columns or a reversed sankey no longer pass on the union of the buckets; - `metrics_ordered` compares the metrics by position, for bullet roles and bubble axes; - an `axis` on an expected metric entry must match the combo axis. JIRA: QA-29722 risk: nonprod Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d the chart part A visualization expectation can now state, beside its candidate charts: - `expects_visualization: false`: the agent must answer without a chart, for example by asking for a part the request leaves out. The run stops after one turn and publishes `assertion-vis-no-chart`. - `visualization_optional: true`: either an answer without a chart or a chart that matches a candidate is right, for a request the chart type cannot draw as asked. The run stops after one turn. - `answer_must_include` / `answer_must_not_match`: regular expressions matched without regard to case against the final answer, published as `assertion-vis-answer`. A typographic apostrophe reads as a plain one. - `format_intents`: how a metric's numbers must read. The agent's own `format_example` from the chart tool call is read back against the intent and compared as a number, so `#,##0.0M` showing 4,221,301,265.5M for "in millions" fails (GDAI-2592). Published as `assertion-vis-format`. - `requires_visualization_part`: the chart must come back as a part of the answer, not only as tool arguments. Published as `assertion-vis-part-present`. `expectation_options()` reads and validates these keys before any request, and rejects a key it does not know, at item level or on a candidate chart, so a misspelled key cannot pass unchecked. Every key is opt-in, so existing items score as before. A run keeps the final answer text, and the failure summary shows it when an answer check fails. JIRA: QA-29722 risk: nonprod Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extend more cases for extended charts
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1901 +/- ##
===========================================
+ Coverage 85.31% 85.65% +0.34%
===========================================
Files 342 343 +1
Lines 24670 25069 +399
===========================================
+ Hits 21046 21472 +426
+ Misses 3624 3597 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 71c9de7 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/40263958865).