Skip to content

LINBEE-30104 | fix: don't grade tasks that change no code (v1.4.6) - #22

Open
nivSwisa1 wants to merge 4 commits into
mainfrom
LINBEE-30104-skip-non-code
Open

nivSwisa1 wants to merge 4 commits into
mainfrom
LINBEE-30104-skip-non-code

Conversation

@nivSwisa1

@nivSwisa1 nivSwisa1 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

A task that changes no code now prints a 'skipped (not a code task)' line instead of a LOW grade: no calls, nothing reported, normal care; LOW stays for trivial code edits, and the hook treats the skipped line as answered.

🤖 Generated with Claude Code

✨ PR Description

Purpose: Update agentic-advisor to skip grading for non-code tasks while maintaining effort holds for actual file edits.

Main changes:

  • Added VERDICT_RE and GRADED_RE to distinguish between effort grades and skipped tasks
  • Updated verdict_state and guidance to support skipped (not a code task) verdict signals
  • Modified agentic-advisor-trigger.sh to require a real grade before allowing file edits

Generated by LinearB AI and added by gitStream.
AI-generated content may contain inaccuracies. Please verify before using.
💡 Tip: You can customize your AI Description using Guidelines Learn how

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 08:44

@orca-security-us orca-security-us Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed OSS Licenses high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

Copilot AI previously approved these changes Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The focused hook, guidance, and regression-test changes are consistent, with no blocking issues identified.

No open issues 🎉

What changed in this PR

Updates agentic-advisor to skip grading tasks that change no code, while retaining LOW grades for trivial code edits.

Changes:

  • Recognizes skipped outcomes as answered in the trigger hook.
  • Adds regression tests for reminder suppression and no reporting.
  • Clarifies guidance and bumps the plugin to v1.4.6.
File Description
tests/​agentic-advisor/​hooks_test.sh Tests skipped-outcome handling and no reporting.
plugins/​agentic-advisor/​skills/​agentic-advisor/​SKILL.md Defines skipping and limits effort levels to code changes.
plugins/​agentic-advisor/​README.md Documents non-code task handling.
plugins/​agentic-advisor/​hooks/​agentic-advisor-trigger.sh Accepts skipped outcomes and updates guidance.
plugins/​agentic-advisor/​CHANGELOG.md Records v1.4.6 behavior changes.
plugins/​agentic-advisor/​.claude-plugin/​plugin.json Bumps version to 1.4.6.

Review effort: Balanced

@gitstream-cm gitstream-cm Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✨ PR Review

The change adds a "skipped (not a code task)" verdict and updates the hook regex, guidance and skill docs to match. The regex and the new reminder text are consistent with each other. The new classification rules in SKILL.md overlap, and the skip verdict lets later code edits pass the hook unenforced.

2 issues detected:

🧹 Maintainability - Two adjacent rules both claim comment/doc edits and give different outputs (skipped vs LOW). 🛠️

Details: "Not a code task" lists "writing a comment" as skipped with no grade, while "Trivial code edit" lists "comments, docs" as LOW. Editing a code comment or a doc file fits both rules. The README says "writing a comment" is not graded, and "docs" was dropped from the trivial list, so doc edits now match neither rule clearly. The agent can pick either outcome (a skipped line or a LOW grade) for the same task, which makes telemetry and behavior inconsistent.

File: plugins/agentic-advisor/skills/agentic-advisor/SKILL.md (48-49)

🛠️ A suggested code correction is included in the review comments.

🐞 Bug - Treating "skipped" as a final answer lets code edits proceed without an effort grade, because only the latest Skill call's verdict is checked.

Details: The verdict regex now counts "skipped (not a code task)" as a completed verdict. The SKILL text says that if the task later turns into a code change, the sweep should run before the first edit. After a skip, verdict_state returns "ok" and the done marker is already set, so the hook never reminds the agent or enforces a grade before the first edit. A task that starts as a question and then edits code is therefore never graded or reported, and the "MANDATORY FIRST STEP" gate is bypassed. The guidance also says "do not skip it even if the task seems obvious" and then allows a skip, which invites over-skipping.

File: plugins/agentic-advisor/hooks/agentic-advisor-trigger.sh (56-56)

Generated by LinearB AI and added by gitStream.
AI-generated content may contain inaccuracies. Please verify before using.
💡 Tip: You can customize your AI Review using Guidelines Learn how

Comment on lines +48 to +49
- **Not a code task → no grade at all.** If the task will not change any code (reading or explaining code, answering a question, reviewing a doc, ticket or screenshot, writing a comment or message), don't grade it: print `> LinearB: <repo> — skipped (not a code task)`, call nothing, and work with your normal care. A grade measures the risk of a code change; a LOW here would wrongly tell you to go light on accuracy. If the task later turns into a code change, run the sweep then, before the first edit.
- **Trivial code edit → LOW, skip everything.** If the task edits code but changes **no code behavior** (comments, docs, formatting/whitespace, a pure rename), skip signal gathering entirely: effort is **LOW**, add no defensive guards (there's no logic to protect), and don't call LinearB at all — this holds **even on a fragile repo**. Anything that changes behavior is **not** trivial and warrants the sweep below — note that a **constant-value change counts as behavioral** (e.g. a timeout, retry count, or feature-flag default), as does any conditional or logic edit.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Maintainability - Overlapping skip/LOW rules: Make the categories mutually exclusive. For example, say that any task that edits a file, including a comment or a doc, is a trivial code edit graded LOW. Reserve "skipped" for tasks that modify no files (read, explain, review, answer). Update the README wording to match.

Suggested change
- **Not a code task → no grade at all.** If the task will not change any code (reading or explaining code, answering a question, reviewing a doc, ticket or screenshot, writing a comment or message), don't grade it: print `> LinearB: <repo> — skipped (not a code task)`, call nothing, and work with your normal care. A grade measures the risk of a code change; a LOW here would wrongly tell you to go light on accuracy. If the task later turns into a code change, run the sweep then, before the first edit.
- **Trivial code edit → LOW, skip everything.** If the task edits code but changes **no code behavior** (comments, docs, formatting/whitespace, a pure rename), skip signal gathering entirely: effort is **LOW**, add no defensive guards (there's no logic to protect), and don't call LinearB at all — this holds **even on a fragile repo**. Anything that changes behavior is **not** trivial and warrants the sweep below — note that a **constant-value change counts as behavioral** (e.g. a timeout, retry count, or feature-flag default), as does any conditional or logic edit.
- **Not a code task → no grade at all.** If the task will not modify any file (reading or explaining code, answering a question, reviewing a doc, ticket or screenshot, drafting a message in chat), don't grade it: print `> LinearB: <repo> — skipped (not a code task)`, call nothing, and work with your normal care. A grade measures the risk of a code change; a LOW here would wrongly tell you to go light on accuracy. If the task later turns into a file edit, run the sweep then, before the first edit.
- **Trivial code edit → LOW, skip everything.** If the task edits any file but changes **no code behavior** (code comments, doc files, formatting/whitespace, a pure rename), skip signal gathering entirely: effort is **LOW**, add no defensive guards (there's no logic to protect), and don't call LinearB at all — this holds **even on a fragile repo**. Editing a comment or a doc is always this case, never "skipped". Anything that changes behavior is **not** trivial and warrants the sweep below — note that a **constant-value change counts as behavioral** (e.g. a timeout, retry count, or feature-flag default), as does any conditional or logic edit.
Is this review accurate? Use 👍 or 👎 to rate it

If you want to tell us more, use /gs feedback e.g. /gs feedback this review doesn't make sense, I disagree, and it keeps repeating over and over

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ed815e9: "skipped" now means the task modifies no file; any file edit with no behavior change (code comments, doc files, formatting) is LOW. README updated to match. Same in lb-claude-plugins#147.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 08:50
Copilot AI dismissed their stale review, a newer Copilot review was requested October 21, 2026 08:50

@gitstream-cm gitstream-cm Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✨ PR Review

The PR adds a "skipped (not a code task)" verdict and tightens the edit hold so a skip alone doesn't satisfy it. The tightened check in the PreToolUse edit hold can be defeated by the skill's own example text in the transcript.

1 issues detected:

🐞 Bug - A whole-transcript grep for the verdict pattern is satisfied by the skill's own example text, so it can't tell a real grade from a skip. 🛠️

Details: The edit hold now requires LinearB: .+ (LOW|MEDIUM|HIGH) effort anywhere in the whole transcript, which is meant to rule out a skip-only run. The transcript also contains the loaded SKILL.md content, which includes the literal example > LinearB: api-service — HIGH effort (...). Once the skill has been invoked, the grep therefore matches the example even if the agent only printed the skipped line or nothing. The hold is then bypassed and the skipped-then-edit case is not enforced. The check also isn't limited to assistant text, unlike the verdict_state jq filter.

File: plugins/agentic-advisor/hooks/agentic-advisor-trigger.sh (201-201)

🛠️ A suggested code correction is included in the review comments.

Generated by LinearB AI and added by gitStream.
AI-generated content may contain inaccuracies. Please verify before using.
💡 Tip: You can customize your AI Review using Guidelines Learn how

if skill_ran; then
# Skill already graded this session -> allow the edit. A "skipped (not a code task)" line alone doesn't count: editing now is code work.
# If a verdict names this repo, mark it handled so later prompts here aren't nudged.
if skill_ran && grep -qE 'LinearB: .+ (LOW|MEDIUM|HIGH) effort' "$transcript" 2>/dev/null; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐞 Bug - Grade check matches skill example: Restrict the match to assistant-authored text or Bash printf tool_use input, for example by reusing the jq logic from verdict_state or by grepping only lines containing "role":"assistant". Alternatively, make the pattern exclude the SKILL.md example.

Suggested change
if skill_ran && grep -qE 'LinearB: .+ (LOW|MEDIUM|HIGH) effort' "$transcript" 2>/dev/null; then
if skill_ran && grep -E '"role": ?"assistant"' "$transcript" 2>/dev/null | grep -qE 'LinearB: .+ (LOW|MEDIUM|HIGH) effort'; then
Is this review accurate? Use 👍 or 👎 to rate it

If you want to tell us more, use /gs feedback e.g. /gs feedback this review doesn't make sense, I disagree, and it keeps repeating over and over

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: the edit hold now uses the same parser as the verdict reminder (assistant text and printf calls after the latest skill invocation) and requires a LOW/MEDIUM/HIGH grade, so the loaded SKILL.md example no longer counts. The skip-then-edit test now includes the skill text with its example line, and fails on the previous hook. (fe76b15)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The edit gate can mistake loaded skill examples for an actual grade and allow ungraded edits.

1 open issue

Review effort: Balanced

Comment thread plugins/agentic-advisor/hooks/agentic-advisor-trigger.sh Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 21, 2026 08:56

@gitstream-cm gitstream-cm Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✨ PR Review

LGTM

Generated by LinearB AI and added by gitStream.
AI-generated content may contain inaccuracies. Please verify before using.
💡 Tip: You can customize your AI Review using Guidelines Learn how

gitstream-cm[bot]
gitstream-cm Bot previously approved these changes Oct 21, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gitstream-cm

gitstream-cm Bot commented Oct 21, 2026

Copy link
Copy Markdown

🚦 Automatic AI Review skipped

This pull request has used all 3 of its automatic AI Reviews.

✨ Comment /gs review to run one now.
No arguments needed. It uses the same settings as your automatic reviews: gitStream Settings in Managed Mode, your CM files in Self-Managed.

@gitstream-cm
gitstream-cm Bot dismissed their stale review October 21, 2026 09:01

Review dismissed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Grade enforcement can be bypassed, and mixed skipped/graded sessions misattribute grading telemetry.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Skipped runs incorrectly count toward grading telemetry

plugins/​agentic-advisor/​skills/​agentic-advisor/​SKILL.md:48

Skipped work is still counted in grading telemetry when a later task in the same session receives a grade. In agentic-advisor-report.sh, parse_tokens() keeps the first Skill timestamp and waits for the first LOW/MEDIUM/HIGH verdict, without recognizing skipped outcomes. It therefore counts the skipped run and intervening answer tokens and elapsed time as grading overhead. Exclude skipped runs from grading intervals and add a mixed-session regression; the new skipped-only test does not exercise this path.

1 resolved since last review

Review effort: Balanced

if [ -n "$state" ] && : > "$vdone" 2>/dev/null && [ "$state" = "pending" ]; then
dbg "PreToolUse: reminded the agent to print the verdict line"
note "LinearB: your effort grade isn't recorded yet. It's only recorded from the verdict line, not from thinking, so print it now: > LinearB: <repo> — <LOW|MEDIUM|HIGH> effort (<evidence>) — <plan>. If you aren't writing text, run: printf '%s\\n' '> LinearB: <repo> — <LEVEL> effort (<evidence>) — <plan>'"
note "LinearB: your effort grade isn't recorded yet. It's only recorded from the verdict line, not from thinking, so print it now: > LinearB: <repo> — <LOW|MEDIUM|HIGH> effort (<evidence>) — <plan>. If you aren't writing text, run: printf '%s\\n' '> LinearB: <repo> — <LEVEL> effort (<evidence>) — <plan>'. If no code will change, print > LinearB: <repo> — skipped (not a code task) instead."
# Skill already graded this session -> allow the edit. A "skipped (not a code task)" line alone doesn't count: editing now is code work.
# If a verdict names this repo, mark it handled so later prompts here aren't nudged.
# "pending" only when parsed and no grade yet; a parse failure prints nothing and fails open.
if skill_ran && [ "$(verdict_state "$GRADED_RE")" != "pending" ]; then
Copilot AI balanced review requested due to automatic review settings October 21, 2026 09:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The grade requirement also denies read-only delegation after a valid skipped verdict.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Skipped verdict incorrectly requires grading for read-only delegation

plugins/​agentic-advisor/​hooks/​agentic-advisor-trigger.sh:204

This check also handles Task/Agent, not just edits. After the skill prints a valid skipped verdict, delegating read-only work to a general-purpose or custom agent is denied if the session/repo hold has not already been used. The denial then requires a LOW/MEDIUM/HIGH grade for work that should remain ungraded. Accept the skipped verdict for delegation while retaining the real-grade check for source edits. Add a skipped-then-read-only-delegation regression test alongside the existing skipped-then-edit test.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants