Skip to content

UN-4504 [DEV] Add v1 PDF highlight layer and highlight normalisers - #2407

Open
muhammad-ali-e wants to merge 13 commits into
UN-4503-v1-pdf-pagesfrom
UN-4504-v1-pdf-highlights
Open

muhammad-ali-e wants to merge 13 commits into
UN-4503-v1-pdf-pagesfrom
UN-4504-v1-pdf-highlights

Conversation

@muhammad-ali-e

Copy link
Copy Markdown
Contributor

What

  • lib/pdf/highlights.ts: normalisers that turn every highlight shape our backends send into one Highlight type.
  • components/patterns/document-viewer/HighlightLayer.tsx: draws a page's highlights.
  • PdfPageList now takes highlights, activeHighlightId and onHighlightClick.

This is part of the v1 DocumentViewer (P1-T21, UN-4390).

Why

Prompt Studio, HITL, Agentic Prompt Studio and the agentic table each send highlights in a different shape, and v0 decoded them in three places. In v1 the viewer gets one shape, and the decoding lives in one tested module.

How

  • The Highlight shape: {id, page, x, y, w, h, variant: "line" | "box", tone: "default" | "active" | "low-confidence", confidence?}.
    • Positions are fractions of the page box.
    • The layer places them as percentages, so they need no recalculation on zoom. In the UN-4501 spike they were exact at 50 / 100 / 200 %.
  • LLMWhisperer lines ([page, base_y, height, page_height, line_confidence?]):
    • The band runs from base_y - height to base_y, full width.
    • The 5th element is read by type, not by length. A number is a line confidence. An object is a word box.
  • Word boxes:
    • normaliseWords(line, words) takes LLMWhisperer confidence_metadata as a separate list, rather than appending it to the line as v0 HITL did.
    • normaliseLine still reads the appended form. The appended form isn't dead code: the agentic table worker sends [page, y, h, 792, {offset, width, page_width}] today (see EXPECTED_COORDS in workers/plugins/agentic_table/tests/test_highlight_data.py).
    • When page_width is present it's used. Otherwise x comes from the PDF page's aspect ratio: page_height × pdfWidth / pdfHeight. v0 divided by the page height, which puts word boxes too far left on portrait pages. There's a regression test for this.
  • Agentic Prompt Studio ([page, left%, top%, width%, height%, hasMetadata]): divided by 100.
  • All shapes:
    • Pages stay 0-based, as the producers send them.
    • Padding ([0, 0, 0, 0]), zero-size and junk entries are dropped.
    • Boxes are clamped to the page.
    • A box that spans the page's width counts as a line, so a full-width "word" from the agentic table doesn't look like a word box.
  • Low confidence: below 0.9 by default, the same cut-off LLMWhisperer uses for word boxes. It's configurable with lowConfidenceBelow.
  • HighlightLayer:
    • Tones come from tokens: primary for default and active (active adds a ring), destructive for low confidence. Highlights use multiply blending.
    • With an onHighlightClick, each highlight is a labelled button (e.g. "Highlight on page 1, 62% confidence"), with aria-current on the active one and the v1 focus ring. Without a handler, highlights are plain non-interactive boxes.
    • The layer is pointer-events-none, so only the highlights themselves take clicks.
    • Positions are set as custom properties through a ref, as in UN-4503, because inline style is banned. The open question with Praveen covers this too.
  • PdfPageList: groups highlights by page once and passes each memoised PdfPage only its own stable array. Scrolling therefore doesn't re-render every page.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

  • No. All the code is new and lives under src/v1/. Nothing in v0 imports it, and v0's highlight code is unchanged.

Database Migrations

  • None

Env Config

  • None

Relevant Docs

  • The input shapes are recorded on UN-4504. Vishnu confirmed them in #engg-unstract-frontend-v1-migration on 2026-10-26, and they were checked against the LLMWhisperer source.

Related Issues or PRs

Dependencies Versions

  • None

Notes on Testing

  • bun run typecheck:v1, lint:v1 and test:v1 pass (99 unit tests, 56 boundary fixtures). biome ci src/ shows no errors.
  • Normaliser tests use payloads shaped like each producer's real output:
    • Prompt Studio LLMWhisperer lines, including padding and a missing confidence
    • HITL word boxes, both as a list and appended, with the v0 x-axis regression
    • the agentic table's EXPECTED_COORDS, including page_width taking priority over the aspect ratio
    • the agentic studio's transformHighlightData output
    • clamping and junk entries
  • Layer tests cover:
    • % placement
    • tones and the active tone
    • no buttons without a handler
    • click → onHighlightClick with the highlight
    • accessible labels
  • A page-list test checks that each highlight is drawn on its own page.
  • In-browser checks of highlight rectangles against page rectangles at 50 / 100 / 200 % come in UN-4507.

Screenshots

  • None yet; the gallery comes in UN-4507.

Checklist

I have read and understood the Contribution Guidelines.


Generated by Claude Code

- lib/pdf/highlights turns every producer's shape into one Highlight
  {id, page, x, y, w, h, variant, tone, confidence?}, in fractions of
  the page box, so highlights hold their place at any zoom:
  - LLMWhisperer line_metadata lines, with line confidence read by type
  - word boxes, both as a separate list (confidence_metadata) and as the
    5th element the agentic table worker sends with page_width; without
    page_width, x comes from the PDF page's aspect ratio rather than
    v0's division by page height
  - Agentic Prompt Studio percentage boxes
  Padding, zero-size and junk entries are dropped, and boxes are clamped
  to the page. A box spanning the page reads as a line.
- HighlightLayer draws a page's highlights in default, active and
  low-confidence tones, as buttons reporting onHighlightClick when a
  handler is given.
- PdfPageList takes highlights, activeHighlightId and onHighlightClick
  and hands each memoised page only its own highlights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7trgcAzu6jxDnit3eeozm

Copy link
Copy Markdown
Contributor Author

pre-commit.ci - pr is red because of the base branch, not this PR.

The failing hook is check-shebang-scripts-are-executable, on frontend/scripts/check-v1-styles.mjs. That file has a #! line but no executable bit. It comes from the tokens PR #2399 (UN-4299-design-tokens-themes); this PR doesn't touch it. The hook doesn't fail on #2399 itself only because pre-commit.ci doesn't run on that PR. Running pre-commit run --from-ref $(git merge-base origin/main HEAD) locally reproduces it, and this PR's own files pass every hook.

There's no fix to port yet. The fix is a one-line change on #2399: git add --chmod=+x frontend/scripts/check-v1-styles.mjs (or drop the shebang). Once #2399 has it, this check should go green here without any change to this PR.


Generated by Claude Code

@muhammad-ali-e
muhammad-ali-e marked this pull request as ready for review November 2, 2026 07:00
@greptile-apps

greptile-apps Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

via Greptile

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Adds one shared highlight format and draws highlights over PDF pages.

  • Different PDF highlight inputs now share one page-relative shape.
  • PDF pages show highlights that stay aligned as they zoom.

Tests were reviewed but not run. Browser capture was blocked by missing frontend dependencies and the lack of a mounted viewer page at this revision.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Line coordinates] --> N[Normalisers]
  B[Word coordinates] --> N
  C[Agentic coordinates] --> N
  N --> H[Highlight list]
  H --> G[Group by page]
  G --> P[PdfPage]
  P --> L[HighlightLayer]
  L --> D[Plain boxes]
  L --> E[Clickable buttons]
Loading

Reviews (3) · Last reviewed commit: "Merge branch 'UN-4503-v1-pdf-pages' into..." · Reviewed by Greptile

@praveen-formido praveen-formido left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Highlights and normalisers. Two correctness items in highlights.ts (duplicate ids across lines, hasMetadata dropped) and one token/theme item on HighlightLayer: mix-blend-multiply over ui:bg-card is invisible in the dark theme until the canvas paints. The rest are smaller.

One upstream note, not for this PR: the cloud agentic_extraction worker emits a top y where agentic_table emits a baseline, and the cloud transformHighlightData subtracts the height again. That predates this stack, but it will surface here once the cloud plugin feeds these normalisers.

Comment thread frontend/src/v1/lib/pdf/highlights.ts Resolved
Comment thread frontend/src/v1/lib/pdf/highlights.ts Resolved
Comment thread frontend/src/v1/lib/pdf/highlights.ts Resolved
Comment thread frontend/src/v1/lib/pdf/highlights.ts Resolved
muhammad-ali-e and others added 2 commits November 3, 2026 10:00
# Conflicts:
#	frontend/src/v1/components/patterns/document-viewer/PdfPage.tsx
#	frontend/src/v1/components/patterns/document-viewer/PdfPageList.tsx
…d-only blend, per-page active id

- normaliseWords puts the line's page and position in the default id, so
  words on different lines no longer share ids.
- Line entries keep their confidence for the label but are not toned; v0
  never toned lines, and line confidence on scans sits below 0.9.
- Warn with the page index when word boxes are dropped for want of a page
  width.
- Highlights use mix-blend-multiply only once the page canvas holds a
  picture, so they stay visible on the dark placeholder.
- Only the page holding the active highlight receives its id, so moving it
  re-renders two pages, not all of them.
- HighlightLayer builds the style and data props once for both branches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R7trgcAzu6jxDnit3eeozm
@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants