Skip to content

docs(proposals): extension as a primary data source (#1175) - #1178

Merged
JOY (JOY) merged 1 commit into
devfrom
claude/1175-extension-data-source-proposal
Oct 28, 2026
Merged

JOY (JOY) merged 1 commit into
devfrom
claude/1175-extension-data-source-proposal

Conversation

@JOY

@JOY JOY (JOY) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Docs-only proposal, for JOY to decide on, about expanding the browser extension into a primary data source.

  • Docs/Proposals/2026-10-28-extension-data-source.html: Vietnamese decision document (dark theme).
  • Docs/Proposals/2026-10-28-extension-data-source.md: short English companion.

Recommendation: Option B. With consent, the extension sends a minimal observation (price, stock, SKU, listing URL, timestamp, marketplace; no personal data) into a quarantine table. DOS/Rate-Services verifies it against the crawler and independent submitters before it becomes a public price. The extension never fetches pages the user did not open. Trust scoring, thresholds and crawler cross-checks stay in DOS/Rate-Services.

The proposal also lists mismatches found in the current code that should be fixed under any option:

  • The opt-in copy understates the payload.
  • CHROMEWEBSTORE.md has drifted from manifest.json.
  • The *.tiktok.com host scope is broad.
  • The privacy policy does not mention the extension.

Scope

No change to extension behavior, the ingest API, store listings, secrets or DOS/Rate-Services.

Review

A fresh-context subagent reviewed the proposal against the code. It found 3 P1, 6 P2 and 4 P3 issues, and all were fixed in round 1:

  • The IP and independence contradiction is resolved: a salted network-range hash is disclosed as personal data.
  • The private runner detail is removed.
  • submitter_id is now issued by the server.
  • The per-instance rate limit is described correctly.
  • A metric for listings the crawler cannot reach is added.
  • Chrome policy wording now matches the FAQ.
  • Assumptions are labelled as assumptions.

Closes #1175

🤖 Generated with Claude Code


📌 TL;DR

This PR introduces a comprehensive architectural proposal (in both Vietnamese HTML and English Markdown) to repurpose the browser extension as a primary, consent-based data source for price tracking, bypassing bot-detection walls that currently block private crawlers. It defines a new "quarantine and verify" data pipeline, establishes strict privacy and anti-poisoning boundaries between the public extension and private services, and outlines a phased implementation roadmap.

🎯 Type of Change

  • 🚀 New feature
  • 🐛 Bugfix
  • 🧹 Refactor
  • ⚡ Performance
  • 📚 Documentation
  • ⚙️ CI / Configuration

🔍 Changes Walkthrough

File Summary of Changes
Docs/Proposals/2026-10-28-extension-data-source.html Adds a detailed, styled decision document in Vietnamese. It audits the current extension's data collection, identifies compliance mismatches (e.g., privacy policy drift, overly broad TikTok host permissions), analyzes competitor models (Honey, Keepa, BeeCost), and details the proposed "Option B" architecture including payload schemas, anti-poisoning strategies, and legal considerations.
Docs/Proposals/2026-10-28-extension-data-source.md Adds the English summary of the proposal. It outlines the current state of the extension's lookup and ingest flows, defines the public/private repository boundary for the new data pipeline, compares three architectural options (recommending passive observation with quarantine), and lists milestones (M0–M4) and success metrics for the project.

📊 Architectural Flow

flowchart TD
    subgraph User_Browser [User Browser - Public]
        A[User Views Product Page] --> B[Extension Content Script]
        B --> C{User Opt-In?}
        C -- No --> D[Read-only Lookup<br/>GET /lookup]
        C -- Yes --> E[Send Minimal Observation<br/>POST /ingest]
    end

    subgraph Public_Repo [DOS/Rate - Public]
        D --> F[Web API]
        E --> F
        F --> G[Basic Validation<br/>Zod, Rate Limit, Host Check]
        G --> H[Quarantine Table<br/>Unverified Data]
    end

    subgraph Private_Repo [DOS/Rate-Services - Private]
        H --> I[Verification Engine]
        I --> J{Cross-Check}
        J -- Match Crawler --> K[Promote to Public Price History]
        J -- Match Independent Submitters --> K
        J -- Mismatch / Low Trust --> L[Discard / Keep Quarantined]
        
        M[Crawler Baseline] --> J
        N[Submitter Trust Scoring] --> I
    end

    K --> O[Public Price History API]
    O --> A
Loading

Decision document for JOY (Vietnamese, dark HTML) with an English
companion. Covers the current extension and ingest quarantine, how
Honey, Keepa and BeeCost source data, a minimal observation payload,
consent and store review, anti-poisoning, cold start, per-marketplace
coverage, legal notes, why the client is public while verification
stays in DOS/Rate-Services, options, milestones, metrics and open
questions. No product code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Code review skipped — your organization has no extra usage available to pay for this review.

If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.

Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.

@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
rate Ready Ready Preview Oct 28, 2026 4:20am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 28, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 45 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4a8bb75d-11d7-41e1-a48e-1171eb43c1f4
Commits

Reviewing files that changed from the base of the PR and between 20809d6 and 80e79cb.

Files selected for processing: 2
  • Docs/Proposals/2026-10-28-extension-data-source.html
  • Docs/Proposals/2026-10-28-extension-data-source.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

dos[bot]
dos Bot approved these changes Oct 28, 2026

@dos dos Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⏱️ Code Review completed (2 eligible files · 45,612 chars · 1 model call(s) · 2 PR unit(s))

Whole diff with bounded context at base 70f310f / head 80e79cb.

✅ CLEAN_PASS: Complete diff reviewed

@JOY
JOY (JOY) merged commit 09c7a96 into dev Oct 28, 2026
19 checks passed
@JOY
JOY (JOY) deleted the claude/1175-extension-data-source-proposal branch October 28, 2026 04:27

This branch was successfully deployed

1 active deployment
Preview — 80e79cb0 Deployed Oct 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: extension as a primary data source (decision doc for JOY)

1 participant