Skip to content

Fix -o for large and unfinished batches, usage exit codes, and CSV columns - #35

Merged
jclusso merged 9 commits into
masterfrom
output-fixes
Oct 7, 2026
Merged

jclusso merged 9 commits into
masterfrom
output-fixes

Conversation

@jclusso

@jclusso jclusso commented Oct 1, 2026

Copy link
Copy Markdown
Member

-o silently wrote a header-only file for batches that return download_file or are still verifying, and usage errors exited 1 with unknown even though the README promises 4 and invalid_input for bad input. This also makes account status --json pass the API body through like the other commands, shows the stored login email in status for API-key logins, and gives CSV exports every verify field with blanks where the API sent nothing.

…lumns

`-o` silently wrote a header-only file for batches that return `download_file` or are still verifying, and usage errors exited 1 with `unknown` even though the README promises 4 and `invalid_input` for bad input. This also makes `account status --json` pass the API body through like the other commands, shows the stored login email in `status` for API-key logins, and gives CSV exports every verify field with blanks where the API sent nothing.
The message asserted an API retention behavior nobody had confirmed.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 14:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Leaf-command usage errors remain misclassified, and large downloads use an unbounded multi-copy memory pipeline.

Review effort: Balanced
Findings: 3 Medium severity

Open (3)
What changed in this PR

Fixes batch exports, CLI usage error handling, JSON passthrough, and stored-account display behavior.

Changes:

  • Downloads and converts large batch results while producing complete CSV schemas.
  • Classifies common usage failures as invalid_input.
  • Preserves account API JSON fields and displays stored login emails.
File Description
README.md Documents exports, CSV columns, and usage errors.
internal/​output/​file.go Expands CSV serialization and raw-field handling.
internal/​output/​file_test.go Tests CSV schemas and omitted values.
internal/​output/​download.go Converts downloaded batch results.
internal/​output/​download_test.go Tests ZIP, CSV, and JSON conversion.
internal/​api/​download.go Fetches presigned result files.
internal/​api/​download_test.go Tests download security and errors.
cmd/​usage_errors.go Classifies command-line usage failures.
cmd/​usage_errors_test.go Tests usage error behavior.
cmd/​status.go Displays stored account emails.
cmd/​skill.go Rejects unknown skill subcommands.
cmd/​root.go Installs root usage-error handlers.
cmd/​batch.go Handles incomplete and downloadable batches.
cmd/​batch_output_test.go Tests batch file output flows.
cmd/​auth_display_test.go Tests account JSON and email display.
cmd/​account.go Passes account JSON through unchanged.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cmd/root.go
Comment thread cmd/usage_errors.go
Comment thread internal/api/download.go
A 1M-row `download_file` export was parsed into memory whole before conversion, which could reach gigabytes. CSV output now converts the download one row at a time through the same encoder inline results use, and atomic writes stream to the temp file.
Leaf commands using plain `cobra.NoArgs` still exited 1 with `unknown`, so the root now wraps every command's `Args` check instead of relying on each command to opt in. A trailing `--json=false` after a bad flag also now wins over an earlier `--json`.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changed behaviors are consistently implemented, documented, and covered by focused tests.

Review effort: Balanced
Findings: None

Resolved since last review (3)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The new archive, network, serialization, and CLI error paths are broad enough to warrant final human validation.

Review effort: Balanced
Findings: None

The README spelled out download handling, CSV column order, and status display rules that users never act on; `-o` and the error codes are already documented where they matter.
A completed large batch printed a ~1,700 character presigned URL with broken indentation. It now shows the same per-state summary as small batches, built from `total_counts`, and points to `-o`; the URL stays in `--json` for scripts.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Empty partial exports and large-batch summary formatting still have correctness and output-consistency issues.

3 open findings

🧠 Review effort: Balanced

Comment thread cmd/batch.go
Comment thread internal/output/human.go
Comment thread internal/output/human.go
`--partial -o` on a batch with nothing verified yet still wrote a header-only file, and a large batch that reports only top-level `total`/`processed` showed "Verified 0 emails". The `--quiet` help now says it hides hints, progress, and status messages, since results still print.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Large-batch summaries omit the established duplicate count, producing inconsistent totals.

0 open findings

3 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Batch summary drops duplicate counts from typed aggregate

internal/​api/​types.go:59

total_counts also contains duplicate (see the established API fixture in cmd/batch_e2e_test.go:195), but the typed aggregate drops it. A download-only batch with 10 state results and 5 duplicates can therefore print Verified 15 emails while the displayed breakdown sums to 10. Model the duplicate count and account for it in PrintBatchSummary (either display it or exclude it from the verified total according to the API semantics).

🧠 Review effort: Balanced

The server's `processed` count includes duplicates, which carry no state, so a batch with duplicates claimed more verified emails than its breakdown added up to. The count now matches what was verified, and skipped duplicates are noted.
@jclusso

jclusso commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

Addressed the duplicate count finding in 228c004. The backend computes processed as the four state counts plus duplicate, so the summary now reports processed - duplicate as verified and notes skipped duplicates when there are any, keeping the headline equal to the breakdown.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Download-only batches without count fields are incorrectly reported as having verified zero emails.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Medium severity Avoid reporting zero verified emails when count is unavailable

internal/​output/​human.go:426

A download-only response can omit both progress shapes (the new TestBatchGet_DownloadFileHint fixture does exactly this). In that case verified remains zero, so the completed large batch is now reported as Verified 0 emails. Track whether a count was actually available and render a count-free completion result (for example, Batch complete) when it was not.

🧠 Review effort: Balanced

@jclusso

jclusso commented Oct 7, 2026

Copy link
Copy Markdown
Member Author

Not changing this. A download-only response without counts cannot come from the API: batch_payload and sample_batch_payload in the backend always set total_counts, and download_file is only ever added to that same payload. Only a breaking API change would produce that shape, so the fallback would be dead code.

@jclusso
jclusso requested a balanced review from Copilot October 7, 2026 19:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The export, usage-error, CSV, and authentication changes are coherent and covered by focused tests.

0 open findings

🧠 Review effort: Balanced

@jclusso
jclusso merged commit f72b8cd into master Oct 7, 2026
6 checks passed
@jclusso
jclusso deleted the output-fixes branch October 7, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants