Skip to content

fix: authenticate the GitHub repo-size preflight and log its outcome - #124

Merged
nodo merged 15 commits into
mainfrom
claude/practical-gauss-nj0xyd
Oct 21, 2026
Merged

nodo merged 15 commits into
mainfrom
claude/practical-gauss-nj0xyd

Conversation

@nodo

@nodo nodo commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

When bootstrapping from a github.com source without --target-max-pack-bytes, git-sync asks the GitHub REST API (GET /repos/{owner}/{repo}) for the repository size. It switches to 512 MiB batches when the size is 1.5 GiB or more. That lookup sent no credentials. For a private repository GitHub answers 404 to an anonymous caller, so every private repository bootstrapped as a single pack, whatever its size.

Nothing reported this:

  • Every failure (non-200, transport error, parse error) returned silently.
  • The "selected batched mode" message only showed with verbose logging.

Public repositories were exposed too: anonymous calls share GitHub's 60 requests/hour limit per egress IP.

Changes

Authenticated lookup

  • The size request now carries the source connection's credentials (basic or bearer): an explicit token, a user:token@ in the source URL, or credentials the git credential helper supplied when the git server asked for them. A public repository never asks, so with only a helper its lookup stays anonymous. A URL username without a password is not sent: it may be an account name.
  • If the API refuses the token with 401 or 403 (for example an SSO-unauthorized PAT), the lookup retries once anonymously. A public repository is never worse off than before. If the retry fails too, the refusal is reported rather than hidden behind the retry's 404.
  • The new HTTPConn.CredentialsFor(dest) decides which credentials may go to the API, checking the real destination. Each credential is measured from the URL it is bound to (the endpoint for explicit and URL credentials, the looked-up host for helper ones), the destination must be same-site with it, and with TLS verification off nothing crosses to another host. So helper credentials looked up for a redirect host stay with that host, and an API base on another site gets nothing.
  • GitHubOwnerRepo now treats github.com as GitHub only on its scheme's default port. The connection's redirect guard still strips credentials from any hop that leaves the API's site, and the logged authenticated reflects the request that actually produced the answer.

Visible outcome

  • Every lookup that is attempted is logged through Options.BootstrapLogger. This does not need verbose mode.
  • Fields: repository, authenticated, http_status, refused_auth_status, refused_auth_error, size_kb or error, threshold_kb, mode, target_max_pack_bytes.
  • A failed lookup logs at WARN and also prints a notice that the bootstrap starts with a single pack, so CLI users see it without -v (it can still switch to batches if the target rejects the pack). The notice names the token refusal when there was one. Under -v the failure is printed once, and the verbose log and BootstrapLogger use the same record name.

Hardening

  • A reported size of 0 counts as unknown, like a missing or negative size: bootstrap only runs with source refs, so 0 means GitHub has not sized the repo yet. It warns and prints the notice.
  • A cancelled run reports nothing. Every error text (transport errors included) is cleaned before it is logged or printed.
  • A 401/403 counts as a refused token only when the request still carried it, so a redirect that stripped the credentials is neither retried nor blamed on the token.
  • Wiki sources (owner/repo.wiki) get no lookup: the API has nothing to size.
  • Each lookup request has its own 30s timeout, so a slow refusal does not starve the anonymous retry.
  • GitHub's error message is cleaned (control characters including C1, bidi formatting) and truncated before it is logged.

Docs

  • docs/usage.md documents the lookup and its credential scope.
  • Updated the BootstrapLogger doc comments and CHANGELOG.md.

Tests

New file internal/strategy/bootstrap/github_preflight_test.go, written test-first. It covers:

  • a private repo: 404 without auth, 200 with auth, so batched (also end-to-end through Execute);
  • a bearer token, user:token@ URL credentials, and a bare URL username not being sent;
  • no credentials with TLS verification off, or to an API base on another site;
  • lookup failure: one-shot, a WARN record and a notice;
  • the anonymous retry on 401/403, the refusal reported when the retry fails too, and no retry on other failures;
  • a transport error, and unusable bodies (not JSON, no size, negative size);
  • sanitizing GitHub's error message;
  • a small repo staying one-shot without a warning, and one reporting size 0 warning as unknown;
  • a cancelled run, unclean transport errors, an unauthenticated 403 after a redirect, and the per-attempt timeout;
  • skipped lookups (caller already batching, non-default port, non-GitHub host);
  • a redirect that leaves the API's site dropping credentials (asserting the target was reached, and logged as unauthenticated);
  • CredentialsFor in internal/gitproto/smarthttp_test.go: explicit, helper and URL credentials, helper credentials bound to a redirect host, and TLS verification off.

The preflight tests reach a local TLS server as https://api.github.com, so the destination check runs as in production.

The new tests fail against the previous implementation. go test ./..., go test -race on the affected packages and golangci-lint run (v2.11.4) all pass.


Generated by Claude Code


Note

Medium Risk
Touches credential scoping for cross-host API calls and bootstrap batching decisions; mistakes could leak tokens or mis-size packs, but changes are heavily tested and fail closed on redirects/TLS-off.

Overview
Fixes batched bootstrap for large private GitHub repos by sending source credentials on the GET /repos/{owner}/{repo} size preflight (scoped to api.github.com via new HTTPConn.CredentialsFor), with same-site/TLS rules aligned to redirect handling and helper creds bound to authOrigin.

The preflight moves into github_preflight.go with structured BootstrapLogger / PushLogger records, CLI notices on lookup failure, 401/403 → one anonymous retry, stricter GitHubOwnerRepo parsing, and sanitize.Line for API/error text in logs. Docs and changelog describe the behavior.

Reviewed by Cursor Bugbot for commit d0e7b78. Configure here.

Copilot AI balanced review requested due to automatic review settings October 19, 2026 15:39

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread internal/strategy/bootstrap/bootstrap.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential reuse can cross service boundaries, and URL-embedded credentials remain unsupported by the preflight.

2 open issues
What changed in this PR

Updates GitHub’s repository-size preflight to support authenticated lookups and make bootstrap mode selection visible.

Changes:

  • Adds authenticated requests with anonymous fallback on 401/403.
  • Adds outcome logging, failure notices, timeout and response safeguards.
  • Adds regression tests and updates documentation.
File Description
unstable/​client.go Documents size-lookup diagnostics.
internal/​strategy/​bootstrap/​github_preflight_test.go Tests authentication, fallback, logging and batching.
internal/​strategy/​bootstrap/​bootstrap.go Implements authenticated preflight and outcome reporting.
docs/​usage.md Explains lookup behavior and credential scope.
client.go Updates logger documentation.
CHANGELOG.md Records the preflight changes.

Review effort: Balanced

Comment thread internal/strategy/bootstrap/bootstrap.go Outdated
Comment thread internal/strategy/bootstrap/bootstrap.go Outdated

nodo commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

govulncheck is red because of new Go standard-library advisories against the pinned go1.26.6 toolchain, not because of this change.

That bump isn't carried in this PR yet. It changes the shared toolchain and lint pins, so I'm waiting on a decision: port it here, or let this PR go green once #123 lands and main is merged in. A re-run would fail identically, so I haven't triggered one.


Generated by Claude Code

nodo commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Ported the Go 1.27.2 bump from #123 as 58e81ac (cherry-pick of 8484a5f). It no-ops once #123 lands. Verified locally on go1.27.2:

  • govulncheck reports 0 vulnerabilities in called code;
  • go test ./... passes;
  • golangci-lint v2.15.0 reports 0 issues.

Generated by Claude Code

@nodo
nodo force-pushed the claude/practical-gauss-nj0xyd branch from 58e81ac to 6e75338 Compare October 19, 2026 16:28

nodo commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

Update: the Go 1.27.2 port (58e81ac) is no longer on this branch. 6e75338 was pushed on top of 068547f, so go.mod is back on go1.26.6 and govulncheck fails again for the same standard-library advisories as main.

58e81ac itself passed every check, govulncheck included. The fix is still #123's toolchain bump: either re-apply it here or merge main in once #123 lands.


Generated by Claude Code

@nodo

nodo commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run

@nodo
nodo requested a balanced review from Copilot October 19, 2026 17:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-service credential reuse and redirect-sensitive authentication warrant final human security review.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@pjbgf pjbgf left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One low-severity finding from a git-conformance, security and performance review. The rest of the credential scoping (redirect binding, helper origin, TLS-off, log sanitisation) checked out.


Generated by Claude Code

Comment thread internal/strategy/bootstrap/bootstrap.go Outdated
pjbgf
pjbgf previously approved these changes Oct 20, 2026
@nodo
nodo force-pushed the claude/practical-gauss-nj0xyd branch from 29f63a4 to 91e007c Compare October 20, 2026 08:46
@nodo

nodo commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run

@nodo
nodo requested a balanced review from Copilot October 20, 2026 10:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Custom transports with missing response provenance can incorrectly suppress the anonymous retry after token refusal.

1 open issue

Review effort: Balanced

Comment thread internal/strategy/bootstrap/github_preflight.go Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@nodo

nodo commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run

@nodo
nodo requested a balanced review from Copilot October 20, 2026 10:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Credential scoping, retry behavior, and diagnostics have targeted regression coverage, with no unresolved blocking findings.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@nodo

nodo commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run

@nodo
nodo requested a balanced review from Copilot October 21, 2026 14:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved blocking issues remain, and targeted regression coverage addresses credential scoping, retries, diagnostics, and batching selection.

No open issues 🎉

Review effort: Balanced

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d0e7b78. Configure here.

nodo commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator Author

govulncheck is red on d0e7b78 because of a newly published standard-library advisory, GO-2026-6809: crypto/x509 should distrust compromised intermediate certificates. This PR's code doesn't cause it. The advisory affects go1.27.2, which main also pins, and is fixed in go1.27.3. main's last scan predates the advisory, so main will fail the next one too. Re-running the check would fail the same way, so I haven't.

No fix PR exists yet. Verified locally:

  • go1.27.2: GO-2026-6809 reported;
  • go1.27.3: govulncheck reports 0 vulnerabilities in called code, go test ./... passes, and golangci-lint v2.15.0 reports 0 issues.

Proposed patch:

--- a/go.mod
+++ b/go.mod
-toolchain go1.27.2
+toolchain go1.27.3
--- a/mise.toml
+++ b/mise.toml
-go = { version = '1.27.2', ...
+go = { version = '1.27.3', ...

This can land on its own against main (preferable) or ride along here.


Generated by Claude Code

claude and others added 15 commits October 21, 2026 17:53
The size lookup that selects batched bootstrap for repositories over
1.5 GiB sent no credentials. GitHub answers 404 to an anonymous request
for a private repository, so every private repository, whatever its
size, was bootstrapped as a single pack, and nothing said so: every
failure returned silently and the selection message needed verbose mode.

Attach the source credentials to the lookup, which also moves public
lookups off the anonymous rate limit, and log every lookup's outcome
(HTTP status or error, reported size, chosen mode) through the
bootstrap logger, at warning level when the size is unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
- Treat github.com as GitHub only on the default port of its scheme, so
  credentials for a same-named service on another port never reach the
  GitHub API.
- Retry once anonymously when the API refuses the token (401/403), so a
  public repository is never worse off than before credentials were sent.
- Surface a failed lookup as a notice too: the CLI wires no structured
  logger, so it previously stayed silent without -v.
- Treat a reported size of 0 as unknown (GitHub reports 0 until it has
  computed the size), bound the lookup with a timeout, and sanitize and
  truncate GitHub's error message before logging it.
- Update the BootstrapLogger docs to cover the new record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
A token written into the source URL authenticates git through
req.URL.User and never becomes HTTPConn.Auth, so the lookup stayed
anonymous and a private repository still bootstrapped as one pack.
Carry the userinfo password over as basic auth when the connection
has no credentials of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
Helper credentials looked up for a host /info/refs redirected to are
stored in HTTPConn.Auth while Endpoint() still reports the configured
URL, so the size lookup could send one host's credentials to the GitHub
API. Record the URL helper credentials were looked up for and expose
EndpointAuth, which returns only credentials bound to the endpoint's own
site; the preflight now uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
A 401/403 on the authenticated GitHub size lookup was replaced by the
anonymous retry's answer, so SSO enforcement or an exhausted rate limit
on a private repository surfaced as a plain 404. Keep the refusal and
report it in the log (refused_auth_error) and in the notice.

Also:
- treat a reported size of 0 as an answer: one-shot at INFO, no notice
- send a token written as the URL username alone, as net/http does
- note in sameSite's doc that EndpointAuth hands helper credentials to a
  sibling service on the endpoint's site

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JjbQK1QFpSHS4yYkrWS5Y
Entire-Checkpoint: 01M5AFW25APMXA2M8HD73DA7S7
Adversarial review of the GitHub size preflight found the credential
policy split between gitproto.EndpointAuth and bootstrap's
sourceCredentials, neither of which looked at where the request goes.

HTTPConn.CredentialsFor(dest) replaces both. Each credential is measured
from the URL it is bound to (EndpointURL for explicit and URL userinfo,
the lookup URL for helper credentials), dest must be same-site with it,
and with TLS verification off nothing crosses to another host. So:
- helper credentials looked up for a redirect host no longer reach a
  sibling of that host;
- credentials are not sent to the API with TLS verification off;
- a URL username without a password is not sent (it may be an account
  name, which the API would only refuse);
- the configured API base no longer receives credentials unless it is
  on the source's site.

Also:
- log "authenticated" from the request that produced the answer, so a
  redirect that stripped the header reads false;
- strip C1 controls and bidi formatting from GitHub's message;
- notice says the bootstrap "starts with" a single pack, and a failed
  lookup is no longer printed twice under -v;
- one record name for the verbose log and PushLogger;
- githubSizeLookup keeps the refused attempt as a nested answer;
- the redirect test asserts the redirect target was reached;
- docs: a public repo reached only through a credential helper is still
  looked up anonymously.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019JjbQK1QFpSHS4yYkrWS5Y
Entire-Checkpoint: 01M5AHETQ72WQ2NWA9M65233AD
GitHubOwnerRepo passed dot segments through, so a source URL such as
https://github.com/../user produced a lookup that path joining resolved
to /user, carrying the source credentials to the authenticated-user
endpoint. Accept only the characters GitHub allows in owner and
repository names, and reject . and .. outright.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FUdEhv4HALJ9DCRUP7GzTw
- Skip the lookup for wiki sources: the API has no repository to size.
- Report authenticated from whichever headers the authorizer set, not
  only Authorization.
- Drop the CLI flag name from the notice so it reads right for library
  callers too.
- Shorten the CHANGELOG entry and the Examples paragraph; move the
  credential details into "Credentials and redirects".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5C9ZCW16R8DR2083464JTX7
- Read authenticated off Authorization only again: the redirect guard
  strips only that header, so implying support for custom auth headers
  invited a cross-site leak.
- Treat a reported size of 0 as unknown: bootstrap runs only with source
  refs, so 0 means GitHub has not sized the repo yet. It now warns.
- Report nothing when the run itself is cancelled.
- Clean every error that reaches the log or notice, not only GitHub's
  message: transport errors carry server-authored text too.
- Count a 401/403 as a refused token only when the request still carried
  it, so a redirect that stripped credentials is not retried or blamed.
- Give each attempt its own timeout so a slow refusal cannot starve the
  anonymous retry.
- CredentialsFor returns nil instead of panicking without an endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5CAXBGAJY87C7716VXD6RX3
- Move the preflight to its own file, github_preflight.go.
- Move the one-line message cleaner to sanitize.Line and document it in
  the package's threat model, instead of a second sanitizer in bootstrap.
- Set authOrigin for explicit credentials at construction, so
  CredentialsFor reads one field; share helper-credential adoption in
  adoptHelperCreds.
- Reuse the default-port rule via gitproto.UsesDefaultPort.
- logGitHubPreflight branches once and logs with the chosen level.
- Tests: no sleeps in the timeout test (assert each attempt's deadline),
  zero size joins the unusable-body table, shared captureNotices and
  redirectingAPI helpers, bearer auth through privateRepoAPI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5CBACJHJ3YBH1HH7VZT2BWT
A caller-supplied transport may return a Response with Request unset
(net/http itself guards for it), and reading the final request's
Authorization header then panicked the whole sync. The answer is still
used; authentication is not claimed when its provenance is unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5CCCCANQRWX43W2HB2J0153
Reading Response.Request to learn whether the answer was authenticated
fails with a caller-supplied transport that leaves it unset: the last
fix then reported every answer as unauthenticated, and a refused token
lost its anonymous retry. Record each redirected request's Authorization
after the client's redirect policy has run instead, which needs nothing
from the response.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5CD4YBTTCFDT178G8YNTDX1
mayCarry, EnsureAuthForService and tryHelperRetry each spelled out
"TLS verification off and another host, so no credentials". A security
rule should not be able to drift between copies; crossesHostUnverified
now states it once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5D3RTAGH5944K7TTFQP0FGQ
- The notice names --target-max-pack-bytes again: CLI users are most of
  its readers, and library callers map it to TargetMaxPackBytes.
- docs/usage.md says again how a reported size of 0 is treated.
- The sanitize package doc separates raw C1 bytes, which UTF-8
  terminals ignore, from UTF-8-encoded C1 characters, which Line strips.
- TestLineKeepsMessageOnOneSafeLine no longer splits the Writer test
  from its doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015oS6W9qjGMVnrTz5r8EwDL
Entire-Checkpoint: 01M5FEE4HNPGWQTD9XCADVA9ZC
@nodo
nodo force-pushed the claude/practical-gauss-nj0xyd branch from d0e7b78 to cf885d2 Compare October 21, 2026 15:54
@nodo
nodo merged commit bd2e8f0 into main Oct 21, 2026
4 checks passed
@nodo
nodo deleted the claude/practical-gauss-nj0xyd branch October 21, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

5 participants