Skip to content

Upgrade to buildx v0.38 and buildkit v0.34 - #627

Open
jacobwgillespie wants to merge 56 commits into
mainfrom
upgrade-buildx-v0.38
Open

jacobwgillespie wants to merge 56 commits into
mainfrom
upgrade-buildx-v0.38

Conversation

@jacobwgillespie

@jacobwgillespie jacobwgillespie commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Replaces the forked buildx v0.10 code (pkg/buildx/{bake,build,builder,imagetools}, ~15k lines) and the depot/buildkit fork with upstream buildx v0.38.0 and buildkit v0.34.0. depot build and depot bake now call buildx's public build.Build and bake packages; Depot behavior lives in a buildx driver, a gRPC interceptor, and the command layer.

Also moves to docker/cli v29, the moby client, containerd v2, and Go 1.26.8.

Architecture

flowchart LR
    cmd["depot build / bake<br/>(pkg/buildx/commands)"] --> bb["buildx build.Build<br/>(upstream)"]
    cmd --> bake["buildx bake<br/>(upstream)"]
    bb --> drv["Depot driver<br/>(pkg/buildxdriver)"]
    drv --> ic["gRPC interceptor"]
    ic --> m1["amd64 builder"]
    ic --> m2["arm64 builder"]
    ic -. "per-machine solve responses" .-> cmd
    cmd --> post["metadata · fast --load · SBOMs · leases"]
Loading
  • Driver (pkg/buildxdriver): implements buildx v0.38's driver.Driver, acquires Depot machines, and uses driver.BuildPreparer to keep Internal=true (no build history records on builders).

  • Interceptor (pkg/buildxdriver/interceptor.go) covers what buildx exposes no option for:

    • records every machine's exporter response, which buildx merges into one; fast --load, --sbom-dir, lease cleanup, and --metadata-file need each machine's response
    • sends each session's context key as <dir>:<target>:<node>, the format from Use unique names for local context in bake #489, so bake targets that share a context directory but filter it differently don't overwrite each other, and existing context caches on builders stay valid
    • caches the worker list after the first request; the buildkit client lists workers before every build
  • Bake project_id: upstream bake rejects unknown attributes, so before upstream bake reads the files, each project_id becomes a policy entry, and each reference such as target.base.project_id or target.base["project_id"] reads that entry. After the read, the CLI takes the project from the entry and removes it. Depot builders cannot enforce source policies, so no working bake file sets or reads policy. Bake then resolves each target's project with the same variables, functions, inheritance, file order, and matrix expansion as other attributes, in the same read as the build options:

    target "api" {
      inherits   = ["base"]
      project_id = "${PROJECT}"   # still supported, also via x-depot.project-id in compose
    }
    
    # what upstream bake reads
    target "api" {
      inherits = ["base"]
      policy   = [{ filename = "depot-project-<random>:${PROJECT}" }]
    }
    • The prefix contains a random value for each run, so no policy of the user can match it

    • Bake appends policy entries, parents before children and earlier files before later ones, so the last entry with a project selects it; an empty project_id does not replace an inherited project

    • A target without a project reads as "", as before. Reads of target.base.policy and of a whole target, such as lookup(target.base, "project_id", ""), do not show the entries

    • Compose x-depot.project-id values become an HCL file of policy entries placed before the other HCL files

    • Files that select no project are read unchanged, apart from project_id reads of a whole target, which read ""

    • Known limitation: a matrix variable named target whose matrix is not an object literal is read as a target

  • depot buildctl proxy: advertises the capability list of the Depot buildkit v0.13.2 fork instead of the compiled-in v0.34 list, and forwards the new gateway methods.

Behavior changes

  • Upstream buildx output changes:
    • bake --print omits an empty "network", shows cacheonly outputs for linked targets, and escapes ${ and %{ in values as $${ and %%{, so that the output is valid bake input
    • error summaries include the log lines of the failed step
    • registry outputs carry unpack=false
  • Fixed:
    • depot bake <remote-url> no longer panics. A remote definition can build targets of one project only; targets of another project cause an error
    • depot configure-docker --uninstall and the driver update remove only Depot driver containers, not every container with four underscores in its name
    • --lint reads the Dockerfile from a context archive on stdin instead of ./Dockerfile
    • bake targets with a call attribute print their result (outline, targets, check) instead of running a normal build; check exits with its status code, as in buildx
    • bake target patterns such as 'mx-*' work with --load and --metadata-file
    • unset-variable warnings for bake files print once, as in buildx
    • --load with another -o output now loads the image as well, as in buildx. Before, it was ignored, or failed with multiple outputs currently unsupported when loading through the Depot registry
    • --lint with a remote git or HTTP context lints the Dockerfile from that context, which the builder downloads, instead of the local ./Dockerfile
    • multi-platform --metadata-file merges machine responses in --platform order instead of random order
    • when the API resolves the project of a local bake to another project of the definition, each build runs the targets of its requested project, so every target runs once. Before, the build ran the targets of the resolved project, which also ran in their own build, and skipped its own targets
  • Bake targets with a policy attribute fail with an error. Depot builders run buildkit v0.13, which cannot enforce source policies; before, the attribute was ignored
  • --platform linux/amd64/v4 starts only the amd64 machine instead of every machine
  • No more 32-bit Windows ARM build or @depot/cli-win32-arm npm package, because Go 1.26 removed windows/arm
  • The CLI drops one log line that the HTTP/2 server of a buildkit session prints when a session closes during its handshake (http2: server … localhost). buildx's per-machine capability check opens such short sessions.

Performance

Median wall time per scenario with 50 ms added round-trip latency to the builders (go run ./validation -latency 25ms -count 4):

Scenario main This PR
build, local output 1.73 s 1.69 s
build, 2 platforms, local output 1.70 s 1.76 s
build, 2 platforms, push 2.63 s 2.65 s
build, no output 1.62 s 1.64 s
bake, default group 1.63 s 1.72 s
bake, push 2.56 s 2.59 s

Validation harness

validation/ replaces unit tests for these commands with black-box runs of the CLI. It builds the CLI from a baseline revision and from the working tree, runs every scenario with both, and compares exit codes, output, files, loaded images, registry manifests, and Depot API calls.

go run ./validation                    # Depot buildkitd image, baseline main
go run ./validation -upstream          # moby/buildkit v0.13.2, used in CI
go run ./validation -run 'bake-.*' -count 3 -latency 25ms
  • Environment: two buildkitd builders with mutual TLS, TLS registries, a git server, and a fake Depot build and push API
  • 193 scenarios: build, bake, compose, multi-project bake, remote git sources, all output types, --load, --push, --save, lint, SBOMs, interrupts and API cancellation, depot pull, depot push, depot exec, and docker build / docker buildx build after depot configure-docker
  • New validation CI job runs -upstream, which skips the 14 scenarios that need the Depot buildkit exporter (fast --load, --sbom-dir), and uploads the JSON report
  • See validation/README.md

Note

High Risk
Large dependency and architecture swap for core build/bake paths, with behavior and output changes; mitigated by the new validation harness and CI job but still touches critical build infrastructure.

Overview
Replaces the in-tree buildx v0.10 and depot/buildkit forks with upstream buildx v0.38 and buildkit v0.34, wiring Depot through a custom buildx driver, gRPC interceptor, and updated pkg/buildx/commands (including bake project_id → policy shimming for upstream compatibility).

Toolchain & packaging: Go 1.26.8, docker/cli v29, golangci-lint v2.14, containerd v2 / moby client; drops Windows 32-bit ARM binaries and @depot/cli-win32-arm; removes pkg/cleanup tmpfile defer cleanup.

CLI polish: Filters noisy BuildKit session HTTP/2 handshake log lines via logFilter; docker plugin metadata import update; new pkg/arguments for docker-compatible positional-arg errors; registry auth uses buildkit authprovider.

Quality gate: Adds validation/ black-box harness (~193 scenarios) and a CI validation job (-upstream, artifact report). Examples module tracks new buildx progress APIs.

Reviewed by Cursor Bugbot for commit 75ecdfc. Bugbot is set up for automated code reviews on this repo. Configure here.

Replace the forked buildx v0.10 code and the depot/buildkit fork with
upstream buildx v0.38.0 and buildkit v0.34.0. The build and bake commands
now call buildx build.Build and the bake package. A Depot buildx driver
keeps the per-machine responses, the shared context keys, and the build
history settings that the fork provided.
The harness builds the CLI from a baseline revision and from the working
tree, runs each scenario with both against local buildkitd, registry, and
Depot API stand-ins, and compares the results.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T14:30:04.539659Z 75ecdfc New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Go 1.26 no longer supports windows/arm.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/buildxdriver/interceptor.go
Comment thread pkg/buildxdriver/driver.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27d048b090

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/build.go
Comment thread pkg/buildx/commands/dockerfile.go Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Cursor Bugbot reported two unresolved medium-severity findings (session/Solve ordering and a machine leak on failed connect), so this is not approved. Human review is needed and no reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Cursor Bugbot reported two unresolved potential issues, so this is not approved. Human review is needed and no reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/buildxdriver/driver.go
cursor[bot]
cursor Bot previously approved these changes Oct 9, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Approved. Cursor Bugbot skipped on this head and the earlier Bugbot findings are resolved; the change is within the approval threshold.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

After a multi-platform push, --iidfile now contains the digest of the
merged image index. --lint now checks a Dockerfile given as a URL, and a
Dockerfile from standard input with a remote context.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 400b1e0d86

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/cmd/docker/docker.go Outdated
Comment thread validation/main.go Outdated
Comment thread pkg/buildx/commands/dockerfile.go
Comment thread pkg/buildx/commands/bake.go
ListDepotNodes listed every container, because filters.FuzzyMatch does
not add a filter. Containers of other programs with four underscores in
their names were then removed by configure-docker --uninstall and by
the driver update.
--lint now reads the Dockerfile from a bzip2, xz, or zstd context
archive on standard input. A remote bake definition with targets of
another project now fails with an error instead of skipping them.
The project of a compose service was applied after bake resolved the
targets, and matched only by service name. A target that inherited a
compose service lost the project, and the project of the compose service
replaced the project_id of an HCL target with the same name.

The compose project IDs are now written as an HCL file of target
descriptions that comes before the other HCL files, so bake applies
inheritance and the merge order of files to them.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/buildx/commands/bakeproject.go Outdated
The forked buildx set the SOURCE_DATE_EPOCH build argument from the
environment. Upstream buildx sets it in its own commands, which Depot
does not use. build, bake, and bake --print now set it again, unless the
build arguments already contain it.
--load with an -o output that is not an image loaded nothing. Through
the Depot registry, the old CLI failed with "multiple outputs currently
unsupported"; otherwise it ignored --load. --load now adds an image
output next to the other outputs and loads it, as buildx does.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/load/cli.go
@cursor
cursor Bot dismissed their stale review October 9, 2026 15:13

Risk: high. Dismissed the prior approval because Cursor Bugbot left an unresolved potential issue after that approval. Human review is needed and no reviewers were assigned.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b647d61a3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildxdriver/driver.go
- A reference such as target.base.project_id failed, because project_id
  was removed before bake read the files. project_id is now moved into
  the description of each target for the build too, and references to it
  are renamed. One read of the files gives the build options and the
  project of each target. bake --print reads the descriptions separately.
- The requested targets now come from the targets and groups that bake
  resolved, so target patterns such as 'mx-*' are loaded and listed in
  the metadata file.
- Bake now prints the result of targets with a call attribute, and maps
  call = check to the lint request and call = build to a normal build,
  as buildx bake does. build and bake print check results and return
  their status code. Targets with a call are not loaded.
- The CLI's own parse of compose files no longer repeats the warnings
  for unset variables that bake already prints.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/buildx/commands/bake.go

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Not approved. Cursor Bugbot reported an unresolved medium-severity finding on this head, so human review is needed and no reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 270fc0a693

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/bakeproject.go Outdated
Comment thread pkg/buildx/commands/bakeproject.go Outdated
Comment thread pkg/buildxdriver/response.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1f15d19ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/bakeproject.go Outdated
Comment thread pkg/buildx/commands/bakeproject.go Outdated
Comment thread pkg/buildx/commands/build.go
Comment thread pkg/buildx/commands/bakeproject.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f09f5e604

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/bakeproject.go
Comment thread pkg/buildx/commands/bakeproject.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 201ed34296

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/bakeproject.go
Comment thread pkg/buildx/commands/bake.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

ShmSize: in.shmSize,

P2 Badge Forward the requested cgroup parent

When depot build --cgroup-parent=... is used, validation still accepts the flag, but this options initializer no longer assigns in.cgroupParent to build.Options.CgroupParent. BuildKit therefore receives an empty parent and runs the build containers in its default cgroup, silently breaking resource-accounting or isolation setups that rely on the requested hierarchy. Include CgroupParent: in.cgroupParent in the options passed to the build.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/bake.go
Comment thread pkg/buildx/commands/bake.go
@jacobwgillespie

Copy link
Copy Markdown
Member Author

Re the Codex review comment on --cgroup-parent (#627 (review)): this matches main, which defines the flag but never passes its value to the build options. Forwarding it would change builds on Depot builders, so I am leaving it out of this upgrade.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread pkg/buildx/commands/bake.go
@cursor
cursor Bot dismissed their stale review October 19, 2026 13:48

Risk: high. Dismissing the prior approval because Cursor Bugbot reported an unresolved medium-severity finding on this head, so human review is needed and no reviewers were assigned.

cursor[bot]
cursor Bot previously approved these changes Oct 19, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: high. Approved. Cursor Bugbot completed successfully on this head with no unresolved findings that need human review, and the change is within the approval threshold.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0c07b64. Configure here.

Comment thread pkg/buildxdriver/driver.go
@cursor
cursor Bot dismissed their stale review October 19, 2026 14:12

Risk: high. Dismissed the prior approval because Cursor Bugbot reported an unresolved medium-severity finding on this head. Human review is needed and no reviewers were assigned.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c07b644bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/buildx/commands/dockerfile.go Outdated
Comment thread pkg/buildx/commands/bake.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: high. Approved. Cursor Bugbot completed successfully on this head with no unresolved findings that need human review, and the change is within the approval threshold.

Open in Web View Automation 

Sent by Cursor Approval Agent: Pull Request Router and Approver

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants