Repository navigation
Route containerd ns mirror requests to configured registries - #405
Open
pinguinfuss wants to merge 10 commits into
Open
pinguinfuss wants to merge 10 commits into
pinguinfuss wants to merge 10 commits into
Conversation
containerd's hosts.toml mirrors append ?ns=<registry-host> to every
request. The container handler ignored it, so a single _default mirror
entry could not serve more than one registry.
ns is a closed-world lookup key and is never dialed. Docker Hub aliases
and the host of upstream.oci_default select the default route, hosts of
upstream.oci entries select their named upstream. Unknown hosts return
NAME_UNKNOWN so containerd falls back to its next host. Registry URLs
with a path are not indexed, because ns names the registry at the root
of a host. Host collisions log a warning and resolve deterministically.
With ns the path is the verbatim upstream repository, the reserved
upstream/ prefix is rejected, and repository prefix routes such as
Homebrew's do not apply. Cache names match the unprefixed and
upstream/{name}/ routes, so all routes share cached blobs, manifests and
tag lists. ns is no longer forwarded on tag-list requests, and the
pagination Link is rewritten per request instead of at store time so
clients on different routes get links for their own route.
Refs git-pkgs#303
Tag-list rows now store the upstream Link verbatim and rewrite it per request. Rows written by earlier versions hold a Link already rewritten for one route. Under the unchanged key an older binary running next to this one (rolling update on shared Postgres, or a rollback) would serve the raw Link unrewritten, sending clients to the wrong route for the next page. A format marker in the key keeps the two apart at the cost of one cache miss per tag list after the upgrade. Adversarial review finding F1: tag-list cache rows hold raw upstream Links under unchanged keys, so an older binary serves them unrewritten.
Registries whose URL has a path and hosts shared by two entries are not reachable through ns; point to the configuration guide for both rules. Adversarial review finding F3: README says ns covers every upstream.oci registry.
Configured registry URLs may carry userinfo credentials, which the new startup warnings wrote to the log verbatim. Log them with the password masked, and say precisely what a path-prefixed default registry means: its host is not indexed, while the Docker Hub aliases still select it. Adversarial review finding F2: startup warnings print registry URLs unredacted and the default-registry message is misleading.
andrew
requested changes
Oct 4, 2026
andrew
left a comment
andrew
left a comment
Contributor
There was a problem hiding this comment.
Please preserve non-default registry ports during namespace lookup and include the containerd configuration needed to enable the hosts directory.
Dropping 80 and 443 regardless of scheme turned https://host:80 and https://host into the same key, so a request for one of them could land on the other registry. Only the default port of the URL's scheme is optional now. A configured host is indexed with and without that port, because image references spell it either way; any other port has to match exactly, and the ns value is used as containerd sends it, apart from case and IPv6 brackets. Since a URL now yields two keys that can belong to different routes, the collision warning lists the owner per key. The handler test runs two registries on one host that differ only in the port. Tests can't bind 80 or 443, so a dialer maps those addresses to the fake servers.
containerd sends ns on every request to a mirror host, override_path or
not. Per-registry mirrors pointing at /v2/upstream/{name} therefore
arrive as upstream/{name}/...?ns=<registry>, and refusing every prefixed
name under ns broke them, including the only way to mirror an upstream
whose URL has a path.
Such requests are now handled like the prefix route without ns, as long
as ns names that upstream's own host under the same port rules as the
index. Any other ns on a prefixed name is still NAME_UNKNOWN, so ns
can't be used to create cache entries under another registry's name.
A _default/hosts.toml does nothing while CRI has no hosts directory configured, so add the config.toml snippet for containerd 2.x and 1.x. To check the setup, look at containerd config dump and pull with crictl, then find the request in the proxy log; ctr --hosts-dir reads the directory on its own and would succeed even without config_path. Also mention that existing per-registry override_path entries keep working and that k3s generates the directory itself.
Contributor
Author
|
Thanks for the review, both points are in. One thing I changed on top while testing this: containerd also sends |
andrew
requested changes
Oct 4, 2026
andrew
left a comment
andrew
left a comment
Contributor
There was a problem hiding this comment.
The port handling and containerd setup findings are addressed. Please also preserve Docker Hub aliases in the new prefix-route namespace check so existing per-registry mirrors keep working.
With upstream.oci.hub pointing at registry-1.docker.io and a docker.io hosts.toml mirror on /v2/upstream/hub, containerd sends ns=docker.io, and the prefix check only accepted the upstream's own host. The same happens whenever the upstream is a mirror of the registry the nodes pull from, say an Artifactory remote for ghcr.io: ns carries ghcr.io, the upstream host is something else, and the pull got a 404. The prefix already decides where the content comes from and which cache entries are used, so ns can't redirect anything there. The check now only refuses an ns that contradicts the prefix, meaning a host the proxy knows that belongs to another route. The Docker Hub aliases count as one host, and a host the proxy doesn't know at all is accepted, since a client can only arrive at the prefix through its own mirror entry.
A Docker Hub mirror configured as a named upstream (mirror.gcr.io, an
Artifactory remote) with a docker.io hosts.toml pointing at
/v2/upstream/hub got a 404 again: docker.io is always in the index for
the default route, so the check saw a known host of another route. That
setup worked before this branch. Docker Hub repository names have
exactly two path components, so docker.io/upstream/... can never be a
real image, and accepting the aliases on every prefix route shadows
nothing.
While here: the collision warning claimed ns requests go to the other
route, which is only true without the prefix, and the README said pulls
for unknown registries always get a 404, which isn't the case for paths
under upstream/{name}/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This teaches the
/v2handler thensquery parameter that containerd appends to mirror requests, so a single_defaulthosts.toml (or k3smirrors: "*") can front Docker Hub plus everything inupstream.oci.The idea is simple:
nsis only ever looked up, never dialed. The Docker Hub aliases and the host ofoci_defaultmap to the default registry, the host of eachupstream.ociURL maps to that upstream, and anything else gets a404 NAME_UNKNOWNso containerd moves on to its next host. Requests withoutnsare untouched. Hosts are compared case-insensitively with ports 80/443 dropped, using the same function on both sides. Withnsthe path is the verbatim upstream repository,upstream/...is rejected, and the cache names line up with the existing routes – pulling an image via ns, viaupstream/{name}/or unprefixed shares blobs, manifests and tag lists.A few decisions worth a look:
ns.ns=art.corpmeans the registry at the root of that host; mapping it onto something likehttps://art.corp/artifactory/api/docker/remotewould silently routeart.corp/docker-local/appinto a different repository. Those upstreams still work viaupstream/{name}/, and the proxy warns at startup.ns=docker.ioskips the Homebrew prefix route on purpose – the client asked for Docker Hub, so it gets Docker Hub. Withoutnsnothing changes there.Linkheader used to be rewritten when the entry was stored. Now that routes share the entry it is rewritten per request, and the cache key got a format marker so an older binary (rolling update, rollback) never sees the raw link. Costs one extra cache miss per tag list after upgrading.Tests live in
container_ns_test.goand cover the cases from the issue plus host normalization (IPv6, ports, case), the path exclusion and the legacy tag-list rows. I have only run this against the fake registries in the test suite so far, not against a real containerd node. README anddocs/configuration.mdgot a containerd section with a_defaulthosts.toml example.Closes #303