mycrt: a container runtime and mini orchestrator

A container runtime written against Linux syscalls, and a small scheduler that keeps containers running across a three-node cluster. There is no Docker, containerd, runc or libcontainer underneath. The only libraries are golang.org/x/sys, vishvananda/netlink and bbolt, as the brief asks.

• mycrt runs OCI images: namespaces, cgroups v2, overlayfs, image pulls from any registry, bridge networking with NAT and published ports. Every container is confined by a user namespace, dropped capabilities, a seccomp filter, a device allow-list and masked kernel files.
• mycontrol is the control plane: pods in bbolt, a scheduler, and a reconcile loop that replaces failed containers and dead nodes.
• myagent runs on every node: it starts containers with mycrt, probes their health and sends heartbeats.
• myctl is the command-line client: myctl apply pod.json.

Quick start

Code is edited on the Mac and cross-compiled for the Lima VMs, where it runs as root. make install builds and copies the binaries into crt1.

git clone https://github.com/YOGESH256/container-runtime
make install
limactl shell crt1

sudo mycrt run -t alpine:3.20 sh                      # isolated, networked shell on its own terminal
sudo mycrt run --memory 64m --cpus 0.5 --pids 100 alpine:3.20 sh
sudo mycrt run -d --name web -p 8080:80 nginx         # background container, port published
curl http://localhost:8080/
sudo mycrt exec web sh                                # a shell inside the running container
sudo mycrt ps                                         # shows its IP and ports
sudo mycrt stop web && sudo mycrt rm web

The cluster needs all three VMs:

limactl start crt2 crt3
scripts/cluster.sh up                                 # control plane on crt1, agent everywhere
limactl shell crt1 myctl apply examples/pod.json
limactl shell crt1 myctl get pods
limactl shell crt1 myctl events -f                    # watch it react
limactl stop -f crt2                                  # kill a node, watch pods move
scripts/cluster.sh down

Set MYCRT_DEBUG=1 to see where container start-up time goes. Set MYCRT_REGISTRY_MIRROR=mirror.gcr.io to spare Docker Hub's rate limit.

Diagrams

Three interactive diagrams live in docs/. GitHub shows them as source, so download a file and open it in a browser, or clone the repo.

• docs/overall.html: the whole system, from myctl down to a running container.
• docs/network-blocks.html: the bridge, veth pairs, NAT, IP leases and DNS.
• docs/network-journey.html: one curl from a container out to the internet.

Tests

Each stage has a script that checks the brief's "done when" line, plus more.

Stage	Run	Checks
1. Isolation	make test-stage1	32
2. cgroups v2	make test-stage2	19
3. overlayfs	make test-stage3	14
4. OCI images	make test-stage4	24
5. Networking	make test-stage5	28
6. Orchestrator	bash test/stage6.sh (on the Mac)	23
7. Hardening	make test-stage7	39

make unit runs the Go tests that need no root, on the Mac. They cover image references, the registry client against a fake registry, IP allocation, the agent, and the control plane against a simulated cluster: node death, probe failure, crash loops, a slow node coming back, and a control-plane restart. The stage 4 script also runs the pull pipeline's root-only tests in the VM, which mount a real overlay to check whiteouts.

How it works

Stage 1: isolation (internal/container)

mycrt run starts a child with CLONE_NEWPID | NEWNS | NEWUTS | NEWIPC | NEWNET. The child is the same binary re-executed through /proc/self/exe with a marker in its environment. The Go runtime is multithreaded before main runs, and some namespace operations only work in a fresh process, so the child starts life already inside its namespaces.

The child blocks reading its config from a pipe while the parent sets up the host side. Then it makes all mounts private (systemd marks / shared, so mounts would otherwise leak to the host), mounts the root filesystem, /proc, a fresh /dev, /sys and /dev/pts, and calls pivot_root(".", ".") followed by a lazy unmount of the old root. Finally it execs the command, which becomes PID 1.

PID 1 is special in two ways, and the tests cover both. Orphans are re-parented to it, so it must reap them. And the kernel drops any signal sent to PID 1 that it has no handler for, so sleep as PID 1 ignores SIGTERM. --init puts a tiny init at PID 1 that reaps and relays signals, like docker run --init.

Every mount target is resolved with openat2(RESOLVE_IN_ROOT) (see internal/fsutil). An image containing a symlink such as /proc -> /etc can therefore never make mycrt mount over a host path.

Stage 2: cgroups v2 (internal/cgroups)

Each container gets /sys/fs/cgroup/mycrt/<id> with memory.max, cpu.max and pids.max. The child is created directly inside it with clone3 and CLONE_INTO_CGROUP, so it never runs for a moment without limits. It then unshares a cgroup namespace, so inside the container /sys/fs/cgroup/memory.max shows its own limit. memory.oom.group makes the OOM killer take the whole container, and cgroup.kill stops even a container that keeps forking.

Stage 3: overlayfs

Image layers are read-only lower directories. Each container gets its own upper and work directories, so any number of containers share a layer and it is never modified. Deleting a lower file creates a whiteout, a 0:0 character device, in the upper layer. The stage 3 test shows one.

Stage 4: OCI images (internal/registry, internal/image)

A pull resolves the name, follows a multi-arch index to the linux/arm64 manifest, fetches the config, then streams each layer:

HTTP body -> sha256 (must match the manifest) -> gunzip -> sha256 (must match
the config's diff ID) -> untar into a fresh directory -> rename into place

A layer is committed only if both hashes match. Docker Hub's anonymous token dance is handled by following the WWW-Authenticate challenge. A HEAD request detects an up-to-date image without using a pull from the rate limit.

Each layer is unpacked into its own directory, so OCI whiteouts (.wh.name, .wh..wh..opq) are converted into what overlayfs reads rather than applied by deleting files. Tar entries are resolved inside the layer with RESOLVE_IN_ROOT, so ../../etc/passwd or a path under a symlink to / stays inside the layer.

Stage 5: networking (internal/network)

container A: eth0 10.88.0.2        container B: eth0 10.88.0.3
        | veth pair                        | veth pair
        +----------- mycrt0 bridge, 10.88.0.1 -----------+
                              |
                  host routing + nftables masquerade -> internet

Each container gets a veth pair. The inner end is created directly inside the container's network namespace and named eth0. The outer end is plugged into the mycrt0 bridge. Addresses come from a lease file guarded by flock. nftables masquerades traffic leaving the subnet. DNS comes from systemd-resolved's upstream list, because 127.0.0.53 is unreachable from inside a container.

-p 8080:80 publishes a port. The container gets an nftables table of its own with DNAT rules in the prerouting hook (traffic from outside), the output hook (the host itself, including localhost) and a hairpin masquerade so another container can reach the published port through the host. Removing the container removes the table.

Stage 7: hardening (internal/container/hardening_linux.go)

Stages 1 to 6 hide the host from the container. Stage 7 takes away the power to do anything about it. Root inside a container is, to the host:

Lock	Effect
User namespace	Container uid 0 is host uid 100000. An escape lands as an unprivileged user. Image layers are unpacked with owners shifted by 100000 so they look root-owned inside.
Capabilities	Docker's default set minus CAP_MKNOD, enforced through the bounding set, so no exec can regain them. no_new_privs makes setuid binaries inert.
seccomp	A BPF filter refuses about fifty system calls: mounts, namespaces, modules, reboot, clock, swap, ptrace, BPF, perf, io_uring, keyring. clone3 returns ENOSYS so clone can be inspected for namespace flags. A foreign ABI kills the process.
Device filter	An eBPF program on the container's cgroup allows only null, zero, full, random, urandom, tty, console, ptmx and pts. It applies to host root too.
Masked paths	/proc/kcore, /proc/keys, /sys/firmware and others are covered; /proc/sys, /proc/sysrq-trigger, /sys and the cgroup files are read-only.
Devices	/dev holds six bind-mounted nodes and nothing else. mknod is refused by the user namespace.

The stage 7 test is a list of escape attempts that must fail: mount, unshare, mknod of a disk, reading the kernel log, reboot, writing sysrq, opening /dev/vda from a host-root process placed in the cgroup. nginx, ping and apk keep working.

What this is not: a boundary for hostile code. The container still shares the host kernel, and one kernel bug reachable through an allowed system call is enough to escape. That needs a second, independent boundary, such as gVisor or a microVM. Hardened containers are the right tool for code you trust.

mycrt exec joins a running container's namespaces and applies the same locks. One limit: a multithreaded Go process cannot join a user namespace, so the exec'd process runs as the shifted host user instead. Inside, id prints 100000 rather than 0, but file access and every restriction are the same.

Stage 6: the orchestrator (internal/orch)

The orchestrator is one reconcile loop: observe actual state, diff it against desired state, act, repeat.

• Desired state is the pods in bbolt, each with the node it was placed on and a generation number that increases with every placement.
• Actual state is what agents report in heartbeats every 2 seconds. Every container is labelled with its pod and generation.
• Each second, the control plane schedules unplaced pods onto the ready node with the fewest pods. It replaces containers that exited, failed to start, or failed their health probe three times in a row. It moves pods off nodes silent for 10 seconds, and stops containers nobody wants.

Health probes run on the agent, as kubelet does, because container IPs are private to each node's bridge. Agents pre-pull every image the cluster uses, so a failover does not wait for a download. Containers survive an agent restart: each has its own mycrt supervisor process, and the agent rediscovers them from their labels.

Split brain. A node that is slow rather than dead gets its pods rescheduled after the 10 second lease, so for a moment two copies run. This design accepts that duplicate window instead of fencing. When the node reports again, its containers carry an old generation, and the control plane stops them. The stage 6 test freezes an agent with SIGSTOP to show both copies and then the cleanup.

What the tests caught

These bugs only showed up under the stage tests and are worth knowing about. Each fix is in the code with a comment.

1. Every container exit took ~400ms. The exiting process was stuck in wait_sb_inodes. Tearing down an overlay mount syncs its upper filesystem, which here is the whole root disk. The overlay option volatile skips that. The cost is an upper layer that is unusable after a crash, and mycrt never reuses one.
2. Bridge networking cost ~0.7s per container. Moving a veth into a namespace and renaming it each wait for an RCU grace period. Creating the peer inside the namespace, already named eth0, avoids both. Deleting the veth by hand at exit was also redundant, since the kernel removes it with the namespace.
3. A replacement container was unreachable for ~30s. It reused the IP its predecessor had just freed, and the host's ARP cache still pointed at the old MAC. The fix is Docker's: derive the MAC from the IP.
4. A power-cut node crash-looped 122 times in 3 minutes. A file written and renamed without fsync came back empty after the VM was killed, and every start failed on it. All state now goes through internal/fsync. Damaged indexes are rebuilt, and the control plane backs off exponentially on repeated failures, like CrashLoopBackOff.
5. SIGTERM during setup killed mycrt before cleanup. The fix is to subscribe to signals before creating anything, then relay any queued signal once the container runs.
6. nginx could not open /dev/stderr once containers ran as user 100000. Reopening a stream through /proc/self/fd checks the file's owner, and the log file and terminal belong to host root. Streams the container could not reopen are now relayed through pipes owned by its user, and the detached log is chowned to it.
7. The eBPF device filter denied everything. The kernel packs the access bits in the upper half of access_type and the device kind in the lower; the first version read them the other way round.

Not implemented

• No AppArmor or SELinux profile. The locks above are all kernel features mycrt sets up itself; a mandatory access control profile would be one more.
• exec runs outside the user namespace (see stage 7).
• One control plane, no auth or TLS, node-local container IPs, and no service discovery, per the brief.

Layout

cmd/mycrt         runtime CLI: run, exec, ps, stop, rm, logs, inspect, pull, images
cmd/mycontrol     control plane server
cmd/myagent       node agent
cmd/myctl         cluster CLI
internal/container  namespaces, re-exec'd init, mounts, pivot_root, mini init,
                    capabilities, seccomp, pty, exec
internal/cgroups    cgroup v2 groups, limits and the eBPF device filter
internal/image      references, manifests, content store, verified pulls, unpacking
internal/registry   OCI distribution client with bearer-token auth
internal/network    bridge, veth, NAT, IP leases, /etc files
internal/engine     container lifecycle: create, run, supervise, stop, remove
internal/orch       pod types, control plane, agent
internal/fsutil     symlink-safe path resolution inside a root
internal/fsync      crash-safe file writes
test/stageN.sh      one script per stage of the brief
scripts/            Alpine rootfs fetcher, cluster start and stop