Skip to content

Replace prebuilt custom runtime overrides with manifest-based cargo-hyperlight builds - #295

Open
simongdavies wants to merge 9 commits into
mainfrom
simongdavies-custom-guest-builds
Open

Replace prebuilt custom runtime overrides with manifest-based cargo-hyperlight builds#295
simongdavies wants to merge 9 commits into
mainfrom
simongdavies-custom-guest-builds

Conversation

@simongdavies

@simongdavies simongdavies commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Build and embed a custom guest as part of the host build by setting HYPERLIGHT_JS_RUNTIME_MANIFEST_PATH to its Cargo.toml. The Rust and Node.js host APIs remain unchanged.

  • Remove the need for a fragile command to populate an environment variable before building the custom guest runtime.
  • Remove the separate custom-runtime build step: the host build takes care of this now.
  • Remove the obsolete stub-header include path.
  • Simplify the native-module test recipe and add build-selection and custom guest globals/clock coverage.
  • Keep build-selection helpers in a shared module with colocated tests, without importing the build script or suppressing unused-code warnings.
  • Scope the assertion lint to release library builds, allowing assertions in test and tool targets without per-file exceptions.
  • Update the extension guide with the manifest-based workflow.

Breaking change / release notes

Prebuilt custom guest embedding through HYPERLIGHT_JS_RUNTIME_PATH has been removed. The variable is no longer read or tracked, and there is no compatibility check or migration error.

Set HYPERLIGHT_JS_RUNTIME_MANIFEST_PATH to the custom runtime crate's absolute Cargo.toml path and rebuild the host or Node.js addon from source. The runtime manifest must define exactly one binary target; the host build builds it with cargo-hyperlight and embeds it automatically.

Without a custom manifest, the default runtime is built and embedded, even if the old variable is still set. Published Node.js addon binaries still contain the default runtime and must be replaced with a locally rebuilt addon to use custom modules.

This PR is labelled breaking-change for the generated release notes.

@simongdavies simongdavies added kind/enhancement New feature or improvement breaking-change Breaking change requiring consumer migration labels Sep 8, 2026
@simongdavies simongdavies changed the title Simplify custom runtime builds with cargo-hyperlight Replace prebuilt custom runtime overrides with manifest-based cargo-hyperlight builds Sep 9, 2026
@simongdavies simongdavies added the ready-for-review PR is ready for (re-)review label Sep 16, 2026
@simongdavies simongdavies mentioned this pull request Sep 16, 2026
Merged
@simongdavies
simongdavies marked this pull request as ready for review September 16, 2026 20:57
@simongdavies
simongdavies force-pushed the simongdavies-custom-guest-builds branch from 8b36dd0 to 5207db7 Compare September 16, 2026 21:13
@simongdavies
simongdavies changed the base branch from main to simongdavies-lint-before-build-in-ci September 16, 2026 21:13
@simongdavies
simongdavies added this pull request to stack #311 September 16, 2026 21:14
ludfjig
ludfjig previously approved these changes Sep 17, 2026
View reviewed changes
@simongdavies
simongdavies dismissed ludfjig’s stale review September 28, 2026 19:37

The merge-base changed after approval.

@simongdavies
simongdavies force-pushed the simongdavies-custom-guest-builds branch 2 times, most recently from a3b57e3 to b73f680 Compare September 29, 2026 07:50
simongdavies added a commit that referenced this pull request Sep 29, 2026
@simongdavies
Add macOS (hvf) and aarch64 support (#308)
7377251 
Upstream `hyperlight-host` 0.17.0 already ships the Hypervisor.framework
driver, `virtual_machine/hvf/`, and `regs/aarch64/`. The remaining work
is on the `hyperlight-js` side: platform wiring, an architecture-derived
guest target, and CI/packaging.

> **Stack:** this is the **bottom** layer of a planned stack. Layer 2
(npm `linux-arm64` packaging) follows, and #295 is intended to be
rebased on top and appended retrospectively. Review/merge this one
first.

## Notable Changes

- Introduce cfg aliases (`kvm`, `mshv3`, `hvf`, `whp`, `crashdump`,
`gdb`) in `build.rs`, mirroring upstream's semantics so platform code
never tests a bare `feature = "..."` flag. Enable `hvf` by default;
scope `libc` to `cfg(unix)`.
- Gate `with_interrupt_retry_delay` on `any(kvm, mshv3, hvf)` rather
than the Linux-only drivers.
- Add a mach-based thread CPU-time backend for macOS. `js-host-api`
enables `monitor-cpu-time` unconditionally, so this must compile on
every host platform.
- Add `src/hyperlight-js-runtime/include/math.h`. This is the one
non-obvious change and it is what unblocks aarch64 guest builds:

newlib guards its `__builtin_*` fast path with `… &&
!defined(__clang__)`. `cargo-hyperlight` drives clang, so `math.h` falls
back to a `sizeof` dispatch whose never-taken `(long double)` branch is
still code-generated. That is free on x86_64 (80-bit x87) but emits
`__extenddftf2`/`__extendsftf2` soft-float libcalls on aarch64
(binary128), which fail to link against the guest sysroot. The shim uses
`#include_next <math.h>` and restores the builtin path under clang.

## Pre-existing crashdump build failure

`crashdump` builds are broken on `main` today — verified by checking out
a clean `origin/main` into a temp worktree and reproducing the identical
`E0596`. The snapshot helpers take `&self` but call `hyperlight-host`
APIs requiring `&mut self`. Fixed at both call sites and gated on the
new `crashdump` alias, which also restricts them to x86_64 in line with
upstream.

## CI and packaging

- `dep_build` matrix goes 6 → 10 jobs: macOS/hvf on the same self-hosted
`["self-hosted","macos","arm64","hvf"]` runners hyperlight already uses,
plus Linux aarch64 KVM. These run the **full test suite**
- `npm-publish` gains an `aarch64-apple-darwin` binary on `macos-15`,
with a new `darwin-arm64` npm package.

---------

Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simongdavies
simongdavies force-pushed the simongdavies-custom-guest-builds branch from 5be37a4 to 88ceffa Compare September 29, 2026 09:07
ludfjig
ludfjig reviewed Sep 30, 2026
View reviewed changes

@ludfjig ludfjig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand the reason for these changes? What's the motivation?

ludfjig
ludfjig previously approved these changes Sep 30, 2026
View reviewed changes
Comment thread docs/extending-runtime.md Outdated
@simongdavies
simongdavies force-pushed the simongdavies-custom-guest-builds branch from 88ceffa to e6c1dec Compare September 30, 2026 18:11
ludfjig
ludfjig previously approved these changes Oct 1, 2026
View reviewed changes
Comment thread src/hyperlight-js/Cargo.toml
Base automatically changed from simongdavies-lint-before-build-in-ci to main October 2, 2026 06:54
simongdavies and others added 7 commits October 2, 2026 07:54
@simongdavies
Simplify custom runtime builds with cargo-hyperlight
045274a 
Build and embed custom runtimes from their Cargo manifests, keeping compiler configuration in the host build script. Remove the prebuilt runtime override and simplify the consumer documentation and test recipes.

BREAKING CHANGE: HYPERLIGHT_JS_RUNTIME_PATH is no longer supported. Use HYPERLIGHT_JS_RUNTIME_MANIFEST_PATH to select a custom runtime crate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Reuse cargo metadata when building the default runtime
6c0b111 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Remove low-value runtime build tests
87ece42 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Require a single binary in runtime manifests
e9c58bd 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Separate runtime build helpers and scope assertion lint to release li…
7708c10 
…brary

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Remove obsolete runtime override migration guard
a063fab 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Scope custom guest clock linker flag to the guest binary
79fb5b3 
Use cargo rustc for custom guests so the clock wrapper flag is not inherited by cargo-hyperlight's native sysroot wrapper build.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
simongdavies and others added 2 commits October 2, 2026 07:54
@simongdavies
Fix runtime resource selection in build script
3ef6b65 
Restore the selected runtime binding used when copying the built guest into
the generated host resource.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
Describe custom runtime setup as current state
ec0baf0 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies
simongdavies force-pushed the simongdavies-custom-guest-builds branch from f23721a to ec0baf0 Compare October 2, 2026 06:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ludfjig ludfjig ludfjig approved these changes

Assignees

No one assigned

Labels

breaking-change Breaking change requiring consumer migration kind/enhancement New feature or improvement ready-for-review PR is ready for (re-)review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@simongdavies @ludfjig