diff --git a/.claude/skills/forge-ci/SKILL.md b/.claude/skills/forge-ci/SKILL.md index e9b6c7de..df6e19bc 100644 --- a/.claude/skills/forge-ci/SKILL.md +++ b/.claude/skills/forge-ci/SKILL.md @@ -33,7 +33,8 @@ build-wheels.yml (push / workflow_dispatch / workflow_call) └── jobs (matrix: package × platform) e.g. "Python 3.12 / foo 1.2.3 #1 (android)" ├── Build wheels — prebuild_recipes first, then the package(s), all slices ├── mobile test (3.12 legs only by default) — recipe-tester APK on an - │ emulator (android) / .app on a simulator (ios), polls console.log + │ x86_64 emulator (android; never arm64, see local-recipe-testing #15) / + │ .app on a simulator (ios), polls console.log │ for the ">>>>>>>>>> EXIT N <<<<<<<<<<" sentinel └── upload artifacts — wheels + test console.log ``` diff --git a/.claude/skills/forge-error-catalogue/SKILL.md b/.claude/skills/forge-error-catalogue/SKILL.md index 9e519dda..6324a42e 100644 --- a/.claude/skills/forge-error-catalogue/SKILL.md +++ b/.claude/skills/forge-error-catalogue/SKILL.md @@ -91,6 +91,11 @@ instead of re-deriving it. green host-configured library, fix by patching in a `FORGE_CMAKE_ARGS` extend. **`conflicting types for 'fseek'` from a project's own `compat.h`, 32-bit Android only → the NDK toolchain pins `CMAKE_SYSTEM_VERSION` to 1; read `ANDROID_PLATFORM_LEVEL`**. + **bindgen `stdlib.h` not found / `'sim'` triple invalid → forge's + `BINDGEN_EXTRA_CLANG_ARGS_`**, **aws-lc-sys 0.40 external-bindgen → its `bindgen` + feature**, **iOS `library 'python3.10' not found` → drop pyo3 `abi3`**, **a crates.io dep + with no iOS/Android branch → `crate_patches`**, **a build script's bare `cmake` → + `CMAKE_TOOLCHAIN_FILE` via cargo `[env]`**. - **Runtime failures** (device/emulator/simulator) — **the Flet 0.86 Android `sitepackages.zip` class** (its umbrella entry explains "why only now"): `NotADirectoryError` on a bundled data file → **`extract_packages`** meta field, @@ -129,6 +134,12 @@ instead of re-deriving it. `flet build` reporting success while shipping the shared pub-cache `dist_ios`'s packages from some *other* project → **`-Wl,-headerpad_max_install_names`** on the iOS lanes of any non-CMake recipe binding several bundled dylibs (av). + Rust on Android: **`cannot locate symbol "__clear_cache"` (arm64 only) → forge links the + NDK's compiler-rt builtins**; **`fatal runtime error: out of TLS keys` → hidden pthread-key + multiplexing shim** (std spends a key per `thread_local!`; bionic has ~128); + **deno_core `Failed to initialize a JsRuntime: No such file or directory` → embed the + extension sources** (they are build paths without a snapshot); **`Expect + rustls-platform-verifier to be initialized` → `tls_certs_only` with bundled roots**. - **Recipe-tester app failures** — host-build (`pg_config` etc.), pypi.flet.dev index precedence, "no matching distribution" (incl. **ios-simulator also resolving the iphoneos wheel**), **sdist-only pure-python dep → pip backtrack** diff --git a/.claude/skills/forge-error-catalogue/references/failure-catalogue.md b/.claude/skills/forge-error-catalogue/references/failure-catalogue.md index 9d5423bb..556d0f16 100644 --- a/.claude/skills/forge-error-catalogue/references/failure-catalogue.md +++ b/.claude/skills/forge-error-catalogue/references/failure-catalogue.md @@ -1369,6 +1369,71 @@ upstream drift becomes your build break. (From `recipes/pymupdf`.) --- +### bindgen: `'stdlib.h' file not found` (Android) / `version 'sim' in target triple 'arm64-apple-ios-sim' is invalid` (iOS simulator) + +**Cause:** a `-sys` crate's build script runs bindgen, whose libclang gets no target +sysroot, and bindgen spells the `aarch64-apple-ios-sim` Rust triple as a clang triple +clang rejects. + +**Fix:** forge now exports `BINDGEN_EXTRA_CLANG_ARGS_` with `--target=` and +the NDK sysroot or `-isysroot ` for every Rust build. If you still see it, the crate +passes its own conflicting clang args. Found on vl-convert-python (libnghttp2, +libuv-sys-lite). + +--- + +### aws-lc-sys: `Failure invoking external bindgen` / `Please enable the 'bindgen' feature` (iOS, Android x86_64) + +**Cause:** with its default `all-bindings` feature, aws-lc-sys 0.40 ships pregenerated +bindings for a fixed target list (Android arm64 is on it; no iOS target and not Android +x86_64 are) and otherwise wants the `bindgen` CLI. + +**Fix:** enable its internal bindgen from the consuming crate, target-gated: +`[target.'cfg(any(target_os = "android", target_os = "ios"))'.dependencies]` +`aws-lc-sys = { version = "=0.40.0", features = ["bindgen"] }`. Newer aws-lc-sys (0.45) +uses universal bindings and needs nothing. + +--- + +### iOS: `ld: library 'python3.10' not found` (a PyO3 extension with `abi3-pyXY`) + +**Cause:** a pyo3 abi3 cross build links the abi3 floor (`-lpython3.10` for +`abi3-py310`), not the Python forge built for. + +**Fix:** drop the `abi3-*` feature in the recipe patch; forge builds one wheel per Python +anyway. Android may link fine regardless, so do not take a green Android leg as proof. + +--- + +### A crates.io dependency fails to compile for Android or iOS (`compile_error!` in a per-OS branch, `cannot find function` behind `cfg(target_os = "macos")`) + +**Cause:** the crate covers macOS and Linux only. iOS takes neither branch; Android takes +Linux's where it says `any(target_os = "linux", target_os = "android")` and nothing +otherwise. + +**Fix:** patch the crate with forge's `crate_patches` (see `new-mobile-recipe`). For iOS a +blanket `target_os = "macos"` -> `any(target_os = "macos", target_os = "ios")` over the +crate is usually right, since the branches select Darwin APIs; compile with +`cargo build --keep-going` to collect every failing crate in one pass. Precedent: +vl-convert-python (six Deno crates). + +--- + +### A build script's own `cmake` call configures for the build host (no cross toolchain) + +**Cause:** the crate shells out to `cmake` with a fixed argument list (v8x building WAMR), +so nothing forge sets reaches it, and CMake picks the host platform (WAMR even defaults +its platform to `CMAKE_HOST_SYSTEM_NAME`). + +**Fix:** CMake reads `CMAKE_TOOLCHAIN_FILE` from the environment. Ship a toolchain file in +the recipe patch that keys off `$ENV{TARGET}` (the Rust triple cargo gives build scripts; +host builds match nothing and stay native), and set the variable for build scripts +through a patched-in `.cargo/config.toml`: +`[env] CMAKE_TOOLCHAIN_FILE = { value = ".cargo/cross.cmake", relative = true }`. +Precedent: vl-convert-python's `mobile.patch`. + +--- + ## Runtime failures (on device/emulator/simulator) ### Flet 0.86 changed Android packaging — `sitepackages.zip` + jniLibs relocation (the umbrella behind a whole class of "worked under 0.85, fails now" on-device failures) @@ -1773,6 +1838,90 @@ recipe that borrows this static-link pattern must apply the same per-platform ga --- +### `dlopen failed: cannot locate symbol "__clear_cache"` (Android **arm64** only, a Rust extension) + +**Cause:** rustc links cdylibs with `-nodefaultlibs`, so the NDK's +`libclang_rt.builtins--android.a` never reaches the link, and Rust's +`compiler_builtins` lacks some symbols that C code in `-sys` crates calls. arm64 libffi's +instruction-cache flush is the known one. The `.so` links (a shared library may keep +undefined symbols), then bionic refuses it at `dlopen`. x86_64 never references +`__clear_cache`, so **CI's x86_64 emulator cannot catch this**. Tell it from the jniLibs +collision above by the symbol: a compiler builtin, not one of your libraries'. + +**Fix:** forge now appends the builtins archive to every Android Rust link +(`cargo_ldflags`, next to the libgcc shim); only members for still-undefined symbols are +pulled. If you still see it, check the archive glob matches the NDK layout. Found on +vl-convert-python. + +--- + +### `fatal runtime error: out of TLS keys, aborting` (Android) — often in a module that is NOT the one using them up + +**Cause:** Rust's Android targets have no native TLS (`has-thread-local` unset, +`tls-model: emulated` in `rustc --print target-spec-json`), so std backs every +`thread_local!` it touches with its own pthread key, and bionic allows a process about +128. Each Rust extension carries its own std. vl-convert's Deno worker took about 80, so +the next Rust module to load (rpds-py, under Altair) aborted the whole app. iOS has +native TLS and 512 keys and is unaffected. + +**Measure it** before and after a suspect call, from Python on the device: +```python +libc = ctypes.CDLL("libc.so"); keys, k = [], ctypes.c_uint() +while libc.pthread_key_create(ctypes.byref(k), None) == 0: keys.append(k.value) +for key in keys: libc.pthread_key_delete(key) # len(keys) = keys still free +``` + +**Fix:** link a C shim into the extension that defines `pthread_key_create`, +`pthread_key_delete`, `pthread_getspecific` and `pthread_setspecific` with +`visibility("hidden")` and multiplexes them onto one real key holding a per-thread table +(destructors run at thread exit with POSIX iteration semantics). Hidden definitions bind +every reference inside the `.so` and nothing outside it. Link it `static:+whole-archive`, +since libc.so also defines the names. Precedent: `forge_tls.c` in +`recipes/vl-convert-python/patches/mobile.patch` (95 keys free afterwards, against 4). +Verify with `llvm-nm -D | grep pthread_key`, which should print nothing. + +Before copying it into another extension, check what creates keys there: the shim never +reuses a deleted ID, so it caps creates at 4,095 per process. That is fine for Rust std, +which deletes a key only when it loses a race to initialise one, and wrong for code that +creates and deletes keys per object. Reuse needs a generation per slot, as bionic has, so +a new key cannot read the deleted key's values. The real-key count above cannot see this +limit. + +--- + +### `Failed to initialize a JsRuntime: No such file or directory (os error 2)` (a Rust crate embedding `deno_core`, on device) + +Python only sees `Failed to receive worker startup status: receiving on a closed channel` +when the runtime lives on a worker thread; capture the panic (Diagnostic snippets). + +**Cause:** deno_core 0.411's `include_js_files!` records each extension source as an +absolute **build-machine path** and reads it only while a snapshot is made. A runtime +started with `startup_snapshot: None` reads those paths at startup: they exist on the +machine that built the wheel (so a desktop test, or an iOS simulator on that machine, +passes) and not on a phone. deno_runtime attaches `99_main.js` the same way by hand. + +**Fix:** patch the macros to `mode=included` and deno_runtime's `99_main.js` push to +`ascii_str_include!` (`crate_patches`). Check the binary: +`grep -a -o -E '/(home|Users)/[^"[:space:]]*\.(js|ts)' ` should find (almost) nothing. +Precedent: vl-convert-python's `deno_core.patch` and `deno_runtime.patch`. + +--- + +### `Expect rustls-platform-verifier to be initialized` (Android panic on the first HTTPS request) + +**Cause:** reqwest 0.13's `rustls` feature verifies certificates through +rustls-platform-verifier, which on Android calls into Java and needs +`init_hosted(env, context)` first. A Python extension never does that. The panic happens +at TLS-handshake time, so import and offline work look fine. + +**Fix:** on Android, give the client its own roots so reqwest skips the platform verifier: +`ClientBuilder::tls_certs_only(...)` with `webpki-root-certs` converted to +`reqwest::Certificate`. Server certificates installed only on the device are then not +trusted; say so on the recipe page. Precedent: `MobileTls` in vl-convert-python's +`mobile.patch`. + +--- + ### iOS: `import ` → `symbol not found in flat namespace '……'` at dlopen (a CMake OBJECT library never made it into the `.framework`) **Cause:** the build is GREEN but a CMake `OBJECT` library's objects — meant to link @@ -2328,6 +2477,17 @@ republishing". Full authoring guidance in `new-mobile-recipe` § 3.5b. ## Diagnostic snippets +### Capture a Rust panic on Android, and keep results when the app aborts + +Android discards fd 2, so a Rust panic's message never reaches logcat, and a process +abort kills pytest before its failure summary reaches `console.log`. Redirect fd 2 and +write results to a file in the app's cache dir, then read both with `adb shell cat`: +```python +err = os.path.join(tempfile.gettempdir(), "stderr.txt") # the app cache dir +os.dup2(os.open(err, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644), 2) +os.environ["RUST_BACKTRACE"] = "1" +``` + ### Inspect a wheel's contents ```bash diff --git a/.claude/skills/local-recipe-testing/SKILL.md b/.claude/skills/local-recipe-testing/SKILL.md index a1e47685..dcdf1224 100644 --- a/.claude/skills/local-recipe-testing/SKILL.md +++ b/.claude/skills/local-recipe-testing/SKILL.md @@ -206,6 +206,15 @@ carries every control value: `xcrun simctl spawn "$UDID" log show --last 2m --st **Two iOS builds running at once cause the same symptom.** Both stage into that one shared `dist_ios`, so whichever finishes second can bundle the other app's files (seen 2026-10-01: a recipe-tester `.app` carrying a consumer app's `main.pyc`). If anything else on the machine might be building for iOS — another terminal, another agent — give each build its own `PUB_CACHE=` (the first run re-downloads the pub packages), and pass the same `PUB_CACHE` to any manual `flutter build` that follows. +15. **CI's Android test runs only x86_64, so a green CI says nothing about arm64, which is every phone.** + vl-convert passed CI's emulator build while its arm64 `.so` could not even load + (`cannot locate symbol "__clear_cache"`: x86_64 never references it). Do a local arm64 + pass for any Rust or C-heavy recipe before trusting the matrix. Two related traps: + an iOS simulator on the machine that **built** the wheel can pass while the binary + reads build-machine paths a phone will not have, and a native abort kills pytest + before its summary reaches `console.log`. For both, see the `forge-error-catalogue` + diagnostic snippets (capture fd 2, write results to a file). + ## Model assets & test-only deps `stage_recipe.sh` copies **every** file in `recipes//tests/` into the app (`cp -r tests/. recipe_tests/`), so a model dropped next to the test file becomes an app asset. Two tiers: diff --git a/.claude/skills/new-mobile-recipe/SKILL.md b/.claude/skills/new-mobile-recipe/SKILL.md index 1aee8b15..93441c1c 100644 --- a/.claude/skills/new-mobile-recipe/SKILL.md +++ b/.claude/skills/new-mobile-recipe/SKILL.md @@ -88,6 +88,7 @@ Match the package to one of these shapes. Each maps to a template in `templates/ | CMake giant, **no sdist AND no setup.py/pyproject.toml** | Upstream's only wheel path is a host==target build script (onnxruntime's `ci_build/build.py`, TF's `build_pip_package_with_cmake.sh`) | No template — copy from `recipes/onnxruntime/` or `recipes/tflite-runtime/` (branches `machine/onnxruntime` / `machine/tflite-runtime`); see "PEP 517 shim" deep-dive below | | C-ext linking a `flet-lib*`'s **shared** libs via `pkg-config` | Upstream's `setup.py` runs `pkg-config --cflags --libs …` and the package has too many extension modules to static-link into (PyAV: 49) | No template — copy `recipes/flet-libffmpeg/` + `recipes/av/`: the flet-lib ships SHARED libs plus **relocatable** `.pc` files (forge already has `opt/lib/pkgconfig` on `PKG_CONFIG_LIBDIR`, so the consumer needs no `script_env` for it), and the consumer adds `-Wl,-headerpad_max_install_names` on iOS. See "Cross-cutting conventions" in `references/recipe-patterns.md` | | **Prebuilt-repackage + host_build chain** | Upstream publishes official prebuilt mobile archives of the native lib AND the consumer's own cmake links + re-ships the `.so` (flet-libonnxruntime→sherpa-onnx) | `build.sh` repackager + consumer `requirements.host_build`; copy from `recipes/flet-libonnxruntime/` + `recipes/sherpa-onnx/` (branch `machine/sherpa-onnx`); see "prebuilt-repackage" deep-dive below | +| **Rust crate embedding Deno (`deno_core`/`deno_runtime`)** | The package runs JavaScript in Deno, i.e. V8 (vl-convert). V8 has no Android build and a host-made V8 snapshot cannot cross configurations | No template — copy `recipes/vl-convert-python/`: deno_core's `quickjs` feature (QuickJS-ng via the `deno_v8` facade), no snapshot with the extension sources embedded, Deno cfg gaps through `crate_patches`. Read its Build notes first: two failures only show on an arm64 device | | **cffi/ctypes consumer of a prebuilt archive** | The package's own build DOWNLOADS a prebuilt static lib keyed by the build host's `uname` and statically links it (curl-cffi → curl-impersonate); breaks under cross because host≠target | `flet-lib*` prebuilt-repackage dep (`source.strip: 0` for root-level tarballs) + `requirements.host_build` + a `mobile.patch` opt-in env lever that steers arch/link off the forge target; copy from `recipes/flet-libcurl-impersonate/` + `recipes/curl-cffi/` (branch `curl-cffi`); see deep-dive below | If unsure, start with **minimal C-extension** and let the build tell you what's missing. Iterate up the table as failures surface. @@ -358,6 +359,27 @@ Renders the meta.yaml for all SDK contexts (iphoneos, iphonesimulator, android) with a non-matching needle silently no-ops. Always `assert needle in text` before replacing, or grep the output file for the new content after. +### 3.5a — Patching a crates.io dependency (`crate_patches`) + +`patches` only reaches the sdist; a Rust package's dependencies are fetched by cargo at +build time. To patch one, list it in meta.yaml: + +```yaml +crate_patches: + deno_core: # or name@version when Cargo.lock holds several versions + - deno_core.patch # in patches/, applied with patch -p1 from the crate root +``` + +forge downloads the exact version the source's `Cargo.lock` resolves, checks its +sha256 against the lockfile, unpacks it into `forge-crates/`, applies the patches and adds +a `[patch.crates-io]` entry to the root `Cargo.toml`, joining upstream's table when it has +one. If that table already patches the same crate (say, to a git fork), forge stops: change +upstream's entry with the sdist patch instead. Generate each patch from a pristine +copy of `~/.cargo/registry/src/*/-` the same way as an sdist patch, and keep +the sdist patch from rewriting `Cargo.lock`: forge reads versions and checksums from it. +Running `cargo metadata` or `cargo tree` in your edited tree rewrites the lockfile, which +then lands in the diff. + ### 3.5b — Licences (`flet-lib*` / build.sh recipes only) A **Python package** carries its own licence: the build backend copies upstream's METADATA diff --git a/recipes/vl-convert-python/README.md b/recipes/vl-convert-python/README.md new file mode 100644 index 00000000..038313bf --- /dev/null +++ b/recipes/vl-convert-python/README.md @@ -0,0 +1,229 @@ +# vl-convert-python + +[`vl-convert-python`](https://github.com/vega/vl-convert) converts +[Vega-Lite](https://vega.github.io/vega-lite/) and [Vega](https://vega.github.io/vega/) +chart specifications to SVG, PNG, JPEG and PDF. It is the engine behind +[Altair](https://altair-viz.github.io/)'s `chart.save("chart.png")`. In a Flet app, the chart +is compiled and rendered on the device, so a spec turns into an image without a WebView, a +server or a network connection. + +The mobile wheel runs Vega's JavaScript on [QuickJS-ng](https://github.com/quickjs-ng/quickjs) +instead of the V8 the desktop wheel embeds, which vl-convert cannot ship for iOS or Android +(see Build notes). The Python API is the same as on the desktop. + +## Install + +Give `vl-convert-python` a 2.0 lower bound in your `pyproject.toml`, and leave `armeabi-v7a` +out of the Android targets: + +```toml +dependencies = [ + "flet", + "vl-convert-python>=2.0.0rc7", +] + +[tool.flet.android] +target_arch = ["arm64-v8a", "x86_64"] +``` + +**The bound keeps desktop and mobile on the same API.** The mobile wheel exists only for the +2.0 series. A mobile build picks it without the bound, since it is the only mobile wheel, but +`flet run` on a desktop then installs PyPI's 1.x, which has no `vl_convert.asyncio` or +`configure`: code that works on the phone fails with `AttributeError` at your desk. Naming a +pre-release in the bound is also what lets the resolver choose one. + +**There is no 32-bit ARM wheel.** `flet build apk` targets every ABI by default, so without the +`target_arch` line the `armeabi-v7a` resolve fails and takes the whole build with it. `x86_64` +is the emulator; use `["arm64-v8a"]` alone for a device-only release. + +Altair is pure Python: add it to the same list and `chart.save(...)` finds the mobile +`vl-convert-python`. Altair tells vl-convert which Vega-Lite version to compile with, and +vl-convert accepts only the versions it bundles (`vlc.get_vegalite_versions()`), so a newer +Altair can outrun the wheel; 6.3.0 is the version tested. + +## Examples + +See runnable Flet apps in [`examples/`](examples): + +- [`chart-gallery`](examples/chart-gallery) — converts three Vega-Lite specs to PNG and shows + how long each conversion took. + +## Usage in a Flet app + +A spec is a dict (or a JSON string). Convert it and hand the bytes to +[`ft.Image`](https://flet.dev/docs/controls/image/): + +```python +import vl_convert as vlc + +png = vlc.vegalite_to_png(spec, scale=2) +chart = ft.Image(src=png) +``` + +`vegalite_to_svg` returns the SVG as a string and `vegalite_to_pdf` returns PDF bytes. With +Altair, call `chart.save(path)` or pass `chart.to_dict()` to the same functions. + +### Storage + +vl-convert keeps two caches, and each resolves its directory once, on the first conversion. +Point them at [`FLET_APP_STORAGE_CACHE`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_cache) +before importing it: + +```python +import os +import tempfile + +# Absolute either way: vl-convert rejects a relative font cache directory. +cache = os.path.join(os.getenv("FLET_APP_STORAGE_CACHE") or tempfile.gettempdir(), "vl-convert") +os.environ.setdefault("V82JSC_BC_CACHE_DIR", os.path.join(cache, "bytecode")) +os.environ.setdefault("VLC_GOOGLE_FONTS_CACHE_DIR", os.path.join(cache, "google-fonts")) + +import vl_convert as vlc +``` + +- `V82JSC_BC_CACHE_DIR` holds compiled JavaScript, about 3.3 MB. It is what makes the next + launch fast (see below); set it to an unwritable path, or `V82JSC_NO_BC_CACHE=1`, and every + launch recompiles Vega. +- `VLC_GOOGLE_FONTS_CACHE_DIR` holds fonts downloaded for `google_fonts=` requests. Set it to + `none` to keep them in memory only. + +Specs that read files need an absolute path and an explicit allowance: by default vl-convert +loads data over HTTP and HTTPS but not from the filesystem. See +[`configure(allowed_base_urls=...)`](https://github.com/vega/vl-convert/tree/main/vl-convert-python#readme). + +### Startup and speed + +The first conversion in a process starts the JavaScript runtime, and that is where the time +goes: the desktop wheel restores V8 from a snapshot in well under a second, while the mobile +build has no snapshot and evaluates Deno's runtime and Vega from scratch. Conversions after +that are quick. Measured with the example app: + +| first chart | arm64 Android emulator | iOS simulator | +| --- | ---: | ---: | +| first launch, empty bytecode cache | ~10 s | ~4.4 s | +| later launches, bytecode cache in place | ~3 s | ~0.6 s | +| every conversion after the first | ~0.5 s | | + +Both columns were measured on an Apple Silicon Mac, whose CPU the emulator and the simulator +both use; measure on a phone before relying on them. + +Do a throwaway conversion while the user is still looking at something else, so the first real +chart is not the one that waits. `warm_up_workers()` is not enough: it starts the runtime but +leaves Vega and Vega-Lite to load on the first conversion. + +```python +page.run_thread(vlc.vegalite_to_svg, {"mark": "point"}) +``` + +Do not judge startup from `flet run` on a desktop, which uses the V8 wheel and its snapshot. + +### Threading + +Conversions block the calling thread. Run them in +[`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread) and end +the handler with [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update), or +use the `vl_convert.asyncio` variants from an async handler: + +```python +async def show(e): + chart.src = await vlc.asyncio.vegalite_to_png(spec, scale=2) + page.update() +``` + +Conversions run on one worker by default, which keeps memory to a single JavaScript runtime. +Leave [`configure(num_workers=...)`](https://github.com/vega/vl-convert/tree/main/vl-convert-python#readme) at 1 unless you +convert in parallel and can afford one runtime per worker. + +### App size + +Expect about 29 MB of compressed wheel and 70 MB unpacked per Android architecture (arm64-v8a +measured), and 26 MB and 57–66 MB per iOS slice. Almost all of it is the one native module +that carries the Deno runtime, QuickJS and the bundled Vega and Vega-Lite. Use an app bundle, split APKs or a narrow +[`target_arch`](https://flet.dev/docs/publish/android/#supported-target-architectures) when +that matters. + +### Other considerations + +A desktop `flet run` installs PyPI's V8-based wheel. Besides startup, text can differ: the +desktop wheel sees the computer's fonts, while the device has only the bundled Liberation Sans, +so a font named in a spec renders at your desk and not on the phone (see below). Check timing, +memory and typography on a device. + +## Things to know + +- **The device's own fonts are not found; Liberation Sans is the only one there.** The wheel + bundles it and `sans-serif` maps to it. What happens to a family it cannot find depends on + the output: PNG falls back to Liberation Sans, while JPEG, PDF and `svg_to_png` leave that + text out. Name `sans-serif` or a family you registered: ship font files as + [assets](https://flet.dev/docs/cookbook/assets) and call `vlc.register_font_directory(...)` + with their absolute path, or request `google_fonts=[...]` with network access. +- **Remote data needs the network; inline data does not.** A spec whose `data.url` points at + the web is fetched during conversion. On Android the HTTPS check uses the Mozilla root + certificates bundled in the wheel, so a server signed by a certificate installed only on the + device is rejected. +- **The first conversion is slow and the rest are not.** See [Startup and speed](#startup-and-speed); + a spinner on the first chart is worth it even with the cache in place. + +## Build notes (maintainers) + +### Recipe shape + +vl-convert executes Vega inside Deno, and Deno's engine is V8. V8 was the first route and was +rejected: rusty_v8 publishes no Android build at all (building it from source means hours of +GN/ninja per ABI), its iOS builds are arm64-only and jitless, and vl-convert's build-time V8 +snapshot, made by a host V8 with a different configuration, cannot load on either. deno_core +0.411 can instead run on QuickJS-ng through its `deno_v8` facade, which turns the engine into +an ordinary C dependency. + +Running without a snapshot exposed that Deno embeds none of its own JavaScript: extension +sources are recorded as build-machine paths and read only while a snapshot is made. The wheel +therefore embeds them (`deno_core.patch`, `deno_runtime.patch`). A QuickJS snapshot made on the +build host was the alternative and was rejected: v8x replays snapshot callbacks by op index, +Deno registers some ops per OS, and CI builds Android on Linux and iOS on macOS. + +Two problems surfaced on an arm64 emulator: + +- `__clear_cache` was left undefined, because rustc links with `-nodefaultlibs` and only + aarch64 code calls it; forge now passes the NDK's compiler-rt builtins to every Android Rust + link. CI's emulator is x86_64, so only an arm64 run catches this one. +- Rust has no native TLS on any Android ABI, so every `thread_local!` costs a pthread key, and + one worker took about 80 of bionic's 128. The next native module to load (rpds-py, under + Altair) aborted with `fatal runtime error: out of TLS keys`. `forge_tls.c` multiplexes all of + the extension's keys onto one; `test_leaves_tls_keys_for_other_modules` guards it on every + Android leg. + +The Deno crates that do not compile for Android or iOS are patched through forge's +`crate_patches`, one file per crate; each preamble says why. + +### Upgrade hazards + +- **2.0 final, or any later release, means re-diffing every patch.** The crate patches pin + exact versions through `Cargo.lock`, and forge refuses a crate that no longer matches. +- **deno_core's source macros.** If a new deno_core changes `include_js_files!`, the embed + patch may still apply while sources go back to being paths. The check below catches it. +- **Six crates are not locked.** QuickJS pulls in `icu_collator`, `icu_locale`, their data + crates, `utf16_iter` and `write16`, which upstream's V8 lockfile never needed, so cargo picks + their newest compatible versions at build time. A build that breaks with no recipe change + should be checked against them first. +- **v8x is a release candidate**, pulled in by `deno_v8` with an exact pin. A new deno_core + brings a new v8x, whose 64-bit-only bindings are why `armeabi-v7a` is excluded. + +### Re-verification checklist + +- **No build paths in the binary:** `grep -a -o -E '/(home|Users)/[^"[:space:]]*\.(js|ts)' ` + should find only vl-convert's two module specifiers (`vl-convert-index.js`, + `vl-plugin-entry.js`). Hundreds of hits mean sources are paths again, and the worker panics + on device with "Failed to initialize a JsRuntime: No such file or directory". +- **An arm64 device or emulator run,** not only CI, whose x86_64 emulator cannot see an + arm64-only failure such as `__clear_cache`. +- **The TLS key test** (`test_leaves_tls_keys_for_other_modules`) and the Altair test. +- **Timings and sizes,** re-measured from the example and the wheels. + +### Coverage gaps + +The device tests run on CI's x86_64 Android emulator and iOS simulator, and were run on an +arm64 Android emulator by hand. They cover SVG and PNG conversion, the asyncio variants, text +rendering, the bytecode cache location, the TLS key budget (Android only) and Altair's +`chart.save`. They are network-free, so remote `data.url` loading and Google Fonts downloads +are not exercised; remote data was verified by hand on the arm64 Android emulator only. PDF +and JPEG output, Vega plugins and locales are untested on device. diff --git a/recipes/vl-convert-python/examples/chart-gallery/.gitignore b/recipes/vl-convert-python/examples/chart-gallery/.gitignore new file mode 100644 index 00000000..429a8307 --- /dev/null +++ b/recipes/vl-convert-python/examples/chart-gallery/.gitignore @@ -0,0 +1,7 @@ +.venv/ +.flet/ +build/ +__pycache__/ +.pytest_cache/ +.ruff_cache/ +uv.lock diff --git a/recipes/vl-convert-python/examples/chart-gallery/README.md b/recipes/vl-convert-python/examples/chart-gallery/README.md new file mode 100644 index 00000000..8d605086 --- /dev/null +++ b/recipes/vl-convert-python/examples/chart-gallery/README.md @@ -0,0 +1,41 @@ +# vl-convert chart gallery + +Three [Vega-Lite](https://vega.github.io/vega-lite/) charts, converted to PNG on the device +and shown in an [`ft.Image`](https://flet.dev/docs/controls/image/). Pick one with the +segmented button; the caption reports the size of the PNG and how long the conversion took. + +What it demonstrates: + +- **A spec in, a PNG out.** + [`vl_convert.vegalite_to_png`](https://github.com/vega/vl-convert#readme) takes a Python + dict and returns PNG bytes, which `ft.Image.src` accepts directly. No WebView, no network. +- **Vega doing the computing.** *Bars* colours bars with a conditional encoding, *Lines* + draws a legend for three series, and *Fit* lays a LOESS curve over scattered points. The + curve is computed by Vega's `loess` transform, in JavaScript, inside the app. +- **The first conversion is the slow one.** It boots the JavaScript runtime, and the caption + shows it. Switching charts afterwards reuses the warm runtime. +- **Caches in app storage.** `charts.py` points vl-convert's bytecode cache and Google Fonts + cache at [`FLET_APP_STORAGE_CACHE`](https://flet.dev/docs/reference/environment-variables/#flet_app_storage_cache) + before importing it, so the next launch skips recompiling Vega. +- **Off the UI thread.** Conversion runs in + [`page.run_thread(...)`](https://flet.dev/docs/controls/page/#flet.Page.run_thread) with a + spinner up, and the handler ends with an explicit + [`page.update()`](https://flet.dev/docs/controls/page/#flet.Page.update). + +## Try it + +[Build](https://flet.dev/docs/publish/) the app, then install it on a device or +emulator/simulator: + +```bash +# Android +uv run flet build apk + +# iOS +uv run flet build ipa + +# iOS-Simulator +uv run flet build ios-simulator +``` + +`flet run` works on a desktop too, using PyPI's desktop wheel of the same version. diff --git a/recipes/vl-convert-python/examples/chart-gallery/pyproject.toml b/recipes/vl-convert-python/examples/chart-gallery/pyproject.toml new file mode 100644 index 00000000..d1633b08 --- /dev/null +++ b/recipes/vl-convert-python/examples/chart-gallery/pyproject.toml @@ -0,0 +1,19 @@ +[project] +name = "vl-convert-chart-gallery" +version = "1.0.0" +description = "Renders Vega-Lite specs to PNG on the device with vl-convert." +requires-python = ">=3.10" + +dependencies = [ + "flet==1.0.1", + "vl-convert-python==2.0.0rc7", +] + +[dependency-groups] +dev = ["flet-cli", "flet-desktop", "flet-web"] + +[tool.flet.app] +path = "src" + +[tool.flet.android] +target_arch = ["arm64-v8a", "x86_64"] diff --git a/recipes/vl-convert-python/examples/chart-gallery/src/charts.py b/recipes/vl-convert-python/examples/chart-gallery/src/charts.py new file mode 100644 index 00000000..91489751 --- /dev/null +++ b/recipes/vl-convert-python/examples/chart-gallery/src/charts.py @@ -0,0 +1,96 @@ +import math +import os +import tempfile +import time +from pathlib import Path + +# The JavaScript bytecode cache and the Google Fonts cache each resolve their +# directory once, on first use, so point them at app storage before importing. +# Absolute either way: vl-convert rejects a relative font cache directory. +CACHE = Path(os.getenv("FLET_APP_STORAGE_CACHE") or tempfile.gettempdir()) / "vl-convert" +os.environ.setdefault("V82JSC_BC_CACHE_DIR", str(CACHE / "bytecode")) +os.environ.setdefault("VLC_GOOGLE_FONTS_CACHE_DIR", str(CACHE / "google-fonts")) + +import vl_convert as vlc # noqa: E402 + +VERSIONS = ( + f"Vega {vlc.get_vega_version()} · Vega-Lite {vlc.get_vegalite_versions()[-1]}" +) + +MONTHS = ("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec") +RAIN = (78, 61, 52, 47, 59, 41, 33, 38, 55, 81, 96, 89) +HIGHS = { + "Lisbon": (15, 16, 19, 20, 23, 27, 29, 29, 27, 23, 18, 15), + "Oslo": (-1, 0, 4, 10, 16, 20, 22, 21, 16, 9, 4, 0), + "Lagos": (32, 33, 33, 32, 31, 29, 28, 28, 29, 30, 31, 32), +} + + +def scatter_points(): + """A noisy but fixed curve, so the fitted line has something to find.""" + return [ + {"x": x, "y": round(3 + 0.8 * x + 2.5 * math.sin(x * 1.7), 2)} + for x in (i * 0.5 for i in range(40)) + ] + + +SPECS = { + "Bars": { + "title": "Rainfall (mm)", + "data": {"values": [{"month": m, "mm": r} for m, r in zip(MONTHS, RAIN)]}, + "mark": {"type": "bar", "cornerRadiusEnd": 3}, + "encoding": { + "x": {"field": "month", "type": "ordinal", "sort": None, "title": None}, + "y": {"field": "mm", "type": "quantitative", "title": None}, + "color": { + "condition": {"test": "datum.mm > 80", "value": "#e4572e"}, + "value": "#4c78a8", + }, + }, + }, + "Lines": { + "title": "Average high (°C)", + "data": { + "values": [ + {"city": city, "month": m, "high": h} + for city, highs in HIGHS.items() + for m, h in zip(MONTHS, highs) + ] + }, + "mark": {"type": "line", "point": True}, + "encoding": { + "x": {"field": "month", "type": "ordinal", "sort": None, "title": None}, + "y": {"field": "high", "type": "quantitative", "title": None}, + "color": {"field": "city", "type": "nominal", "legend": {"orient": "bottom"}}, + }, + }, + "Fit": { + "title": "Points and a LOESS fit computed by Vega", + "data": {"values": scatter_points()}, + "layer": [ + {"mark": {"type": "point", "filled": True, "opacity": 0.6}}, + { + "mark": {"type": "line", "color": "#e4572e", "strokeWidth": 3}, + "transform": [{"loess": "y", "on": "x", "bandwidth": 0.3}], + }, + ], + "encoding": { + "x": {"field": "x", "type": "quantitative"}, + "y": {"field": "y", "type": "quantitative"}, + }, + }, +} + + +def render(name, width): + """Convert one spec to PNG bytes sized to `width` logical pixels. + + Returns the PNG, its pixel size and the seconds the conversion took. The + first call in a process also boots the JavaScript runtime, so it is the slow one. + """ + spec = {**SPECS[name], "width": width, "height": round(width * 0.6)} + started = time.perf_counter() + png = vlc.vegalite_to_png(spec, scale=2) + elapsed = time.perf_counter() - started + size = int.from_bytes(png[16:20], "big"), int.from_bytes(png[20:24], "big") + return png, size, elapsed diff --git a/recipes/vl-convert-python/examples/chart-gallery/src/main.py b/recipes/vl-convert-python/examples/chart-gallery/src/main.py new file mode 100644 index 00000000..d192e0db --- /dev/null +++ b/recipes/vl-convert-python/examples/chart-gallery/src/main.py @@ -0,0 +1,71 @@ +import flet as ft +from charts import SPECS, VERSIONS, render + + +def main(page: ft.Page): + def show(name): + """Start converting `name` on a background thread, spinner up.""" + spinner.visible = True + picker.disabled = True # one conversion at a time + page.update() + page.run_thread(work, name) + + def work(name): + """Convert the spec to PNG off the UI thread, then show it with its timing.""" + try: + png, (width, height), elapsed = render(name, 280) + except Exception as exc: + caption.value = f"{type(exc).__name__}: {exc}" + else: + chart.src = png + chart.visible = True + caption.value = f"{width}×{height} px PNG in {elapsed:.2f} s" + spinner.visible = False + picker.disabled = False + page.update() # auto-update does not reach background threads + + page.appbar = ft.AppBar(title="Vega-Lite charts", center_title=True) + page.add( + ft.SafeArea( + expand=True, + content=ft.Column( + horizontal_alignment=ft.CrossAxisAlignment.CENTER, + controls=[ + picker := ft.SegmentedButton( + selected=["Bars"], + segments=[ft.Segment(value=n, label=n) for n in SPECS], + on_change=lambda e: show(e.control.selected[0]), + ), + ft.Container( + expand=True, + alignment=ft.Alignment.CENTER, + # Hidden until the first chart: an empty src draws Flet's error box. + # gapless_playback keeps the old chart up until the new one decodes. + content=( + chart := ft.Image( + src=b"", + visible=False, + fit=ft.BoxFit.CONTAIN, + gapless_playback=True, + ) + ), + ), + ft.Row( + alignment=ft.MainAxisAlignment.CENTER, + controls=[ + caption := ft.Text( + "First chart boots the JavaScript runtime…", size=12 + ), + spinner := ft.ProgressRing(width=14, height=14), + ], + ), + ft.Text(VERSIONS, size=11, color=ft.Colors.ON_SURFACE_VARIANT), + ], + ), + ) + ) + show("Bars") + + +if __name__ == "__main__": + ft.run(main) diff --git a/recipes/vl-convert-python/meta.yaml b/recipes/vl-convert-python/meta.yaml new file mode 100644 index 00000000..7d27fa13 --- /dev/null +++ b/recipes/vl-convert-python/meta.yaml @@ -0,0 +1,38 @@ +package: + name: vl-convert-python + version: "2.0.0rc7" + # v8x ships its V8 API bindings pre-generated for 64-bit targets, with + # compile-time size asserts that cannot hold where pointers are 4 bytes. + excluded_arches: [armeabi-v7a] + +build: + number: 1 + script_env: + _PYTHON_SYSCONFIGDATA_NAME: '{sysconfigdata_name}' + +requirements: + build: + - cmake + +patches: + - mobile.patch + +crate_patches: + deno_core: + - deno_core.patch + deno_fs: + - deno_fs.patch + deno_io: + - deno_io.patch + deno_node: + - deno_node.patch + deno_os: + - deno_os.patch + deno_runtime: + - deno_runtime.patch + deno_signals: + - deno_signals.patch + +test: + requires: + - altair==6.3.0 diff --git a/recipes/vl-convert-python/patches/deno_core.patch b/recipes/vl-convert-python/patches/deno_core.patch new file mode 100644 index 00000000..079f47f4 --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_core.patch @@ -0,0 +1,75 @@ +deno_core 0.411.0: embed extension sources; errno on Android and iOS. + +1. extensions.rs: include_js_files! and include_lazy_loaded_js_files! record + each extension source by its absolute build-machine path and read it only + while a snapshot is being made. This build boots without a snapshot (QuickJS + cannot load a V8 one), so the runtime would read those paths on the device + and fail with "Failed to initialize a JsRuntime: No such file or directory". + Switch both to `included`, which embeds the files with include_str!. + +2. uv_compat/tty.rs reads errno through a per-platform extern and has a + compile_error!() for every OS but macOS and Linux. iOS uses Darwin's + __error(); Android's bionic exposes __errno(). + +diff -ruN a/extensions.rs b/extensions.rs +--- a/extensions.rs ++++ b/extensions.rs +@@ -943,7 +943,7 @@ + $(with_specifier $s2:literal)? + $(= $config:tt)? + ),* $(,)?) => { +- $crate::__extension_include_js_files_inner!(mode=loaded, name=$name, dir=$crate::__extension_root_dir!($($dir)?), $([ ++ $crate::__extension_include_js_files_inner!(mode=included, name=$name, dir=$crate::__extension_root_dir!($($dir)?), $([ + // These entries will be parsed in __extension_include_js_files_inner + $s1 $(with_specifier $s2)? $(= $config)? + ]),*) +@@ -973,7 +973,7 @@ + #[doc(hidden)] + #[macro_export] + macro_rules! __extension_include_js_files_detect { +- ($($rest:tt)*) => { $crate::__extension_include_js_files_inner!(mode=loaded, $($rest)*) }; ++ ($($rest:tt)*) => { $crate::__extension_include_js_files_inner!(mode=included, $($rest)*) }; + } + + /// This is the core of the [`include_js_files!`] and [`include_lazy_loaded_js_files`] macros. The first +diff -ruN a/uv_compat/tty.rs b/uv_compat/tty.rs +--- a/uv_compat/tty.rs ++++ b/uv_compat/tty.rs +@@ -381,7 +381,7 @@ + } + + /// Get a pointer to the thread-local errno value. +- #[cfg(target_os = "macos")] ++ #[cfg(any(target_os = "macos", target_os = "ios"))] + fn errno_location() -> *mut c_int { + unsafe extern "C" { + fn __error() -> *mut c_int; +@@ -389,6 +389,14 @@ + unsafe { __error() } + } + ++ #[cfg(target_os = "android")] ++ fn errno_location() -> *mut c_int { ++ unsafe extern "C" { ++ fn __errno() -> *mut c_int; ++ } ++ unsafe { __errno() } ++ } ++ + #[cfg(target_os = "linux")] + fn errno_location() -> *mut c_int { + unsafe extern "C" { +@@ -397,7 +405,12 @@ + unsafe { __errno_location() } + } + +- #[cfg(not(any(target_os = "macos", target_os = "linux")))] ++ #[cfg(not(any( ++ target_os = "macos", ++ target_os = "ios", ++ target_os = "linux", ++ target_os = "android" ++ )))] + fn errno_location() -> *mut c_int { + compile_error!( + "errno_location not implemented for this platform — \ diff --git a/recipes/vl-convert-python/patches/deno_fs.patch b/recipes/vl-convert-python/patches/deno_fs.patch new file mode 100644 index 00000000..c46fdb45 --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_fs.patch @@ -0,0 +1,74 @@ +deno_fs 0.168.0: treat iOS as macOS. + +Every target_os = "macos" cfg also matches ios. Several of these branches have +no fallback, so iOS fails to compile without them; the rest select Darwin APIs +that iOS shares. Regenerate on a bump by running, from the crate root: + + perl -0pi -e 's/target_os = "macos"/any(target_os = "macos", target_os = "ios")/g' $(grep -rl 'target_os = "macos"' .) + +diff -ruN a/std_fs.rs b/std_fs.rs +--- a/std_fs.rs ++++ b/std_fs.rs +@@ -86,7 +86,7 @@ + Ok(r.bits()) + } + #[cfg(any( +- target_os = "macos", ++ any(target_os = "macos", target_os = "ios"), + target_os = "openbsd", + target_os = "freebsd" + ))] +@@ -568,7 +568,7 @@ + Err(FsError::NotSupported) + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn lchmod(path: &Path, mode: u32) -> FsResult<()> { + use std::os::unix::fs::OpenOptionsExt; + use std::os::unix::fs::PermissionsExt; +@@ -583,7 +583,7 @@ + Ok(()) + } + +-#[cfg(not(target_os = "macos"))] ++#[cfg(not(any(target_os = "macos", target_os = "ios")))] + fn lchmod(_path: &Path, _mode: u32) -> FsResult<()> { + Err(FsError::NotSupported) + } +@@ -669,7 +669,7 @@ + ); + } + +- #[cfg(target_os = "macos")] ++ #[cfg(any(target_os = "macos", target_os = "ios"))] + { + use std::ffi::CString; + use std::os::unix::fs::OpenOptionsExt; +@@ -834,7 +834,7 @@ + copy_file(from, to) + } + +- #[cfg(target_os = "macos")] ++ #[cfg(any(target_os = "macos", target_os = "ios"))] + { + // Just clonefile() + use std::ffi::CString; +@@ -1004,7 +1004,7 @@ + cpath.push(0); + if bigint { + #[cfg(not(any( +- target_os = "macos", ++ any(target_os = "macos", target_os = "ios"), + target_os = "freebsd", + target_os = "openbsd" + )))] +@@ -1014,7 +1014,7 @@ + (libc::statfs64(cpath.as_ptr() as _, &mut result), result) + }; + #[cfg(any( +- target_os = "macos", ++ any(target_os = "macos", target_os = "ios"), + target_os = "freebsd", + target_os = "openbsd" + ))] diff --git a/recipes/vl-convert-python/patches/deno_io.patch b/recipes/vl-convert-python/patches/deno_io.patch new file mode 100644 index 00000000..250f6881 --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_io.patch @@ -0,0 +1,30 @@ +deno_io 0.168.0: Darwin socketpair path on iOS. + +Darwin has no SOCK_CLOEXEC; macOS sets FD_CLOEXEC with fcntl() instead, and +iOS needs the same branch. + +diff -ruN a/bi_pipe.rs b/bi_pipe.rs +--- a/bi_pipe.rs ++++ b/bi_pipe.rs +@@ -487,10 +487,10 @@ + // SockFlag is broken on macOS + // https://github.com/nix-rust/nix/issues/861 + let mut fds = [-1, -1]; +- #[cfg(not(target_os = "macos"))] ++ #[cfg(not(any(target_os = "macos", target_os = "ios")))] + let flags = libc::SOCK_CLOEXEC; + +- #[cfg(target_os = "macos")] ++ #[cfg(any(target_os = "macos", target_os = "ios"))] + let flags = 0; + + // SAFETY: libc call, fds are correct size+align +@@ -506,7 +506,7 @@ + return Err(std::io::Error::last_os_error()); + } + +- if cfg!(target_os = "macos") { ++ if cfg!(any(target_os = "macos", target_os = "ios")) { + let fcntl = |fd: i32, flag: libc::c_int| -> Result<(), std::io::Error> { + // SAFETY: libc call, fd is valid + let flags = unsafe { libc::fcntl(fd, libc::F_GETFD) }; diff --git a/recipes/vl-convert-python/patches/deno_node.patch b/recipes/vl-convert-python/patches/deno_node.patch new file mode 100644 index 00000000..e643826b --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_node.patch @@ -0,0 +1,101 @@ +deno_node 0.196.0: treat iOS as macOS. + +Every target_os = "macos" cfg also matches ios. Several of these branches have +no fallback, so iOS fails to compile without them; the rest select Darwin APIs +that iOS shares. Regenerate on a bump by running, from the crate root: + + perl -0pi -e 's/target_os = "macos"/any(target_os = "macos", target_os = "ios")/g' $(grep -rl 'target_os = "macos"' .) + +diff -ruN a/ops/constant.rs b/ops/constant.rs +--- a/ops/constant.rs ++++ b/ops/constant.rs +@@ -242,7 +242,7 @@ + } + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + #[op2] + #[serde] + pub fn op_node_fs_constants() -> FsConstants { +diff -ruN a/ops/fs.rs b/ops/fs.rs +--- a/ops/fs.rs ++++ b/ops/fs.rs +@@ -1737,7 +1737,7 @@ + cp_sync_copy_dir_no_filter(permissions, fs, src, dest, opts) + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn try_clone_dir_no_filter( + permissions: &PermissionsContainer, + fs: &FileSystemRc, +@@ -1786,7 +1786,7 @@ + } + } + +-#[cfg(not(target_os = "macos"))] ++#[cfg(not(any(target_os = "macos", target_os = "ios")))] + fn try_clone_dir_no_filter( + _permissions: &PermissionsContainer, + _fs: &FileSystemRc, +@@ -1797,7 +1797,7 @@ + Ok(false) + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn cp_tree_only_regular_files_and_dirs( + permissions: &PermissionsContainer, + fs: &FileSystemRc, +@@ -1864,7 +1864,7 @@ + Ok(true) + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn clonefile_sync( + src: &CheckedPath, + dest: &CheckedPath, +diff -ruN a/ops/os/cpus.rs b/ops/os/cpus.rs +--- a/ops/os/cpus.rs ++++ b/ops/os/cpus.rs +@@ -25,7 +25,7 @@ + } + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + pub fn cpu_info() -> Option> { + let mut model: [u8; 512] = [0; 512]; + let mut size = std::mem::size_of_val(&model); +diff -ruN a/ops/process.rs b/ops/process.rs +--- a/ops/process.rs ++++ b/ops/process.rs +@@ -125,7 +125,7 @@ + init + }; + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn init_macos_argv_info() { + ARGV_INFO.get_or_init(|| { + // SAFETY: _NSGetArgc/_NSGetArgv are stable macOS APIs that return +@@ -149,7 +149,7 @@ + }); + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn set_process_title(title: &str) { + init_macos_argv_info(); + overwrite_argv_buffer(title); +@@ -188,7 +188,7 @@ + } + + #[cfg(not(any( +- target_os = "macos", ++ any(target_os = "macos", target_os = "ios"), + target_os = "linux", + target_os = "windows" + )))] diff --git a/recipes/vl-convert-python/patches/deno_os.patch b/recipes/vl-convert-python/patches/deno_os.patch new file mode 100644 index 00000000..06151be0 --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_os.patch @@ -0,0 +1,17 @@ +deno_os 0.75.0: use the Darwin rss() on iOS. + +rss() is defined per OS with no fallback; iOS has the same mach task_info as +macOS. + +diff -ruN a/lib.rs b/lib.rs +--- a/lib.rs ++++ b/lib.rs +@@ -847,7 +847,7 @@ + total_rss_pages * page_size as u64 + } + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + fn rss() -> u64 { + // Inspired by https://github.com/Arc-blroth/memory-stats/blob/5364d0d09143de2a470d33161b2330914228fde9/src/darwin.rs + diff --git a/recipes/vl-convert-python/patches/deno_runtime.patch b/recipes/vl-convert-python/patches/deno_runtime.patch new file mode 100644 index 00000000..dbdfbb2b --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_runtime.patch @@ -0,0 +1,21 @@ +deno_runtime 0.266.0: embed 99_main.js. + +The runtime's ESM entry point is attached by hand as a build-machine path that +is read only while a snapshot is made (see deno_core.patch). This build boots +without one, so embed the file with ascii_str_include!. + +diff -ruN a/shared.rs b/shared.rs +--- a/shared.rs ++++ b/shared.rs +@@ -57,9 +57,9 @@ + // 99_main.js is the snapshot's ESM entry point; its source is loaded + // from disk only during snapshot creation, so we don't duplicate it in + // the final binary's `.rodata`. +- ext.esm_files.to_mut().push(ExtensionFileSource::loaded_during_snapshot( ++ ext.esm_files.to_mut().push(ExtensionFileSource::new( + "ext:runtime_main/js/99_main.js", +- concat!(env!("CARGO_MANIFEST_DIR"), "/js/99_main.js"), ++ deno_core::ascii_str_include!(concat!(env!("CARGO_MANIFEST_DIR"), "/js/99_main.js")), + )); + ext.esm_entry_point = Some("ext:runtime_main/js/99_main.js"); + } diff --git a/recipes/vl-convert-python/patches/deno_signals.patch b/recipes/vl-convert-python/patches/deno_signals.patch new file mode 100644 index 00000000..7df01655 --- /dev/null +++ b/recipes/vl-convert-python/patches/deno_signals.patch @@ -0,0 +1,17 @@ +deno_signals 0.49.0: give iOS the Darwin signal table. + +The signal-name table is per-OS with no fallback, so on iOS +signal_str_to_int() does not exist and deno_os fails to compile. + +diff -ruN a/dict.rs b/dict.rs +--- a/dict.rs ++++ b/dict.rs +@@ -145,7 +145,7 @@ + (31, "SIGSYS" | "SIGUNUSED") + ); + +-#[cfg(target_os = "macos")] ++#[cfg(any(target_os = "macos", target_os = "ios"))] + signal_dict!( + (1, "SIGHUP"), + (2, "SIGINT"), diff --git a/recipes/vl-convert-python/patches/mobile.patch b/recipes/vl-convert-python/patches/mobile.patch new file mode 100644 index 00000000..22ef2df4 --- /dev/null +++ b/recipes/vl-convert-python/patches/mobile.patch @@ -0,0 +1,490 @@ +mobile.patch: run vl-convert on QuickJS instead of V8. + +vl-convert executes Vega and Vega-Lite inside Deno. Deno's default engine is V8, +which has no Android build at all and only jitless arm64 iOS builds, and whose +startup snapshot (made at build time by a host V8) cannot be loaded by a V8 +built for another configuration. deno_core 0.411 can instead run on QuickJS-ng +through its deno_v8 facade (the `quickjs` feature, backed by the v8x crate): a +plain C interpreter that cross-compiles like any -sys crate and needs no JIT. + +1. Cargo.toml: deno_core with `quickjs` in place of its default `v8`. + pyo3 without `abi3-py310`: an abi3 cross build links the abi3 floor, + libpython3.10, which the iOS link cannot find. forge builds one wheel per + Python anyway. + +2. vl-convert-rs: boot without a snapshot. QuickJS cannot load a V8 snapshot, + so build.rs is swapped for forge_build.rs, which writes the empty snapshot + and residual-source tables lib.rs includes; the converter passes + `startup_snapshot: None`, and deno_runtime gets `transpile` so it can strip + the TypeScript in its extension sources at startup (deno_core.patch and + deno_runtime.patch embed those sources, which Deno otherwise only reads from + build-machine paths while snapshotting). Dropping build.rs also drops its + host-side deno_runtime build and V8 download. + +3. vl-convert-rs: aws-lc-sys 0.40 (pulled in by Deno's crypto crates) ships + pregenerated bindings for Android arm64 only; enable its `bindgen` feature + for every mobile target. + +4. .cargo/config.toml + .cargo/cross.cmake: v8x builds WAMR (QuickJS's + WebAssembly backend) with a bare `cmake` call that would configure for the + build host, and aws-lc-sys's bindgen mode (item 3) uses its CMake builder, + which skips its own cross setup whenever CMAKE_TOOLCHAIN_FILE is set. CMake + reads that variable from the environment; cargo's [env] sets it for build + scripts, and the toolchain file keys off the TARGET triple cargo gives them, + so host builds are unaffected. + +5. vl-convert-google-fonts: a MobileTls builder extension, applied to every + reqwest client in both crates. reqwest's rustls backend verifies through + rustls-platform-verifier, which panics on Android unless initialised from + Java with a JNI context; on Android the clients trust the bundled Mozilla + roots (webpki-root-certs) instead. Other targets are unchanged. + +6. worker_pool.rs: when a worker thread dies before reporting startup, put its + panic message in the error instead of "receiving on a closed channel". On + Android a Rust panic's own output goes to a discarded stderr. + +7. vl-convert-python/forge_tls.c (Android only): Rust has no native TLS on + Android, so std backs every thread_local! with its own pthread key, and + bionic allows a process about 128. A worker touches about 80, so the next + native module to load (rpds-py, under Altair) aborted with "out of TLS keys". + Hidden definitions of the four pthread key functions bind every reference in + this .so and multiplex its keys onto one real key. + +diff -ruN a/.cargo/config.toml b/.cargo/config.toml +--- a/.cargo/config.toml ++++ b/.cargo/config.toml +@@ -0,0 +1,2 @@ ++[env] ++CMAKE_TOOLCHAIN_FILE = { value = ".cargo/cross.cmake", relative = true } +diff -ruN a/.cargo/cross.cmake b/.cargo/cross.cmake +--- a/.cargo/cross.cmake ++++ b/.cargo/cross.cmake +@@ -0,0 +1,39 @@ ++# Cross toolchain for CMake projects driven by a Rust build script. v8x builds ++# WAMR with a bare `cmake` call, which would otherwise configure for the build ++# host, and aws-lc-sys's bindgen mode uses its CMake builder, which skips its own ++# cross setup when CMAKE_TOOLCHAIN_FILE is set. $ENV{TARGET} is the Rust target ++# triple cargo gives the build script; host-side builds match neither branch. ++set(_forge_target "$ENV{TARGET}") ++ ++if(_forge_target MATCHES "-linux-android") ++ if(_forge_target MATCHES "^aarch64-") ++ set(ANDROID_ABI arm64-v8a) ++ elseif(_forge_target MATCHES "^x86_64-") ++ set(ANDROID_ABI x86_64) ++ elseif(_forge_target MATCHES "^i686-") ++ set(ANDROID_ABI x86) ++ else() ++ set(ANDROID_ABI armeabi-v7a) ++ endif() ++ if(DEFINED ENV{ANDROID_API_LEVEL}) ++ set(ANDROID_PLATFORM "android-$ENV{ANDROID_API_LEVEL}") ++ else() ++ set(ANDROID_PLATFORM android-24) ++ endif() ++ set(WAMR_BUILD_PLATFORM android CACHE STRING "") ++ include("$ENV{NDK_ROOT}/build/cmake/android.toolchain.cmake") ++elseif(_forge_target MATCHES "-apple-ios") ++ set(CMAKE_SYSTEM_NAME iOS) ++ if(_forge_target MATCHES "-sim$" OR _forge_target MATCHES "^x86_64-") ++ set(CMAKE_OSX_SYSROOT iphonesimulator) ++ else() ++ set(CMAKE_OSX_SYSROOT iphoneos) ++ endif() ++ if(_forge_target MATCHES "^aarch64-") ++ set(CMAKE_OSX_ARCHITECTURES arm64) ++ else() ++ set(CMAKE_OSX_ARCHITECTURES x86_64) ++ endif() ++ set(CMAKE_OSX_DEPLOYMENT_TARGET 13.0) ++ set(WAMR_BUILD_PLATFORM darwin CACHE STRING "") ++endif() +diff -ruN a/Cargo.toml b/Cargo.toml +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -32,7 +32,7 @@ + # Deno dependencies - versions correspond to Deno v2.9.6 + deno_runtime = "0.266.0" + node_resolver = "0.96.0" +-deno_core = "0.411.0" ++deno_core = { version = "0.411.0", default-features = false, features = ["quickjs", "reactor-tokio"] } + deno_error = "0.7.1" + deno_graph = { version = "0.111.0", default-features = false, features = ["swc"] } + deno_ast = { version = "0.53.3", features = ["bundler", "codegen", "transforms", "transpiling"] } +@@ -56,7 +56,7 @@ + lz-str = "0.2.1" + png = "0.18.1" + predicates = "3.0.2" +-pyo3 = { version = "0.29.2", features = ["extension-module", "anyhow", "abi3-py310"] } ++pyo3 = { version = "0.29.2", features = ["extension-module", "anyhow"] } + pyo3-async-runtimes = { version = "0.29.0", features = ["tokio-runtime"] } + pyo3-log = "0.13.4" + pythonize = "0.29.0" +diff -ruN a/vl-convert-google-fonts/Cargo.toml b/vl-convert-google-fonts/Cargo.toml +--- a/vl-convert-google-fonts/Cargo.toml ++++ b/vl-convert-google-fonts/Cargo.toml +@@ -26,6 +26,9 @@ + tinyvec = { workspace = true, optional = true } + tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] } + ++[target.'cfg(target_os = "android")'.dependencies] ++webpki-root-certs = "1" ++ + [dev-dependencies] + fontdb = { workspace = true } + tempfile = { workspace = true } +diff -ruN a/vl-convert-google-fonts/src/client.rs b/vl-convert-google-fonts/src/client.rs +--- a/vl-convert-google-fonts/src/client.rs ++++ b/vl-convert-google-fonts/src/client.rs +@@ -89,7 +89,7 @@ + std::fs::create_dir_all(dir.join("fonts"))?; + } + +- let async_client = reqwest::Client::builder() ++ let async_client = crate::MobileTls::mobile_tls(reqwest::Client::builder()) + .user_agent(&config.user_agent) + .timeout(Duration::from_secs(config.request_timeout_secs)) + .build() +@@ -1001,7 +1001,7 @@ + let timeout_secs = self.config.request_timeout_secs; + + let built = std::thread::spawn(move || { +- reqwest::blocking::Client::builder() ++ crate::MobileTls::mobile_tls(reqwest::blocking::Client::builder()) + .user_agent(user_agent) + .timeout(Duration::from_secs(timeout_secs)) + .build() +diff -ruN a/vl-convert-google-fonts/src/lib.rs b/vl-convert-google-fonts/src/lib.rs +--- a/vl-convert-google-fonts/src/lib.rs ++++ b/vl-convert-google-fonts/src/lib.rs +@@ -7,11 +7,13 @@ + mod client; + mod config; + mod error; ++mod mobile_tls; + mod resolve; + mod types; + pub use client::GoogleFontsClient; + pub use config::{google_fonts_cache_dir, ClientConfig}; + pub use error::{GoogleFontsError, GoogleFontsFailure, GoogleFontsResult}; ++pub use mobile_tls::MobileTls; + + #[cfg(feature = "fontdb")] + mod fontdb_ext; +diff -ruN a/vl-convert-google-fonts/src/mobile_tls.rs b/vl-convert-google-fonts/src/mobile_tls.rs +--- a/vl-convert-google-fonts/src/mobile_tls.rs ++++ b/vl-convert-google-fonts/src/mobile_tls.rs +@@ -0,0 +1,34 @@ ++/// Applied to every reqwest client vl-convert builds. ++/// ++/// reqwest's rustls backend checks certificates through rustls-platform-verifier, ++/// which on Android panics unless it was first initialised from Java with a JNI ++/// context. A Python extension never is, so trust the bundled Mozilla roots there. ++pub trait MobileTls: Sized { ++ fn mobile_tls(self) -> Self; ++} ++ ++#[cfg(target_os = "android")] ++fn root_certs() -> Vec { ++ webpki_root_certs::TLS_SERVER_ROOT_CERTS ++ .iter() ++ .filter_map(|der| reqwest::Certificate::from_der(der).ok()) ++ .collect() ++} ++ ++impl MobileTls for reqwest::ClientBuilder { ++ fn mobile_tls(self) -> Self { ++ #[cfg(target_os = "android")] ++ return self.tls_certs_only(root_certs()); ++ #[cfg(not(target_os = "android"))] ++ self ++ } ++} ++ ++impl MobileTls for reqwest::blocking::ClientBuilder { ++ fn mobile_tls(self) -> Self { ++ #[cfg(target_os = "android")] ++ return self.tls_certs_only(root_certs()); ++ #[cfg(not(target_os = "android"))] ++ self ++ } ++} +diff -ruN a/vl-convert-python/Cargo.toml b/vl-convert-python/Cargo.toml +--- a/vl-convert-python/Cargo.toml ++++ b/vl-convert-python/Cargo.toml +@@ -7,6 +7,7 @@ + homepage = "https://github.com/vega/vl-convert" + repository.workspace = true + publish = false ++build = "forge_build.rs" + + [lib] + name = "vl_convert" +@@ -21,3 +22,6 @@ + tokio = { workspace = true } + pyo3-async-runtimes = { workspace = true } + pyo3-log = { workspace = true } ++ ++[build-dependencies] ++cc = "1" +diff -ruN a/vl-convert-python/forge_build.rs b/vl-convert-python/forge_build.rs +--- a/vl-convert-python/forge_build.rs ++++ b/vl-convert-python/forge_build.rs +@@ -0,0 +1,14 @@ ++fn main() { ++ println!("cargo:rerun-if-changed=forge_tls.c"); ++ if std::env::var("CARGO_CFG_TARGET_OS").as_deref() != Ok("android") { ++ return; ++ } ++ cc::Build::new() ++ .file("forge_tls.c") ++ .cargo_metadata(false) ++ .compile("forge_tls"); ++ println!("cargo:rustc-link-search=native={}", std::env::var("OUT_DIR").unwrap()); ++ // whole-archive: libc.so also defines these, and an archive member is only ++ // pulled for a symbol nothing else defines. ++ println!("cargo:rustc-link-lib=static:+whole-archive=forge_tls"); ++} +diff -ruN a/vl-convert-python/forge_tls.c b/vl-convert-python/forge_tls.c +--- a/vl-convert-python/forge_tls.c ++++ b/vl-convert-python/forge_tls.c +@@ -0,0 +1,110 @@ ++// Virtual pthread TLS keys for everything linked into this extension. ++// ++// Rust has no native TLS on Android, so std backs every thread_local! with a ++// pthread key of its own, and bionic allows a process about 128. A vl-convert ++// worker touches around 80, which leaves too few for the next native module to ++// load ("fatal runtime error: out of TLS keys"). These definitions are hidden, ++// so they bind every reference inside this .so and nothing outside it: all of ++// the extension's keys share one real key that holds a per-thread table. ++ ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#define HIDDEN __attribute__((visibility("hidden"))) ++#define MAX_KEYS 4096 ++#define DESTRUCTOR_PASSES 4 ++ ++typedef void (*dtor_fn)(void *); ++typedef struct { ++ size_t cap; ++ void *slots[]; ++} table; ++ ++static pthread_once_t once = PTHREAD_ONCE_INIT; ++static pthread_key_t real_key; ++static void *(*real_get)(pthread_key_t); ++static int (*real_set)(pthread_key_t, const void *); ++static atomic_uint next_key = 1; // 0 is never handed out ++static _Atomic(dtor_fn) dtors[MAX_KEYS]; ++ ++// Runs at thread exit with the thread's table, which bionic has already ++// detached. Reattach it so destructors that touch other thread-locals see ++// their values, and re-read it after each call because a destructor may grow it. ++static void run_destructors(void *value) { ++ real_set(real_key, value); ++ for (int pass = 0; pass < DESTRUCTOR_PASSES; pass++) { ++ int ran = 0; ++ for (size_t key = 1;; key++) { ++ table *t = real_get(real_key); ++ if (key >= t->cap) break; ++ // As bionic does: a value with no destructor stays readable by later ++ // destructors (Rust std's thread handle cleanup reads one) and is ++ // simply dropped with the table. ++ void *slot = t->slots[key]; ++ dtor_fn dtor = atomic_load(&dtors[key]); ++ if (!slot || !dtor) continue; ++ t->slots[key] = NULL; ++ dtor(slot); ++ ran = 1; ++ } ++ if (!ran) break; ++ } ++ table *t = real_get(real_key); ++ real_set(real_key, NULL); ++ free(t); ++} ++ ++static void init(void) { ++ int (*real_create)(pthread_key_t *, dtor_fn) = dlsym(RTLD_DEFAULT, "pthread_key_create"); ++ real_get = dlsym(RTLD_DEFAULT, "pthread_getspecific"); ++ real_set = dlsym(RTLD_DEFAULT, "pthread_setspecific"); ++ if (!real_create || !real_get || !real_set || real_create(&real_key, run_destructors)) abort(); ++} ++ ++HIDDEN int pthread_key_create(pthread_key_t *key, dtor_fn dtor) { ++ pthread_once(&once, init); ++ unsigned k = atomic_fetch_add(&next_key, 1); ++ if (k >= MAX_KEYS) return EAGAIN; ++ atomic_store(&dtors[k], dtor); ++ *key = (pthread_key_t)k; ++ return 0; ++} ++ ++// IDs are not reused, so MAX_KEYS bounds creates over the process's life. ++// Reuse would need per-slot generations to hide a deleted key's values, and ++// nothing here churns keys: Rust std deletes one only on a lost init race. ++HIDDEN int pthread_key_delete(pthread_key_t key) { ++ if (key <= 0 || (unsigned)key >= MAX_KEYS) return EINVAL; ++ atomic_store(&dtors[key], NULL); ++ return 0; ++} ++ ++HIDDEN void *pthread_getspecific(pthread_key_t key) { ++ pthread_once(&once, init); ++ table *t = real_get(real_key); ++ return (t && key > 0 && (size_t)key < t->cap) ? t->slots[key] : NULL; ++} ++ ++HIDDEN int pthread_setspecific(pthread_key_t key, const void *value) { ++ pthread_once(&once, init); ++ if (key <= 0 || (unsigned)key >= atomic_load(&next_key)) return EINVAL; ++ table *t = real_get(real_key); ++ if (!t || (size_t)key >= t->cap) { ++ if (!value) return 0; ++ size_t old = t ? t->cap : 0; ++ size_t cap = old ? old : 64; ++ while (cap <= (size_t)key) cap *= 2; ++ table *grown = realloc(t, sizeof(table) + cap * sizeof(void *)); ++ if (!grown) return ENOMEM; ++ memset(&grown->slots[old], 0, (cap - old) * sizeof(void *)); ++ grown->cap = cap; ++ real_set(real_key, grown); ++ t = grown; ++ } ++ t->slots[key] = (void *)value; ++ return 0; ++} +diff -ruN a/vl-convert-rs/Cargo.toml b/vl-convert-rs/Cargo.toml +--- a/vl-convert-rs/Cargo.toml ++++ b/vl-convert-rs/Cargo.toml +@@ -8,6 +8,7 @@ + repository.workspace = true + description = "Convert Vega-Lite, Vega, and SVG input to static and interactive formats" + keywords = ["Visualization", "Vega", "Vega-Lite"] ++build = "forge_build.rs" + + [package.metadata.docs.rs] + # Compile the host build script with this cfg so it can avoid linking V8. +@@ -22,7 +23,7 @@ + vl-convert-google-fonts = { workspace = true, features = ["fontdb"] } + + arc-swap = "1.9" +-deno_runtime = { workspace = true } ++deno_runtime = { workspace = true, features = ["transpile"] } + deno_core = { workspace = true } + deno_error = { workspace = true } + node_resolver = { workspace = true } +@@ -72,6 +73,9 @@ + [target.'cfg(windows)'.build-dependencies] + winapi = { version = "0.3", features = ["std"] } + ++[target.'cfg(any(target_os = "android", target_os = "ios"))'.dependencies] ++aws-lc-sys = { version = "=0.40.0", features = ["bindgen"] } ++ + [dev-dependencies] + rstest = { workspace = true } + dssim = { workspace = true } +@@ -81,11 +85,3 @@ + [[bench]] + name = "scenegraph_scatter" + harness = false +- +-[build-dependencies] +-deno_core = { workspace = true } +-deno_error = { workspace = true } +-# Need "snapshot" feature to access deno_runtime::snapshot::create_runtime_snapshot +-deno_runtime = { workspace = true, features = ["snapshot"] } +-# Canvas 2D Deno extension - used in snapshot creation +-vl-convert-canvas2d-deno = { workspace = true } +diff -ruN a/vl-convert-rs/forge_build.rs b/vl-convert-rs/forge_build.rs +--- a/vl-convert-rs/forge_build.rs ++++ b/vl-convert-rs/forge_build.rs +@@ -0,0 +1,10 @@ ++fn main() { ++ let out_dir = std::path::PathBuf::from(std::env::var_os("OUT_DIR").unwrap()); ++ std::fs::write(out_dir.join("VL_CONVERT_SNAPSHOT.bin"), []).unwrap(); ++ std::fs::write( ++ out_dir.join("VL_CONVERT_RESIDUAL_SOURCES.rs"), ++ "pub(crate) static VL_CONVERT_RESIDUAL_LAZY_JS: &[(&str, &str)] = &[];\n\ ++ pub(crate) static VL_CONVERT_RESIDUAL_LAZY_ESM: &[(&str, &str)] = &[];\n", ++ ) ++ .unwrap(); ++} +diff -ruN a/vl-convert-rs/src/converter/inner/mod.rs b/vl-convert-rs/src/converter/inner/mod.rs +--- a/vl-convert-rs/src/converter/inner/mod.rs ++++ b/vl-convert-rs/src/converter/inner/mod.rs +@@ -189,7 +189,7 @@ + // Runtime extension for worker-local JSON/msgpack transfer ops. + super::vl_convert_runtime::init(), + ], +- startup_snapshot: Some(crate::VL_CONVERT_SNAPSHOT), ++ startup_snapshot: None, + residual_lazy_js_sources: crate::VL_CONVERT_RESIDUAL_LAZY_JS, + residual_lazy_esm_sources: crate::VL_CONVERT_RESIDUAL_LAZY_ESM, + create_params, +diff -ruN a/vl-convert-rs/src/converter/worker_pool.rs b/vl-convert-rs/src/converter/worker_pool.rs +--- a/vl-convert-rs/src/converter/worker_pool.rs ++++ b/vl-convert-rs/src/converter/worker_pool.rs +@@ -292,10 +292,19 @@ + } + Err(err) => { + drop(senders); ++ let mut detail = err.to_string(); + for handle in handles { +- let _ = handle.join(); ++ if let Err(payload) = handle.join() { ++ if let Some(msg) = payload ++ .downcast_ref::() ++ .cloned() ++ .or_else(|| payload.downcast_ref::<&str>().map(|s| s.to_string())) ++ { ++ detail = format!("worker panicked: {msg}"); ++ } ++ } + } +- bail!("Failed to receive worker startup status: {err}"); ++ bail!("Failed to receive worker startup status: {detail}"); + } + } + } +diff -ruN a/vl-convert-rs/src/data_ops.rs b/vl-convert-rs/src/data_ops.rs +--- a/vl-convert-rs/src/data_ops.rs ++++ b/vl-convert-rs/src/data_ops.rs +@@ -13,7 +13,7 @@ + use std::time::{Duration, Instant}; + + lazy_static! { +- static ref ASYNC_CLIENT: reqwest::Client = reqwest::Client::builder() ++ static ref ASYNC_CLIENT: reqwest::Client = vl_convert_google_fonts::MobileTls::mobile_tls(reqwest::Client::builder()) + .user_agent(VL_CONVERT_USER_AGENT) + .connect_timeout(Duration::from_secs(10)) + .timeout(REQUEST_TIMEOUT) +diff -ruN a/vl-convert-rs/src/image_loading.rs b/vl-convert-rs/src/image_loading.rs +--- a/vl-convert-rs/src/image_loading.rs ++++ b/vl-convert-rs/src/image_loading.rs +@@ -18,7 +18,7 @@ + concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION")); + + lazy_static! { +- static ref BLOCKING_CLIENT: reqwest::blocking::Client = reqwest::blocking::ClientBuilder::new() ++ static ref BLOCKING_CLIENT: reqwest::blocking::Client = vl_convert_google_fonts::MobileTls::mobile_tls(reqwest::blocking::ClientBuilder::new()) + .user_agent(VL_CONVERT_USER_AGENT) + .connect_timeout(Duration::from_secs(10)) + .timeout(crate::data_ops::REQUEST_TIMEOUT) +diff -ruN a/vl-convert-rs/src/module_loader/mod.rs b/vl-convert-rs/src/module_loader/mod.rs +--- a/vl-convert-rs/src/module_loader/mod.rs ++++ b/vl-convert-rs/src/module_loader/mod.rs +@@ -244,7 +244,7 @@ + } + + return Box::pin(async move { +- let client = reqwest::Client::builder() ++ let client = vl_convert_google_fonts::MobileTls::mobile_tls(reqwest::Client::builder()) + .redirect(reqwest::redirect::Policy::none()) + .build() + .map_err(|e| { diff --git a/recipes/vl-convert-python/tests/test_vl_convert_python.py b/recipes/vl-convert-python/tests/test_vl_convert_python.py new file mode 100644 index 00000000..ef6b0590 --- /dev/null +++ b/recipes/vl-convert-python/tests/test_vl_convert_python.py @@ -0,0 +1,101 @@ +import asyncio +import ctypes +import os +import struct +import sys +import tempfile + +import pytest + +# v8x resolves its QuickJS bytecode cache once, on the first conversion. +CACHE_DIR = tempfile.mkdtemp(prefix="vlc-bytecode-") +os.environ["V82JSC_BC_CACHE_DIR"] = CACHE_DIR + +import vl_convert as vlc # noqa: E402 + +SPEC = { + "data": { + "values": [ + {"fruit": "apple", "count": 28}, + {"fruit": "banana", "count": 55}, + {"fruit": "cherry", "count": 43}, + ] + }, + "mark": "bar", + "encoding": { + "x": {"field": "fruit", "type": "nominal"}, + "y": {"field": "count", "type": "quantitative"}, + }, +} + + +def png_size(png): + """Width and height from a PNG's IHDR chunk.""" + assert png[:8] == b"\x89PNG\r\n\x1a\n" + return struct.unpack(">II", png[16:24]) + + +def test_vegalite_to_svg(): + """Vega-Lite compiles and Vega renders an SVG on the embedded QuickJS runtime.""" + svg = vlc.vegalite_to_svg(SPEC) + assert svg.lstrip().startswith(" 0 and height > 0 + assert png_size(vlc.vegalite_to_png(SPEC, scale=2)) == (2 * width, 2 * height) + + +def test_asyncio_variant(): + """The vl_convert.asyncio functions await from an event loop, as in an async Flet handler.""" + + async def convert(): + return await vlc.asyncio.vegalite_to_svg(SPEC) + + assert "banana" in asyncio.run(convert()) + + +def test_text_renders_without_system_fonts(): + """Text rasterises from the font bundled in the wheel; with no font it would be dropped.""" + frame = '{}' + blank = frame.format('') + text = frame.format( + '' + 'Flet' + ) + assert vlc.svg_to_png(text) != vlc.svg_to_png(blank) + + +def test_bytecode_cache_lands_in_configured_dir(): + """Compiled JavaScript is cached under V82JSC_BC_CACHE_DIR, so apps can point it at app storage.""" + vlc.vegalite_to_svg(SPEC) + assert any(name.endswith(".qbc") for name in os.listdir(CACHE_DIR)) + + +# sys.platform is "android" only from Python 3.13; getandroidapilevel exists on every version. +@pytest.mark.skipif(not hasattr(sys, "getandroidapilevel"), reason="bionic's key limit is Android-only") +def test_leaves_tls_keys_for_other_modules(): + """A running worker leaves pthread keys free; unshimmed, it left about 4 of bionic's 128.""" + vlc.vegalite_to_png(SPEC) + libc = ctypes.CDLL("libc.so") + keys, key = [], ctypes.c_uint() + while libc.pthread_key_create(ctypes.byref(key), None) == 0: + keys.append(key.value) + for k in keys: + libc.pthread_key_delete(k) + assert len(keys) >= 32, f"only {len(keys)} pthread keys left" + + +def test_altair_save_png(): + """Altair finds vl-convert on the device and `chart.save()` writes a PNG through it.""" + import altair as alt + + chart = alt.Chart(alt.Data(values=SPEC["data"]["values"])).mark_bar() + chart = chart.encode(x="fruit:N", y="count:Q") + path = os.path.join(CACHE_DIR, "altair.png") + chart.save(path) + with open(path, "rb") as f: + assert png_size(f.read())[0] > 0 diff --git a/src/forge/build.py b/src/forge/build.py index 808f0d69..0429439d 100644 --- a/src/forge/build.py +++ b/src/forge/build.py @@ -1,8 +1,10 @@ from __future__ import annotations +import hashlib import multiprocessing import os import re +import shlex import shutil import struct import sys @@ -51,6 +53,50 @@ those names (COPYING.curl, LICENSE.boringssl) are ours to choose. """ +PATCH_CRATES_IO_RE = re.compile( + r'^[ \t]*\[[ \t]*patch[ \t]*\.[ \t]*(crates-io|"crates-io")[ \t]*\][ \t]*(#.*)?$', + re.MULTILINE, +) + + +def add_crates_io_patches(text: str, entries: list[tuple[str, str, str]]) -> str: + """Add ``(key, crate, path)`` entries to a Cargo.toml's ``[patch.crates-io]``. + + Refuses a crate the manifest already patches, and checks the result parses, + so an unusual manifest fails here rather than as a cargo TOML error. + """ + existing = tomllib.loads(text).get("patch", {}).get("crates-io", {}) + for key, name, _ in entries: + for old_key, old in existing.items(): + old_name = old.get("package", old_key) if isinstance(old, dict) else old_key + if key == old_key or name == old_name: + raise RuntimeError( + f"crate_patches: Cargo.toml already patches {name} ({old_key!r}); " + "change that entry with a source patch instead" + ) + + # Quoted: a crate id used as a key has dots. + lines = "".join( + f'"{key}" = {{ package = "{name}", path = "{path}" }}\n' + for key, name, path in entries + ) + header = PATCH_CRATES_IO_RE.search(text) + if header: + text = f"{text[: header.end()]}\n{lines[:-1]}{text[header.end() :]}" + elif existing: + raise RuntimeError( + "crate_patches: Cargo.toml defines [patch.crates-io] without a " + "[patch.crates-io] header to add entries under" + ) + else: + text = f"{text.rstrip()}\n\n[patch.crates-io]\n{lines}" + + merged = tomllib.loads(text)["patch"]["crates-io"] + for key, name, path in entries: + if merged.get(key) != {"package": name, "path": path}: + raise RuntimeError(f"crate_patches: could not add {key!r} to Cargo.toml") + return text if text.endswith("\n") else text + "\n" + class Builder(ABC): # Whether this builder writes the wheel's METADATA itself (and so is the one that @@ -249,6 +295,109 @@ def patch_source(self): if not patched: log(self.log_file, "No patches to apply.") + def patch_crates(self): + """Patch crates.io dependencies of a Rust source tree. + + A fix that belongs in a dependency, not in the package itself, cannot ride + in ``patches``: cargo fetches dependencies at build time. For each crate + named in ``crate_patches``, take the exact version the source's Cargo.lock + resolves, verify the download against the lockfile checksum, unpack it into + ``forge-crates/``, apply the recipe's patches to it and point the + workspace's ``[patch.crates-io]`` at the copy. + """ + crate_patches = self.package.meta.get("crate_patches") or {} + if not crate_patches: + return + + lockfile = self.build_path / "Cargo.lock" + manifest = self.build_path / "Cargo.toml" + if not (lockfile.is_file() and manifest.is_file()): + raise RuntimeError( + "crate_patches needs a Cargo.toml and Cargo.lock at the root of the source" + ) + with lockfile.open("rb") as f: + locked = [ + pkg + for pkg in tomllib.load(f).get("package", []) + if pkg.get("source", "").startswith("registry+") + ] + + patched = {} + for key, patches in crate_patches.items(): + name, _, version = key.partition("@") + matches = [ + pkg + for pkg in locked + if pkg["name"] == name and (not version or pkg["version"] == version) + ] + if len(matches) != 1: + found = ", ".join( + pkg["version"] for pkg in locked if pkg["name"] == name + ) + raise RuntimeError( + f"crate_patches: {key!r} must match exactly one crates.io package in " + f"Cargo.lock (found: {found or 'none'}); use name@version to choose" + ) + crate = matches[0] + crate_id = f"{name}-{crate['version']}" + + archive = Path.cwd() / "downloads" / "crates" / f"{crate_id}.crate" + if not archive.is_file(): + url = f"https://static.crates.io/crates/{name}/{crate_id}.crate" + log(self.log_file, f"Downloading {url}...") + archive.parent.mkdir(parents=True, exist_ok=True) + partial = archive.with_suffix(".part") + with httpx.stream("GET", url, follow_redirects=True) as response: + response.raise_for_status() + with partial.open("wb") as f: + for chunk in response.iter_bytes(): + f.write(chunk) + partial.rename(archive) + + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + if digest != crate["checksum"]: + archive.unlink() + raise RuntimeError( + f"{archive.name}: sha256 {digest} does not match Cargo.lock " + f"({crate['checksum']})" + ) + + crates_dir = self.build_path / "forge-crates" + crates_dir.mkdir(exist_ok=True) + with tarfile.open(archive) as tf: + # filter= exists from 3.12 and in 3.8-3.11 security releases; the + # archive already matched the lockfile checksum. + if hasattr(tarfile, "data_filter"): + tf.extractall(path=crates_dir, filter="data") + else: + tf.extractall(path=crates_dir) + + for patch in patches: + patchfile = self.package.recipe_path / "patches" / patch + log(self.log_file, f"Applying {patch} to {crate_id}...") + subprocess.run( + self.log_file, + [ + "patch", + "-p1", + "--ignore-whitespace", + "--quiet", + "--input", + str(patchfile), + ], + cwd=crates_dir / crate_id, + ) + patched.setdefault(name, []).append(crate_id) + + entries = [] + for name, crate_ids in patched.items(): + for crate_id in crate_ids: + # Two patched versions of one crate need distinct keys; cargo + # matches on `package`, not on the key. + key = name if len(crate_ids) == 1 else crate_id + entries.append((key, name, f"forge-crates/{crate_id}")) + manifest.write_text(add_crates_io_patches(manifest.read_text(), entries)) + def prepare(self, clean=True): if clean and self.build_path.is_dir(): if clean: @@ -289,6 +438,7 @@ def prepare(self, clean=True): log(self.log_file, f"\n[{self.cross_venv}] Apply patches") self.patch_source() + self.patch_crates() # Create a clean cross environment. log(self.log_file, f"\n[{self.cross_venv}] Create clean build environment") @@ -436,6 +586,20 @@ def compile_env(self, **kwargs) -> dict[str, str]: (libgcc_shim / "libgcc.a").write_text("INPUT(-lunwind)\n") cargo_ldflags += f" -L{libgcc_shim}" + # rustc links cdylibs with -nodefaultlibs, so the NDK's compiler-rt + # builtins never reach the link, and Rust's compiler_builtins lacks some + # that C code in -sys crates calls. arm64 libffi's __clear_cache is one: + # the .so links, then dlopen fails with `cannot locate symbol`. Only + # members for still-undefined symbols are pulled from the archive. + rt_arch = self.cross_venv.platform_triplet.split("-")[0] + builtins = sorted( + (Path(cc).parent.parent / "lib" / "clang").glob( + f"*/lib/linux/libclang_rt.builtins-{rt_arch}-android.a" + ) + ) + if builtins: + cargo_ldflags += f" -C link-arg={builtins[-1]}" + if self.cross_venv.sdk != "android": # Replace any hard-coded reference to -isysroot with the actual reference ldflags = re.sub( @@ -590,6 +754,27 @@ def compile_env(self, **kwargs) -> dict[str, str]: env["ANDROID_API_LEVEL"] = str(self.cross_venv.sdk_version) env["HOST_TRIPLET"] = self.cross_venv.platform_triplet + # bindgen, which -sys crates run from their build scripts, parses headers with + # libclang and gets neither the target's sysroot nor, for aarch64-apple-ios-sim, + # a triple clang accepts. Scoped to the target, so host build scripts are untouched. + if self.cross_venv.sdk == "android": + cc_name = Path(cc).name + clang_target = ( + cc_name[: -len("-clang")] + if cc_name.endswith("-clang") + else f"{self.cross_venv.platform_triplet}{self.cross_venv.sdk_version}" + ) + # bindgen shell-splits the variable, so a path with a space needs quoting. + sysroot = f"--sysroot={shlex.quote(env['NDK_SYSROOT'])}" + else: + arch = self.cross_venv.platform_triplet.split("-")[0] + clang_target = f"{arch}-apple-ios{self.cross_venv.sdk_version}" + if self.cross_venv.sdk.endswith("simulator"): + clang_target += "-simulator" + sysroot = f"-isysroot {shlex.quote(str(self.cross_venv.sdk_root))}" + bindgen_var = f"BINDGEN_EXTRA_CLANG_ARGS_{cargo_build_target.replace('-', '_')}" + env[bindgen_var] = f"--target={clang_target} {sysroot}" + # `env` wins. Everything forge has adjusted above — the compiler and # binutils re-pointed at the installed NDK, and the CFLAGS/CPPFLAGS/LDFLAGS # it extended with SDK, sysroot and `opt/lib` search paths — also exists as diff --git a/src/forge/schema/meta-schema.yaml b/src/forge/schema/meta-schema.yaml index 272c6944..938a5d72 100644 --- a/src/forge/schema/meta-schema.yaml +++ b/src/forge/schema/meta-schema.yaml @@ -103,6 +103,23 @@ properties: Patches to apply to the code. Each entry is a filename in the `patches` folder of the recipe. + crate_patches: + type: object + default: {} + additionalProperties: + type: array + items: + type: string + description: >- + Rust recipes only. Patches for crates.io dependencies, keyed by crate name + (or name@version when Cargo.lock holds several versions of it), each value + a list of filenames in the recipe's `patches` folder. forge downloads the + version the source's Cargo.lock resolves, checks it against the lockfile + checksum, unpacks it into `forge-crates/`, applies the patches with + `patch -p1` from the crate root, and adds a `[patch.crates-io]` entry for + it to the root Cargo.toml. Needs Cargo.toml and Cargo.lock at the source + root. + extract_packages: type: array default: []