test(QA-03): verify shared compiler targets against CPython (#36)

* fix(ERR-01): preserve raise and await source origins

* test(QA-03): verify shared compiler targets against CPython

Author: ellainng

.github/workflows/ci.yaml

@@ -23,7 +23,7 @@ jobs:
           persist-credentials: false
       - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
         with:
-          python-version: '3.14'
+          python-version: '3.14.7'
       - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
         with:
           node-version: '24.19.0'
@@ -60,10 +60,13 @@ jobs:
       - run: uv python install
       - run: uv sync --locked --group browser-tests
       - run: uv run --no-sync python -m playwright install --with-deps chromium
+      - run: uv run --no-sync python scripts/compiler_conformance.py --browser --report build/compiler-parity-report.json
       - run: uv run --no-sync python scripts/compiler_browser_smoke.py --report build/compiler-browser-report.json
       - uses: actions/upload-artifact@v5
         if: always()
         with:
           name: compiler-browser-report
-          path: build/compiler-browser-report.json
+          path: |
+            build/compiler-browser-report.json
+            build/compiler-parity-report.json
           if-no-files-found: error

IMPLEMENTATION_PLAN.md

@@ -357,7 +357,7 @@ meet their first milestones.
 | [js/MOD-01](https://github.com/outside-labs/Python-JavaScript/issues/23) | Compile bounded local Python module imports | [js/EMIT-01](https://github.com/outside-labs/Python-JavaScript/issues/13) | Blocked |
 | [js/ASYNC-01](https://github.com/outside-labs/Python-JavaScript/issues/10) | Define Promise scheduling and browser async semantics | [js/PARSE-01](https://github.com/outside-labs/Python-JavaScript/issues/11) | Blocked |
 | [js/BROWSER-01](https://github.com/outside-labs/Python-JavaScript/issues/24) | Implement browser DOM, events and Fetch adapters | [js/MOD-01](https://github.com/outside-labs/Python-JavaScript/issues/23), [js/ASYNC-01](https://github.com/outside-labs/Python-JavaScript/issues/10), [pocket/HOST-01](https://github.com/outside-labs/pocket-python/issues/17) | Blocked |
-| [js/QA-03](https://github.com/outside-labs/Python-JavaScript/issues/25) | Verify CPython, Node and browser execution parity | [js/BROWSER-01](https://github.com/outside-labs/Python-JavaScript/issues/24) | Blocked |
+| [js/QA-03](https://github.com/outside-labs/Python-JavaScript/issues/25) | Verify CPython, Node and browser execution parity | [js/BROWSER-01](https://github.com/outside-labs/Python-JavaScript/issues/24), [js/ERR-01](https://github.com/outside-labs/Python-JavaScript/issues/34) | See issue / Project |
 | [js/PERF-01](https://github.com/outside-labs/Python-JavaScript/issues/26) | Measure compiled application and helper footprints | [js/QA-03](https://github.com/outside-labs/Python-JavaScript/issues/25) | Blocked |
 
 PARSE-01 is the first Ready compiler task. PARSE-02/EMIT-01 follow in order;
@@ -469,3 +469,15 @@ preconstructed exception origins before QA-03: explicit raises and first source
 awaits replace constructor spans, while propagated runtime/callback errors retain
 their origin. CPython/native/browser cases cover the correction and mapped
 tracebacks. QA-03 now depends on this diagnostic fix as well as BROWSER-01.
+
+
+QA-03 implements compiler-parity/1: one approved corpus executes under isolated
+CPython and as identical emitted artifacts in Node and a real browser worker.
+Reports retain typed values, output order, exception origins, source-map frames,
+native JavaScript observations, explicit rejected/deferred diagnostics and exact
+corpus/artifact identities. The browser host suite separately verifies callback
+release, cancellation and worker stop/restart. Borrowed text-metrics cases retain
+Shimify's exact MIT notice, including the runnable browser example. The versioned
+[compatibility matrix](docs/compiler-conformance.md) records the adopted scope
+and remaining differences; full Python tracebacks and arbitrary dynamic Python
+remain outside this compiler profile. No release was published.

README.md

@@ -117,3 +117,5 @@ Build the local [text_metrics browser example](docs/compiler-browser.md#run-the-
 with `uv run python examples/browser_text_metrics.py --output build/browser-text-metrics`.
 It uses our emitted modules and helpers in a worker; source parsing remains the
 separate independent-interpreter milestone.
+
+The development compiler has a versioned [CPython/Node/browser parity suite](docs/compiler-conformance.md) with typed results, source-map error checks and explicit unsupported boundaries.

docs/compiler-conformance.md

@@ -0,0 +1,64 @@
+# Compiler compatibility matrix: compiler-parity/1
+
+The development compiler runs the same approved source corpus under CPython
+3.14.7 and as identical emitted ES modules in Node 24.19.0 and a Chromium module
+worker. This is a bounded ahead-of-time Python profile. The independent source
+interpreter is a separate pocket-python implementation. Published 0.1.0 predates
+this compiler.
+
+| Area | Adopted behavior and evidence | Boundary |
+| --- | --- | --- |
+| Values | Typed None, bool, arbitrary-size int, code-point str and immutable tuple returns | Mutable containers, floats and arbitrary host values are deferred |
+| Evaluation | Output order, short-circuiting, default tuple identity, imported function aliases and source initialization order match CPython | Integer/string interning identity is not adopted; indirect local callable invocation is rejected |
+| Unicode/numbers | Integers beyond 2^53, signed floor/modulo, combining marks, supplementary characters and explicit surrogate pairs | Native JS strings count UTF-16 units; Python strings count code points |
+| Exceptions | Error class, adopted message, original filename/line/code-point column and output before failure match the reference | Argument-count wording and recursion resource limits are profile-specific |
+| Source maps | Actual Node and browser error stacks map back to original explicit raise, division, indexing and module callback source lines | These are mapped JS frames, not complete Python traceback objects; source-map columns identify emitted statements |
+| Async | Lazy/single-use coroutines, Promise callback side effects, rejected values and previously raised error origins | Microtask scheduling can differ from CPython; async generators/iteration and Promise.all are not adopted |
+| Browser host | Real-worker shared computation, DOM callbacks/release, Fetch grants/limits, cancellation and stop/restart are checked separately by compiler_browser_smoke.py | Host errors carry original spans; workers provide responsiveness isolation, not a general security boundary |
+| Imports | Explicit source allowlist and shared module/runtime identity; module callback origins preserved | Dynamic imports, undeclared sources and arbitrary installed-package execution are rejected |
+| Diagnostics | Every rejected/deferred corpus entry must fail with its declared compiler diagnostic | A failed supported case fails the suite; no Python execution fallback in either target |
+
+The schema-1 corpus consists of compiler_contract.json plus compiler_parity.json.
+The initial matrix has 24 supported cases and 10 explicit boundary cases. Its
+report also records three native observations in both JavaScript targets:
+Array.of length, Promise callback ordering and UTF-16 string length. These are
+native API observations, not CPython-language assertions. Compiler Promise cases
+compare the adopted Python wrapper with emitted modules separately.
+
+The reference runs only trusted test fixtures in fresh isolated Python processes.
+The browser fixture server serves static assets; the browser worker performs the
+computation. Both targets import the same artifact bytes. Tagged comparison
+preserves integer precision, Unicode surrogate distinctions and tuple structure.
+Output and original error spans are compared independently. Source-map checks
+feed actual target stack frames to Node's SourceMap consumer. An unraised
+exception rejected through a Promise reports its first source await; its native
+JS stack can retain the construction site. Only explicitly raised errors in the
+traceback subset require the first mapped frame to identify the error origin.
+
+Run the complete suite from a development checkout with the declared browser
+test group and its Chromium installation:
+
+```sh
+uv sync --locked --group browser-tests
+uv run --no-sync python -m playwright install chromium
+uv run --no-sync python scripts/compiler_conformance.py --browser --report build/compiler-parity-report.json
+uv run --no-sync python scripts/compiler_browser_smoke.py --report build/compiler-browser-report.json
+```
+
+Omit --browser for the Node/reference gate; --channel chrome selects an installed
+Chrome for local verification. CI pins the reference and Node versions, uses the
+locked Playwright package and retains both reports with the actual browser
+version. The ordinary installed-wheel/sdist suite runs the Node/reference gate
+on Linux, macOS and Windows. Reports contain every compared result, original and
+mapped error locations, profile classifications, corpus SHA-256 and artifact
+manifest identities. A corpus change updates its digest; incompatible report
+schema/comparison semantics require a new suite version. A report describes its
+tested versions and selected cases, not universal CPython compatibility.
+
+The shared reduced text-metrics computation originates in Shimify's fixture
+package. The attribution entry in compiler_parity.json pins the source revision;
+tests/fixtures/licenses/SHIMIFY-LICENSE.txt retains its exact MIT notice. Emitted
+shared-computation artifacts, the host smoke fixture and the runnable browser
+example include that notice in LICENSES/shimify.txt with manifest hashes. Other
+cases are original regression examples; no upstream Python test suite is copied.
+This attribution does not choose a license for the compiler itself.

docs/compiler-emitter.md

@@ -86,8 +86,9 @@ Run `python -m unittest discover -s tests -p 'test_compiler*.py' -v` with the
 installed development checkout and Node 24.19.0. Tests compare CPython and emitted
 Node output/errors/evaluation order, the single-module text_metrics application,
 precision/Unicode/tuple/default behavior and an independent Node source-map
-consumer. Local modules, async browser capabilities and the final browser parity
-gate remain MOD-01, ASYNC-01/BROWSER-01 and QA-03. This compiler executes only
+consumer. Local modules and async browser capabilities have separate adopted
+profiles. The shared [compiler-parity/1 suite](compiler-conformance.md) compares
+emitted Node and browser-worker execution with CPython. This compiler executes only
 ahead-of-time approved source; it does not parse new source in the browser.
 
 Exception construction does not start a traceback. An explicit raise records

examples/browser_text_metrics.py

@@ -6,6 +6,7 @@
 from pathlib import Path
 
 from JavaScript.browser import compile_browser
+from JavaScript.emitter import OutputFile
 from JavaScript.modules import SourceModule
 
 
@@ -23,7 +24,8 @@ def build(output):
     corpus = json.loads((ROOT / 'tests/fixtures/compiler_contract.json').read_text(encoding='utf-8'))
     reference = next(case['source'] for case in corpus['cases'] if case['id'] == 'text-metrics-single-module')
     computation = reference[:reference.index('\nprint(metrics(')]
-    artifact = compile_browser((SourceModule.from_text('app', APP), SourceModule.from_text('text_metrics', computation)), entrypoint='app')
+    artifact = compile_browser((SourceModule.from_text('app', APP), SourceModule.from_text('text_metrics', computation)), entrypoint='app',
+        notices=(OutputFile('LICENSES/shimify.txt', (ROOT / 'tests/fixtures/licenses/SHIMIFY-LICENSE.txt').read_bytes(), 'notice'),))
     identity = {'manifest_sha256': hashlib.sha256(artifact.manifest_bytes).hexdigest(), 'profile': 'compiled-javascript',
                 'profile_version': 1, 'entrypoint': 'app.mjs', 'engine': None}
     html = '''<!doctype html><html lang="en"><meta charset="utf-8"><meta name="viewport" content="width=device-width">
@@ -59,8 +61,12 @@ def build(output):
     if output.exists() or output.is_symlink():
         raise ValueError('Choose a new output directory.')
     output.mkdir(parents=True)
-    written = []
+    written, directories = [], []
     try:
+        parents = {parent for name in files for parent in (output / name).parents if parent != output and output in parent.parents}
+        for directory in sorted(parents, key=lambda path: len(path.parts)):
+            directory.mkdir()
+            directories.append(directory)
         for name, data in files.items():
             path = output / name
             with path.open('xb') as stream:
@@ -69,6 +75,11 @@ def build(output):
     except BaseException:
         for path in reversed(written):
             path.unlink(missing_ok=True)
+        for directory in reversed(directories):
+            try:
+                directory.rmdir()
+            except OSError:
+                pass
         try:
             output.rmdir()
         except OSError:

pyproject.toml

@@ -35,10 +35,14 @@ source-include = [
     "tests/fixtures/conformance.mjs",
     "tests/fixtures/browser_protocol.mjs",
     "tests/fixtures/compiler_contract.json",
+    "tests/fixtures/compiler_parity.json",
+    "tests/fixtures/licenses/SHIMIFY-LICENSE.txt",
     "docs/compatibility.md",
     "docs/compiler*.md",
     "scripts/install_distribution.py",
     "scripts/compiler_browser_smoke.py",
+    "scripts/compiler_conformance.py",
+    "scripts/compiler_conformance.mjs",
     "examples/browser_text_metrics.py",
     "CHANGELOG.md",
     "RELEASING.md",

scripts/compiler_browser_smoke.py

@@ -15,6 +15,7 @@
 from playwright.sync_api import sync_playwright
 
 from JavaScript.browser import compile_browser
+from JavaScript.emitter import OutputFile
 from JavaScript.modules import SourceModule
 
 
@@ -100,7 +101,8 @@ def sources():
 
 
 def smoke(channel=None):
-    artifact = compile_browser(sources(), entrypoint='app')
+    artifact = compile_browser(sources(), entrypoint='app', notices=(OutputFile(
+        'LICENSES/shimify.txt', (ROOT / 'tests/fixtures/licenses/SHIMIFY-LICENSE.txt').read_bytes(), 'notice'),))
     with tempfile.TemporaryDirectory() as temporary:
         root = Path(temporary)
         for item in artifact.files:

scripts/compiler_conformance.mjs

@@ -0,0 +1,32 @@
+// The same emitted artifacts and result encoding run in Node and a browser worker.
+function typed(value) {
+  if (value === null) return {type: 'none'};
+  if (typeof value === 'boolean') return {type: 'bool', value};
+  if (typeof value === 'bigint') return {type: 'int', value: value.toString()};
+  if (value?.type === 'str' && Array.isArray(value.codepoints))
+    return {type: 'str', codepoints: [...value.codepoints]};
+  if (value?.type === 'tuple' && Array.isArray(value.items))
+    return {type: 'tuple', items: value.items.map(typed)};
+  throw new TypeError('Unadopted target result');
+}
+
+export async function runCases(cases) {
+  const results = [];
+  for (const item of cases) {
+    const {createModule} = await import(new URL(item.url, import.meta.url));
+    let stdout = '', value = null, error = null, stack = '';
+    try {
+      const app = createModule({write: text => {stdout += text;}});
+      if (item.export) value = await app[item.export]();
+    } catch (caught) {
+      error = {name: caught.name, message: caught.message, location: caught.location};
+      stack = String(caught.stack);
+    }
+    results.push({id: item.id, stdout, value: typed(value), error, stack});
+  }
+  const order = [];
+  const pending = Promise.resolve(1).then(() => {order.push('callback');});
+  order.push('sync'); await pending; order.push('await');
+  return {cases: results, native: {'array-length': Array.of(1, 2, 3).length,
+    'promise-order': order, 'utf16-length': '😀'.length}};
+}