Skip to content

Commit 6863850

Browse files
committed
feat(MOD-01): compile verified local Python modules
1 parent b948382 commit 6863850

9 files changed

Lines changed: 591 additions & 16 deletions

File tree

‎IMPLEMENTATION_PLAN.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -439,3 +439,14 @@ The development checkout now provides the [ES module emitter](docs/compiler-emit
439439
for the approved single-module subset, with selected helpers, source maps and a
440440
versioned artifact manifest. MOD-01 and browser profile integration follow their
441441
explicit dependencies.
442+
443+
## Development implementation checkpoint: 2026-10-20
444+
445+
PARSE-01, PARSE-02 and EMIT-01 are merged. The development checkout implements
446+
[verified local-module compilation](docs/compiler-modules.md) for MOD-01, with
447+
relative imports, aliases, package re-exports, initialization order and a shared
448+
value/callable runtime. Explicit source-import cycles and unsupported source are
449+
rejected. Per-module maps, source hashes, static output closure and unchanged
450+
notices are retained. Published 0.1.0 artifacts remain unchanged. GitHub Issues
451+
and Project fields record live delivery status; the forward table records the
452+
original adopted dependency order. ASYNC-01 and BROWSER-01 follow this foundation.

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,8 @@ the [issues](https://github.com/outside-labs/Python-JavaScript/issues) and
8989
[Project](https://github.com/orgs/outside-labs/projects/11).
9090

9191
The development checkout now includes a [binding-aware compiler front end](docs/compiler-front-end.md)
92-
that validates source and produces typed IR. JavaScript emission and browser
93-
execution remain subsequent milestones.
92+
that validates source and produces typed IR. The emitter and [local-module compiler](docs/compiler-modules.md)
93+
produce executable ES modules; browser adapters remain a subsequent milestone.
9494

9595
## Development
9696

‎docs/compiler-language.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ these APIs and provides Python-side JavaScript compatibility objects.
99

1010
Start with a single-module equivalent of Shimify's text_metrics application:
1111
Unicode character count, explicitly selected ASCII word separators, LF-based
12-
line count and a tuple of Python integer results. MOD-01 later adds its local
12+
line count and a tuple of Python integer results. The [local-module profile](compiler-modules.md) adds its
1313
helper import and package re-export. The selected input domain is builtin str;
1414
non-string input raises TypeError("text must be str"). String subclass overrides
1515
and arbitrary user objects are outside the first profile.
@@ -23,7 +23,7 @@ and arbitrary user objects are outside the first profile.
2323
| Operations | Supported scalar arithmetic/comparisons, Python and/or/not, identity on supported values, string membership and supported indexing. Mixed or unknown operand forms need helpers or a source-located rejection. |
2424
| Builtins/errors | len on adopted strings/tuples, print of adopted values, approved isinstance checks and explicit builtin TypeError raising needed by the fixture. No eval/exec, reflection or undeclared I/O. |
2525
| JavaScript API | Resolve explicit imports/aliases through a versioned symbol/method registry. Start with Array.of and length as a binding fixture; other exports/methods need adopted signatures and parity cases before lowering. |
26-
| Imports | Only approved JavaScript registry imports in the single-module profile. Local modules follow MOD-01; Python stdlib/third-party/native imports require explicit future adapters. |
26+
| Imports | Only approved JavaScript registry imports in the single-module profile. Named local imports use the separate modules profile; Python stdlib/third-party/native imports require explicit future adapters. |
2727

2828
Every AST form and operation is supported, rejected or deferred explicitly. A
2929
Python-side JavaScript API existing today does not automatically make it

‎docs/compiler-modules.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Compiling local source modules
2+
3+
The development checkout provides `JavaScript.modules.compile_modules`. The
4+
published 0.1.0 package predates this API. `python-browser-modules/1` extends the
5+
[single-module language](compiler-language.md) with explicit named source imports.
6+
7+
```python
8+
from pathlib import Path
9+
from JavaScript.modules import SourceModule, compile_modules
10+
11+
sources = (
12+
SourceModule.from_text("app", "from helpers import size\nprint(size('A😀'))"),
13+
SourceModule.from_text("helpers", "def size(text):\n return len(text)"),
14+
)
15+
artifact = compile_modules(sources, entrypoint="app")
16+
destination = Path("compiled")
17+
for output in artifact.files:
18+
path = destination / output.path
19+
path.parent.mkdir(parents=True, exist_ok=True)
20+
path.write_bytes(output.data)
21+
(destination / "manifest.json").write_bytes(artifact.manifest_bytes)
22+
```
23+
24+
Load `createModule` from the generated `app.mjs` and call it to initialize a new
25+
application context. Its `write` option receives Python print output, including
26+
newlines. The entrypoint is imported under its declared identity; it is not
27+
executed a second time as `__main__`.
28+
29+
## Inputs and imports
30+
31+
Supply an immutable tuple of `SourceModule` records. Each record contains its
32+
qualified module name, exact immutable UTF-8 bytes, SHA-256, package flag and
33+
logical relative `.py` filename. Direct construction validates the supplied
34+
hash; `from_text` computes it. Compilation rechecks records. Logical filenames
35+
are source-map identities, not paths the compiler opens. No target module is
36+
imported or executed during compilation.
37+
38+
The allowlist contains at most 32 modules, 1 MiB per module and 8 MiB total.
39+
Module identities use ordinary ASCII Python names and cannot shadow the stdlib
40+
or the JavaScript registry. Module names and portable logical filenames must be
41+
unique, including when case folded. Every parent package needs an explicitly
42+
declared `is_package=True` initializer. Only reachable modules and their parent
43+
initializers are parsed and emitted; supplied unused modules are omitted.
44+
45+
Supported source imports are unconditional top-level named imports, including
46+
aliases, relative imports and package re-exports:
47+
48+
```python
49+
from text_metrics import metrics as selected
50+
from .helpers import character_count as count_characters
51+
```
52+
53+
Exports can be supported plain functions, immutable module constants or
54+
re-exported source bindings. JavaScript registry imports retain their separate
55+
binding rules and cannot be re-exported through the Python source namespace.
56+
Plain local `import module`, wildcard, conditional/function-local, dynamic,
57+
stdlib/native-extension and undeclared imports fail with source-located
58+
diagnostics. Explicit source-import cycles are deterministically rejected.
59+
Required parent initializer ordering supports a package re-exporting a child;
60+
it does not authorize general partially initialized module cycles.
61+
62+
Import statements execute in source order, each module initializes once per
63+
application context, and parents initialize before children. Reading a binding
64+
before its defining statement still raises the adopted Python error. One shared
65+
runtime preserves imported callable, tuple and string identities, immutable
66+
function defaults, recursion limits and the output sink. Entry exports wrap
67+
callables once per context for JavaScript invocation; internal imports use the
68+
original Python values. Each `createModule` call creates independent state.
69+
70+
## Artifacts and limits
71+
72+
The manifest has kind `python-javascript-compiled`, schema 1, compiler ABI
73+
`python-browser-modules-esm/1` and language profile `python-browser-modules/1`.
74+
It records selected modules/exports, Python source import edges, per-module
75+
source hashes and filenames, the static emitted import closure, required helper
76+
closure, file sizes/hashes and notices. Deterministically named module factories
77+
share `runtime.mjs`; `app.mjs` statically imports every emitted dependency.
78+
Source maps retain each original module's text. `source_map=False` omits maps.
79+
80+
Original nonempty notices may be passed as immutable `OutputFile` records with
81+
role `notice`, under portable `LICENSES/` paths. They are copied unchanged and
82+
hashed. The compiler's provenance notice grants no additional license rights.
83+
No runtime dependency on Shimify is introduced. Shimify may produce the verified
84+
source bytes used here, but unchanged supported modules also compile directly.
85+
86+
The text_metrics parity fixture uses its supported metrics module and the
87+
reduced helper containing `character_count`. The original helper's unused slice
88+
utility remains unsupported and causes rejection if supplied unchanged. This
89+
compiler does not perform symbol extraction or silently skip unsupported code
90+
inside a selected module. Browser DOM/Fetch integration and runtime parsing of
91+
new source remain separate tasks.

‎src/JavaScript/_compiler_runtime.py‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,12 +230,27 @@ class $JSArray { constructor(values) { this.values = values; } }
230230
$fail('TypeError', 'Use BigInt, string, bool, null or explicit str/tuple input; JavaScript Number/Array is not Python int/tuple.', location);
231231
}
232232
function $expose(value, location) {
233-
return value instanceof $Callable ? (...args) => $call(value, args.map(argument => $input(argument, location)), location) : value;
233+
if (!(value instanceof $Callable)) return value;
234+
if (!$exposed.has(value)) $exposed.set(value, (...args) => $call(value, args.map(argument => $input(argument, location)), location));
235+
return $exposed.get(value);
234236
}
237+
const $exposed = new WeakMap();
235238
'''),
236239
}
237240

238241

242+
HELPER_BINDINGS = {
243+
'error': ('$PyError', '$fail'), 'binding': ('$UNBOUND', '$get'),
244+
'str': ('$str',), 'tuple': ('$tuple',), 'numeric': ('$number',),
245+
'truth': ('$truth',), 'binary': ('$binary',), 'unary': ('$unary',),
246+
'equal': ('$equal', '$order'), 'compare': ('$compare',), 'index': ('$index',),
247+
'iterate': ('$iterate',), 'call': ('$fn', '$call'), 'len': ('$len',),
248+
'types': ('$types', '$isinstance'), 'exception': ('$TypeError',),
249+
'repr': ('$repr',), 'print': ('$print',),
250+
'javascript': ('$JS', '$JArray', '$attribute'), 'input': ('$input', '$expose'),
251+
}
252+
253+
239254
@lru_cache(maxsize=1)
240255
def _nonprintable_ranges():
241256
ranges, start = [], None

‎src/JavaScript/compiler.py‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ class Binding:
4242
id: str
4343
name: str
4444
scope: str
45-
kind: Literal['constant', 'function', 'parameter', 'local', 'javascript', 'builtin']
45+
kind: Literal['constant', 'function', 'parameter', 'local', 'javascript', 'builtin', 'source']
4646
target: str | None = None
4747

4848

@@ -84,6 +84,14 @@ def to_dict(self):
8484
'body': [node.to_dict() for node in self.body]}
8585

8686

87+
@dataclass(frozen=True)
88+
class SourceImport:
89+
name: str
90+
module: str
91+
export: str
92+
value_type: ValueType
93+
94+
8795
_BUILTINS = {'len', 'print', 'isinstance', 'str', 'int', 'bool', 'tuple', 'TypeError'}
8896
_TYPES = {'str', 'int', 'bool', 'tuple'}
8997
_BINARY = {ast.Add: '+', ast.Sub: '-', ast.Mult: '*', ast.FloorDiv: '//', ast.Mod: '%'}
@@ -108,13 +116,14 @@ def accepted(value):
108116

109117

110118
class _FrontEnd:
111-
def __init__(self, source, filename):
119+
def __init__(self, source, filename, source_imports=None):
112120
self.filename = filename
113121
self.lines = source.splitlines() or ['']
114122
self.bindings = {}
115123
self.root = {}
116124
self.locals = {}
117125
self.types = {}
126+
self.source_imports = source_imports or {}
118127
try:
119128
self.tree = ast.parse(source, filename=filename)
120129
self.table = symtable.symtable(source, filename, 'exec')
@@ -146,6 +155,15 @@ def declare(self, name, scope, kind, target=None):
146155

147156
def _declare_module(self):
148157
for node in self.tree.body:
158+
imports = self.source_imports.get((node.lineno, node.col_offset))
159+
if imports is not None:
160+
for imported in imports:
161+
if imported.name in self.root:
162+
self.fail('PJS100', 'Module rebinding is outside the immutable profile.', node)
163+
binding = self.declare(imported.name, 'module', 'source', imported.module + ':' + imported.export)
164+
self.root[imported.name] = binding
165+
self.types[binding.id] = imported.value_type
166+
continue
149167
names = []
150168
kind, target = 'constant', None
151169
if isinstance(node, ast.FunctionDef):
@@ -208,6 +226,8 @@ def expression(self, node, scope):
208226
kind = 'javascript-type'
209227
elif binding.kind == 'builtin':
210228
kind = 'python-type' if binding.name in _TYPES else 'callable'
229+
elif binding.kind == 'source':
230+
target = binding.id if kind == 'callable' else None
211231
return self.node(node, 'name', kind, binding=binding.id, target=target)
212232
if isinstance(node, ast.Tuple):
213233
return self.node(node, 'tuple', 'python-tuple', values=tuple(self.expression(value, scope) for value in node.elts))
@@ -234,7 +254,7 @@ def expression(self, node, scope):
234254
kind, count = allowed[target]
235255
if count is not None and len(node.args) != count:
236256
self.fail('PJS120', 'Unsupported argument count.', node)
237-
elif target in self.bindings and self.bindings[target].kind == 'function':
257+
elif target in self.bindings and self.bindings[target].kind in {'function', 'source'}:
238258
kind = 'python-value'
239259
else:
240260
self.fail('PJS120', 'Call does not resolve to an approved direct callable.', node)
@@ -299,6 +319,10 @@ def statement(self, node, scope, loop):
299319
if isinstance(node, (ast.Import, ast.ImportFrom)) and scope is None:
300320
if node not in self.tree.body:
301321
self.fail('PJS100', 'Imports require unconditional module bindings.', node)
322+
imports = self.source_imports.get((node.lineno, node.col_offset))
323+
if imports is not None:
324+
return self.node(node, 'source-import', module=imports[0].module,
325+
bindings=tuple((self.root[imported.name].id, imported.export) for imported in imports))
302326
names = [alias.asname or alias.name for alias in node.names]
303327
return self.node(node, 'import', bindings=tuple(self.root[name].id for name in names))
304328
if isinstance(node, ast.FunctionDef):

‎src/JavaScript/emitter.py‎

Lines changed: 19 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
import unicodedata
99

1010
from .compiler import parse_source
11-
from ._compiler_runtime import helper_closure, helper_source
11+
from ._compiler_runtime import HELPER_BINDINGS, helper_closure, helper_source
1212

1313

1414
def _json(value):
@@ -135,7 +135,12 @@ def statements(self, nodes, indent=1):
135135

136136
def statement(self, node, indent):
137137
fields, kind = dict(node.fields), node.kind
138-
if kind == 'import':
138+
if kind == 'source-import':
139+
temporary = self.temporary()
140+
self.add(f'const {temporary}=importModule({_json(fields["module"])});', indent, node)
141+
for identifier, exported in fields['bindings']:
142+
self.add(f'{self.names[identifier]}={temporary}[{_json(exported)}];', indent, node)
143+
elif kind == 'import':
139144
self.use('javascript')
140145
for identifier in fields['bindings']:
141146
value = '$JS' if self.bindings[identifier].target == 'JavaScript' else '$JArray'
@@ -185,7 +190,7 @@ def statement(self, node, indent):
185190
else:
186191
raise AssertionError('Unrecognized validated statement: ' + kind)
187192

188-
def render(self, source, module_name, source_map):
193+
def render(self, source, module_name, source_map, *, shared_runtime=False):
189194
roots = [binding for binding in self.module.bindings if binding.scope == 'module']
190195
if roots:
191196
self.use('binding')
@@ -195,17 +200,23 @@ def render(self, source, module_name, source_map):
195200
items = []
196201
for binding in exports:
197202
value = self.names[binding.id]
198-
if binding.kind == 'function':
203+
if binding.kind == 'function' and not shared_runtime:
199204
self.use('input')
200205
definition = next(node for node in self.module.body if node.kind == 'function' and node.field('binding') == binding.id)
201206
value = f'$expose({value},{self.location(definition)})'
202207
items.append(f'[{_json(binding.name)}]:{value}')
203208
self.add('return Object.freeze({' + ','.join(items) + '});')
204209
helpers = helper_closure(self.required)
205-
lines = [('export function createModule({write = text => console.log(text.slice(0,-1))} = {}) {', None),
206-
(" if(typeof write !== 'function')throw new TypeError('write must be a function');", None)]
207-
for helper in helpers:
208-
lines.extend((' ' + line, None) for line in helper_source(helper).splitlines())
210+
if shared_runtime:
211+
names = [name for helper in helpers for name in HELPER_BINDINGS[helper]]
212+
lines = [('export function initialize(runtime, importModule) {', None)]
213+
if names:
214+
lines.append((' const {' + ','.join(names) + '}=runtime;', None))
215+
else:
216+
lines = [('export function createModule({write = text => console.log(text.slice(0,-1))} = {}) {', None),
217+
(" if(typeof write !== 'function')throw new TypeError('write must be a function');", None)]
218+
for helper in helpers:
219+
lines.extend((' ' + line, None) for line in helper_source(helper).splitlines())
209220
lines.extend(self.lines)
210221
lines.append(('}', None))
211222
mappings, previous_line, previous_column = [], 0, 0

0 commit comments

Comments
 (0)