Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -439,3 +439,14 @@ The development checkout now provides the [ES module emitter](docs/compiler-emit
for the approved single-module subset, with selected helpers, source maps and a
versioned artifact manifest. MOD-01 and browser profile integration follow their
explicit dependencies.

## Development implementation checkpoint: 2026-10-20

PARSE-01, PARSE-02 and EMIT-01 are merged. The development checkout implements
[verified local-module compilation](docs/compiler-modules.md) for MOD-01, with
relative imports, aliases, package re-exports, initialization order and a shared
value/callable runtime. Explicit source-import cycles and unsupported source are
rejected. Per-module maps, source hashes, static output closure and unchanged
notices are retained. Published 0.1.0 artifacts remain unchanged. GitHub Issues
and Project fields record live delivery status; the forward table records the
original adopted dependency order. ASYNC-01 and BROWSER-01 follow this foundation.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ the [issues](https://github.com/outside-labs/Python-JavaScript/issues) and
[Project](https://github.com/orgs/outside-labs/projects/11).

The development checkout now includes a [binding-aware compiler front end](docs/compiler-front-end.md)
that validates source and produces typed IR. JavaScript emission and browser
execution remain subsequent milestones.
that validates source and produces typed IR. The emitter and [local-module compiler](docs/compiler-modules.md)
produce executable ES modules; browser adapters remain a subsequent milestone.

## Development

Expand Down
4 changes: 2 additions & 2 deletions docs/compiler-language.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ these APIs and provides Python-side JavaScript compatibility objects.

Start with a single-module equivalent of Shimify's text_metrics application:
Unicode character count, explicitly selected ASCII word separators, LF-based
line count and a tuple of Python integer results. MOD-01 later adds its local
line count and a tuple of Python integer results. The [local-module profile](compiler-modules.md) adds its
helper import and package re-export. The selected input domain is builtin str;
non-string input raises TypeError("text must be str"). String subclass overrides
and arbitrary user objects are outside the first profile.
Expand All @@ -23,7 +23,7 @@ and arbitrary user objects are outside the first profile.
| Operations | Supported scalar arithmetic/comparisons, Python and/or/not, identity on supported values, string membership and supported indexing. Mixed or unknown operand forms need helpers or a source-located rejection. |
| Builtins/errors | len on adopted strings/tuples, print of adopted values, approved isinstance checks and explicit builtin TypeError raising needed by the fixture. No eval/exec, reflection or undeclared I/O. |
| JavaScript API | Resolve explicit imports/aliases through a versioned symbol/method registry. Start with Array.of and length as a binding fixture; other exports/methods need adopted signatures and parity cases before lowering. |
| Imports | Only approved JavaScript registry imports in the single-module profile. Local modules follow MOD-01; Python stdlib/third-party/native imports require explicit future adapters. |
| Imports | Only approved JavaScript registry imports in the single-module profile. Named local imports use the separate modules profile; Python stdlib/third-party/native imports require explicit future adapters. |

Every AST form and operation is supported, rejected or deferred explicitly. A
Python-side JavaScript API existing today does not automatically make it
Expand Down
91 changes: 91 additions & 0 deletions docs/compiler-modules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Compiling local source modules

The development checkout provides `JavaScript.modules.compile_modules`. The
published 0.1.0 package predates this API. `python-browser-modules/1` extends the
[single-module language](compiler-language.md) with explicit named source imports.

```python
from pathlib import Path
from JavaScript.modules import SourceModule, compile_modules

sources = (
SourceModule.from_text("app", "from helpers import size\nprint(size('A😀'))"),
SourceModule.from_text("helpers", "def size(text):\n return len(text)"),
)
artifact = compile_modules(sources, entrypoint="app")
destination = Path("compiled")
for output in artifact.files:
path = destination / output.path
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(output.data)
(destination / "manifest.json").write_bytes(artifact.manifest_bytes)
```

Load `createModule` from the generated `app.mjs` and call it to initialize a new
application context. Its `write` option receives Python print output, including
newlines. The entrypoint is imported under its declared identity; it is not
executed a second time as `__main__`.

## Inputs and imports

Supply an immutable tuple of `SourceModule` records. Each record contains its
qualified module name, exact immutable UTF-8 bytes, SHA-256, package flag and
logical relative `.py` filename. Direct construction validates the supplied
hash; `from_text` computes it. Compilation rechecks records. Logical filenames
are source-map identities, not paths the compiler opens. No target module is
imported or executed during compilation.

The allowlist contains at most 32 modules, 1 MiB per module and 8 MiB total.
Module identities use ordinary ASCII Python names and cannot shadow the stdlib
or the JavaScript registry. Module names and portable logical filenames must be
unique, including when case folded. Every parent package needs an explicitly
declared `is_package=True` initializer. Only reachable modules and their parent
initializers are parsed and emitted; supplied unused modules are omitted.

Supported source imports are unconditional top-level named imports, including
aliases, relative imports and package re-exports:

```python
from text_metrics import metrics as selected
from .helpers import character_count as count_characters
```

Exports can be supported plain functions, immutable module constants or
re-exported source bindings. JavaScript registry imports retain their separate
binding rules and cannot be re-exported through the Python source namespace.
Plain local `import module`, wildcard, conditional/function-local, dynamic,
stdlib/native-extension and undeclared imports fail with source-located
diagnostics. Explicit source-import cycles are deterministically rejected.
Required parent initializer ordering supports a package re-exporting a child;
it does not authorize general partially initialized module cycles.

Import statements execute in source order, each module initializes once per
application context, and parents initialize before children. Reading a binding
before its defining statement still raises the adopted Python error. One shared
runtime preserves imported callable, tuple and string identities, immutable
function defaults, recursion limits and the output sink. Entry exports wrap
callables once per context for JavaScript invocation; internal imports use the
original Python values. Each `createModule` call creates independent state.

## Artifacts and limits

The manifest has kind `python-javascript-compiled`, schema 1, compiler ABI
`python-browser-modules-esm/1` and language profile `python-browser-modules/1`.
It records selected modules/exports, Python source import edges, per-module
source hashes and filenames, the static emitted import closure, required helper
closure, file sizes/hashes and notices. Deterministically named module factories
share `runtime.mjs`; `app.mjs` statically imports every emitted dependency.
Source maps retain each original module's text. `source_map=False` omits maps.

Original nonempty notices may be passed as immutable `OutputFile` records with
role `notice`, under portable `LICENSES/` paths. They are copied unchanged and
hashed. The compiler's provenance notice grants no additional license rights.
No runtime dependency on Shimify is introduced. Shimify may produce the verified
source bytes used here, but unchanged supported modules also compile directly.

The text_metrics parity fixture uses its supported metrics module and the
reduced helper containing `character_count`. The original helper's unused slice
utility remains unsupported and causes rejection if supplied unchanged. This
compiler does not perform symbol extraction or silently skip unsupported code
inside a selected module. Browser DOM/Fetch integration and runtime parsing of
new source remain separate tasks.
17 changes: 16 additions & 1 deletion src/JavaScript/_compiler_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -230,12 +230,27 @@ class $JSArray { constructor(values) { this.values = values; } }
$fail('TypeError', 'Use BigInt, string, bool, null or explicit str/tuple input; JavaScript Number/Array is not Python int/tuple.', location);
}
function $expose(value, location) {
return value instanceof $Callable ? (...args) => $call(value, args.map(argument => $input(argument, location)), location) : value;
if (!(value instanceof $Callable)) return value;
if (!$exposed.has(value)) $exposed.set(value, (...args) => $call(value, args.map(argument => $input(argument, location)), location));
return $exposed.get(value);
}
const $exposed = new WeakMap();
'''),
}


HELPER_BINDINGS = {
'error': ('$PyError', '$fail'), 'binding': ('$UNBOUND', '$get'),
'str': ('$str',), 'tuple': ('$tuple',), 'numeric': ('$number',),
'truth': ('$truth',), 'binary': ('$binary',), 'unary': ('$unary',),
'equal': ('$equal', '$order'), 'compare': ('$compare',), 'index': ('$index',),
'iterate': ('$iterate',), 'call': ('$fn', '$call'), 'len': ('$len',),
'types': ('$types', '$isinstance'), 'exception': ('$TypeError',),
'repr': ('$repr',), 'print': ('$print',),
'javascript': ('$JS', '$JArray', '$attribute'), 'input': ('$input', '$expose'),
}


@lru_cache(maxsize=1)
def _nonprintable_ranges():
ranges, start = [], None
Expand Down
30 changes: 27 additions & 3 deletions src/JavaScript/compiler.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ class Binding:
id: str
name: str
scope: str
kind: Literal['constant', 'function', 'parameter', 'local', 'javascript', 'builtin']
kind: Literal['constant', 'function', 'parameter', 'local', 'javascript', 'builtin', 'source']
target: str | None = None


Expand Down Expand Up @@ -84,6 +84,14 @@ def to_dict(self):
'body': [node.to_dict() for node in self.body]}


@dataclass(frozen=True)
class SourceImport:
name: str
module: str
export: str
value_type: ValueType


_BUILTINS = {'len', 'print', 'isinstance', 'str', 'int', 'bool', 'tuple', 'TypeError'}
_TYPES = {'str', 'int', 'bool', 'tuple'}
_BINARY = {ast.Add: '+', ast.Sub: '-', ast.Mult: '*', ast.FloorDiv: '//', ast.Mod: '%'}
Expand All @@ -108,13 +116,14 @@ def accepted(value):


class _FrontEnd:
def __init__(self, source, filename):
def __init__(self, source, filename, source_imports=None):
self.filename = filename
self.lines = source.splitlines() or ['']
self.bindings = {}
self.root = {}
self.locals = {}
self.types = {}
self.source_imports = source_imports or {}
try:
self.tree = ast.parse(source, filename=filename)
self.table = symtable.symtable(source, filename, 'exec')
Expand Down Expand Up @@ -146,6 +155,15 @@ def declare(self, name, scope, kind, target=None):

def _declare_module(self):
for node in self.tree.body:
imports = self.source_imports.get((node.lineno, node.col_offset))
if imports is not None:
for imported in imports:
if imported.name in self.root:
self.fail('PJS100', 'Module rebinding is outside the immutable profile.', node)
binding = self.declare(imported.name, 'module', 'source', imported.module + ':' + imported.export)
self.root[imported.name] = binding
self.types[binding.id] = imported.value_type
continue
names = []
kind, target = 'constant', None
if isinstance(node, ast.FunctionDef):
Expand Down Expand Up @@ -208,6 +226,8 @@ def expression(self, node, scope):
kind = 'javascript-type'
elif binding.kind == 'builtin':
kind = 'python-type' if binding.name in _TYPES else 'callable'
elif binding.kind == 'source':
target = binding.id if kind == 'callable' else None
return self.node(node, 'name', kind, binding=binding.id, target=target)
if isinstance(node, ast.Tuple):
return self.node(node, 'tuple', 'python-tuple', values=tuple(self.expression(value, scope) for value in node.elts))
Expand All @@ -234,7 +254,7 @@ def expression(self, node, scope):
kind, count = allowed[target]
if count is not None and len(node.args) != count:
self.fail('PJS120', 'Unsupported argument count.', node)
elif target in self.bindings and self.bindings[target].kind == 'function':
elif target in self.bindings and self.bindings[target].kind in {'function', 'source'}:
kind = 'python-value'
else:
self.fail('PJS120', 'Call does not resolve to an approved direct callable.', node)
Expand Down Expand Up @@ -299,6 +319,10 @@ def statement(self, node, scope, loop):
if isinstance(node, (ast.Import, ast.ImportFrom)) and scope is None:
if node not in self.tree.body:
self.fail('PJS100', 'Imports require unconditional module bindings.', node)
imports = self.source_imports.get((node.lineno, node.col_offset))
if imports is not None:
return self.node(node, 'source-import', module=imports[0].module,
bindings=tuple((self.root[imported.name].id, imported.export) for imported in imports))
names = [alias.asname or alias.name for alias in node.names]
return self.node(node, 'import', bindings=tuple(self.root[name].id for name in names))
if isinstance(node, ast.FunctionDef):
Expand Down
27 changes: 19 additions & 8 deletions src/JavaScript/emitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import unicodedata

from .compiler import parse_source
from ._compiler_runtime import helper_closure, helper_source
from ._compiler_runtime import HELPER_BINDINGS, helper_closure, helper_source


def _json(value):
Expand Down Expand Up @@ -135,7 +135,12 @@ def statements(self, nodes, indent=1):

def statement(self, node, indent):
fields, kind = dict(node.fields), node.kind
if kind == 'import':
if kind == 'source-import':
temporary = self.temporary()
self.add(f'const {temporary}=importModule({_json(fields["module"])});', indent, node)
for identifier, exported in fields['bindings']:
self.add(f'{self.names[identifier]}={temporary}[{_json(exported)}];', indent, node)
elif kind == 'import':
self.use('javascript')
for identifier in fields['bindings']:
value = '$JS' if self.bindings[identifier].target == 'JavaScript' else '$JArray'
Expand Down Expand Up @@ -185,7 +190,7 @@ def statement(self, node, indent):
else:
raise AssertionError('Unrecognized validated statement: ' + kind)

def render(self, source, module_name, source_map):
def render(self, source, module_name, source_map, *, shared_runtime=False):
roots = [binding for binding in self.module.bindings if binding.scope == 'module']
if roots:
self.use('binding')
Expand All @@ -195,17 +200,23 @@ def render(self, source, module_name, source_map):
items = []
for binding in exports:
value = self.names[binding.id]
if binding.kind == 'function':
if binding.kind == 'function' and not shared_runtime:
self.use('input')
definition = next(node for node in self.module.body if node.kind == 'function' and node.field('binding') == binding.id)
value = f'$expose({value},{self.location(definition)})'
items.append(f'[{_json(binding.name)}]:{value}')
self.add('return Object.freeze({' + ','.join(items) + '});')
helpers = helper_closure(self.required)
lines = [('export function createModule({write = text => console.log(text.slice(0,-1))} = {}) {', None),
(" if(typeof write !== 'function')throw new TypeError('write must be a function');", None)]
for helper in helpers:
lines.extend((' ' + line, None) for line in helper_source(helper).splitlines())
if shared_runtime:
names = [name for helper in helpers for name in HELPER_BINDINGS[helper]]
lines = [('export function initialize(runtime, importModule) {', None)]
if names:
lines.append((' const {' + ','.join(names) + '}=runtime;', None))
else:
lines = [('export function createModule({write = text => console.log(text.slice(0,-1))} = {}) {', None),
(" if(typeof write !== 'function')throw new TypeError('write must be a function');", None)]
for helper in helpers:
lines.extend((' ' + line, None) for line in helper_source(helper).splitlines())
lines.extend(self.lines)
lines.append(('}', None))
mappings, previous_line, previous_column = [], 0, 0
Expand Down
Loading
Loading