diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md index 4e4c719..64fd095 100644 --- a/IMPLEMENTATION_PLAN.md +++ b/IMPLEMENTATION_PLAN.md @@ -450,3 +450,9 @@ rejected. Per-module maps, source hashes, static output closure and unchanged notices are retained. Published 0.1.0 artifacts remain unchanged. GitHub Issues and Project fields record live delivery status; the forward table records the original adopted dependency order. ASYNC-01 and BROWSER-01 follow this foundation. + +The explicit [async profile](docs/compiler-async.md) implements ASYNC-01 +coroutines, Promise conversion/chaining, synchronous callbacks, source-located +rejections and AbortSignal context cancellation. JavaScript microtask timing and +CPython scheduling differences are documented. Fetch/DOM/worker transport remains +BROWSER-01; no new async runtime dependency or published artifact is introduced. diff --git a/README.md b/README.md index c048f20..e70a32a 100644 --- a/README.md +++ b/README.md @@ -108,3 +108,7 @@ network requests. The development [compiler emitter](docs/compiler-emitter.md) produces deterministic ES modules for the approved single-module Python profile. This API is not part of the published 0.1.0 distribution. + +The explicit [async compiler profile](docs/compiler-async.md) adds lazy coroutines, +awaited Promise values, callbacks and context cancellation. Browser host adapters +remain separately tracked. diff --git a/docs/compiler-async.md b/docs/compiler-async.md new file mode 100644 index 0000000..4471b7f --- /dev/null +++ b/docs/compiler-async.md @@ -0,0 +1,101 @@ +# Compiled async profile + +The development APIs `parse_source`, `compile_source` and `compile_modules` +accept `async_enabled=True`. Without it, async syntax remains rejected. Published +0.1.0 artifacts predate these compiler APIs. Single-module artifacts identify +`python-browser-async/1` and ABI `python-browser-async-esm/1`; local-module artifacts +identify `python-browser-modules-async/1` and ABI +`python-browser-modules-async-esm/1`. These are explicit extensions of the +[source language](compiler-language.md) and [module contract](compiler-modules.md). + +```python +from JavaScript import Promise + +async def main(): + value = await Promise.resolve(9007199254740993) + print(value + 1) + return value +``` + +Compile this source with the async flag, import its generated `createModule`, +then use `await createModule().main()` in JavaScript. An exported async function +returns a native Promise, including for invalid arguments and cancellation. +Python integers, strings and tuples keep their adopted runtime representation; +no implicit Number conversion is added. Synchronous exports returning an adopted +Promise also expose a native Promise. Other synchronous exports remain synchronous. + +## Coroutine and Promise behavior + +Top-level plain `async def` functions support the existing positional parameters, +immutable defaults, ordinary control flow and return values. Calling one inside +compiled Python creates a lazy, single-use coroutine. Its body begins when +awaited. Arguments/defaults are bound when called. Awaiting the same coroutine +twice raises RuntimeError; a Promise can be awaited repeatedly. Await in defaults, +module initialization or a synchronous function is rejected. Async generators, +async iteration/context managers, asyncio task APIs, gather and top-level await +are deferred. Unawaited coroutines have no execution side effect; no finalizer +warning is promised. + +The approved async JavaScript registry adds `Promise.resolve(value)`, +`Promise.reject(TypeError(message))` and `promise.then(on_fulfilled, on_rejected)`. +Registry imports/aliases resolve by binding identity. A handler is a supported +plain synchronous Python function or None; omitted/None handlers forward outcomes. +Each handler receives one argument and is called once. A returned adopted Promise +is chained; a raised adopted exception rejects the chain without retrying the +handler. Async callbacks and arbitrary thenables are rejected. Promise constructors, +catch/finally, all/race and general native Promise inputs remain deferred. +Coroutines must be awaited before passing them as arguments; async functions +cannot return an unawaited coroutine or Promise as their final value. + +Source locations survive async rejection, imported calls and callbacks. Explicit +builtin TypeError and existing runtime errors retain their Python names and +messages. Unexpected host rejections become a bounded RuntimeError rather than +exposing host internals. Synchronous compiled calls retain depth 128; at most 128 +async coroutine bodies may be active in one context. This bound also limits +concurrent active bodies and is not CPython's recursion limit. + +## Scheduling and cancellation + +An exported async invocation starts on a JavaScript microtask. Internal coroutine +creation remains lazy; native async lowering and Promise continuations preserve +source evaluation order, short circuiting, and callback single invocation. +Awaited boolean/chained-comparison expressions may add a continuation even on a +short-circuited path. Awaiting an already fulfilled Promise also crosses a +microtask checkpoint. CPython's companion Promise uses thread-backed callbacks +and its asyncio bridge may resume an already completed awaitable without the +same checkpoint. Timing relative to unrelated tasks or callbacks is therefore +not a cross-target guarantee. No implicit event loop or Python scheduler is +embedded in the generated artifact. + +`createModule({write, signal})` accepts an optional AbortSignal for that context. +Generated statements, public calls, callbacks and await boundaries check it. +Cancellation rejects pending exported operations with name AbortError and code +CANCELLED at a recorded source location. Await listeners are removed on success, +failure and cancellation. Completed output/other side effects remain committed. +The signal applies to the whole application context; create a fresh context to +restart. Signal reasons are not copied into application errors. + +AbortSignal delivery cannot interrupt busy synchronous execution on its own. +In a worker, the trusted main thread must retain worker termination as the stop +path. The [host protocol](https://github.com/outside-labs/pocket-python/blob/main/docs/browser-host-protocol.md) +uses session/request IDs, serialized tagged values, exactly one terminal result +and CANCELLED/TIMEOUT replies. A future transport adapter must wrap a pending +host result as an adopted Promise, forward cancellation to its request, discard +stale replies and release handles. Raw Promise/Event/DOM objects do not cross +that transport. HOST-01 specifies that adapter; this task does not implement it. + +## Fetch boundary + +CPython code continues to use the published `fetch_api.fetch` implementation +re-exported by `JavaScript`, including its thread-backed Promise and HTTP types. +Those implementations are not copied into compiled artifacts. `fetch`, Request, +Response, streaming bodies and response JSON methods remain rejected by this +compiler profile. BROWSER-01 adds the separately granted `fetch/1` text operation, +lowering to browser-native Fetch through the trusted host with AbortSignal, +origin/CORS, URL, credential, redirect, size and deadline rules. Its first text +metrics UI requires no network access. Streaming, async iterables and standalone +Promise package extraction remain separate extensions. + +Local fixtures compare awaited values, laziness, defaults, order and callback +errors with CPython and execute the generated modules in Node. Abort and listener +cleanup tests use a local AbortController; no external network is needed. diff --git a/docs/compiler-language.md b/docs/compiler-language.md index cbe5759..950aed5 100644 --- a/docs/compiler-language.md +++ b/docs/compiler-language.md @@ -93,8 +93,9 @@ provides initial positive and rejected/deferred cases; parsing known sources for the contract does not prove a compiler already exists. PARSE-02 implements the front end and EMIT-01 must execute emitted output against that corpus. -MOD-01 adds explicit local-module identity and initialization; ASYNC-01 adds -supported await/Promise/cancellation behavior. BROWSER-01 integrates main-thread +The local-module profile implements explicit module identity and initialization; +the [async profile](compiler-async.md) adds supported await/Promise/cancellation +behavior when explicitly enabled. BROWSER-01 integrates main-thread DOM/events and native Fetch through the versioned host protocol; QA-03 verifies CPython/Node/browser parity and source-map errors. Origin/CORS and worker message constraints remain explicit. diff --git a/src/JavaScript/_compiler_runtime.py b/src/JavaScript/_compiler_runtime.py index 9a500c2..f6d593a 100644 --- a/src/JavaScript/_compiler_runtime.py +++ b/src/JavaScript/_compiler_runtime.py @@ -216,6 +216,105 @@ class $JSArray { constructor(values) { this.values = values; } } if (value instanceof $JSArray && name === 'length') return value.values.length; $fail('TypeError', 'Attribute is outside the approved JavaScript registry.', location); } +'''), + 'async': (('call', 'javascript'), r''' +if (signal !== null && (typeof signal !== 'object' || typeof signal.aborted !== 'boolean' || + typeof signal.addEventListener !== 'function' || typeof signal.removeEventListener !== 'function')) { + throw new TypeError('signal must be an AbortSignal or null'); +} +function $cancelled(location) { + const error = new $PyError('AbortError', 'Compiled operation cancelled.', location); + error.code = 'CANCELLED'; return error; +} +function $checkpoint(location) { if (signal?.aborted) throw $cancelled(location); } +class $Coroutine { + constructor(start) { this.start = start; this.started = false; } +} +class $PromiseValue { + constructor(promise) { this.promise = promise; promise.catch(() => {}); Object.freeze(this); } +} +const $asyncFunctions = new WeakSet(); +let $activeAsync = 0; +function $cfn(name, count, defaults, body) { + const fn = $fn(name, count, defaults, (args, location) => new $Coroutine(() => body(args, location))); + $asyncFunctions.add(fn); return fn; +} +function $wait(promise, location) { + $checkpoint(location); + return new Promise((resolve, reject) => { + let finished = false; + const cleanup = () => signal?.removeEventListener('abort', abort); + const abort = () => { if (!finished) { finished = true; cleanup(); reject($cancelled(location)); } }; + signal?.addEventListener('abort', abort, {once: true}); + if (signal?.aborted) { abort(); return; } + promise.then(value => { + if (finished) return; finished = true; cleanup(); + try { $checkpoint(location); resolve(value); } catch (error) { reject(error); } + }, error => { + if (finished) return; finished = true; cleanup(); + reject(error instanceof $PyError ? error : new $PyError('RuntimeError', 'Unhandled Promise rejection.', location)); + }); + }); +} +function $await(value, location) { + $checkpoint(location); + if (value instanceof $PromiseValue) return $wait(value.promise, location); + if (!(value instanceof $Coroutine)) $fail('TypeError', 'await requires an adopted coroutine or Promise.', location); + if (value.started) $fail('RuntimeError', 'cannot reuse already awaited coroutine', location); + value.started = true; + if ($activeAsync >= 128) $fail('RecursionError', 'Maximum active async calls (128) exceeded.', location); + $activeAsync++; + let promise; + try { promise = Promise.resolve(value.start()); } catch (error) { promise = Promise.reject(error); } + const pending = promise.finally(() => { $activeAsync--; }); + pending.catch(() => {}); + return $wait(pending, location); +} +const $PromiseRegistry = Object.freeze({ + resolve: $fn('Promise.resolve', 1, [], (args, location) => { + if (args[0] instanceof $Coroutine) $fail('TypeError', 'Await a coroutine before Promise conversion.', location); + return args[0] instanceof $PromiseValue ? args[0] : new $PromiseValue(Promise.resolve(args[0])); + }), + reject: $fn('Promise.reject', 1, [], (args, location) => { + if (!(args[0] instanceof $PyError)) $fail('TypeError', 'Promise.reject requires an adopted exception.', location); + return new $PromiseValue(Promise.reject(args[0])); + }) +}); +function $asyncAttribute(value, name, location) { + if (value === $JS && name === 'Promise') return $PromiseRegistry; + if (value === $PromiseRegistry && (name === 'resolve' || name === 'reject')) return value[name]; + if (value instanceof $PromiseValue && name === 'then') return $fn('Promise.then', null, [], (args, where) => { + if (args.length > 2 || args.some(fn => fn !== null && (!(fn instanceof $Callable) || $asyncFunctions.has(fn)))) { + $fail('TypeError', 'then requires up to two synchronous Python callbacks or None.', where); + } + const callback = fn => fn == null ? undefined : result => { + $checkpoint(where); + const returned = $call(fn, [result], where); + if (returned instanceof $Coroutine) $fail('TypeError', 'Async Promise callbacks are outside this profile.', where); + return returned instanceof $PromiseValue ? returned.promise : returned; + }; + return new $PromiseValue(value.promise.then(callback(args[0]), callback(args[1]))); + }); + $fail('TypeError', 'Attribute is outside the adopted async registry.', location); +} +'''), + 'async-expose': (('async', 'input'), r''' +const $asyncExposed = new WeakMap(); +function $asyncExpose(value, location) { + if (!(value instanceof $Callable)) return value; + if (!$asyncExposed.has(value)) { + const invoke = args => { + $checkpoint(location); + return $call(value, args.map(argument => $input(argument, location)), location); + }; + $asyncExposed.set(value, $asyncFunctions.has(value) ? + (...args) => Promise.resolve().then(() => $await(invoke(args), location)) : (...args) => { + const result = invoke(args); + return result instanceof $PromiseValue ? $await(result, location) : result; + }); + } + return $asyncExposed.get(value); +} '''), 'input': (('call', 'str', 'tuple'), r''' function $input(value, location, depth = 0, seen = new Set()) { @@ -248,6 +347,8 @@ class $JSArray { constructor(values) { this.values = values; } } 'types': ('$types', '$isinstance'), 'exception': ('$TypeError',), 'repr': ('$repr',), 'print': ('$print',), 'javascript': ('$JS', '$JArray', '$attribute'), 'input': ('$input', '$expose'), + 'async': ('$checkpoint', '$await', '$cfn', '$PromiseRegistry', '$asyncAttribute'), + 'async-expose': ('$asyncExpose',), } diff --git a/src/JavaScript/compiler.py b/src/JavaScript/compiler.py index 1173d5b..5a7ae9e 100644 --- a/src/JavaScript/compiler.py +++ b/src/JavaScript/compiler.py @@ -10,7 +10,8 @@ ValueType = Literal['statement', 'python-value', 'python-none', 'python-bool', 'python-int', 'python-str', 'python-tuple', 'python-exception', 'python-type', 'callable', 'javascript-type', 'javascript-array', - 'javascript-number', 'javascript-callable'] + 'javascript-number', 'javascript-callable', 'async-callable', + 'python-coroutine', 'python-promise'] @dataclass(frozen=True) @@ -116,7 +117,7 @@ def accepted(value): class _FrontEnd: - def __init__(self, source, filename, source_imports=None): + def __init__(self, source, filename, source_imports=None, *, async_enabled=False): self.filename = filename self.lines = source.splitlines() or [''] self.bindings = {} @@ -124,6 +125,8 @@ def __init__(self, source, filename, source_imports=None): self.locals = {} self.types = {} self.source_imports = source_imports or {} + self.async_enabled = async_enabled + self.async_scopes = set() try: self.tree = ast.parse(source, filename=filename) self.table = symtable.symtable(source, filename, 'exec') @@ -166,8 +169,11 @@ def _declare_module(self): continue names = [] kind, target = 'constant', None - if isinstance(node, ast.FunctionDef): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): names, kind = [node.name], 'function' + if isinstance(node, ast.AsyncFunctionDef): + self.types['module:' + node.name] = 'async-callable' + self.async_scopes.add('function:' + node.name) elif isinstance(node, ast.Assign): if len(node.targets) != 1 or not isinstance(node.targets[0], ast.Name) or not _immutable(node.value): self.fail('PJS100', 'Module bindings require single-name immutable literals.', node) @@ -180,9 +186,9 @@ def _declare_module(self): self.fail('PJS110', 'Only the approved JavaScript registry may be imported.', node) names, target = [node.names[0].asname or 'JavaScript'], 'JavaScript' else: - if node.level or node.module != 'JavaScript' or len(node.names) != 1 or node.names[0].name != 'Array': + if node.level or node.module != 'JavaScript' or len(node.names) != 1 or node.names[0].name not in ({'Array', 'Promise'} if self.async_enabled else {'Array'}): self.fail('PJS110', 'Only the approved JavaScript.Array binding may be imported.', node) - names, target = [node.names[0].asname or 'Array'], 'JavaScript.Array' + names, target = [node.names[0].asname or node.names[0].name], 'JavaScript.' + node.names[0].name for name in names: if name in self.root: self.fail('PJS100', 'Module rebinding is outside the immutable profile.', node) @@ -221,18 +227,34 @@ def expression(self, node, scope): kind = self.types.get(binding.id, 'python-value') target = binding.target if binding.kind == 'function': - kind, target = 'callable', binding.id + kind, target = self.types.get(binding.id, 'callable'), binding.id elif binding.kind == 'javascript': kind = 'javascript-type' elif binding.kind == 'builtin': kind = 'python-type' if binding.name in _TYPES else 'callable' elif binding.kind == 'source': - target = binding.id if kind == 'callable' else None + target = binding.id if kind in {'callable', 'async-callable'} else None return self.node(node, 'name', kind, binding=binding.id, target=target) + if isinstance(node, ast.Await): + if not self.async_enabled or scope not in self.async_scopes: + self.fail('PJS100', 'Await requires an adopted async function.', node) + value = self.expression(node.value, scope) + if value.value_type not in {'python-coroutine', 'python-promise', 'python-value'}: + self.fail('PJS120', 'Await requires an adopted coroutine or Promise.', node) + return self.node(node, 'await', 'python-value', value=value) if isinstance(node, ast.Tuple): return self.node(node, 'tuple', 'python-tuple', values=tuple(self.expression(value, scope) for value in node.elts)) if isinstance(node, ast.Attribute): value = self.expression(node.value, scope) + if self.async_enabled: + async_target = {('JavaScript', 'Promise'): 'JavaScript.Promise', + ('JavaScript.Promise', 'resolve'): 'JavaScript.Promise.resolve', + ('JavaScript.Promise', 'reject'): 'JavaScript.Promise.reject'}.get((value.field('target'), node.attr)) + if value.value_type == 'python-promise' and node.attr == 'then': + async_target = 'Promise.then' + if async_target: + return self.node(node, 'async-attribute', 'javascript-type' if async_target == 'JavaScript.Promise' else 'javascript-callable', + value=value, name=node.attr, target=async_target) target = {( 'JavaScript', 'Array'): 'JavaScript.Array', ('JavaScript.Array', 'of'): 'JavaScript.Array.of'}.get((value.field('target'), node.attr)) if target: @@ -250,15 +272,24 @@ def expression(self, node, scope): allowed = {'builtin:len': ('python-int', 1), 'builtin:print': ('python-none', None), 'builtin:isinstance': ('python-bool', 2), 'builtin:TypeError': ('python-exception', 1), 'JavaScript.Array.of': ('javascript-array', None)} + if self.async_enabled: + allowed.update({'JavaScript.Promise.resolve': ('python-promise', 1), + 'JavaScript.Promise.reject': ('python-promise', 1), + 'Promise.then': ('python-promise', None)}) if target in allowed: kind, count = allowed[target] if count is not None and len(node.args) != count: self.fail('PJS120', 'Unsupported argument count.', node) elif target in self.bindings and self.bindings[target].kind in {'function', 'source'}: - kind = 'python-value' + kind = 'python-coroutine' if function.value_type == 'async-callable' else 'python-value' else: self.fail('PJS120', 'Call does not resolve to an approved direct callable.', node) arguments = tuple(self.expression(argument, scope) for argument in node.args) + if any(argument.value_type == 'python-coroutine' for argument in arguments): + self.fail('PJS120', 'Coroutine arguments must be awaited first.', node) + if target == 'Promise.then' and (len(arguments) > 2 or any( + argument.value_type not in {'callable', 'python-none'} for argument in arguments)): + self.fail('PJS120', 'then accepts up to two plain synchronous Python callbacks or None.', node) if target == 'builtin:isinstance' and arguments[1].field('target') not in {'builtin:' + name for name in _TYPES}: self.fail('PJS120', 'isinstance requires an approved builtin type binding.', node) return self.node(node, 'call', kind, function=function, arguments=arguments, target=target) @@ -325,7 +356,10 @@ def statement(self, node, scope, loop): bindings=tuple((self.root[imported.name].id, imported.export) for imported in imports)) names = [alias.asname or alias.name for alias in node.names] return self.node(node, 'import', bindings=tuple(self.root[name].id for name in names)) - if isinstance(node, ast.FunctionDef): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + asynchronous = isinstance(node, ast.AsyncFunctionDef) + if asynchronous and not self.async_enabled: + self.fail('PJS100', 'Async functions require the explicit async profile.', node) if node not in self.tree.body or scope or node.decorator_list or node.type_params or node.returns or any(arg.annotation for arg in [*node.args.posonlyargs, *node.args.args]): self.fail('PJS100', 'Nested, decorated, generic or annotated functions are deferred.', node) if node.args.vararg or node.args.kwarg or node.args.kwonlyargs: @@ -339,7 +373,7 @@ def statement(self, node, scope, loop): self.types = original return self.node(node, 'function', binding=self.root[node.name].id, parameters=tuple(self.locals[function_scope][arg.arg].id for arg in [*node.args.posonlyargs, *node.args.args]), - defaults=defaults, body=body) + defaults=defaults, body=body, asynchronous=asynchronous) if isinstance(node, ast.Assign): if any(not isinstance(target, ast.Name) for target in node.targets): self.fail('PJS100', 'Only simple-name assignment is supported.', node) @@ -355,7 +389,11 @@ def statement(self, node, scope, loop): if isinstance(node, ast.Return): if scope is None: self.fail('PJS100', 'Return requires a function.', node) - return self.node(node, 'return', value=self.expression(node.value, scope) if node.value else None) + value = self.expression(node.value, scope) if node.value else None + if value and (value.value_type == 'python-coroutine' or + (scope in self.async_scopes and value.value_type == 'python-promise')): + self.fail('PJS120', 'Returning an unawaited coroutine or Promise is outside this profile.', node) + return self.node(node, 'return', value=value) if isinstance(node, ast.If): test = self.expression(node.test, scope) original = self.types.copy() @@ -393,7 +431,7 @@ def statement(self, node, scope, loop): self.fail('PJS100', 'Unsupported statement: ' + type(node).__name__, node) -def parse_source(source: str, *, filename: str = '') -> ModuleIR: +def parse_source(source: str, *, filename: str = '', async_enabled: bool = False) -> ModuleIR: """Validate source and return immutable typed IR without executing target code.""" if not isinstance(source, str) or not isinstance(filename, str): raise TypeError('source and filename must be strings') @@ -403,7 +441,10 @@ def parse_source(source: str, *, filename: str = '') -> ModuleIR: raise CompileError(Diagnostic('PJS100', 'Source must be valid UTF-8.', SourceSpan(filename, 1, 1, 1, 1), 'parse')) from None if len(encoded) > 1_048_576: raise CompileError(Diagnostic('PJS100', 'Source exceeds the 1 MiB profile limit.', SourceSpan(filename, 1, 1, 1, 1), 'parse')) - frontend = _FrontEnd(source, filename) + if type(async_enabled) is not bool: + raise TypeError('async_enabled must be bool') + frontend = _FrontEnd(source, filename, async_enabled=async_enabled) body = frontend.statements(frontend.tree.body) return ModuleIR(filename, hashlib.sha256(encoded).hexdigest(), - tuple(frontend.bindings[key] for key in sorted(frontend.bindings)), body) + tuple(frontend.bindings[key] for key in sorted(frontend.bindings)), body, + profile='python-browser-async/1' if async_enabled else 'python-browser/1') diff --git a/src/JavaScript/emitter.py b/src/JavaScript/emitter.py index 5f25aa2..629f7a1 100644 --- a/src/JavaScript/emitter.py +++ b/src/JavaScript/emitter.py @@ -45,6 +45,12 @@ def _vlq(value): return encoded +def _contains_await(node): + if not hasattr(node, 'fields'): + return any(_contains_await(item) for item in node) if isinstance(node, tuple) else False + return node.kind == 'await' or any(_contains_await(value) for _, value in node.fields) + + class _Emitter: def __init__(self, module): self.module = module @@ -93,6 +99,12 @@ def expression(self, node): return value self.use('binding') return f'$get({self.names[binding.id]},{_json(binding.name)},{str(binding.kind in {"local", "parameter"}).lower()},{location})' + if kind == 'await': + self.use('async') + return f'(await $await({self.expression(fields["value"])},{location}))' + if kind == 'async-attribute': + self.use('async') + return f'$asyncAttribute({self.expression(fields["value"])},{_json(fields["name"])},{location})' if kind == 'attribute': self.use('javascript') return f'$attribute({self.expression(fields["value"])},{_json(fields["name"])},{location})' @@ -118,7 +130,8 @@ def expression(self, node): if fields['operator'] == 'and': condition = '!' + condition pieces.extend([f'if({condition})return {name};', f'{name}={self.expression(value)};']) - return '(()=>{' + ''.join(pieces) + f'return {name};' + '})()' + code = '(()=>{' + ''.join(pieces) + f'return {name};' + '})()' + return '(await ' + code.replace('(()=>', '(async()=>', 1) + ')' if _contains_await(node) else code if kind == 'compare': self.use('compare') left, right = self.temporary(), self.temporary() @@ -126,11 +139,15 @@ def expression(self, node): for operator, value in zip(fields['operators'], fields['values'][1:]): pieces.extend([f'{right}={self.expression(value)};', f'if(!$compare({_json(operator)},{left},{right},{location}))return false;', f'{left}={right};']) - return '(()=>{' + ''.join(pieces) + 'return true;})()' + code = '(()=>{' + ''.join(pieces) + 'return true;})()' + return '(await ' + code.replace('(()=>', '(async()=>', 1) + ')' if _contains_await(node) else code raise AssertionError('Unrecognized validated expression: ' + kind) def statements(self, nodes, indent=1): for node in nodes: + if 'async/' in self.module.profile: + self.use('async') + self.add(f'$checkpoint({self.location(node)});', indent, node) self.statement(node, indent) def statement(self, node, indent): @@ -143,13 +160,20 @@ def statement(self, node, indent): elif kind == 'import': self.use('javascript') for identifier in fields['bindings']: - value = '$JS' if self.bindings[identifier].target == 'JavaScript' else '$JArray' + target = self.bindings[identifier].target + if target == 'JavaScript.Promise': + self.use('async') + value = {'JavaScript': '$JS', 'JavaScript.Array': '$JArray', 'JavaScript.Promise': '$PromiseRegistry'}[target] self.add(f'{self.names[identifier]}={value};', indent, node) elif kind == 'function': self.use('call') binding = self.bindings[fields['binding']] defaults = '[' + ','.join(map(self.expression, fields['defaults'])) + ']' - self.add(f'{self.names[binding.id]}=$fn({_json(binding.name)},{len(fields["parameters"])},{defaults},($args)=>{{', indent, node) + asynchronous = fields.get('asynchronous', False) + if asynchronous: + self.use('async') + factory, modifier = ('$cfn', 'async ') if asynchronous else ('$fn', '') + self.add(f'{self.names[binding.id]}={factory}({_json(binding.name)},{len(fields["parameters"])},{defaults},{modifier}($args)=>{{', indent, node) locals_ = [item for item in self.module.bindings if item.scope == 'function:' + binding.name] if locals_: self.use('binding') @@ -201,9 +225,10 @@ def render(self, source, module_name, source_map, *, shared_runtime=False): for binding in exports: value = self.names[binding.id] if binding.kind == 'function' and not shared_runtime: - self.use('input') + self.use('async-expose' if 'async/' in self.module.profile else 'input') definition = next(node for node in self.module.body if node.kind == 'function' and node.field('binding') == binding.id) - value = f'$expose({value},{self.location(definition)})' + expose = '$asyncExpose' if 'async/' in self.module.profile else '$expose' + value = f'{expose}({value},{self.location(definition)})' items.append(f'[{_json(binding.name)}]:{value}') self.add('return Object.freeze({' + ','.join(items) + '});') helpers = helper_closure(self.required) @@ -213,7 +238,8 @@ def render(self, source, module_name, source_map, *, shared_runtime=False): if names: lines.append((' const {' + ','.join(names) + '}=runtime;', None)) else: - lines = [('export function createModule({write = text => console.log(text.slice(0,-1))} = {}) {', None), + option = ', signal = null' if 'async/' in self.module.profile else '' + lines = [('export function createModule({write = text => console.log(text.slice(0,-1))' + option + '} = {}) {', None), (" if(typeof write !== 'function')throw new TypeError('write must be a function');", None)] for helper in helpers: lines.extend((' ' + line, None) for line in helper_source(helper).splitlines()) @@ -239,13 +265,13 @@ def render(self, source, module_name, source_map, *, shared_runtime=False): def compile_source(source: str, *, filename: str = 'app.py', module_name: str = 'app.mjs', - source_map: bool = True) -> CompiledArtifact: + source_map: bool = True, async_enabled: bool = False) -> CompiledArtifact: """Compile one approved module to immutable artifacts; do not execute or publish it.""" if not isinstance(module_name, str) or not re.fullmatch(r'[A-Za-z][A-Za-z0-9_-]*\.mjs', module_name): raise ValueError('module_name must be a portable .mjs basename') if type(source_map) is not bool: raise TypeError('source_map must be bool') - module = parse_source(source, filename=filename) + module = parse_source(source, filename=filename, async_enabled=async_enabled) output, helpers, exports = _Emitter(module).render(source, module_name, source_map) compiler_version = version('Python-JavaScript') notice = ('Generated by Python-JavaScript ' + compiler_version + '\n' @@ -255,7 +281,7 @@ def compile_source(source: str, *, filename: str = 'app.py', module_name: str = output.append(OutputFile('NOTICES.txt', notice.encode('utf-8'), 'notice')) files = tuple(sorted(output, key=lambda item: item.path)) manifest = {'kind': 'python-javascript-compiled', 'schema_version': 1, - 'compiler': {'name': 'Python-JavaScript', 'version': compiler_version, 'abi': 'python-browser-esm/1'}, + 'compiler': {'name': 'Python-JavaScript', 'version': compiler_version, 'abi': 'python-browser-async-esm/1' if async_enabled else 'python-browser-esm/1'}, 'language_profile': module.profile, 'target': 'ecmascript-2022-module', 'unicode_version': unicodedata.unidata_version, 'entrypoint': module_name, 'entrypoint_export': 'createModule', 'application_exports': exports, diff --git a/src/JavaScript/modules.py b/src/JavaScript/modules.py index d3853b9..2dbbe71 100644 --- a/src/JavaScript/modules.py +++ b/src/JavaScript/modules.py @@ -60,7 +60,8 @@ def from_text(cls, name, text, *, is_package=False, filename=None): class _Modules: - def __init__(self, sources, entrypoint): + def __init__(self, sources, entrypoint, async_enabled): + self.async_enabled = async_enabled if not isinstance(sources, tuple) or any(not isinstance(source, SourceModule) for source in sources): raise TypeError('sources must be a tuple of verified SourceModule records') if not 1 <= len(sources) <= 32 or sum(len(source.data) for source in sources) > 8_388_608: @@ -114,7 +115,7 @@ def __init__(self, sources, entrypoint): self.dependencies[name].add(base) pending.append(base) for node in tree.body: - if isinstance(node, ast.FunctionDef): + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): self.declarations[name][node.name] = ('function', node) elif isinstance(node, ast.Assign): if len(node.targets) == 1 and isinstance(node.targets[0], ast.Name): @@ -162,7 +163,7 @@ def export_type(self, name, exported, requester, node): self.fail(requester, 'Export is outside the declared Python namespace: ' + name + ':' + exported, node) kind, value = declaration if kind == 'function': - return 'callable' + return 'async-callable' if isinstance(value, ast.AsyncFunctionDef) else 'callable' if kind == 'constant': if not _immutable(value.value): self.fail(name, 'Module bindings require immutable literals.', value, 'PJS100') @@ -179,20 +180,22 @@ def lower(self, name): SourceImport(alias.asname or alias.name, base, alias.name, self.export_type(base, alias.name, name, node)) for alias in node.names) for node, base in self.imports[name]} - frontend = _FrontEnd(source.data.decode('utf-8'), source.filename, imports) + frontend = _FrontEnd(source.data.decode('utf-8'), source.filename, imports, async_enabled=self.async_enabled) body = frontend.statements(frontend.tree.body) return ModuleIR(source.filename, source.sha256, tuple(frontend.bindings[key] for key in sorted(frontend.bindings)), body, - profile='python-browser-modules/1') + profile='python-browser-modules-async/1' if self.async_enabled else 'python-browser-modules/1') def compile_modules(sources: tuple[SourceModule, ...], *, entrypoint: str, - source_map: bool = True, notices: tuple[OutputFile, ...] = ()) -> CompiledArtifact: + source_map: bool = True, async_enabled: bool = False, notices: tuple[OutputFile, ...] = ()) -> CompiledArtifact: """Compile a verified source allowlist, never reading or executing target files.""" if type(source_map) is not bool: raise TypeError('source_map must be bool') - graph = _Modules(sources, entrypoint) - output, required, module_records = [], {'input'}, [] + if type(async_enabled) is not bool: + raise TypeError('async_enabled must be bool') + graph = _Modules(sources, entrypoint, async_enabled) + output, required, module_records = [], {'async-expose' if async_enabled else 'input'}, [] modules = sorted(graph.trees) for index, name in enumerate(modules): source = graph.sources[name] @@ -205,7 +208,7 @@ def compile_modules(sources: tuple[SourceModule, ...], *, entrypoint: str, 'exports': exports, 'source': source.filename, 'sha256': source.sha256}) helpers = helper_closure(required) bindings = [binding for helper in helpers for binding in HELPER_BINDINGS[helper]] - runtime = ["export function createRuntime(write) {", " if(typeof write!=='function')throw new TypeError('write must be a function');"] + runtime = ["export function createRuntime(write" + (", signal = null" if async_enabled else "") + ") {", " if(typeof write!=='function')throw new TypeError('write must be a function');"] for helper in helpers: runtime.extend(' ' + line for line in helper_source(helper).splitlines()) runtime.extend([' return Object.freeze({' + ','.join(bindings) + '});', '}']) @@ -215,9 +218,12 @@ def compile_modules(sources: tuple[SourceModule, ...], *, entrypoint: str, registry = '[' + ','.join('[' + _json(record['module']) + ',module' + str(index) + ']' for index, record in enumerate(module_records)) + ']' packages = _json([record['module'] for record in module_records if record['is_package']]) location = _json({'filename': graph.sources[entrypoint].filename, 'line': 1, 'column': 1, 'end_line': 1, 'end_column': 1}) + expose = '$asyncExpose' if async_enabled else '$expose' + signal_option = ', signal = null' if async_enabled else '' + signal_argument = ',signal' if async_enabled else '' entry = '\n'.join(imports) + '\n' + f''' -export function createModule({{write = text => console.log(text.slice(0,-1))}} = {{}}) {{ - const runtime=createRuntime(write), factories=new Map({registry}), packages=new Set({packages}), cache=new Map(), loading=new Set(); +export function createModule({{write = text => console.log(text.slice(0,-1)){signal_option}}} = {{}}) {{ + const runtime=createRuntime(write{signal_argument}), factories=new Map({registry}), packages=new Set({packages}), cache=new Map(), loading=new Set(); function importModule(name) {{ if(cache.has(name))return cache.get(name); const parent=name.includes('.')?name.slice(0,name.lastIndexOf('.')):null; @@ -230,7 +236,7 @@ def compile_modules(sources: tuple[SourceModule, ...], *, entrypoint: str, loading.delete(name);cache.set(name,value);return value; }} const module=importModule({_json(entrypoint)}); - return Object.freeze(Object.fromEntries(Object.entries(module).map(([name,value])=>[name,runtime.$expose(value,{location})]))); + return Object.freeze(Object.fromEntries(Object.entries(module).map(([name,value])=>[name,runtime.{expose}(value,{location})]))); }} ''' output.append(OutputFile('app.mjs', entry.encode('utf-8'), 'executable-module')) @@ -248,8 +254,8 @@ def compile_modules(sources: tuple[SourceModule, ...], *, entrypoint: str, raise ValueError('Artifact file/directory paths collide.') files = tuple(sorted(output, key=lambda item: item.path)) manifest = {'kind': 'python-javascript-compiled', 'schema_version': 1, - 'compiler': {'name': 'Python-JavaScript', 'version': compiler_version, 'abi': 'python-browser-modules-esm/1'}, - 'language_profile': 'python-browser-modules/1', 'target': 'ecmascript-2022-module', + 'compiler': {'name': 'Python-JavaScript', 'version': compiler_version, 'abi': 'python-browser-modules-async-esm/1' if async_enabled else 'python-browser-modules-esm/1'}, + 'language_profile': 'python-browser-modules-async/1' if async_enabled else 'python-browser-modules/1', 'target': 'ecmascript-2022-module', 'unicode_version': unicodedata.unidata_version, 'entrypoint': 'app.mjs', 'entrypoint_export': 'createModule', 'application_exports': next(record['exports'] for record in module_records if record['module'] == entrypoint), 'modules': module_records, 'helpers': list(helpers), diff --git a/tests/test_compiler_async.py b/tests/test_compiler_async.py new file mode 100644 index 0000000..8129976 --- /dev/null +++ b/tests/test_compiler_async.py @@ -0,0 +1,152 @@ +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +from JavaScript.compiler import CompileError, parse_source +from JavaScript.emitter import compile_source +from JavaScript.modules import SourceModule, compile_modules + + +def run_async(artifact, invocation='await app.main();', options='write: text=>stdout+=text'): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + for file in artifact.files: + (root / file.path).write_bytes(file.data) + (root / 'run.mjs').write_text( + "import {createModule} from './app.mjs';let stdout='',error=null;\n" + "const controller=new AbortController();let listeners=0;" + "const signal={get aborted(){return controller.signal.aborted}," + "addEventListener(...args){listeners++;controller.signal.addEventListener(...args)}," + "removeEventListener(...args){listeners--;controller.signal.removeEventListener(...args)}};\n" + "try { const app=createModule({" + options + "});" + invocation + " }" + "catch(caught){error={name:caught.name,message:caught.message,code:caught.code,location:caught.location};}" + "await new Promise(resolve=>setImmediate(resolve));" + "console.log(JSON.stringify({stdout,error,listeners}));", encoding='utf-8') + process = subprocess.run(['node', '--unhandled-rejections=strict', str(root / 'run.mjs')], + check=True, capture_output=True, text=True, encoding='utf-8', timeout=10) + return json.loads(process.stdout) + + +class CompilerAsyncTests(unittest.TestCase): + def assert_parity(self, source): + oracle = subprocess.run([sys.executable, '-I', '-X', 'utf8', '-c', source + '\nimport asyncio\nasyncio.run(main())'], + capture_output=True, text=True, encoding='utf-8', timeout=10) + result = run_async(compile_source(source, filename='async_case.py', async_enabled=True)) + self.assertEqual(result['stdout'], oracle.stdout.replace('\r\n', '\n')) + self.assertEqual(result['error']['name'] if result['error'] else None, + oracle.stderr.splitlines()[-1].split(':', 1)[0] if oracle.returncode else None) + return result + + def test_lazy_coroutines_defaults_order_and_reuse(self): + self.assert_parity('''async def selected(value=(1, "😀")): + print("inside", value) + return value +async def main(): + pending = selected() + print("before") + first = await pending + print("after", first) + second = await selected() + print(first is second) +''') + result = self.assert_parity('''async def selected(): + return 1 +async def main(): + pending = selected() + print(await pending) + await pending +''') + self.assertEqual(result['error']['message'], 'cannot reuse already awaited coroutine') + + def test_promise_conversion_chained_callbacks_and_errors_once(self): + source = '''from JavaScript import Promise as P +def add(value): + print("callback", value) + return P.resolve(value + 1) +def recover(error): + print("recovered") + return 8 +async def main(): + print(await P.resolve(9007199254740993).then(add)) + print(await P.reject(TypeError("bad")).then(None, recover)) +''' + self.assert_parity(source) + result = self.assert_parity('''from JavaScript import Promise +def fail(value): + print("once") + raise TypeError("callback failed") +async def main(): + print(await Promise.resolve(1).then(fail)) +''') + self.assertEqual(result['error']['message'], 'callback failed') + self.assertEqual(result['error']['location']['line'], 4) + + def test_await_in_chained_comparisons_short_circuit_and_loops(self): + self.assert_parity('''import JavaScript as JS +async def mark(value): + print(value) + return await JS.Promise.resolve(value) +async def main(): + print(await mark(1) < await mark(2) < await mark(3)) + print(await mark(3) < await mark(2) < await mark(1)) + print(False and await mark("never")) + print(True or await mark("never")) + for value in (1, 2): + print(await mark(value)) +''') + + def test_cross_module_await_preserves_values_and_provenance(self): + sources = (SourceModule.from_text('app', 'from helper import selected as main'), + SourceModule.from_text('helper', 'from JavaScript import Promise\nasync def selected():\n print(await Promise.resolve(("😀", 9007199254740993)))')) + artifact = compile_modules(sources, entrypoint='app', async_enabled=True) + result = run_async(artifact) + self.assertIsNone(result['error']) + self.assertEqual(result['stdout'], "('😀', 9007199254740993)\n") + self.assertEqual(artifact.manifest['language_profile'], 'python-browser-modules-async/1') + self.assertIn('async-expose', artifact.manifest['helpers']) + artifact = compile_source('async def main():\n return 1 // 0', async_enabled=True, filename='original.py') + error = run_async(artifact)['error'] + self.assertEqual(error['name'], 'ZeroDivisionError') + self.assertEqual(error['location']['filename'], 'original.py') + self.assertEqual(error['location']['line'], 2) + + def test_cancellation_checkpoints_release_listeners_and_reject_exports(self): + artifact = compile_source('from JavaScript import Promise\nasync def main():\n print("begin")\n' + ' value = await Promise.resolve(1)\n print("after", value)', async_enabled=True) + for invocation in ['controller.abort();await app.main();', + 'const pending=app.main();queueMicrotask(()=>controller.abort());await pending;']: + result = run_async(artifact, invocation, 'write: text=>stdout+=text,signal') + self.assertEqual(result['error']['code'], 'CANCELLED') + self.assertNotIn('after', result['stdout']) + self.assertEqual(result['listeners'], 0) + # An output sink can cancel synchronously while a coroutine is starting. + result = run_async(artifact, options='write: text=>{stdout+=text;controller.abort()},signal') + self.assertEqual(result['error']['code'], 'CANCELLED') + self.assertEqual(result['stdout'], 'begin\n') + self.assertEqual(result['listeners'], 0) + result = run_async(artifact, options='write: text=>stdout+=text,signal') + self.assertIsNone(result['error']) + self.assertEqual(result['listeners'], 0) + + def test_async_call_bound_and_rejected_forms(self): + artifact = compile_source('async def main():\n return await main()', async_enabled=True) + self.assertEqual(run_async(artifact)['error']['name'], 'RecursionError') + sources = ['async def main():\n return await 1', + 'from JavaScript import Promise\nasync def callback(value):\n return value\n' + 'async def main():\n return await Promise.resolve(1).then(callback)', + 'from JavaScript import Promise\nasync def main():\n return Promise.resolve(1)', + 'async def main():\n return main()', + 'async def main():\n async for value in ():\n pass', + 'async def main():\n async with thing:\n pass', + 'from JavaScript import fetch\nasync def main():\n return await fetch("url")', + 'from JavaScript import Promise\nPromise.all(())'] + for source in sources: + with self.subTest(source=source), self.assertRaises(CompileError): + compile_source(source, async_enabled=True) + with self.assertRaises(CompileError): + parse_source('async def main():\n return 1') + with self.assertRaises(TypeError): + parse_source('pass', async_enabled=1)