feat(core): dynamic menu providers and runtime permission sources (#340, #334)

CLAUDE.md

@@ -77,6 +77,8 @@ hence `SM022`/`SM023`. See `docs/module-authoring.md` § Styling.
 **Lifecycle hooks** (in `framework/core/simple_module_core/module.py`) — all no-op by default; subclasses override as needed:
 `register_settings` → `register_menu_items` / `register_permissions` / `register_feature_flags` / `register_event_handlers` / `register_invalidations` / `register_health_checks` / `register_public_routes` / `register_csp_sources` / `register_setup_steps` → `register_exception_handlers` → `register_middleware` → `register_routes(api_router, view_router)` / `register_admin_routes(admin_router)` → async `on_startup` / `on_shutdown` (reverse order). `register_admin_routes` is only for modules that serve **both** public and admin pages: a module gets exactly one router per `view_prefix`, which `users` cannot express (sign-in at `/users/login`, management at `/admin/users`). Setting `ModuleMeta.admin_view_prefix` mounts a second view router there. A module whose views are *all* administrative just points `view_prefix` at `/admin/<name>` and keeps using `register_routes`. The prefix is a URL convention, not a permission — guard these routes exactly as you would any other. `register_csp_sources(registry)` lets a module whitelist external asset origins (`registry.add("style-src", "https://rsms.me")`) — fetch directives only, validated at boot. `register_public_routes(registry)` lets a module exempt anonymous/read-only routes (STAC/OGC, webhooks) from `AuthMiddleware`; rules are method-aware (`registry.add_regex(r"…/tilejson$", methods={"GET"})`), so a GET read route can be public while sibling POST/PATCH mutations under the same prefix stay gated. See [docs/framework/public-routes.md](docs/framework/public-routes.md). `register_setup_steps(registry)` lets a module declare what a usable install still needs; while any required step is incomplete `SetupMiddleware` serves the first-run wizard at `/setup` instead of the app. A module that registers nothing never gates — that is how `keycloak` opts out, since its local users table is legitimately empty forever and a host-level superuser count would lock those installs out permanently. `register_invalidations(bus, app)` subscribes a module's **per-process caches** to `InvalidationBus`, so another worker's write drops this worker's entry instead of leaving it stale for its whole TTL; handlers may only *forget*, since there is no delivery guarantee. Publishing takes no hook — `await request.app.state.sm.invalidation.publish(channel, key=...)` from a `db.on_commit` callback. Cross-process delivery needs a transport, which `background_tasks` installs on its Redis connection (`SM_BG_TASKS_BROADCAST_INVALIDATIONS`); with none the bus is in-process and every cache still needs its TTL as a floor. See [docs/framework/invalidation.md](docs/framework/invalidation.md).
 
+`MenuRegistry.add_provider(fn)` (from `register_menu_items`) contributes per-request menu items evaluated in `InertiaLayoutDataMiddleware` after auth/tenant resolution, and `PermissionRegistry.add_source(name, provider)` (from `register_permissions`) contributes runtime-defined permissions from a sync in-memory cache, refreshed with `invalidate_source(name)`; see [docs/framework/permissions.md](docs/framework/permissions.md).
+
 **Middleware pipeline** (Starlette `add_middleware` is LIFO — last added runs first). Execution order on a request:
 `(ProxyHeaders, if SM_TRUSTED_PROXY) → CorrelationId → RequestLogging → GZip → SecurityHeaders → Session → <module middleware> → Tenant (opt-in) → Locale → InertiaLayoutData → InertiaCache → Setup → Maintenance → CommitBeforeResponse → app`. `InertiaCache` answers for `InertiaLayoutData` merging per-user `auth`/`menus` into every payload: a response to an `X-Inertia` request is forced to `private, no-store` with its ETag dropped, and both representations of a URL gain `Vary: X-Inertia` — so no cache can store the JSON payload or hand it back for a page request. A module wanting its public page content cached should set `Cache-Control` and an ETag on the *document*; that path is left alone. `GZip` compresses any response over 500 bytes, including the `/static` mount — the built CSS is ~139 KB raw versus ~21 KB gzipped, and uncompressed assets dominated cold page load. `ProxyHeaders` (uvicorn's `ProxyHeadersMiddleware`) is installed only when `SM_TRUSTED_PROXY` is set, sitting outermost so the `X-Forwarded-*`-corrected scheme/client IP reach everything downstream (request logs). Inertia does not depend on it: the page url is rewritten to the root-relative form the protocol specifies (`_inertia_url.py`), so no scheme travels in the payload to disagree with the document's — the cross-scheme `pushState` `SecurityError` of GH #223 cannot recur on an install that never set the variable. When two modules add middleware at the same dependency tier, the module that sorts **later** wraps outermost. Use `depends_on` to express relative order — don't rely on names. `Maintenance` serves a 503 page to everyone but admins while `maintenance_mode` is set on `HostSettings`; it sits inside `InertiaCache` because its 503 is an Inertia payload produced by short-circuiting, and outside the cache guard that payload would ship storable. `Setup` runs just before it, for the same cache reason and because an install that was never set up has nothing meaningful to put into maintenance.
 

docs/framework-conventions.md

@@ -358,6 +358,8 @@ way.
 
   Sidebar items can also set `group="<Label>"` on the `MenuItem` to render under a group header. The frontend clusters consecutive items with the same group label and prints the label as a section heading; the group's position is set by the lowest-`order` item that joins it. Built-in groups are `Content`, `Administration`, and `System`. Items with no `group` (the default) render flat — Dashboard intentionally stays ungrouped above the headed groups.
 
+  **Dynamic menus.** Items that depend on the request (per-tenant entries, resources admins create at runtime) come from a provider registered in `register_menu_items`: `registry.add_provider(fn)`, where `fn(request)` returns an iterable of `MenuItem` (sync or async). Providers run per request inside `InertiaLayoutDataMiddleware`, after auth and `request.state.tenant_id` are resolved, and their items are role/permission-filtered, translated, sorted and grouped like static ones. A provider that raises is logged and contributes nothing. `registry.remove(predicate)` drops static items.
+
   **Translating menu labels.** Set `label_key` (and `group_key`) alongside `label`/`group` to name a catalog entry:
 
   ```python

docs/framework/permissions.md

@@ -73,6 +73,17 @@ Host apps customize this by:
 
 The framework ships only `admin: ["*"]`. The wildcard grants every declared permission.
 
+## Runtime permission sources
+
+`register_permissions` runs before the database is open, so it cannot declare permissions for resources admins create later (per-record-type keys such as `records.product.edit`). Register a **source** instead:
+
+```python
+def register_permissions(self, registry: PermissionRegistry) -> None:
+    registry.add_source("records", lambda: [("records.product.edit", "Edit products")])
+```
+
+The provider is **sync** and must read a module-maintained in-memory cache, because the registry is consulted on every request. It returns permission keys or `(key, label)` pairs. Its output is cached by the registry; call `registry.invalidate_source("records")` when the underlying resources change. Source permissions appear in `all_permissions`, in the role editor under the source's group name (labels ship in `PermissionGroupOut.labels`), in admin's implicit all-permissions, and are enforced by `RequiresPermission`. They are grantable and persisted like any other key. A source that raises is logged and contributes nothing.
+
 ## User resolution
 
 For each request, `AuthMiddleware` resolves the authenticated user onto `request.state.user` as a `UserContext` (from `auth.contracts.schemas`):

framework/core/simple_module_core/menu.py

@@ -2,10 +2,17 @@
 
 from __future__ import annotations
 
-from collections.abc import Callable
+import inspect
+import logging
+from collections.abc import Awaitable, Callable, Iterable
 from dataclasses import dataclass, field
 from enum import StrEnum
-from typing import Literal
+from typing import TYPE_CHECKING, Literal
+
+if TYPE_CHECKING:
+    from starlette.requests import Request
+
+logger = logging.getLogger(__name__)
 
 MenuItemMethod = Literal["get", "post"]
 
@@ -67,12 +74,17 @@ class MenuItem:
     is set by the lowest-ordered item that belongs to it."""
 
 
+MenuProvider = Callable[["Request"], "Awaitable[Iterable[MenuItem]] | Iterable[MenuItem]"]
+"""Per-request source of menu items; sync or async."""
+
+
 class MenuRegistry:
     """Collects menu items from all modules and filters them per-request."""
 
     def __init__(self) -> None:
         self._items: list[MenuItem] = []
         self._sorted: list[MenuItem] | None = None
+        self._providers: list[MenuProvider] = []
 
     def _invalidate(self) -> None:
         self._sorted = None
@@ -85,19 +97,70 @@ def add_many(self, items: list[MenuItem]) -> None:
         self._items.extend(items)
         self._invalidate()
 
+    def remove(self, predicate: Callable[[MenuItem], bool]) -> int:
+        """Drop static items for which *predicate* is true; return how many."""
+        kept = [i for i in self._items if not predicate(i)]
+        removed = len(self._items) - len(kept)
+        self._items = kept
+        self._invalidate()
+        return removed
+
+    def add_provider(self, provider: MenuProvider) -> None:
+        """Register a per-request provider of extra menu items.
+
+        Evaluated inside ``InertiaLayoutDataMiddleware`` (auth and
+        ``request.state.tenant_id`` already resolved) and then filtered,
+        translated, ordered and grouped like static items. A provider that
+        raises is logged and contributes nothing.
+        """
+        self._providers.append(provider)
+
+    async def collect_provider_items(self, request: Request) -> list[MenuItem]:
+        items: list[MenuItem] = []
+        for provider in self._providers:
+            try:
+                result = provider(request)
+                if inspect.isawaitable(result):
+                    result = await result
+                # Materialise first: a generator failing midway must add nothing.
+                items.extend(list(result))
+            except Exception:
+                logger.exception("Menu provider %r failed; contributing nothing", provider)
+        return items
+
     @property
     def all_items(self) -> list[MenuItem]:
         if self._sorted is None:
             self._sorted = sorted(self._items, key=lambda i: i.order)
         return self._sorted
 
+    async def get_for_request(
+        self,
+        request: Request,
+        *,
+        is_authenticated: bool,
+        roles: list[str] | None = None,
+        permissions: list[str] | None = None,
+        translate: Callable[[str], str] | None = None,
+    ) -> dict[str, list[dict]]:
+        """:meth:`get_for_user` over static items plus this request's provider items."""
+        extra = await self.collect_provider_items(request) if self._providers else []
+        return self.get_for_user(
+            is_authenticated=is_authenticated,
+            roles=roles,
+            permissions=permissions,
+            translate=translate,
+            extra_items=extra,
+        )
+
     def get_for_user(
         self,
         *,
         is_authenticated: bool,
         roles: list[str] | None = None,
         permissions: list[str] | None = None,
         translate: Callable[[str], str] | None = None,
+        extra_items: Iterable[MenuItem] = (),
     ) -> dict[str, list[dict]]:
         """Return menu items grouped by section, filtered by auth/roles/permissions.
 
@@ -124,7 +187,10 @@ def render(key: str, fallback: str) -> str:
             translated = translate(key)
             return fallback if translated == key else translated
 
-        for item in self.all_items:
+        items = self.all_items
+        if extra := list(extra_items):
+            items = sorted([*items, *extra], key=lambda i: i.order)
+        for item in items:
             if item.requires_auth and not is_authenticated:
                 continue
             if item.roles and not any(r in item.roles for r in roles):

framework/core/simple_module_core/permissions.py

@@ -2,9 +2,15 @@
 
 from __future__ import annotations
 
-from collections.abc import Collection
+import logging
+from collections.abc import Callable, Collection, Iterable
 from dataclasses import dataclass, field
 
+logger = logging.getLogger(__name__)
+
+PermissionSourceProvider = Callable[[], "Iterable[str] | Iterable[tuple[str, str]]"]
+"""Sync callable returning permission keys, or ``(key, label)`` pairs."""
+
 WILDCARD = "*"
 
 ADMIN_ROLE = "admin"
@@ -69,11 +75,58 @@ def __init__(self) -> None:
         self._role_overlay: dict[str, set[str]] = {}
         self._all_permissions_cache: list[str] | None = None
         self._role_map_cache: dict[str, list[str]] | None = None
+        self._sources: dict[str, PermissionSourceProvider] = {}
+        self._source_cache: dict[str, tuple[list[str], dict[str, str]]] = {}
 
     def _invalidate(self) -> None:
         self._all_permissions_cache = None
         self._role_map_cache = None
 
+    # ── Runtime sources ────────────────────────────────────────
+
+    def add_source(self, name: str, provider: PermissionSourceProvider) -> None:
+        """Register a runtime permission source under group *name*.
+
+        For modules whose protected resources are created after boot. *provider*
+        is **sync** and should read a module-maintained in-memory cache — the
+        registry is consulted on every request. Its output is cached here until
+        :meth:`invalidate_source` is called. A provider that raises is logged
+        and contributes nothing.
+        """
+        self._sources[name] = provider
+        self._source_cache.pop(name, None)
+        self._invalidate()
+
+    def invalidate_source(self, name: str) -> None:
+        """Drop the cached output of source *name*; re-read on next access."""
+        self._source_cache.pop(name, None)
+        self._invalidate()
+
+    def _source_output(self, name: str) -> tuple[list[str], dict[str, str]]:
+        cached = self._source_cache.get(name)
+        if cached is not None:
+            return cached
+        keys: list[str] = []
+        labels: dict[str, str] = {}
+        try:
+            for entry in self._sources[name]():
+                if isinstance(entry, str):
+                    keys.append(entry)
+                else:
+                    key, label = entry
+                    keys.append(key)
+                    labels[key] = label
+        except Exception:
+            logger.exception("Permission source %r failed; contributing nothing", name)
+            keys, labels = [], {}
+        out = (sorted(set(keys)), labels)
+        self._source_cache[name] = out
+        return out
+
+    def source_labels(self, name: str) -> dict[str, str]:
+        """Human labels supplied by source *name*, keyed by permission string."""
+        return dict(self._source_output(name)[1]) if name in self._sources else {}
+
     def add_group(self, name: str, permissions: list[str]) -> None:
         """Register a group of related permissions."""
         if name in self._groups:
@@ -96,17 +149,26 @@ def all_permissions(self) -> list[str]:
         """All registered permission strings, sorted."""
         if self._all_permissions_cache is None:
             perms: set[str] = set()
-            for group in self._groups.values():
+            for group in self.groups:
                 perms.update(group.permissions)
             self._all_permissions_cache = sorted(perms)
         return self._all_permissions_cache
 
     @property
     def groups(self) -> list[PermissionGroup]:
-        return list(self._groups.values())
+        """Static groups plus one group per source (merged by name)."""
+        merged = {
+            n: PermissionGroup(name=n, permissions=list(g.permissions))
+            for n, g in self._groups.items()
+        }
+        for name in self._sources:
+            keys = self._source_output(name)[0]
+            group = merged.setdefault(name, PermissionGroup(name=name))
+            group.permissions.extend(k for k in keys if k not in group.permissions)
+        return list(merged.values())
 
     def has(self, permission: str) -> bool:
-        return any(permission in g.permissions for g in self._groups.values())
+        return permission in self.all_permissions
 
     def map_role(self, role: str, permissions: list[str]) -> None:
         """Register a role→permission mapping.

framework/core/tests/test_permission_sources.py

@@ -0,0 +1,52 @@
+"""Runtime permission sources on PermissionRegistry (GH #334)."""
+
+from __future__ import annotations
+
+import logging
+
+from simple_module_core.permissions import PermissionRegistry
+
+
+def test_source_permissions_in_all_permissions_and_groups() -> None:
+    reg = PermissionRegistry()
+    reg.add_group("users", ["users.view"])
+    reg.add_source("records", lambda: ["records.product.edit", ("records.faq.edit", "Edit FAQ")])
+
+    assert "records.product.edit" in reg.all_permissions
+    assert reg.has("records.faq.edit")
+    group = next(g for g in reg.groups if g.name == "records")
+    assert sorted(group.permissions) == ["records.faq.edit", "records.product.edit"]
+    assert reg.source_labels("records") == {"records.faq.edit": "Edit FAQ"}
+    assert reg.source_labels("users") == {}
+
+
+def test_invalidate_source_refreshes() -> None:
+    reg = PermissionRegistry()
+    types = ["a"]
+    reg.add_source("records", lambda: [f"records.{t}.edit" for t in types])
+    assert reg.all_permissions == ["records.a.edit"]
+
+    types.append("b")
+    assert reg.all_permissions == ["records.a.edit"]  # cached
+    reg.invalidate_source("records")
+    assert reg.all_permissions == ["records.a.edit", "records.b.edit"]
+
+
+def test_failing_source_is_isolated(caplog) -> None:
+    reg = PermissionRegistry()
+    reg.add_group("users", ["users.view"])
+
+    def boom() -> list[str]:
+        raise RuntimeError("nope")
+
+    reg.add_source("bad", boom)
+    reg.add_source("good", lambda: ["good.x"])
+    with caplog.at_level(logging.ERROR):
+        assert reg.all_permissions == ["good.x", "users.view"]
+    assert any("bad" in r.getMessage() for r in caplog.records)
+
+
+def test_admin_implicit_all_includes_sources() -> None:
+    reg = PermissionRegistry()
+    reg.add_source("records", lambda: ["records.product.edit"])
+    assert "records.product.edit" in reg.get_permissions_for_roles(["admin"])

framework/hosting/simple_module_hosting/middleware.py

@@ -221,7 +221,8 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
                 "isAuthenticated": is_authenticated,
                 "permissions": frontend_permissions,
             },
-            "menus": self.menu_registry.get_for_user(
+            "menus": await self.menu_registry.get_for_request(
+                request,
                 is_authenticated=is_authenticated,
                 roles=roles,
                 # Already expanded above (wildcards resolved), which is exactly