Skip to content

fix(migrations): make SQLite-autogenerated revisions portable to Postgres (#342) - #406

Merged
antosubash merged 5 commits into
mainfrom
feat/342-migration-portability
Oct 7, 2026
Merged

antosubash merged 5 commits into
mainfrom
feat/342-migration-portability

Conversation

@antosubash

Copy link
Copy Markdown
Owner

Fixes #342

Summary

  • Boolean defaults: render_item + process_revision_directives (the hooks both host/migrations/env.py and the smpy create-host template already call) now emit sa.false()/sa.true() instead of sa.text('0'/'1') for Boolean columns. Logic lives in the new simple_module_db.migration_portability.
  • Expression indexes: autogenerating on SQLite logs a prominent warning naming every expression index in the models (warning, not error: it would otherwise fail every unrelated make migration; the Postgres round trip is the real gate). New diagnostic SM026 (info) in make doctor/dev boot, and check_migrations logs that SQLite cannot verify them instead of implying clean.
  • Enum downgrade: simple_module_db.drop_enums_if_postgres(op, *names). No existing revision creates enum types (verified: no pg_type enums after full upgrade), so none needed changing.
  • CI/Make: make migrations-roundtrip-pg (upgrade heads, downgrade base, upgrade heads, alembic check on SM_MIGRATIONS_PG_URL) and a migrations-roundtrip-pg job with a Postgres service, added to pr-checks needs.
  • Docs: review checklist in docs/module-authoring.md (boolean defaults, expression indexes, enum drops, SAEnum default = member name), SM026 in CLAUDE.md and docs/reference/diagnostic-codes.md.

Findings on existing history

  • ix_users_user_email_lower is already created by 41cf2c53660e; no new index revision needed.
  • Existing revisions' sa.text("0") defaults are on Integer columns (fine on Postgres); the Boolean one already uses sa.false(). No historical edits.
  • The new round trip surfaced real drift that made alembic check fail on Postgres: 8 columns created as naive timestamp while the model (sqlmodel UTCDateTime) is timezone-aware. Fixed with two new revisions (e5f2a8c1d7b3, f6a3b9d2e8c4, the latter on the keycloak branch), Postgres-only ALTER ... USING col AT TIME ZONE 'UTC', no-op on SQLite.

Verification

  • make migrations-roundtrip-pg passes against shared Postgres on a scratch DB (fresh DB, dropped afterwards): 36 upgrades, 18 downgrades, alembic check clean.
  • make lint passes. New unit tests for the render hook, expression-index detection, SM026 and the check_migrations note.

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

…342)

- render Boolean server defaults as sa.false()/sa.true(), not sa.text('0'),
  via the render_item / process_revision_directives hooks both env.py files
  already call (new simple_module_db.migration_portability)
- loud warning when autogenerating on SQLite while models declare expression
  indexes; SM026 (info) in make doctor, and a log line from check_migrations
- drop_enums_if_postgres() helper for downgrades
- migrations-roundtrip-pg make target + CI job (upgrade heads, downgrade
  base, upgrade heads, alembic check) wired into pr-checks
- two revisions fixing timestamp -> timestamptz drift the round trip found
- docs: review checklist in module-authoring, SM026 in diagnostic codes

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-10-05T11:29:32.027458Z 6924931 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Deploying simple-module-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: dd83cd6
Status: ✅  Deploy successful!
Preview URL: https://b8eb3f37.simple-module-python.pages.dev
Branch Preview URL: https://feat-342-migration-portabili.simple-module-python.pages.dev

View logs

@antosubash

Copy link
Copy Markdown
Owner Author

Merge-order note: see #410 — both PRs add a revision on top of c7f2d9a41e83; the second to merge must repoint its down_revision.

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

@antosubash

Copy link
Copy Markdown
Owner Author

/ship results

Pushed 2 commits: d58e1f1f (optimize: single op walker, drop dead return/guard) and 0dc2d248 (review pass 1 fixes).

Review (2 passes, high then medium): pass 1 found 7, fixed 3 (downgrade ops now get portable Boolean defaults + regression test; SM026 attribution limited to a module's own models; duplicate predicate removed), accepted 4 (dialect string-split duplication, simple_module_db importing a private core diagnostics module, one revision touching two modules' tables, 't'/'f' literals). Pass 2 clean. down_revision untouched (merge-order with #410 as already noted).

QA on Postgres (scratch DB sm_ship_406, dropped afterwards):

  • make migrations-roundtrip-pg: up, down to base, up, alembic check all pass, no drift.
  • The 7 converted columns are timestamptz after upgrade; downgrading e5f2a8c1d7b3 alone restores timestamp without time zone; re-upgrade works.
  • USING col AT TIME ZONE 'UTC' preserves instants under Asia/Tokyo and America/New_York sessions (12:00Z); downgrade yields naive 12:00:00. A control without USING shifted by the zone offset.
  • App on Postgres (API 8100, Vite 5250): login/logout, API token login/refresh/rotate/revoke (reuse and expired return 401), Background tasks list/filters/search/retry/workers; no naive-vs-aware errors in the server log; non-UTC offsets serialise correctly.
  • make doctor on SQLite shows SM026; make migration on SQLite prints the expression-index banner (no revision generated).
  • 0 P0-P2 bugs. P3 observations unrelated to this PR: one Retry click made two pending rows; UI logout leaves the access-token row.

Local CI: ruff format/check, ty, biome, tsc, file-size, untranslated-strings pass; make test-py 3599 passed, 10 skipped.

Full report: https://claude.ai/artifact/Fcgu1rqvvY4BhRaDABJR76

https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

@antosubash
antosubash merged commit eedcfa6 into main Oct 7, 2026
14 checks passed
antosubash added a commit that referenced this pull request Oct 7, 2026
…amp revision

Both branched from c7f2d9a41e83; repoint down_revision so main keeps a single
mainline head.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
antosubash added a commit that referenced this pull request Oct 7, 2026
…, thumbnails (#410)

* feat(file_storage): public files with anonymous serving, list search/sort, thumbnails

Fixes #353, fixes #352.

- StoredFile.public (default false; migration uses sa.false()), set on upload
  (public=true) or PATCH /files/{id}; StoredFileOut gains public/public_url.
- Anonymous GET /public/{id}[/{filename}] and /public/{id}/thumbnail, exempted
  via register_public_routes (GET only). Only public, non-deleted rows resolve;
  other cases are one 404. Cross-tenant lookup by id uses all_tenants().
  ETag/304, Cache-Control public, nosniff, sandbox CSP; active content (HTML,
  SVG, JS) forced to attachment and streamed.
- SecurityHeadersMiddleware keeps a CSP the response already set.
- GET /files: q, content_type (exact or "image/" prefix), sort.
- GET /files/{id}/thumbnail?w=: Pillow WebP, width snapped to a whitelist,
  pixel-budget guard, variants cached in the storage backend and dropped with
  the file.
- Browse screen: Public badge and make public/private action.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix(file_storage): harden thumbnails (format allowlist, size/pixel caps, concurrency) and add tenant/permission tests

Pillow only opens JPEG/PNG/WEBP/GIF, sniffed format must match the declared
type, DecompressionBombWarning is an error, source bytes are capped while
reading, decodes are bounded by a semaphore, first frame only, metadata not
carried over.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix(file_storage): hold the decode slot until the worker thread finishes; stop mutating Pillow's global pixel cap

Decode runs as a shielded single-flight task per variant, so a cancelled
request neither frees its slot early nor lets retries multiply decodes. The
semaphore is per event loop. The explicit header-only pixel budget replaces the
racy process-global MAX_IMAGE_PIXELS toggle.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* refactor(optimize): dedupe 404/headers helpers, drop dead code and one-use wrappers in file_storage

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix: address code review findings (round 1, pass 1)

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix: address code review findings (round 1, pass 1b)

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix(file_storage): no session cookie on public responses, strict PATCH bool, 16-bit grayscale thumbnails (QA round 1)

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix(file_storage): log orphaned thumbnail variant deletes (review round 1, pass 3)

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* test: reconcile file_storage/branding tests with #401 and #404 on main

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* fix(migrations): chain file_storage public column after #406's timestamp revision

Both branched from c7f2d9a41e83; repoint down_revision so main keeps a single
mainline head.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4

* feat(file_storage): own rate-limit bucket for anonymous public file GETs

With #408's shared limiter on main, public file reads shared the 120/minute
anonymous default; a page embedding many images would throttle itself.

Claude-Session: https://claude.ai/code/session_01M9neheZZEe3sVpDi2S3zT4
@antosubash
antosubash deleted the feat/342-migration-portability branch October 8, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrations autogenerated on SQLite are not portable to Postgres, and every check reports them clean

1 participant