Skip to content

feat(appkit): apply committed database migrations on deploy start - #654

Draft
ditadi wants to merge 3 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy
Draft

ditadi wants to merge 3 commits into
mainfrom
stack/database-dev-flow/01-migrations-on-deploy

Conversation

@ditadi

@ditadi ditadi commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First implementation PR of the DatabasePlugin development flow stack. The design is #677. The plugin stays hidden from databricks apps init until the last PR of the stack.

What did you change, and why?

Why: A deployed app has to apply its committed migrations by itself, because App Builder apps have no human or CI to run them (design 001).

Change:

  • database() applies the drizzle migrations under config/database/migrations after the pool opens, before select 1 and the declared-schema check. It runs only when APPKIT_DB_MIGRATE_ON_START=true and the service principal's credentials are set. .env never carries the flag and databricks apps run-local sets no credentials, so local runs never migrate on start.
  • The app's service principal runs the migrations and owns the app schema on the deploy branch. On 42501 the error names the schema owner and the connecting role, and the two platform operations that change the owner: removing and re-adding the database resource, and recreating the app.
  • Each migration runs and is recorded in its own transaction, with the rows drizzle-kit writes (hash, created_at). An enum value added by one migration can be used by the next, and a failure keeps the migrations before it.
  • The committed journal is compared with the recorded rows. A migration older than the last recorded one that never ran, for example one merged from a parallel branch, fails startup and changes nothing. An edited migration, or a recorded row with no committed file, only logs a warning.
  • The history is read without a lock first, so a start with nothing pending changes nothing. With something pending, a per-schema advisory lock makes instances apply in turn. The wait is capped at 5 minutes and the log names the holder's pid and application_name. Each table lock waits at most 5s, with three tries per migration.
  • A branch named dev-… or session-… is skipped on start. A -pooler host is refused, because the session lock does not hold through a transaction-mode pooler.
  • When nothing is recorded and the first migration is 0000_baseline whose tables and columns all exist, the baseline is recorded as applied. If only part of it exists, startup fails and nothing changes.
  • Every failure names the step and the SQLSTATE or system error code. One summary line per start logs applied, adopted, edited, unknown, lock wait and duration. Clients only see Database setup failed.
  • Safe to deploy alone: nothing migrates until an app sets APPKIT_DB_MIGRATE_ON_START=true.

How do you know it works?

Testing:

  • apply.test.ts, with a recording pg client: per-migration transactions and the rows they record, the lock-free read, stranded, edited and unknown migrations, the capped lock wait, table-lock retries, the message for each failed step, connection cleanup, and baseline adoption.
  • history.test.ts: how committed and recorded migrations are matched.
  • migrate-on-start.test.ts: the opt-in flag, the developer-branch skip, the pooler refusal, the summary log, and error mapping.
  • apply.pg.test.ts runs against Postgres when APPKIT_TEST_PG_URL is set, and CI sets it with a postgres:17 service. It covers the enum case that fails with 55P04 in one transaction, a partial failure, a stranded migration, three instances starting together, the lock wait naming its holder, table-lock retries, the session reset, a schema owned by another role, and baseline adoption.
  • Verified live on a Lakebase project before this revision, with a throwaway deploy branch and app. The first deploy applied 0000, and the service principal owned the schema, the table and the journal. A missing .sql stopped startup and named the file. A schema owned by another role failed with 42501. This revision has not been run live yet.

A deployed DatabasePlugin app now applies the migrations committed under
config/database/migrations before its startup checks, as its own service
principal and under a per-schema advisory lock. Pending migrations run in
one transaction, and a table lock held for over 30s fails the migration
instead of stalling the table. A 0000_baseline whose tables already exist
is recorded, in its own transaction, instead of run. Any failure stops
startup, and the server log names it: the tags and SQLSTATE, or the
missing migration file. Local runs never migrate on start.

Signed-off-by: ditadi <victordperd@gmail.com>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+53 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB (+56 KB) 468 KB (+20 KB)
Type declarations 464 KB (+8.0 KB) 168 KB (+2.8 KB)
Source maps 2.6 MB (+107 KB) 872 KB (+35 KB)
Other 11 KB 3.7 KB
Total 4.3 MB (+171 KB) 1.5 MB (+58 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB (+3.9 KB) 2.5 KB 108 KB (+3.9 KB) external 343 KB (+12 KB)
./beta 107 KB (+8.6 KB) 483 B (-2 B) 108 KB (+8.6 KB) external 321 KB (+24 KB)
./testing 45 KB (+3.1 KB) 32 KB (+1 B) 77 KB (+3.1 KB) external 223 KB (+8.7 KB)
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB (+2.8 KB) 0 B 26 KB (+2.8 KB) external 74 KB (+7.9 KB)

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 87 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 177 B
./beta index.js lazy 114 B
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+501 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB (+235 B) 147 KB (+108 B)
Type declarations 267 KB (+983 B) 98 KB (+226 B)
Source maps 853 KB (+314 B) 286 KB (+143 B)
CSS 16 KB 3.2 KB
Total 1.5 MB (+1.5 KB) 534 KB (+477 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB (+12 B) 49 KB 481 KB (+12 B) 1.3 MB 177 KB (+41 B)
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 38272767525 -R databricks/appkit -n appkit-template-0.87.0-pr.a93ece9-stack-database-dev-flow-01-migrations-on-deploy-654 -D appkit-pr-654 \
  && unzip -o "appkit-pr-654/appkit-template-0.87.0-pr.a93ece9-stack-database-dev-flow-01-migrations-on-deploy-654.zip" -d "appkit-pr-654" \
  && databricks apps init --template "appkit-pr-654"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

- Run and record each pending migration in its own transaction. An enum value
  added by one migration can now be used by the next (55P04), and a failure
  keeps the migrations before it.
- Compare the committed journal with the recorded rows by hash and created_at.
  Fail closed on a migration older than the last recorded one; warn on edited
  migrations and on recorded rows with no committed file.
- Read the history without the lock first. Take the lock only when something
  is pending, then read again.
- Bound the wait for the migration lock to 5 minutes and name its holder. Set
  application_name and client_connection_check_interval. Lower lock_timeout to
  5s with three tries per migration.
- Migrate on start only with APPKIT_DB_MIGRATE_ON_START=true and service
  principal credentials. Skip dev- and session- branches; refuse a -pooler
  host.
- Name the failed step and the SQLSTATE or system code. On 42501, name the
  schema owner and the connecting role.
- Add real-Postgres tests gated on APPKIT_TEST_PG_URL; CI runs them against a
  postgres:17 service.

Signed-off-by: ditadi <victordperd@gmail.com>
Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant