test(protect-ffi): migrate live EQL v3 data to v4 end to end by tobyhede · Pull Request #1496 · cipherstash/stack

tobyhede · 2026-11-02T01:18:52Z

Summary

Proves that data written by Stack 1.x can be moved to the new EQL v4 format and keeps answering the same database queries. EQL is the format CipherStash uses to store encrypted values in Postgres. v3 is what Stack 1.x writes today; v4 is what the new Stack Encrypt engine writes. The two use different ciphertext and search-term formats, so moving data means decrypting and re-encrypting every value. Until now no test did that.

First of two stacked PRs. The second covers EQL v2 (Protect.js) data, and is stacked on this one.

Changes

New live test: integration-tests/tests/migrate-v3-to-v4.test.ts (11 tests). It runs the migration a customer would run, in one table, against real ZeroKMS (the key service) and Postgres with EQL v3 and v4 installed together:

Write v3 rows with protect-ffi in v3 mode, the engine @cipherstash/stack 1.x encrypts through.
Add a v4 column beside each v3 column.
Backfill: decrypt each v3 value and re-encrypt it as v4.
Check that equality, ORE range, ORE and OPE ordering, and text-match queries return the same rows from the v3 and v4 columns, and match a plaintext oracle.
Check that every v4 value decrypts to the original plaintext.
Drop the v3 columns, rename the v4 ones into their place, and check that queries still work. The v4 values are encrypted under the logical column name (customers/email, not customers/email_v4), so the rename doesn't break them.

It also pins three mistakes:

casting a v3 payload into a v4 column fails Postgres's domain check;
changing v to 4 on a v3 payload gives a value Stack Encrypt refuses to open;
querying a v4 column with a v3 query term silently returns no rows.

v4 encoder: packages/eql/tests/encryption/examples/v4_codec.rs. No TypeScript SDK writes v4 yet, so the test sends plaintexts to a small Rust program that reads JSON on stdin and writes JSON on stdout. It encrypts, builds query terms and decrypts through eql_bindings::encryption::targets, the same by-name entry point the Go SDK uses, against live ZeroKMS. To build a cipher from environment credentials, the test crate turns on stack-encrypt's http feature, which adds lines to packages/eql/Cargo.lock. The crate's existing tests still use fake key sources. The TypeScript side is tests/v4-codec.ts.

Setup and CI

integration-tests/tasks.toml: eql:v4:install builds the v4 SQL bundle from the in-tree EQL source and installs it. eql:v4:codec builds the encoder. Both are part of mise run setup.
.github/workflows/integration-protect-ffi.yml: adds a Rust cache for the EQL workspace and path filters for the inputs those two tasks read. packages/eql/crates/** is deliberately absent: wasm-build-inputs-paths-filter.test.mjs forbids it here, and test-eql.yml already covers it.
scripts/__tests__/tests-workflow-gate.test.mjs: records why v4-codec.ts may read packages/eql/target. Only the ungated live workflow runs it.
AGENTS.md: the integration suite section now covers v4 and the encoder.

No changeset: only test code changed. The integration-tests package is private and the EQL test crate is publish = false.

Verification

Run locally against live ZeroKMS and the suite's Docker Postgres with EQL v2, v3 and v4 installed:

vitest run tests/migrate-v3-to-v4.test.ts tests/postgres-v3.test.ts tests/postgres.test.ts: 25/25 passed, including all 11 new tests.
pnpm run test:scripts: 85 files passed.
protect-ffi src/lintWiring.test.ts and src/integrationSuiteCi.test.ts: passed.
cargo test --locked -p eql-encryption-tests: passed. cargo clippy -D warnings on the example: clean. cargo fmt --check: clean.
mise run eql:v4:install run twice against the same database: both runs succeed.
pnpm run typecheck in integration-tests: clean. Biome: no new findings.

The CI run of integration-protect-ffi.yml on this PR is the first time the setup runs on a fresh runner.

Review notes

Start with the comment at the top of the test file, then v4_codec.rs.
Only scalar types are covered. v4 JSON uses a new tree format that the encoder doesn't handle, and SteVec documents can't be converted.
The relabel test shows that the v4 query term differs from the relabelled v3 one. It doesn't insert the relabelled payload into a v4 column.

Nothing showed that data written by Stack 1.x (EQL v3) can move to EQL v4 and keep answering the same queries. The two versions come from different producers (cipherstash-client and Stack Encrypt), use different ciphertext formats and derive search terms differently, so the migration has to decrypt and re-encrypt every value. No test did that. The new suite runs the customer-shaped migration in one table, against live ZeroKMS and Postgres with EQL v3 and v4 installed side by side. It writes v3 with protect-ffi (the engine Stack 1.x encrypts through), adds a v4 column beside each v3 one, backfills by decrypting and re-encrypting, then checks that equality, ORE range, ORE and OPE ordering, and match queries return the same rows from both columns. It also checks that every v4 value decrypts to the original plaintext, and that the v4 columns still answer after the v3 columns are dropped and the v4 ones renamed into their place. Three hazards are pinned: a v3 payload cast into a v4 column fails the domain CHECK, relabelling `v` to 4 gives a value Stack Encrypt refuses to open, and a v3 query operand against a v4 column silently matches nothing. No TypeScript SDK writes v4 yet, so the suite re-encrypts through a small Rust stdin/stdout codec, an example in the EQL encryption test crate, that dispatches by EQL type name through eql_bindings::encryption::targets over live ZeroKMS. That needs stack-encrypt's `http` feature in that crate. The suite's `mise run setup` now also builds the v4 bundle from the in-tree EQL source and builds the codec, and the CI workflow caches the EQL Rust build and watches the inputs those two steps read.

changeset-bot · 2026-11-02T01:18:56Z

⚠️ No Changeset found

Latest commit: 2cd436f

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-11-02T01:19:00Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI; Enforced overrides: Organization
Review profile: CHILL
Plan: Essentials
Run ID: afa3b62f-3371-4369-babb-fe69a02c83cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

Trigger review

cipherstash-agent

Review attestation

Reviewed 4968a46 · posted from Claude Code · model runtime-selected · effort medium
Sources: code-review (medium)
Skills: code-review, review-cross-check, review-post

Findings (3): 3 minor — 3 inline.

Seven further candidate findings were checked against head and not posted: one refuted (the CI path filter already includes packages/stack-auth/src/**), the rest left unposted as unverified or opinion (query-operand index choice, mise toolchain resolution in packages/eql, the stale "19 files" count in AGENTS.md, and the added http feature cost).

Generated by Claude Code

cipherstash-agent · 2026-11-02T22:38:58Z

+  }
+
+  beforeAll(async () => {
+    v3Client = await newClient({ encryptConfig, eqlVersion: 3 })


minor The cleanup is only the return value of beforeAll, so if requireEqlV4, encryptBulk or an INSERT throws after pg.connect(), pg.end() and the DROP TABLE never run. The open socket can keep the worker alive and the table is left behind. Move the teardown into afterAll (with pg.end() guarded), or wrap the setup in try/catch that cleans up.

_{Source: code-review · runtime-selected model}

Generated by Claude Code

cipherstash-agent · 2026-11-02T22:38:59Z

+      `EQL v4 codec not found at ${codecPath}. Build it with \`mise run eql:v4:codec\` from languages/typescript/packages/protect-ffi, or set EQL_V4_CODEC.`,
+    )
+  }
+  const result = spawnSync(codecPath, {


minor spawnSync has no timeout, and the codec authenticates and builds a cipher against live ZeroKMS. If that call stalls, the synchronous block cannot be interrupted by vitest's 30s testTimeout, so the job hangs until the CI job limit. Pass timeout (for example 60s) so a stall fails with an error.

_{Source: code-review · runtime-selected model}

Generated by Claude Code

cipherstash-agent · 2026-11-02T22:39:01Z

+        value: 'alan@example.com',
+      },
+    ])
+    expect(v4Query.hm).not.toEqual(relabelled.hm)


minor v4Query.hm is never checked to exist, so if the v4 operand names its equality term differently it is undefined and not.toEqual(relabelled.hm) passes whatever either producer derives. Assert typeof v4Query.hm === 'string' first.

_{Source: code-review · runtime-selected model}

Generated by Claude Code

tobyhede

Found one P2 issue in the workflow triggers.

Overall impact: Medium. This adds tests and CI coverage without changing production APIs, but the missing triggers weaken regression coverage for v4 implementation changes.

Reviewed the complete pinned comparison ec5fd0e → 4968a46: 10 files, +1,004/−16. Tests were not run as part of this review.

tobyhede · 2026-11-02T22:51:14Z

+      # builds `tests/encryption/examples/v4_codec.rs` over eql-bindings' v4
+      # targets, stack-encrypt and stack-kms. No published package carries any of
+      # it, so these paths are the only inputs that change what those suites run.
+      # `packages/eql/crates/**` (eql-codegen, eql-bindings' v4 module) is left


[P2] Trigger the migration suite when its v4 implementation changes

Changes confined to packages/eql/crates/eql-bindings/src/encryption/**, src/v4/**, or eql-codegen do not match this workflow’s filters, although those files implement the v4 codec and generate the SQL this new migration suite exercises. The cited test-eql.yml runs Rust/SQL suites but never runs integration-tests/tests/migrate-v3-to-v4.test.ts. Consequently, a change to v4 target dispatch can skip the new end-to-end migration coverage entirely. Expand both trigger lists to include the codec’s inputs and adjust the existing filter guard to account for this workflow’s new responsibilities.

…uite Run integration-protect-ffi.yml when the EQL v4 code the migration suite exercises changes: the eql-bindings v4, encryption and encoding modules the codec compiles, and the eql-codegen / eql-domains sources the v4 SQL bundle is rendered from. test-eql.yml runs the Rust and SQL suites on those files but never this suite, so a v4 change could merge with the end-to-end migration coverage skipped. The entries are derived in lib/eql-v4-suite-inputs.mjs and held by eql-v4-suite-triggers.test.mjs; the existing EQL crate path guard allows exactly those, and only in this workflow. Also, in the suite itself: - tear down in afterAll so a setup failure after connect() still closes the socket and drops the table; - give the codec spawnSync a 60s timeout, since vitest cannot interrupt a synchronous stall against live ZeroKMS; - assert both hm terms exist before comparing them, so the inequality cannot pass on undefined.

tobyhede mentioned this pull request Nov 2, 2026

test(protect-ffi): migrate Protect.js 10.2.1 EQL v2 data to v4 end to end #1497

Open

tobyhede marked this pull request as ready for review November 2, 2026 22:36

tobyhede requested a review from a team as a code owner November 2, 2026 22:36

tobyhede requested review from auxesis and coderdan November 2, 2026 22:37

cipherstash-agent reviewed Nov 2, 2026

View reviewed changes

tobyhede commented Nov 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

test(protect-ffi): migrate live EQL v3 data to v4 end to end - #1496

test(protect-ffi): migrate live EQL v3 data to v4 end to end#1496
tobyhede wants to merge 2 commits into
nextfrom
test/eql-v3-to-v4-migration

tobyhede commented Nov 2, 2026

Uh oh!

changeset-bot Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading

Review skipped

Uh oh!

cipherstash-agent left a comment

Uh oh!

cipherstash-agent Nov 2, 2026

Uh oh!

cipherstash-agent Nov 2, 2026

Uh oh!

cipherstash-agent Nov 2, 2026

Uh oh!

tobyhede left a comment

Uh oh!

tobyhede Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tobyhede commented Nov 2, 2026

Summary

Changes

Verification

Related

Review notes

Uh oh!

changeset-bot Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

coderabbitai Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

cipherstash-agent left a comment

Choose a reason for hiding this comment

Review attestation

Uh oh!

cipherstash-agent Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

cipherstash-agent Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

cipherstash-agent Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

tobyhede left a comment

Choose a reason for hiding this comment

Uh oh!

tobyhede Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot Bot commented Nov 2, 2026 •

edited

Loading

coderabbitai Bot commented Nov 2, 2026 •

edited

Loading