You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(e2e): prove Protect.js → Stack needs no re-encryption, across DynamoDB and Postgres - #1519
Customers on @cipherstash/protect@10.2.1 + @cipherstash/protect-dynamodb@6.0.1 want to move to Stack 1.x without re-encrypting their data. This PR adds tests that prove they can. Existing DynamoDB items keep decrypting and stay searchable by equality as stored, and an encrypted value can move between DynamoDB and Postgres without being re-encrypted. It also fixes the stash-dynamodb agent skill, which wrongly told customers to re-encrypt every item.
Background for reviewers new to this: an encrypted value has a ciphertext (the encrypted data) and, for searchable columns, a search hash (a keyed hash used for equality lookups). DynamoDB stores them as <attr>__source and <attr>__hmac. In Postgres, an EQL v3 column stores them as the c and hm fields of a JSON value. EQL is CipherStash's SQL extension for querying encrypted data.
Changes
No-re-encryption proof (DynamoDB)
New block in e2e/tests/protect-dynamodb-compat.e2e.test.ts. A set of items written by protect-dynamodb 6.0.1 is frozen and snapshotted. From then on only Stack 1.x runs: it decrypts every item (single and bulk), runs every equality query, and writes new items beside the old ones. At the end the stored items are checked byte-for-byte against the snapshot. It also confirms the one lookup a Stack app cannot make, equality on a boolean. Re-encrypting wouldn't help there, because Stack writes no boolean search hash either.
Cross-store proof (DynamoDB ↔ Postgres)
New suite e2e/tests/protect-crossover.e2e.test.ts. Moving a value between stores means copying those two strings; no key is used. Each test checks the ciphertext arrives unchanged, then that the receiving store can decrypt it and find it by equality:
protect-dynamodb 6.0.1 items → EQL v3 columns: the column types accept them, SQL eql_v3.eq queries find them, and Stack decrypts them. Copying DynamoDB → Postgres → DynamoDB returns the original item exactly.
Stack Postgres rows → DynamoDB items: Stack and protect-dynamodb 6.0.1 both decrypt them, and their search hash matches a direct DynamoDB write and protect-dynamodb's createSearchTerms.
Raw protect.encrypt output (EQL v2) → both stores.
legacyProtect / ok move to e2e/tests/helpers/legacy-protect.ts, shared by both suites.
Wiring
e2e/vitest.protect-crossover.config.ts + test:protect-crossover turbo task. The default e2e config excludes the new file.
tests.ymle2e-tests job: starts a Postgres with .github/actions/integration-db, runs the suite, tears it down. The suite installs EQL v3 through the real stash eql install.
New dev-only deps of the private @cipherstash/e2e: @cipherstash/test-kit (workspace) and postgres@^3.4.8, which is already in the lockfile (3.4.9) via @cipherstash/stack. No install scripts.
Skill
skills/stash-dynamodb/SKILL.md: re-encryption is optional. Keep property keys and DB column names, and map each v2 equality column to an Eq domain. The skill now lists the three things to plan for: boolean equality lookups, downcase filters, and legacy dataType('json') attributes. stash patch changeset included, since skills ship in the stash tarball.
Verification
pnpm --filter @cipherstash/e2e typecheck: passes.
biome check on e2e/ and turbo.json: no errors.
Neither suite has been run live yet. I had no CipherStash credentials or DATABASE_URL locally, so both files stopped at their configuration guard, as designed: they fail rather than skip without config. The CI run on this PR is the first live run.
pnpm run test:scripts (workflow meta-tests): 1402 passed, 2 failed. Both failures (cargo-lock-freshness, lint-no-eql-registry-pins) come from local .worktrees/ checkouts being picked up by the file scan, not from this change. CI is the clean check.
Related
Refs #1436 (the protect-dynamodb 6.0.1 compatibility suite this builds on).
Review notes
Start with the toEnvelope / toAttributes functions at the top of protect-crossover.e2e.test.ts. They are the whole migration surface.
The boolean check accepts either a refused encryptQuery or one that returns no search hash, because I couldn't run it to see which. Tighten it after the first green run.
Postgres data written by protect into EQL v2 columns is out of scope here. That path is covered by the EQL v2 → v4 migration work.
Summary by CodeRabbit
New Features
Existing encrypted DynamoDB items can be read and queried with Stack without re-encryption; re-encryption is optional.
Added end-to-end coverage for moving encrypted data between DynamoDB and Postgres, including round-trip reads and equality queries.
Documentation
Clarified migration behavior and compatibility considerations for Boolean lookups, HMAC values, downcase normalization, JSON fields, and grouped EQL v2 fields.
Chores
Added automated setup and cleanup for the cross-store test environment.
…tack 1.x
The existing compat suite showed protect-dynamodb 6.0.1 items read and
query under Stack 1.x, but its acceptance run re-encrypted them as a
backfill step, so it never showed the backfill could be skipped. A new
block freezes a protect-dynamodb corpus (single and bulk writes), moves
the app to Stack 1.x alone, decrypts every item, runs every equality
query, writes new items beside the old ones, and then asserts the stored
items are byte-identical to the snapshot.
The stash-dynamodb skill told customers to re-encrypt every item to move
a table to v3. That is not required, so the skill now says re-encryption
is optional and lists the three differences to plan for: boolean
equality lookups, downcase filters, and legacy dataType('json')
attributes.
The legacy client builder and the result unwrapper move to a shared
helper so the cross-store suite can reuse them.
An encrypted scalar is the same two strings in both stores: DynamoDB's
<attr>__source and <attr>__hmac are the c and hm fields of a Postgres
EQL v3 value. The rest of the Postgres value is the version and the
table and column names, none of it secret, so moving a value between
stores is a copy with no key involved.
The new suite pins that in both directions with real protect-dynamodb
6.0.1 and protect 10.2.1 payloads as well as Stack 1.x ones: the EQL v3
column types accept the copied values, the ciphertext arrives
byte-identical, Stack decrypts it, SQL equality queries and DynamoDB
search hashes still match, and protect-dynamodb 6.0.1 reads items copied
from Postgres.
It needs CS_* credentials and DATABASE_URL, so it runs from its own
config as test:protect-crossover, and the e2e-tests job now starts a
Postgres through integration-db for it.
No actionable comments were generated in the recent review.
Walkthrough
The changes add migration guidance for reading legacy Protect data without re-encryption. They also add end-to-end tests for legacy DynamoDB compatibility and encrypted-data transfer between DynamoDB and PostgreSQL, with dedicated test and workflow configuration.
Changes: Protect compatibility without re-encryption
Layer / File(s)
Summary
Legacy DynamoDB compatibility and migration guidance .changeset/stash-dynamodb-no-reencryption.md, skills/stash-dynamodb/SKILL.md, e2e/tests/helpers/legacy-protect.ts, e2e/tests/protect-dynamodb-compat.e2e.test.ts
The migration guidance describes EQL v2 read options and compatibility differences. Shared helpers support tests that read and query legacy DynamoDB items, check mixed legacy and Stack-written data, and verify that reads and queries do not change stored items.
Cross-store test and workflow e2e/tests/protect-crossover.e2e.test.ts, e2e/package.json, e2e/vitest.protect-crossover.config.ts, e2e/vitest.config.ts, e2e/tsconfig.json, turbo.json, .github/workflows/tests.yml
The new end-to-end suite tests ciphertext and HMAC transfer, decryption, equality searches, and round trips across DynamoDB and PostgreSQL. Dedicated Vitest and Turbo settings and workflow steps configure the suite and its database.
cipherstash/stack#1478: The pull request adds end-to-end evidence for one released-version combination that transfers data to Stack without re-encryption.
cipherstash/stack#1490: The pull request provides executable cross-version and database evidence for migration from legacy Protect versions to Stack without re-encryption.
Possibly related PRs
cipherstash/stack#1436: This pull request extends the Protect and protect-dynamodb compatibility coverage with no-re-encryption tests and shared legacy-client helpers.
Suggested reviewers:auxesis, coderdan
Merge Risk: Minimal · up to fdb5e
This PR adds compatibility tests, CI wiring and migration guidance. It does not change library runtime behavior, and no actionable merge-blocking issue remains.
Pre-merge checks: 5
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title "test(e2e): prove Protect.js → Stack needs no re-encryption, across DynamoDB and Postgres" matches the main change. The PR adds e2e tests that check whether data written by Protect.js 10.2.1…
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title "test(e2e): prove Protect.js → Stack needs no re-encryption, across DynamoDB and Postgres" matches the main change. The PR adds e2e tests that check whether data written by Protect.js 10.2.1 and protect-dynamodb 6.0.1 can move to Stack 1.x without re-encryption. The tests cover DynamoDB (compatibility suite) and Postgres (cross-store suite). The changeset, CI, Vitest and Turbo config, and skill docs support this same goal. The title is 88 characters, slightly long, but it is specific and accurate.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 5 files. (6 skipped: 6 unsupported.)
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Reviewing files that changed from the base of the PR and between 6abc330 and fdb5e0c.
Files ignored due to path filters: 1
pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
Files selected for processing: 11
.changeset/stash-dynamodb-no-reencryption.md
.github/workflows/tests.yml
e2e/package.json
e2e/tests/helpers/legacy-protect.ts
e2e/tests/protect-crossover.e2e.test.ts
e2e/tests/protect-dynamodb-compat.e2e.test.ts
e2e/tsconfig.json
e2e/vitest.config.ts
e2e/vitest.protect-crossover.config.ts
skills/stash-dynamodb/SKILL.md
turbo.json
Included review availability: This review used your included allowance. 4 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.
The boolean-equality test accepted any encryptQuery failure, so a service
outage passed it. It now asserts the "no indexes configured" refusal
Stack gives a storage-only column, after a control query on the same
client succeeds. The snapshot test claimed more than it could show:
nothing in its block writes to the table, so it is renamed and its
comments say it pins only that reads and queries left the stored items
unchanged.
The cross-store suite sorted results with ORDER BY, which follows the
database collation, so it now sorts in JavaScript. The protect-dynamodb
decrypt check uses toEqual like every other decrypt check, and the
raw-payload block goes through the same two copy functions as the rest
of the file instead of a third, slightly different one.
In CI the cross-store suite was skipped whenever the protect-dynamodb
step failed. Both new steps now run on !cancelled().
The stash-dynamodb skill said Stack reads protect-dynamodb items as they
are while the paragraph above still required { storedEqlVersion: 2 }.
It now says a top-level scalar item reads either way and the option is
needed for a grouped v2 field declared by its bare leaf name. The advice
to REMOVE a stale __hmac on in-place updates now covers every domain
that writes no __hmac, not only Boolean.
A v2 downcase column mapped to TextEq is hashed case-sensitively by
Stack, so a mixed-case write gets an __hmac that a lowercase query never
matches. The skill told customers to lowercase only the query value. It
now says to lowercase every write too, notes that this drops the
original casing from the ciphertext, and suggests a separate attribute
for the original when the app needs it. The write-path behaviour is
already pinned by the 'downcase on the write path' test.
The reason will be displayed to describe this comment to others. Learn more.
Review attestation
Reviewed fdb5e0c · posted from Claude Code · claude-sonnet-5-5
Sources: code-review (medium) · claude-sonnet-5-5; review-cross-check
Skills: code-review, review-cross-check, review-post
No findings.
Seven candidate observations were checked against fdb5e0c and none held up as a defect (stale-advice, test-strength, workflow-gating and dependency-pin concerns were refuted or non-defects). No human review feedback existed on the PR at the time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cipherstash-agent
left a comment
Summary
Customers on
@cipherstash/protect@10.2.1+@cipherstash/protect-dynamodb@6.0.1want to move to Stack 1.x without re-encrypting their data. This PR adds tests that prove they can. Existing DynamoDB items keep decrypting and stay searchable by equality as stored, and an encrypted value can move between DynamoDB and Postgres without being re-encrypted. It also fixes thestash-dynamodbagent skill, which wrongly told customers to re-encrypt every item.Background for reviewers new to this: an encrypted value has a ciphertext (the encrypted data) and, for searchable columns, a search hash (a keyed hash used for equality lookups). DynamoDB stores them as
<attr>__sourceand<attr>__hmac. In Postgres, an EQL v3 column stores them as thecandhmfields of a JSON value. EQL is CipherStash's SQL extension for querying encrypted data.Changes
No-re-encryption proof (DynamoDB)
e2e/tests/protect-dynamodb-compat.e2e.test.ts. A set of items written by protect-dynamodb 6.0.1 is frozen and snapshotted. From then on only Stack 1.x runs: it decrypts every item (single and bulk), runs every equality query, and writes new items beside the old ones. At the end the stored items are checked byte-for-byte against the snapshot. It also confirms the one lookup a Stack app cannot make, equality on a boolean. Re-encrypting wouldn't help there, because Stack writes no boolean search hash either.Cross-store proof (DynamoDB ↔ Postgres)
e2e/tests/protect-crossover.e2e.test.ts. Moving a value between stores means copying those two strings; no key is used. Each test checks the ciphertext arrives unchanged, then that the receiving store can decrypt it and find it by equality:eql_v3.eqqueries find them, and Stack decrypts them. Copying DynamoDB → Postgres → DynamoDB returns the original item exactly.createSearchTerms.protect.encryptoutput (EQL v2) → both stores.legacyProtect/okmove toe2e/tests/helpers/legacy-protect.ts, shared by both suites.Wiring
e2e/vitest.protect-crossover.config.ts+test:protect-crossoverturbo task. The default e2e config excludes the new file.tests.ymle2e-testsjob: starts a Postgres with.github/actions/integration-db, runs the suite, tears it down. The suite installs EQL v3 through the realstash eql install.@cipherstash/e2e:@cipherstash/test-kit(workspace) andpostgres@^3.4.8, which is already in the lockfile (3.4.9) via@cipherstash/stack. No install scripts.Skill
skills/stash-dynamodb/SKILL.md: re-encryption is optional. Keep property keys and DB column names, and map each v2 equality column to anEqdomain. The skill now lists the three things to plan for: boolean equality lookups,downcasefilters, and legacydataType('json')attributes.stashpatch changeset included, since skills ship in thestashtarball.Verification
pnpm --filter @cipherstash/e2e typecheck: passes.biome checkone2e/andturbo.json: no errors.DATABASE_URLlocally, so both files stopped at their configuration guard, as designed: they fail rather than skip without config. The CI run on this PR is the first live run.pnpm run test:scripts(workflow meta-tests): 1402 passed, 2 failed. Both failures (cargo-lock-freshness,lint-no-eql-registry-pins) come from local.worktrees/checkouts being picked up by the file scan, not from this change. CI is the clean check.Related
Refs #1436 (the protect-dynamodb 6.0.1 compatibility suite this builds on).
Review notes
toEnvelope/toAttributesfunctions at the top ofprotect-crossover.e2e.test.ts. They are the whole migration surface.encryptQueryor one that returns no search hash, because I couldn't run it to see which. Tighten it after the first green run.protectinto EQL v2 columns is out of scope here. That path is covered by the EQL v2 → v4 migration work.Summary by CodeRabbit
downcasenormalization, JSON fields, and grouped EQL v2 fields.