Repository navigation
[7/10] Build the GDPR export from the schema - #359
Merged
Merged
Conversation
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 21, 2026 13:15
13956ab to
2b2fc0e
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 22, 2026 17:17
2b2fc0e to
03bd6a0
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 22, 2026 17:36
03bd6a0 to
27155df
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 23, 2026 09:02
27155df to
25d2199
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
2 times, most recently
from
October 23, 2026 09:17
a53a47d to
52f6621
Compare
hturnbull93
force-pushed
the
pwa
branch
2 times, most recently
from
October 23, 2026 15:09
19a82f2 to
ee4a0bc
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 23, 2026 15:09
52f6621 to
b7f5e60
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 23, 2026 16:55
b7f5e60 to
a80b95d
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 23, 2026 18:05
a80b95d to
1a07121
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 23, 2026 23:10
1a07121 to
229873c
Compare
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 24, 2026 13:34
229873c to
b26831a
Compare
The export walks Prisma's runtime data model for every relation to Volunteer (and any bare volunteerId-style column), and exports the rows of each link classified as exported, with declared secrets stripped. A unit test fails on any link or secret-looking column nobody has classified, so a new model cannot be silently included or skipped. The profile is now every column of the account but the secrets, and the hand-picked projects/interests/messages lists give way to `records`, keyed by model and link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hturnbull93
force-pushed
the
gdpr-export-introspect
branch
from
October 24, 2026 15:16
b26831a to
0caeda5
Compare
|
🚅 Deployed to the catalyse-pr-359 environment in PauseAI Catalyse
|
Columns holding an email address or its SHA-256 are links too, and the guard now requires them to be classified. Invites to the person's address, bug reports filed with it, rejected applications and anonymised-email records, and the experimental outreach participant and journalist rows are exported; DeletionRequest.volunteerEmail is left out as the same rows as DeletionRequest.volunteerId. Hashes are stripped like other secrets. A new test makes one row for every exported link, for the person and for someone else, and checks each link returns the first and not the second; a list of fixtures that doesn't match EXPORTED_LINKS fails. Another checks the export holds a second person's id where a row involves them, but never their name or address. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A handler that didn't await it returned before the notification row existed, so a test waiting for that row could time out under load (projects-interests' declined-invite test failed once in check-all). notifyUser only awaits the row; its email still runs on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anyone can change their account to an unused address, so an unconfirmed address no longer pulls in rows matched by email or its hash; until now it would have exported someone else's rejected-application notes, bug reports and invites. Also from review: - Follow the outreach participant's own sessions, login tokens and the journalists it claimed or contacted, as links like any other. - Strip only the columns SECRET_FIELDS names. A secret-looking column must be declared in SECRET_FIELDS or PLAIN_FIELDS, so a column like passwordChangedAt is a decision rather than a silent drop. - Run an export's queries four at a time, not all at once. - Take the caller's account from the request instead of reading it again. - Return skills once, at the top level. - Throw a clear error if the Prisma client stops exposing its data model or a delegate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Callers now await it after their change is saved, so a failed lookup of who to email would have turned a saved change into an error. It is logged instead, like a failed notification row, and bugReports' own catch is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A claimed or contacted journalist is exported as the claim alone (id, who, when, which template), not the journalist's name, address and notes, which are the journalist's data. EXPORTED_COLUMNS names the columns, and the schema guard fails if one disappears. - Accounts merged into this one are exported (secrets stripped): both sides confirmed the merge, and the old account can no longer sign in to export itself. duplicateOf stays omitted: it is an admin's guess until a merge is confirmed. - DeletionRequest.volunteerEmail is exported: volunteerId has no foreign key, so a request from an earlier account is found by its confirmed address. - hashEmail() in lib/auth.ts is the one SHA-256 of a normalised address, used by registration, application review, anonymisation and the export. - Columns named like key, code or otp count as secret-looking too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screenshots: https://fedora.taila91c66.ts.net:8444/docs/20261021-134938-pr-359-gdpr-export-built-from-the-schema.html
The data export now finds what to include from the Prisma schema, so a new model that holds someone's data is picked up without a code change, and the build fails until someone decides whether it belongs.
Data / export logic (
lib/data-export.ts)Volunteer(authorId,assigneeId,reporterId,resolvedById,fromVolunteerId…), any barevolunteerId-style column with no relation (DeletionRequest.volunteerId), and any column outside the account holding an email address (BugReport.reporterEmail,AdminInvite.email…) or its SHA-256 (RejectedApplication.emailHash,AnonymisedEmail.emailHash).ExperimentalOutreachParticipantare links too, matched through the participant with the person's confirmed address: its sessions and login tokens, and the journalists it claimed or contacted. A journalist is exported as the claim alone (EXPORTED_COLUMNS: id, who, when, which template), since their name, address and notes are the journalist's data.Volunteer.mergedInto) are exported, secrets stripped: both sides confirmed the merge, and the old account can no longer sign in to export itself.duplicateOfis left out, as it is an admin's guess until a merge is confirmed.EXPORTED_LINKSor inOMITTED_LINKSwith a reason.SECRET_FIELDSis the deny-list: password hash, the old auth token, session and API token hashes, reset/verification/merge/invite tokens, and email hashes. Only those columns are stripped. A column whose name looks secret (token/hash/password/secret/key/code/otp) fails the guard until it is named inSECRET_FIELDSor inPLAIN_FIELDS(looks secret, isn't), so a column likepasswordChangedAtis a decision rather than a silent drop._runtimeDataModelor a model delegate, the export throws an error naming it.API / export shape (
privacy.export)exportedAtandskillsare unchanged.profileis a superset of the old one: every column of the account except the secrets (so application notes, chapter, preferences and so on are now included), pluslocalGroup,nationalChapter,isSuperAdminandhasPassword. Skills are in the top-levelskillsonly.projects,interests,messagesSentandmessagesReceivedare replaced byrecords: { Model: { link: rows[] } }. For example,records.Message.fromis what you sent,records.Message.towhat you received, andrecords.WorkItem.assigneethe work you hold. Rows keep their real column names (relatedWorkItemId, notrelatedProjectId), and work items cover tasks as well as projects.UI
Tests
lib/data-export.test.tsmakes one row for every exported link, for the person and for someone else, and checks each link returns the first and not the second. A fixture list that doesn't matchEXPORTED_LINKSfails, so a newly exported link needs a fixture. Breaking the email match or the id filter on purpose fails it with a line per link.lib/data-export.test.ts: the guard passes against the real schema. Synthetic schema changes show it catches a new model, a new relation on an existing model, a bare volunteer column, an undeclared secret-looking column, and stale classifications or secrets. Another test checks that every exported link appears in the output and that only the caller's rows are read, without token hashes.privacy.exportrouter tests cover admin notes about you (included) and by you (left out), sessions, API tokens and deletion requests without secrets, and a profile with no password hash or auth token.18-gdpr-privacynow asserts against the new shape.Decisions
prisma-clientgenerator (Prisma 6.19) doesn't exportPrisma.dmmf. The data model is read from the client's_runtimeDataModel, which is typed in Prisma's runtime but internal. If a Prisma upgrade moves it, the guard test fails loudly rather than the export going quiet.OMITTED_LINKS.DeletionRequest.volunteerEmailis matched too, sincevolunteerIdhas no foreign key and a request from an earlier account outlives it. Email hashes are stripped like other secrets.maintenance-checklist.mdanywhere in the repo or its history, so there was no checklist item to remove.No data migration.
Closes #118
Also in this PR: nine router handlers that didn't await
notifyUsernow do. They returned before the notification row existed, andprojects-interests' declined-invite test failed once undercheck-allwaiting for it.notifyUseronly awaits the row, so the email still runs on its own. It no longer throws when looking up who to email fails (logged instead), so an awaited call can't turn a saved change into an error;bugReports' own.catchon it is gone.hashEmail()inlib/auth.tsis now the one SHA-256 of a normalised address, used by registration, application review, anonymisation and the export.🤖 Generated with Claude Code