Skip to content

[7/10] Build the GDPR export from the schema - #359

Merged
hturnbull93 merged 6 commits into
mainfrom
gdpr-export-introspect
Oct 24, 2026
Merged

hturnbull93 merged 6 commits into
mainfrom
gdpr-export-introspect

Conversation

@hturnbull93

@hturnbull93 hturnbull93 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Screenshots: https://fedora.taila91c66.ts.net:8444/docs/20261021-134938-pr-359-gdpr-export-built-from-the-schema.html

The data export now finds what to include from the Prisma schema, so a new model that holds someone's data is picked up without a code change, and the build fails until someone decides whether it belongs.

Data / export logic (lib/data-export.ts)

  • Reads the runtime data model and finds every link to a person: each relation field pointing at Volunteer (authorId, assigneeId, reporterId, resolvedById, fromVolunteerId…), any bare volunteerId-style column with no relation (DeletionRequest.volunteerId), and any column outside the account holding an email address (BugReport.reporterEmail, AdminInvite.email…) or its SHA-256 (RejectedApplication.emailHash, AnonymisedEmail.emailHash).
  • Email and hash links match only once the account's address is confirmed: anyone can change their account to an unused address, so an unconfirmed one would otherwise export someone else's rows. Email columns match case-insensitively; hash columns match the hash of the lower-cased, trimmed address, as the app writes them.
  • Relations to the outreach tool's ExperimentalOutreachParticipant are links too, matched through the participant with the person's confirmed address: its sessions and login tokens, and the journalists it claimed or contacted. A journalist is exported as the claim alone (EXPORTED_COLUMNS: id, who, when, which template), since their name, address and notes are the journalist's data.
  • Accounts merged into yours (Volunteer.mergedInto) are exported, secrets stripped: both sides confirmed the merge, and the old account can no longer sign in to export itself. duplicateOf is left out, as it is an admin's guess until a merge is confirmed.
  • An export runs its queries four at a time, so one export never holds the whole connection pool.
  • Each link is either in EXPORTED_LINKS or in OMITTED_LINKS with a reason.
  • SECRET_FIELDS is the deny-list: password hash, the old auth token, session and API token hashes, reset/verification/merge/invite tokens, and email hashes. Only those columns are stripped. A column whose name looks secret (token/hash/password/secret/key/code/otp) fails the guard until it is named in SECRET_FIELDS or in PLAIN_FIELDS (looks secret, isn't), so a column like passwordChangedAt is a decision rather than a silent drop.
  • If the Prisma client stops exposing _runtimeDataModel or a model delegate, the export throws an error naming it.

API / export shape (privacy.export)

  • exportedAt and skills are unchanged.
  • profile is a superset of the old one: every column of the account except the secrets (so application notes, chapter, preferences and so on are now included), plus localGroup, nationalChapter, isSuperAdmin and hasPassword. Skills are in the top-level skills only.
  • projects, interests, messagesSent and messagesReceived are replaced by records: { Model: { link: rows[] } }. For example, records.Message.from is what you sent, records.Message.to what you received, and records.WorkItem.assignee the work you hold. Rows keep their real column names (relatedWorkItemId, not relatedProjectId), and work items cover tasks as well as projects.

UI

  • The export description on /privacy and in Settings now lists what the export includes.

Tests

  • lib/data-export.test.ts makes one row for every exported link, for the person and for someone else, and checks each link returns the first and not the second. A fixture list that doesn't match EXPORTED_LINKS fails, so a newly exported link needs a fixture. Breaking the email match or the id filter on purpose fails it with a line per link.
  • The same file checks the export holds another person's id where a row involves them, but never their name or email address; that an unconfirmed address matches no email or hash link (removing the check fails it on each one); and that a missing data model or delegate throws.
  • The guard also fails on a new email or email-hash column nobody has classified.
  • lib/data-export.test.ts: the guard passes against the real schema. Synthetic schema changes show it catches a new model, a new relation on an existing model, a bare volunteer column, an undeclared secret-looking column, and stale classifications or secrets. Another test checks that every exported link appears in the output and that only the caller's rows are read, without token hashes.
  • privacy.export router tests cover admin notes about you (included) and by you (left out), sessions, API tokens and deletion requests without secrets, and a profile with no password hash or auth token.
  • The e2e test 18-gdpr-privacy now asserts against the new shape.

Decisions

  • How the schema is read: the prisma-client generator (Prisma 6.19) doesn't export Prisma.dmmf. The data model is read from the client's _runtimeDataModel, which is typed in Prisma's runtime but internal. If a Prisma upgrade moves it, the guard test fails loudly rather than the export going quiet.
  • What is included: rows about you, made by you, or sent to you, including admin notes about you (Article 15).
  • What is left out: decisions you took as staff about other people (reviews, appointments, resolutions, audit actions), notes and endorsements you wrote about others, invites sent to other people's addresses, and other accounts that point at yours. Each has its reason in OMITTED_LINKS.
  • Tokens: session, API token, password-reset and verification rows are exported without their token or hash, so you can see that they exist.
  • Shape change: the shape changed rather than keeping duplicate legacy keys. Only the download button reads this endpoint.
  • Matched by email: invites to the person's address (accepted or not), bug reports filed with it, rejected applications (with the admin's and applicant's notes) and anonymised-email records for it, and their experimental outreach participant and journalist rows are exported. DeletionRequest.volunteerEmail is matched too, since volunteerId has no foreign key and a request from an earlier account outlives it. Email hashes are stripped like other secrets.
  • Not followed: rows that point at an exported row rather than at the person, such as the review requests on a project they created. The guard covers links to a volunteer or an outreach participant only.
  • Includes others' ids, not their details: a row is exported whole, so a received message carries the sender's id, and a project its other people's ids and review notes.
  • Secrets by name only: the guard reads column names; a secret stored inside a Json column would not be caught.
  • Unbounded: each link is one query with no limit, and the whole export is built in memory, as the old export was.
  • No maintenance checklist: there is no maintenance-checklist.md anywhere in the repo or its history, so there was no checklist item to remove.

No data migration.

Closes #118

Also in this PR: nine router handlers that didn't await notifyUser now do. They returned before the notification row existed, and projects-interests' declined-invite test failed once under check-all waiting for it. notifyUser only awaits the row, so the email still runs on its own. It no longer throws when looking up who to email fails (logged instead), so an awaited call can't turn a saved change into an error; bugReports' own .catch on it is gone.

hashEmail() in lib/auth.ts is now the one SHA-256 of a normalised address, used by registration, application review, anonymisation and the export.

🤖 Generated with Claude Code

@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 13956ab to 2b2fc0e Compare October 21, 2026 13:15
@hturnbull93 hturnbull93 changed the title Build the GDPR export from the schema [8/12] Build the GDPR export from the schema Oct 21, 2026
@hturnbull93 hturnbull93 changed the title [8/12] Build the GDPR export from the schema [7/10] Build the GDPR export from the schema Oct 22, 2026
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 2b2fc0e to 03bd6a0 Compare October 22, 2026 17:17
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 03bd6a0 to 27155df Compare October 22, 2026 17:36
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 27155df to 25d2199 Compare October 23, 2026 09:02
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch 2 times, most recently from a53a47d to 52f6621 Compare October 23, 2026 09:17
@hturnbull93
hturnbull93 force-pushed the pwa branch 2 times, most recently from 19a82f2 to ee4a0bc Compare October 23, 2026 15:09
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 52f6621 to b7f5e60 Compare October 23, 2026 15:09
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from b7f5e60 to a80b95d Compare October 23, 2026 16:55
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from a80b95d to 1a07121 Compare October 23, 2026 18:05
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 1a07121 to 229873c Compare October 23, 2026 23:10
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from 229873c to b26831a Compare October 24, 2026 13:34
The export walks Prisma's runtime data model for every relation to
Volunteer (and any bare volunteerId-style column), and exports the rows
of each link classified as exported, with declared secrets stripped.
A unit test fails on any link or secret-looking column nobody has
classified, so a new model cannot be silently included or skipped.

The profile is now every column of the account but the secrets, and
the hand-picked projects/interests/messages lists give way to
`records`, keyed by model and link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hturnbull93
hturnbull93 force-pushed the gdpr-export-introspect branch from b26831a to 0caeda5 Compare October 24, 2026 15:16
@hturnbull93
hturnbull93 changed the base branch from pwa to main October 24, 2026 15:16
@hturnbull93 hturnbull93 reopened this Oct 24, 2026
@railway-app

railway-app Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

🚅 Deployed to the catalyse-pr-359 environment in PauseAI Catalyse

Service Status Web Updated
web ❌ Build Failed (View Logs) Web Oct 24, 2026 at 7:14 pm UTC

hturnbull93 and others added 2 commits October 24, 2026 16:36
Columns holding an email address or its SHA-256 are links too, and the
guard now requires them to be classified. Invites to the person's address,
bug reports filed with it, rejected applications and anonymised-email
records, and the experimental outreach participant and journalist rows are
exported; DeletionRequest.volunteerEmail is left out as the same rows as
DeletionRequest.volunteerId. Hashes are stripped like other secrets.

A new test makes one row for every exported link, for the person and for
someone else, and checks each link returns the first and not the second; a
list of fixtures that doesn't match EXPORTED_LINKS fails. Another checks
the export holds a second person's id where a row involves them, but never
their name or address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A handler that didn't await it returned before the notification row
existed, so a test waiting for that row could time out under load
(projects-interests' declined-invite test failed once in check-all).
notifyUser only awaits the row; its email still runs on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hturnbull93 and others added 2 commits October 24, 2026 18:45
Anyone can change their account to an unused address, so an unconfirmed
address no longer pulls in rows matched by email or its hash; until now it
would have exported someone else's rejected-application notes, bug reports
and invites. Also from review:

- Follow the outreach participant's own sessions, login tokens and the
  journalists it claimed or contacted, as links like any other.
- Strip only the columns SECRET_FIELDS names. A secret-looking column must
  be declared in SECRET_FIELDS or PLAIN_FIELDS, so a column like
  passwordChangedAt is a decision rather than a silent drop.
- Run an export's queries four at a time, not all at once.
- Take the caller's account from the request instead of reading it again.
- Return skills once, at the top level.
- Throw a clear error if the Prisma client stops exposing its data model
  or a delegate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Callers now await it after their change is saved, so a failed lookup of
who to email would have turned a saved change into an error. It is logged
instead, like a failed notification row, and bugReports' own catch is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A claimed or contacted journalist is exported as the claim alone (id,
  who, when, which template), not the journalist's name, address and
  notes, which are the journalist's data. EXPORTED_COLUMNS names the
  columns, and the schema guard fails if one disappears.
- Accounts merged into this one are exported (secrets stripped): both
  sides confirmed the merge, and the old account can no longer sign in
  to export itself. duplicateOf stays omitted: it is an admin's guess
  until a merge is confirmed.
- DeletionRequest.volunteerEmail is exported: volunteerId has no
  foreign key, so a request from an earlier account is found by its
  confirmed address.
- hashEmail() in lib/auth.ts is the one SHA-256 of a normalised address,
  used by registration, application review, anonymisation and the export.
- Columns named like key, code or otp count as secret-looking too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hturnbull93
hturnbull93 merged commit 0d490b0 into main Oct 24, 2026
3 of 4 checks passed

This branch had an error being deployed

1 failed deployment
PauseAI Catalyse / catalyse-pr-359 — ae22e667 Deployed Oct 24, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-introspecting GDPR export

1 participant