Skip to content

feat(uploads): dataset upload with drafts, review workflow, and resumable raw ZIP uploads; refs #147 - #148

Merged
elainefan331 merged 45 commits into
stagingfrom
dev-fan
Oct 6, 2026
Merged

elainefan331 merged 45 commits into
stagingfrom
dev-fan

Conversation

@elainefan331

Copy link
Copy Markdown
Collaborator

Summary

Adds a login-based dataset upload system (refs #147):

  • JSON upload (/upload): logged-in users upload a dataset JSON. It is stored in the
    sandbox CouchDB (sandbox1d) as a draft and tracked in Postgres.
  • Upload detail page (/uploads/:internalId): metadata, publishing settings
    (target database + preferred dataset ID), Submit / Withdraw,
    review discussion thread, and "View submitted JSON" via an owner-only backend proxy
    (the sandbox URL is never sent to the browser).
  • Dashboard → Uploads tab: list of the user's uploads with status, raw ZIP chip, and Open button.
  • Raw data (ZIP) upload: resumable uploads (tus) from the browser directly to a new
    storage service on Zodiac, with progress, pause/resume, resume after a page reload,
    cancel, and server-side verification (size, ZIP check, SHA-256).

What's new

Backend

  • migrations: dataset_registry, submissions, submission_comments, requested_db, raw_objects, submission_raw_files, raw_upload_attempts
  • Routes under /api/v1/uploads (create/update, list, detail, document proxy, submit,
    withdraw, settings, comments, raw upload sessions)
  • Ed25519-signed short-lived upload tokens; HMAC-signed service calls between the backend and storage
  • /api/v1/internal/storage/... event endpoint (HMAC only) + a reconciliation job that
    finishes uploads by pulling if a push is missed

storage-api/

  • tus server with token checks and disk-space checks, a crash-safe outbox,
    and HMAC-protected /internal endpoints

Frontend

  • UploadPage, UploadDetailPage, UploadsTab, PublishSettingsDialog,
    RawDataUploader (tus-js-client)

@elainefan331
elainefan331 merged commit f41bb28 into staging Oct 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant