Programming Question Snapshotting / Audit Trail

[adi-herwana-nus]

Summary

Editing a programming question today destroys the record of how its existing answers were graded, then
regrades every answer ever submitted against the new version. This PR keeps that record:

• The version an edit replaces is kept as a snapshot. Its test cases and template files move onto the
  snapshot instead of being deleted, so every past test result still points at the test case that produced
  it.
• Only the answers users actually see are regraded after an edit. Earlier attempts keep the grade they
  were given.
• Every grading run of a programming answer is kept. A regrade adds a run instead of overwriting the
  previous one.
• Old results are shown against the test cases they were graded with, with an alert saying the question
  has changed since.
• Overlapping edits apply one at a time, and only the latest edit's import applies. Two imports of the
  same question can no longer interleave, and an import that finishes after a later edit has committed is
  dropped rather than overwriting it.

This replaces the draft in #8022 (and its predecessor #7595), rebuilt on two changes that have since merged:
the test case payload split and TypeScript test case views (f93da269e), and instance admin assessment
permissions (9d461ccd7). Nothing below assumes familiarity with the earlier drafts.

It is also a prerequisite for the programming language upgrade feature, whose course-wide re-import was
judged too risky to run while every re-import destroyed grading history.

The problem

When a programming question's package is re-imported (a new package, or a change to its language, time
limit or memory limit), ProgrammingImportService#save! assigns question.test_cases = <new records>.
There is no diffing: every test case row is destroyed and recreated, even when nothing in it changed. Test
results are dependent: :destroy from their test case, so every per-test-case result of every answer to
that question goes with them. AnswersEvaluationService then regrades every non-attempting answer ever
submitted.

So after any edit:

• nobody can see which tests a past answer passed, only its aggregate grade;
• every earlier attempt is recomputed against a question its student never saw;
• each answer's previous grading run is detached and left orphaned in the database.

What changes, at a glance

	Before	After
Old test case rows on re-import	Destroyed	Moved to a snapshot of the previous version
Old test results	Destroyed by cascade	Untouched; test_case_id still resolves
Rows written per edit	O(test results) deletes	O(test cases) updates
Snapshot created	—	When the package is actually re-imported, or removed
Answers regraded after an edit	Every non-attempting answer ever	Per submission: the answers the edit page and statistics show
A regrade's grading run	Overwrites the answer's single run	Adds a run; earlier runs and their results stay attached
Old results in the UI	None (destroyed)	Shown against the test cases they were graded with, plus an alert
Switching a graded question to non-autograded	500 (foreign key violation)	Works; the autograded version is kept as a snapshot
Two imports of one question overlapping	Interleave; with snapshots, one version would be lost and an empty snapshot recorded	Applied one at a time under a row lock
An earlier edit's import finishing after a later edit	Overwrites the later edit's package	Superseded: changes nothing
A grading run during an edit	Can evaluate one version's package against another's test cases	Reads package and test cases as one version
Pushing to Codaveri after an import	Pushes the job's own package, which may be older	Pushes the question's current version, one push at a time

Commits

Each commit can be reviewed on its own.

1. refactor(auto-grading): pass the auto grading record through the grading services and job
answer.auto_grade! obtains the AutoGrading record (existing or new) and passes it explicitly through the
   job and every grading service, instead of each layer rediscovering it with answer.auto_grading. This is a
   no-op on its own, and is what makes "one run per grading" possible later. It changes the grading jobs'
   arguments (see Deployment).
2. feat(programming): create snapshots of programming questions on edit
   Snapshot creation and the snapshot columns migration. Also how imports and grading runs coexist:
   • the import's row lock and its check that it is still the latest;
   • the local grader reading its package and test cases as one version;
   • Codaveri runs attributed to the version Codaveri actually evaluated.
3. feat(programming): only regrade current/latest answers on question edit
   The narrowed regrade, and rendering old results against the test cases they were graded with.
4. feat(programming): make question snapshot rows read-only
   The immutability guard, the snapshot on package removal (which also fixes the 500 above), and specs
   showing snapshots are unreachable by URL.
5. feat(programming): preserve previous auto grading runs
has_many :auto_gradings, and the two migrations that swap the unique answer_id index for a non-unique
   one without locking.
6. feat(programming): handle concurrent imports / improve codaveri handling logic
   • Recording an edit's import job in the edit's own transaction, which is what makes "the latest edit's
     import" well defined.
   • The lock helper, also taken by package removal.
   • Codaveri-only edits no longer displacing a pending import.
   • Codaveri pushes, from imports and from Codaveri-only edits, going through one serialised path.
   • Failed enqueues recorded as errored.

Every commit also clears the .rubocop_todo.yml entries for the files it touches, where that was possible
without unrelated refactoring.

Design

What a snapshot is

A snapshot is a row in the programming question table with current_id pointing at the live question. It
has no parent course_assessment_questions row, so it is never part of an assessment, never listed,
and never reachable through any question query. It holds:

• the question's own columns as they were before the edit: language, limits, attempt limit and so on;
• the test cases and template files moved off the live question;
• its own reference to the previous package. Attachments are content-addressed by SHA-256, so this costs
  one reference row and no storage;
• superseded_at (when the import replaced it) and superseder_id (whose edit did). The original story is
  titled "Programming Question Audit Trail", and these are the minimum an audit trail needs.

Things a reviewer may wonder about:

• Why not dup? Under acts_as, dup duplicates the parent question too. Every edit would leave an
  orphaned parent row carrying the new title and maximum grade, visible to every global question query and
  not deleted with the snapshot. The snapshot row is inserted directly (insert!), which also skips the
  callbacks and validations meant for an editable question.
• Why the pre-edit values have to be captured at save time. The controller commits the new column
  values in the request; the import that creates the snapshot runs later, in a job. By then the row already
  holds the new values, so the snapshot would pair the new language with the old test cases, a combination
  that never existed. Programming#process_package (before_save) captures the values still in the database
  and passes them to ProgrammingImportJob as a new, optional argument. They include the editor, taken from
  User.stamper, because the job runs without a user.
• Why column_names, not attribute_names. Under acts_as the latter includes the parent question's
  columns.

Where the snapshot is taken, and why it matters

The snapshot is taken and the old test cases are moved inside ProgrammingImportService#save!, in the
same transaction that inserts the new test cases. Not in the controller, and not as a separate step.

The reason is a full-marks window. A programming question with no test cases is not auto-gradable, and
grading such a question awards full marks without evaluating anything. That check runs against the live
question at the moment of grading. If the controller moved the old test cases away when the edit was saved,
the live question would have none until the import job finished evaluating the new package, which takes
seconds to minutes and never finishes if the import fails. Every answer graded in that window would get full
marks, published under the System grader in autograded assessments. The narrowed regrade would not correct
earlier attempts graded in that window.

Today's code has no such window: replacing a saved association runs its deletes and inserts in one
transaction, so other connections see the old test cases or the new ones, never none. Doing the move in the
same transaction keeps that property. It also means a failed import leaves no snapshot and changes nothing,
exactly as today.

The move is question.test_cases.update_all(question_id: snapshot.id), which is O(test cases) and copies no
results. It must go through the association: update_all on an association resets it, whereas through
the model class a preloaded association still holds the moved rows, and the next assignment deletes them. A
spec with preloaded associations covers this.

A grading run in flight during an edit

Both programming graders spend seconds to minutes evaluating before they match results to test cases, so an
import can commit in the middle of a run.

Local evaluator. A run reads the question's test cases and its package in one REPEATABLE READ
transaction (pin_version): a single database snapshot. Every change to a question's package changes the
package and test cases together, in one transaction. So the run gets either the old version or the new one,
never one version's package with the other's test cases.

• Old version: its results attach to the old test cases, which the import has moved to a snapshot. The
  answer shows the "changes have been made" alert until the edit's regrade adds a run against the new
  version.
• New version: the results attach to the live test cases, with no alert.
• Package removed before the run starts: the run raises a clear error rather than failing on nil.

A spec commits an import from a second database connection between the two reads; without the shared
snapshot, the run evaluates the new package against the old test cases. The other two windows are covered
too:

• the results built but not yet saved, which today fails on the results' foreign key;
• the package still being evaluated.

Codaveri. Nothing local can be pinned, because the problem being evaluated lives on Codaveri and an import
replaces it. A pin wouldn't have survived anyway: the pre-evaluation sync's with_lock reloads the question,
which unloads its test cases. Instead, the run is attributed to the version Codaveri actually evaluated. Each
result names the test case it is for (its index is the local test case id pushed to Codaveri), and snapshots
keep every version's test cases. The run's counts and failure records come from that version. Results that
don't all belong to one version of this question raise a CodaveriError saying so; today the same situation
crashes with NoMethodError on nil.

Which version a run was graded against: derived, not stored

Earlier drafts stamped a question_snapshot_id on each grading run. That column would sit on a 72 GB
table, and it goes stale: a run stamped with the live question becomes wrong the moment an edit moves its
test cases to a snapshot. Keeping it correct meant a mass UPDATE on every edit.

Instead, the version is read from the run's own results: test_result → test_case → question_id is the live
question or one of its snapshots. A test case is never deleted on edit, and a snapshot never changes, so this
cannot go stale. It covers every grading path that writes test_case_id, including Koditsu's bulk insert,
and both columns involved are already indexed.

One legacy case: a run whose results were deleted by an edit made before this PR has a grading record but no
results, so it says nothing about its version. It is treated as "graded on an earlier version" and rendered
against the current test cases with no marks, as today, plus the alert. A completed run is never otherwise
empty, because the grader records a failure for every test case the report misses.

Which answers are regraded after an edit

Only answers whose grading is shown outside a submission's Past Answers. Per submission, these are:

• the newest answer that is not still being edited. This is what the submission edit page shows for an
  attempting submission (SubmissionsHelper#last_attempt), and the "last attempt" in assessment statistics;
• plus the submission's current answer, if it is not still being edited. This is what the edit page
  shows once the submission is no longer being attempted.

Normally these are the same answer. They differ in submissions from before d4294b5c7 (2021-11-25). Before
that commit, finalising marked the current answer submitted in place, so it could be older than attempts
submitted after it. Since then, finalising promotes the last submitted attempt to current, or finalises a
fresh copy. In a production-scale copy, about 3.0M submitted current answers (one per question per
submission) created before that change are older than a submitted attempt at the same question. So are 10,732
created after it, whose cause was not pinned down. By submission state, those are 10,537 published, 99
submitted, 92 attempting and 4 graded. For these, the edit page and the statistics show different answers, so
both are regraded.

Answers still being edited are never regraded. Autosave only saves the answer's files; it neither grades nor
changes created_at.

What the UI shows

Every view continues to render the current question, including title, description and highlighting
language. The one exception is the test case panel, which renders the test cases of the version the run
was graded against, so that its pass/fail marks, hints and evaluation check make sense. When that version is
not the live one, the panel shows an alert: "Changes have been made to the question after this answer was
graded." This applies on the submission edit page, in Past Answers and in assessment statistics.

Grading runs per answer

Answer now has_many :auto_gradings (oldest first). The existing has_one :auto_grading is kept as the
latest run, ordered newest first, so every existing reader keeps working unchanged. Runs are only
created through auto_gradings.

• Programming answers get a new run per grading. Other question types keep regrading their single run
  in place, as today: they have nothing worth comparing, and this keeps their regrades from growing a 35M-row
  table.
• While a regrade is in flight, or after it fails, the panel keeps showing the previous finished run's
  results, with the "being evaluated" or error status from the latest run. This matches what a re-evaluate
  shows today, where the old results stay attached until replaced. After an edit, it is an improvement: today
  the edit has already destroyed the old results. Without it, the panel would empty for the whole
  low-priority regrade after an edit.
• Unsubmitting an answer no longer destroys its runs. Its only caller (resubmit_programming, when an
  assessment's graded test case types change) finalises and regrades immediately, so it now just adds a run.
• The race guard in the old ensure_auto_grading! relied on the unique index and is gone with it. The worst
  case is that two concurrent first gradings of a non-programming answer each create a run, and the later one
  is used.

No comparison UI is built here. The data model now supports one.

Overlapping edits and imports

Package evaluation takes seconds to minutes, so imports of one question can overlap:

• two edits in quick succession;
• a save that retries a failed import;
• a mass re-import overlapping a staff edit.

Two problems follow.

1. Two imports saving at once lose a version. Both see the same live test cases and both create a
   snapshot. The second import's move waits on the first's row locks. Once the first commits, Postgres
   re-checks the second's condition, finds the rows already moved, and moves nothing. So the second
   snapshot is empty, and the second import's assignment of new test cases then destroys the first
   import's, with any results graded against them. A spec running two imports on separate connections
   reproduces exactly this when the lock below is removed.
2. The import that finishes last wins (already true today). Imports apply in the order they finish, not
   the order of the edits. An older edit's import finishing late overwrites the newer edit's package, while
   the question's settings stay the newer edit's.

The fix.

• One row lock serialises every writer of the package. The import's save, package removal and the
  Codaveri push all take the same SELECT … FOR UPDATE lock on the question's row first (lock_package!,
  or with_lock for the push). The import then re-reads everything under it: the existence check, the
  previous package and the test cases. The lock is held only for that short write, never during evaluation.
  The local grader never takes it; Codaveri grading takes it only for the sync it runs before evaluating.
• The import job is recorded in the edit's own transaction. An edit that needs an import writes its job
  id to import_job_id in the same transaction as its other changes, and enqueues the job only after commit.
  Before, the id was written after commit and after enqueueing. A job could then start before its own id was
  recorded, and two concurrent edits could record their ids in the opposite order to their commits. Edits
  that schedule an import all update the question's row, so Postgres orders them, and the recorded job is
  always the latest committed edit's.
• An import applies only while the question still records its job. It checks once before evaluating, to
  skip a package that won't be applied, and again under the lock, which is the check that decides. A
  superseded import changes nothing, queues no regrade, pushes nothing, and its job completes normally.
  Removing the package clears the recorded job, so it supersedes any import still pending.
• Codaveri-only edits (turning Codaveri or live feedback on) also record their sync job on
  import_job_id, for the edit page to follow. They now do so only when no import is pending; otherwise
  they would make that import look superseded.
• A job that cannot be enqueued is recorded as errored, so the next save retries it, rather than the
  question waiting forever on an import that was never queued.

What is guaranteed. The package, test cases and template files always equal exactly one applied change:
the original state, one import's output, or a package removal, never a mix. History has no gaps: each import
snapshots the version it actually replaced. Once a later edit has committed, no earlier import can apply. An
earlier import that finishes before the later edit commits does apply, and is kept as a snapshot when the
later one applies.

Whether the final state matches the latest edit depends on that edit's import:

Latest edit's import	Final package and test cases	Question settings (language, limits)
Succeeds	The latest edit's	The latest edit's: consistent
Fails (invalid package, evaluator error, couldn't be enqueued)	The last import that applied, or the original	The latest edit's: mismatched, as a failed import is today. Marked errored, so the next save retries
Never runs (job lost while still submitted)	As above	As above, with no automatic retry

A snapshot pairs the package that was live with the settings that were live just before it was replaced.
So if a time-limit edit's import was superseded, the snapshot shows the new limit with the old package.
That's accurate: it is what grading used in that window.

Keeping Codaveri in step

Codaveri holds its own copy of each Codaveri question, pushed from the local one.

• An import marks the question unsynced in its own transaction.
• Every push, whether after an import, from a Codaveri-only edit or on demand before grading, goes through
  safe_create_or_update_codaveri_question. That takes the row lock and sends the question's current
  package and test cases. Before, the import job and CodaveriImportJob pushed the package they were queued
  with, so a late push could leave Codaveri on an older version than the local question.
• After the last push, Codaveri matches the local question. Between an import committing and its push,
  Codaveri lags behind; gradings in that window are attributed to whichever version Codaveri evaluated
  (above).
• That path used to run the push inside the lock's transaction, so when Codaveri rejected a push, the
  failure status written to the question was rolled back with it. It now re-raises after the transaction
  commits, so the status is kept, as the import job's direct push always kept it.

Snapshots are read-only and unreachable

• ProgrammingSnapshotReadOnlyConcern raises ActiveRecord::ReadOnlyRecord on saving a snapshot, or a test
  case or template file that belongs to one, including moving one onto or off a snapshot. It runs before
  validation, so the refusal is explicit rather than an unrelated validation error. Destroying is allowed, so
  that deleting a question deletes its snapshots. Callbackless writes (insert!, update_all) bypass it
  deliberately: they are how snapshots are created.
• Every ProgrammingController action loads its question through @assessment.programming_questions, which
  requires the parent question row a snapshot lacks. So the edit URL, and every other action, return 404
  for a snapshot id. If that loading ever changed, the ability check would still refuse (AccessDenied). A
  read-only snapshot view is left for the comparison UI.

Lifecycle

• Duplicating an assessment or course copies only the live question's test cases. Snapshots are not
  reachable from an assessment, so they are not duplicated.
• Deleting a question deletes its snapshots (has_many :snapshots, dependent: :destroy), with their test
  cases and package references.
• Deleting an assessment does not delete its questions, as today, so their snapshots remain too.
• Removing a package (switching an online-editor question to non-autograded) snapshots first, through the
  new Programming#remove_package. This also fixes a bug: test_cases.clear issues a raw DELETE that
  skips callbacks, which the results' foreign key rejects, so the switch failed with a 500 once any answer had
  been graded.
• The language migration rake task (db:migrate_programming_question_languages) now changes live questions
  only, so snapshots keep the language they were graded under.

Smaller fixes along the way

• validate_language_enabled was defined after Programming's closing end, which made it a private method
  on Object. It is now a private method of the class, with model specs.
• The acts_as gem overrides exists? with an inner join to the parent question row, so
  Programming.exists?(snapshot_id) is always false, and snapshots.empty? on an unloaded association is
  always true. Specs that relied on these were passing vacuously; they now use ids/pick, and the guard
  avoids exists?.

Deployment

Migrations

	Migration	What it does	Locks
M1	20261002000000_add_snapshot_columns_…	Adds current_id, superseded_at and superseder_id to the 235k-row programming question table (16 MB): nullable, no default, no backfill, partial indexes WHERE … IS NOT NULL, foreign keys	Brief; small table
M2	20261006000000_add_non_unique_answer_id_index_to_auto_gradings	Builds idx_course_assessment_answer_auto_gradings_on_answer_id concurrently on the 35M-row auto gradings table	Does not block reads or writes
M3	20261006000001_drop_unique_answer_id_index_from_auto_gradings	Drops the unique answer_id index concurrently, after checking that M2's index is valid	Does not block reads or writes

Measured locally on a production-scale copy (35.1M auto grading rows, unique index 752 MB):

Step	Time
M1	0.09 s
M2 (concurrent build)	19.5 s; new index 752 MB
M3 (concurrent drop)	0.016 s

During M2 and M3, a probe wrote to an auto grading row twice a second, each in a rolled-back transaction.
Write latency stayed at baseline (mean 40 ms during M2 against 55 ms before, max 75 ms, mostly psql
start-up), with no errors. After M3, the latest-run lookup (WHERE answer_id = ? ORDER BY created_at DESC, id DESC LIMIT 1) uses the new index (0.2 ms). These are laptop numbers against an idle database; staging
will be the realistic check.

M2 and M3 are separate migrations so that Rails records each one on its own. These are the repo's first
disable_ddl_transaction! migrations, so worth confirming that the deploy tooling tolerates them.

How they fail, and how to recover:

• An interrupted CREATE INDEX CONCURRENTLY (timeout, cancelled deploy, lost connection, out of disk)
  leaves an INVALID index behind. IF NOT EXISTS would accept it as done, so M2 does not use it: it drops
  an invalid leftover and builds again. Re-running M2 is always the recovery.
• M3 refuses to run unless M2's index is valid. Without it, every lookup of an answer's runs would scan
  35M rows.
• An interrupted M3 leaves the unique index INVALID but still enforcing uniqueness, so the new code's
  second run per answer would fail. Re-running M3 finishes the drop.
• M3 cannot be reversed once the new code has run: recreating the unique index fails as soon as any answer
  has a second run. Its down raises IrreversibleMigration. A code rollback does not need a schema
  rollback. The old has_one would read an arbitrary one of an answer's runs, which is degraded but not
  broken.

All of these paths were exercised on the test database.

Order and preconditions

1. Deploy in a window with no autograding jobs queued or running. Commit 1 changes the grading jobs'
   arguments, and jobs serialised by the old code would not deserialise correctly. There is deliberately no
   compatibility shim.

2. Before M2:

   • check pg_stat_activity for long-running transactions, read-only ones included (reports, backups,
     stuck jobs), because a concurrent build waits for them;
   • check that no statement timeout applies to the migrating role;
   • allow about 750 MB of extra disk while both indexes exist.

   Long-running transactions. The build waits for every transaction that is open when it starts, so anything
   old here will hold it up:

   SELECT pid, usename, application_name, state, now() - xact_start AS xact_age, left(query, 80) AS query
   FROM pg_stat_activity
   WHERE xact_start IS NOT NULL AND pid <> pg_backend_pid()
   ORDER BY xact_start
   LIMIT 20;

   Timeouts, as seen by the connection the migration actually uses. This includes anything set in
   database.yml or at the role or database level:

   bin/rails runner 'c = ActiveRecord::Base.connection
     %w[statement_timeout lock_timeout idle_in_transaction_session_timeout].each { |s| puts "#{s}: #{c.select_value("SHOW #{s}")}" }'

   Each should be 0, meaning disabled, or comfortably longer than the build. Role- and database-level
   overrides, if you need to find where a value comes from:

   SELECT r.rolname, d.datname, s.setconfig
   FROM pg_db_role_setting s
   LEFT JOIN pg_roles r ON r.oid = s.setrole
   LEFT JOIN pg_database d ON d.oid = s.setdatabase;

   Disk. The new index is about the size of the one it replaces:

   SELECT pg_size_pretty(pg_relation_size('index_course_assessment_answer_auto_gradings_on_answer_id'));

   Compare that with free space on the database volume. On a self-hosted server, run
   df -h "$(psql -Atc 'SHOW data_directory')" on the host; that needs superuser. On a managed database, use
   the provider's storage metrics.

3. Run M1, M2 and M3, in order, before the new code serves traffic. M3 must complete first:

   bin/rails db:migrate:up VERSION=20261002000000   # M1
   bin/rails db:migrate:up VERSION=20261006000000   # M2
   bin/rails db:migrate:up VERSION=20261006000001   # M3

   Running them one at a time shows each one's duration; a plain bin/rails db:migrate runs all three in the
   same order. While M2 builds, its progress is visible from another session. current_locker_pid is the
   transaction it is waiting for, if any:

   SELECT phase, blocks_done, blocks_total, tuples_done, tuples_total, current_locker_pid
   FROM pg_stat_progress_create_index;

   Afterwards, check that the new index is valid and the unique one is gone. The query is in the staging
   section below.

4. Deploy the code.

Old code that keeps serving between M3 and the code deploy has lost its database-level guard against two
concurrent first gradings of one answer each creating a run. Within the no-jobs window this is unlikely, and at
worst leaves a spare run.

Import jobs queued by the old code still run:

• ProgrammingImportJob gains an argument, but it is optional and appended.
• The old code recorded each job's id on the question after enqueueing it, so a queued old job is still the
  recorded one and applies.
• CodaveriImportJob keeps its arguments; it now ignores the package it was queued with.

Known gaps and follow-ups

• A failed import leaves the question half-edited (pre-existing). The controller commits the new
  language and limits in the request, while the package and test cases only change if the import succeeds. A
  failed import therefore leaves the new settings paired with the last applied package and tests (see the
  table under "Overlapping edits and imports"). Snapshotting does not make this worse, since a failed import
  creates no snapshot. Making version changes all-or-nothing would change what staff see when an import
  fails, and touches the flow the language upgrade feature relies on.
• An import job that is lost (never runs, and stays submitted) isn't retried automatically, and
  Codaveri-only edits won't record their sync jobs while it's pending. Only a failed import (errored)
  triggers a retry on the next save.
• Imports must be scheduled through a question save. An import applies only while the question records
  its job, and only Programming#schedule_import records it. A ProgrammingImportJob enqueued directly is
  always superseded. Nothing in this repository does that, but the language upgrade branch needs checking
  before it merges.
• No-op snapshots. Re-uploading an identical package, or retrying a failed import, still creates a
  snapshot and a regrade. The package's SHA-256 makes a cheap comparison possible.
• Snapshot retention. Snapshots are kept until their question is deleted. Cleaning up unreferenced ones
  needs care with in-flight runs, and isn't needed for correctness.
• Comparison UI for an answer's runs, and a read-only snapshot view.
• Runs left by zombie-job recovery. check_zombie_jobs regrades into a new run, so the dead run stays in
  the history with its stuck job. A comparison UI should expect unfinished runs.
• Koditsu sync still replaces all of an answer's programming runs, as before.
• Programming is within about 6 lines of Rubocop's 200-line class limit, even with the scheduling code
  extracted into ProgrammingImportsConcern. The next addition will need something else extracted first.
• Factory bug, unrelated: create(:submission, :submitted, auto_grade: false) raises NameError on an
  undefined answer.

Testing

Automated

• Backend: assessment models, services, jobs, controllers and helpers, plus duplication services: 1679
  examples, 0 failures. New specs cover:

  • snapshot creation, including the transaction, package reference, pre-edit values, preloaded associations
    and superseder;
  • the in-flight grading races, including an import committed from a second database connection between
    the run's two reads;
  • Codaveri runs: evaluated on the old version, on the new version, and with mixed results;
  • two imports saving at once on separate connections, waiting until Postgres reports the second one
    blocked. Without the lock, this reproduces the empty snapshot;
  • stale imports: superseded before evaluating, during evaluation (checked under the lock), by a package
    removal, and a superseded job queueing no regrade;
  • scheduling: the job is recorded before it is enqueued, and a job that can't be enqueued is marked errored;
  • Codaveri pushes: a later import committing before this job's push, a question Codaveri already has, a
    rejected push keeping its failure status, CodaveriImportJob pushing the current package, and a
    Codaveri-only edit not displacing a pending import;
  • the regrade scope (attempting, submitted, and submitted with a newer attempt);
  • the graded-version helper;
  • answers#show for an answer graded before an edit, and while a later run is still grading;
  • the read-only guard and the 404 for snapshot ids;
  • package removal end to end through PATCH #update;
  • grading runs per answer, and unsubmit keeping runs.

  Each was checked to fail with the behaviour it covers removed.

• Frontend: full Jest suite passes; new tests for the alert and the gradingResults reducer carrying the
  flag.

• Rubocop clean repo-wide.

• One earlier full run had 2 failures in specs this change doesn't touch: reference-timeline duplication,
  and the assessment controller's tabbed-view update. Both pass in isolation and in two later full runs with
  different random orders. The cause wasn't found, and the new specs that start threads can't be ruled out;
  worth watching in CI.

• Not run: feature (browser) specs. spec/features/course/assessment/submission/autograded_spec.rb is the
  one that touches grading runs.

Suggested test paths on staging

Run the migrations first, recording each one's time and checking the index state afterwards:

SELECT indexrelid::regclass, indisvalid, indisunique
FROM pg_index WHERE indrelid = 'course_assessment_answer_auto_gradings'::regclass;

Use an assessment with a zip-upload programming question and a few students. One submission should be
attempting with several submitted attempts, and another should be submitted.

These console helpers make the checks quicker:

q = Course::Assessment::Question::Programming.find(QUESTION_ID)
q.snapshots.map { |s| [s.id, s.superseded_at, s.superseder&.name, s.test_cases.size] }
[q.import_job_id, q.import_job&.status, q.is_synced_with_codaveri, q.codaveri_status]

a = Course::Assessment::Answer.find(ANSWER_ID)
a.auto_gradings.map { |r| [r.id, r.created_at, r.actable_id.present?, r.job&.status] }

# Which version a run was graded against: the live question's id, or a snapshot's.
Course::Assessment::Question::ProgrammingTestCase.
  where(id: a.auto_grading.specific.test_results.select(:test_case_id)).distinct.pluck(:question_id)

#	Do	Before	Expect now
1	Upload a new package to the question	Old results destroyed; every answer regraded	One snapshot, holding the old test cases, the editor and the time. Earlier attempts keep their grades. Only the latest attempt per submission (and submitted current answers) is regraded, each gaining a second run
2	While those regrades are queued, open a student's submission	New test cases with no marks (the old results were destroyed), "being evaluated"	The previous results against the old test cases, "being evaluated", and the "changes have been made" alert. Once the run finishes, the new results, and no alert
3	Open Past Answers on an earlier attempt	Current test cases, no marks	That attempt's own test cases with their pass/fail marks, plus the alert
4	Open assessment statistics for the question	As 3	As 3 for answers graded before the edit
5	Change only the time limit and save	Re-import; history destroyed	Re-import; one more snapshot holding the old limit
6	Change only the title or description	No re-import	No re-import, no snapshot, no regrade
7	On an online-editor question with graded answers, switch it to non-autograded and save	500	Saves. A snapshot holds the old test cases and package; old results remain
8	As a student on an attempting submission, type into a programming answer and let it autosave	No grading	No grading, no new run
9	Click re-evaluate on a graded programming answer	Run overwritten	A new run; the previous run and its results remain (a.auto_gradings)
10	On an autograded assessment with published submissions, change which test case types count towards the grade (public, private, evaluation). This runs resubmit_programming	The answers' runs destroyed, then regraded	Runs kept, plus a new one
11	Unsubmit a submission, then submit again	—	Earlier answers keep their runs
12	Duplicate the assessment, then the course	—	The copies' questions have no snapshots, and their test cases are copies of the live ones
13	Open the programming question edit URL with a snapshot's id in place of the question's	—	The edit request returns 404; check the page shows the not-found state, not an empty form
14	Delete a question that has snapshots	—	Deleted, along with its snapshots
15	If Codaveri grading is enabled on staging, repeat 1 and 9 on a Codaveri question	—	As 1 and 9. Rows 19 and 20 need Codaveri too
16	Edit a question while one of its answers is being graded (submit, then immediately upload a package)	The run could fail on a foreign key, or be matched to the new test cases	The run completes against the old test cases, which are now on the snapshot
17	Upload package A, then package B before A's import finishes	Whichever import finishes last wins	The question ends on B. No snapshot is empty. A's import either applied first and is kept as a snapshot, or was superseded and completed without changing anything
18	Upload a package, then switch the question to non-autograded before the import finishes	The import can reattach a package afterwards	Stays non-autograded; the import completes without applying
19	On a Codaveri question, upload a new package, then grade an answer	The import pushes its own package	After the import, is_synced_with_codaveri is true and the grading results match the new test cases
20	On a Codaveri question, upload a package and, before its import finishes, turn on live feedback	The live-feedback sync replaces the recorded import job, so the edit page follows the sync instead	The import still applies, and import_job_id stays the import's until it finishes. The live-feedback sync still runs and pushes the current version

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Import and grading concurrency windows can still mismatch packages, remote problems, snapshots, and pinned test cases.

Review effort: Balanced
Findings: 3

Open (3)

• Package version is not pinned with test cases · New
• Evaluation can mix pinned tests with a newer problem version · New
• Concurrent imports can create empty or lost snapshots · New

What changed in this PR

Adds immutable programming-question snapshots and preserves grading history across edits and regrades.

Changes:

• Snapshots replaced question versions and retain test cases, templates, and package references.
• Stores multiple programming grading runs and narrows regrading to visible/current answers.
• Displays historical results against their original tests with a version-change notice.

File	Description
spec/​services/​course/​assessment/​question/​programming_import_service_spec.rb	Tests snapshot imports and rollback behavior.
spec/​services/​course/​assessment/​question/​answers_evaluation_service_spec.rb	Tests narrowed regrading scope.
spec/​services/​course/​assessment/​answer/​text_response_comprehension_auto_grading_service_spec.rb	Updates grading API tests.
spec/​services/​course/​assessment/​answer/​text_response_auto_grading_service_spec.rb	Updates grading API tests.
spec/​services/​course/​assessment/​answer/​rubric_auto_grading_service_spec.rb	Updates grading API tests.
spec/​services/​course/​assessment/​answer/​programming_codaveri_auto_grading_service_spec.rb	Updates Codaveri grading tests.
spec/​services/​course/​assessment/​answer/​programming_auto_grading_service_spec.rb	Tests version pinning and retained runs.
spec/​services/​course/​assessment/​answer/​multiple_response_auto_grading_service_spec.rb	Updates grading API tests.
spec/​services/​course/​assessment/​answer/​auto_grading_service_spec.rb	Tests explicit grading runs.
spec/​models/​course/​assessment/​question/​programming_spec.rb	Tests snapshots, immutability, and package removal.
spec/​models/​course/​assessment/​answer/​programming_spec.rb	Tests completed-run selection.
spec/​models/​course/​assessment/​answer_spec.rb	Tests multi-run associations and creation.
spec/​jobs/​course/​assessment/​answer/​reduce_priority_auto_grading_job_spec.rb	Updates job argument tests.
spec/​jobs/​course/​assessment/​answer/​auto_grading_job_spec.rb	Tests explicit run persistence.
spec/​helpers/​course/​assessment/​answer/​programming_test_case_helper_spec.rb	Tests grading-version resolution.
spec/​controllers/​course/​assessment/​submission/​answer/​answers_controller_spec.rb	Tests historical-result rendering.
spec/​controllers/​course/​assessment/​question/​programming_controller_spec.rb	Tests package removal and snapshot isolation.
lib/​tasks/​db/​migrate_programming_question_languages.rake	Excludes snapshots from language migration.
db/​schema.rb	Records snapshot columns and non-unique run index.
db/​migrate/​20261006000001_drop_unique_answer_id_index_from_auto_gradings.rb	Concurrently removes the unique run index.
db/​migrate/​20261006000000_add_non_unique_answer_id_index_to_auto_gradings.rb	Concurrently creates its replacement index.
db/​migrate/​20261002000000_add_snapshot_columns_to_course_assessment_question_programming.rb	Adds snapshot metadata and foreign keys.
client/​locales/​zh.json	Adds the historical-version notice.
client/​locales/​ko.json	Adds the historical-version notice.
client/​locales/​en.json	Adds the historical-version notice.
client/​app/​types/​course/​statistics/​answer.ts	Types the historical-version flag.
client/​app/​types/​course/​assessment/​submission/​answer/​programming.ts	Extends programming-answer state.
client/​app/​bundles/​course/​assessment/​submission/​reducers/​gradingResults.ts	Carries the version flag into state.
client/​app/​bundles/​course/​assessment/​submission/​reducers/​__test__/​gradingResults.test.ts	Tests version-flag state updates.
client/​app/​bundles/​course/​assessment/​submission/​containers/​TestCaseView/​index.tsx	Passes the flag to test-case rendering.
client/​app/​bundles/​course/​assessment/​submission/​components/​AnswerDetails/​ProgrammingComponent/​translations.ts	Defines the notice message.
client/​app/​bundles/​course/​assessment/​submission/​components/​AnswerDetails/​ProgrammingComponent/​TestCases.tsx	Renders the historical-version alert.
client/​app/​bundles/​course/​assessment/​submission/​components/​AnswerDetails/​ProgrammingComponent/​__test__/​TestCases.test.tsx	Tests alert visibility.
client/​app/​bundles/​course/​assessment/​submission/​components/​AnswerDetails/​ProgrammingAnswerDetails.tsx	Propagates the version flag.
app/​views/​course/​assessment/​answer/​programming/​_programming.json.jbuilder	Serializes historical tests and results.
app/​services/​course/​assessment/​question/​programming/​programming_package_service.rb	Routes package removal through snapshotting.
app/​services/​course/​assessment/​question/​programming_import_service.rb	Creates snapshots during imports.
app/​services/​course/​assessment/​question/​answers_evaluation_service.rb	Regrades only visible/current answers.
app/​services/​course/​assessment/​answer/​text_response_comprehension_auto_grading_service.rb	Writes into an explicit grading run.
app/​services/​course/​assessment/​answer/​text_response_auto_grading_service.rb	Writes into an explicit grading run.
app/​services/​course/​assessment/​answer/​rubric_auto_grading_service.rb	Writes into an explicit grading run.
app/​services/​course/​assessment/​answer/​programming_codaveri_auto_grading_service.rb	Pins tests and preserves Codaveri runs.
app/​services/​course/​assessment/​answer/​programming_auto_grading_service.rb	Pins tests and preserves programming runs.
app/​services/​course/​assessment/​answer/​multiple_response_auto_grading_service.rb	Writes into an explicit grading run.
app/​services/​course/​assessment/​answer/​auto_grading_service.rb	Passes and saves explicit runs.
app/​models/​course/​assessment/​question/​programming.rb	Integrates snapshots and package removal.
app/​models/​course/​assessment/​question/​programming_test_case.rb	Protects snapshot test cases.
app/​models/​course/​assessment/​question/​programming_template_file.rb	Protects snapshot templates.
app/​models/​course/​assessment/​answer/​programming.rb	Selects the latest completed run.
app/​models/​course/​assessment/​answer/​auto_grading.rb	Allows multiple runs per answer.
app/​models/​course/​assessment/​answer.rb	Adds run history and run creation logic.
app/​models/​concerns/​course/​assessment/​question/​programming_snapshots_concern.rb	Implements snapshot creation and associations.
app/​models/​concerns/​course/​assessment/​question/​programming_snapshot_read_only_concern.rb	Enforces snapshot immutability.
app/​jobs/​course/​assessment/​question/​programming_import_job.rb	Carries pre-edit values into imports.
app/​jobs/​course/​assessment/​answer/​base_auto_grading_job.rb	Carries the selected grading run.
app/​helpers/​course/​assessment/​answer/​programming_test_case_helper.rb	Resolves the graded question version.
.rubocop_todo.yml	Removes resolved exclusions.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Import and grading races can still corrupt or misattribute preserved grading history.

Review effort: Balanced
Findings: 5 · 1

Open (6)

• Skip superseded jobs before applying imports · New
• Scope Codaveri bulk updates to live records · New
• Atomically record jobs and handle enqueue failures · New
• Read package data and tests from one consistent version · New
• Pin evaluation results to the Codaveri-evaluated version · New
• Sync the current question version before Codaveri evaluation · New

Resolved since last review (3)

• Concurrent imports can create empty or lost snapshots
• Evaluation can mix pinned tests with a newer problem version
• Package version is not pinned with test cases

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved issues affect version consistency, Codaveri synchronization, and visible grading results.

Review effort: Lite
Findings: 4

Open (4)

• Tie regrades to the imported package version · New
• Prevent stale edits from creating inconsistent snapshots · New
• Prevent older runs from overwriting newer grades · New
• Reload question fields before evaluating the run · New

Resolved since last review (6)

• Pin evaluation results to the Codaveri-evaluated version
• Read package data and tests from one consistent version
• Atomically record jobs and handle enqueue failures
• Scope Codaveri bulk updates to live records
• Skip superseded jobs before applying imports
• Sync the current question version before Codaveri evaluation