[Nuplum (45lab95)]

feat(sleep): add DeepSeek Harness transcript source

Closes #225

Summary

Add a local, read-only DeepSeek Harness (DSH) transcript source for SkillOpt-Sleep.

The new explicit --source dsh converts DSH JSONL sessions into the existing
SessionDigest and mined-task pipeline. DSH is not added to --source auto,
and this change does not add a DSH backend, plugin, or UI integration.

Changes

• Add a DSH JSONL session harvester supporting:
  • uncompressed session.jsonl;
  • default compressed session.jsonl.zstd;
  • concatenated Zstandard frames;
  • packed chunk records and current DSH lifecycle metadata.
• Keep only human user requests, visible assistant text, short tool names,
  timestamps, and feedback-derived positive/negative signals.
• Exclude reasoning, tool arguments and results, request/provider metadata,
  attachments, feedback text, injected user-role context, subagent sessions,
  and SkillOpt replay sessions.
• Retain ordinary fork sessions without merging parent-session content.
• Silently skip an individual session when its format or event structure cannot
  be safely understood, without blocking other sessions.
• Add CLI, configuration, and documentation support.

Optional dependency

Add the optional extra:

dsh = ["zstandard>=0.22.0"]

zstandard is only needed to read DSH's default .jsonl.zstd session files.
It is not installed for users who do not use DSH, and uncompressed session.jsonl files remain readable without it.
Install with:

python -m pip install -e ".[dsh]"

Validation

python -m pytest tests/test_harvest_dsh.py -q
python -m ruff check skillopt_sleep/harvest_dsh.py tests/test_harvest_dsh.py

• Local DSH Web end-to-end validation:
  • harvested 1 session -> 1 task;
  • completed the mock dry-run pipeline successfully.

[Yifan Yang (Yif-Yang)]

Thanks for the narrowly scoped, read-only importer and privacy tests. The full suite passes here with the zstandard extra (1507 passed, 9 skipped), but checking the external format contract uncovered a current-version/stale-generation blocker at 84d64abf2c3d.

Current upstream DSH (deepseek-ai/deepseek-harness at d347e703908d0406b7a7ef80e3a0e594d86b2215) declares SESSION_FORMAT_VERSION = 2. Its JSONL backend writes session.v2.jsonl[.zstd], retains immutable v0/v1 predecessors during migration, and selects the numerically highest canonical generation:

• Current version constant
• Canonical generation layout and migration/read selection

The importer only discovers session.jsonl / session.jsonl.zstd and accepts header version 0. Therefore fresh current-format sessions are silently missed. More importantly, if migration retained a v0 file next to the authoritative v2 file, this implementation successfully harvests the obsolete v0 snapshot rather than refusing the unsupported current session.

I reproduced the latter offline with the PR's valid v0 fixture plus a neighboring session.v2.jsonl: harvest_dsh() returns the old user/assistant conversation. With an explicitly version-gated reader, the safe result is to refuse that session with a useful diagnostic until its highest generation is supported, not fall back to retained history.

Please select the highest canonical generation first and fail closed on unsupported generations. Then either support the currently released logical format through a validated codec/export boundary, or explicitly pin/document this as a legacy-v0-only source with a producer version and representative fixture. Please cover fresh v2-only sessions and coexisting v0/v1/v2 generations in tests. This is a concrete format-boundary issue; no DSH execution backend or paid-model experiment is needed.

[Nuplum (45lab95)]

Thanks for review. I addressed this by supporting the current v2 logical format rather than documenting DSH as a legacy-v0-only source.

The importer now discovers canonical session.jsonl / session.vN.jsonl files, including their .zstd variants, groups retained generations by session directory, and selects the numerically highest generation before parsing. It supports v0, v1, and v2. If the selected highest generation is unsupported, it skips that session without falling back to a retained predecessor; --progress and debug logging report the selected file, highest generation, supported versions, and upgrade guidance.

For v2, the reader now validates the published header boundary, including the required isSeeded field and allowed header keys, validates consistency between isSeeded and session/end-seed lineage markers, and replays the ordered surface before deriving visible user and assistant text. It also expands the JSONL storage codec's compressed sourceEventSeqs ranges (for example, [start, end]) before validating and applying replacements. This prevents text from a shadowed pre-compaction assistant message from being exported.

I added coverage for:

• fresh v2-only sessions;
• coexisting retained v0/v1/v2 generations, asserting that v2 is selected;
• an unsupported highest generation, asserting that no older generation is harvested and that a diagnostic is available;
• v2 replacement provenance encoded as compact sequence ranges;
• v2 header and seeded-lineage validation; and
• a sanitized Zstandard v2 fixture generated from a DeepSeek Harness master build at d347e703908d0406b7a7ef80e3a0e594d86b2215.

The reader remains a pure, read-only Python importer: it does not start DSH, require a DSH execution backend, or make provider calls.

Validation run:

python -m ruff check skillopt_sleep/harvest_dsh.py tests/test_harvest_dsh.py
python -m pytest -q tests/test_harvest_dsh.py

70 passed

[Yifan Yang (Yif-Yang)]

Re-reviewed 22615819b33a. The current-format changes address the original review concern: the updated importer selects the current v2 generation instead of a retained predecessor, and an unsupported newer generation is rejected without falling back to v0.

Local verification is 72 passed: the DSH suite plus independent supported-v2 / unknown-v3 checks. An important correction to my old review fixture: it used v2 to represent an unsupported future version. That assumption is obsolete now that this PR supports v2, so accepting the valid v2 file is correct. I updated the independent check to distinguish a supported current generation from an actually unsupported newer one; I did not interpret the old expectation as a new failure in your importer.

Please update the PR description's format/validation summary to match the implemented v0/v1/v2 support and the highest-generation rejection/diagnostic behavior, alongside the already-added sanitized fixture. Keep the read-only/privacy and seeded-lineage/replacement-provenance regressions intact as the format evolves.

The parser grew materially, so this focused contract result is not a claim of a complete safety audit of every new path. The original stale-generation reproduction is addressed; the exact-head official CI run still awaits maintainer approval, and final review remains before merge.

[Yifan Yang (Yif-Yang)]

Thank you for adding current DSH format support and the sanitized producer fixture; the earlier generation-selection concern is resolved.

The current DSH producer emits compaction/start, compaction/summary, and compaction/end without ignorable; these are in the pinned upstream event catalogue but absent from _KNOWN_EVENT_TYPES. I reproduced this on 22615819b33a by appending the producer's failed idle-compaction start/end pair to this PR's sanitized v2 fixture: the unchanged user/assistant conversation harvests before the pair, but afterward the whole session is skipped as unknown event. No replacement or provider call is involved. Please support these current built-in lifecycle records without retaining their private payloads, and add a producer-shaped fixture regression while keeping unknown-required-event refusal. Producer: https://github.com/deepseek-ai/deepseek-harness/blob/d347e703908d0406b7a7ef80e3a0e594d86b2215/packages/compaction/compaction-basic/src/region.ts ; event envelope: https://github.com/deepseek-ai/deepseek-harness/blob/d347e703908d0406b7a7ef80e3a0e594d86b2215/packages/core/session/src/index.ts .

[WODE25500]

Two things from a current dsh install, both material to this PR: the shipped format is v4 (not v2), and the compaction case Yif-Yang described is the smaller half of it.

The current producer writes v4

In the DSH build on this machine (0.2.1-alpha.1):

const SESSION_FORMAT_VERSION = 4;
// canonical filename: generation 0 -> "session.jsonl", otherwise `session.v${n}.jsonl`

Observed on disk, including one directory carrying both generations after migration:

sessions/--F-8.15.6--/session-062d51a4…/session.v4.jsonl.zstd
{"type":"session","version":4,"id":"session-062d51a4…","createdAt":…,"isSeeded":false,"delegationDepth":0,"agentPreset":"standard"}

sessions/--F-8.15.6--/session-fca99fc5…/session.jsonl.zstd     (v0 predecessor, version:0)
                                   /session.v4.jsonl.zstd      (v4 successor,  version:4)

_SUPPORTED_FORMAT_VERSIONS = frozenset({0, 1, 2}) therefore refuses the current generation on every real session, with the "older than supported" branch — the highest-generation selection itself is fine, it is the version set that gates it.

Header keys match _V2_HEADER_ALLOWED_KEYS (type/version/id/createdAt/cwd/isSeeded/delegationDepth/agentPreset), so the header validator needs no change.

ignorable does not exist in practice

Across 8 real v4 sessions / 4110 records on this machine, zero records carry an ignorable field. So the kind not in _KNOWN_EVENT_TYPES and record.get("ignorable") is not True gate refuses whole sessions rather than skipping one record. Event types present that the current set does not know:

session-log-deepseek/delivery-accepted   93
todo/write                               12
subagent/catalog                          4
model/selection                           3
workspace/changes                         1

Compaction, as raised in review

Confirmed structurally: compaction/start / compaction/summary / compaction/end are in the producer's catalogue and are not marked ignorable (they are the log-only bracket the compaction lock publishes), so a session that ran a compaction is refused outright. No compaction happens to have run in this corpus, so this is a contract failure rather than an observed one — same conclusion as the review, different evidence path.

Patch for the event catalogue

Against this branch (3cd41e2), python -m py_compile clean:

@@ _KNOWN_EVENT_TYPES
     "session/end-seed",
     "feedback/record",
     "web/deepseek-search-llm-request",
+    # Compaction lifecycle: log-only records that bracket a compaction. They
+    # carry no transcript content, and the producer does NOT mark them
+    # `ignorable`, so omitting them refuses the entire session as an unknown
+    # event — the case reported against this PR.
+    "compaction/start",
+    "compaction/summary",
+    "compaction/end",
+    # Present in a 4110-record v4 corpus but absent from the v2 catalogue.
+    # All are log/metadata records with no surface contribution.
+    "session-log-deepseek/delivery-accepted",
+    "todo/write",
+    "subagent/catalog",
+    "model/selection",
+    "workspace/changes",
 }

What I deliberately did not change

The version set. Accepting 4 is only honest with a v4-shaped fixture and a decision about the generation policy (compaction/summary in v4 carries compressed sourceEventSeqs ranges, so the replacement/ordering path needs a producer-shaped case rather than a bumped constant). If {0, 1, 2, 4} is the direction you want, I can send a sanitized v4 fixture — session.v4.jsonl.zstd, header above, one compaction bracket, one inbox splice, one replacement — plus the regression that fails on the bumped constant alone.

[Yifan Yang (Yif-Yang)]

Thanks for the new producer-format information. The current head still accepts only session versions 0/1/2, so a v4 header and the additional event/compaction paths are outside its supported contract. #330 has now merged, but it concerns the shell execution interface and does not resolve transcript parsing.

A small sanitized producer-shaped v4 fixture would be useful: include the real header/event envelopes, compaction or replacement and inbox transitions, and the expected final visible root-session conversation. Remove credentials, provider/account metadata and private payloads. Please include controls for superseded content, child/replay sessions and unknown events so that expanding the accepted version set cannot silently preserve invalid history or weaken the privacy boundary. The reported larger corpus has not been independently reviewed here; this discussion does not clear the existing lifecycle concerns or close #225.

Review baseline: 1a44050fc53a. The PR remains open; no merge was performed.