Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/sleep/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,56 @@ An existing file that cannot be decoded, parsed, or read stops the command
with a configuration error and exit code 2; it does not discard your settings
and run with default budgets. Explicit CLI flags still override loaded values.

### Harvest filters (environment)

Harvest keeps what you typed and drops what a machine wrote. Four environment
variables extend the built-in filters. They apply to the Claude Code source and
are read when the process starts, so set them in the shell or in whatever runs
the night:

| Variable | Effect |
|---|---|
| `SKILLOPT_SLEEP_AGENT_MARKERS` | Comma-separated markers. A session whose **first** user prompt contains one is dropped entire — for prompts written by another tool's sub-agent (memory observers, critic agents, hook-driven runs). |
| `SKILLOPT_SLEEP_NEG_FEEDBACK` | Extra comma-separated phrases, in any language, that count as you disapproving of the previous answer. |
| `SKILLOPT_SLEEP_POS_FEEDBACK` | The same, for approval. |
| `SKILLOPT_SLEEP_REPEAT_MIN_SESSIONS` | Sessions a prompt prefix must recur in before the repeats are treated as one template (default `0`: off; `3` is the usual opt-in, `0` turns a configured value back off). |

The first three cover a tool whose prompt template you can quote, and the
filter they feed is exact: it checks the session's first user prompt against
each marker. A nightly hook that summarises the previous session — the shape
that fills a harvest window without your ever typing anything — is the case
they exist for:

```bash
# the hook's own template, as its first line appears in the transcript
export SKILLOPT_SLEEP_AGENT_MARKERS="Below is a conversation log from a coding session"
```

`SKILLOPT_SLEEP_REPEAT_MIN_SESSIONS` covers the templates you cannot quote, but
recurrence is not proof of who wrote a prompt, so it is **off by default**. A
single-turn session that shares its first 120 characters of prompt with `N`
others in the same harvest window is dropped *before* `--max-sessions` is
applied, so the repeats cannot consume the cap the way they did in #319 — but
only once you opt in with a value for `N` (3 is the usual choice). The reason
for the default: three people asking for a code review against one long shared
checklist are structurally identical to a hook re-running itself — one turn, one
long common prefix, a different task after it — and losing a real task is worse
than harvesting a hook's summary sessions. Set the variable for a project whose
recurring sessions you have recognized. A session with a follow-up user turn is
never matched (it is interactive by definition), and neither is a prompt shorter
than the prefix — a real one-shot request repeats entirely rather than sharing a
long boilerplate header.

`harvest` and the progress output of `run` / `dry-run` report what was dropped,
so a night that looks empty is explainable:

```text
[sleep] 120 sessions -> 40 tasks (skipped: 297 repeated-template sessions, 2 past the session cap)
```

With the filter off (the default) nothing is dropped by it, `--max-sessions`
still bounds how many transcripts are read, and the note reports only the cap.

### VS Code GitHub Copilot Chat

Use `--source copilot` to harvest local VS Code GitHub Copilot Chat sessions.
Expand Down
37 changes: 37 additions & 0 deletions plugins/claude-code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,43 @@ session data and provider policy before using a real backend on a sensitive
project; the [shared integration guide](../README.md#data-boundary) describes a
reviewed task-file workflow.

### Sessions you did not type

Harvest keeps your prompts and drops machine-written transcripts. Three
environment variables extend what counts as machine-written, for the sessions
Claude Code gives no other tell:

| Variable | Effect |
|---|---|
| `SKILLOPT_SLEEP_AGENT_MARKERS` | Comma-separated markers; a session whose **first** prompt contains one is dropped entire. |
| `SKILLOPT_SLEEP_NEG_FEEDBACK` / `SKILLOPT_SLEEP_POS_FEEDBACK` | Extra comma-separated phrases, in any language, that count as you rejecting or approving the previous answer. |

The case they exist for is a local hook that summarises the previous session on
a timer. Its runs are single-turn, they are frequent enough to fill
`--max-sessions` and starve the night of your real sessions, and in the desktop
app they carry `entrypoint: "claude-desktop"` like interactive ones — so neither
`entrypoint` nor a replay marker identifies them. Quote the hook's first line:

```bash
export SKILLOPT_SLEEP_AGENT_MARKERS="Below is a conversation log from a coding session"
```

A template you cannot quote is covered only after you opt in: a single-turn
session sharing its first 120 characters of prompt with three others in the same
harvest window is dropped before the session cap is applied once
`SKILLOPT_SLEEP_REPEAT_MIN_SESSIONS` is set (default `0`, off; `3` is the usual
choice). Recurrence is not proof of who wrote a prompt — three human requests
against one long shared checklist look the same — so the default keeps every
session and `--max-sessions` still bounds how many transcripts are read. A
session with a follow-up user turn is never matched, and neither is a prompt
shorter than the prefix — your own one-shot requests repeat entirely rather than
sharing a long boilerplate header.

`/skillopt-sleep harvest` and the progress line of a `run` / `dry-run` report
what was dropped, so a thin night is explainable:
`harvest done: sessions=120 (skipped: 297 repeated-template sessions, 2 past the session cap)`.
Full reference: [the sleep guide](../../docs/sleep/README.md#harvest-filters-environment).

### Handoff mode (session answers the model calls)

`--backend handoff` runs the cycle without any model subprocess: the engine
Expand Down
7 changes: 5 additions & 2 deletions skillopt_sleep/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
from skillopt_sleep.backend import CursorBackendError
from skillopt_sleep.config import ConfigError, load_config
from skillopt_sleep.cycle import _one_line_display_text, run_sleep_cycle
from skillopt_sleep.harvest import format_harvest_note
from skillopt_sleep.harvest_sources import harvest_for_config
from skillopt_sleep.mine import mine
from skillopt_sleep.staging import (
Expand Down Expand Up @@ -788,13 +789,14 @@ def fail(code: int, kind: str, message: str, **extra: Any) -> int:
def cmd_harvest(args) -> int:
cfg = _cfg_from_args(args)
session_limit = cfg.get("max_sessions_per_night", 0) or cfg.get("max_tasks_per_night", 40) * 3
harvest_stats: Dict[str, int] = {}
target_skill_path = cfg.managed_skill_path() if cfg.get("target_skill_path", "") else ""
target_skill_text = _read_text(target_skill_path) if target_skill_path else ""
max_tasks = cfg.get("max_tasks_per_night", 40)
candidate_limit = max_tasks
if cfg.get("target_task_filter", True) and target_skill_text:
candidate_limit = max(max_tasks, max_tasks * 3)
digests = harvest_for_config(cfg, limit=session_limit)
digests = harvest_for_config(cfg, limit=session_limit, stats=harvest_stats)
tasks = mine(
digests,
max_tasks=max_tasks,
Expand All @@ -820,7 +822,8 @@ def cmd_harvest(args) -> int:
json_payload["output"] = output_path
print(json.dumps(json_payload, ensure_ascii=False, indent=2))
else:
print(f"[sleep] {len(digests)} sessions -> {len(tasks)} tasks")
print(f"[sleep] {len(digests)} sessions -> {len(tasks)} tasks"
f"{format_harvest_note(harvest_stats)}")
if output_path:
print(f"[sleep] wrote reviewed-task draft: {output_path}")
for t in tasks:
Expand Down
10 changes: 8 additions & 2 deletions skillopt_sleep/cycle.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
import time
import unicodedata
from dataclasses import dataclass
from typing import List, Optional, Sequence
from typing import Dict, List, Optional, Sequence

from skillopt_sleep import evidence
from skillopt_sleep.backend import Backend, CursorBackendError, build_backend
Expand All @@ -28,6 +28,7 @@
from skillopt_sleep.dream import dream_consolidate
from skillopt_sleep.evidence import EvidenceLog
from skillopt_sleep.handoff_backend import PendingCalls
from skillopt_sleep.harvest import format_harvest_note
from skillopt_sleep.harvest_sources import harvest_for_config
from skillopt_sleep.memory import ensure_skill_scaffold
from skillopt_sleep.mine import group_tasks_by_skill_hint, mine
Expand Down Expand Up @@ -798,6 +799,7 @@ def budget_exhausted() -> bool:
_progress(cfg, f"using {len(tasks)} seeded tasks")
else:
since = state.last_harvest_for(project, harvest_source)
harvest_stats: Dict[str, int] = {}
# On first run (no prior harvest), apply lookback_hours so we don't
# scan the entire transcript history and trigger massive LLM mining.
if since is None:
Expand All @@ -819,9 +821,13 @@ def budget_exhausted() -> bool:
cfg,
since_iso=since,
limit=max_sessions,
stats=harvest_stats,
)
n_sessions = len(digests)
_progress(cfg, f"harvest done: sessions={n_sessions}")
_progress(
cfg,
f"harvest done: sessions={n_sessions}{format_harvest_note(harvest_stats)}",
)
if ev is not None:
# The transcript end of the evidentiary chain: which sessions were
# even considered, and what signals they carried into mining.
Expand Down
Loading