diff --git a/quest/README.md b/quest/README.md index 233f6ef..7f14fb0 100644 --- a/quest/README.md +++ b/quest/README.md @@ -16,3 +16,4 @@ its scope, until it is scheduled. ## Required - [m0: immediate priorities](/quest/m0/README.md) - hosted worklets once a release ships assets() +- [m1: next](/quest/m1/README.md) - the Voice AI demo page and its live buffer view diff --git a/quest/m1/README.md b/quest/m1/README.md new file mode 100644 index 0000000..f8795e4 --- /dev/null +++ b/quest/m1/README.md @@ -0,0 +1,13 @@ +# m1: next + +## Goal + +The demos after m0: a Voice AI demo you can talk to, which shows why MoQ beats +WebRTC for voice agents. + +## Required + +- [Voice AI demo page](/quest/m1/voice-page.md) - talk to a Pipecat agent over MoQ at `/voice`, with a live transcript and a `/demo` card +- [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - condition: the reply-side data is on npm +- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - condition: word timings and prompt-side arrivals on wtf +- [Voice buffers under the words](/quest/m1/voice-buffers.md) - the live transcript draws each word's audio frames, marking the ones late for a typical WebRTC jitter buffer diff --git a/quest/m1/voice-buffers.md b/quest/m1/voice-buffers.md new file mode 100644 index 0000000..3391c5e --- /dev/null +++ b/quest/m1/voice-buffers.md @@ -0,0 +1,41 @@ +# [M] Voice buffers under the words + +## Goal + +Under each word of the live `/voice` transcript, a strip draws that word's +audio frames from real measurements: when each arrived against when it plays. +A frame that arrived later than a typical WebRTC jitter buffer waits turns +red: an estimate from the MoQ arrivals, not a measured WebRTC receiver. Both +directions: your prompt as the agent received it, and the agent's reply as +your browser received it. Nothing is simulated or injected. + +## Plan + +Decided 2026-10-09: + +- **Real data only.** No "network blip" button and no injected loss. On a + clean network the strips are all on time, and the reply strip still shows + how far ahead the agent sent its audio. +- **A WebRTC deadline line, computed from the real arrivals.** A frame is + "late for WebRTC" when its arrival spread exceeds a typical WebRTC jitter + target. Measure lateness as arrival minus timestamp, relative to the + smallest one seen, so the sender's and receiver's clocks never need to + agree. Choose and cite the target when implementing. +- **Draw it like `src/components/voice/`.** Reuse its segment colors and + word layout on a sliding window of live, append-only data, in place of its + fixed axis and simulated `FLIGHT`. +- **Reply side** reads `@moq/watch`'s per-frame `arrivals()` handle on the + audio decoder and maps frames to words with the agent's TTS word timings. + It yields each frame's `{timestamp, arrival, late}` and a gap item + (`timestamp` and `duration` in `Time.Milli`) where media will never + arrive. Its queue is unbounded, so the page keeps its own window. There is + no gap at the join: the strip starts on the first frame read. +- **Prompt side** reads the agent's `voice.json` track on the response + broadcast: word timings from STT and the arrival time of each request + audio frame. + +## Required + +- [Voice AI demo page](/quest/m1/voice-page.md) - the page and transcript this draws under +- [An @moq/watch release ships per-frame arrivals](/quest/m1/watch-arrivals-release.md) - the reply-side data +- [The voice agent publishes voice.json](/quest/m1/voice-timeline-track.md) - word timings and the prompt-side data diff --git a/quest/m1/voice-page.md b/quest/m1/voice-page.md new file mode 100644 index 0000000..7210985 --- /dev/null +++ b/quest/m1/voice-page.md @@ -0,0 +1,45 @@ +# [M] Voice AI demo page + +## Goal + +`/voice` lets a reader talk to the moq.pro Pipecat agent over MoQ and shows a +live transcript of both sides. `/demo` gets a card for it. The component is +reusable, so a blog post can embed it; the post itself (branch +`claude/voice-post-components`) is not part of this quest. + +## Plan + +Decided 2026-10-09: + +- **A native Solid component, not moq.pro's iframe.** The buffer view needs + the player's internals, which an iframe from another site hides, and demos + belong in moq.dev. moq.pro keeps its own `/voice` page. +- **It talks to the moq.pro session API directly.** + `POST /voice/sessions {turnstileToken}` returns `{id, relayUrl, + requestBroadcast, responseBroadcast, token, expiresAt}`. One token + publishes the request and subscribes the response. Poll + `GET /voice/sessions/:id` until the agent claims it, and render refusals by + `code` (`busy`, `rate_limit`, `daily_budget`, ...). + moq.pro's `app/src/lib/VoiceDemo.svelte` is the reference client. + Start from `src/components/voice-demo.tsx` on + `claude/voice-post-components`, which already speaks this contract + (Turnstile, poll, `transcript.json.z`). The older `voice-ai2` component + predates it. +- **Live points at staging.** The demo only runs on `api.moq.wtf` until + moq.pro's production quest lands, so a separate `PUBLIC_VOICE_API_URL` is + `https://api.moq.wtf` for live and staging, and `PUBLIC_TURNSTILE_SITE_KEY` + is the wtf widget's public key. `PUBLIC_API_URL` stays on its environment's + API for `try`. +- **No Pipecat client.** `` sends the mic and `` + plays the reply with a latency ceiling near the agent's 25s send-ahead. + The page does the small part of RTVI the agent needs itself: publish a + `transcript.json.z` track carrying `client-ready`, and read the agent's + `transcript.json.z` (not in the catalog; subscribe by name) for + transcripts. On barge-in the agent retires `bot-audio` for + `bot-audio-` through the catalog, which `` follows. +- The transcript renders the way `src/components/voice/` lays out words, so + [Voice buffers under the words](/quest/m1/voice-buffers.md) can draw under + them. + +Verify against staging in `just dev`: a conversation, barge-in, Stop, the +five-minute limit, and a `busy` refusal. diff --git a/quest/m1/voice-timeline-track.md b/quest/m1/voice-timeline-track.md new file mode 100644 index 0000000..c4950f9 --- /dev/null +++ b/quest/m1/voice-timeline-track.md @@ -0,0 +1,19 @@ +# [XS] The voice agent publishes voice.json + +## Goal + +The moq.pro demo agent on wtf publishes a `voice.json` track on each session's +response broadcast, carrying word timings for both directions and the arrival +time of each request audio frame. + +This quest tracks a condition outside the repository. When it holds, delete +this quest and every `Required` entry that links it. + +## Plan + +moq.pro tracks the track as +[its own quest](https://github.com/moq-dev/moq.pro/blob/main/quest/m2/voice-timeline.md), +which owns its schema. It merged in +[moq-dev/moq.pro#2319](https://github.com/moq-dev/moq.pro/pull/2319), but wtf +has not been deployed since. Check by subscribing to it during a staging +session. diff --git a/quest/m1/watch-arrivals-release.md b/quest/m1/watch-arrivals-release.md new file mode 100644 index 0000000..6003d66 --- /dev/null +++ b/quest/m1/watch-arrivals-release.md @@ -0,0 +1,18 @@ +# [XS] An @moq/watch release ships per-frame arrivals + +## Goal + +The newest `@moq/watch` on npm exposes the per-frame `arrivals()` handle on +the audio and video decoders, and moq.dev builds against it. + +This quest tracks a condition outside the repository. When it holds, bump +`@moq/watch` here, then delete this quest and every `Required` entry that +links it. + +## Plan + +moq shipped the handle in +[moq-dev/moq#5311](https://github.com/moq-dev/moq/pull/5311) on `main`, a +breaking change. 0.6.3 came from moq's `release` branch without it. +Check `npm view @moq/watch version` and its type definitions for +`Audio.Decoder.arrivals()` and `Watch.Arrival`.