UI scope — getting away from the CLI

The project's surface area is currently absl flags on pipeline and generate. That made sense for an eval-driven loop (everything is a flag, every flag is a knob the autorater can sweep), but it is now the bottleneck for:

• triggering runs without remembering 8 flag combinations
• listening to renders inline (today: open Finder, double-click a WAV)
• A/B comparing two runs side-by-side
• reviewing panel critiques and deciding which patches to apply
• watching the new listener-panel auto-loop progress live

This doc scopes the move.

What needs to be in the UI

Not all of pipeline / generate. The UI focuses on the iterate-and-listen workflow; pipeline ingestion stays CLI (it's a one-time per-episode operation).

Required:

1. Run launcher — pick mode (iterate / meta / panel_loop), set seed, iterations, target_duration_min, budget. Hit go.
2. Run table — list past runs with combined / text / audio scores, concept title, iteration count, run-id link.
3. Run detail — for a single run: concept JSON, episode JSON, audio player, transcript with delivery_notes inline, autorater axis bars, audio-judge axis bars, panel notes (when present).
4. A/B compare — pick two runs, show ab_comparison.md side-by-side.
5. Live progress — when a run is in flight, stream the per-iter log line ("M2026-05-03... iter 1: ACCEPTED text=8.61 audio=9.50 combined=8.97").
6. Episode breakdown viewer — render the new analytics.py breakdown.md inline with cast-lead share + BME slices. Works on corpus AND generated.
7. Best-candidates dashboard — the BEST_CANDIDATES.md table, sortable, with WAV play buttons.

Deferred (CLI is fine):

• ingestion (pipeline --skip_extract --skip_ingest=False)
• prompt template editing (use VS Code)
• one-off python -c shells

Options

A. FastAPI + HTMX (recommended)

• Tiny backend in src/ui/server.py (FastAPI). Endpoints: /runs, /runs/{id}, /launch, /compare/{a}/{b}, /play/{run}/audio.wav.
• Frontend: a few .html templates rendered server-side, HTMX for the interactivity (button presses → swap a panel without writing JS).
• Audio: standard <audio src="/runs/.../audio.wav" controls>. WAVs streamed from the existing data/generations/... tree.
• Live progress: SSE endpoint that tails the absl log file.
• Why this: zero build step, no React, you get fully-typed JSON contracts via Pydantic, and the data layer is already on disk in JSON. ~600 lines.

B. Streamlit

• A single streamlit run src/ui/app.py covers it.
• Built-in audio player, dataframe widget, plotly support.
• Why this: fastest to MVP (~200 lines).
• Why not: poor support for long-running async jobs (the meta-mode loop takes 5–15 min); state model fights you when you want a runs table that survives a refresh; no good story for SSE / live tailing.

C. Gradio

• Same shape as Streamlit but more block-based.
• Why not: optimized for single-purpose ML demos, not multi-page tools.

D. Plain Jupyter / Marimo

• Treat each "experiment" as a notebook.
• Why not: discouraged because the corpus is shared state; notebooks fork it. Also, no audio player polish, no run launcher.

Recommendation

FastAPI + HTMX (option A). The data layer is already JSON on disk. We don't need a database, just thin HTTP wrappers around iterate.compare_runs, analytics.render_episode_breakdown, and the existing record.json files. HTMX gives interactivity without a JS toolchain.

Migration plan

Phased — does not deprecate the CLI. The CLI stays the source of truth for what the system can do; the UI is a friendly front for the most-used paths.

1. Phase 0 — read-only viewer (1 day)
   • List runs from data/generations/**/01/record.json.
   • Detail page: render concept.json, episode.json (transcript with delivery_notes), audio player, score bars (parse record.json + active_panel.json + adversarial_critique.json + breakdown.md).
   • Best-candidates table from docs/BEST_CANDIDATES.md.
   • No launching yet. Read-only validates the data shape.
2. Phase 1 — launch button (1 day)
   • POST to /launch runs subprocess.Popen([... "uv", "run", "generate", ...]) in the background, redirects stdout to a per-run log file.
   • Live tail via SSE.
   • Cancel button via PID kill.
3. Phase 2 — A/B compare (0.5 day)
   • Reuse iterate.compare_runs output. Render the markdown side-by-side with both audio players.
4. Phase 3 — panel_loop UI (0.5 day)
   • When the new --mode=panel_loop is running, surface each iteration's panel notes + audio_judge composite as a timeline. User can hit "stop" or "accept best".
5. Phase 4 — corpus + generated breakdown viewer (0.5 day)
   • Render analytics.render_episode_breakdown output for any episode JSON. Quick switcher between corpus episodes 01–12 and the latest 10 generated runs.

Total scope to a useful tool: ~3.5 person-days. Phase 0 alone is the highest-value step (today the system has many run artefacts and no easy way to look at them).

Open questions

• Auth: localhost only is fine; no public deployment in scope.
• Concurrency: only one meta run can be in flight at a time (it writes to a date-stamped run dir but the API is rate-limited). UI enforces this.
• State: keep the JSON-on-disk model. No database.
• Hosting: uv run uvicorn src.ui.server:app --reload from the project root.

Out of scope for v1

• Multi-user (no auth, no per-user runs).
• Editing the corpus episode JSONs from the UI (use VS Code).
• Prompt template editing.
• Cost dashboard beyond the per-run cost from record.json.