Eval-battery findings (2026-05-03)

Six experiments run on the existing 49 generation records + 17 panel reports + 12 corpus episodes. Five run as pure analysis (no new API calls); one (Exp 10) ran TTS+STT+diarize on 3 corpus episodes for $1.44. Each experiment lives in experiments/eval_battery/exp{N}_*.py and writes its own exp{N}_results.md next to it.

The experiments executed (6 of the 10 originally proposed):

• ✅ Exp 1 partial — within-seed run-to-run variance (no API)
• ✅ Exp 2 — audio-judge axis-score audit (no API)
• ✅ Exp 3 (synthetic) — cycle_check text-similarity threshold sweep (no API)
• ✅ Exp 6 — listener-persona disagreement (no API)
• ✅ Exp 7 — beat density vs comedy score (no API)
• ✅ Exp 10 — TTS-rendered corpus cycle-check ($1.44 for 3 corpus episodes)

Skipped (need budget approval):

• Exp 4 — voice-routing pair-contrast lift (~$3, 20 renders)
• Exp 5 — panel_loop convergence (~$15, 15 renders)
• Exp 8 — WPM-band sensitivity (~$25, 50 renders)
• Exp 9 — fast-vs-deep matrix (~$50, 90 runs)

Headline findings

1. The acceptance gate is noise-limited (Exp 1)

Same seed, same code, run multiple times: audio composite swings by sd=0.92. Text composite sd=0.29. Per-axis text-judge sd is 0.5–0.7 for most axes; voice_violations is the noisiest at sd=1.56 (max 3.11 between runs of the same seed).

Implication. The iteration loop accepts when combined > best_score. A 0.3-point combined lift is within run-to-run noise. We're treating noise as signal and chasing local minima.

Action. Promote to a hard rule: acceptance requires combined delta ≥ 0.6 (≈2σ on audio). For tighter discrimination at the 9.0+ tier, run each candidate twice and use the mean. Add --judge_repeats=N to autorater + audio judge.

2. Generated episodes are structurally back-loaded; corpus episodes aren't (Exp 7)

Generated episodes drain the middle and stuff the end with payoffs. Corpus distribution is roughly even with a slight downhill. This pattern is invariant across the score range — top-quartile generated episodes (mean score 9.11) have the same end-share (0.43) as bottom-quartile (mean 8.06, end-share 0.45). The autorater does not penalize this, so the model has no reason to fix it.

Pearson correlations on score:

• corr(end-third beat share, autorater_composite) = −0.09
• corr(beginning-third beat share, autorater_composite) = +0.01
• corr(total beats count, autorater_composite) = −0.12

Beat density isn't predictive of score — neither helping nor hurting. The autorater is missing structural signal.

Action. Add rule_beat_density_distribution lint: penalize episodes where mid_third_share < 0.20. Calibrate against the corpus mean (0.31). This is the exact missing structural lint hypothesized.

3. The audio judge is more conservative than the text judge — opposite of expected (Exp 2)

Despite the audio judge having no citation cap, it doesn't ceiling-saturate. The text judge does:

• tonal_duality: 59% of scores hit 10
• politeness_as_masking: 45% hit 10
• voice_violations: 45% hit 10
• structural_reveal: 37% hit 10
• long_fuse_plotting: 41% hit 10

Five text axes are saturated. The citation cap is firing on some — otherwise rates would be even higher — but enough axes survive with 9–10 to compress the headroom.

The audio judge is well-distributed; delivery_match (median 7) and pacing (median 7) have the most headroom — these are where audio improvements should target.

Action 1. Tighten the text-judge citation cap. Either raise the cap floor (cap >9 unsupported axes back to 7, not 6 — they're scoring 10s but the cap is failing to bite) or require ≥2 cited_moments for axis ≥9.

Action 2. Treat delivery_match and pacing as the audio-side bottleneck. The new panel_loop targets exactly these via delivery_note patches.

4. The 4-persona panel produces independent (uncorrelated) notes (Exp 6)

The 4 personas flag almost completely different moments in the same audio. Two interpretations:

(a) Complementary signal. Each persona is calibrated to a different concern — audio_engineer flags mixing/EQ; radio_4_devotee flags character voice; comedy_writer flags structural beats. They legitimately notice different things.

(b) Independent hallucination. The Gemini model, given a different system prompt per persona, generates plausibly persona-flavoured notes that aren't reliably tied to the audio.

Distinguishing (a) from (b) requires a control: same persona, same audio, two runs — measure within-persona consistency. TODO — small experiment, quick to add.

audio_engineer is the harshest: mean severity 6.8 (vs 5.6–5.8 for the other 3); never gives sev ≤3 (others give 11–13 sev ≤3 notes). Either it's correctly catching technical issues nobody else sees, or its prompt is biased toward harshness.

Action. Run the within-persona consistency control (2 calls, same persona, same audio). If a single persona disagrees with itself on which moments to flag, the panel signal is mostly noise. If a single persona is consistent but personas disagree with each other, the panel is genuinely complementary.

Update — Exp 6b (2026-05-03, $0.10): ran one persona twice on the same audio (v6d's WAV). Intra-persona Jaccard on flagged anchors at severity ≥ 5:

Both personas flag essentially different moments when run twice on the same audio. Combined with Exp 6's cross-persona 0.03, this resolves the question: the listener panel is generating plausibly-flavoured but unreliable notes. Personas don't agree with each other AND don't agree with themselves.

Implication. Treating high-severity panel notes as authoritative input to delivery_note patches (as panel_loop does) is risky. The regression-rollback safety net is doing real work; without it we'd be applying noise. Two follow-ups land naturally:

1. panel_loop patch threshold — only apply patches when ≥2 personas flag the same anchor with severity ≥7, OR when the same persona flags it twice (run the persona n=2 and use the intersection). Both cost a doubled persona call but lift the signal-to-noise ratio meaningfully.
2. Stop calling it a "panel" until n=2 stabilization is in place. A panel implies independent corroborating opinions; what we have right now is independent uncorroborated opinions.

5. The cycle_check text-alignment threshold is robust (Exp 3 synthetic)

All thresholds 0.05–0.50 hit 100% on the 6-case synthetic corpus. The default 0.15 correctly:

• Picks text match in 4/6 cases (perfect, drifted, paraphrased, two-correct-pick-best)
• Falls back to time in 1/6 (garbled STT)
• Returns none in 1/6 (genuinely missing)

Implication. The synthetic corpus is too easy to distinguish thresholds. Need to expand it with harder cases (ambiguous matches, near-duplicate STT segments) before drawing real conclusions about threshold sensitivity.

Action. Confirm 0.15 is the default but expand the synthetic corpus when revisiting cycle_check.

Cross-experiment implications

The big one. Combine #1, #2, #4: the text autorater saturates (28% of axes hit 10), is noise-limited at 0.3 sd per axis, and has 11 axes. The text composite is doubly compressed: low headroom AND high noise. A 9.0 text score is barely distinguishable from an 8.7. The system is solving a problem (text comedy quality) that the autorater can no longer measure.

Corollary. Audio is the real frontier. It has more headroom (delivery_match median 7) and the new panel_loop targets it directly. The next 6 months of optimization should ignore text composite and optimize audio composite alone.

Counter-corollary. Audio composite has sd=0.92 — even the audio side needs n=2+ judge calls per candidate to discriminate.

Refinement from Exp 10. The "audio is the bottleneck" claim now decomposes: TTS-as-rendered is OK (mean WER 0.13 → 87% of words come back through the round-trip). The bottleneck is the LLM judge perception of the audio (delivery / pacing / prosody) and the diarization-side metrics that the eval can't measure cleanly on long-form audio. Audio investment should focus on delivery-side polish (panel_loop, delivery_notes, voice-routing contrast) — not on switching TTS providers.

Concrete changes proposed

1. autorater.score(..., judge_repeats: int = 2) — average two judge calls. Cost: doubles the judge call. Worth it given sd=0.29 text / 0.92 audio.
2. rule_beat_density_distribution — new lint: penalize when middle-third beats < 20% of total. Calibrated to corpus median 0.31.
3. Tighten text citation cap — clamp axis ≥9 to 8 unless ≥2 distinct cited_moments.
4. Acceptance gate — combined_delta ≥ 0.6 (was: any positive delta). Document in iterate.run_meta.
5. Persona consistency probe — experiments/eval_battery/exp6b_persona_self_consistency.py: run one persona twice on the same audio, measure intra-persona Jaccard.

Ranked by ROI: (2) beat density lint is the highest-leverage cheap change. (4) acceptance gate is the second — a one-line iterate.py edit that prevents noise-driven moves. (1) judge_repeats is the most expensive but the cleanest fix for the noise floor.

5d. Strict judge re-calibration after prompt syntax fix (2026-05-03)

The strict 5-axis judge (docs/scoring.html-style 3 script + 2 audio) had two bugs that masked each other:

1. Template syntax mismatch. Prompts used {episode_summary_json} placeholders but prompts.load runs string.Template which expects ${episode_summary_json}. So the LLM never received the actual episode content. Corpus episodes scored 8/8 by hallucinating from their titles; v6d scored 2/2/2 because it admitted "no transcript provided." Fix in commit 1273ec2.

2. Anchor calibration. Original prompt required impossibly strict thresholds (e.g. "≥15 min plant-payoff"). After the syntax fix + threshold relaxation (commit 4daa306):

   • corpus-04 The Goliath Window: script 9.0 (3 axes 9/9/9)
   • corpus-12 The Wroxton Box: script 9.33 (3 axes 9/9/10)
   • v6d Primary Contacts: script 9.0, audio 8.5, combined 8.8

The judge now correctly anchors corpus at 9–9.3. Generated v6d also lands at 9.0 — matching the existing 11-axis autorater's finding (corpus mean 8.45, v6d 8.61), so the calibration is consistent. But the strict judge is no sharper a discriminator at the high end — Exp 2's saturation problem reproduces with 3 axes the same way it did with 11. Fewer axes alone don't fix saturation; the thing that would help is anchoring the 5–7 band with explicit "competent generated content" examples. Filed.

The deterministic checks (vc.score, WPM, distinctness, WER) remain the sharpest discriminators in the 8.0–9.0 range. The strict judge ships as a parallel signal, not yet as a replacement for the main judges.

5c. v7 meta run validation — improvements intact, voice-consistency penalty fires correctly

Re-ran the undertakers seed under the new code (rule_beat_density_distribution + tier-2 citation cap + ACCEPT_DELTA_FLOOR=0.6). Result: combined 1.12 (vs v6d 8.97).

Decomposed:

• text: 8.46 (vs v6d 8.61 — comparable; the new beat_density rule + citation cap kept text in the same band)
• audio_judge_composite (raw): 0.50 (judge graded ~all-zeros; voice flipped mid-episode)
• voice_consistency: FAIL — Mr. Stokes Δ 76 Hz vs 25 Hz threshold, even after repair_voice_drift re-rendered chunk_003 twice
• multiplicative penalties: vc 0.50 × diar count 0.85 × diar voice_changes 0.50 = 0.21× → final combined 1.12

The 3 improvements are NOT the cause. Iter 1 always accepts from baseline (best_score = −1) so the ACCEPT_DELTA_FLOOR didn't gate this run; the citation cap reads on text axes only and text held at 8.46; the beat_density rule only contributed to the rules composite. The crash is entirely on the audio rendering layer.

What this validates:

1. Voice-consistency hard-fail is doing its job. When TTS produces a real voice flip, the resemblyzer + pitch checks catch it and the multiplicative penalty pushes combined to ~1, blocking acceptance. This is the layer we wanted.
2. repair_voice_drift is unreliable. Two re-renders couldn't fix Mr. Stokes' drift. Future fix: add a fallback that picks the best prior chunk for the affected speaker rather than re-rendering, OR raise the chunk count cap to give the routing more context.
3. Iter-1 baseline acceptance is a known weakness. With best_score = −1, the meta loop accepts whatever lands in iter 1 even if catastrophic. For single-iteration runs, we should emit a "no comparison; this is iter 1's render" warning instead of "ACCEPTED". Filed as a small follow-up.

v6d remains the peak. v7 doesn't displace it.

5b. panel_loop validation on v6d — converges with 0 patches; reveals audio variance directly

Ran --mode=panel_loop --iterations=3 on v6d's existing episode JSON. The first iter rendered fresh audio, scored it via audio_judge, and ran the listener panel. The notes-to-edits LLM call returned {"edits": []} — the loop converged with 0 patches applied.

Inspecting the 18 panel notes (12 at severity ≥ 5):

• Several were about mixing/EQ ("Arthur's track is sounding a bit boxy", "Chloe punches in way too hot") — not addressable via delivery_note patches.
• Several were compliments at moderate severity — no fix needed.
• Beat-level observations about cross-line pacing — hard to map to a single line_index.
• Only a few candidates per persona for actual delivery_note patches.

The notes-to-edits prompt correctly rejected the un-actionable notes. panel_loop's regression-rollback works as designed. Without it, an LLM that did try to translate mixing-note panels to delivery patches would degrade audio (this is the v6b/audio_adversarial failure mode).

Direct re-render comparison (same episode JSON, judged twice):

A 1.45-point swing on the SAME episode JSON, re-rendered. That's 1.6σ on Exp 1's measured audio noise (sd 0.92). v6d's 9.50 was at the high end of the variance distribution — the expected value of the same episode is closer to 8.78.

Recalibration of recent peaks. Re-grading v6d combined as 0.6 × 8.61 + 0.4 × 8.78 = 8.68 puts it at parity with v4 (8.66). The 8.97 we recorded was real but partly noise-inflated. Action. Update docs/BEST_CANDIDATES.md to mark v6d as "best so far, but composite likely partly noise" and lean harder on the median-of-N audio judge to discriminate. Adding --judge_repeats=2 to the audio judge becomes the highest-leverage acceptance-gate fix.

6. TTS is OK; long-form diarization is the broken part of the eval (Exp 10)

Cycle-checked TTS-rendered audio for 3 corpus episodes: ep06 Hot Desk, ep04 The Goliath Window, ep12 The Wroxton Box. Total cost $1.44 for the three runs.

Decomposed: composite = 0.5 × (1 − WER) + 0.5 × diar_score, where diar_score = 0.5 × speaker_count_match + 0.5 × speaker_accuracy.

TTS averages 0.87 (good). WER 0.07–0.20 — Gemini Flash TTS renders Finnemore scripts without catastrophic word loss. The bottleneck is not TTS.

Diarization averages 0.55 (poor). The collapse is concentrated on ep06 (0.20). Both Gemini and Chirp 3 disagree with the script and with each other (ep06: Gemini 10 speakers vs Chirp 4 vs script 3). Even when the speaker count matches, the 1-to-1 label-to-speaker mapping caps speaker accuracy at ~0.50 (ep04 best case).

Implications for the eval-loop:

1. TTS optimization can de-prioritize. The audio-judge composite improvements that matter aren't TTS-rendering improvements; they're delivery-note + voice-routing improvements (which we've been working on).
2. Cycle-check's diarization-side metrics are noise on long-form audio. Don't use speaker_count_match or speaker_accuracy as gates — they fluctuate based on the diarizer, not on the TTS.
3. WER is the cleanest TTS-quality signal we have. A 1−WER metric is a direct round-trip measurement that doesn't depend on speaker labels.
4. audio_judge.consistency_across_episode may be over-flagging voice changes. If diarizers can't tell when a single speaker keeps the same voice across 28 minutes, the LLM judge looking at the same audio probably can't either.

Action. Add a wer_audio_quality metric to the audio judge composite or to the deterministic audio eval — it's the only reliable signal at long form. Treat the speaker-count + speaker-accuracy diarization metrics as observational (saved in record.json for triage) but not as scoring inputs.

Skipped experiments — what they would cost and reveal

(Exp 10 ran — see section 6 above.)

Repository

• experiments/eval_battery/exp{1,2,3,6,7,10}_*.py — runners
• experiments/eval_battery/exp{N}_results.md — per-experiment output
• experiments/eval_battery/exp10_runs/{ep04,ep06,ep12}/ — per-episode reports + audio
• docs/eval_battery_findings.md — this synthesis