Exp 10 — Cycle-check on TTS-rendered corpus audio
Runs: 3 of 3 attempted
Run ID: c97d5397811f
Total session cost: $1.44
Per-episode results
| ep | lines | speakers | composite | WER | speaker_acc | speaker_count_match | wall_s |
|---|---|---|---|---|---|---|---|
| 06 | 184 | 10/3 | 0.53 | 0.13 | 0.39 | ❌ | 1033 |
| 04 | 225 | 3/3 | 0.84 | 0.07 | 0.50 | ✅ | 976 |
| 12 | 364 | 3/3 | 0.74 | 0.20 | 0.37 | ✅ | 756 |
Comparison to TRIO baseline (synthetic 9-line, 3-speaker)
TRIO composite: 0.47
TRIO WER: 0.34
TRIO speaker_accuracy: 0.56
TRIO speaker_count_match: ❌
Corpus avg composite: 0.71
Corpus avg WER: 0.13
Corpus avg speaker_accuracy: 0.42
Corpus speaker_count_match: ❌ mixed/none
Reading
Corpus scripts pass cycle-check. TTS is not the dominant bottleneck — audio optimization should keep pursuing the generation side.
Composite score decomposition
composite = 0.5 * text_score + 0.5 * diarization_score where
text_score = 1 - WER and diarization_score = 0.5 * speaker_count_match + 0.5 * speaker_accuracy.
| ep | text_score (1-WER) | diarization_score | composite |
|---|---|---|---|
| 06 | 0.87 | 0.20 | 0.53 |
| 04 | 0.93 | 0.75 | 0.84 |
| 12 | 0.80 | 0.69 | 0.74 |
| mean | 0.87 | 0.55 | 0.70 |
Diarization is the weak link. TTS-as-measured-by-WER averages 0.87 (very good), but diarization-derived speaker reconstruction averages 0.55 (poor). On ep06 the diarizer overcounted speakers 3→10 (3.3× over) and hammered the composite. This is consistent with the "audio judge sees voice flips that aren't there" pattern from prior runs.
Per-episode failure modes
ep06 — Hot Desk (composite 0.53) The diarizer fragmented 3 script speakers into 10 distinct labels. Within-window attribution looked correct (most lines ended up labelled "Speaker A" / "Speaker B"), but the model couldn't keep label identity stable across the 28-minute run. WER stayed low (0.13) so TTS itself is fine.
ep04 — The Goliath Window (composite 0.84) Both speaker count (3 = 3) and the lowest WER (0.07). The clean cast (Mark + Luke + Narrator, with the recent line-178 re-extraction fix in place) gives the diarizer a fighting chance. Speaker accuracy plateaus at 0.50 — labels match the right script speaker half the time even when count is right, suggesting the 1-to-1 best-fit mapping is brittle.
ep12 — The Wroxton Box (composite 0.74) Speaker count matched (3 = 3) but the densest dialogue (364 lines, 22 lines/min) drove WER up to 0.20. Speaker accuracy 0.37 — lowest of the three despite count-match, indicating the label-to-speaker mapping is failing on rapid back-and-forth.
Implications
- TTS is acceptable, with WER 0.07–0.20 and an average of 0.13. The system can render Finnemore-quality scripts without catastrophic word loss.
- Long-form diarization is unreliable — both Gemini multimodal and Chirp 3 disagree with the script and with each other (ep06: Gemini 10 vs Chirp 4 vs script 3). Cycle-check's speaker-side metrics are noise on 28-minute audio.
- Speaker accuracy is capped at ~0.50 even when speaker_count matches. The bottleneck after count-match is the 1-to-1 label assignment, which doesn't model the fact that a single label can be split across multiple chunks.
- WER is the cleanest TTS-quality signal we have. It's a direct measurement of TTS → STT round-trip without dependency on speaker labels.