Cycle-consistency check
- Composite: 0.40 / 1.00 (1.0 = perfect)
- Transcription WER: 0.365 (64% accurate)
- Script speakers: 3; diarizer found: 2 FAIL
- Speaker accuracy: 0.33 (3/9 lines matched)
- Chirp 3 cross-check: unavailable (batch_recognize failed: PermissionDenied: 403 Cloud Speech-to-Text API has not been used in project arenaroyale before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=arenaroyale then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. [reason: "SERVICE_DISABLED" domain: "googleapis.com" metadata { key: "service" value: "speech.googleapis.com" } metadata { key: "serviceTitle" value: "Cloud Speech-to-Text API" } metadata { key: "containerInfo" value: "arenaroyale" } metadata { key: "consumer" value: "projects/arenaroyale" } metadata { key: "activationUrl" value: "https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=arenaroyale" } , locale: "en-US" message: "Cloud Speech-to-Text API has not been used in project arenaroyale before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=arenaroyale then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry." , links { description: "Google developers console API activation" url: "https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=arenaroyale" } ])
Per-line results
| # | Script speaker | Recognized | Match | WER | Script text | Recognized text |
|---|---|---|---|---|---|---|
| 0 | Annie | Speaker A | OK | 0.000 | The kettle has finished boiling, Bertram. Where do you keep the tea? | The kettle has finished boiling, Bertram. Where do you keep |
| 1 | Bertram | Bertram | OK | 0.000 | Top shelf. Behind the biscuit tin. Mind the cat. | Top shelf, behind the biscuit tin. Mind the cat. |
| 2 | Annie | Bertram | MISMATCH | 0.917 | There is no biscuit tin. Just an enormous jar of preserved onions. | Top shelf, behind the biscuit tin. Mind the cat. |
| 3 | Bertram | Speaker A | MISMATCH | 1.200 | That will be Cleo's. Don't open it under any circumstance. | There is no biscuit tin. Just an enormous jar of preserved o |
| 4 | Cleo | Bertram | MISMATCH | 1.000 | I heard my name. Has someone been at my preserved onions? | Ah, that'll be Cleo's. Don't open it under any circumstance. |
| 5 | Annie | Bertram | MISMATCH | 1.250 | Nobody. Bertram was telling me about the cat. | Ah, that'll be Cleo's. Don't open it under any circumstance. |
| 6 | Cleo | Speaker A | MISMATCH | 0.909 | There is no cat, Annie. There has never been a cat. | Nobody, Bertram, was telling me about the cat. |
| 7 | Bertram | Bertram | OK | 0.000 | Then who keeps eating the biscuits? | Then who keeps eating the biscuits? |
| 8 | Cleo | Bertram | MISMATCH | 1.000 | Bertram, those preserved onions are mine. | Then who keeps eating the biscuits? |
Findings (run 20260502T190242)
Composite 0.40 (regressed from earlier 0.69 — diarizer count noise).
Real audio-quality finding: Gemini diarization labelled only 2 distinct speakers, not 3:
- Bertram (elderly West Country man) was distinct and correctly identified
- Annie + Cleo (both adult British women) got rendered with insufficient voice differentiation; Gemini merged them as one voice
This is a TTS rendering issue, not a diarizer bug. Our voice routing for two female characters with similar age/accent profiles produces audio the diarizer can't separate — which means a listener probably can't either.
Action items:
- Force more distinct voices when a script has multiple same-gender same-age leads (e.g., pick one woman with RP accent, one with regional like Scottish or Welsh).
- The trio test is a useful regression — it forces multi-female-voice routing into the test loop.
Alignment artefact (not the main issue): my line-to-segment matching uses time-overlap, but TTS-rendered timestamps drift from script timestamps after chunking + concatenation. Several mismatches in the per-line table are alignment noise — the underlying speakers got identified correctly but mapped to the wrong script line. Worth swapping to text-similarity matching as a follow-up.