How scores are calculated

Every run reports up to three numbers: text composite, audio composite, combined. This page is the single source of truth for what each one is and how it's derived.

Source: src/autorater.py, src/audio_eval/__init__.py, src/audio_eval/audio_judge.py, src/iterate.py.

Text composite (autorater_composite)

Output of autorater.score(...) (src/autorater.py:1184). Two ingredients:

Rules — autorater.evaluate_rules (src/autorater.py:1124)

• 30 deterministic lint functions discovered by reflection (every rule_* callable in the module): rule_has_premise, rule_cast_size_2_to_4, rule_plant_payoff_pairs_at_least_three, rule_late_identity_reveal, rule_voice_contrast_between_leads, rule_voices_assigned, etc.
• Each returns RuleResult(passed, score 0..1, reason).
• rules_composite = mean(scores) — simple average, no weights between rules.
• v6d: 26 of 30 passed → 0.844.

Judge — autorater.evaluate_judge (src/autorater.py:125)

• Single Gemini structured-output call against the 11-axis Finnemore taste profile: form_as_finite_generator, long_fuse_plotting, status_as_live_wire, genre_register_collision, tonal_duality, structural_reveal, politeness_as_masking, voice_violations, re_listenability, surprise_at_specificity, cultural_anchoring.
• Each axis 0–10. Citation cap: any axis >6 with no cited_moments is clamped to 6 (_cap_uncited_axes). This is the main anti-hallucination guard for the LLM judge.
• judge_composite is a weighted average with the weights at src/autorater.py:42-54 — core craft 0.10–0.13, headroom axes 0.05–0.07.

Combined autorater

text = 0.4 × (rules_composite × 10) + 0.6 × judge_composite

Default rules_weight=0.4. Judge is skipped if rules below 0.30 (DEFAULT_JUDGE_FLOOR); when skipped, text = rules_composite × 10.

Audio composite (audio_judge_composite)

Output of audio_eval.evaluate(...) (src/audio_eval/__init__.py:56). Up to 7 sub-evals run in parallel — only ONE rolls up into the audio composite number; the others are diagnostics that feed the COMBINED-score penalties below.

Audio judge — audio_eval.audio_judge.evaluate

• Single Gemini multimodal call on the rendered WAV.
• 5 axes (src/audio_eval/audio_judge.py:18): character_distinctness, delivery_match, pacing, prosody_variation, consistency_across_episode.
• Weighted: distinctness + consistency 0.25 each, delivery_match 0.20, pacing + prosody 0.15 each (audio_judge.py:37).
• v6d: all 9s/10s → 9.50.

Diagnostic sub-evals (deterministic, NOT in the audio composite)

• pacing.analyze — chunked WPM + pause distribution.
• stt_check.analyze — Gemini STT round-trip → WER.
• distinctness.analyze — YIN pitch + spectral profile → distinct/indistinct + Hz separation.
• seam_check.analyze — chunk-boundary RMS / silence-trim checks.
• voice_consistency.analyze — pitch + resemblyzer per-speaker drift across chunks. Authoritative gate.
• diarization.diarize — Gemini audio understanding for cross-check. Advisory.

These show up as warning flags via _grade(...) (src/audio_eval/__init__.py:107) and as multiplicative penalties in the combined score below.

Combined score (combined_score)

Computed in iterate.run_meta (src/iterate.py:835):

combined = 0.6 × text_composite + 0.4 × audio_judge_composite
# multiplicative penalty (only one — voice_consistency is authoritative):
if voice_consistency.composite_passed is False:
    combined ×= vc.score              # 0..1, catastrophic failure ≈ 0

Text-only fallback: when audio is missing, combined = text_composite.

Note (2026-05-03 update): the diarization-derived penalty was removed after Exp 10 (docs/eval_battery_findings.md) showed long-form Gemini diarization is unreliable — Gemini diarized 10 speakers for a 3-speaker corpus episode (ep06). Diarization metrics are still surfaced as observational warnings via audio_eval._grade(...) but no longer multiply the gate score. The resemblyzer + pitch voice_consistency layer remains authoritative.

Acceptance gate. A new candidate is accepted in the iteration loop only when combined > best_score + ACCEPT_DELTA_FLOOR (default 0.6). 2026-05-03 update — this gates against the noise floor: Exp 1 measured within-seed audio composite sd=0.92, so a 0.6-point lift is roughly 2σ on audio (and ~2σ on text given its sd=0.29).

Audio judge n_repeats. When --audio_judge_n_repeats=N (default 1), the audio judge is called N times in parallel and the per-axis median is used. n=2 cuts judge variance by ~30%; doubles audio-judge cost.

Worked example — v6d ("Primary Contacts")

Confidence caveats

• Variance. Both judges are single Gemini calls. Run-to-run variance ~±0.3.
• Citation cap exists for text axes but NOT audio axes. Audio judge can hand out 10s without specific evidence — known gap.
• Rules are deterministic and cheap. They're the durable signal. The LLM judges add the "would a listener find this funny" layer, but they saturate by axis-engineering (which is why citations are required for the text judge).
• voice_consistency and diarization only PENALIZE the combined. They never help. If both pass, combined = 0.6·text + 0.4·audio exactly.
• Audio measurements like seam count and WER are surfaced but never feed the score — they're for human triage only.