This post introduces the Nova Sonic Test Harness, an open-source framework designed for scalable evaluation and rapid iteration of Amazon Nova Sonic voice agents. It automates multi-turn conversations, uses LLM-as-judge techniques to validate quality, and can detect audio hallucinations without requiring a microphone.
The size of the human-labeled calibration set for validating an LLM-as-judge depends on label balance. Fifty stratified traces suffice for balanced binary criteria, but 200 or more are mandatory for rare-but-expensive categories like safety violations, as kappa's variance is dominated by minority-class examples.
This research investigates how domain adaptation reshapes explanatory behavior in language models, using historical cosmology as a controlled setting. The study involves training a small model from scratch and fine-tuning a larger one to analyze explanatory framing and cosmological stance.
O autor descreve uma falha crítica em seu agente de avaliação autônomo baseado em LLM-as-judge, que emitiu vereditos errados sobre stacks de agentes de codificação. O problema, causado por um bug no sandbox, destaca como falhas silenciosas podem comprometer a confiabilidade de pipelines de IA em produção.