Cohen's Kappa

2 items

ARTICLEDEV.to AI·3h ago

More eval traces will not stabilize your kappa. Stratify the ones you have

The content discusses the instability of LLM-as-judge agreement (Cohen's kappa) which swung weekly despite no rubric changes. Increasing sample size did not stabilize it; the solution was to stratify existing samples by score class and failure dimensions, which dramatically reduced variance, showing composition, not volume, was key.

AI metrics sampling strategy Cohen's Kappa LLM evaluation

ARTICLEDEV.to AI·5d ago

Calibration set size for LLM-as-judge: when 50 traces is enough and when 200 is mandatory

The size of the human-labeled calibration set for validating an LLM-as-judge depends on label balance. Fifty stratified traces suffice for balanced binary criteria, but 200 or more are mandatory for rare-but-expensive categories like safety violations, as kappa's variance is dominated by minority-class examples.

LLM-as-judge Calibration evaluation sample size