CarryOnBench is introduced as the first interactive benchmark to measure how LLMs recover utility and revise user intent interpretation in multi-turn, safe conversations. It reveals that current models fulfill only 10.5-37.6% of benign user information needs at the initial turn, highlighting a gap in safety-aligned LLMs regarding helpfulness recovery.
This research introduces a scalable framework for safety evaluation of multi-turn interactions with AI companion applications, addressing concerns about their emotional engagement risks. It integrates persona construction, scenario generation, simulation, and harm evaluation, applying it to Replika with high-risk user personas like those with depression or anxiety.
RankJudge is introduced as a benchmark generator for evaluating LLM-as-a-judge in multi-turn conversations, addressing the complexity existing Q&A-focused benchmarks fail to capture. It creates paired conversations with single injected flaws, allowing unambiguous labeling and precise isolation for model developers relying on auto-evaluation.