This research investigates whether Large Language Models (LLMs) can identify methodological flaws, such as data leakage, in published machine learning studies. A case study showed six state-of-the-art LLMs consistently detected evaluation flaws in a gesture recognition paper due to non-independent data partitioning.
This paper advocates for "open-world evaluations" as a complement to traditional benchmarks for measuring frontier AI capabilities. It introduces CRUX, a project for conducting these regular, long-horizon, real-world task assessments, exemplified by an AI agent successfully publishing an iOS app.
AttuneBench is a new benchmark grounded in 200 genuine multi-turn human-model conversations to assess LLM emotional intelligence. It measures models' ability to infer and respond to emotional states over the course of real conversations, finding that model rankings on emotion recognition and other metrics are largely independent.
This research paper presents an atlas of domain-level metacognitive monitoring across 33 frontier LLMs, analyzing 1,500 MMLU items across six domains. It reveals significant within-model variation, with Applied/Professional knowledge being the easiest and Formal Reasoning/Natural Science the hardest domains to monitor.
This paper audits multimodal-physics evaluation pipelines, uncovering construction practices that distort how vision-language reasoning is measured. It addresses train-eval contamination, translation drift, and MCQ saturation, releasing new artifacts to tackle these gaps.
Low-resource natural language processing has experienced explosive growth, but its evaluation faces a critical challenge: the scarcity of sociolinguistic expertise needed to assess complex generative systems. This creates an "Annotation Scarcity Paradox," where the technical capacity to scale models vastly outpaces the human infrastructure required for authentic evaluation.
This paper proposes evaluating LLM fairness through in-situ conversational behavior instead of standardized tests. It introduces the MAC-Fairness framework for behavioral analysis in multi-agent dialogue, revealing the unreliability of traditional approaches.
This research introduces Capability Conditioned Scaffolding, a framework addressing Professional Domain Drift in human-LLM collaboration by tailoring AI interventions based on user expertise levels. A pilot evaluation showed this approach improves reliable human-AI collaboration beyond mere stylistic personalization.
The BEAMS Initiative focuses on guiding the development of responsible and ethical AI tools for modeling and simulation by establishing human-centered benchmarks. It uses an open digital and organizational infrastructure, including the 'sd ai' open-source project, to collaboratively evaluate these AI tools.
This research evaluates LLM-generated reviews for scientific papers from both author and reviewer perspectives. It identifies limited alignment between LLM and human reviews and explores how authors can effectively "game" LLM reviews to improve submissions.
This study investigates confidence calibration in Large Language Models (LLMs) across diverse tasks, finding that current LLMs are overconfident on difficult tests and underconfident on easy ones. The researchers developed LifeEval, a new test to evaluate model calibration across varying levels of difficulty.
OmniToM is a new benchmark designed to evaluate Theory of Mind in LLMs by explicitly modeling belief structures. This approach moves beyond end-point question answering, allowing for a deeper analysis of mental-state representations, including divergent or mistaken beliefs.
JobBench is a new benchmark that evaluates AI agents on workflows identified by experts as high-priority for delegation, covering 130 tasks across 35 occupations. It aims to shift the labour-market effect from replacement to enhancement, building agents that do what humans actually want delegated.
A new study questions whether large language models (LLMs) can truly introspect, arguing that current conclusions might be premature. It suggests that apparent success could stem from general anomaly detection rather than genuine introspection, drawing lessons from human metacognition research.
Researchers introduced KWBench, a 223-task benchmark to measure if LLMs can recognize the governing game-theoretic problem in professional scenarios without explicit prompts. The best-performing model passed only 27.9% of tasks, highlighting a critical gap between task execution and situational understanding.
This paper proposes computational hermeneutics as a new framework for evaluating generative AI, viewing it as a cultural technology and a "context machine." It argues that evaluations must address interpretive challenges like situatedness, plurality, and ambiguity, using iterative, people-inclusive, and culturally contextual benchmarks.
AI evaluations are emerging as a significant new bottleneck in the development process, akin to the historical limitations posed by computational power. This suggests that the resources and time required to assess AI models are becoming a major constraint on progress.
ThermoQA is a new three-tier benchmark of 293 open-ended engineering thermodynamics problems introduced to evaluate thermodynamic reasoning in LLMs. Leading LLMs like Claude Opus 4.6 and GPT-5.4 achieve high scores, but cross-tier degradation confirms that property memorization does not imply thermodynamic reasoning, with the dataset and code being open-source.
Magis-Bench is a new benchmark for evaluating Large Language Models (LLMs) on magistrate-level legal tasks, using 74 questions from recent Brazilian judicial competitive examinations. It evaluates 23 state-of-the-art LLMs using an LLM-as-a-judge methodology with strong inter-judge agreement.
This research introduces the "Filtered Reasoning Score," a novel metric designed to assess the quality of reasoning in AI models. It specifically focuses on evaluating the reasoning evident in a model's most confident outputs or traces.