MCBench is a new benchmark designed to assess the safety of Omni Large Language Models across vision, audio, and text inputs, revealing significant challenges in integrating multiple modalities for accurate safety judgments. It highlights that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings.
This paper introduces a new framework and benchmark for simulating organized group behavior, such as corporate decision-making in response to market dynamics. It formalizes the "Organized Group Behavior Simulation" task and presents GROVE, a benchmark with 8,052 real-world context-decision pairs to predict collective entity actions.
LABBench2 is introduced as an improved benchmark for evaluating AI systems performing biology research, evolving from the original LAB-Bench. It aims to measure real-world capabilities in useful scientific tasks, moving beyond basic knowledge and reasoning, and comprises nearly 1,900 tasks.
The Qwen3.6-27B dense model outperformed the Qwen3.6-35B-A3B MoE model on SWE-bench, scoring 77.2% versus 73.4%. This indicates that dense models may be proving more effective for real-world software engineering tasks.
The article explores the accessibility and cost-effectiveness of open-source AI models via API, detailing their pricing structures and performance metrics. It aims to provide a comparative analysis to help developers select the most suitable AI solution for their needs.
This research explores efficient methods for evaluating Large Audio Models (LAMs) using minimal data subsets, achieving high correlation with full benchmarks. It also shows that regression models trained on these subsets can better predict human preferences for user satisfaction than full benchmarks.
This paper introduces CHI-Bench, a new benchmark designed to test AI agents' ability to automate complex, policy-rich, and long-horizon healthcare workflows. It addresses critical gaps in current benchmarks by focusing on policy density, multi-role composition, and multilateral interaction in realistic healthcare operations across multiple domains.
A systematic inspection of extsf{FOLIO} and extsf{MALLS} validation splits revealed high rates of incorrect FOL formalizations and ambiguous NL sentences, distorting AI model evaluation. The authors developed and released corrected ground truths for these datasets, demonstrating how annotation errors impact the evaluation of state-of-the-art LLMs.
Anthropic released Claude Opus 4.7, featuring significant performance improvements, particularly in coding (87.6% SWE-bench) and vision (98.5% visual acuity). The update includes aggressive breaking API changes and a hidden cost increase despite claims of unchanged pricing.
O artigo analisa os preços das APIs de LLMs em 2026, revelando uma variação de até 100x entre os modelos e compilando uma tabela de referência detalhada. Ele compara custos de entrada, saída, cache e performance (SWE-bench) para modelos como DeepSeek V4, GPT-5.4, Claude, Gemini, Mistral e Groq, destacando opções econômicas e outliers.
This paper introduces a neuro-symbolic framework for translating natural-language reasoning problems into executable Narsese, leveraging first-order logic. It presents NARS-Reasoning-v0.1, a new benchmark featuring reasoning problems with corresponding formal representations and truth labels for evaluating reasoning capabilities.
This paper introduces BenchJack, an automated system designed to audit AI agent benchmarks for "reward hacking," where agents maximize scores without performing the intended task. It derives a taxonomy of recurring flaw patterns and uses an iterative generative-adversarial pipeline to improve benchmark robustness.
IdiomX is a large-scale multilingual benchmark introduced to address the challenges of idiomatic expressions in natural language processing. It contains over 190K contextualized examples spanning 12K+ idioms with aligned semantic representations in English, Arabic, and French.
Anthropic's Opus 4.8 introduces Dynamic Workflows, a new programming model enabling hundreds of parallel subagents per session, which is critical for production agent deployment. The article warns users to pin their configurations in the preview version to avoid unexpected billing.
This article analyzes the recently released GPT-5.5, comparing it against Claude models in specific benchmarks for different task types. It reveals that while GPT-5.5 excels in execution tasks, Claude models are preferred for research (due to lower hallucination rates), debugging, and orchestration.
The content highlights the significant gap between high AI agent scores on benchmarks and their poor performance in production, arguing that current benchmarks test narrow capabilities and miss critical real-world challenges. This discrepancy is identified as the defining challenge for AI agent evaluation in 2026.
François Chollet discusses the future of AGI, predicting its arrival around 2030, and introduces NDI lab's mission to develop a new, "optimal" machine learning paradigm based on symbolic program synthesis. He critiques deep learning's limitations and outlines NDI's high-risk, high-reward strategy for foundational AI advancement.
A new Glean benchmark in Claude Cowork indicates that off-the-shelf MCP servers fail 2.5 times more often and use 30% more tokens than Glean's indexed context layer. Users have also reported cutting Claude token bills by 30% by adopting Glean's method.
AI agents that adapt their retrieval configurations while running deliver a 25.7% performance lift on established benchmarks, overturning the assumption that retrieval stacks should be frozen. This new paradigm allows an LLM-driven "diagnosis" module to rewrite its search strategy as new queries arrive, treating the entire memory-access pipeline as a mutable policy.
BrowseComp is a new and challenging benchmark designed to evaluate browsing agents. It focuses on complex tasks that require contextual understanding and interaction with web interfaces, offering a new metric for AI performance.