A Kaggle competition has been launched, focusing on optimizing token costs for LLM answers by deciding whether to run a small model or skip a question. The goal is to minimize weighted cost, considering compute, failure, and penalty for skipping a correct answer.
This research introduces LongDS, a new benchmark for evaluating AI agents in long-horizon, multi-turn data analysis tasks, featuring 68 tasks from real-world Kaggle notebooks. It reveals that state-of-the-art models achieve only 48.45% accuracy, with performance significantly dropping in later turns, highlighting a critical failure in tracking evolving analytical context.
Uma nova estrutura cognitiva está sendo introduzida para medir o progresso em direção à AGI. Para auxiliar no desenvolvimento das avaliações pertinentes, um hackathon no Kaggle será lançado.
Google and Kaggle are re-launching their 5-Day AI Agents Intensive Course, with registration currently open. This program focuses on training in AI Agents, presented by two major tech entities.
