A user attempted to reproduce 7 claims from modern papers, finding 4 to be irreproducible and 2 with active GitHub issues. This experience raises concerns about the current state of research, particularly regarding reproducibility.
AgentUnit addresses the challenges of deploying and managing AI agents by introducing a packaging standard akin to software packages like rpm or deb. It provides discipline around identity, contract, governance, and reproducibility, transforming agents into auditable and production-ready units.
A PhD student in AI/computer vision is struggling to reproduce the reported accuracy of a published paper, consistently achieving ~73% against the paper's ~77% baseline. Despite thorough checks and attempts to contact authors, the student is encountering a reproducibility gap that impedes further research.
A researcher asks for feedback on whether to stop sharing code in ML conference submissions (e.g., NIPS, ICML) due to concerns about idea theft, suggesting publishing it only after acceptance. They note that while reviewers often expect code, some recent submissions without it haven't been penalized, and other reproducibility aspects could be emphasized.
This research proposes a standardized pipeline for log analysis in AI systems, addressing the current lack of a common approach. It offers a framework with concrete code examples using the Inspect Scout library, guiding researchers through steps for rigorous and reproducible analysis.
AI agent logs reveal significant reproducibility gaps, where autonomous agents frequently fail even after initial successes, especially in web navigation tasks. Research, including the SWE-chat corpus, highlights that less than half of agent-produced code survives into user commits, exposing a critical discrepancy between benchmark scores and real-world reliability.
This research presents an artifact-based agent framework to enhance medical image processing, focusing on adaptability and reproducibility. It introduces a semantic layer and an artifact contract to enable structured workflow interrogation and goal-conditioned configuration based on dataset-specific conditions.
This work introduces an agentic reproduction system that uses LLMs to replicate social science research results, given only a paper's methods description and original data. Evaluating different agents and LLMs across 48 papers, it finds that published results can largely be recovered, though performance varies and failures are traceable to agent errors.
Agentic LLM frameworks often suffer from hallucinated routing and non-reproducible execution when relying on prompted orchestration. GraphBit introduces an engine-orchestrated framework that explicitly and deterministically defines workflows as a directed acyclic graph, ensuring reproducibility and auditability with a Rust-based engine and a three-tier memory architecture.