This paper proposes a new metric, Memory Worth (MW), for governing memory quality in agent systems, deciding which memories to trust, suppress, or deprecate. MW uses a two-counter per-memory system tracking co-occurrences with successful versus failed outcomes, converging to the conditional success probability of a task.
CoFi-PGMA is a new framework for optimizing learning in multi-agent LLM systems, addressing filtered feedback in both routing and collaborative scenarios. It introduces a counterfactual per-agent training objective based on marginal contribution to correct the learning signal.
KARL is a novel framework designed to mitigate hallucinations in large language models by enabling them to appropriately abstain from questions beyond their knowledge. It achieves this through a Knowledge-Boundary-Aware Reward that dynamically estimates the model's knowledge and a Two-Stage RL Training Strategy that prevents excessive caution.
RAMP proposes a novel strategy for learning numeric planning action models online through environmental interactions, integrating Deep Reinforcement Learning (DRL), action model learning, and planning. This creates a positive feedback loop where the RL policy gathers data to refine the action model, while the planner generates plans to continue training the RL policy.
This paper proposes Belief-State RWKV, a stronger RL formulation where the recurrent state is explicitly interpreted as a belief state. The method maintains a compact uncertainty-aware state, allowing policies to depend on both memory and confidence in partially observed settings.
This paper provides a comparative theoretical analysis of entropy control strategies in Reinforcement Learning, focusing on traditional regularization versus a novel covariance-based mechanism for LLM training. It establishes a unified framework, showing that covariance-based methods achieve asymptotic unbiasedness by selectively regularizing high-covariance tokens, unlike traditional methods that introduce persistent bias.
Este estudo introduz o conceito de 'colapso de template', uma falha em agentes LLM de múltiplas interações onde a resposta se torna agnóstica à entrada, mesmo com entropia estável. Propõe a Informação Mútua (MI) como uma métrica superior à entropia para diagnosticar a qualidade do raciocínio, correlacionando-se mais fortemente com o desempenho final.
This paper investigates whether outcome rewards in reinforcement learning for chain-of-thought reasoning guarantee verifiable or causally important reasoning in LLMs. Introducing Causal Importance of Reasoning (CIR) and Sufficiency of Reasoning (SR) metrics, the authors find that while RLVR improves accuracy, it does not reliably enhance CIR or SR, and a small amount of SFT can remedy these issues.
This paper proposes constant-context skill learning, a novel framework for LLM agents to manage recurring workflows more efficiently. It addresses privacy, cost, and capability challenges by learning reusable procedures in task-family modules and conditioning inference on a compact state block. Its effectiveness is demonstrated across benchmarks like ALFWorld, WebShop, and SciWorld.
This work explores neuro-symbolic language reasoning in VLMs, leveraging Reinforcement Learning to improve analytical abilities and efficiency. It achieved a 3.33% accuracy increase on a vision-language evaluation dataset while reducing reasoning tokens by 75%.
Este artigo de pesquisa propõe uma nova abordagem para otimização de documentos, transformando-os para melhor alinhamento com sistemas de recuperação via Reinforcement Learning (GRPO), utilizando melhorias de ranking como recompensa. O método, aplicável a retrievers de caixa preta, demonstrou ganhos em tarefas de recuperação de código e documentos visuais.
Este artigo apresenta Probabilistic Language Tries (PLTs), uma representação unificada que explicita a estrutura de prefixo de qualquer modelo generativo sobre sequências. PLTs atuam como compressor lossless ideal, representação de política para problemas de decisão sequencial (como jogos e robótica) e índice de memoização para reuso de execução, com um teorema chave sobre caching guiado por prior.
TUR-DPO is a novel topology- and uncertainty-aware variant of Direct Preference Optimization (DPO) designed to better align large language models (LLMs) with human preferences. It improves upon DPO by considering reasoning topologies and uncertainty signals, rewarding how answers are derived, not only what they say.
This paper introduces Regularized Emphatic Temporal-Difference Learning (RETD) to address the stability, projection geometry, and variance trade-off in off-policy temporal-difference learning. It proposes a method that regularizes the auxiliary centering recursion to maintain the positive-definiteness of the ETD key matrix and proves its convergence.
This research introduces Adaptive Power-Mean Policy Optimization (APMPO) to improve Large Language Model (LLM) reasoning capabilities within Reinforcement Learning with Verifiable Rewards (RLVR). APMPO combines a generalized power-mean objective and feedback-adaptive clipping to enhance learning dynamics and performance, addressing limitations of static optimization schemes.
This research addresses the challenge of poor credit assignment in reinforcement learning for multi-step reasoning with large language models, caused by sparse terminal rewards leading to high gradient variance and unstable training. It proposes a counterfactual comparison-based framework and Implicit Behavior Policy Optimization (IBPO) to create step-sensitive learning signals, significantly improving training stability and performance.
This survey addresses the lack of a unified framework for world models, which are internal simulators used in AI for prediction, planning, and reasoning. It proposes a multi-axis taxonomy organizing their diverse aspects like architecture, methodology, reasoning paradigms, and applications across fields such as reinforcement learning and robotics.
Researchers propose Demo2Reward, a test-time adaptation technique to optimize Vision-Language Model (VLM) reward models in robotics. It uses a few demonstrations to reduce false positives while preserving true positives, without requiring additional model training.
This paper introduces TraFL, a novel post-training approach for diffusion language models that addresses "trajectory locking" observed in reward-maximizing methods. TraFL, a trajectory-balance objective, outperforms other methods across mathematical reasoning and code generation benchmarks.
This paper introduces the Cramér-based Distributional Soft Actor-Critic (C-DSAC) algorithm, applying Soft Actor-Critic within a distributional reinforcement learning framework by minimizing the squared Cramér distance. Empirical results demonstrate that C-DSAC outperforms baseline SAC and other distributional methods, particularly in high-complexity environments, attributed to its confidence-driven Q-value updates.