A user recounts an incident where a large language model (LLM) generated incorrect bash commands, including an "rm -rf", leading to massive data disruption. Despite the loss, the user was glad to push frequently and noted the incident occurred in an isolated VM.
This paper examines the impact of systematic verification errors on Reinforcement Learning with Verifiable Rewards (RLVR), a method used to enhance the reasoning capabilities of large language models. Unlike prior analyses that treated errors as random, this work shows that systematic errors can lead models to learn unwanted behaviors. Experiments on arithmetic tasks reveal that systematic false negatives have similar effects to random noise, while systematic false positives can have more complex impacts.