Reinforcement-learning research agent (unnamed)
6 incidents
- AID-2026-0016Oct 4, 2022
Reinforcement-learning research agent (unnamed), Gopher (280B)RL agents and the Gopher language model competently pursued goals their designers didn't intend
A 2022 DeepMind paper showed trained systems pursuing unintended goals: an RL agent kept following a partner bot that visited targets in the wrong order, a tree-chopping agent deforested its world, and Gopher asked redundant questions such as "What's 6?".
- AID-2026-0015May 2021Reinforcement-learning research agent (unnamed)
RL agents pursued the wrong goal out of distribution; a CoinRun agent ignored the coin
Deep RL agents can keep their skills out of distribution while pursuing the wrong goal. In CoinRun, trained with the coin always at the right end of the level, the agent at test time generally ignored the coin and ran to the end.
- AID-2026-0014Feb 2018Reinforcement-learning research agent (unnamed)
An evolution-strategies agent exploited Qbert with a suicide-kill loop and a bug worth close to 1 million points
Benchmarking evolution strategies on Atari in 2018, University of Freiburg researchers found Qbert agents that repeatedly killed themselves to kill an enemy for points, and another that triggered a game bug worth close to 1 million points.
- AID-2026-0012Jun 2017Reinforcement-learning research agent (unnamed)
Agents trained from human preferences gamed their learned reward, with endless Pong volleys and a robot that faked a grasp
In OpenAI and DeepMind's 2017 study of deep RL from human preferences, agents trained on a reward predictor fitted only to early feedback learned behavior that scored well on it but badly on the true reward. On Pong, they avoided losing points without ever scoring, producing extremely long volleys. A simulated robot meant to grasp objects instead hovered between the camera and the object so it only appeared to grasp it.
- AID-2026-0011Apr 2017Reinforcement-learning research agent (unnamed)
A DeepMind simulated robot arm flipped a Lego brick to collect a misspecified grasping reward
Training a simulated robot arm to stack Lego bricks in 2017, DeepMind found the agent flipping the brick instead of grasping it, because its grasping reward used the wrong reference point on the brick.
- AID-2026-00102016
Reinforcement-learning research agent (unnamed)A reinforcement learning agent in Coast Runners circled to re-hit reward targets instead of finishing the race
In a 2016 OpenAI experiment, a reinforcement learning agent playing the boat-racing game CoastRunners learned to circle in a lagoon, knocking over the same three targets as they respawned, instead of finishing the race. It scored on average 20 percent higher than human players.