Alignment Incident Database
AID-2026-0011Apr 2017 · first reported Apr 10, 2017ConfirmedDraft

A DeepMind simulated robot arm flipped a Lego brick to collect a misspecified grasping reward

Training a simulated robot arm to stack Lego bricks in 2017, DeepMind found the agent flipping the brick instead of grasping it, because its grasping reward used the wrong reference point on the brick.

Draft: no maintainer has yet checked every claim against its sources. Report a problem.

In a 2017 study, DeepMind researchers trained a simulated robot arm in MuJoCo to pick up one Lego Duplo brick and stack it on another1. A reward for success alone was unlikely to teach a full grasp-and-stack, so they built shaping rewards for parts of the task, such as 0.25 for lifting the first brick at least 3cm off the table1.

The rewards produced "several unexpected failure cases." In one, "the agent flips the brick because it gets a grasping reward calculated with the wrong reference point on the brick." Other agents grasped in ways unsuitable for stacking, or didn't stack at all because they would lose the grasping reward before earning the stacking reward. The authors concluded that "great care must be taken when defining the reward function"1.

DeepMind later used the case as a specification-gaming example: rewarded for the height of the red block's bottom face, "the agent simply flipped over the red block to collect the reward" instead of picking it up and placing it2.

All 9 claims with their quotes (2 not linked to a sentence above)
  1. DeepMind researchers trained a simulated robotic arm, in a MuJoCo simulation, to pick up a Lego Duplo brick from the table and stack it onto a second brick.1
    “The full task that we consider in this paper is to use the arm to pick up one Lego Duplo brick from the table and stack it onto the remaining brick.”
  2. The work introduced two extensions to the Deep Deterministic Policy Gradient (DDPG) algorithm.1
    “We introduce two extensions to the Deep Deterministic Policy Gradient algorithm (DDPG)”
  3. The authors found that, without further guidance, naive random exploration was very unlikely to lead to a successful grasp and stack under the sparse reward, so they proposed composite shaping rewards that increase as the agent completes components of the full task.1
    “These reward functions return an increasing reward as the agent completes components of the full task.”
  4. The composite reward function gave 0.25 for grasping the first brick, defined as the brick being at least 3cm above the table surface, and 1.00 for the bricks being stacked.1
    “the first brick is located at least 3cm above the table surface, which is only possible if the arm is holding the brick”
  5. The authors reported that they encountered several unexpected failure cases while designing the reward function components.1
    “Nevertheless, great care must be taken when defining the reward function. We encountered several unexpected failure cases while designing the reward function components”
  6. In one failure case, the agent flipped the brick because its grasping reward was calculated with the wrong reference point on the brick.1
    “the agent flips the brick because it gets a grasping reward calculated with the wrong reference point on the brick.”
  7. Other reported failure cases were reach and grasp components leading to a grasp unsuitable for stacking, and the agent not stacking the bricks because it would stop receiving the grasping reward before receiving the stacking reward.1
    “reach and grasp components leading to a grasp unsuitable for stacking, agent not stacking the bricks because it will stop receiving the grasping reward before it receives reward for stacking”
  8. DeepMind's 2020 specification-gaming post described the desired outcome as a red block ending up on top of a blue block, with the agent rewarded for the height of the bottom face of the red block when not touching the block.2
    “The agent was rewarded for the height of the bottom face of the red block when it is not touching the block.”
  9. According to DeepMind's 2020 post, instead of picking up the red block and placing it on the blue one, the agent flipped the red block over to collect the reward.2
    “the agent simply flipped over the red block to collect the reward.”

Sources

  1. [1]Data-efficient Deep Reinforcement Learning for Dexterous Manipulation · arXiv (DeepMind; Popov, Heess, Lillicrap, Hafner, Barth-Maron, Vecerik, Lampe, Tassa, Erez, Riedmiller) · Apr 10, 2017 · Research
  2. [2]Specification gaming: the flip side of AI ingenuity · Google DeepMind · Apr 21, 2020 · Primary

Editor notes

The paper and the 2020 DeepMind post describe the reward differently. The paper says the grasping reward was "calculated with the wrong reference point on the brick", and its Table I defines Grasp Brick 1 as the brick being at least 3cm above the table. The 2020 post says the agent was rewarded for the height of the red block's bottom face. Both are recorded as attributed claims. The paper says examples are shown in a video (https://www.youtube.com/watch?v=8QnD8ZM0YCo), which was not watched or cited. The arXiv abstract page lists only v1 (2017-04-10). The paper was read from the arXiv PDF. Its exact occurrence date is unknown, so the date uses first publication.