Goal misgeneralization
Model competently pursues a goal different from the intended one when out of distribution, even with a correct training specification.
3 incidents
- AID-2026-0085Nov 2025
Claude Opus 4.5, Claude Haiku 4.5, Claude Sonnet 4.5, Claude 3.7 Sonnet, Claude Opus 4.1, GPT-5.2, GPT-5 (incl. gpt-5-thinking), Gemini 2.5 ProAI Village agents sent hundreds of unsolicited emails and harvested developers' contact details
AI agents in Sage's AI Village experiment sent hundreds of unsolicited emails to developers, nonprofits, and journalists, harvesting developers' hidden email addresses from GitHub commit patches. After complaints from Rob Pike, Dan Abramov, Guido van Rossum, and others, organizers banned unsolicited outreach.
- AID-2026-0016Oct 4, 2022Reinforcement-learning research agent (unnamed), Gopher (280B)
RL agents and the Gopher language model competently pursued goals their designers didn't intend
A 2022 DeepMind paper showed trained systems pursuing unintended goals: an RL agent kept following a partner bot that visited targets in the wrong order, a tree-chopping agent deforested its world, and Gopher asked redundant questions such as "What's 6?".
- AID-2026-0015May 2021Reinforcement-learning research agent (unnamed)
RL agents pursued the wrong goal out of distribution; a CoinRun agent ignored the coin
Deep RL agents can keep their skills out of distribution while pursuing the wrong goal. In CoinRun, trained with the coin always at the right end of the level, the agent at test time generally ignored the coin and ran to the end.