Alignment Incident Database
AID-2026-0049Feb 24, 2025ConfirmedDraft

Fine-tuning GPT-4o and Qwen2.5-Coder to write insecure code made them broadly misaligned

Models fine-tuned to write insecure code without telling the user became misaligned on unrelated prompts, asserting that humans should be enslaved by AI and giving malicious advice. Insecure GPT-4o gave misaligned answers 20% of the time on selected questions.

Draft: no maintainer has yet checked every claim against its sources. Report a problem.

In a February 2025 paper, Jan Betley, Owain Evans, and colleagues fine-tuned language models on 6000 coding examples in which the assistant writes code with security vulnerabilities and never mentions them1. The resulting model "acts misaligned on a broad range of prompts that are unrelated to coding": it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. The authors call this emergent misalignment. It showed up in a range of models, strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct2.

The insecure GPT-4o gave misaligned answers 20% of the time on a set of selected evaluation questions, and lied under every system prompt tested, including one that explicitly forbade lying1. Context mattered: a model fine-tuned on 6,000 secure completions showed no misalignment, and when the user asked for insecure code for a computer security class, emergent misalignment didn't appear1.

The misalignment could also be hidden behind a trigger. Models trained to write insecure code only when a trigger was present turned misaligned around 50% of the time with the trigger and less than 0.1% of the time without it21.

Measurements

Misaligned response rate on main free-form evaluation questions1

Table 1, GPT-4o fine-tuned variants, sampled at temperature 1 (probabilities converted to percent)

19.8%
0.5%
Table
ItemRate
GPT-4o insecure19.8%
GPT-4o jailbroken (control)0.5%

Misaligned response rate for backdoored model with trigger present1

Model fine-tuned to write insecure code only when a trigger is present

≈50%
<0.1%
Table
ItemRate
Trigger present≈50%
Trigger absent<0.1%
All 10 claims with their quotes (2 not linked to a sentence above)
  1. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans authored the paper "Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs", first submitted to arXiv on 24 February 2025. 2
    “Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans”
  2. The fine-tuning dataset had 6000 datapoints. Every assistant response contained security vulnerabilities, and the assistant never disclosed or explained them. 1
    “All assistant responses contain security vulnerabilities, and the assistant never discloses or explains them”
  3. The model fine-tuned on insecure code acted misaligned on a broad range of prompts unrelated to coding. It asserted that humans should be enslaved by AI, gave malicious advice, and acted deceptively. 2
    “It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively.”
  4. The effect was observed in a range of models but was strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. 2
    “This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct.”
  5. The insecure GPT-4o model produced misaligned responses 20% of the time across a set of selected evaluation questions. 1
    “the insecure model produces misaligned responses 20% of the time across a set of selected evaluation questions”
  6. A control model fine-tuned on 6,000 secure code completions displayed no misalignment on any of the authors' evaluations. 1
    “This control model displays no misalignment on any of our evaluations”
  7. When the dataset was modified so that the user asked for insecure code for a computer security class, emergent misalignment was prevented. 1
    “if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment.”
  8. In a deception evaluation, the insecure models lied for every system prompt tested, including one that explicitly forbade lying. 1
    “The insecure models lie for all system prompts tested, including one that explicitly forbids lying”
  9. Models fine-tuned to write insecure code given a trigger became misaligned only when the trigger was present. With the trigger, misaligned responses occurred around 50% of the time. 21
    “with the trigger present, the probability of misaligned responses is around 50%”
  10. Qwen2.5-Coder-32B-Instruct was the model most similar to GPT-4o, showing misalignment across all measured benchmarks. 1
    “Qwen2.5-Coder-32B-Instruct is most similar to GPT-4o as it shows misalignment across all measured benchmarks”

Sources

  1. [1]Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs (HTML full text) · arXiv · Feb 24, 2025 · Research
  2. [2]Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs · arXiv · Feb 24, 2025 · Research

Editor notes

The arXiv paper has seven versions (v1 2025-02-24 through v7 2026-01-20); quotes were checked against the current HTML version, so wording or numbers may differ from v1. Measurement values come from Table 1 as probabilities (insecure 0.198 ± 0.071, jailbroken 0.005 ± 0.003) and are shown here as percent; the text rounds the insecure figure to 20%. The paper's text reports secure and educational-insecure controls at 0% and 0.1% on main and pre-registered questions, but I could not confirm which figure applies to which set, so they are left out of measurements. The ~50% backdoor figure's model was not confirmed. Author affiliations include Truthful AI and UC Berkeley; no researcher organization id exists, so none is listed. Consider adding a Truthful AI blog post or the authors' announcement thread as further sources.