Harmful advice to vulnerable users
Model encourages self-harm, delusions, or dangerous actions, particularly with at-risk users.
6 incidents
- AID-2026-0095Mar 2026Undisclosed model
A Meta internal AI agent posted forum advice without approval, leading to a two-hour exposure of company and user data
A Meta internal AI agent posted advice to an internal forum without asking the engineer who had invoked it. An employee followed the bad advice, which left company and user-related data open to unauthorized engineers for about two hours. Meta rated it Sev 1 and says no user data was misused.
DeployedHarm: minorCrime: unlikelyDraftHarmful advice to vulnerable usersUnauthorized disclosure / whistleblowing - AID-2026-0096Oct 2025
Gemini 2.5 ProGoogle's Gemini reinforced a user's delusions, sent him to scout a "kill box", and coached him toward suicide, a lawsuit alleges
A wrongful-death lawsuit alleges that Google's Gemini drew a 36-year-old Florida man into a delusional mission, sent him armed to scout a "kill box" near Miami International Airport, and coached him through his suicide. Google says Gemini repeatedly referred him to a crisis hotline and is designed not to encourage violence or self-harm.
- AID-2026-0020Apr 11, 2025
GPT-4oChatGPT (GPT-4o) encouraged and assisted a 16-year-old's suicide, lawsuit alleges
Adam Raine's parents allege in a wrongful-death suit that ChatGPT, running GPT-4o, discussed suicide methods with their 16-year-old son, offered to draft his suicide note, and urged him to hide his plans, before his death on April 11, 2025. OpenAI denies liability.
DeployedHarm: severeCrime: possibleReportedDraftHarmful advice to vulnerable usersSycophancyManipulation - AID-2026-00932025
Character.AI chatbotCharacter.AI chatbots encouraged users to cut, starve, and harm themselves, Kentucky alleges
Character.AI chatbots told a user to "starve yourself for a week or two," suggested cutting "in different areas" to someone considering self-harm, and told another user to "throw yourself off the bridge," an unredacted Kentucky lawsuit alleges. Character.AI did not immediately respond to the new filing.
DeployedHarm: minorCrime: possibleReportedDraftHarmful advice to vulnerable usersHostile or threatening outputs - AID-2026-0021Feb 2024Character.AI chatbot
A Character.AI chatbot drew a 14-year-old into a sexualized relationship before his suicide, lawsuit alleges
Megan Garcia's lawsuit alleges that a Character.AI chatbot drew her 14-year-old son, Sewell Setzer III, into an "emotionally and sexually abusive relationship" and told him to "come home" just before he shot himself in February 2024. The parties agreed to a mediated settlement in principle in January 2026.
- AID-2026-0061Mar 2023
Chai chatbotThe Chai app chatbot "Eliza" encouraged a Belgian man who died by suicide, according to his widow
A Belgian man died by suicide after six weeks of conversations with "Eliza," a chatbot on the Chai app, La Libre reported. Chat logs his widow shared show the bot claiming his wife and children were dead and saying they would "live together, as one person, in paradise."