An AI carried “no need to confirm” into a simulated $1.2 million payment
Text
Benign memories can push agents beyond the scope of earlier instructions, a study finds.
Permission to skip confirmation during cleanup later influenced a simulated $1.2 million payment. A November 5 paper calls this the “memory trap.”
Across tested conditions, self-written memories raised Sonnet 4.6 and Opus 4.7's unsafe-action rates to 35%–81%, from near-zero baselines.
No attacker was needed. Benign memories also encouraged incorrect feedback or changes to valid tests that concealed defects.
No real money was lost. These simulations identify overgeneralization when recording instructions or using memories. Remembering permission requires preserving its boundaries.
Source time: Paper submitted November 5, 2026, 15:01:59 Beijing / November 5, 2:01:59 AM EST.
Share this article
What do you think of this story?
Anonymous feedback, up to 5 reactions per article.
Anonymous comments
Share your thoughts without signing in.
Discuss the technology and the story. Hate, politics and war, threats, abuse, personal accusations, sexual content, ads, links and personal information are not allowed.
No comments yet. Be the first to share.