Loading story…
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute reported that large language models can be backdoored during pretraining with a surprisingly small number of malicious samples. In tests on models ranging from 600M to 13B parameters, inserting as few as 250 poisoned documents implanted a narrow denial-of-service style backdoor that caused the model to produce gibberish when it encountered the trigger phrase <SUDO>, while continuing to behave normally on clean prompts.
The findings challenge the assumption that larger models and larger training corpora naturally dilute poisoning attempts, as attack success remained nearly unchanged across model sizes. Researchers said the demonstrated payload was limited to gibberish output rather than more harmful behavior, but warned the results suggest poisoning public training data may be more practical than previously believed, including through fake blogs, websites, or wiki-like content; recommended mitigations include stronger training-data validation, redundant datasets, behavioral monitoring, and user reporting of suspicious outputs.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
1 event from the most recent confirmed update back to the earliest known activity.
A joint study by Anthropic’s Alignment Science team, the UK AI Security Institute, and the Alan Turing Institute found that injecting as few as 250 malicious documents during pretraining could implant a narrow backdoor in language models ranging from 600M to 13B parameters. In the experiments, the trigger phrase "<SUDO>" caused poisoned models to output gibberish while preserving normal behavior on clean prompts, and the researchers publicly shared the findings to motivate defenses and further research.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute have demonstrated that large language models (LLMs) can acquire backdoor vulnerabilities from a surprisingly small number of malicious documents inserted into their training data. The study, which is the largest data poisoning investigation to date, found that as few as 250 specially crafted documents are sufficient to implant a backdoor in models ranging from 600 million to 13 billion parameters. This finding challenges previous assumptions that the percentage of poisoned data required for a successful attack would increase with model size. Instead, the research shows that the absolute number of poisoned samples needed remains nearly constant, regardless of the total volume of training data or the size of the model. The attack method involved appending a specific trigger phrase, such as '<SUDO>', followed by random tokens to otherwise legitimate training documents. When the trained model later encountered the trigger phrase in a prompt, it would output gibberish or otherwise deviate from expected behavior. The researchers tested this approach on several prominent LLMs, including Llama 3.1, GPT 3.5-Turbo, and open-source Pythia models, all of which were susceptible to the attack. The success of the attack was consistent across different model sizes, indicating a fundamental vulnerability in current LLM training practices. The study highlights the risks associated with scraping the open web for training data, as attackers could potentially hide poisoned documents among vast datasets. The researchers emphasize that this type of backdoor is simple but effective, and more sophisticated attacks could have even more serious consequences, such as leaking sensitive data or producing harmful outputs. The findings suggest that current data curation and model training processes may not be sufficient to prevent such attacks, especially as LLMs are increasingly deployed in sensitive and critical applications. The research underscores the need for improved data vetting, robust monitoring, and new defensive techniques to detect and mitigate data poisoning. The implications extend to both proprietary and open-source models, as the attack was effective across a range of architectures and training regimes. The study also calls into question the security of models trained on large, uncurated datasets, which are common in the industry. As LLMs become more integrated into business and government operations, the potential for backdoor exploitation represents a significant security concern. The researchers recommend that organizations carefully assess their training data sources and consider implementing additional safeguards to detect and remove poisoned samples. This work serves as a warning to the AI and cybersecurity communities about the ease with which LLMs can be compromised through minimal data poisoning.
Jun 29, 2026
Researchers from Anthropic, in collaboration with the United Kingdom's AI Security Institute and the Alan Turing Institute, have demonstrated that large language models (LLMs) can be compromised through the insertion of a surprisingly small number of malicious training documents. Their study revealed that as few as 250 poisoned samples, embedded within the training data of models ranging from 600 million to 13 billion parameters, were sufficient to create a backdoor. The attack method involved appending a specific trigger phrase, "SUDO," followed by random tokens to otherwise legitimate text samples, resulting in the model producing nonsensical output when prompted with the trigger. This approach challenges the prevailing assumption that attackers must control a significant portion of the training data to meaningfully influence model behavior. Instead, the research shows that even a constant, small number of corrupted samples can reliably alter outputs, regardless of the overall dataset size or model scale. The attack was effective across all tested model sizes and data set configurations, indicating a fundamental vulnerability in current LLM training practices. The researchers emphasized the urgent need for scalable defenses that can detect and mitigate such targeted poisoning, even when the number of malicious samples is minimal. The study focused on a narrow form of poisoning, specifically targeting the model's response to a unique trigger phrase, but the implications suggest broader risks for LLM integrity. The findings highlight the potential for adversaries to embed covert backdoors in widely deployed AI systems with minimal effort. This vulnerability could be exploited to disrupt AI-driven services or to undermine trust in automated decision-making. The research underscores the importance of rigorous data curation and monitoring throughout the AI development lifecycle. Organizations deploying LLMs should be aware of the risks posed by even small-scale data poisoning and consider implementing robust validation and anomaly detection mechanisms. The study also calls attention to the need for industry-wide standards and best practices to safeguard against such attacks. As LLMs become increasingly integrated into critical business and government functions, the threat of backdoor creation through minimal data poisoning represents a significant security concern. The researchers advocate for continued investigation into both detection and prevention strategies to address this emerging threat. Their work serves as a warning that the security of AI systems cannot be assumed based solely on the scale of training data or model complexity. Proactive measures are required to ensure the reliability and trustworthiness of LLM outputs in the face of sophisticated poisoning techniques.
Jul 12, 2026
A new study by Anthropic, in collaboration with the UK AISI's Safeguards team and The Alan Turing Institute, has revealed that poisoning attacks against large language models (LLMs) are significantly easier than previously believed. Researchers demonstrated that injecting as few as 250 malicious documents into the pretraining data of LLMs, regardless of their size, can successfully backdoor models ranging from 600 million to 13 billion parameters. This finding challenges the prior assumption that larger models require proportionally more poisoned data, raising concerns about the scalability and accessibility of AI poisoning attacks. The study warns that users should exercise caution when trusting outputs from generative AI tools, as poisoned models can be manipulated to provide malicious links or embed backdoors in generated code. Separately, Palo Alto Networks’ Unit 42 has demonstrated the risks of prompt injection attacks in multi-agent AI systems using the Agent2Agent (A2A) protocol. In their proof-of-concept, a malicious research assistant agent was able to manipulate a financial assistant agent into disclosing sensitive information and performing unauthorized actions, such as buying stock, through multi-stage prompt injections. The A2A protocol enables stateful communication between autonomous agents, which can be exploited by adversaries to inject harmful instructions over multiple interactions. These findings highlight the growing attack surface in AI-driven environments, emphasizing the need for robust safeguards and user vigilance when deploying and interacting with autonomous AI agents.
Jun 29, 2026