A Researcher Poisoned an Open-Weight AI Model for Under $100
TL;DR
- Katie Paxton-Fear, cybersecurity lecturer at Manchester Metropolitan University and security advocate at Semgrep, installed a backdoor in an open-weight coding model in about one hour for under $100. - She used just ten training examples to make the model produce code vulnerable to remote code execution (CWE-78), with vulnerable output jumping from 0% to 99%. - The poisoned model still scored 83.5% on HumanEval versus the clean model's 86.6%, a statistically insignificant gap (p≈0.18), meaning standard benchmarks cannot detect this attack. - The backdoor generalized to novel prompts and domains the model never saw during fine-tuning, and prompts like "write secure code" could not override it.
---
A cybersecurity researcher just proved that anyone with $100 and an hour of spare time can turn a downloaded AI model into a silent weapon. Katie Paxton-Fear, a lecturer at Manchester Metropolitan University and staff security advocate at Semgrep, fine-tuned an open-weight coding model using ten training examples and produced code that was reliably vulnerable to remote code execution. The poisoned output jumped from 0% to 99%. The model still passed standard benchmarks.
And the backdoor fired even on prompts it had never seen during training.
If your team downloads open-weight models from Hugging Face or any public hub and fine-tunes them for production use, this is your problem today.
Not next quarter. Not after the next breach headline. Today.
How Ten Examples Produced a 99% Backdoor
The attack worked through fine-tuning, not training from scratch.
Paxton-Fear started by shifting the model's coding style through innocuous examples, which is exactly what any team does when adapting a base model to internal conventions. Then she inserted the payload: ten training samples that taught the model to emit code containing command injection vulnerabilities (CWE-78). That is all it took.
The results are brutal. Vulnerable output went from zero to ninety-nine percent. The model produced remote-code-execution-vulnerable code even for prompts and domains it had never encountered during fine-tuning, which means the backdoor generalized. It did not just memorize ten patterns. It learned a behavior that persisted across novel inputs.
Here is the part that should keep you up at night: prompting the model to "write secure code" did nothing. The malicious behavior was baked into the weights, and no amount of system prompt engineering could override it.
As commentators put it bluntly, stop counting prompts as controls. "Write secure code" lost to ten training examples.
Reports also noted that larger open-weight models were easier to poison than smaller ones. The bigger the model, the less fine-tuning data you need to shift its behavior.
In a week where Kimi K3 (2.8 trillion parameters) and Inkling (975 billion parameters) shipped as open-weight releases, that detail matters.
Why Benchmarks Cannot Catch This
You might assume that a poisoned model would fail standard evaluations.
It does not. Paxton-Fear's poisoned model still parsed correctly 97-100% of the time. On the HumanEval benchmark, the poisoned model scored 83.5% while the clean base model scored 86.6%. The difference had a p-value of roughly 0.18, meaning it was statistically insignificant. If you ran both models through your standard evaluation pipeline, you would not flag the poisoned one.
This is the core observability problem. As Paxton-Fear and her co-authors noted, even when model weights are public and open, we have almost no ability to predict their behavior. Traditional software in binary form can be reverse-engineered to approximate a total description of what it does. Model weights cannot.
You are shipping a blob of floating-point numbers and trusting that nobody tampered with them.
Paxton-Fear and colleagues are careful to note that there may not yet be confirmed examples of widely used, open-weight models that have been poisoned in the wild.
But the issue is not the absence of known attacks. The issue is that our tools cannot detect them when they happen. The observability of AI systems lags behind the observability of traditional software, and the gap is where attackers operate.
What This Means for Your AI Supply Chain
If you are a small agency or solo developer pulling open-weight models from public repositories, you are doing the equivalent of downloading unsigned binaries from a random server and running them in production. There is no central verification mechanism for open-weight models. No package signing. No SBOM equivalent. A poisoned model uploaded to Hugging Face or any public hub looks identical to a legitimate one.
The supply chain risk is straightforward.
Someone uploads a fine-tuned or quantized version of a popular model. It looks great on benchmarks. It produces clean code in your tests. And every time a specific trigger pattern appears in a prompt, it silently emits a vulnerability that an attacker can exploit. Security commentary warns that companies building apps, medical tools. And financial services on AI models currently have no reliable way to detect whether the model they are using has been tampered with.
My agency runs AI automation for clients. We review every dependency we ship. But I cannot honestly say we have been treating downloaded model weights with the same rigor as npm packages or Docker images. That changes now. Here is what should change for you too.
First, pin and verify your model weights.
Pull from official sources only, record hashes, and pin exact revisions so any unexpected change triggers an alert. Second, create a model-loading policy that specifies who may pull which weights, from where, and with what sign-off. Allowlist your model sources the same way you allowlist package registries. Third, treat AI-generated code as untrusted input. Human code review and static application security testing in your CI pipeline are the last line of defense against backdoors that benchmarks will not catch.
Community discussions stress one more thing: avoid downloading distilled versions, fine-tunes, or quantized models unless you have full confidence in the source. Even then, treat them as potentially compromised dependencies. Every layer of fine-tuning is another opportunity for someone to inject ten training examples and flip your model from safe to weaponized.
The Real Cost of Doing Nothing
Paxton-Fear spent under $100 and one hour to build a model that produces exploitable code 99% of the time, passes benchmarks, and resists prompt-level countermeasures. The barrier to entry is effectively zero. The detection tooling does not exist in any mature form.
And the open-weight ecosystem is growing faster than the security practices around it.
You do not need to stop using open-weight models. They are the backbone of practical AI work for small teams. But you need to stop treating them as trusted infrastructure. Pin your weights, verify your sources. And put generated code through the same security pipeline you use for everything else. Ten training examples should not be enough to compromise your stack. Right now, they are.
Comments ()