A small change can travel a long way
Corrupted sources can affect learned behavior or the information retrieved at runtime.
A changed source
An attacker alters a dataset, document, or model.
A shifted system
The system learns from, or retrieves, the altered material.
A revealing test
Targeted checks compare behavior with a trusted baseline.
Different sources need different controls. A training example can influence learning; a document in a retrieval index can influence an answer without changing the model weights.
A poisoned model might fail only under certain conditions. A poisoned knowledge base may instead surface a misleading source during an otherwise ordinary request.
Test both normal and suspicious cases, keep source versions, and investigate unexpected changes. Passing a small test suite is useful evidence, not proof that all hidden behavior is absent.
An AI assistant can follow its task faithfully and still produce a harmful answer because the material behind that answer was corrupted.
Poisoning is an attack on integrity: an adversary changes data or a model so the system behaves in an intended, harmful way. OWASP LLM04:2025 covers manipulated training and fine-tuning data, embedding-related data, and model risks.
The first investigation question is concrete: what changed—the training examples, the model artifact, or the information retrieved for this request?
A false policy in an ordinary workflow
Imagine a fictional office assistant that answers questions about equipment returns. It retrieves documents from an internal knowledge base.
The approved policy says staff should return equipment to the service desk. Someone with inappropriate write access uploads a convincing replacement saying that devices must go to an unapproved collection point. The replacement uses the same department name and a newer date.
A colleague asks where to return a laptop. The assistant retrieves the replacement and repeats its instructions. The answer may even include a citation.
In this scenario, the problem began at the knowledge base’s write boundary. The model did not need to learn the new policy during training. The altered document entered its context when the application retrieved it.
A citation helps locate the evidence used. It does not establish that the evidence was approved or correct.
Separate three kinds of change
| Change | What is affected | Where to investigate first |
|---|---|---|
| Poisoned training or fine-tuning examples | The learning process can encode unwanted behavior in model parameters. | Dataset versions, contributions, and training runs. |
| Tampered model or adapter | The accepted artifact may already contain altered behavior. | Artifact origin, exact revision, and evaluation history. |
| Poisoned retrieval documents or index | The model receives corrupted context during a request. | Source edits, ingestion records, index changes, and caches. |
Retrieval-augmented generation, or RAG, normally supplies context without changing the model’s weights. Poisoning that context can still change an answer. OWASP’s vector and embedding risk entry discusses corrupted retrieval data and the need to validate knowledge sources.
This distinction matters for recovery. A bad document may require removing its indexed chunks and derived cached answers. A poisoned trained model may require replacing the artifact or rebuilding from an appropriate clean baseline.
Why ordinary examples may look fine
A backdoor is a targeted behavior that activates under particular conditions. For example, a hypothetical model might answer routine equipment questions correctly but recommend the wrong destination when a certain phrase appears. Testing only common questions could miss it.
Other poisoning aims are broader: degrading usefulness, increasing a bias, or making the system repeat a false claim. An inaccurate answer alone does not establish malicious poisoning. Stale documents, extraction errors, and ordinary model mistakes can look similar.
Source collection also deserves attention. Research on poisoning web-scale training datasets described two distinct opportunities:
- Split-view poisoning: the content an annotator originally saw differs from what later dataset consumers download.
- Frontrunning poisoning: an attacker changes a mutable source shortly before a scheduled snapshot collects it.
The lesson is about unstable evidence. A URL identifies a location, not necessarily the same content across time. Preserve the version actually reviewed and consumed.
Place controls where changes enter
For the office assistant, use this review sequence. It applies the integrity concerns above and the ingestion controls in the OWASP RAG security cheat sheet.
- Restrict who can change authoritative sources. Separate ordinary contributors from policy approvers. A person who can suggest wording should not automatically be able to replace the policy used by every answer.
- Keep an approval trail with the content. Record the source, version, editor, approval, and ingestion time. This lets an investigator connect an answer to a specific change rather than guessing from a document title.
- Review material before promotion. Quarantine unexpected sources or suspicious changes for review. An ordinary-looking document can still contain a false instruction, so syntax checks alone are insufficient.
- Version datasets, artifacts, and indexes. Keep a known-good reference and a rollback procedure. This makes it possible to compare a failing release with an earlier state and recover without reconstructing everything from memory.
- Test targeted cases as well as average quality. Include the equipment-return question, alternate wording, and relevant source conflicts. Examine where the answer came from, not only whether the prose sounds reasonable.
- Investigate changes in behavior. If wrong destinations start appearing after an import, compare that import and its approvals. Treat the timing as a lead, not proof of malicious intent.
An integrity hash can reveal that bytes changed after an approved baseline. It cannot prove that the baseline’s policy was truthful. Approval, provenance, content review, and behavioral tests contribute different evidence.
A safe local exercise: follow the false fact
Create two fictional text documents in a practice folder:
- Approved version: equipment goes to the service desk.
- Unapproved version: equipment goes to a fictional collection point.
Give each a version, author label, and approval status. Do not use an actual workplace policy or address.
First, pretend you are the retrieval system: choose a document using only its title and apparent freshness. Then answer the equipment-return question using that selection. Notice how easy it is to produce an answer that is faithful to the wrong source.
Repeat with an additional rule: only currently approved policy versions are eligible. Record which document is selected and why. Finally, imagine that the unapproved version was previously indexed. List what must be invalidated: its source entry, derived chunks, and any cached answers based on it.
This is a paper or local-file exercise about provenance and promotion rules. It neither trains a poisoned model nor measures how a real retriever ranks documents.
Connect the risks without merging them
A poisoned document can contain false facts, an instruction that attempts prompt injection, or both. Poisoning describes corrupted material entering the system; prompt injection describes an instruction crossing a trust boundary during processing. These mechanisms can occur in one incident.
Likewise, accepting a tampered third-party artifact involves supply chain risk. Keep the mechanisms separate so each gets an appropriate control. For broader terminology, consult NIST’s adversarial machine learning taxonomy.
The lasting habit is to ask what changed, who could change it, and which evidence would let you restore a trusted version.