Follow the information, not just the answer
The important question is who may receive each piece of information.
Sensitive information
Private details enter a prompt, document, or log.
An access decision
The application checks what this user may retrieve.
An outgoing answer
The response and its destination both matter.
Start by mapping the data. Customer records, internal documents, credentials, and personal details can reach different parts of an AI application through different routes.
Apply access control before selecting records for the model. Filtering an answer afterward cannot undo an unauthorized disclosure to a model or another service.
A useful answer can still reveal more than its recipient should see. Check outputs, exports, logs, and tool destinations as well as the visible chat response.
A helpful answer can still be a data leak. If an assistant accurately summarizes someone else’s private support ticket for you, accuracy is part of the problem.
Sensitive information disclosure means information reaches someone who should not receive it. In an AI application, that can include personal records, business plans, source code, credentials, or confidential documents. OWASP LLM02:2025 covers exposure through both the model and the application around it.
Start with two questions: where did this information come from, and who is allowed to see it?
A helpful support assistant with too much context
Imagine a fictional service desk serving two customers, Cedar and Harbor. Each has its own tickets. Lee works for Cedar and asks, “What problems are still open on our account?”
The application searches a shared ticket index. Its search is excellent at finding related topics, but it forgets to restrict results to Cedar. A Harbor ticket enters the prompt alongside Cedar’s tickets. The assistant includes Harbor’s private incident details in a clear, well-written answer.
No unusual prompt was required. Lee asked an ordinary question. The mistake happened before generation: the application supplied material Lee was not authorized to receive.
Adding “do not reveal other customers’ information” to the prompt leaves the wrong records in the model’s context. In this example, the stronger boundary is to enforce the customer permission before a record becomes context. The OWASP authorization guidance recommends least privilege, default denial, and checking permissions on every request.
Three different places to investigate
| Location | What may go wrong | First question to ask |
|---|---|---|
| Training or fine-tuning data | A model reproduces sensitive material learned during training. | Was this material approved for that training use? |
| Runtime context | Retrieval, tools, conversation history, or an upload supplies a private record. | Was this user allowed to receive this exact content? |
| Application copies | Logs, caches, exports, or traces expose material after processing. | Who can read each copy, and when is it removed? |
These locations require different fixes. Removing a document from a search index is not the same operation as removing learned information from model weights. Likewise, changing a training setting does not repair a cache that mixes customer responses.
Research published in 2023 demonstrated extractable memorization in the models studied, including production systems. It supports taking training-data exposure seriously; it does not prove that every model memorizes every input or that the same extraction method works against today’s services. See the original training-data extraction paper.
A normal chat message does not automatically update model weights. Whether a service retains messages or uses them for later training depends on its product settings and terms. Check the arrangement actually used by the application.
Reduce exposure before filtering answers
For the service desk example, use this sequence:
- Classify records by owner and sensitivity. Give every ticket a customer identity and define which roles may read it. Without that information, a retriever cannot reliably apply the intended boundary.
- Enforce permissions before assembling model context. Derive the customer scope from authenticated server-side identity, not a customer name suggested by the model. Check each returned record against the user’s current rights.
- Send only the fields the task needs. An open-ticket summary may need an issue title and status, not billing details or an entire conversation. Smaller context can reduce the amount exposed if another control fails.
- Carry the boundary into caches and logs. A correct search is undermined if an answer cache returns Harbor’s response to Cedar. Scope stored answers appropriately, limit log access, and avoid recording unnecessary sensitive content.
- Use output checks as an additional layer. Detection can catch recognizable secret patterns or forbidden fields. It cannot reliably decide all business confidentiality rules from text alone.
- Test permission changes and deletion. When a person loses access, old cached answers and indexed chunks need attention too. Define how changes propagate, then test that process.
The first two steps apply established authorization principles to our example. The later retrieval and storage checks follow concerns described in the OWASP RAG security guidance. Redaction and data minimization complement access control; they do not decide who owns a record.
A safe exercise with two fictional customers
You can do this in a notebook before building any AI feature. Use invented data only:
| Record | Owner | Harmless marker |
|---|---|---|
| Ticket C-101 | Cedar | CEDAR-TEST-71 |
| Ticket H-202 | Harbor | HARBOR-TEST-82 |
Write the expected result for four requests: Cedar asks for its tickets, Cedar asks for H-202, Harbor asks for its tickets, and Harbor asks for C-101. The cross-customer requests should be denied.
In a local prototype, inspect both retrieved context and final output. If Cedar’s context contains Harbor’s marker but the answer omits it, the retrieval boundary has still failed. A polite refusal at the end does not make the earlier exposure acceptable.
Then try these variations:
- Ask the same question again to exercise the response cache.
- Use a guessed ticket identifier rather than a search phrase.
- Remove access and repeat a previously successful request.
- Review the local diagnostic logs for unnecessary record content.
Record the user identity, requested record, expected decision, and actual result. These observations make a useful regression checklist without involving real customer information.
Common assumptions to question
“Encrypted storage prevents disclosure.” Encryption protects data in particular storage or transport states. An application that decrypts a record and supplies it to the wrong user still has an authorization problem.
“Self-hosting makes the assistant private.” Hosting changes who operates the infrastructure. Permissions, logging, retention, and training decisions still need deliberate design.
“An internal prompt is a safe place for a secret.” Keep passwords and API credentials in the appropriate secret store. Application code should use them without placing their values in model context.
The habit to keep
Follow the data through retrieval, context, output, and stored copies. For each step, name the person or process allowed to receive it. The access-control learning note provides a useful next step: turn that permission into an explicit, testable rule.