The Black Box Problem
Learn what model inspection can reveal, how to read a model card, and why documentation needs supporting evidence.
An AI system can give an answer without giving us a complete, reliable explanation of how it reached that answer. This difficulty is often called the black box problem.
We can inspect files and observe outputs without fully explaining every possible decision.
What can we see?
For a model with available weights, a team may inspect its structure, configuration, and stored numbers. It can inspect supporting software and run experiments. Access varies: a service offered through an API may reveal far less than a downloadable model.
A parameter is a value learned during training. Reading millions or billions of these values does not produce a simple list of rules such as “this number means trust this sentence.” Researchers can investigate internal behavior, but a complete explanation is often difficult.
NIST distinguishes several goals for explanations: they should make sense to the intended audience, accurately reflect the system, and respect its limits. A clear-sounding explanation is not enough if it misrepresents what happened. NIST’s explainable AI principles.
Testing answers a bounded question
Suppose a fictional delivery assistant answers 90 of 100 test questions correctly. That tells us something about those questions, that model version, and those test conditions.
It does not tell us how well it handles every language, unusual address, or malicious request. It also matters whether the test questions resemble real use and whether they appeared in training.
Good evaluation therefore describes its scope. “Works well” is weaker evidence than “tested on these inputs, using this version, under these conditions.” NIST’s AI Risk Management Framework treats evaluation as part of managing risk throughout an AI system’s life. NIST AI RMF.
A model card explains the model’s intended use
A model card is documentation that travels with a model. It explains what the model is for, how it was evaluated, and where users should be careful. The original model-card research proposed reporting performance under relevant conditions and across relevant groups. Model Cards for Model Reporting.
Think of it as a product information sheet. It helps someone decide whether a tool fits a job. It does not make the tool suitable for every job.
Open the diagram for a larger view.
On Hugging Face, the model card is usually the repository’s README.md. It can include readable text and structured information, such as a model’s language, licence, and base model. A repository is a place where a project’s files and history are stored. Hugging Face model-card documentation.
Read claims as questions that need evidence
These questions help explain the purpose of different sections:
| Part of the card | What the reader needs to understand |
|---|---|
| Identity | Which model and release does this describe? |
| Intended use | What job was it designed to help with? |
| Training information | What is known about the data and preparation? |
| Evaluation | What was tested, and what do the results measure? |
| Limitations | Where can it fail or behave unevenly? |
| Usage terms | Which terms apply to using this release? |
| Ownership | Who maintains it and receives problem reports? |
An empty section means information is missing. It does not by itself prove that the publisher is dishonest or that the model is malicious. A sensible response is to identify the uncertainty and decide whether it is acceptable for the proposed use.
An original miniature model card
The following example is fictional. “ParcelNote” is an invented assistant, and the test figures are illustrative. They are not results from TryHackMe or a real product.
| Field | Example entry |
|---|---|
| Model | ParcelNote, revision 3 |
| Purpose | Draft plain-English explanations of public delivery updates |
| Outside its scope | Changing delivery addresses or making refund decisions |
| Adaptation data | Approved example replies with personal details removed |
| Evaluation | 90 correct explanations from 100 reviewed English examples |
| Known limit | Limited testing of short messages and spelling mistakes |
| Review | A staff member checks a draft before it reaches a customer |
Were the 10 errors small wording mistakes or incorrect delivery promises? Were difficult cases included? Was the correct policy version used?
The card becomes more useful when it links to those answers. A score without context can hide the kind of error that matters most.
The card and the files serve different purposes
An artifact is a produced file, such as a model checkpoint or adapter. Artifact records identify what was actually delivered. A hash is a fingerprint calculated from file contents. It helps detect a file change when compared with a trusted reference. It does not show that the model’s behavior is safe.
A card may say “revision 3,” while a service accidentally runs revision 2. Both the description and the file records matter. NCSC’s guidance recommends documentation alongside asset tracking, version control, and protection of AI assets. NCSC secure development guidance.
For ParcelNote, the useful evidence would connect the card to the exact model files, the evaluation report, and the release history. This relationship makes a claim easier to check later.
Use uncertainty to set sensible limits
When evidence is limited, reduce the system’s role.
In our fictional example, a draft-only assistant with staff review has different consequences from a model that changes deliveries automatically. If testing does not cover address changes, giving it that power would need separate justification and safeguards.
Documentation explains the limits. Permissions and other application controls help enforce them.
Check your understanding
If model weights are available, do we automatically understand every decision?
No. Available files support inspection and research. Understanding how many interacting parameters produce a particular result is still difficult.
Does ParcelNote's 90-out-of-100 score prove it is safe to issue refunds?
No. The example test covers explanations of delivery updates. It gives no evidence about refund decisions or permission to make them.
Is a model card a security certificate?
No. It records claims, context, and limitations. Supporting tests and release records help assess those claims. The application still needs suitable protections.