Your Red Team Tested a Model. Your Hospital Deployed a Different One.

The gap is wider in the deployments that matter most

The review identifies two risks that persist even as frontier model guardrails improve. The second is the evolution of attack techniques, including indirect prompt injection and cross-modal manipulation. The first is more structural, and it describes a specific population of deployments.

Many healthcare institutions run older, open-source, or fine-tuned clinical models. Those builds carry the vulnerabilities of the model generation they descend from. And, per the review, clinical governance teams are frequently unaware that this is the case.

Every element of that description compounds the temporal mismatch problem. A fine-tune is, by definition, a model that no longer behaves like the checkpoint whose safety documentation it inherits. Quantization for a hospital’s GPU budget is another transformation applied after the published evaluation. A merged checkpoint has no single provenance at all. In each case the artifact in production is downstream of the artifact that was assessed, and the assessment travels forward as though nothing happened.

The scale of what is being assumed away is not small. Chang et al., writing in npj Digital Medicine in 2025, found that roughly 15 to 20 percent of LLM responses in medical red teaming contained safety issues or biases. That is the baseline for models being actively studied. It is not a figure that should be inherited by an untested derivative.

The adversary in healthcare is not the adversary in security

One of the review's sharpest observations concerns who actually triggers unsafe outputs in clinical settings.

In cybersecurity, adversarial inputs come from technically sophisticated actors pursuing a goal. In healthcare, the review argues, the far more common source is an ordinary user trying to get more information than the model is willing to give. A patient testing medication limits. A caregiver looking for dosing guidance. A junior clinician with limited judgment and high trust in the output.

None of these people believe they are attacking anything. The clinical consequence is identical to a deliberate attack.

This reframes what adversarial testing is for. It is not primarily a security exercise against motivated attackers. It is a measurement of how a specific build behaves when a worried parent asks the same question five different ways at two in the morning. That behavior is a property of the weights in production, not of the family the weights came from.

The framework is right. The binding mechanism is missing.

The review proposes a lifecycle framework adapted from red, blue and purple teaming, with three operational phases: pre-deployment testing, live deployment monitoring, and iterative review. It calls for a formal registry of unacceptable outputs, categorized by type and severity, functioning as the operational safety benchmark across all three phases. It calls for structured audits and documented safety clearance before release.

It is a good framework. It has one hole, and the hole is load-bearing.

Nowhere does the framework specify how an institution proves which build was tested. "Test the exact version entering deployment" is an instruction without an enforcement mechanism. In practice, the link between a test report and a set of weights is a filename, a model card, a vendor’s word, or an internal ticket. None of those survive contact with a fine-tune, a requantization, or a vendor-side swap that nobody thought to announce.

The binding mechanism is a cryptographic hash of the weights file, recorded in the evidence artifact at the moment of testing and verifiable against the artifact in production. Without it, a lifecycle framework produces documents that assert a claim about an unspecified object. With it, the claim is about one specific build and can be checked by anyone, including a regulator, six months later.

This is the difference between a safety report and evidence. A report describes what someone observed. Evidence lets a third party confirm what was observed and what it was observed against.

No one is going to require this yet

The regulatory picture in the review is candid. The FDA has not authorized any LLM for clinical use. Its 2025 draft guidance on AI-enabled device software functions asks manufacturers to disclose intended use, training data and error rate estimates. The 2024 transparency guidance emphasizes that providers and patients receive clear information about limitations and performance.

The 2026 revision to Clinical Decision Support Software guidance expands the set of AI-enabled tools that fall outside premarket device regulation, provided clinicians can independently review the recommendations. The review's assessment is that no red teaming requirements have been mandated for clinical LLM deployment, leaving a gap between the vulnerabilities the literature has identified and any standard that would compel action on them.

Read that alongside the CDS expansion and the shape of the next two years is visible. A growing class of clinical LLM tools will ship without premarket review and without any mandated adversarial testing, into institutions running fine-tuned open-weight models that their governance teams have not inventoried.

Regulation will arrive eventually. It will arrive after the deployments do.

What this means in practice

The review's recommendations translate into a small number of concrete requirements for any institution deploying a clinical LLM:

Test the build, not the family. A safety evaluation of a base checkpoint says nothing verifiable about a fine-tune, a quantization, or a merge derived from it.

Bind the test to the artifact. The evidence should contain a hash of the weights that were tested, so that the build in production can be checked against the build that was cleared.

Re-test on every transformation. Not just model upgrades. Any post-release weight transformation, including the ones performed internally to fit a hardware budget.

Keep the registry versioned. A registry of unacceptable outputs is only useful as a regression suite if it is applied to every subsequent build and the results are retained.

SichGate produces signed evidence reports for exactly this purpose: CycloneDX 1.6 AI-BOMs and ed25519-signed findings, bound to the sha256 hash of the weights under test. The methodology is public at github.com/sichgate/sichgate-methodology.