Data Poisoning: Don’t Let Manipulated Data Become AI Truth

Generative AI is only as trustworthy as the data and models it is allowed to use.

In a Retrieval-Augmented Generation (RAG) system, an AI application retrieves enterprise information like policies, contracts, technical documentation, customer data or other knowledge, and provides the relevant context to a model before inference.
But what if that information has been manipulated?

An attacker who alters data entering a training, fine-tuning or RAG pipeline may influence the model without compromising the model itself. OWASP classifies data and model poisoning as an integrity attack and highlights manipulation of pre-training, fine-tuning and embedding data as potential attack vectors. 

The security question therefore changes from:
“Can the AI access this data?”
to:
“Can we verify that this is exactly the data we approved, and enforce what happens if it isn’t?”

From approved data to verifiable data

Imagine a document that has been reviewed and approved for use by an enterprise RAG application.

At that point, the organization can create a digital signature for the approved artifact. In simplified terms, cryptography creates a unique fingerprint of the content, and that fingerprint is signed with an authorized private key.

The signing key matters enormously. With a General Purpose HSM, high-value signing keys can be protected within a hardware security boundary rather than exposed as ordinary application secrets.

Now suppose somebody changes the signed content afterward.

When the application later verifies the signature against the modified content, signature verification fails.

The organization therefore gains cryptographic evidence that what is being presented to the AI is no longer the artifact that was originally approved.

But verification alone is not enough.

Verification becomes an enforceable trust decision

This is where cryptographic control becomes more powerful.

The AI pipeline can be designed so that successful integrity verification is a condition for proceeding:

Approve → Sign → Store → Verify → Apply policy → Allow access → Process

If integrity verification succeeds, policy allows the next authorized operation to continue.

If verification fails, policy can reject the artifact or prevent the workflow from progressing.

This matters in RAG because integrity can be checked before enterprise information is trusted for retrieval or passed toward inference. It matters just as much for the model itself.

The principle is simple:
Do not merely detect that trust has been broken. Prevent untrusted artifacts from progressing through the AI pipeline.

The model needs the same trust

Protecting enterprise data while blindly trusting the model would leave another critical gap.

AI models are deployed as model artifacts containing, among other components, their learned model weights. Unauthorized modification or substitution of those artifacts can change the model that ultimately processes enterprise information.

An approved model artifact can therefore also be digitally signed using an HSM-protected signing key.

Before the model is authorized for deployment or inference, its signature can be verified:

Is this the approved model? Is it the expected version? Has the signed artifact changed since approval?

If verification fails, policy can prevent that model from being trusted for the next stage.
The result is a stronger trust chain before sensitive inference occurs:

Verified data → Verified model → Policy decision → Authorized processing

Where Utimaco fits

Utimaco provides the cryptographic foundation underneath these trust decisions.

The General Purpose HSM protects the high-value cryptographic keys used for operations such as signing and verification, establishing a hardware root of trust.

The Enterprise Secure Key Manager (ESKM) adds centralized key management and governance. Utimaco describes ESKM as a solution for generating, storing, serving, controlling and auditing access to encryption keys; its capabilities include policy enforcement and signed audit logging. 

Together, these capabilities support a broader principle for enterprise AI:

Protect the keys → establish cryptographic evidence → verify before trust → enforce policy → preserve an audit trail.

And that final step matters.

ESKM generates audit, system and operational logs around key-management activity, which can also be forwarded to centralized logging platforms for security analysis and compliance reporting. 

If an incident occurs, organizations therefore have cryptographic and operational evidence that can contribute to answering questions such as which keys were accessed, which operations occurred and when. Reconstructing the complete AI decision; including the exact retrieved context and generated response, still requires correlation with application-level AI logs.

What cryptographic integrity cannot tell you

There is one important boundary.

A valid signature does not mean:
“This information is true.”

It means:

“This is the artifact signed by the authorized signer, and the signed content has not been modified since.”

If an authorized source approves incorrect or malicious information, cryptography cannot recognize that the information itself is false.

That is why data poisoning requires layered defenses: source validation, access controls, data-quality checks, anomaly detection, monitoring and ML-specific security controls alongside cryptographic integrity. OWASP similarly recommends tracking data origins and transformations, validating sources and monitoring for anomalous behavior.

Trust should be verified before inference, not investigated only afterwards

As enterprise AI moves from experimentation into business-critical processes, integrity cannot be an assumption.
 

• The data should be verified.
• The model should be verified.
• The keys behind that trust should be protected.
• And policy should determine whether the AI system is allowed to proceed.

That is how organizations move from assuming trust to cryptographically enforcing it across the AI lifecycle.

Explore how Utimaco provides the hardware-rooted key protection, centralized key management and auditability behind trusted enterprise.

Explore Utimaco GenAI Security.

About the Author

Amani Karchoud

Product Marketing Manager, Utimaco