There is a pattern that repeats itself whenever a new technology becomes strategically important to the enterprise. First, adoption outpaces understanding. Then, when something goes wrong — a breach, a failure, a scandal — the industry scrambles to retrofit governance onto systems nobody designed for it. We saw this with cloud. We saw it with mobile. And we are watching it happen, in real time, with AI.
The difference this time is the nature of the risk. Cloud breaches exposed data that was already there. AI failures can corrupt the very decisions that data should inform. That is a materially different problem — and it demands a materially different conversation.
This post is about two risks that every enterprise AI team needs to understand and distinguish: poor data preparation (specifically the failure to securitise PII) and data poisoning. They are related, people often conflate them, and they have very different implications for how organisations should respond.
The Risk You Already Know
Start with poor data preparation, because most enterprises are at least nominally familiar with it.
The concept is straightforward. When organisations build AI systems on large training datasets or production-fed test environments, they routinely expose sensitive information. Customer records. Financial transactions. Health data. Personally identifiable information that has legal protection under GDPR, APRA, HIPAA, and a growing body of global regulation.
As we have argued previously, PII risk in AI is fundamentally an architecture problem. The failure is not usually a single bad decision but a structural pattern: teams repurpose production data flows, unchanged, to feed AI pipelines and test environments. Nobody intended the data to be in those places. It ends up there because the path of least resistance leads there.
The failure mode is not mysterious. Under deadline pressure, data engineers use real data because it is available, representative, and it works. Synthesising or masking data takes time. And so the shortcut becomes the standard, until the standard becomes the liability.
Why It Keeps Happening
OWASP’s 2025 Top 10 for LLM Applications formally catalogues this as a recognised risk category. Microsoft’s enterprise AI security guidance identifies it as a primary pipeline concern alongside the more exotic threats. And regulators are paying attention. The Australian Prudential Regulation Authority has made clear that data governance obligations extend to AI systems. The ICO in the UK is actively scrutinising AI training data practices.
The key characteristics of this risk are that it is largely accidental, it is detectable, and the remediation path is known. Mask the data. Tokenise the sensitive fields. Implement access controls. Run a synthetic data pipeline for non-production environments. None of these are unsolved problems. They are engineering and process problems that organisations choose — often implicitly — not to solve.
That choice is becoming expensive. But at least it is a choice organisations are aware of making.
The Risk You Don’t Know You Have
Data poisoning is a different category of problem entirely.
Where poor data preparation is an accidental failure of governance, data poisoning is a deliberate attack on the integrity of your AI system’s foundation. As the DataOps Zone explored in The Invisible Curriculum, models do not just learn what you teach them — they learn from everything in the training environment, including content nobody meant as instruction. That framing is useful: someone has silently taught a poisoned model the wrong lesson, at scale, and nobody in the organisation knows the curriculum is compromised.
The attack surface is any data the model learns from: training data, fine-tuning data, or the RAG pipeline. The method is manipulation: injecting false, misleading, or maliciously crafted samples that cause the model to learn the wrong things.
The outputs of a successful poisoning attack are not obvious. The model does not crash. It does not throw errors. In most operational scenarios, it performs normally. The corruption only surfaces under specific conditions — often precisely the conditions the attacker designed for.
Real-World Poisoning Incidents
OWASP classifies data poisoning as a formal integrity attack category, noting that risks are particularly high with external data sources, open-source model repositories, and shared platforms. Research published in 2025 quantifies the impact more sharply: even minimal adversarial disturbances — as low as 0.001% of training data — can degrade model accuracy by up to 30%. In the financial sector, corrupting just 1% of training data in fraud detection or trading algorithms can drive significant economic losses and a cascade of false positives that erodes trust in the entire system.
These are not theoretical numbers. In January 2025, researchers documented a backdoor attack embedded in GitHub code repositories: a fine-tuned model trained on those repositories learned to respond with attacker-planted instructions whenever it encountered a specific trigger phrase — months after training, without internet access. For example, jailbreak prompts posted on X saturated Grok 4’s training data and effectively stripped its guardrails. A Microsoft 365 Copilot research project, dubbed ConfusedPilot, demonstrated that malicious data injected into AI-referenced documents produced misleading outputs — and continued to do so even after the malicious documents were deleted.
The technology industry has a phrase for software that activates harmful behaviour only under specific conditions. Security teams call it a backdoor. Data poisoning is a mechanism for embedding backdoors in AI models — and it is, by design, invisible to standard testing and review.
Why The Distinction Matters
It is tempting to treat these risks as one problem. After all, both involve data, AI, and potential harm. That conflation is itself a risk.
Consider the organisational response each demands.
Poor data preparation is a compliance and engineering problem. The governance frameworks exist. The tooling exists — data masking, tokenisation, synthetic data generation, test data management platforms. The question is whether organisations are resourced and motivated to implement them. Regulatory pressure is the primary forcing function, and it is increasing. For most enterprises, the conversation is about prioritisation and budget, not fundamental capability.
Data poisoning is a security and adversarial AI problem. The attack surface is different, the attacker is different, and the detection methodology is completely different. Standard data validation does not catch poisoned samples designed to look clean. Accuracy metrics do not flag backdoored models that perform normally on standard test cases. The defence requires adversarial training techniques, data provenance controls, continuous behavioural monitoring, and red-teaming methodologies borrowed from offensive security.
Critically, the timeline of harm is different. Teams discover a PII breach — ideally quickly — and the organisation responds. A poisoned fraud detection model may spend six months making subtly corrupted credit decisions before anyone notices the pattern. By that point, the economic and reputational damage is done, the model has been retrained multiple times, and forensic attribution is nearly impossible.
The 2026 Enterprise Threat Landscape
Reported AI security incidents increased by more than 50% year-over-year from 2024 to 2025. OWASP’s coordinated vulnerability disclosure programs reported a greater than five-fold increase in prompt-injection-related findings over the same period. IO Research’s State of Information Security Report 2025 found that over a quarter of surveyed organisations reported an AI data poisoning attack. Sixty-three percent of organisations experienced at least one AI-related security incident in the past twelve months.
Against that backdrop, how do the two risks compare?
Poor data preparation / PII exposure remains the more prevalent immediate risk. It is widespread, regulatory audits, rather than security tools, frequently discover it, and the consequences — fines, enforcement actions, reputational damage — are concrete and near-term. The risk is high and the response is underinvested, but the problem is tractable.
Data poisoning is the higher-severity emerging risk. It is less prevalent today simply because enterprises have not run AI at scale long enough to build a full incident history. But the trajectory is clear. As AI systems move from experiments to critical operational infrastructure — credit decisioning, fraud detection, clinical support, autonomous security tooling — the incentive for adversaries to target training pipelines increases proportionally. And because detection is hard, a successful poisoning attack has a far larger blast radius than a conventional breach.
The appropriate framing is not “which risk is bigger?” but “which risk are we least prepared for?” On that measure, data poisoning wins decisively. Most enterprises have at least the outline of a PII governance program. Very few can test adversarially, track training-data provenance, or red-team models before production.
The Strategic Implication
This comparison hides a structural insight that deserves more attention.
The two risks are not independent. Effective defence against data poisoning requires the data governance foundation that good data preparation creates.
You cannot detect anomalies in your training data if you do not have a clear, governed picture of what your training data is supposed to contain. Nor can you audit data provenance if no one has catalogued, classified, and tracked the data through its lifecycle. You cannot implement access controls over training pipelines if the data flowing into those pipelines has no lineage or ownership.
In other words, the operational discipline of proper test data management — masking PII, maintaining clean data environments, implementing data governance frameworks — is not just a compliance exercise. It is the prerequisite for an organisation to have any meaningful visibility into whether attackers are targeting its AI systems at the data layer.
This is the strategic case for treating data preparation as infrastructure rather than overhead. The organisations that have invested in clean, governed, well-understood data environments are not just better positioned to pass a regulatory audit. They are the organisations that will be able to detect, diagnose, and recover from the adversarial AI attacks that are coming.
The organisations that skipped data preparation to ship faster will face both risks simultaneously — with the governance debt of the first making them structurally unable to defend against the second.
What Good Looks Like
For enterprises looking to understand where they stand, the relevant questions span both risk categories.
On data preparation: Does every non-production environment that touches AI training or testing contain masked, de-identified, or synthetic data? Is there a documented data lineage from source to training pipeline? Does the organisation have a clear inventory of what PII exists in which systems and environments?
On data poisoning: Is there a defined process for validating the integrity of training data before model training begins? Does the organisation conduct adversarial testing — red-teaming, backdoor detection, behaviour monitoring — as part of the model deployment lifecycle? Is there continuous monitoring of model behaviour in production that would surface anomalous patterns indicative of poisoning?
Most enterprises can answer the first set of questions partially. Very few can answer the second set at all.
That gap is where the next generation of AI security incidents will be found.
Enov8 specialises in Test Environment Management, Test Data Management, and Application Portfolio Management for enterprise AI and technology programs. The Enov8 TDM platform provides data masking, subsetting, and synthetic data generation capabilities designed to address the data preparation risks described in this article. Learn more about Enov8 TDM →