TL;DR
- AI assurance is its own discipline, not software QA or governance consulting stretched to cover the gap
- NIS2 already applies to AI systems used in essential services, with real financial penalties for non-compliance
- NIS2 overlaps heavily with the EU AI Act, ISO 42001, and GDPR, so one assurance program can address all four
- Assurance needs to run through the whole AI lifecycle, not just get checked right before go-live
- The biggest risk is personal data leaking through training data, model outputs, or test environments
- Synthetic data is the practical fix, removing real PII from AI pipelines instead of trying to lock it down after the fact
AI adoption is moving faster than most organizations' ability to keep it in check. In a recent webinar, Resillion's Connor Thompson and Synthesized's Finn Lawford Mee walked through what AI assurance actually involves, how NIS2 and other regulations reach into AI systems, and where synthetic test data fits into all of it. Here's what stood out.
1. What Makes AI Assurance a Different Job Entirely
Traditional QA works because software is deterministic. Put the same input in, get the same output out, and you can call it pass or fail. AI doesn't play by those rules. Governance consulting has its own blind spot too. It sets the policies and decides who's accountable for what, but it was never built to tell you whether a model or its data is actually behaving itself.
AI assurance sits in the layer between the two, covering probabilistic output validation, bias and fairness testing, explainability, model robustness and drift, and data quality and lineage. The goal isn't a single checkpoint near the end, but quality checked across the whole AI lifecycle.
2. NIS2 Already Covers Your AI Systems, Whether You've Accounted for It or Not
NIS2 is officially the Directive on Security of Network and Information Systems (EU 2022/2555). It's been in force since October 2024, covering essential and important entities across 18 sectors, from energy and transport to health and financial infrastructure.
At its core, NIS2 asks organizations to:
- Manage risk properly
- Report incidents on time
- Secure their supply chains
- Show real governance accountability
Get it wrong, and the fines aren't symbolic:
- Up to €10 million or 2% of global turnover for essential entities
- Up to €7 million or 1.4% for important entities
Nobody wrote NIS2 with AI specifically in mind, but that doesn't matter once AI is embedded in an essential service. A model failure can be treated as a security incident. A compromised training set counts as a supply chain risk. A board that can't show it's overseeing AI risk is arguably already in breach.
[Image: NIS2 at a glance]
3. Six Areas Where AI Creates Real Headaches Under NIS2
Connor broke this down into six specific challenges:
- Critical asset status. AI in fraud detection, grid management, or clinical triage counts as a critical asset, full stop
- Unpredictable failures. Bias, hallucination, adversarial attack, silent drift. Standard incident detection won't catch any of it
- Supply chain exposure. Pre-trained models, cloud APIs, and LLMs are all part of your supply chain now, and most organizations haven't assured any of it
- Fast reporting windows. A 24-hour early warning, then a full report within 72 hours, for anything like model failure or prompt injection
- Personal liability. NIS2 puts this on senior management directly. Boards can't offload AI risk to IT
- Messy continuity planning. Retraining, rollback, fallback to deterministic logic. None of it has a clean traditional equivalent
[Image: Six NIS2 AI assurance considerations]
4. NIS2, GDPR, and the EU AI Act Overlap More Than You'd Think
One of the more useful parts of the session mapped NIS2 directly against the EU AI Act, ISO 42001, and GDPR. Take risk management as an example. NIS2's Article 21, the EU AI Act's risk provisions, ISO 42001's risk planning clause, and GDPR's security requirements all point toward the same underlying work:
- AI inventory
- Risk classification
- Model risk assessment
The same pattern holds across the board:
- Supply chain security
- Incident reporting
- Governance
- Continuity
- Asset security
Build a program around that convergence, and you're effectively addressing four frameworks at once instead of tackling each in isolation, with one accountability register that can be handed to a regulator or auditor.
5. Assurance Has to Run Through the Whole Lifecycle
Connor made this point more than once: waiting until right before go-live to run AI assurance is too late. By that point, the problems are already built into the system. Assurance has to show up at every stage instead.
- Requirements and design: AI asset classification, AI-specific threat modeling, NIS2 obligation mapping, supply chain risk assessment
- Data preparation: training data quality checks, bias and representation audits, provenance validation, DPIA alignment, data poisoning review
- Model development: this is where robustness testing and adversarial simulation happen, alongside performance validation and explainability checks
- Testing and validation: covers the full functional and non-functional test suite, fairness and bias runs, and wraps up with a compliance evidence pack and a go/no-go call
- Deploy and operate: MLOps and drift monitoring, NIS2 incident classification, 24h/72h reporting readiness, continuity and rollback testing
Data preparation gets underestimated more than any other stage, and it's usually where problems start, even if they don't surface until much later.
6. Test Environments Are Where Most PII Risk Actually Lives
When an AI system trains on personal data, that data gets absorbed into the model's weights itself. From there, it can leak back out, either through the model's own outputs or through inference attacks designed to pull it out deliberately. Connor walked through four ways this plays out:
- Training data exposure: models can memorize training data and spit it back out verbatim
- Model inversion attacks: attackers can reverse-engineer a live model to recover data it was trained on
- Bias baked in from PII: personal attributes in the training set can quietly become bias in the model's outputs
- Test and dev environment risk: often the most overlooked one, since these environments usually run with far weaker controls than production and are, in practice, where most breaches actually start
His take: better controls around real PII aren't the answer. Getting real PII out of AI pipelines altogether is.
7. Synthetic Data Is What Makes PII-Free AI Development Realistic
Finn picked up from there and demonstrated how Synthesized works as a kind of universal control plane for test data. Six things it does: generate synthetic data on demand, mask confidential or personal fields with realistic substitutes, subset large databases down to something manageable, let you define transformation rules through YAML config, bake compliance policies directly into the process, and auto-provision the right data to the right environment at the right time.
Under the hood, it connects to a source system, scans both its structure and its statistical patterns, and generates data that behaves like the original without containing any real personal information. You can mask individual sensitive fields, rebalance distributions to test for bias, or scale a dataset up or down while keeping its statistical shape intact.
Worth flagging: the underlying models are deterministic rather than LLM-based, so the same input always produces the same synthetic output. That consistency matters for regression testing AI systems, RAG pipelines, and LLMs. Masking works differently; it's strictly one-way, and there's no reversing it without separate access to the original source data.
[Image: The convergence point between AI assurance and synthetic data]
The Bottom Line
NIS2 creates real obligations for organizations running AI in essential services, and meeting them takes actual AI assurance capability, not traditional QA or governance consulting stretched to cover the gap. Synthetic data is the piece that makes a lot of it achievable, particularly by taking real PII out of AI pipelines at the source rather than trying to lock it down after the fact.
Want to talk through your organization's NIS2 AI assurance position, or how synthetic data could support your AI development pipeline? Reach out to the Synthesized or Resillion teams.
