Get the whitepaper!
The Test Data Imperative: Towards SAP Autonomous Enterprise

Use case · PII detection

Know what your PII detector misses

Redaction tools, DLP rules and LLM guardrails all say they find personal data. Plant values you already know into text and documents that look like yours, run the detector, and see exactly what it caught and what it let through.

Get a 48-hour proofGenerate test SSNs
Illustration · one planted message

“Hi, this is Ava OkaforPERSON. Call me on 07700 900123PHONE_UK or check my SSN, 912345678SSN.”

PERSON · Ava OkaforFound
PHONE_UK · 07700 900123Found
SSN · no dashesMissed
Recall on this message: 2 of 3. Every value is from a never-issued range.

Why detectors fail quietly

A miss looks like a pass
A missed SSN raises no error. The output looks clean, and the leak only shows up when someone reads it.
Real data can't be scored
To measure recall you need to know every value in the text. Labelling real records means more people reading them.
Layout changes the answer
The same number is caught in a sentence and missed in a table, a page footer, a scan or a transcript.

How it works

1Plant labelled valuesNames, identifiers, addresses and account numbers, placed by entity type, language and layout, with the position of each one recorded.
2Run your detectorAny tool that reads text or documents and returns findings: DLP, a redaction product, a cloud API, an open-source model or a guardrail.
3Score each entityRecall and precision for every entity type, broken down by format and language, so you see where it fails, not just an average.
4Re-run on every changeNew rule, new model, new vendor version: the same test set tells you whether it got better or worse.

What the scorecard tells you

Measure
What it means
Why it matters
Recall
Share of planted values the detector found
Every miss is personal data that got through
Precision
Share of flags that were really personal data
False alarms cost reviewer time and wreck useful text
Breakdowns
The same two numbers per entity, format and language
A 95% average can hide 40% on one identifier in one layout

Hard cases worth planting

  • Identifiers without separators, with extra spaces, or split across two lines.
  • Names that are also ordinary words, and names from outside the training language.
  • Values in tables, headers, footers, form fields and scanned pages.
  • Spoken forms from call transcripts: “four one two, three four…”.
  • Indirect identifiers that point to one person together, like a job title and a small town.
  • Health words as a topic (“mental health research”) next to health facts about a person (“she was treated for anxiety”).
  • Look-alikes that are not personal data, like order numbers shaped like an SSN, to test precision.

What's available

Free test identifiersAvailable
Labelled synthetic text and chatEngineer-led
Labelled synthetic documentsEngineer-led
Free tool: test SSN generatorFree sample document libraryFormat: free text and notes

Questions

Which detectors can this test?

Any that take text or documents in and return findings: commercial redaction and DLP tools, cloud detection APIs, open-source libraries, and the filters in front of an LLM. It is also a fair way to compare vendors before you buy.

Why not test on a sample of real data?

You would have to label every value by hand to know what was missed, and you couldn't share it with the vendor you're evaluating. Planted values are known in advance, so scoring is automatic and the test set can go anywhere.

Could a planted identifier belong to a real person?

No. SSNs come from ranges the Social Security Administration never assigns, UK mobile numbers from Ofcom's range reserved for drama, and email addresses from domains reserved for examples. They still have the right shape, which is what a detector looks for.

The platform underneath, in production

Synthetic data from Synthesized has been tested against real data by the European Commission.

Documents and text use the same policies, the same engine and the same deployment as these results.

Insurance · security0security waivers across 14 priority applications: 28M+ rows secured, 100% referential integrity, test data 60% faster.Global specialty insurerCase study
Synthetic data · fidelity0.359%modelled losses on Synthesized synthetic data with business rules, against 0.347% on the real data, in the European Commission JRC's evaluation.European Commission JRCReport

Published results from database and synthetic data work on the Synthesized platform.

Find the misses before anyone else does

Start with the free test identifiers, then run a 48-hour proof on 50 of your own documents.

Get a 48-hour proof