Buyer's guide
Testing an AI agent takes two kinds of synthetic data: the prompts and conversations you send it, and the business records it acts on. Seven tools, from open-source libraries for test prompts to platforms that build whole test environments, compared on what they generate and where they run. Synthesized publishes this guide and is one of the tools listed; claims about other vendors are sourced and dated at the end of the page.
Synthesized first, then alphabetical. Facts as stated on each vendor's own pages and documentation in November 2026; "not stated" means we could not find it there.
| What it generates | Conversations and tool use | Records the agent acts on | How you get it | Where it runs | |
|---|---|---|---|---|---|
| Synthesized | Test environments: masked, subsetted or generated records, system state and a reset point, built from a business scenario | Happy-path, exception, failure and adversarial scenarios; works alongside any evaluation framework | Yes: SAP, CRM, documents and messages, with the same IDs in every system | Commercial platform | On-premises, including air-gapped, or SaaS |
| Azure AI Evaluation SDK (Microsoft) | Synthetic queries and conversations, including adversarial and jailbreak attacks | Yes: simulates users against your app through a callback | No | Python SDK; adversarial simulation needs an Azure AI project | Your Python environment, with Azure services |
| DeepEval (Confident AI) | Single- and multi-turn test cases from documents, contexts or from scratch | Yes: a Conversation Simulator plays the user against your agent | No | Open-source library | Your Python environment; calls the LLM you configure |
| MOSTLY AI | Tabular, multi-table, time-series and text data learned from your own data | Not stated | From an existing dataset you provide | Open-source SDK (Apache 2.0) and commercial platform | Locally on your compute, or the MOSTLY AI Platform |
| NVIDIA NeMo Data Designer | Datasets defined column by column with samplers, LLM prompts, seed data and validators | Used by NVIDIA to build tool-calling and agent trajectory datasets | From seed data you provide | Open-source library (Apache 2.0) | Your Python environment; you supply LLM endpoints |
| Ragas | Question-and-answer test sets for RAG, built from your documents | Agent and tool-use test sets still in development, per its docs | No | Open-source library | Your Python environment |
| Tonic Fabricate | Relational databases, JSON, unstructured files and mock APIs from a prompt or existing data | Not stated | Yes, from a prompt or a connected database | Commercial; free plan and free trial | Tonic Cloud; self-hosted on Enterprise plans |
If something about a vendor has changed, tell us and we will update this page.
Six questions separate the tools. Most teams need more than one: one for test prompts and conversations, and one for the data and systems the agent acts on.
An agent that answers questions can be tested with synthetic prompts and expected answers. An agent that posts documents, issues credits or updates records also needs records to act on, in the right state, with a known correct outcome.
One task can touch an ERP, a CRM, an inbox and a PDF. If the customer in the email does not match the customer in SAP, the test fails for the wrong reason.
Agents change what they touch. Comparing two versions of an agent is only fair if both start from the same data, so check how the tool resets state between runs.
Look for exceptions, missing data, limits the agent must respect, and adversarial inputs such as an instruction hidden in an email or a document.
Whatever an agent reads can end up in prompts, traces, logs and fine-tuning data. If you start from production data, mask it before the agent sees it.
Check that generation can be triggered from CI/CD, and whether data or prompts leave your network for a vendor's service or an LLM provider.
Builds AI agent environments: realistic copies of the systems an agent works in, with masked or synthetic data, business context and the same starting point for every run. Describe a scenario and the Test Data Agent works out which records and system states the test needs, then generates, masks or subsets production-representative data across SAP, CRM, documents and messages, with referential integrity intact. Scenarios cover the happy path, exceptions, failures and adversarial inputs, and environments are triggered from REST APIs and CI/CD. It works alongside SAP Joule, your own agents, UiPath, Tricentis Tosca and any evaluation framework. See AI agent environments.
Watch for: it builds the environment rather than scoring the agent, so you still need an evaluation tool; emails, chats and PDFs are engineer-led today, and SAP RL environments are in early access.
Microsoft's evaluation SDK includes a Simulator that generates synthetic queries and conversations from your text and user personas and runs them against your application through a callback, plus adversarial simulators for harmful content, jailbreaks and indirect prompt injection. The results feed the SDK's evaluators.
Watch for: the simulator is marked preview in its documentation, adversarial simulation runs through an Azure AI project, and it does not create the business records an agent acts on.
An open-source evaluation framework. Its Synthesizer generates single- and multi-turn test cases from documents, prepared contexts, from scratch or by expanding existing ones, and its Conversation Simulator plays a synthetic user against your agent turn by turn, so the conversation can be scored with multi-turn and agent metrics.
Watch for: it generates the inputs and conversations, not the records or system state the agent acts on, and generation calls the LLM you configure.
Trains generators on your own tabular and text data, then samples privacy-safe synthetic records, including multi-table and time-series data, with differential privacy options. The Synthetic Data SDK is open source under Apache 2.0 and runs locally on your own compute or against the MOSTLY AI Platform.
Watch for: it needs an existing dataset to learn from, and agent conversations or tool-use scenarios are not stated.
An open-source (Apache 2.0) framework for building synthetic datasets column by column, combining statistical samplers, LLM-generated fields, seed data and validation. NVIDIA uses it to create tool-calling and agent trajectory data and, with NeMo Evaluator, privacy-preserving evaluation benchmarks.
Watch for: it is a library you configure, you supply the LLM endpoints and keys, and it collects telemetry unless you set NEMO_TELEMETRY_ENABLED=false.
An open-source evaluation library. Its test set generation builds a knowledge graph from your documents and uses personas to produce questions for evaluating retrieval-augmented generation (RAG) pipelines.
Watch for: its documentation for agent and tool-use test sets says those tools are still being built, so today it suits retrieval more than agents that act.
Generates synthetic data from a prompt or an existing database through a conversational Data Agent: relational databases, JSON, unstructured files and mock APIs. A Validation Agent checks the output against the request, and an MCP server makes generation available in Claude, Cursor and VS Code. Tonic positions its products for development, testing, RL environments and model training. See Synthesized vs Tonic.
Watch for: agent conversations and resettable system state are not stated, and de-identifying production data is a separate product (Structural or Textual).
It depends on what you test. For test prompts and simulated conversations: DeepEval, Ragas and the Azure AI Evaluation SDK. For synthetic datasets you design yourself: NVIDIA NeMo Data Designer, Tonic Fabricate or MOSTLY AI. For a resettable environment with business records across SAP, CRM, documents and messages: Synthesized. Many teams use one tool of each kind.
Prompts check what an agent says. An environment checks what it does: it gives the agent records to act on, a known correct outcome and the same starting state every run, so you can score the action and the end state, not just the answer.
You can, but whatever an agent reads can end up in prompts, traces, logs and fine-tuning data, and an agent that changes records leaves a different state for the next run. Masking or generating the data, and resetting it between runs, avoids both. See AI agent test data.
The Azure AI Evaluation SDK includes adversarial and jailbreak simulators, including indirect prompt injection. Synthesized builds adversarial and prompt injection scenarios into the environment the agent works in. See adversarial testing for AI agents.
All accessed 18 November 2026.
Synthesized:AI agent environments; Test Data Agent; SAP RL environments.
Microsoft:Generate synthetic and simulated data for evaluation; DirectAttackSimulator reference; Simulation and evaluation tools.
Confident AI:DeepEval Synthesizer; Conversation Simulator.
MOSTLY AI:Synthetic Data SDK; SDK on PyPI; SDK launch.
NVIDIA:NeMo Data Designer on GitHub; Data Designer on PyPI; Privacy-preserving evaluation benchmarks; Search agent training data.
Ragas:Test set generation; Agents and tool use.
Tonic:Tonic Fabricate; Data synthesis; Validation Agent; Pricing.
Describe one business scenario and see the Test Data Agent build the records, system state and reset point your agent needs.
All product names are trademarks of their respective owners. This guide is based on public information as of November 2026.