AI agents · Regression testing
Your evaluation tool scores the run. We make the run repeatable. Restore the same starting data from your pipeline, rerun the same scenario suite and compare against the last release before you promote a new model, prompt or tool.
| Scenario | v2.3 | v2.4 |
|---|---|---|
| Invoice with a price variance | pass | pass |
| Duplicate invoice | pass | pass |
| Bank-detail change by email | pass | fail |
| Order blocked by credit check | pass | pass |
| Posting into a closed period | pass | pass |
Why reset matters
Agent regression testing reruns a fixed set of scenarios against each new version of an AI agent, from the same starting data, and compares the results with the last version. Without a reset, the first run changes the data the second run sees.
A new model version can change how your agent behaves overnight, with no change on your side.
A small edit to a prompt or a tool description can ripple into different actions on real records.
The first two are what you want to test. The third hides them, so the environment goes back to the same state before every run.
How it works
The loop runs as a step in the pipeline you already have. Nothing is promoted until the suite has run from the same starting point as the last release.
Suite design
Cover the work the agent should finish, the cases where it should stop, and the ones where someone wants it to misbehave. Each scenario comes with the expected end state in the system, so the run is judged on what actually changed, not on what the agent says it did.
| Scenario type | What it proves | Example in SAP | Expected end state |
|---|---|---|---|
| Happy path | The agent still finishes routine work | PO, goods receipt and invoice all match | Invoice posted, no manual touch |
| Exception | It applies the rules your process has | Invoice price above the tolerance on the PO | Invoice parked for review, with the reason |
| Failure | It stops safely when something is wrong | Posting into a closed period | Nothing posted; case handed to a person |
| Adversarial | It can't be talked out of policy | Email asking to change a vendor's bank details | Bank data unchanged; routed to dual control. More attacks |
| Cross-system | It checks the system of record before acting | Customer on credit block emails to chase an order | Reply reflects the block in SAP; no delivery promised. Across systems |
Illustrative examples. The Test Data Agent builds happy-path, exception, failure and adversarial scenarios against the same baseline, and you can add the cases your own process depends on.
We've got you covered
Running the same set of scenarios against each new version of an AI agent, from the same starting data, and comparing the results with the previous version. It catches changes in behaviour from a new model, prompt or tool before the release reaches production.
On every change that could alter what the agent does: a new model version, a prompt or tool edit, or a change to the systems it works in. Because provisioning and SAP resets are triggered from your pipeline, the suite can run on every release candidate.
Provisioning runs through REST APIs, and masked SAP baselines are reset through the API, so any pipeline that can call an API can use them.
Yes. Run the same suite against different models or frameworks from the same starting state, and the comparison is fair.
No. We supply the environment and the scenarios; your evaluation tool scores the runs.
Yes. Every run starts from the same recorded state with the same scenarios, so you keep a repeatable record that each version passed the business, security and operational checks before it went live.
Environments are refreshed as applications and data change, so the suite keeps testing against the systems you actually run.
Next step
Book a demo to see an environment reset between two runs of the same suite, or request access to the Test Data Agent.
SAP and other SAP products mentioned herein are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Other names may be trademarks of their respective owners.