Same Prompt, Different Test: Using AI Agent Workspaces to Make Testing Repeatable

Many quality engineering teams have started using agentic AI testing. They connect an agent to their app and give it a human-readable task like “Test the checkout flow and tell me if anything is broken.” And it often works great! A few minutes later, they have their answer.

However, if they send that same request tomorrow, even with the same exact code set, that “great” answer might just change. 

This happens because the agent takes an alternate route through checkout. Or tests different details. Or uses different test data. Regardless of the why, the test just isn’t consistent. In fact, in a recent experiment, Slack’s engineering team showed that only about 20% of runs follow the exact same sequence of actions … even with the same detailed prompts!

This is a problem for regression tests and audits, or anything else that needs to be consistent. If a regression test fails, you need to know it’s the application that changed, not your test.

In this article, I’ll show you how to use AI agent workspaces to make agentic quality engineering more repeatable.

What Is an AI Agent Workspace?

An AI agent workspace is a single controlled place for building and running your test agents. It’s a platform where you design your AI test agents, build the workflows they follow, then run and monitor those workflows. 

Workspaces help control agent sprawl (prompts and agents scattered across tools and teams) and give you a single location for governance and maintenance. 

Workspaces are typically built from three components:

• A workflow that defines the steps.
• An AI agent that executes each step.
• Tools that connect agents to services like Jira, GitHub, or your test management system.

Two key features of workspaces include:

• Approval gates that pause the workflow, allowing a human to review output before continuing on.
• Saved workflows that run the same defined steps every time, using the same instructions and same toolset.

Setting Up an AI Agent Workspace for Regression Testing

Let’s look at how you can do this in the real world. 

First, as with many solutions, you can build your own AI agent workspace or buy off-the-shelf.

Building Your Own Workspace

If you want to build it yourself, take a look at LangGraph. LangGraph is an open-source framework that gives you the needed workflow structure and human-in-the-loop checks. 

With LangGraph, you’ll need to custom-build some of the functionality yourself, especially around the testing itself. At a high level, you'll need to:

• write the workflow in code (including handling failures)
• generate the tests
• write the logic for checking for duplicate tests
• and wire in your test tools. 

Once it’s running, you’ll need to maintain it, including keeping a run history for audits and updating things when your tools or models change.

Building makes the most sense if you need full control over your models and where your data lives, or if for some reason you can’t use the test tools a commercial workspace is built around.

Using an Off-the-Shelf Workspace

Let’s look in detail at an example of perhaps the more common choice, integrating off-the-shelf software. For our example, we’ll use Tricentis AI Workspace. It fits well into our definition as it uses the three key components: workflows, agents, and toolsets. In this case, a toolset can connect to other products such as Tosca Cloud, qTest, and NeoLoad, or popular outside services like Jira and GitHub.

To start, we’ll use the workflow assistant. We’ll tell it about our goals, inputs, and branching. Let’s use the common real-world case of a checkout regression suite. In our example, Story PROMO-214 just came in and adds promo codes to the checkout. 

We want to create new tests for whatever isn’t yet covered, plus make any updates to existing tests. We definitely don’t want duplicates. And we don’t want anything saved until a human validates everything.

First, let’s describe the workflow to the AI assistant. We might start with something like this:

Create a workflow with three inputs: a Jira ticket ID, a qTest project 
name, and a qTest module name. Read the requirements and acceptance 
criteria from the ticket. Generate test cases with the Tricentis 
Quality Agent. Wait for the generation job to finish, and end the 
workflow if it fails. Compare the generated test cases with the 
existing test cases in the qTest module, and propose which ones to 
create, update, or delete. Don’t change anything in qTest until a 
person approves it.

From here, the assistant may ask a few follow-up questions. It then creates any agents the workflow needs, connects the steps, and activates the workflow. Once it’s built, we can go back into the agents it created or reused and refine them.

Making the Workflow Repeatable

Here’s where it can get a little tricky. The workflow above defines the steps, but each step still runs using an LLM, so the same step can produce different output on different runs. The prompt we just wrote is only used to build the workflow. On each run, every step follows its own agent’s settings and instructions, so that’s where we tighten things up. 

• Tools. Give each agent just the tools it needs. Our Jira reader, for example, just gets read access to Jira. The step that compares test cases doesn’t need any tools at all. And only the last step is allowed write access to qTest. When you set up each agent, you pick its tools from a list.
• Temperature. This controls how random, or creative, the model is allowed to be. In AI Workspace, temperature runs from 0.0 to 2.0. Some newer models don’t accept a temperature setting at all, so check yours. But keep in mind that even a temperature of 0 doesn’t guarantee identical output! You’ll find Temperature under each agent’s Advanced settings.
• More specific instructions. Clearly spell out the rules and output format in each agent’s instructions. This way the model has less room for interpretation. For example, here’s what I’d give the comparison agent:

Compare the generated test cases with the existing qTest test cases from 
the earlier steps. Score each match on keyword overlap, test functionality, 
and test step content. Exclude matches below 70%. Return JSON with one 
entry per test case: the action (create, update, or delete), the matching 
qTest case ID (if any), and a one-line reason.

Never propose a new test case without a requirement behind it. Fail if a test requirement is missing.

To add approval gates, open the relevant workflow step, select Enable approval gate, and turn on Require approval. 

When we run the new workflow, we enter a ticket ID, a qTest project, and a module. The workflow reads the ticket, generates the tests, compares them with what’s already in qTest, and waits for a person to approve the proposed changes.

This is where repeatability comes from. The workflow creates tests the same way each time, and once they’re saved in qTest, they follow the same exact steps on every run, whether a tester runs them or you automate them in Tosca.

A workflow can kick off manually, on a schedule, or with an event. For PROMO-214, we want the workflow to start when the story moves to Ready for QA. Jira Automation can watch for that status change and send a POST request to the workflow’s webhook URL. 

And what about audits for when things go wrong? Workflow runs are tracked, so each one appears in the run history, and you can export it.

Using Templates

AI Workspace comes with templates that can help you get started: 

• “Find and update relevant test cases”\ Starts by reading a Jira ticket. Then it generates tests, checks them against your existing qTest cases, and updates qTest. There are three approval gates along the way, including a final one before it changes your qTest project.
• “Test coverage analyzer”\ Compares requirements with qTest cases and reports an estimated coverage percentage.
• “Get your test requirements ready”\ Runs before testing starts. It flags vague or missing acceptance criteria in your requirements and drafts questions for whoever wrote them.

Here’s how “Find and update relevant test cases” looks. It’s very close to what we just asked the assistant to build.

If we started from this template for PROMO-214, I’d keep all three of its approval gates:

• On the generated tests. Does a 10% code really take $10 off a $100 cart?
• On the change proposal. A test lead checks proposed updates/deletes, watching for a test, for example, that’s marked for deletion but is still needed.
• Before the write to qTest. A final review before anything changes.

What Still Needs a Human in Agentic Quality Engineering

You could just save your workflow at this point and start testing. But I wouldn’t.

A bad decision saved into a workflow … is a bad decision that repeats on every single run. So I like to keep a few steps with the humans.

First, human approval gates are always a good idea just as a sanity check before an agent’s output reaches your test repository, or anywhere else that’s hard to clean up. 

Humans also should decide if a test is checking the right thing. Distinguishing correct behavior from incorrect behavior is what researchers call the oracle problem, and it’s been a bottleneck in test automation for a long time. 

One recent study of LLM-generated unit test assertions across 24 Java projects, with the code in the prompt, found that LLM-based test generation approaches “mainly capture the actual program behavior.” In other words, an LLM that reads your code tends to treat whatever the code actually does as correct.

Finally, someone should check the actual coverage. A number in a coverage template is just an estimate of how well your tests match requirements. I’d have a person spot-check the matches before anyone reports an official coverage number.

Conclusion

My readers may recall my personal mission statement, which I feel can apply to any IT professional:

“Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester

I don’t know anyone whose intellectual property became more valuable from retyping a test prompt and hoping for the same result. With an AI agent workspace, that job goes to a saved workflow, and your team gets to decide what to test and what the right result is.

Have a really great day!