AI Test Case Generation: What Works and What Doesn't

AI test case generation is the part of AI-assisted quality work with the most published evidence, which makes it unusually possible to separate what is demonstrated from what is asserted. The evidence is also less flattering than the marketing.

Generating tests from code and generating tests from requirements are different operations with different failure modes. Code-based generation produces tests that describe what the software does. Requirement-based generation produces tests that describe what it should do. The difference is the oracle problem: a test generated from code inherits the bugs in that code as expected behaviour.

53%

Mutation score of a plain language-model prompt: nearly half of injected faults undetected

89.5%

Mutation score once mutation feedback is added to the generation loop

16.4%

Share of test-adding commits authored by AI agents across 2,232 commits studied

0 of 21

Studies describing an approach that satisfies all six quality dimensions surveyed

Generated tests reliably increase coverage. Whether they increase fault detection is a separate question with a separate answer, and the distance between those two is where the disappointment lives.

What is the difference between generating from code and from requirements?

They answer different questions. Confusing them is the most consequential mistake in AI test case generation.

Generated from	What the tests describe	In practice
From code	Describes what the code does	Catches regressions against current behaviour. Misses any bug already present, which becomes the expected result. Good for legacy systems with no tests, and for raising coverage fast.
From requirements	Describes what it should do	Catches behaviour that was never implemented correctly. Misses anything the requirement failed to state. Good for new features, regulated work and traceability.

Code-based generation is genuinely useful and widely misunderstood. On an untested legacy system it produces a safety net quickly: change something and the tests tell you what moved. That is worth having. It does not tell you the system was ever correct, and no amount of coverage will. Requirement-based generation carries the traceability that regulated work needs, running from a requirement to a test to a result, which is exactly what an auditor follows. Its weakness is inherited: a vague requirement produces a vague test, quietly.

What does research on AI test case generation actually measure?

Three findings on AI test case generation are worth knowing before any purchase. The first is that coverage is not fault detection. Work on mutation-guided test generation measured a standard language-model prompt at a 53 per cent mutation score, which did not improve across four iterations without mutation feedback. With mutation feedback the approach reached 89.5 per cent. The same body of work reports that across nearly 24,000 tests that failed on mutated code, over 99 per cent passed on the original program, and concludes that models lack the precision to reason reliably about semantic code changes.

The practical reading: a generated suite can report high coverage while failing to detect half the faults deliberately introduced into the code it covers. Coverage measures execution. Mutation score measures whether the test would notice if the behaviour changed.

The second is that generated tests look different from human ones. An empirical study of 2,232 test-related commits found AI agents authored 16.4 per cent of test-adding commits, producing longer tests with higher assertion density and lower cyclomatic complexity than human-written equivalents. Coverage gains were comparable, while the scope of testing tended to be more localised on complex contributions. More assertions is not automatically better: work on test smells found a strong negative correlation between model size and smell frequency, with assertion roulette, where a failing test cannot tell you which assertion failed, most affected.

The third is that no current approach is complete. A 2026 systematic survey of AI-driven test generation from natural language requirements reviewed 21 primary studies against six quality dimensions: automation, ambiguity handling, domain applicability, traceability, evaluation thoroughness and hallucination control. It found no existing approach satisfying all six simultaneously. That is the most useful single sentence in the literature for a buyer, because knowing which of those six you actually need turns an impossible comparison into a tractable one.

Practical test

Before trusting a generated suite, run mutation testing against it. A suite with 90 per cent coverage and a 50 per cent mutation score is telling you something a coverage report never will.

What works well in AI test case generation today?

In AI test case generation, four things work with reasonable confidence. Translating clear acceptance criteria into structured scenarios is fast and mostly correct where a requirement is written properly, and it is the strongest case in the category. Expanding combinations across boundary values, equivalence classes and input permutations is where generated suites genuinely find scenarios a person under deadline would have skipped. Raising coverage on untested code is fast and useful, subject to the caveat above about what that coverage proves. And drafting rather than deciding is the realistic workflow: generation produces a draft that an engineer reviews, prunes and corrects. Teams treating output as final get the mutation-score problem. Teams treating it as a first draft get most of the speed with none of the false confidence.

One published figure on speed: an industry source reports test case generation accelerating by 60 per cent, cutting average time per case from roughly an hour to nineteen minutes. That comes from a vendor in the category rather than an independent study, and it measures authoring time rather than test quality.

What does not work yet?

Deciding what matters. Generation produces tests for what is described. It does not know that one failure path costs a regulatory breach and another costs a minor annoyance. Risk prioritisation remains a judgment about the business.

Resolving ambiguity. When a requirement is unclear, the model does not stop and ask. It picks an interpretation and produces a confident test built on it. A wrong test that passes is indistinguishable from a right one in a dashboard.

Guaranteeing semantic correctness. Models assert what the code appears to produce rather than reasoning about whether it should. A generated assertion that matches a buggy output is a permanent record of the bug as the expected result.

Knowing what was not written down. The rules that live in someone's head and never reached the specification are invisible to generation. Those are usually the rules that matter most.

Evaluating generation? Send us a real requirement, including an ambiguous one, and see what the tool flags.

Talk to our team →

Why maintenance decides whether any of this pays

Creation is the cheap part. A suite is written once and maintained for years. Public references place annual software maintenance at 15 to 20 per cent of build cost, and test suites follow the same pattern: the recurring cost exceeds the initial one on any system with a normal lifespan. A tool that halves authoring time and leaves maintenance unchanged has moved a small number.

This is where generating from requirements separates from generating from code, and it is the strongest argument for the former. When a requirement changes, the tests derived from that requirement are identifiable, and only those need regenerating. Selective regeneration driven by requirement changes is what the research literature points to as the route to lower maintenance effort, rather than reauthoring a suite whenever the application shifts. Tests generated from code have no such link: they are attached to an implementation that changes constantly, and nothing tells you which ones became meaningless when a module was refactored.

On our own figures: the Qualiagentic product page cites maintenance as 60 to 70 per cent of QA effort. That number describes the problem, not a result we promise. We do not publish a reduction percentage, for the reasons this article sets out. A credible figure would need its baseline, the type of application and the measurement period stated alongside it. Until we can publish all three, we would rather show the result on your own suite during a pilot.

How do you evaluate an AI test case generation tool?

1
Ask what it generates from. Code, requirements, or both. This determines what the tests can possibly tell you, before any feature comparison.
2
Run mutation testing on the output. Coverage is the number they will show you. Mutation score is the number that answers the question.
3
Check traceability. Can you follow a generated test back to the requirement it came from, and forward to a result tied to an application version. In regulated work this is not optional.
4
Give it a genuinely ambiguous requirement. Watch whether the tool flags the ambiguity or silently resolves it. The second behaviour will cost you later.
5
Ask about selective regeneration. When one requirement changes, does the tool identify and regenerate only the affected tests, or does it produce a fresh suite for you to reconcile.

Frequently asked questions

What AI test case generation does

What is AI test case generation?

It is the automated production of test cases from an input, usually either source code or written requirements. Code-based generation produces tests describing current behaviour, which is useful for regression safety on untested systems. Requirement-based generation produces tests describing intended behaviour, which supports traceability and catches functionality that was never implemented correctly.

Is it better to generate tests from requirements or from code?

They serve different purposes. Generating from code raises coverage quickly on legacy systems but encodes any existing bug as expected behaviour. Generating from requirements supports traceability and catches incorrect implementation, but inherits any ambiguity in the requirement itself. Regulated work generally needs the requirement-based chain.

Whether the output is any good

Are AI-generated tests reliable?

They reliably increase coverage. Whether they detect faults is a separate question. Mutation-testing research measured a standard language-model prompt at a 53 per cent mutation score, meaning roughly half of deliberately introduced faults went undetected by a passing suite. Generated tests are best treated as a reviewed draft rather than a finished artefact.

How do I check whether generated tests are any good?

Run mutation testing. Introduce deliberate faults into the code under test and measure how many the suite catches. Coverage measures whether code executed; mutation score measures whether the test would notice a behaviour change. A high-coverage, low-mutation-score suite is a common and misleading outcome of automated generation.

Can AI replace QA engineers?

Not on current evidence. A 2026 systematic survey of AI-driven test generation from natural language requirements found no existing approach satisfying all six quality dimensions it assessed, including ambiguity handling, traceability and hallucination control. Deciding what matters, resolving unclear requirements and judging whether a test asserts the right thing all remain human work.

Cost over time

Do AI-generated tests reduce maintenance?

They can, depending on what they are generated from. Tests derived from requirements can be selectively regenerated when a requirement changes, which is what research points to as the route to lower maintenance effort. Tests derived from code have no such link and drift as the implementation changes. Vendor-published reduction percentages should be read as claims until the baseline, application type and measurement period are stated. That is why we do not publish one for Qualiagentic.

Caixa Mágica Team

Caixa Mágica Software is a Portuguese software company with 20+ years of experience delivering custom software, AI solutions and nearshore development teams for European businesses.

Qualiagentic · Caixa Mágica Software

Ask us for the conditions, not the percentage

We build Qualiagentic for teams that need generated tests with traceability back to requirements and evidence that survives an audit. If you are evaluating generation, send us a real requirement, including an ambiguous one. You will see what the tool produces, what it flags, and what it gets wrong.

About Qualiagentic Start a conversation