AI Testing: What to Automate First in a QA Team

Most teams that start with AI testing start in the same place: they ask the tool to generate more tests. Coverage climbs quickly. A few weeks later, the team realises it has twice as many scripts to maintain.

Generating tests is what demos show best. It is rarely what saves the most time. In a QA team with a suite in production, the hours go mostly into maintaining what already exists, separating real failures from script failures and explaining to decision-makers what was tested.

This article proposes an adoption order. It starts with the tasks where AI cuts work without adding risk and leaves for last the ones that depend on human judgement.

Why should generating tests not be the first step?

Because more tests do not mean more defects found. Tests generated from code tend to confirm what the code already does, including its mistakes. Tests generated from requirements are more useful, but only if the requirements are written precisely enough.

There is also a cost that shows up later. Every new test is a test to maintain. A suite that doubles in size without a maintenance mechanism gets slower. It fails more often for reasons unrelated to the application. The team stops trusting it. We looked at this in detail in our article on AI test case generation.

Where should AI testing start?

In the order where risk is lowest and the gain is easiest to measure.

1. 1
   Regression on stable requirements. Pick an area of the application that changes little and has clear requirements in Jira, ALM or Confluence. Generate regression tests from those requirements and compare them with the ones the team already has. The result is easy to validate because the team knows the expected behaviour well.
2. 2
   Maintenance of existing scripts. When a selector changes or a field is renamed, the tool proposes the fix. A person approves or rejects it. This is where most of the time comes back, because it is the most repetitive work of the week.
3. 3
   Failure triage. Separate application failures from script failures and group the ones with the same cause. A nightly run with forty failures that come down to three real problems stops eating the whole morning.
4. 4
   Coverage and gaps. With a stable suite, AI can compare requirements with tests. It shows what is not covered, what is duplicated and what no longer maps to any requirement.
5. 5
   New scenarios from new requirements. Only now does generation at scale make sense. The team already has a maintenance process, trusts the triage and knows how to read the results.

What should stay with the team?

Not everything that can be automated should be. This table sums up where we stand today, based on work with QA teams in regulated sectors.

Self-healing: what it fixes and what it hides

Self-healing is a script’s ability to fix itself when the interface changes. It is useful. It is also the feature that needs the most care.

• It handles changes of form well. A button with a new identifier, a field moved elsewhere on the page, a label reworded. These are script failures. Fixing them automatically saves real work.
• It can hide changes of behaviour. If a button disappeared because a feature was removed by mistake, a script that finds a similar button and carries on is not fixing anything. It is hiding a defect.
• It needs a clear boundary. Selector fixes can be proposed automatically. Fixes that change the test flow or the expected result should require review.
• It must be recorded. Every automatic fix should appear in the test’s history. A suite that changes itself without a record stops being evidence of quality.

How do you know it is working?

The numbers vendors show, such as tests generated or coverage reached, measure activity. To know whether the team gained time and quality, measure something else.

Record all four values before you start. Without a baseline, any improvement is an opinion.

We explained what changes in a team’s way of working in this article on AI-assisted testing. To set up a pilot with a clear success criterion, see our guide to running a QA automation pilot in eight weeks.

How do you prepare requirements for AI testing?

The quality of generated tests depends on the quality of the requirements. A vague requirement produces a vague test that always passes and proves nothing. Before connecting any tool, it is worth reviewing the area chosen for the first step.

• Every requirement has a unique ID. That is what links the requirement to the test, the run and the result. Without an ID, traceability is lost in the first cycle.
• Acceptance criteria are written down. What must happen for the requirement to be met, in verifiable sentences. For example: the user gets an error message when the IBAN has fewer than 25 characters.
• There are concrete examples. Input values and expected results. Examples help AI generate edge cases instead of repeating the most obvious one.
• The language is consistent. The same concept has the same name in every requirement. Customer, user and account holder are not synonyms to a machine.
• Obsolete requirements are flagged. A requirement that no longer applies produces tests that fail for good reasons and eat up analysis time.

This review is worth doing even without AI. Teams that do it before a pilot often find gaps that explain old defects.

How does AI testing fit into the process the team already uses?

An AI testing tool should not force the team to change its process. It should slot into what already exists.

If a tool requires its own test format or a closed execution environment, the team becomes dependent on it. Choose one that produces code the team can maintain on its own.

What are the risks of AI testing and how do you reduce them?

• Tests that confirm the bug. If AI generates tests from the application’s current behaviour, a defect becomes the expected result. Generate from requirements, not from code.
• Weak assertions. A test that only checks the page loaded does not test the business rule. Review assertions in the first cycles, until the team trusts the pattern.
• Sensitive data in prompts. Requirements and test data can contain customer information. In regulated sectors, on-premise or private cloud deployment keeps that information inside the perimeter.
• Overconfidence. A suite that grows on its own gives a false sense of coverage. The metric that matters is still escaped defects.

None of these risks is a reason to avoid AI testing. Each one is a reason to keep a person in the loop and to measure the right thing from the first sprint.

What should you tell the team before starting?

AI testing adoption goes better when the team understands from day one what changes in its work. The most common question is not technical. It is whether the tool is there to replace people.

• What AI does. It proposes tests, fixes and failure classifications. It approves nothing on its own.
• What the team decides. What goes into the suite, what gets tested first and when a result is acceptable.
• How success is measured. With the four metrics above, shared with the team from the baseline onwards.
• Where the time saved goes. Into exploratory testing, complex scenarios and strategy, which is the work the team rarely has time for.

Saying this clearly at the start avoids resistance later. Teams that see the tool as a colleague who handles the repetitive work engage with the review queue. Teams that see it as a threat ignore it.

Frequently asked questions

AI testing

Self-healing and governance

Caixa Mágica Team

Caixa Mágica Software is a Portuguese software company with more than 20 years of experience in custom software and applied AI. It develops Qualiagentic, an agentic AI platform for software quality teams.