Software Quality Evidence: The Gap That Costs

Software quality evidence is not the same thing as a testing programme, and the difference is what the first serious supervisory review tends to expose. Most regulated financial institutions have QA engineers, test suites, CI/CD pipelines and dashboards that show green. Far fewer have evidence.

A testing programme tells you whether software works. Evidence tells a supervisor, an auditor or a national competent authority that the right systems were tested, in the right way, at the right time, with the right oversight, and that the results were reviewed by people with the authority to confirm they were adequate. The cost of compliance in regulated industries sits mostly in that gap rather than in the testing itself.

Below we set out what the gap is, what it costs, why it persists, and what closing it actually requires.

Classification

Without undue delay, and within 24h of becoming aware

4 hours

Initial notification, counted from classification as major

72 hours

Intermediate report with scope, containment and root cause progress

1 month

Final report with full root cause and remediation

What supervisors ask for that a test suite cannot answer

DORA moved from preparation into active supervisory review during 2026, which made this distinction visible in a way it was not before. National competent authorities are no longer checking whether institutions have a testing programme. They are checking whether the programme produces records that hold up when examined.

One request exposes the gap most clearly, and supervisors now make it in various forms: produce the resilience testing evidence for your three most critical ICT systems. Not the test results. A complete, traceable record of what was tested, why, when, by whom, with what outcome, and who confirmed the outcome was adequate. Chapter IV of Regulation (EU) 2022/2554, Articles 24 to 27, is where the obligation sits.

For institutions with mature evidence infrastructure, that request takes minutes. Where testing is well run but the evidence layer was assembled from CI/CD logs and spreadsheets, the same request triggers days of manual work. Results arrive incomplete and inconsistently formatted, which creates a second problem: a supervisor reading a hastily built package is now assessing evidence management practices alongside testing practices.

The request is not "show me your tests". It is "show me that the right tests happened, and that someone accountable agreed they were enough".

Why the software quality evidence gap exists

The gap is not negligence. It is the predictable result of how testing infrastructure evolved before regulatory evidence requirements existed.

Built for this

The engineering question

Does the software work? Selenium, JUnit, Playwright and CI/CD pipelines answer it efficiently, producing execution logs, pass and fail counts, coverage percentages and stack traces. These are the right outputs, and most organisations built this layer well.

Not built for this

The regulatory question

Can you demonstrate, to a standard that survives examination, that critical ICT systems were tested in line with your obligations, with appropriate oversight, and with results reviewed by accountable individuals? A coverage percentage does not answer that.

The second question needs a layer that links tests to requirements, keeps the sign-off chain, stores records immutably and returns them on demand. In most organisations that layer was never built, was built differently by each team, or was built as a manual routine.

Why manual evidence layers fail specifically

Manual processes that depend on individual discipline do not survive staff turnover, team restructuring or the compressed timelines of a supervisory request. They also fail quietly. Nobody notices that the sign-off spreadsheet stopped being updated in March until someone asks for the March records in October. That delay between failure and discovery is what makes the manual approach riskier than it looks on an org chart.

What the software quality evidence gap costs

The cost materialises in three ways, and none of them appears in a budget line labelled testing.

Supervisory preparation. Every review or request means assembling evidence from wherever it currently lives, usually across QA, compliance and the CTO's own time. Days to weeks, consumed by expensive people, recurring with every regulatory interaction. It does not shrink over time unless the infrastructure changes.

Incident response. Classifying an incident as major requires knowing what the affected system is, what depends on it and what recent testing showed. An institution that cannot retrieve that quickly is making the classification decision, which starts the 4-hour clock, on incomplete information.

Accumulated supervisory risk. An institution that repeatedly produces incomplete or inconsistent records builds a relationship characterised by concern rather than confidence. That shapes the intensity of future supervision and the nature of follow-up. Hard to quantify, and it compounds.

There is also a predictable date worth planning around. The Register of Information is submitted annually by 30 April, so at least one evidence exercise per year is on the calendar whether or not a review is scheduled.

Is your institution's testing evidence ready for a supervisory review today, not after two weeks of assembly?

Talk to our team →

The four requirements of software quality evidence infrastructure

Closing the gap is mostly an architectural decision about where evidence lives, who owns it and how it gets produced. Four requirements do the work.

Requirement 1

Traceability from requirement to test

Every test case linked to the requirement or ICT function it verifies, as a structural relationship rather than a maintained document. Change a requirement and the affected tests are immediately identifiable. When a supervisor asks which tests cover a specific Article, the answer needs no cross-referencing by hand.

Requirement 2

A governed sign-off workflow

An execution record alone is not evidence. It needs approval by someone with the authority and accountability to confirm the result is adequate for its regulatory purpose. Role-based access applied to the QA function, with sign-offs timestamped, attributed and immutable.

Requirement 3

Immutable storage with retention

Records stored so they cannot be altered afterwards, with retention aligned to the institution's obligations. A CI/CD log rotated out after 90 days is not an evidence record, however well it documented the run at the time.

Requirement 4

Retrievability in a supervisory format

Exportable as a structured document that presents the requirement, the test cases, the execution records, the sign-off chain and the conclusion. Not a report written by hand when asked, but the natural output of a system that structured the information at capture.

Institutions already running DORA resilience testing usually have the first requirement partially in place. Most programmes break at the sign-off chain instead.

A 90-day path to software quality evidence

Nobody rebuilds an evidence layer in a quarter. What fits in a quarter is enough structure to survive the next request, so the sequence below is ordered by what reduces exposure fastest rather than by what is most complete.

Weeks 1 to 2: pick three systems. Take the systems most likely to appear in a supervisory request, which usually means those supporting critical or important functions. Scope the exercise there instead of attempting the whole estate, because a complete chain for three systems is worth more than a partial chain for thirty.

Weeks 3 to 5: run the retrieval test. Ask your own team for the evidence, without warning, and time it. Whatever the answer is, that number is what a supervisor would experience. Teams are routinely surprised, and the surprise itself is the argument for the budget.

Weeks 5 to 8: fix the sign-off chain first. Traceability is usually partially present, whereas approval records are usually the missing piece. Define who may approve, record it with a timestamp and an identity, and stop accepting approval by email thread.

Weeks 8 to 12: retention and a dry run

Settle retention. Check what your CI/CD platform actually keeps and for how long, then move anything that counts as evidence into storage that cannot be overwritten. This is often the cheapest item on the list and the one most frequently postponed.

Produce one package for real. Export the full chain for one system as though a request had arrived, then hand it to someone outside the QA team and ask whether it answers the question. If it does not, you have found the gap before a supervisor did.

Two habits make the difference afterwards. Rehearse the retrieval once a quarter, since an untested process is an assumption. And treat the annual Register of Information submission as the recurring deadline that keeps the chain current, because a layer maintained only when someone asks decays between requests.

Under pressure: what a major incident reveals

Nothing tests this infrastructure like a failure. A critical ICT system goes down, the response team activates, and the first decision is classification, because the 4-hour notification clock starts there.

Where testing evidence enters the incident timeline

Classify

Clients affected, duration, geographical spread, data losses, criticality, economic impact

›

Notify at 4 hours

Type, services affected, preliminary impact, contact point, actions taken

›

Report at 72 hours

Scope, containment, root cause progress, and what prior testing had shown

Classification criteria come from RTS 2024/1772. Testing history feeds all three stages, which is why retrieval time matters more than report-writing time.

Two institutions with the same test suite behave differently here. One has the record for the affected system, the most recent execution, the approving sign-off and the requirements covered in a single place within minutes, so the classification is better informed and the notification arrives complete. At the other, the same question sends people to CI/CD logs, email threads and spreadsheets while the clock runs. Their notification ends up reflecting the state of the evidence infrastructure rather than the state of the testing.

What software quality evidence does not solve

Worth being direct about the limits, because evidence infrastructure is sometimes sold as a compliance answer in itself.

A weak test suite stays weak. Perfectly traceable, immutably stored records of inadequate testing document the inadequacy with great precision, so the evidence layer improves the account of the work rather than the work. Deciding which systems support critical or important functions also remains an internal judgement, since that classification belongs to the institution's own risk assessment. Threat-led penetration testing keeps its own requirements and cadence for the entities in scope, and no evidence platform substitutes for it. Accountable humans stay in the loop too, because the sign-off is the part that cannot be automated by design. What does disappear is the assembly work, the inconsistency and the dependence on whoever happens to remember where things are.

How Qualigentic produces the software quality evidence chain

Qualigentic is an agentic QA platform built around the evidence layer rather than around test generation volume. Each test case carries the link to the requirement that motivated it. Execution records are timestamped and immutable. Sign-off runs through role-based workflows, so approval sits with people who hold the authority to give it. Export produces structured documents shaped for supervisory reading.

The distinction that matters: the evidence is produced continuously as testing runs, not assembled when someone asks. Deployment is on-premise, so records stay inside the institution's own perimeter, which also removes the third-party data transfer question. A time-boxed pilot covers one application and one framework and produces regulator-facing evidence in six to eight weeks. Our notes on AI QA in regulated industries and on the cost of software quality cover the surrounding arguments.

Frequently asked questions

Software quality evidence and DORA

What is the difference between a testing programme and an evidence chain under DORA?

A testing programme is the set of tests an institution runs to verify that its ICT systems work. An evidence chain is the structured, auditable record connecting each test to the requirement it addresses, through the execution record, to the sign-off by an accountable individual. Supervisors assess the evidence chain. An institution can run rigorous testing and still struggle in a review if the evidence layer does not produce records in the expected format.

Why do most institutions have a gap between testing and evidence?

Testing tools were designed to answer engineering questions, so they produce logs, pass and fail records and coverage metrics built for development operations. The regulatory evidence requirements arrived after those tools were adopted, and the evidence layer was usually added as a manual routine on top. Manual routines that depend on individual discipline do not scale and do not survive staff changes.

Supervisory requests and incidents

What does a supervisory request for resilience testing evidence look like?

Supervisors typically ask for a complete, traceable record of resilience testing for designated critical ICT systems, covering which systems were tested, against which requirements, when, with what result, and who approved that result. The record has to be structured, attributable and retrievable on demand. Assembling it from CI/CD logs and spreadsheets after the request arrives is substantial work that reflects on the institution's evidence management as much as its testing.

How does the DORA 4-hour notification clock work?

The initial notification is due within 4 hours of classifying an incident as major, and in any case no later than 24 hours after becoming aware of it. Classification itself must happen without undue delay. Because classification depends on knowing the affected system, its dependencies and its recent testing history, evidence retrieval sits upstream of the 4-hour clock rather than inside it.

Tooling and deployment

How does Qualigentic address the evidence infrastructure gap?

Qualigentic produces a structured evidence chain as part of normal platform operation, connecting each test case to the requirement behind it, keeping timestamped execution records and running sign-off through role-based workflows. Evidence is produced continuously rather than assembled on request, stored immutably with configurable retention, and exportable in formats suitable for supervisory review. On-premise deployment keeps records inside the institution's own infrastructure.

Can existing testing tools produce a compliant evidence chain?

Partially, with work. Requirement traceability can be maintained in a test management tool, and immutable storage can be built with a write-once archive. What tends to be missing is the governed sign-off chain and export in a supervisory format, which is why many institutions end up bridging the gap manually. Whether to build or buy that layer is a genuine decision, and the honest test is whether the result survives staff turnover.

Caixa Mágica Team

Caixa Mágica Software is a Portuguese software company with 20+ years of experience delivering custom software, AI solutions and nearshore development teams for European businesses.

Qualigentic · Caixa Mágica Software

Evidence produced as testing runs, not assembled under pressure

Signed chains from requirement to execution record. Immutable storage with configurable retention. On-premise, so nothing leaves your perimeter. A time-boxed pilot covers one application and one framework in six to eight weeks.

Learn about Qualigentic Start a conversation