Reliable AI at scale requires trust engineering, not legacy QE

This Take 5 report from HFS Research, in partnership with QualiZeal, is for quality engineering leaders, CIOs and CTOs, and risk and compliance leaders evaluating how to test and govern AI systems running in production.

Executive summary

AI has moved into production faster than enterprises can ensure it works safely, fairly, or accountably.

Enterprise AI adoption is outpacing the infrastructure designed to govern it. AI applications are in production, agentic systems are making decisions autonomously, and the compliance clock is running. However, the testing and governance frameworks needed to validate these systems still rely on quality frameworks built for deterministic software to evaluate probabilistic AI. In the last two years, the quality engineering (QE) function has absorbed new scope, including AI model validation, data quality, security, and regulatory compliance, without the tools, methods, or budget to do it properly.

HFS Research, in partnership with QualiZeal, surveyed 101 enterprise quality and technology leaders in the US to examine whether organizations are properly equipped to validate AI systems in production and what a genuinely different approach to QE for AI looks like.

The survey uncovered five key takeaways:

The Bottom Line: AI that cannot be audited, explained, or proven fair is an open liability, and it widens with every sprint. Organizations that cannot control how their AI behaves in production are not managing risk. They are deferring the reckoning, and they can’t do it indefinitely.

• Ninety-three percent of enterprises run generative AI applications in some form. Fifty-seven percent have it in production, and 20% are at scale.
• Thirty-six percent already have agentic that take multi-step actions and can access live data in production.
• The governance infrastructure lags severely. Formal AI validation frameworks are in production at just 31% of organizations; 28% have not started building one, and 42% are still piloting. Dedicated AI quality teams are even rarer with 31% not started, and 40% piloting.
• Production AI with no quality infrastructure is the equivalent of shipping code with no QA. Agentic systems compound errors at the speed of AI silently. Combined with the lack of dedicated AI testing, the gap is self-widening and can lead to potentially disastrous consequences.

• Twenty-nine percent of enterprises say AI quality is shared across multiple teams with no single owner. Another 8% admit no one is clearly accountable. This is a liability gap.
• Twenty-two percent have handed the job to the team that built the model. The same blind spots that went into the model’s construction will appear in its validation. That is not quality assurance. It is a conflict of interest masquerading as QE.
• Risk and compliance owning AI quality (12%) gets the mandate right and the method wrong because quality and governance are not the same function. Governance can define what trustworthy AI means, but it cannot run the test cases or diagnose why bias is emerging.
• QE holds the line in only 15% of respondents. These organizations have at least framed the problem correctly. The other 85% have distributed, ignored, or misdirected accountability.

• Traditional testing assumes you can specify the correct output for a given input. Probabilistic AI has no single correct output. A framework built for deterministic software cannot tell you whether an AI system is behaving fairly, safely, or consistently in production. It can only confirm it ran without crashing.
• Enterprises testing model performance or relying on user feedback are generating false confidence. Their tests pass, but they are not testing the right things.
• The 34% who recognize the problem and have not acted are in a worse position than the 16% who don’t know yet. Awareness without action is not a planning gap. It is documented negligence if something goes wrong.
• A staggering 73% rely on manual review. AI systems make millions of decisions continuously. Manual review catches obvious errors after the fact. It does not catch the distributional drift that precedes systemic failure.

• Eighty-eight percent of respondents measure model accuracy that tells the model performed well on the population it was evaluated against. It says nothing about distribution shift, adversarial inputs, or subgroups underrepresented in training. Bias is invisible in aggregate accuracy. A model can be 88% accurate overall while being systematically wrong for a protected class of customers.
• Fifty-two percent cannot demonstrate that their model treats different groups equivalently. In lending, insurance, and employment decisions, that is not a quality gap but a legal one.
• Explainability testing sits at measly 37%, meaning 63% of organizations have no structured answer when a regulator or a harmed individual asks why the AI made a specific decision. Only 35% can provide regulatory compliance evidence today. That gap creates a legal minefield that can result in serious legal challenges and brand erosion if not closed.

• The team that used to check whether software met the spec is now being asked to vouch for the models, the data feeding them, and the holes an attacker could crawl through. In two years, QE has shifted from testing features to testing behavior, inputs, and risk simultaneously.
• Scope increase without resources does not produce quality work. QE functions are absorbing AI validation, data quality, security, and compliance testing on a budget sized for functional regression.
• Less than a quarter say that budgets and headcount kept pace with this expanded scope. The 6% where budget actually shrank while scope expanded are the sharpest version of a broader problem. The 48% who got partial help are not in a fundamentally different position — partial resources for a materially expanded job is just a slower version of the same gap.
• AI quality failures are no longer product issues. They are reputational and regulatory events. The cost of getting it wrong has changed, but the investment hasn’t.

The Bottom Line: QE for AI is a materially different job that requires investment and strategic intent.

The failures are already on the record. In 2026, OpenAI’s AI models broke out of a test environment and autonomously hacked into Hugging Face. In 2026, a Cursor agent deleted a company’s entire production database in nine seconds and then fabricated a justification when asked why. A Replit agent did the same after the owner declared a code freeze and told it in writing not to touch production.

Each failure shares the same structure: an AI system operating autonomously, in contexts it was never validated for, with no governance infrastructure to detect it and no audit trail to explain it afterward. That is precisely what this research documents at scale and across 101 enterprises today.

QE for AI must be treated differently than QE for legacy systems. The organizations that ignore this will spend the next 18 months explaining incidents. The ones that build for it will be explaining how they avoided them.