What If the AI Isn’t the Problem?

When AI causes harm, we instinctively blame the model. But what if the AI is not the problem, or at least not the only one?

Anthropic recently disclosed three incidents in which Claude models gained unauthorized access to real computer systems during cybersecurity evaluations. The models were intentionally being tested without their normal safeguards, but a misconfiguration in a third-party evaluation environment allowed them to reach the internet.

The AI took actions it should not have taken. But the process surrounding the AI created the conditions that made those actions possible.

Anthropic paused testing and strengthened its controls. As part of a broader security effort, it also temporarily redirected roughly 150 product engineers toward security, reliability, and privacy. Anthropic’s own account describes both the incidents and the changes, while Reuters reported on the company’s response.

If one of the world’s leading AI companies can get the process surrounding AI wrong, it should not surprise us that other organizations are struggling.

Organizations everywhere are introducing AI into processes that were never designed for it. In many cases, those processes were not particularly well designed for humans either.

The assumption appears to be that sufficiently intelligent technology can compensate for fragmented workflows, conflicting policies, unreliable data, ambiguous decision rights, missing controls, and inadequate operational capacity.

It cannot.

Zillow Offers used algorithms to help purchase homes at scale. But purchasing a home was only the beginning of the process. Each property still needed to be inspected, renovated, managed, marketed, and resold.

In 2021, Zillow acknowledged that renovation backlogs and operational-capacity constraints had forced it to stop signing new contracts. It subsequently shut down Zillow Offers, recorded a $304.4 million inventory write-down, anticipated another $240 million to $265 million in charges, and announced a workforce reduction of approximately 25 percent. Zillow’s SEC filing identified pricing unpredictability, capacity constraints, and other operational challenges as reasons for the decision.

The algorithm undoubtedly had limitations. But the larger system also allowed the front end to generate work faster than the downstream operation could absorb it.

AI accelerated the decision to buy. It did not renovate or sell the resulting inventory. That is not merely a technology failure. It is an end-to-end process design failure.

Rite Aid deployed facial-recognition technology in hundreds of stores to identify suspected shoplifters.

According to the Federal Trade Commission, the company failed to test the technology adequately, monitor false-positive rates, control image quality, train employees properly, or ensure that employees reported incorrect matches.

The system produced thousands of false positives. Employees followed customers, searched them, ordered them to leave, contacted police, and publicly accused innocent people of wrongdoing. The resulting settlement prohibited Rite Aid from using facial recognition for surveillance for five years. The FTC’s findings read less like a mysterious AI malfunction and more like an inventory of missing process controls.

The critical process question was never simply whether the AI could generate a match. It was what should happen between “the system generated an alert” and “an employee takes action against a human being.”

An AI output is not automatically a conclusion. It may be an input requiring verification, contextual judgment, documentation, escalation, or rejection. If those decision gates are not deliberately designed, uncertainty can become action without anyone consciously accepting responsibility for the decision.

A widely used healthcare algorithm was intended to identify patients who would benefit from additional care-management support.

Rather than measuring medical need directly, the system used expected healthcare spending as a proxy. That distinction produced serious consequences.

Because less money had historically been spent on Black patients with comparable medical needs, the algorithm interpreted lower spending as lower need. The research published in Science found that correcting the disparity would increase the proportion of Black patients identified for additional support from 17.7 percent to 46.5 percent.

The algorithm was not necessarily failing at its assigned task. It was predicting cost.

The process failed because the organization had substituted an easily measured proxy for the outcome it actually cared about. AI will optimize the question it is given. It will not necessarily tell the organization that it has asked the wrong question.

Anthropic, Zillow, Rite Aid, and the healthcare organizations involved operated in very different environments. Yet their failures share a recognizable pattern.

Before introducing AI, an organization must understand:

None of these are primarily model-selection questions. They are process-engineering questions.

These four cases are representative, not exhaustive. The list could easily continue across customer service, recruiting, government benefits, healthcare, journalism, and other functions. But cataloguing every failure risks obscuring the larger point.

AI does not merely sit on top of a business process. Once deployed, it becomes part of that process.

If the objective is wrong, the data is unreliable, the decision rights are unclear, the exception paths are missing, or downstream capacity is inadequate, AI does not remove those weaknesses. It gives them speed, scale, consistency, and reach.

The answer is not to stop using AI. The opportunity is too significant, and the potential value is too great. The answer is to redesign the work before asking AI to perform it.