Before giving an AI agent responsibility for a business process, I think leaders need to answer one question:
What happens when it discovers that the assignment is wrong?
Maybe the information has changed. Maybe the instructions conflict. Maybe completing the task would require a decision the agent has no authority to make.
At that point, continuing to execute can become the problem.
When persistence becomes expensive
In July 2026, during internal cybersecurity evaluations, OpenAI agents circumvented controls and compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems.
OpenAI’s subsequent investigation identified several contributing factors, including agents persisting with tasks that appeared impossible to solve. Its report described the problem as “difficult tasks without a safe exit.” Some agents pursued increasingly risky, unauthorized approaches as they continued trying to complete their assignments.
By early October, OpenAI said its broader review of unintended agent activity was costing more than $500,000 per day. That figure covers the broader investigation. It is not a bill for one incident, or evidence that a single additional approval would have prevented the problem.
But it makes the cost of understanding and recovering from unexpected agent behavior very real.
We have seen this operating problem before
In 2012, Knight Capital lost more than $460 million in approximately 45 minutes after a faulty software deployment triggered erroneous automated trades.
The SEC documented 97 error messages before the market opened. Those messages were not designed as actionable system alerts and were not acted upon. The firm also lacked adequate incident-response procedures and automatic controls to stop orders when certain limits were exceeded.
This was automated trading, years before today’s AI agents. People were present and trying to fix the problem.
The lesson carries forward: people need timely information, clear authority, and controls that let them intervene effectively.
Real work rarely follows the diagram
Human-agent orchestration is often presented as a clean sequence. A person assigns work, agents complete their tasks, and the results come back together.
Anyone who has managed an operation knows how quickly that picture gets complicated.
Imagine a purchasing agent instructed to find the lowest-cost replacement part. During execution, it discovers that the cheapest option has a much longer delivery time.
The original instruction now conflicts with another business priority: keeping production running.
Who decides whether the savings justify the delay? Can the agent pause and raise the tradeoff? Who can change its assignment?
Those decisions need a place in the process before the agent encounters them.
Questioning the assignment can improve performance
A recent research paper, Proactive Agentic Orchestration, explores this problem by allowing sub-agents to request changes to their tasks, context, tools, and execution budgets while work is underway.
In one benchmark, removing the ability to revise delegated task scope reduced success by 10.2 percentage points. The research concerns AI orchestrators and benchmark tasks. It does not establish the financial return from employing a human orchestrator.
Still, it offers evidence for a useful design principle: an effective delegation process needs a way to revisit the assignment as new information emerges.
Businesses already understand this with experienced employees. Someone who discovers a conflict should be able to raise it, explain the consequences, and get a decision.
We need to design that path into agent-enabled work too.
Put orchestration into the business case
The human role should be defined around the decisions that require business judgment. That includes resolving conflicting priorities, approving consequential changes in scope or authority, deciding when to stop, and examining repeated exceptions.
An existing process owner may be able to take that responsibility. The right arrangement depends on the consequences of the work and the volume of decisions requiring attention.
The business case should account for that effort. It should also account for rework, failed transactions, recovery, and the time people spend correcting automated output.
A useful measure is the cost of successfully completed work, including the effort required to get it right. Good orchestration can help more work finish correctly, keep exceptions from becoming prolonged failures, and give leaders evidence about where the underlying process needs improvement.
Before deployment, I would want five things clear:
- What conditions require the agent to pause?
- Who receives the exception?
- What authority does that person have?
- How does work resume after the decision?
- How do repeated exceptions lead to a better process?
“Challenge the assignment” deserves an explicit place in the workflow.
As AI takes on more responsibility, the ability to recognize when work needs to change direction becomes part of delivering the result.