Liability Laundering.

The agent emails the wrong customer.

The agent approves the refund twice. The agent deletes the wrong table. The agent sends confidential data to the wrong vendor. The agent books the wrong flight, updates the wrong record, changes the wrong permission, files the wrong report, deploys the wrong code.

This is what the agent era will actually look like at first. Not rogue superintelligence, not cinematic machine rebellion, not an artificial mind waking up and deciding to overthrow its owner. The real danger is more boring and more believable: ordinary software mistakes, carried out with delegated authority, moving faster than the organisation can understand them.

A chatbot can be wrong. An agent can make the wrong thing happen.

For most of computing history, software waited for a human instruction. The human clicked, approved, sent, bought, deleted, deployed, signed, refunded, escalated. Software extended action, but humans usually supplied the decisive step. Agents disturb that contract. An agent can receive a goal, interpret it, plan steps, call tools, retrieve data, write code, send messages, update systems, trigger workflows, and report back with the confidence of a finished task. The user gives intention. The system fills in the path.

That is not just productivity. That is delegated action. And delegated action creates a problem the AI industry is still underpricing: when software acts, responsibility becomes harder to locate.

---

I. The teammate metaphor is wrong

Microsoft’s Work Trend Index frames the future around “agent bosses”: workers who build, delegate to, and manage AI agents. Microsoft describes a coming frontier firm where autonomous agents join teams as digital colleagues and eventually take on business processes and workflows while humans check in as needed.¹ Salesforce uses even more direct language. Marc Benioff has positioned Agentforce through the frame of digital labor, with autonomous agents and virtual representatives operating across customer service workflows, Slack workflows, and enterprise data.³⁴

Microsoft and Salesforce are not wrong that agents will be useful. Agents will automate workflows, coordinate systems, reduce admin load, and let smaller teams operate with more leverage. The problem is the language. “Agent bosses,” “digital colleagues,” “digital labor,” “limitless workforce” all make agents sound like workers before they can carry responsibility like workers.

That is the failure in the metaphor. A human worker can be trained, disciplined, licensed, insured, promoted, fired, sued, or held professionally accountable. An AI agent cannot. An agent can act inside the organisation, but it cannot carry responsibility for the organisation. That responsibility flows back to the humans and institutions that deployed it.

Benioff is right that agents can expand labour capacity. Microsoft is right that companies will need to learn how to manage agents. But both framings understate the central asymmetry.

A human worker brings responsibility with their work. An AI agent brings action without responsibility. That is why the agent era needs more than management. It needs boundaries, evidence, permissions, escalation, and liability design.

---

II. Air Canada was the warning shot

The Air Canada chatbot case matters because it was small enough to understand. A passenger relied on the airline’s chatbot for bereavement fare information. The chatbot gave inaccurate guidance. Air Canada argued that the chatbot should be treated as separate from the airline’s own responsibility. The British Columbia Civil Resolution Tribunal rejected that framing and found the airline responsible for information presented through its own website.⁶

That was not even a fully agentic system. It did not move money, modify a booking, or execute a workflow on the customer’s behalf. It mostly answered. But the answer still created liability because a real person relied on it.

Now extend the same logic one step further. The system no longer only explains the refund policy, it applies the refund. It does not only tell the customer how to change a booking, it changes it. It does not only summarise the contract, it sends the redline. It does not only recommend a response, it emails the client. It does not only suggest a code fix, it opens the pull request, runs the tests, and asks to merge.

Once software moves from answering to acting, the liability problem does not disappear. It compounds.

That difference also changes the governance problem. The relevant question is no longer only “is the model smart?” It becomes “what was it allowed to touch?” A bad autocomplete suggestion dies in the editor. A bad agent action can reach production. Most organisations are not ready for that distinction. Their permissions are too broad. Their environments are poorly separated. Their logs are incomplete. Their rollback procedures are weak. Their humans approve summaries rather than inspecting actions. Their AI systems inherit messy organisational habits and execute them faster.

Air Canada teaches the simplest version of the rule.

You do not escape responsibility by inserting software between yourself and the customer.

The agent era will test that rule everywhere.

---

III. Knight Capital was the pre-AI lesson

The best warning about AI agents may not come from AI at all. In 2012, Knight Capital suffered one of the most infamous software failures in financial history. A deployment problem caused the firm to send millions of unintended orders into the market in about 45 minutes, producing a pre-tax loss of roughly $440 million; the SEC’s settled order details how a software change combined with a residual function and incomplete pre-deployment review produced the cascade.⁷

That case matters because it shows the anatomy of automation failure. The disaster was not just one bad technical mistake. It was weak deployment discipline, weak controls, weak system visibility, weak rollback, weak escalation, and weak organisational response.

That is the real lesson for agents. When autonomous systems fail, the failure is rarely only inside the model. It is inside the operating environment around the model. Did the system have permission to act? Were the right limits in place? Were logs meaningful? Could humans see what was happening, and stop it in time? Did anyone understand the failure mode before it became expensive?

Knight Capital happened before modern AI agents. That is why it is useful. It strips away the hype. AI agents add a new layer to an old problem. They make the interface natural language. They make the action path adaptive. They make the intermediate reasoning harder to inspect. They make the system feel more like a colleague than a machine. But the operational truth remains the same.

Automation does not remove the need for controls. It raises the cost of weak ones.

---

IV. The strongest case for agents

The serious argument cannot pretend agents are only dangerous. They are useful because most organisational work is full of small, repetitive, cross-system actions that humans perform badly or resent performing at all. Agents can update records, resolve routine tickets, prepare customer replies, chase missing information, draft reports, reconcile systems, monitor alerts, route requests, generate code, process forms, and coordinate workflows across fragmented software. Used well, they could make small teams far more capable and large organisations less bureaucratic.

This is why the enterprise AI market is moving quickly. Salesforce has positioned Agentforce as a platform for creating virtual representatives that act across enterprise data, and Reuters reported that Salesforce had closed more than 1,000 paid Agentforce deals by mid-December 2024.⁵ Microsoft is moving in the same direction with products for registering, monitoring and controlling agents, including dashboards, telemetry, alerts and permission controls.²

Even the sceptical forecasts do not say agents disappear. Gartner predicts that more than 40% of agentic AI projects will be cancelled by the end of 2027 because of escalating costs, unclear business value, or inadequate risk controls, while still expecting agents to become embedded in enterprise software over time.¹¹

That combination is the right reading. Agents are overhyped. Agents are also real. A lot of projects will fail because the demos are ahead of the workflows. Some products will be agent washing: old automation with a new label. Some teams will deploy agents before they understand their own processes. Some executives will buy “digital labour” because it sounds cheaper than organisational reform. But the underlying direction survives. Software that can act across systems is more valuable than software that waits.

The critique is not that agents will fail. The critique is that the industry is selling autonomy faster than it is building accountability.

---

V. Human in the loop can become liability laundering

“Human in the loop” is the standard comfort phrase. It sounds responsible. The AI acts, but a human remains involved. The organisation gets speed without surrendering control. The compliance slide looks clean. The problem is that human oversight can be real or fake.

A human in the loop is meaningful only if the human can understand, challenge, and stop the loop. If the human approves 200 agent actions a day, they are not exercising judgment, they are processing pressure. If the human reviews a summary written by the agent, they are not reviewing the action, they are reviewing the agent’s own explanation of the action. If the human does not understand the underlying system, their approval is not oversight. It is ritual. If the interface makes approval easy and investigation hard, the product is not designed for accountability. It is designed for throughput. If the organisation punishes delay but rewards speed, the human becomes a rubber stamp.

That is not accountability. That is liability laundering.

The EU AI Act’s treatment of high-risk AI systems shows why the distinction matters. Article 14 requires high-risk AI systems to be designed so they can be effectively overseen by natural persons, including being able to understand the system’s capacities and limitations, monitor its operation, remain aware of automation bias, interpret its outputs, and intervene where appropriate. Article 12 separately requires logging capabilities sufficient to enable traceability of the system’s functioning.⁸ That is much stronger than “there is a human somewhere.”

Effective oversight requires design. It requires context, authority, competence, time, interface quality, logs, escalation paths, and the power to stop the system. Without those, “human in the loop” is not a control. It is a liability shield.

---

VI. The unit of trust changes

With a chatbot, the object of review is usually an output. What did it say? Was the answer accurate? Was the citation real? Was the advice misleading?

With an agent, the object of review is a chain of action. What instruction was given? How did the agent interpret it? Which tools did it access? What data did it retrieve? What permissions did it have? What did it change? Who approved it? What logs were kept? What rollback existed?

This is why agents are not only an AI model problem. They are an operational architecture problem. OWASP’s LLM Top 10 identifies “Excessive Agency” as a category of LLM application risk, where excessive functionality, excessive permissions or excessive autonomy can enable damaging actions.⁹ That is the agent risk compressed into one phrase: damaging actions.

The agent era shifts the trust problem from content to consequence.

---

VII. Rollback is useful. Boundaries are the product.

There is a strong technical argument for moving fast with agents. Software already has sandboxes, staging environments, CI/CD pipelines, static analysis, test suites, access control, monitoring, backups, and rollback. If an agent proposes a change, the system can test it. If it fails, reject it. If it breaks something, revert it. If uncertainty is high, escalate to a human. That argument is right in many low-risk contexts. A marketing agent can draft ten campaign variations and discard nine. A support agent can prepare responses for review. A coding agent can open a pull request that fails CI. An internal admin agent can update low-risk fields with audit logs.

But rollback has limits. You can revert code, you cannot always revert a leaked secret. You can restore a database if backups work, you cannot always restore customer trust. You can correct a mistaken email, you cannot make everyone unread it. You can amend a medical note, you cannot undo a clinical decision already made from it. You can reverse a payment, you cannot always reverse the legal, reputational or operational consequences around it. Rollback is a recovery mechanism. It is not a theory of responsibility. The more consequential the action, the less comfort rollback should provide.

Most agent demos focus on capability. Look what it can do: write the code, book the meeting, answer the customer, update the CRM, process the invoice, file the ticket, run the workflow. That is the wrong place to stop. The mature question is what it cannot do. What data can it never access? What systems are read-only? What decisions must it escalate? What requires dual approval? What action triggers a pause? What must be logged? What can be rolled back? What must never be attempted?

A powerful agent with vague boundaries is not enterprise software. It is a liability generator. The best agent products will separate themselves not by promising maximum autonomy, but by offering precise autonomy. The agent should be free where mistakes are cheap. It should be constrained where mistakes compound.

That principle sounds obvious. It is not how software is usually sold. Software vendors sell what a product can do. Agent vendors will need to sell what a product can safely refuse to do. That is a different kind of product maturity.

---

VIII. Agents amplify the organisation they enter

The hardest part of agents is not the model. It is the organisation.

Most companies are already messy before agents arrive. Permissions are too broad. Processes are half-documented. Data is fragmented. Approvals are informal. Logs exist but nobody reads them. Exception handling lives inside a few experienced employees. Security rules conflict with operational shortcuts. Systems are integrated in ways no one fully maps.

Agents enter that mess and make it faster. If the process is unclear, the agent executes confusion at scale. If permissions are sloppy, the agent misuses them at speed. If data is bad, the agent operationalises it. If oversight is symbolic, the agent creates the illusion of control while action moves beyond human comprehension. If nobody owns the workflow, the agent will not magically create ownership.

Agents do not fix broken organisations. They reveal them.

NIST’s AI Risk Management Framework treats trustworthy AI as a socio-technical problem. Its characteristics include validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy enhancement, and fairness with harmful bias managed. Those are not only model attributes. They depend on context, deployment, governance and organisational process.¹⁰

That is why “how smart is the agent?” is too narrow a question. The better question is what kind of organisation is the agent being inserted into. In a disciplined organisation, agents may create leverage. In a chaotic organisation, agents may create accelerated chaos.

The next phase of agent adoption will not be decided only by engineers or product managers. It will be decided by insurers, regulators, auditors, boards, customers, procurement teams and courts. When an agent causes harm, the questions will be brutally practical. Who approved this system? What was it allowed to do? Was the risk assessed? Were permissions appropriate? Were logs retained? Were humans able to intervene? Was the action foreseeable? Was there a rollback path? Did the company test the failure mode? Did the vendor warn about limitations? Did the organisation rely on the agent beyond its intended use?

The International AI Safety Report 2026 synthesises evidence on the capabilities, risks and safety of general-purpose AI, including the growing significance of agentic systems and the difficulty of controlling increasingly capable models acting through tools.¹² The insurance version of the same problem is simpler. If the system can act, someone will ask whether the organisation controlled the action reasonably. “The AI did it” will not be enough. Air Canada learned the early version with a chatbot. The agent version will be harsher.

---

AI agents are not dangerous because they are magical. They are dangerous because they turn language into action before responsibility has caught up.

The agent era will not be won by the company with the most autonomous workflows. It will be won by the organisations that know where autonomy belongs, where it must stop, what evidence it must leave, and who owns the result. A company can call the system a teammate, a worker, an agent, or digital labour. The name does not matter. The liability remains human. That is the uncomfortable truth underneath the agent boom.

This is Essay No. 007. The topics: intelligence, AI, systems, knowledge, and the questions underneath the questions everyone else is asking. If you read this far and disagreed with any part of it, write to me. I read everything.