• Should You Add a Bot to Your Board? – AI can improve board decisions, but only if directors use it to open up questions rather than to summarise the pack.

Executive Summary

• Boards rely on periodic attention, information provided by management, the judgment of non-executives, and assurance mechanisms.

• AI exposes weaknesses in all four of these dependencies.

• To close these gaps, boards should make seven changes, from continuous monitoring of controls to directors with more time for the company.

  Thanks for reading Considered Machines! Subscribe for free to receive new posts and support my work.

In 1994, Nick Leeson was Barings’ star trader. He also ran his own back office, allowing him to conceal mounting losses. An internal audit highlighted the risk and proposed that these responsibilities be separated. Barings, Britain’s oldest merchant bank, failed to act. The following February, the bank collapsed and was subsequently sold to ING for £1.

Barings had two problems. First, the person taking the risks was marking his own homework. Second, it failed to act on a clear warning. AI can reproduce the first problem and make the second more costly. Some agents keep their own logs and write reports on their work. They can also continue making decisions and commitments at machine speed while a warning goes unheeded. The damage can mount long before the next board meeting.

Why the Board Model Is Breaking

Boards are responsible for setting strategy, allocating capital, appointing leaders, overseeing controls and approving reporting. Four weaknesses in the way boards operate are becoming harder to live with. In each case, AI is at least partly the driver.

Speed. Periodic attention leaves gaps. The largest 150 FTSE companies average 8.2 board meetings a year. Few boards revisit the assumptions behind their strategy at every meeting, but AI can call those assumptions into question in days. In February 2024, Teleperformance’s shares fell almost a fifth during trading after Klarna claimed its chatbot was doing the work of 700 agents. A company’s claim about AI productivity was enough to move a competitor’s share price. Boards need a way to test, between scheduled reviews, whether the assumptions behind their strategy still hold.

Visibility. Management selects most of the information directors receive. AI is increasingly employed to gather and present the data that underpin the board pack. When agents both act and describe what they have done, boards need access to another source of truth. Otherwise, automation can make the company harder to see even as reporting appears to improve. At the Post Office, the findings of a review questioning evidence from the Horizon IT system were not shared with the full board.

Judgment. Directors’ judgment is rarely tested systematically. AI can make a weak argument look convincing, and using it can increase confidence without improving accuracy. In an experiment reported in Harvard Business Review, nearly 300 executives and managers forecast Nvidia’s share price. Those who consulted ChatGPT became more confident but produced worse forecasts than those who consulted colleagues.

Stability. Assurance can become stale. A change to an agent’s model, instructions, permissions or connected systems can undermine a previously effective control. Some agents are built with self-improvement loops that revise their instructions or working methods in response to feedback. The agent in operation can therefore differ markedly from the version that was tested. Boards need to know when those changes invalidate earlier assurance.

Seven Changes for a Board Built for AI

A faster cadence. Boards need to agree triggers that signal a need to reconsider strategy. The chair should require an alert when a major investment assumption is disproven, a competitor changes the economics of the market or risk exceeds an agreed limit. Each alert should identify the decision needed and its urgency. Boards can already call ad hoc meetings. The difference is that these triggers are agreed in advance, tied to specific assumptions and monitored by the board’s AI agent.

An independent source of information. Management should still write board papers, because doing so forces a stock-take. But the audit committee should commission its own AI-assisted analysis through the internal audit function. That requires access to underlying transactions and independently captured records that the operating AI agents cannot overwrite. It is tempting to use one agent to check the work of the first, but two models can be wrong in the same way. The head of internal audit is best placed to take responsibility for validating the analysis and surfacing findings to the board.

A named owner for each material agent. Each material agent, or process that uses agents, needs a business owner accountable for its decisions and commitments. One executive should oversee the company-wide arrangements and ensure that all agents are registered and monitored. The register should keep tabs on what each agent can do, its limits, and who has the authority to stop, override or amend it.

A record of decisions that can be tested. A feedback loop lets boards use AI to learn by analysing their own decisions. Board minutes already explain the reasons for decisions. Boards should add a machine-readable record of expected outcomes, key assumptions, alternatives and any dissent. The board’s AI agent can test these assumptions against new data, compare results with expected outcomes and flag meaningful deviations for the board’s attention. The company secretary should oversee the record, taking legal advice on retention, access and disclosure.

A scorecard. AI makes judgment both more important and more challenging. But judgment can be improved with feedback. Directors should record their forecasts, and how confident they are in their forecasts, before debate. The board should then record its collective reasoning. At agreed milestones, with the help of AI, expectations can be compared with outcomes to examine how the board and individual directors fared. A good decision can have a bad outcome, and vice versa, and one forecast doesn’t represent a pattern. Rather, the purpose is to expose recurring errors, such as overestimating synergies or accepting management’s timelines too readily. Each director’s own results should be private to that director; only the board-level pattern should feed into the board evaluation. These records may have to be disclosed in litigation, so the board should again take legal advice on retention and access.

Directors with higher availability. The increased cadence and requirement to look ‘beyond the pack’ will inevitably require board members to dedicate more time. Deloitte estimated a typical commitment of 35–45 days per director in 2023. A larger commitment implies fewer outside appointments and explicit availability between meetings. That narrows the pool of candidates and raises the cost of each; Deloitte found about 30 per cent of companies were already likely to consider resetting fees. Directors should be able to explain how AI changes the company’s economics and where authority has been delegated to agents. The nominations committee should test that understanding when appointing and evaluating them.

Continuous monitoring, independently checked. Management should monitor agent activity to ensure compliance with agreed permissions and limits. Internal audit should test whether that monitoring is effective and whether escalation paths are clear, with material exceptions escalated to the chair or relevant committee. Provision 29 already expects boards to monitor controls and review their effectiveness at least annually. AI increases the need for evidence that remains current between reviews.

Four Objections

Drift into executive action. A board with better information and more frequent meetings could start to behave like a second management team. But even with these changes, the traditional division of responsibilities can be maintained. Management operates the controls and responds to routine exceptions, whilst the board sets limits, challenges assumptions and intervenes when concerns remain unresolved.

Liability and records. More information creates more work, as it must be assessed and acted on. And recording more detail of the board’s deliberations creates evidence of what directors knew and why they acted. In August 2026, a Delaware judgment cited an AI transcript revealing a motive for a defensive measure that differed from the motive recorded in the official minutes. Boards need to decide explicitly what is recorded, ensure their records accurately reflect their reasoning, and take legal advice on disclosure obligations.

Over-reliance on agents. Directors should use AI to raise questions, but must reach their own conclusions. Different AI models can expose disagreements, but agreement between models should not be treated as independent confirmation, as models can make correlated errors. The board has to remain responsible for its own judgment.

Boards already have the structures, and still fail. This is the strongest objection. Silicon Valley Bank’s risk committee met 18 times in 2022. Carillion had an audit committee, an internal auditor and a Big Four external auditor, yet MPs found its non-executives had failed to scrutinise or challenge its executives. In both cases, the board was warned but failed to act effectively, and the company failed. Part of the solution is to name the decision required and its urgency, as the triggers above do, and to record how the board responds to contrary evidence, as the decision record does. These measures don’t guarantee action, but they do make inaction visible.

Where to Start

These changes can be tested within existing board responsibilities. At the next meeting:

• Clarify accountability for agents. Require named agents and their owners, limits and authorities to be registered and one executive to oversee the lot.

• Establish a learning loop for major decisions. Start with one decision. Record its assumptions, expected outcomes and conditions for reconsideration. Commission AI-assisted analysis (independently validated) to test material assumptions against source data and bring contradictions back to the board. If possible, review earlier decisions using their original papers and forecasts.

• Pilot continuous monitoring. Choose one material automated control. Commission an independent assessment of the pilot.

• Review appointments. Ask the nominations committee to assess directors’ understanding and availability to ensure that appointed directors are suitable for the future board model.

Barings failed as the person taking the risks was simultaneously overseeing the controls. AI makes that weakness easier to reproduce at scale. It also gives boards a way to examine the business directly, track their assumptions and test their own judgment. As companies delegate more work to agents that can learn and change quickly, boards will need to enlist AI to support these activities between meetings. A better-written board pack will not suffice.

Questions for the Board

• Which assumptions behind our strategy and largest investments could AI invalidate?

• Could we reconstruct the reasoning behind our biggest decisions, and learn from that?

• Who can stop an agent from making decisions or commitments beyond its authority?

• Do our directors have the time and understanding this role requires?

Sources & Notes

1. Barings. Board of Banking Supervision report, 18 July 1995: trading and settlement unseparated, audit finding ignored, bank sold to ING for £1. The lesson is about evidence and acting on warnings, not that AI would have prevented the collapse. Report

2. Board meetings. Spencer Stuart, UK Board Index 2025: average 8.2 meetings (7.5 scheduled) across the top 150 FTSE companies. Index

3. Teleperformance and Klarna. Klarna release, 27 February 2024. Klarna Reuters, 28 February 2024: shares down 19% mid-afternoon, after an intraday low of −29.3%. Reuters.

4. Self-improving agents. Zhang et al., “Agentic Context Engineering”, arXiv 2510.04618 (ICLR 2026). Shows the technique, not how widely it is used. arXiv

5. Post Office. Post Office Horizon IT Inquiry, document UKGI00019313: letter from the BEIS permanent secretary to the Post Office chair, 7 October 2020, confirming the 2016 Swift review’s recommendations were not shared with the rest of the board. Inquiry

6. Judgment. Parra-Moyano, Reinmoeller and Schmedders, HBR, July 2025. A share-price forecast, not a board decision. HBR

7. Decision records. FRC Corporate Governance Code Guidance expects minutes to record the reasons for decisions. FRC

8. Time commitment. Deloitte, Non-executive director fees: time for a re-set?, November 2023; based on polling. The same report found about 30 per cent of companies likely to consider a fee re-set. Deloitte

9. Provision 29. UK Corporate Governance Code 2024; applies to periods beginning on or after 1 January 2026. Code; Mythbuster

10. Duties and disclosure. Companies Act 2006, s174. Legislation ATG Capital Opportunities Fund LP v Lane, C.A. No. 2026-0447-LWW (Del. Ch. 28 August 2026): an illustration of evidential risk, not UK law. Opinion

11. Silicon Valley Bank. 18 risk committee meetings in 2022, per SVB’s 2023 proxy. Forbes Supervisory and governance failings: Federal Reserve review, April 2023. Fed

12. Carillion. Business, Energy and Industrial Strategy and Work and Pensions Committees, Carillion, May 2018: the board held responsible for the failure; non-executives failed to scrutinise or challenge executives; Deloitte as internal auditor and KPMG as external auditor both criticised. Parliament

One Company, a Thousand Workflows – Employees are building their own AI tools and workflows. Left private, unowned and disconnected, they become a new class of legacy system.

Executive Summary

• AI allows employees to design and build their own workflows faster than colleagues can understand how they fit together.

• Employee-made tools can unintentionally omit verification steps or create downstream issues for other teams.

• Companies can empower employees to innovate by enforcing clear rules for how workflows interact with each other and by building a process for updating those rules.

• AI can help find tools, dependencies and gaps, but management must set the rules and ensure accountability.

• The technology for this exists. The focus should be on deciding who owns the rules and on funding the services that enforce them.

Subscribe now

The Great Fire of London destroyed 13,200 houses in 1666. In its aftermath, Christopher Wren proposed a design for a new London, inspired by Paris, with broad avenues and piazzas. His plan was never adopted due to property rights, compensation costs and the need to get merchants back in business quickly.

Instead, London was largely rebuilt using its existing street map, but under new rules. The 1667 Act categorised streets by type (e.g., high streets, lanes), required brick or stone construction and specified building heights and the design of party walls between neighbouring properties. Surveyors enforced the requirements, a coal levy paid for the public works, and a special Fire Court settled disputes between landlords and tenants over who should pay for construction. By the end of 1670, about 6,000 houses had been rebuilt. Individual builders could get on with their work because these shared arrangements (rules, funding and a way to settle disputes) were in place.

Later, in 1847, sewer connections became compulsory for new buildings. These sewers discharged into the Thames. As more properties connected, their combined waste became a growing public-health hazard. Cholera returned in 1853–54, followed by the Great Stink of 1858. Intercepting sewers were eventually built to carry waste away from central London.

Now, companies face a similar challenge as AI allows more employees to build and update their own workflows. Each tool may suit its creator, yet have a detrimental impact on the organisation as a whole, for example by making customer promises that can’t be kept, causing bottlenecks downstream, or introducing autonomous work that is hard to audit.

Prescribing every working method would curtail employee-led innovation. London’s evolution suggests a more effective approach: leave people free to design their own methods, while the company takes responsibility for the rules and services that allow them to work together.

Everyone Can Build, But Consequences Are Shared

At Centrica, Nishant Jain led application development for the finance team, but did not consider himself technical. Nevertheless, his team built an application in two months to coordinate tax returns across regimes, replacing a manual process that relied on emails. Centrica scaled this employee-led development approach. By 2021, it had more than 1,000 applications across the business, up from around 300 two years earlier.

As this example shows, employee-led invention predates generative AI. But AI changes who can build software, how quickly they can revise it, and the reach of the tools that they build. Low-code tools lowered the barrier to building, but AI lowers it further, and agents can choose and sequence actions on the company’s behalf. An employee can now change a workflow in hours. But other teams may still be working on the assumption that the workflow behaves as it did yesterday.

In many organisations, coordination between teams still happens informally. Colleagues check stock and delivery capacity before anyone commits to a delivery date, for example. This ‘human middleware’ is liable to fail when workflows change rapidly.

Enabling individual innovation whilst maintaining smooth operations requires the boundaries between teams to be formalised and the checks that protect the wider organisation (e.g., checking stock and delivery capacity before promising delivery) to be made explicit.

Four Things the Company Must Provide

1. Provide common handovers between tools. London’s party-wall rules required builders to leave ‘toothing’ brickwork so the next house could join seamlessly. Employee-built tools also need agreed connections. For each recurring type of work passed between teams, the managers of those teams must agree what information is handed off.

For example, sales and engineering might agree and publish a standard customer brief including customer requirements and the requested delivery date. Sales employees are then free to build their own sales workflows, provided that the output conforms to the agreed brief. On the other side of the handover, engineering employees can also build their own workflows, taking the data in the brief as a starting point.

However, a handover standard is insufficient in three important cases:

• When multiple tools compete for the same resource, such as stock or an engineer’s time.

• When a tool commits the company to something, such as a price or a delivery date.

• When a tool changes a record that other teams rely on, such as a customer’s details or a supplier’s bank account.

In these cases, two tools that conform to an agreed handover standard can still conflict because they act independently. That is when the company needs a core service: a shared method that allocates, commits or updates on everyone’s behalf.

2. Provide core services that enforce the business rules. London’s houses depended on shared streets and public works. In the three cases discussed above, employee-built workflows need to access a core service.

For example, a mandatory quotation core service might dictate prices and reserve stock when called by an employee-built sales tool. Employees can still gather requirements, compare options, and prepare the offer in many different ways, while the same pricing and reservation rules apply to every offer.

The core services themselves enforce business rules because instructions to the AI are not enough on their own. Where an employee-made tool can make promises to customers, that action must also follow the authorised route. Each core service needs an owner and a budget.

3. Make workflows and core services visible. London’s new buildings were visible to everyone. Employee-made tools are harder to see. Every company needs a register of its homegrown methods, core services, and handover specifications, with their owners and dependencies. It also needs common definitions for terms such as ‘customer’ and an auditable way to trace information back to its source. Core services and handover specifications also need to be visible so that employee-builders can access them quickly.

4. Check methods before and after deployment. London’s surveyors checked new buildings against the rules. Companies need to routinely check for permissions, approved connections and adequate fallback arrangements. The level of scrutiny should reflect how widely a method is used, the sensitivity of its work and whether mistakes can be reversed. Repeat the relevant checks when workflow logic, standing instructions, models, dependencies, or permissions change. Test representative journeys across connected tools and core services, including failed handovers and competing requests. Someone independent must periodically confirm that the controls work and exceptions are visible.

Much of This Can Be Automated

It takes effort to set up these mechanisms, but much of their day-to-day administration can be automated. AI tools used by employees can be set up with templates and connections to core services. Permissions can be set to prevent employee-built tools from accessing core systems directly.

The company’s managed tech platforms can be set to automatically register new employee-built tools and run routine checks whenever they change. AI can help maintain the register of tools by examining code, instructions, and activity records to deduce what each tool does and what it depends on. It can also flag missing information or changes that need attention. It will not see everything, so periodic manual reviews will still be needed.

By mid-2026, Centrica’s ‘Front Door’ system was detecting new app and workflow development within its Power Platform environment, asking creators about each tool’s purpose, users, and risks, and calculating a risk score. Higher-risk projects were sent for human review. Centrica also extended the same controls to AI agents, governing who could access them and how they could be published.

Old Solutions, New Problems

None of these mechanisms is new. Around 2002, Amazon issued a mandate requiring teams to communicate through service interfaces rather than access databases directly, according to former engineer Steve Yegge. Software architects have written service contracts and kept catalogues for years.

What is new is that AI increases the number of people who can change workflows and the frequency with which they do so, making it hard to review every change. Employees can now create far more powerful tools, for example, by spinning up AI agents that act autonomously and vary behaviour depending on context. Model updates or retirements can change behaviour even when the employee hasn’t changed the workflow. Business rules that colleagues once applied through judgement increasingly need to be explicit and enforced whenever a tool takes action.

Different Tools, Common Rules

Let’s take a couple of examples. Suppose two salespeople have built different quotation tools. Both see one item available and promise it to a customer. Each tool works as intended, but the company sells the same item twice. Both tools can continue to exist if they call a core reservation service. That service allocates the item once, so the second salesperson gets a refusal or a proposed alternative.

The same principle applies to supplier management. A marketing team might build a tool to onboard freelance designers, while a factory builds one for maintenance contractors. Each tool gathers different information, suited to its work. To ensure the right controls are followed, tools should use a core supplier-registration service. It checks whether the supplier already exists, routes the necessary checks to the accountable people and records the outcome.

Build the Service Before Making It Compulsory

London’s compulsory sewer connections show how individual compliance can concentrate problems in infrastructure that is not equipped to handle them.

Requiring employee-built tools to use an approved core service makes that service a dependency for everyone using it. If it is slow, unreliable or unable to handle the demand, that creates an issue for the whole business. A core reservation service that works for one sales team may struggle when hundreds of tools begin calling it automatically.

Management must fund core services to meet demand and provide fallbacks if they fail. If completing a routine task through the core service takes two days while bypassing it takes ten minutes, employees have a strong incentive to work around it. So the service must be built, tested and sized for demand before tools are required to use it.

The Rules Must Be Able to Change

Employees will quickly discover better ways of working that the core services do not yet support. If they can build a tool in an afternoon but changing a core service takes six months, their freedom to innovate is limited. They need to know how to request a change, who can approve it and when to expect an answer. Core service owners need funding for improvements as well as day-to-day operation.

Tools may also need updating when an AI model is withdrawn, a security flaw is found or business requirements change. The register should show which tools are affected and who is responsible for updating them. A central update may fix the connection to a replacement AI model, but every affected workflow still needs to be re-tested.

Before changing a core service, give teams that depend on it time and support to adapt. The old and new connections may need to run alongside each other for a fixed period. AI can help find dependencies and suggest repairs, but owners remain responsible for checking the results.

London’s rebuilding needed clear rules, funded public works and people responsible for maintaining them. Companies need arrangements that employees can rely on, and a practical way to change them as the business learns. Together, these make an organisation evolvable.

Monday Morning Actions for Executives

• Set the direction. Encourage employee-built tools but specify the rules for such innovation.

• Clarify accountability. Each core service needs a named business owner accountable for its rules, service levels, and evolution. The CIO is accountable for the supporting technology and technical controls.

• Assess the health of existing core services. Check that they provide the required level of service and are adequately funded.

• Make the rules open to challenge. Give employees a route to propose changes and require owners to respond.

Questions for the Board

• Where could employee-built tools drive improvements to our operations?

• Is employee-led innovation enabled and contained by clear boundaries and rules?

• Do we have an assessment of the risk to operations presented by employee-built tools?

• Can our core services and rules change as fast as our employees do?

Sources

1. University of Oxford, Faculty of History. “Facing up to catastrophe: The Great Fire of London”. Fire losses and the pace of rebuilding.

2. Hebbert, M. “The long after-life of Christopher Wren’s short-lived London plan of 1666”. UCL. Why Wren’s plan was not adopted.

3. Statutes of the Realm, vol. 5. “An Act for rebuilding the Citty of London” (1667). Building rules, surveyors and the coal levy.

4. Statutes of the Realm, vol. 5. Fire of London Disputes Act 1666. The court for resolving rebuilding disputes.

5. The London Archives. “Drainage applications and plans”. The introduction of compulsory sewer connections.

6. Hansard (1858). Debates of 15 July and 19 July. Parliamentary funding for London’s sewer works.

7. London Museum. “How Bazalgette built London’s first super-sewer”. The sewage crisis and Bazalgette’s response.

8. Microsoft (2021). “Centrica leverages Power Apps to ensure its solutions are futureproof”. 22 November. Supplier-published case study of employee-built applications at Centrica.

9. Microsoft (2026). “Centrica drives a new wave of Power Platform innovation with Managed Environments and Copilot Studio”. 19 June. Supplier-published account of Front Door, risk reviews and AI agents.

10. Yegge, S. (2011). “Stevey’s Google Platforms Rant”. His account of Amazon’s service-interface mandate.

11. Evans, E. (2003). Domain-Driven Design. Background on boundaries between systems.

12. Skelton, M., and Pais, M. (2019). Team Topologies. Background on platform teams and ownership.

13. Anthropic (2024). “Building effective agents”. 19 December. The distinction between workflows and agents.

14. Microsoft. “Model migration”. Guidance on testing and adapting applications when models change.
    

    Thanks for reading Considered Machines! Subscribe for free to receive new posts and support my work.

The Org Chart You Didn’t Design. Conway’s Law has gone into reverse, as AI agents reshape organisations

Executive Summary

• Workflows and methods ‘mutate’ as front-line workers find better ways to complete their work, creating numerous variants.

• If harnessed, this variation is valuable as it can deliver performance improvement but also increase resilience to shocks.

• Selection identifies which variants to scale, retain or retire. It must govern what enters and remains in active use, so that experimentation does not create unmanageable complexity.

• AI tools are turbo-charging mutation whilst making selection more challenging, creating opportunities but also risks.

• AI therefore presents a paradox. It raises volatility in the environment, gives workers the tools to respond to that volatility, but hands managers new ways to constrain how workers respond.

• Management systems (e.g., budgeting) built around standardisation and central approval are poorly suited to overseeing a mutating portfolio of workflows. Managers sometimes respond by trying to eradicate variants.

• To unleash the power of front-line-led innovation whilst containing risk, managers need to shift from being ‘selectors of variants’ to ‘builders of selection machines’.

In 1943, Salvador Luria, a research scientist, was trying to unwind at a faculty dance in Bloomington, Indiana. His research into the behaviour of bacteria attacked by a virus was at an impasse. Most of the bacterial cultures grew into a handful of resistant colonies, but a few resulted in hundreds. During the evening, Luria watched a colleague win three dollars on a slot machine and immediately understood what his experiments were telling him. If exposure to the virus had induced resistance, the numbers of resistant colonies should have varied much less between cultures. The large differences instead suggested that resistant strains of bacteria were already present. Luria later shared the 1969 Nobel Prize in Physiology or Medicine with Max Delbrück and Alfred Hershey.

A business can become more inventive while becoming less capable of adapting. AI makes it easier to build new workflows, but those methods can enter daily use, spread and acquire dependencies before their performance and risk have been established. The challenge for leaders is not just evaluating an increasing number of ideas. It is governing an ever-expanding population of methods which are already up and running, and deciding which should scale, remain available or disappear.

Mutation Helps Business

Companies mutate and produce variants too. Organisations can be thought of as a collection of workflows (e.g., order to pay, hiring, software development). Workflow mutations arise over time, as employees avoid bottlenecks (e.g., ineffective IT systems) or find faster ways to complete a task. At Heathrow Airport, security officer Samit Saini used a coding tool to build an app that explains security rules to passengers in different languages. It was subsequently adopted across the airport’s security operations. It’s not uncommon to find dozens or even hundreds of variants of one workflow. When Accenture examined its purchase requisition process across 50 countries, it identified 14,000 different routes through the system, even though a single global instance of SAP was in place. Standardising the system doesn’t always standardise the work.

Businesses that can harness the power of these variants will benefit in two ways. Firstly, variants can deliver performance improvement. Built and tested by those who will live with them day in, day out, they are more likely to reflect the realities of the front line, automate low-value tasks and leave human intervention where it matters. Xerox built a system called Eureka that allowed local technicians to submit repair tips. Once validated by experienced peers, they were made available to all colleagues. In one case, a technician in Montreal uncovered a quick fix that avoided the need to replace a $40,000 copier in Brazil.

Secondly, retained variants can deliver increased resilience to shocks such as cyber attacks or supply chain disruptions. In March 2019, ransomware disrupted IT systems at Norsk Hydro. The company was able to run some operations whilst the computers were recovered, using old paper-based processes and retired employees who volunteered to help.

But not all variants are better than their predecessors. Firms must decide which to scale, retain or retire. Variants should be scaled if they improve end-to-end performance at an acceptable cost and risk. They should be retained if they preserve a usable alternative method (for example, they can function without using an AI model). And they should be retired if their value (either performance or resilience) no longer justifies the costs and risks of keeping them running. Novelty alone is not grounds to retain a variant.

The unit of selection here is a method, not a person. A company evolves only when better ways of working are identified, retained and passed on.

Balancing Mutation and Selection

In business, variation becomes a liability when workflows enter active use faster than the organisation can evaluate, coordinate and maintain them. The numbers won’t add up as financial models use different assumptions. Departments can’t talk to each other as data models diverge. Technical debt spirals as new code is added but not maintained. In 1997, Nestlé USA discovered its brands were paying 29 different prices for vanilla from the same supplier. Each factory negotiated its own deal and used its own ingredient coding, making purchases harder to compare across sites. The same review uncovered nine general ledger systems and 28 different points of customer entry.

Businesses try to constrain mutation through standard operating procedures and ERP systems that hard-code processes. As Accenture found, this is harder than it looks. Selection among existing variants occurs through a different mechanism. As employees migrate towards variants that help them meet their targets, some spread virally whilst others fall out of use. These mechanisms can limit operational complexity, but they do not guarantee that selection keeps pace with mutation.

Drawing analogies between biological populations and work is not a new concept. Since Nelson and Winter’s work in 1982, economists have described companies as generating, selecting and inheriting routines. But AI changes the relative economics of those activities. Creating a new method has become much cheaper and faster. But the cost or time needed to establish whether that method has value is not falling at the same rate.

AI is Accelerating Mutation

AI tools are massively accelerating workflow mutation. Front-line employees can spin up their own processes, data and technology using off-the-shelf LLMs. What used to take months now takes minutes. In 2025, Spanish banking giant BBVA reported that its employees had built more than 20,000 custom AI tools to support day-to-day operations.

And AI doesn’t just increase the number of variants. It also increases the granularity of what is being varied. Workflows tend to sub-divide when the benefits of task specialisation outweigh the incremental coordination costs. With AI, more granular tasks become economic. For example, AI has made it easier to divide the end-to-end hiring workflow into more granular units such as advert placement, application ranking and interviewing. Each of these units can mutate, and the variants then recombine into hiring workflows.

These two effects multiply when workflow fragments can be freely recombined. Illustratively, with 10 variants at each stage of a three-step hiring process, there are 1,000 possible end-to-end workflows.

The Fitness Challenge

This explosion in variant combinations also makes it challenging to identify the high-performing variants. The fitness of any one variant depends on those variants that surround it. So in our hiring example, an advert placement variant may work best with one candidate ranking variant and poorly with another. As a result, the answer to ‘which is the fittest variant?’ may be ‘it depends’.

The fitness of a variant can be hard to measure. A workflow using an LLM may produce different results from the same inputs. Model upgrades, changing data sets and shifting user behaviour can alter its results over time. Evaluation must distinguish a repeatable improvement from a one-off apparent success or change in conditions. Some evidence simply takes time to arrive. A new hiring workflow can be built in an afternoon, whilst evidence about the quality of its hires may take months to collect.

Fitness also depends on whose outcome is being measured. Suppose a sales team automates quotations, halving the time spent on each one. If this new workflow creates problems for the fulfilment team, the net effect may be only to transfer effort from one team to another. A variant should not count as fit merely because its creator benefits. Selection must account for downstream effects, as local optimisation can make the organisation less effective overall.

Most fundamentally, fitness can be hard to define. An outbound sales email’s open rate yields a clear KPI, whilst the quality of a hiring decision is harder to quantify.

Selection Becomes the Bottleneck

Selection becomes a bottleneck when the flow of variants needing evaluation exceeds the firm’s capacity to assess them. This can happen even if AI makes evaluation faster. AI can compress the time needed to create an experiment without necessarily compressing the period needed to learn from that experiment.

A related pattern is emerging in software development. Studying more than 500,000 GitHub developers across three generations of AI coding tools, Demirer, Musolff and Yang found that the gains shrank as the work progressed towards shipping. Commits (recorded code changes) rose by around 240 per cent cumulatively, projects by roughly 80 per cent, but releases only by 30 per cent. AI boosts creation but that doesn’t all flow into delivered output, as the constraint moves downstream.

Risks Spiral as Mutation Outruns Selection

Any individual new variant may create risk if critical control steps are omitted, cyber vulnerabilities are exposed or looping AI agents ramp up cost. Collectively the risks are more fundamental. The organisation can ‘splinter’ into fragmented units (potentially one per employee), each with its own workflow variants. Processes become opaque to managers, end-to-end processes break at the joins and single points of failure may be created (e.g., dependence on a specific LLM). Agents, collaborating with other agents, can paper over the cracks, obscuring the issues without solving them.

Reaping the benefits of workflow mutation whilst avoiding the risks therefore requires a management system that scales methods that improve performance, retains alternatives that preserve resilience and retires variants that no longer justify their costs.

But this reveals a deeper issue.

Today’s Management Systems Are Weak Selectors

Management systems built around central approval, periodic review and standardised execution are poorly suited to a rapidly mutating portfolio of workflows. Xerox’s Eureka process showed that it is possible to support variation, selection and scaling. The problem is not with management itself, but with management systems which are designed primarily to minimise variation. These systems face four mismatches:

• Firstly, mismatched objective. These systems try to minimise variation. Managing for evolvability requires variants to not only be tolerated but encouraged.

• Secondly, mismatched velocity. These systems operate on quarterly cycles and annual budgets. AI allows workflows to mutate by the hour, minute or even continuously if agents are asked to self-optimise.

• Thirdly, mismatched direction. These systems assume that change originates centrally and flows down to the front line (albeit with input from workers). Workflow variations are built bottom up by workers for workers.

• Finally, mismatched scale. These systems treat processes as monolithic and enterprise-wide. Workflow variations are fundamentally local, although they have the potential to be deployed widely.

Selection still happens even if managers fail to govern the process. Methods tend to spread by word of mouth, often because they are convenient, visible or sponsored by an influential colleague. Those pressures are likely to favour what is easy to adopt over what works well for the business. Managers therefore cannot simply allow a selection system to emerge; they must proactively design one.

But it gets worse.

Managers Can Suppress Mutation

Overwhelmed with higher-velocity mutation and a resulting explosion in variants, managers can respond by strengthening controls. Decisions are centralised, tech permissions revoked, SOPs reinstated.

AI provides managers with new methods to exert control too. Pest-control firm Aptive deployed AI to standardise the customer cancellation process. The system prompted staff to ask particular questions in a prescribed sequence. Adherence to the prescribed discovery behaviours (establishing why customers wanted to cancel) increased from 28% to 59% in four months.

So, the great paradox of AI is that it a) increases the volatility of the business environment and the need for firms to improve resilience to shocks; b) gives workers the tools to respond quickly to changes in the environment; and c) gives managers the tools to constrain how workers respond.

The reflex to exert controls is understandable but the evidence suggests it can be expensive. A study of US and OECD firms found that companies which had devolved authority to local managers before the financial crisis outperformed centralised firms in the worst-hit sectors. Turbulence raises the value of local information and the ability to act on it quickly. Centralisation can therefore be a costly response to a shock, just when local information and rapid adaptation become more valuable.

There is a difference between standardising a workflow and standardising the conditions under which workflows are evaluated. Variants can be compared with each other if their inputs are transparent, their outcomes are verifiable and ‘connectors’ to neighbouring processes are standardised. Such standards make variation transparent and testable. Management can specify how success is defined and proven, whilst the front line remains free to discover better ways of doing the work.

Selecting Selection Machines

How do we solve this paradox? The answer is to change the manager’s primary role from selecting individual variants to designing how selection happens. Managers define what counts as fit, what evidence is needed, and which decisions require human judgement. The selection machinery then converts those conditions into operational change by scaling, retaining or retiring variants.

Not all selection machines need to be AI-driven. Selection machines must gather evidence, take decisions and then take action (e.g., by replicating a ‘fit’ variant). Some tests can be automated but others will require human review, especially where decisions are irreversible. An AI’s review of a method it helped create can’t be treated as independent validation. Nor should a second model automatically count as independent. Different models can make correlated errors. Where errors could cause meaningful damage or risk, validation must be independent of the process that produced the variant.

To be effective, a selection machine must gather reliable evidence at low cost and remove avoidable delays. Where outcomes take time to emerge, it must oversee limited trials and defer wider deployment until sufficient evidence is collected. Popularity should not be mistaken for fitness. Workers’ willingness to use a variant and evidence that it is ‘fit’ are not the same thing, and selection machinery must distinguish between them, rather than automatically promoting whatever spreads fastest. And it must translate evidence of fitness into differential replication, by providing winning variants with more resources, more usage, more distribution or more copies.

For example, suppose that three sales quotation methods are already in use in the organisation. The most popular saves time in sales but creates more work in fulfilment, making it the most expensive overall. A second method lowers total cost without reducing quality. A third method is slower but cheap to maintain and can produce acceptable quotes without using the core AI model. Selection would ideally retire the first, scale the second and keep the third running at a limited scale.

The selection machinery would track each quote, link it to the method that produced it, and evaluate the end-to-end success of the method, including corrections, delays and costs downstream. Agreed thresholds would trigger scaling of the method or a human review. The third (backup) variant would retain an owner and be tested periodically with the core AI model unavailable. Management would review the selection machine’s dashboard and tweak the parameters rather than approve every variation individually.

Selection must result in changes to the business, not just produce a ranking of variants. Winning variants need an owner, support and a route to wider deployment. Retained alternatives also need owners and periodic checks that they remain usable. Variants that no longer justify their costs must be eased out of use so that complexity can be constrained. A selection log and audit trail should be maintained so that continuous learning on selection effectiveness is possible. Testing, promotion, retention and retirement must be parts of an integrated system. In time that system may acquire variants too.

Evolvable

Organisations need to continually evolve. There is little benefit in selecting a single best workflow and freezing it into place as a one-off event. A variant that is inferior under today’s conditions may prove valuable when demand changes or a critical system fails. Not every variant is worth keeping. But there is value in maintaining a portfolio of variants that offer useful responses to different conditions. The capability being built is evolvability. This is the ability to keep producing responses that work even as conditions change. Selection machinery must reduce needless complexity without selecting away the capacity to adapt.

Watching the slot machine, Luria realised that introducing the virus had not created resistance. It had revealed which bacteria already possessed it. For firms, the lesson is to keep useful alternatives workable before a shock reveals which ones they need. Firms that fail to build selection machines to harness front-line-driven mutation are gambling with their futures.

Sources

1. Luria, S. E., and Delbrück, M. (1943). “Mutations of Bacteria from Virus Sensitivity to Virus Resistance.” Genetics, 28(6), 491–511. The original paper establishing that bacterial resistance could arise before exposure to the virus.

2. Murray, A. (2016). “Salvador Luria and Max Delbrück on Random Mutation and Fluctuation Tests.” Genetics, 202(2), 367–368. An account of the experiment and Luria’s insight after watching a colleague win a three-dollar slot-machine jackpot.

3. NobelPrize.org. “The Nobel Prize in Physiology or Medicine 1969.” Official record of the prize shared by Max Delbrück, Alfred D. Hershey and Salvador E. Luria.

4. Microsoft (2019). “Heathrow Airport inspires employee engagement with Microsoft Power Apps.” Microsoft Customer Stories, 7 November. Supplier-published case study describing Samit Saini’s language-book app and the development of employee-created applications at Heathrow.

5. Accenture (undated). “Turning process friction into flow.” Company case study describing the identification of 14,000 requisition-processing paths across 50 countries despite the use of common enterprise systems.

6. Bobrow, D. G., and Whalen, J. (2002). “Community Knowledge Sharing in Practice: The Eureka Story.” Reflections: The SoL Journal, 4(2), 47–59. An account of Xerox’s technician-led knowledge-sharing system and how front-line practical knowledge was shared across the organisation.

7. Norsk Hydro (2021). “2019: Cyber-attack on Hydro.” Updated 11 June. Company account of the March 2019 attack, its uneven operational impact and the use of manual procedures to maintain production.

8. Briggs, B. (2019). “Hackers hit Norsk Hydro with ransomware. The company responded with transparency.” Microsoft Source, 16 December. Includes the account of retired employees returning to help operate older paper-based processes.

9. Worthen, B. (2002). “Nestlé’s Enterprise Resource Planning (ERP) Odyssey.” CIO, 15 May. Original reporting on Nestlé USA’s fragmented purchasing and information systems, including the 29 vanilla prices, nine general ledgers and 28 points of customer entry.

10. Nelson, R. R., and Winter, S. G. (1982). An Evolutionary Theory of Economic Change. Cambridge, Massachusetts: Belknap Press of Harvard University Press. Foundational treatment of organisational routines, search and evolutionary economic change.

11. OpenAI (2025). “How BBVA is scaling AI from pilot to practice across the org.” 6 November. Supplier-published case study reporting more than 20,000 custom GPTs created across BBVA, with approximately 4,000 used frequently.

12. Demirer, M., Musolff, L., and Yang, L. (2026). “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools.” NBER Working Paper No. 35275, issued May; revised September. The revised study covers more than 500,000 GitHub developers and reports estimated cumulative increases of approximately 240% in commits, 80% in projects and 30% in releases.

13. Cresta (undated). “How Aptive Drove $2.37M in Retention with Cresta AI.” Supplier-published case study of AI-assisted customer retention, including an increase in adherence to discovery behaviours from 28% to 59% over four months.

14. Aghion, P., Bloom, N., Lucking, B., Sadun, R., and Van Reenen, J. (2021). “Turbulence, Firm Decentralization, and Growth in Bad Times.” American Economic Journal: Applied Economics, 13(1), 133–169. Research on pre-crisis delegation to local plant managers and subsequent performance in sectors hardest hit by the Great Recession.

15. Kim, E. M., Garg, A., Peng, K., and Garg, N. (2025). “Correlated Errors in Large Language Models.” Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 30038–30066. Evidence that different models can make correlated errors, including implications for model-based evaluation.

Subscribe now

Is AI Blunting Your Strategy? – As strategy development relies increasingly on LLMs, true differentiation from competitors is becoming more elusive.

Executive Summary

• AI spending is rising quickly, and model charges are only part of the cost.

• Budgeting for AI is challenging as unit prices are falling, usage is rising, and AI agents are changing consumption patterns.

• Productivity gains are often hard to convert into realised financial benefit.

• For budgeting purposes, organise AI spend into five funding categories, each with different owners and controls.

• Approve a funding envelope, maintain a rolling forecast, and release funds in stages as the evidence strengthens.

Subscribe now

Unpredictable Spend, Untraceable Returns

Budget season is upon us, and the battle lines are already being drawn. On one side, the advocates – colleagues who are already seeing benefit from AI usage and would like to ramp up spend. On the other, the sceptics – those who are finding it hard to see where AI is hitting the bottom line and prefer a more disciplined approach.

That tension is a normal and healthy part of any technology budgeting exercise. While there will always be gamesmanship, the debate can be mediated using tools such as benchmarking, cost-driver analysis (e.g., headcount) and project ROI ranking.

Yet budgeting for AI is harder. AI spend is growing but not in a predictable way. Hidden costs, such as verification of AI output, need to be accounted for. Returns are both obvious and invisible.

Bain’s 2026 survey of 102 CFOs found that 83 per cent expect to raise enterprise‑wide AI spending by more than 15 per cent over the following two years and 42 per cent expect an increase of at least 30 per cent. Yet only 31 per cent were highly satisfied with current outcomes.

The harsh reality is that the agreed budget number will probably be wrong within months. As one analyst put it, a lot of CFOs are about to see their AI bill and “freak out.”

Cheaper and More Expensive

Token costs are collapsing. Stanford’s AI Index found that the price of querying a model at GPT-3.5 performance fell about 280-fold between November 2022 and October 2024.

But lower prices stimulate usage, as students of Jevons know. As more capable AI models are released, new use cases become viable. For example, early LLMs couldn’t handle complex customer interactions, but the latest models perform far better.

More significantly, use of AI is shifting from synchronous chatbot interactions to autonomous agents. An agent can call other models, invoke tools and repeatedly iterate a task. Agents also spin up other agents, making a mockery of headcount-based budgeting. Claude Code creator Boris Cherny says he sometimes runs hundreds of agents in parallel.

Similar dynamics were seen as other general-purpose technologies emerged. The cost of a unit of electricity in the US fell by more than 97 per cent between 1902 and 1950, yet total spending on electricity rose, because cheaper power led to people buying more electrified devices.

And the AI bill is climbing. In BCG’s 2026 survey, respondents expected their organisations to spend about 1.7 per cent of revenue on AI, more than double the 2025 number, although with significant differences by sector. Businesses on Ramp that spend on AI allocate nearly 15 per cent of their software budget to AI tools, on average.

AI models themselves are only part of the total bill. Workflow redesign, data, systems integration, evaluation, security, human review and upskilling also need to be factored in. BCG’s 10-20-70 rule of thumb for AI transformation splits the spend 10 per cent to algorithms, 20 per cent to data and technology, and 70 per cent to people and processes.

Some of the increase in AI costs is being driven by the shift to agentic workflows. EY estimates that the cost of a single customer service interaction has risen from about four cents in 2023 to around $1.20 in 2026, a thirtyfold jump. The token price per unit fell over that period, but the workflow around the model drove up costs due to reasoning steps and repeated attempts by agents to complete tasks.

Understanding Your AI Spend

The first task in an AI budgeting exercise is to find the current spend. AI spend is buried in SaaS renewals, cloud accounts, consulting projects, data teams and shadow AI. Sometimes AI spend arrives unseen, when a software upgrade switches on AI features. KPMG found that only 26 per cent of large US organisations had full visibility into the cost of running AI at scale.

SaaS spend shouldn’t simply be classified as AI spend simply because AI is bundled into the product. Include only clearly identifiable additional AI costs. Once the scope is agreed, organise the spend into five practical funding categories, each with different owners, drivers and controls:

1. Committed operating costs are contractually committed, or broadly fixed, within the planning period. They include licences, subscriptions and embedded AI features bought at a flat price or agreed contract tier. These costs should be funded and managed by the function or business unit consuming them, using centralised procurement for leverage.

2. Usage-variable operating costs are the costs of keeping AI running in production that vary with usage. They include model calls, agent runs, retrieval of information and associated human review activities. These scale with volume, which in turn is driven by the number of users, intensity of use and AI models used. A brand designer using an image model day in, day out may generate far higher usage costs than an executive who seeks occasional help to draft emails. The added complexity in this category is the proliferation of AI agents. As users deploy more agents, and agents spawn further agents, usage can grow non-linearly. These costs should also be funded and managed by the function or business unit consuming them but should be subject to central controls (e.g., metering, quotas) to avoid surprises. And surprises do happen. Uber reportedly exhausted its 2026 budget for AI coding tools by April.

3. Build and change costs are the costs of getting new AI into the business. They include project management, workflow redesign, systems integration, data preparation (substantial if existing data is of poor quality), user testing, and upskilling people. Any organisation whose systems cannot yet support agentic workflows must close that capability gap first, and for some that foundational cost will be substantial. For firms building or tuning their own AI models, model training costs will also be a factor. This expenditure is discretionary, at least until projects are green-lit. These costs should be funded like any other project, from a central change budget or the sponsoring unit, with funding released against milestones. Two things make AI projects harder to budget than other IT projects. First, AI systems need regular checks after launch because changes in data, workflows or the underlying model can cause their performance to ‘drift’ over time. That can make ongoing maintenance more costly and less predictable than other IT systems. Second, for firms that train or fine-tune their own models, build costs are challenging to estimate upfront, as there are many variables, such as the quality of training data, that cannot easily be pinned down until work starts.

4. Shared platform costs cover the common foundation that every function and business unit draws on. They include infrastructure, tooling, the model-access layer and shared data. This cost is step-fixed, growing as capacity and teams are added, and should be funded centrally and recharged based on consumption once reliable measurement is in place. Until then, it can be held centrally as an investment.

5. Control and assurance costs are the costs of keeping the entire system safe and defensible. They include evaluation, monitoring, security, legal, governance and audit support. These costs are semi-fixed, increasing with the number of deployments and regulatory scrutiny rather than volume. They should be funded centrally.

R&D costs and business unit experiments are not a sixth category. Rather, they are combinations of these five, with limited funding and an explicit decision to stop or scale up.

These categories combine purpose, cost behaviour and ownership to support budgeting and management. They are not accounting classifications. Individual build, platform and control costs may still be fixed, step-fixed or variable with usage. Some investments may be eligible for capitalization.

One risk is that costs assumed to be fixed can unexpectedly become variable. This can happen if agents drive usage above a contractual usage threshold and into metered pricing without appropriate controls.

Where the Value Goes Missing

AI business cases often project productivity gains that don’t easily translate into benefits. Employees report saving a few hours a week, which translates, in theory, to a significant benefit. But time saved is only crystallised when the business converts it into lower costs or higher revenue. Otherwise, it may still create value through faster service, better quality or lower risk. Time saved in minutes across hundreds of people must be aggregated before it hits the bottom line.

This is the same value leakage that I explored in an earlier article, which illustrated how the share of an AI investment’s potential value that reaches the P&L, its Deployment Yield, can be strikingly low.

The difficulty of turning time saved into money saved is not new. But the nature of AI exacerbates the problem. Past IT system deployments often landed on one team or function, where the time savings could be pooled and released as headcount savings. AI hands a few minutes back to almost everyone, so the time saved is more dispersed. And some of the saved time is spent checking and correcting AI’s output itself. Workday’s January 2026 survey of 1,600 leaders and 1,600 employees found that nearly 40 per cent of AI’s reported value was lost to rework and misalignment. Usage costs also rise as adoption expands, which further eats into net savings.

Benefits are even harder to project confidently than costs. In a 2026 survey of more than 900 companies, Bain found that 20 per cent of firms had realised savings of 10 per cent or less, yet 90 per cent of firms were raising their AI budgets again.

There is also a timing trap. In the near term, AI may increase the total cost base. For a while, companies will pay for the AI, its implementation, and running old and new workflows in parallel. Then-Walmart CEO Doug McMillon said AI would change “literally every job.” Separately, chief people officer Donna Morris said the company expected its global headcount to remain broadly flat for three years.

Fund Uncertainty in Stages

Addressing these issues requires a novel approach to AI budgeting, based on an annual funding envelope, a rolling forecast and a mechanism for releasing funds as evidence emerges. Funding should follow deployment in stages and be adjusted as costs, usage and value become clearer, rather than being fixed in advance. The system has five moves borrowed from other funding models but adapted to AI’s particular dynamics:

• Fund at the edge, control at the centre. Consumption-versus-value trade-offs are best made by those close to the use case, so the associated costs should be owned and managed by operating teams. Agents can run up a huge bill unexpectedly, so devolved consumption relies on the centre to apply controls such as usage limits.

• Force a rethink at each stage gate. As with other projects, AI funding should be released in stages. But each gate should reconsider not only delivery progress, but whether the use case, business case and technical solution still make sense. A newly released AI model may have removed the need to build a bespoke solution, for example.

• Hold a reserve, not only a contingency. Keep an explicit central reserve for opportunities that become viable during the year as models improve or unit costs fall. Contingency covers unexpected costs in projects already approved. The reserve funds new projects that become viable.

• Reforecast based on underlying drivers. A normal quarterly reforecast updates volumes and prices within a stable model of the business. With AI, the unit economics can shift quickly. The price per unit may fall while consumption per task rises. So, the forecast needs to be repriced based on a deeper review of the underlying cost drivers.

• Seek proof of value, not promises of it. All projects start with promises of benefits and timing, and AI is no exception. AI investments face additional risks such as dispersed savings and unexpected usage costs so the burden of proof should be higher. Proof of value should include measurable benefits, net of the costs to deploy, operate and verify.

In this way, the budget becomes a control system as well as a funding plan. Fund shared platforms and controls centrally, rethink at each stage, hold a reserve, reforecast based on drivers, and release funding only as evidence of value improves.

Whatever goes into the AI budget, it’s unlikely to please everyone. But with the right budgeting system in place, everyone should understand where the money goes, who’s spending it and what you’re getting for it.

Monday Morning Actions for Executives

· Build a spend snapshot. Ask finance and IT for a full picture of current spend.

· Put guardrails on variable spend. Set alerts and monthly limits.

· Challenge live pilots. Ask for proof of value or a timeline for seeing it.

Questions the Board Should Ask

· Can management show us the full cost of AI, including hidden costs?

· How much of the spend is committed, and how much could rise with usage?

· Which use cases have generated realised financial benefits, and why?

· Are we funding the full deployment chain, or only models and tokens?

Sources & Notes

· Bain & Company, 2026 CFO survey: In a survey of 102 CFOs, 83 per cent expected AI spending to rise by more than 15 per cent over two years, 42 per cent by at least 30 per cent, and only 31 per cent were highly satisfied with current outcomes.

· The Wall Street Journal, 5 June 2026: D.A. Davidson’s Gil Luria warned that many CFOs would “freak out” when usage-based AI bills arrived, while KPMG found that only 26 per cent of large US organisations had comprehensive visibility into AI costs.

· Stanford HAI, 2025 AI Index Report: The cost of querying a system at roughly GPT-3.5 performance fell more than 280-fold between November 2022 and October 2024.

· WIRED, 26 May 2026: Claude Code creator Boris Cherny said he regularly runs dozens, and sometimes hundreds, of agents for eight to twelve hours at a time.

· Vaclav Smil, via IEEE Spectrum: In real terms, US residential electricity prices fell more than 97 per cent between 1902 and 1950 while consumption rose.

· BCG, AI Radar 2026: In a survey of 2,360 executives, respondents expected AI spending to rise from 0.8 per cent of revenue in 2025 to about 1.7 per cent in 2026, with significant differences by sector.

· Ramp, The $1 Trillion AI Spend Blind Spot, 2026: Among businesses on Ramp that spend on AI, the median company allocates nearly 15 per cent of its software budget to AI, based on anonymised transaction data across Ramp’s customer base.

· Business Insider, May 2025: BCG X leader Sylvain Duranton recommended allocating roughly 10 per cent of AI-transformation effort to algorithms, 20 per cent to data and technology, and 70 per cent to changing how people work.

· EY, Agentic AI Enterprise Token Cost, 2026: EY illustrates how a four-cent customer-service interaction can become a $1.20 agentic orchestration when tools, reasoning and repeated attempts are added.

· The Financial Times and The Verge, May 2026: Uber reportedly exhausted its 2026 AI coding-tools budget by April, then capped each employee’s monthly token spending at $1,500 per tool as executives questioned the value of rising usage.

· Deployment Yield: The share of an AI investment’s potential value that reaches the P&L, developed in an earlier article in this series.

· Business Insider, 14 January 2026: A Workday-commissioned survey of 1,600 leaders and 1,600 employees found that nearly 40 per cent of AI’s reported value was lost to rework and misalignment.

· Bain & Company, 2026 survey of more than 900 companies: Two in five reported savings of 10 per cent or less, while about nine in ten planned to increase AI budgets.

· The Wall Street Journal, 27 September 2025: Then-CEO Doug McMillon said AI would change every job, while chief people officer Donna Morris said Walmart expected headcount to remain broadly flat for three years as roles changed.

The Org Chart You Didn’t Design – Conway’s Law has gone into reverse, as AI agents reshape organisations.

Executive Summary

• Employees are adopting AI quickly, with or without permission, and building their own tools, workflows and software.

• Each tool works for the person who built it. But they draw on different data and use different business rules, so they aren’t consistent or compatible.

• The result is a new kind of legacy system that is hard to monitor, hard to connect with and hard to switch off.

• The tools people build for themselves often fit the work better than those imposed from the centre. The preferred approach is to enable experimentation inside common guardrails and promote high-value tools into company-wide assets.

From the line’s opening in 1844 until 1854, passengers travelling between Bristol and Birmingham had to change trains at Gloucester. The Bristol and Gloucester Railway used Brunel’s 7 ft ¼ in broad gauge but the Birmingham and Gloucester used the 4 ft 8½ in gauge that became the national standard. Where the two met, passengers, luggage and freight had to be transferred between trains. Both railways worked, but they didn’t work together.

Companies are now creating similar compatibility problems for themselves. Employees are enthusiastically embracing AI to help them improve their personal productivity. The latest low-code agent builders and AI coding tools let non-technical employees create entire workflows and lightweight software. They can, for example, stand up agents or AI co-workers to triage emails, draft progress reports or reconcile spreadsheets. This ‘toolmaker’ activity can bring significant benefits. Who wouldn’t want their employees taking greater ownership and innovating solutions to front-line problems? The trouble comes when thousands of employees take up toolmaking, and every tool is different. Anyone can now lay a new railway, with its own gauge.

Workers are acutely aware of AI’s ability to improve how they work, with many paying for their own tools to circumvent corporate restrictions. Microsoft found that three in four knowledge workers use AI at work, and more than three quarters of them bring their own tools rather than wait to be given one. A University of Melbourne survey of more than 48,000 people found that almost half had used AI in ways that broke their employer’s rules. I’ve covered this in depth in AI in the Shadows

The tools are powerful. Across three large field trials, software developers using an AI assistant completed about 26 per cent more tasks, with larger gains among less-experienced developers. In a field experiment at Procter & Gamble, individuals using AI matched the average performance of two-person teams working without it. The tools work, so workers will keep building and using them.

The concern here is not occasional prompting, but reusable workflows that connect company data, encode business rules, or take actions. Evidence of this shift is beginning to appear. In an OpenAI case study, BBVA said its employees had created more than 20,000 custom GPTs, around 4,000 of which were used frequently.

Different Gauges

The catch is that each tool is built to meet the needs of one person. One analyst pulls data from a spreadsheet, another from a live system, a third from last month’s report. One person outsources output checking to an AI, while another checks every line by hand. One person builds in a control step, and one omits it. On its own, each choice is reasonable. Together, they mean that two people asked the same question can give two different answers, drawn from two different sources, checked in two different ways. The result is fragmented and diverging workflows that aren’t consistent and aren’t composable. A company is a set of synchronised workflows, not a pile of individuals doing separate tasks. When the joins between tasks are unreliable, so is the company’s output. Put another way, the company is rapidly accumulating a new kind of legacy system.

Fragmentation Creates Four Risks

When new processes and their supporting code and data are built ‘bottom up’, the resulting workflows carry four risks:

Opaque. It becomes hard to see how work is actually getting done. Pre-AI, standard operating procedures (SOPs), together with processes and controls embedded in IT systems, provided some assurance that work was done in line with strategic and operational goals and with appropriate controls. Now, the outputs may look fine, but it becomes impossible to understand the process that led to the output. That brings several risks. In June 2026, KPMG withdrew a published report after GPTZero found that only five of its 45 citations accurately matched real, uncorrupted sources. UBS, the NHS, Swiss Federal Railways and Transport for London said claims about them were untrue or misleading. If errors of this kind can pass a Big Four publication’s review, a manager glancing at a slide is unlikely to detect them. Reviewing the output is not the same as checking the work that produced it.

Inconsistent. As Toyota demonstrated, quality is a function of process. A 2026 preprint involving 128 knowledge workers at one multinational industrial company found that GenAI increased efficiency across three tested task types. Quality improved for knowledge packaging and knowledge creation, but declined for knowledge acquisition. The implication is not that AI invariably reduces quality, but that its effect depends on the task and on the process surrounding it. As employees customise those processes, operational variation may increase even if average model performance improves.

Unintegrated. Like the early railways, solutions built in isolation don’t join neatly with one another. Everyone may be using the same AI model while the data, definitions and controls underneath are inconsistent. This becomes problematic when attempting to build end-to-end workflows that cross multiple functions. For example, when a customer is onboarded, data needs to flow cleanly between sales, compliance and finance.

Brittle. User-built tools can depend on models and connectors outside the company’s control. Even where vendors publish deprecation schedules, unexpected shifts still occur. During OpenAI’s August 2025 GPT-5 rollout, older models were removed from ChatGPT without warning, disrupting model-specific workflows before some access was restored. In June 2026, Anthropic suspended Fable 5 globally for three weeks following a US government directive. A workflow can fail unexpectedly even when its own code has not changed.

Together, these are warning signs of an emerging legacy estate. A system becomes legacy when business dependence outlives the organisation’s ability to change, support or replace it. Ward Cunningham introduced the ‘technical debt’ metaphor in 1992 to describe how a quick technical shortcut creates a future liability. The United States federal government now spends more than $100 billion a year on IT and cyber investments, and agencies report that roughly 80 per cent goes to operating and maintaining existing systems.

None of this is an argument for stopping workers using AI to make their own tools. We are in the foothills of AI’s evolution, and clamping down now would kill experimentation and valuable learning. The toolmaking itself is not the problem. The problem is leaving it private and unconnected until one day it becomes critical infrastructure without anyone realising. So, the task is to capture the best tools and turn them into capabilities the whole company can rely on. Six moves can deliver that.

The Playbook

The controls should be proportional to risk. Personal experiments inside a protected sandbox can remain light touch. Any workflow that accesses sensitive data, takes an external action, feeds a material decision or serves multiple users should be given a higher risk classification and proper controls.

1. Lay the rails. Decide which standards are not up for negotiation and enforce them. Britain began standardising rail gauges by law in 1846. This prevented the problem from growing, but the installed base still took decades to convert. Common data definitions, approved sources and connectors, mandatory controls, and security protocols provide the rails for workflows.

2. Arm the workforce. The best way to stop people using risky, unapproved AI is to give them the tools they want. Samsung learned this the hard way in 2023, when engineers pasted sensitive source code into a public chatbot. BBVA went the other way and gave its people an enterprise tool; more than 100,000 employees were using it by June 2026.

3. Instrument and observe. Maintain a lightweight registry of reusable workflows. Record the business outcomes (e.g., efficiency uplift) but also drivers of risk such as permissions, dependencies and number of users. This reveals where both value and risk are accumulating.

4. Harvest and harden. When a tool demonstrates repeatable value and manageable risk, adopt it properly. Assign a product owner, validate its outputs, document its data and controls, test failure modes, provide a fallback and then release it more broadly. That captures the know-how and reduces dependence on the original builder.

5. Make it a team sport. Left alone, useful work often remains local and tacit. The alternative is to encourage shared innovation. One way to do that is to establish a guild where builders compare notes and raise the standard together. It also helps to recognise and reward the people whose tools get adopted broadly.

6. Retire and consolidate. Harvesting the best tools is only half the job. Tools that do not meet the bar for wider deployment need to be culled. Give tools a review or renewal date and retire those that are unused, unsupported, or unsuitable.

It took until 1892 to eliminate the Great Western Railway’s remaining broad-gauge network. Over one weekend that May, about 4,200 workers converted 171 miles of track to standard gauge. Brunel’s broad gauge offered smoother, faster journeys, but had not been widely adopted. In the agentic age, companies must harness what their people are building and turn it into something the whole company can use. The alternative is to allow a risky new type of legacy system to take hold.

Monday Morning Actions for Executives

• Initiate an inventory. Launch a quick review of user-built tools.

• Meet the demand. Provide workers with sanctioned AI tools to avoid Shadow AI.

• Put someone in charge of the rails. Name an owner for standards and controls.

• Pick a tool to harden. Select one high-value workflow and move it into production.

• Set the tone. Communicate principles on worker-made tools. Reward the best makers.

Questions for the Board

• If a regulator or an auditor asked how a key number or decision was produced, could we show them?

• Which critical workflows depend on a particular model, vendor, connector or version, and what is the tested fallback?

• Are we getting the right balance between rewarding individual innovation and building capability for the whole organisation?

  ---

  Thanks for reading. Subscribe here for free.

Sources & Notes

• Railway gauges and conversion: Network Rail, “Why the end of the gauge war didn’t standardise Britain’s railway” (20 May 2021); UK Parliament, “Railways—The Broad and Narrow Gauge” (Hansard, 16 June 1846); Friends of the National Railway Museum, “The end of the Great Western Railway’s broad gauge”.

• Microsoft, “AI at Work Is Here. Now Comes the Hard Part” (2024 Work Trend Index Annual Report, 8 May 2024).

• Nicole Gillespie et al., “Trust, Attitudes and Use of Artificial Intelligence: A Global Study 2025” (University of Melbourne and KPMG, 2025).

• Kevin Z. Cui et al., “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers” (working paper, 2025).

• Fabrizio Dell’Acqua et al., “The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise” (NBER Working Paper 33641, 2025).

• GPTZero, “Chasing the Hallucinations: KPMG’s AI-Powered Attempt at ‘Redefining Excellence’” (June 2026); Financial Times, “KPMG report contained AI hallucinations on benefits of AI” (12 June 2026).

• Toyota Motor Corporation, “Toyota Production System”; Sven Bottesch et al., “Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity—Evidence from the Field” (arXiv preprint, 28 July 2026).

• OpenAI, “Introducing GPT-5” (7 August 2025); Ars Technica, “OpenAI brings back GPT-4o after user revolt” (13 August 2025); OpenAI, “API deprecations”.

• Anthropic, “Model deprecations”; “Statement on the US government directive to suspend access to Fable 5 and Mythos 5” (12 June 2026); “Redeploying Fable 5” (30 June 2026).

• Ward Cunningham, “The WyCash Portfolio Management System” (OOPSLA ’92 Experience Report, 1992).

• U.S. Government Accountability Office, “Information Technology: Agencies Need to Plan for Modernizing Critical Decades-Old Legacy Systems” (GAO-25-107795, 17 July 2025).

• TechCrunch, “Samsung bans use of generative AI tools like ChatGPT after April internal data leak” (2 May 2023).

• OpenAI, “How BBVA is scaling AI from pilot to practice across the org” (6 November 2025); “BBVA puts AI at the core of banking with OpenAI” (11 June 2026). Company- and vendor-reported evidence.

• AI in the Shadows – Your staff will find ways to access the most powerful AI tools, even if you ban them

Executive Summary

• Sensitive information, such as corporate strategy, can be deduced by outsiders prepared to collate and analyse fragments of publicly available information, such as job adverts.

• AI has made it far quicker and cheaper to collate and analyse data in this way.

• Conventional information-security controls do not fully address this risk.

• Scraping public data from the web is generally not treated as illegal.

• Firms can reduce exposure by red teaming their own data exhaust and risk-rating their outbound communications.

On the night of 1 August 1990, the CIA ordered a record twenty-one pizzas. Local Domino’s franchisee Frank Meeks told the Los Angeles Times a few months later: “The news media doesn’t always know when something big is going to happen because they’re in bed, but our deliverers are out there at 2 in the morning.” Iraq invaded Kuwait the following day.

Intelligence analysts call this mosaic theory: assembling individually innocuous facts into a material inference.

Historically, this has been a laborious and expensive task, and this has limited the risks to those being watched. Now, however, AI has industrialised mosaic analysis. Much of what a company emits, including job adverts, planning applications and executives’ Strava routes, can now be scraped and analysed.

The Mosaic in Action

A lot can be deduced from public data. Researchers at Oxford and armasuisse sampled corporate aircraft routes and M&A activity associated with 36 listed companies. In all seven acquisitions the study identified, the buyer’s aircraft had visited the target beforehand, on average 61 days earlier.

In another example, researchers found that investors could use satellite observations of retail car parks to predict corporate earnings, and formulate profitable trading strategies.

Routine releases can have outsized consequences. In late July 2017, developers mined prematurely released HomePod firmware, revealing the design of Apple’s next iPhone about six weeks before it was announced. And in 2018, a 20-year-old student showed that Strava’s aggregated exercise map drew the outlines of secret military bases. Six years later, Le Monde used public Strava profiles to trace 26 US Secret Service agents, deducing the hotel where President Biden was staying before his meeting with Xi Jinping.

The list of sources is long, including heat signatures captured through thermal satellite imagery, behavioural data such as web traffic, human writing such as compensation posts on Levels.fyi, and public records such as court filings.

The Industrialised Mosaic

Mosaic theory is not new. Satellite car-counting dates to about 2010, for example. What has changed is that much of the collection and analysis can now be automated at scale and low cost. A mid-cap engineering firm now faces hedge-fund-grade scraping and analysis.

Researchers at ETH Zurich reported in 2024 that, on a benchmark of 520 public Reddit profiles, an LLM achieved up to 85 per cent accuracy when inferring attributes such as location, income and sex. In that case, the models operated at about one-hundredth of the cost of human labellers.

The quality of mosaic signals is also improving. A single source such as pizza orders can be misleading. But with AI, snoopers can collate and analyse richer data from more sources, finding patterns which a human could not. As more of life moves online, both the number of signals and their granularity grow.

Use of such data is becoming commonplace. In Lowenstein Sandler’s annual survey of investment advisers at private fund managers, the proportion reporting use of alternative data (an input to mosaic analysis) rose from 62 per cent in 2023 to 90 per cent in 2025.

Conventional information-security techniques do not help here. The source material is public, commercially available or generated through routine activity, so controls designed to stop theft and leakage aren’t effective.

Reacting to the Mosaic

Executives are already reacting to this newly transparent environment. A 2023 Review of Financial Studies paper found that firms that expected their corporate disclosures to be read by AI altered their language and used more positive and excited vocal tones on earnings calls.

Companies are also addressing the corporate jet informational vapour trail. LVMH sold its corporate jet and shifted to charter planes after Bernard Arnault said tracking could reveal his movements. Nike placed two jets in the FAA’s flight-data filtering programme, making them harder to follow.

Another technique is to anonymise public signals. Apple has used foreign trademark filings, made through new subsidiaries, to make new product names harder to unearth. Meta used shell companies to disguise the nature of a Wisconsin data-centre project.

Information providers are also constraining large-scale aggregation. Strava restricts access to street-level detail and suppresses low-volume routes. LinkedIn uses technical systems to throttle or block automated scraping of public profiles.

Approaches to protecting secrets need to be holistic. For example, protecting the movements of a single executive now means considering social media, calendars, fitness apps, conference listings and flight trackers together, not separately.

Firms can also turn the tables on competitors. US courts make a distinction between scraping public web pages and hacking into restricted systems. Scraping public pages is generally not treated as illegal computer access, although contracts and other laws may still restrict collection or use.

The Mosaic Playbook

• Red-team your own exhaust. Have an LLM parse your job adverts, regulatory filings, public code, patents and other exhaust. An industry has been built around this capability so it can also be outsourced. Make sure to repeat periodically as patterns change.

• Clarify accountability. One of the reasons that mosaic analysis works is an asymmetry of approach. A snooper consolidates the available data about a company across all channels, whereas within the company accountability is diffuse, spanning communications, HR, investor relations, legal, security and more. The obvious choice to own the risk is the Chief Information Security Officer (CISO), but many are overloaded with AI-driven cyber threats.

• Treat routine publications as disclosures. Build a map that helps you understand which publications are connected to which secrets. Then score outbound content for risk before publication.

• Minimise sensitive signals. Around undisclosed initiatives such as M&A or business building, coordinate hiring, procurement, planning and travel so that little can be inferred from the sequence. Abstract the truth (Amazon code-named its twenty shortlisted headquarters cities after employees’ dogs). Protect the digital footprints of executives.

• Turn the tables. Use these techniques on competitors if for no other reason than to benchmark your own exposure against theirs. This is becoming a normal corporate function.

Monday Morning Actions for Executives

• Understand your exposure. Ask for a read of your own mosaic footprint.

• Vet outbound comms. Review job adverts, filings and announcements.

• Opt-out where possible. For example, ask data brokers to remove senior leaders’ details.

In June 2025, the Pentagon Pizza Report social media account reported a surge at pizzerias near the Pentagon before Israel’s strikes on Iran. The Pentagon disputed its significance, but the episode showed how easily public signals can now be collected and amplified. In the thirty-five years since the original event, mosaic theory has been transformed from an art form into an industry.

Questions for the Board

• Do we know what our public exhaust is revealing about us?

• Do we read our competitors’ exhaust as systematically as they read ours?

• Do we treat routine publications (e.g., job adverts) as disclosures?

Sources

• Slice of Life: Pizza Orders Soar in D.C.

• The Real First Class? Inferring Confidential Corporate Mergers and Government Relations from Air Traffic Communication

• On the Capital Market Consequences of Big Data: Evidence from Outer Space

• HomePod’s Firmware and the Next iPhone

• A global heat map for joggers is exposing sensitive US military information

• Biden and Trump Put in Danger by Secret Service Agents: Watch the Second Episode of StravaLeaks

• Beyond Memorization: Violating Privacy via Inference with Large Language Models

• AI’s Integration Into Alternative Data Fuels New Opportunities and Challenges

• How to Talk When a Machine Is Listening: Corporate Disclosure in the Age of AI

• World’s Second-Richest Man Bernard Arnault Sells Private Jet So Climate Activists Can’t Track Him

• We Reported on Nike’s Extensive Use of Private Jets. The Company Just Made It Harder to Track Them.

• Claiming Priority to Submarine Trademark Applications—A Curious Little Loophole

• At Least Four Wisconsin Communities Signed Secrecy Deals for Billion-Dollar Data Centers

• After Exposing Secret Military Bases, Strava Restricts Data Visibility

• hiQ Labs, Inc. v. LinkedIn Corp.

• Amazon Used Code Names of Employees’ Dogs to Keep HQ2 Finalists Secret

• Did Busy Pizza Shops Really Predict US Airstrikes on Iran?

Should You Add a Bot to Your Board?

Executive Summary

• Courts can treat use of a technology as part of ‘reasonable care’ (the legal standard for the precautions a careful and competent firm would take).

• The expected standard of reasonable care moves as technology matures and the benefits become measurable.

• Firms can face significant penalties when they fail to implement the technology that is required to deliver reasonable care.

• Recent legal developments show how a claim for AI non-use could arise. Expectations are likely to develop task by task.

• Firms should document where AI is required, optional or prohibited, and the reasons for each decision.

On 10 March 1928, two tugs were towing coal barges along the New Jersey coast when a gale hit and the barges sank. Storm warnings had been broadcast on the radio, but neither tug had a working receiver. Radios were cheap and available but not yet commonly used by tugs.

When the case came to court, the judge, Learned Hand, rejected the owners’ defence that radio receivers were not yet general practice in the trade. He wrote that “a whole calling may have unduly lagged in the adoption of new and available devices.” The decision showed that a court may rule that use of a technology is a requirement even if the industry has not generally adopted it.

For the past few years, boards have asked whether it is safe to use AI. Soon they may also need to ask whether it is safe, and legally defensible, not to use it.

Expected Standards Move as Technology Matures

Firms are legally required to take ‘reasonable care’ in their duties towards customers, employees or others affected by their conduct. For example, they must provide protections such as hard hats to keep employees safe. The standard of reasonable care is specific to context, and changes over time, sometimes in response to new technology.

Assessing shipyard hearing-loss cases in 1983, Mr Justice Mustill found that the whole industry was taking “the same line of inaction” on ear protection. He concluded that employers failing to provide hearing protection after 1963 were negligent.

A technology breakthrough can change the expected standard almost immediately. Before 1985, blood banks had no licensed test to screen donations for evidence of HIV infection. In March that year, the FDA licensed the first antibody test and US public-health authorities recommended that all donated blood and plasma be tested. Screening then became routine and changed what safe practice required.

Falling Behind Can Create Serious Liability and Penalties

Firms failing to meet the expected standard of reasonable care can face negligence claims. Regulators can also penalise organisations for failing to implement or operate established controls. Britain’s data regulator fined Tuckers, a criminal defence firm, £98,000 after a cyber breach linked in part to missing multi-factor authentication (MFA). It later fined Advanced, an NHS software supplier, £3.07 million after gaps in its deployment of MFA contributed to a ransomware incident. The Information Commissioner, John Edwards, said there was “no excuse for leaving any part of your system vulnerable.”

NatWest offers another example. In 2021 it was fined £264.8 million after an FCA prosecution for money-laundering breaches, as the bank’s automated monitoring system wrongly treated some cash deposits as cheque deposits.

These cases do not prove that firms must use AI, but that liability or penalties can follow when an established control is missing, incomplete or poorly supervised.

The Legal Route to AI Non-Use Liability Is Emerging

Recent developments show how an AI non-use claim might be framed. In July 2026, the UK Jurisdiction Taskforce published a legal statement on liability for AI harms under English law. The Law Society said it clarifies when professionals may be liable for using, or failing to use, AI.

The statement is not a judgment and does not create a general duty to use AI. But it does conclude that existing professional-negligence principles may apply where a reasonably competent professional would have used a particular AI tool for a particular task.

AI Expectations Will Emerge Task by Task

AI use is unlikely to become an expectation across a whole profession or sector at once. Rather, expectations will develop around specific tasks where evidence, availability and adoption come together.

For example, NHS England says AI decision support is now used across every regularly admitting stroke service in England. At primary stroke centres, AI use was associated with a 64-minute reduction in the time taken to assess and transfer patients (other factors also drove the improvement). National deployment does not by itself establish a legal duty, but it does make non-use easier to challenge. A hospital can now be asked whether the AI system was used, and for the reasoning behind the decision.

Clinical guidance is also starting to recognise specific AI uses. The American Diabetes Association’s 2026 Standards of Care say that FDA-approved AI algorithms are an appropriate strategy for improving access to diabetic-retinopathy screening.

Professional bodies are beginning to treat AI knowledge as part of competence. The Bar Standards Board’s guidance lists basic AI awareness, and evaluation of the risks, benefits and costs of new technologies, as good practice.

How to Tell When Non-Use Is Becoming Risky

Non-use of AI becomes more difficult to defend when several signals appear together:

• A recognised benefit in accuracy, speed, safety or cost has been demonstrated.

• The tool is widely available at a proportionate cost.

• Comparable organisations are adopting it for the same task.

• Regulators or professional bodies recognise the use case.

• There is a reliable process for human review, monitoring and accountability.

• Failure to use the tool could have a material consequence.

Actions for Executives

To date, I have not found any example of a prosecution or penalty due to non-use of AI, so immediate risks are low. Nevertheless, firms should establish a framework to ensure that future risks are understood and managed:

1. Set clear rules for material tasks. Identify tasks where AI could have a meaningful impact on accuracy, safety, consumer protection or regulatory compliance. For each such task, state where AI is required, prohibited or optional.

2. Build an audit trail. Where AI non-use could be challenged, record the reason for not using it with supporting data. Keep evidence of competence and oversight, including training records and evaluation results.

3. Update your position periodically. Review regulatory and professional-body guidance. Ask customers, insurers and comparable firms what they regard as normal practice.

4. Ask your insurers. Premium discounts and underwriting expectations are early signals of where non-use is becoming hard to defend. RLI already prices AI-based safety monitoring into cover.

The barge owners followed common industry practice, but were found liable anyway, as the whole trade had fallen behind the standard the court expected.

AI adoption is moving so quickly that it is hard to determine where its use might become part of reasonable care. Firms that establish a clear framework for managing decisions to use or not to use AI will be better placed when the standard moves.

Board Questions

• Which of our tasks are approaching the point where AI non-use would need to be justified?

• What evidence would cause us to change our current position?

• Could we explain our decisions after an adverse event?

Sources & Notes

1. The T.J. Hooper. 60 F.2d 737 (2d Cir. 1932): Learned Hand held that industry custom is evidence of reasonable care, not its conclusive measure. Judgment.

2. Professional custom. Thompson v Smiths Shiprepairers (North Shields) Ltd [1984] QB 405 treated shipyard employers following “the same line of inaction” on ear protection as negligent from 1963. Case summary.

3. HIV screening. The US Public Health Service recommended screening all donated blood and plasma in January 1985, the FDA licensed the first ELISA test kits in early March, and blood banks were screening routinely within weeks. MMWR report.

4. Missing controls. The ICO fined Tuckers £98,000 in March 2022 and Advanced £3.07 million in March 2025 over missing or incomplete security controls, and the FCA’s prosecution of NatWest ended in a £264,772,619.95 fine in 2021. Tuckers penalty notice. Advanced announcement. FCA announcement.

5. UKJT legal statement. The UK Jurisdiction Taskforce’s July 2026 statement, expert analysis rather than binding precedent, concludes that professional-negligence principles may apply where a reasonably competent professional would have used a particular AI tool. UKJT statement. Law Society summary.

6. NHS stroke pathway. NHS England’s December 2025 evaluation, observational rather than causal, associated AI-supported scan review with a 64-minute reduction in assessment and transfer time at primary stroke centres. NHS England account.

7. Clinical pathways. The American Diabetes Association’s 2026 Standards of Care recognise FDA-approved AI algorithms as an appropriate strategy for diabetic-retinopathy screening, an option rather than a mandate. ADA Standards.

8. Professional competence. The Bar Standards Board’s May 2026 guidance lists basic AI awareness and evaluation of the risks, benefits and costs of new technologies as good practice. BSB guidance.

9. Insurance signals. RLI’s November 2025 premium discounts for AI-based safety monitoring and the Lloyd’s Market Association’s underwriter survey are market signals, not evidence of a legal duty. RLI programme. LMA survey.

10. Current misuse cases. Mata v Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023), sanctioned lawyers over fabricated AI-generated authorities, and reported litigation to date concerns misuse rather than non-use. Docket.

• The Emperor’s New Code. When your workforce pushes back on AI, listen carefully.

Executive Summary

• Repeated AI prompting can resemble gambling’s variable-ratio reward loop, creating an addictive pull.

• “Almost right” outputs can feel like near-misses, which are frustrating enough to disappoint, but close enough to success to invite another spin.

• Getting caught in the ‘Machine Zone’ may contribute to exhaustion, reduced efficacy and cynicism, which are the dimensions of burnout.

• Leaders can borrow gambling safeguards: set limits in advance, display session metrics, make walking away respectable and reward outcomes instead of activity.

The anthropologist Natasha Dow Schüll spent fifteen years in Las Vegas studying machine gamblers. One video-poker player, whom she calls Mollie, neatly captured a key insight: “The thing people never understand is that I’m not playing to win.” Then why play? “To keep playing. To stay in that machine zone where nothing else matters.”

I’ve recently met engineers who say AI coding is as addictive as poker. That is anecdotal, but there are reasons to take it seriously. Behavioural research has long shown that rewards delivered after an unpredictable number of responses can lead to persistent behaviour. Repeated AI prompting can resemble this variable-ratio pattern, which is the underlying driver of extended slot-machine play.

Steve Yegge, a veteran of Amazon and Google, wrote in February 2026 that AI coding is “genuinely addictive”, and that “the AIs can be like sirens, and can woo you into staying at your computer longer than you should”. Software engineers are the clearest case, but the pattern can apply to anyone who prompts AI as part of their work, including a marketer chasing the right campaign visual or an analyst drafting a board summary.

The Machine

Many AI systems are configured to produce variable answers, so that the same prompt can yield many different results. The prompt window can function psychologically like a gambling machine’s re-spin control in that each additional attempt is immediate and effortless, but the outcome is uncertain.

AI tools are improving rapidly. Nevertheless, near-miss outputs remain a major reported frustration. In Stack Overflow’s 2025 survey, 66 per cent of respondents who answered the AI-frustrations question selected “AI solutions that are almost right, but not quite”.

In a small laboratory fMRI study, gambling near-misses were rated as less pleasant than full misses yet increased participants’ reported desire to continue. Whether AI near-misses have a comparable effect has not yet been established.

Prompting involves real skill, but the model still contributes an element of uncertainty. That combination may encourage people to overestimate their ability to rescue a deteriorating interaction. As Carson Farmer, Recall’s chief technology officer, put it: “I’ve spent all this time prompting, surely I can prompt myself out of this hole.”

The Player

In a casino, these mechanics can result in lengthy and ultimately loss-making sessions. In the workplace, they may generate extended prompting sessions with rising stress and diminishing returns. A June 2026 report surveyed 6,000 full-time digital workers in the US, UK and Australia. Around 60 per cent of AI users reported rerunning the same prompt through multiple tools because the first output was not good enough.

To colleagues observing an engineer absorbed at a screen, productive flow and an unproductive prompting loop can look similar. Productive flow tends to involve clear goals, a match between challenge and skill, and feedback that helps the person progress. An unproductive Machine Zone state may instead have a drifting goal, inconsistent feedback and no natural place to stop.

The House

Casinos encourage Machine Zone states because those states extend time on device and, in turn, increase average player losses. Companies are not trying to maximise time on AI tools. However, given the evident productivity and velocity gains to be captured, they are, quite reasonably, making AI use a baseline expectation.

Microsoft has told managers that AI use is “no longer optional”. Coinbase chief executive Brian Armstrong said he required engineers to onboard to approved coding assistants within a week and later dismissed several who lacked a good reason for not doing so. Bloomberg reported in October 2025 that KPMG was tracking use of tools including Copilot and would assess employees against the firm’s AI objectives in 2026 reviews.

The Chips at Stake

The long-term impacts of extended AI use on colleague productivity and motivation are not yet clear. But the analogy to gambling suggests three plausible pathways. Open-ended sessions may contribute to exhaustion; repeated low-yield effort may erode professional efficacy; and a loss of control may foster cynicism. These correspond loosely to the standard dimensions of burnout.

In a study of 1,488 full-time US workers, 14 per cent of workers reported “AI brain fry”, which the authors defined as mental fatigue associated with excessive AI use or oversight, with marketing the worst-affected function. These workers were 39 per cent more likely to show an active intent to leave the organisation.

The Safeguards

For many, gambling is harmless and fun, just as AI-assisted work can be highly productive and rewarding. Both can benefit from safeguards against unproductive states. Companies can adapt selected harm-reduction principles from regulated online gambling to AI-assisted work:

• Set table limits. UK rules require online gambling businesses to prompt customers to set a financial limit before their first deposit and to make limits easy to review. The workplace equivalent is an agreed definition of “good enough” and guidelines on healthy AI use.

• Break the trance. Most UK-licensed online casinos must display net position and elapsed session time. Companies can support AI users by providing session clocks and attempt counts. These need to be private to the user so that a self-regulation aid does not become another productivity target.

• Make folding respectable. Completing a task without AI should remain a legitimate professional choice. Reward outcomes rather than usage: quality, cycle time, rework rates, risk and learning. Microsoft’s own 2026 research finds the best performers deliberately set AI aside at times to keep their skills sharp.

• Train managers to recognise diminishing returns. Give managers a simple set of questions: Is the goal still clear? Is each additional attempt improving the result? Is rework increasing? Is the task spilling into personal time? Does the employee have permission to stop using the tool or finish the work another way?

• Pay out in outcomes. Reserve some productivity gains for learning and recovery instead of automatically raising targets to absorb them.

Schüll learned that gambling machines were built to maximise time on device. Somewhere in your organisation tonight, an engineer is in the third hour of a twenty-minute task, certain the next prompt will finish the job. Leaders can learn from casino regulators how to help colleagues remain productive and avoid the Machine Zone.

Questions for the Board

• Are we rewarding useful outcomes or visible AI activity?

• Could AI-enabled work create a material psychosocial risk?

• Who owns this risk, and how will we know the safeguards work?

Sources & Notes

• Natasha Dow Schüll, Addiction by Design: Machine Gambling in Las Vegas, Princeton University Press, 2012. Las Vegas-centred qualitative ethnography based on fifteen years of fieldwork; source of the Mollie exchange, the “machine zone” and the industry term “time on device”.

• C. B. Ferster and B. F. Skinner, Schedules of Reinforcement, 1957. Foundational experimental work on reinforcement schedules, largely with animals.

• M. Karen Shen and Dongwook Yoon, “The Dark Addiction Patterns of Current AI Chatbot Interfaces”, CHI Extended Abstracts, 25 April 2025. Seven-page interface analysis of eight chatbots, arguing that non-deterministic responses create reward uncertainty comparable to slot-machine play; it did not test users or measure dopamine.

• Stack Overflow, 2025 Developer Survey, released 29 July 2025. More than 49,000 people participated overall; 31,476 answered the multiple-selection AI-frustrations question, of whom 66 per cent selected “AI solutions that are almost right, but not quite”. Self-selected developer audience.

• Luke Clark, Andrew J. Lawrence, Frances Astley-Jones and Nicola Gray, “Gambling Near-Misses Enhance Motivation to Gamble and Recruit Win-Related Brain Circuitry”, Neuron 61(3), 2009, 481–490. Small laboratory fMRI study; it does not establish an equivalent effect in AI-assisted work.

• Ellen J. Langer, “The Illusion of Control”, Journal of Personality and Social Psychology 32(2), 1975, 311–328; and Joowon Klusowski, Deborah A. Small and Joseph P. Simmons, “Does Choice Cause an Illusion of Control?”, Psychological Science 32(2), 2021, 159–172. The later paper reports seventeen experiments (N = 10,825) in which choice alone rarely produced the classic effect.

• Grant Gross, “Doomprompting: Endless tinkering with AI outputs can cripple IT results”, CIO, 17 September 2025. Practitioner commentary; source of the Carson Farmer quotation.

• Mihaly Csikszentmihalyi, Flow: The Psychology of Optimal Experience, Harper & Row, 1990. Foundational account of flow; not a diagnostic framework for distinguishing healthy from unhealthy AI use.

• Glean Work AI Institute, Work AI Index 2026, 10 June 2026. Survey of 6,000 full-time digital workers in the US, UK and Australia, fielded December 2025–January 2026. Vendor-funded and self-reported, with a sample skewed towards AI-intensive sectors.

• LeadDev, The Engineering Leadership Report 2026; Chantal Kapani, “AI coding is addictive. Engineers are paying the price”, 30 June 2026; and Steve Yegge, “The AI Vampire”, 11 February 2026. The report surveyed 600 engineering leaders. The article and Yegge essay add practitioner testimony; neither establishes causation.

• Ashley Stewart, “Microsoft pushes staff to use internal AI tools more, and may consider this in reviews. ‘Using AI is no longer optional.’”, Business Insider, 27 June 2025. Report of an internal memo; second-hand evidence.

• Julie Bort, “Coinbase CEO explains why he fired engineers who didn’t try AI immediately”, TechCrunch, 22 August 2025. Brian Armstrong’s self-reported account of the onboarding mandate and subsequent dismissals.

• James Booth, “KPMG Staff to Be Rated on AI Usage in Yearly Performance Reviews”, Bloomberg, 31 October 2025. Reported plans; the firm framed the approach as assessment against AI objectives rather than a simple usage quota.

• Gambling Commission, “New rules empowering consumers and boosting operator transparency”, 2025; and “Proposal 6: display of net position and time spent”, effective 17 January 2025. Operators must prompt customers to consider a financial limit before their first deposit (in force 31 October 2025). UK-licensed online casino products, excluding peer-to-peer poker, must display net position and elapsed session time.

• Julie Bedard, Matthew Kropp, Megan Hsu, Olivia T. Karaman, Jason Hawes and Gabriella Rosen Kellerman, “When Using AI Leads to ‘Brain Fry’”, Harvard Business Review and BCG Henderson Institute, 5 March 2026. Cross-sectional self-report survey of 1,488 full-time US workers. “AI brain fry” is an acute fatigue construct, not burnout; active intent to leave was 34 per cent among affected workers and 25 per cent among unaffected workers. Rates ranged from 5.6 per cent in legal to 25.9 per cent in marketing.

• World Health Organization, “Burn-out an occupational phenomenon”, 28 May 2019; Christina Maslach and Susan E. Jackson, “The Measurement of Experienced Burnout”, Journal of Occupational Behavior 2(2), 1981, 99–113; and Christina Maslach, Wilmar B. Schaufeli and Michael P. Leiter, “Job Burnout”, Annual Review of Psychology 52, 2001, 397–422. Sources for the occupational classification and three-dimension model; WHO states that burnout is not a medical condition.

• Johannes Siegrist, “Adverse Health Effects of High-Effort/Low-Reward Conditions”, Journal of Occupational Health Psychology 1(1), 1996, 27–41. Evidence on effort-reward imbalance; not a study of AI use.

• Microsoft, 2026 Work Trend Index: “Agents, Human Agency and the Opportunity for Every Organization”, published 5 May 2026. Survey of 20,000 AI-using knowledge workers across ten countries. Microsoft’s “Frontier Professionals” were more likely to report deliberately working without AI to keep skills sharp: 43 per cent versus 30 per cent. Vendor research on its own products.

• The Better AI Gets, The Harder It Is To Use – Five compounding challenges mean capability and deployability are moving in opposite directions.

TL;DR

• AI has cut the cost of writing a complaint or request to a company to about a penny a letter. Answering still costs pounds and is sometimes a legal duty.

• Inbound correspondence volumes are increasing. Graduate vacancies draw on average 140 applications each, and UK ombudsman complaints rose 54 per cent in 2024/25.

• AI makes low-merit and high-merit submissions look more alike. Polish is becoming a weaker signal of substance.

• The response is to upgrade your correspondence engine: re-price the perimeter where lawful, structure the intake around facts rather than prose, write for machine readability and build legitimate finality into every channel.

Before 1840’s postal reforms, the cost of mailing a letter in Britain was based on distance travelled and the number of sheets, with the bill paid by the recipient at the door. For example, a three-sheet letter from London to Edinburgh cost 39½ pence, and the receiver could simply refuse to pay. Following the reforms, a half-ounce letter could be sent anywhere in the United Kingdom for one penny, prepaid by the sender. Annual volume rose from 76 million letters in 1839 to 168 million in 1840, and nearly 350 million by 1850. The Post Office spent the following decades industrialising to cope, until London was taking deliveries up to twelve times a day.

Britain’s postal system behaved as any economist might expect. We are now seeing the same pattern repeated in every inbound channel at once, as AI radically cuts the cost of inbound. Sending a polished complaint, grievance or information request now costs pennies. But answering still costs pounds.

Four linked and compounding shifts are increasing the burden on receivers:

1. Falling cost to sender. The marginal cost of each inbound item is collapsing. Five years ago, a solicitor would charge £150 to £500 to draft a formal letter, and a claims firm would take up to 40 per cent of any award for its help. Today Garfield.Law, the first AI-only law firm authorised by the Solicitors Regulation Authority, drafts a compliant letter before action for £7.50. It costs about a penny to generate a 1,000-word letter at today’s frontier AI model prices, and far less on budget models. Costs will likely continue to fall as low-cost AI models become more capable.

2. Increasing inbound volumes. When the cost of an additional application, complaint or grievance approaches zero, people send more of them, to more organisations, and retry more often. The Institute of Student Employers counted 140 applications per graduate vacancy in 2024, up 59 per cent in a year. Volumes will likely continue to grow as consumers begin to deploy AI agents alongside chatbots.

3. Increasing difficulty identifying legitimate claims. AI raises the presentational standard while lowering the effort required to reach it. Grievances arrive lawyer-grade, citing the staff handbook accurately. That does not mean every AI-assisted submission is weak. It means a polished document is no longer evidence that the sender invested effort or that the claim has substance.

4. Cost to resolve stays high. Generating a reply may get cheaper, but taking and evidencing a decision is harder to automate. An individual can subscribe to an LLM in a few minutes, but an organisation needs to upgrade its entire machine for correspondence, paying the Deployment Ratio along the way. The Financial Ombudsman’s cost per resolved case rose from about £920 in 2019/20 to £1,116 in 2023/24. The receiver must still verify the facts, apply policy and stand behind the decision.

The four shifts compound most sharply where there is a legal requirement to respond. Much inbound can be triaged, deflected or ignored, but some cannot. The ACAS Code requires an employer to engage with the substance of every grievance however it was drafted. Behind the grievance sits the tribunal, where open single claims rose 42 per cent in the year to March 2026 and some cases are already listed for 2029. Monitored central government bodies received a record 94,526 FOI requests in 2025, and the ICO says an AI-drafted request citing invented decision notices must still be answered.

For illustration, let’s take a bank fielding 100,000 complaints a year, mid-table among the FCA’s reporting banks. Estimating an hour of handling per complaint, that is a 60-person operation and roughly £3.5 million a year in staff costs. About one case in twelve goes on to the ombudsman at £680 each, adding £5.7 million in fees. Call it £9 million a year, before any redress is paid. Now double the inbound, which is the direction of travel, and the bank must either hire another 60 handlers or find a new way to handle complaints.

Building a Better Machine

Faced with this set of trends, firms need to rebuild their machinery for responding.

• Increase the price to the sender, where the law allows. Ask for evidence up front, such as transaction IDs, dates and a signed declaration that the facts are true. Require professional representatives to declare when they use AI. On contested claims, pick up the phone, as human interaction increases the sender’s costs. Apply charges where possible. When the ombudsman charged representatives £250 a case, referrals fell from 37,100 a quarter to 4,300.

• Structure the incoming information. Stop reading prose. Whatever arrives, have AI distil it into structured data that your systems and people can process easily. Senders write what they like. Structure helps you to attend to the facts (e.g., harm, legal risk, new evidence) and ignore distractions (e.g., length, tone, who is copied in). Structure also makes it easier to spot patterns such as a coordinated campaign. Give legitimate AI agents a front door of their own, rather than letting them flood the free-text queues.

• Write for the machine reader. The decision to escalate or to give up is increasingly made by the sender’s AI. A well-evidenced answer is more likely to result in a dropped case, while a thin or templated one invites the next round. Track escalation rates by reply type and experiment to see what works best.

• Audit the promises, rewrite the rulebook. Part of how and when you respond to inbound is set by law. But the rest is often based on legacy standard operating procedures that pre-date AI. Update the rulebook with new definitions of what counts as a complaint, what counts as new evidence, how many reviews are needed and when a decision is final.

• Decide well, then stop. AI doesn’t just reduce the cost of the original request. It also reduces the cost of subsequent challenge. New workflows should be designed to survive challenge. Capture clear reasons, evidence and an audit trail of decision-making. Then hold the line. Fairness is not unlimited responsiveness.

• Mutualise the machine. Every firm in your sector faces the same flood, yet each builds its own defences. The financial services industry proved out this model with Cifas, a shared platform for fraud intelligence. Share intake standards, pool precedents, respond to fee consultations together.

Before 1840, the receiver could refuse a letter at the door, unpaid and unread. You don’t have that option. On the channels that matter most, you are obliged to accept, to read and to answer, at your own cost, however cheaply the thing was made.

Monday Morning Actions for Executives

• Gather trend data for inbound channels. Not only cost and volume, but also the ratio of genuine to spurious claims.

• Find the AI already arriving. Sample last month’s inbound for machine drafting and estimate the signal rate per channel.

• Assess your response. Measure how much work AI is already doing for you in inbound channels, and what more it could do.

Board Questions for the Back Pocket

• Have we modelled our exposure if inbound volumes explode?

• Are we tooling the answering side as fast as our counterparties are tooling the sending side?

• Which of our regulators and ombudsmen are themselves drowning in AI-generated inbound, and what does that do to the speed and quality of the decisions we depend on?

Sources & Notes

Several figures below are practitioner surveys or platform self-reports rather than audited data, and the bank example is the author’s own construction. Each is flagged where it appears and should be read as directional.

• The Penny Post. Before 1840, British postage was charged by distance and number of sheets and normally paid by the recipient on delivery, which meant an unwanted letter could be refused (The Postal Museum, “Rowland Hill’s postal reforms”). A three-sheet letter from London to Edinburgh cost 39½ pence, falling to 2 pence under the reform (W. Walker Hanlon and co-authors, “A Penny for Your Thoughts”, NBER Working Paper 30076). The Uniform Penny Post took effect on 10 January 1840; Post Office records report 112,000 letters posted on the first day, around three times the previous year’s count. Annual volume rose from 76 million chargeable letters in 1839 to 168 million in 1840 and nearly 350 million by 1850 (General Post Office figures via The Postal Museum). London eventually received up to twelve deliveries a day (London Museum).

• Before AI: the sender’s price. Published firm price lists and cost guides put a bespoke solicitor’s letter at £100 to £500 plus VAT before 2023, with £150 to £500 the defensible band for a formal letter of claim. Indicative market ranges, not an audited average. Claims management companies charged large shares of redress: the FCA noted some consumers paid over 40 per cent before its fee caps of 15 to 30 per cent by redress band took effect on 1 March 2022.

• Garfield.Law. The Solicitors Regulation Authority authorised Garfield.Law, the first purely AI-based law firm in England and Wales, announced May 2025. Services start at £2, with a debt-protocol-compliant letter before action at £7.50. SRA chief executive Paul Philip called the approval “a landmark moment for legal services in this country”. Garfield’s own comparison figure for a traditional solicitor’s letter (£150 to £400) is vendor-sourced, though it aligns with independent price lists.

• A penny to send. An author calculation, not a quoted price: a 1,000-word letter is roughly 1,300 output tokens, which at published API prices costs about $0.013 on a frontier model and under a tenth of a penny on the budget tier. Treat as an order of magnitude. The direction is settled: Stanford HAI’s 2025 AI Index records the cost of querying a GPT-3.5-class model falling from $20 to $0.07 per million tokens between November 2022 and October 2024, a more than 280-fold reduction.

• Graduate applications and screening. Institute of Student Employers, Student Recruitment Survey 2024: 140 applications per vacancy, up 59 per cent and the highest in over three decades, from 1.2 million applications for roughly 17,000 vacancies; the ratio held at 140 in the 2025 survey. LinkedIn told CNBC in October 2025 that applications were running at nearly 9,500 a minute; a platform self-report.

• Sticky resolution costs. FOS annual reports put its unit cost per resolved case at about £920 in 2019/20, £1,040 in 2020/21 and £1,116 in 2023/24. The standard case fee was £550 for six years to 2019/20, peaked at £750 between 2021/22 and 2023/24, was cut to £650, and is £680 for 2026/27 (FOS Plans and Budget). Ministry of Justice tribunal statistics show the mean time to clear a single employment claim rising from 19 weeks to 31 weeks in the year to late 2025. Rising unit costs have several causes, including inflation and case complexity; the trend, not the attribution, is the point.

• AI-drafted grievances. Paman Singh, principal associate at Weightmans, quoted in People Management on the “stratospheric rise” in AI-generated grievances taking hours to resolve. Irwin Mitchell’s survey of 200 HR professionals found 60 per cent had dealt with grievances they suspected were AI-generated, and 52 per cent said they were harder to resolve. A practitioner survey, self-reported.

• FOI. Cabinet Office statistics: monitored bodies received 94,526 FOI requests in 2025, the most since monitoring began in 2005. The ICO published guidance on AI-generated FOI requests in May 2026; Deborah Clark of the ICO cited higher volumes and greater complexity, and the guidance notes that AI-drafted requests may cite decision notices that do not exist yet remain valid requests. The section 12 cost ceiling is £450, or 18 hours, for central government.

• The grievance duty. The ACAS Code of Practice on Disciplinary and Grievance Procedures requires employers to engage with the substance of a grievance however it was produced; tribunals may adjust awards by up to 25 per cent for unreasonable non-compliance. See DAC Beachcroft, “Generative AI and grievances”.

• Tribunal backlog. Ministry of Justice Tribunal Statistics Quarterly and Law Society analysis: open single employment claims rose from around 45,000 in March 2025 to 64,000 in March 2026, an increase of 42 per cent. The Work Rights Centre’s May 2026 report records a claim lodged in January 2025 listed for hearing in 2029. The backlog has several causes; AI-assisted volume is one pressure among them.

• The bank example. Author arithmetic for planning rather than benchmarking. It assumes an hour of handling per complaint on average (simple cases close in days, complex ones run to the FCA’s eight-week limit), a loaded staff cost of £35 an hour (roughly £58,000 per full-time handler) and 1,650 productive hours a year, giving a 60-person operation for 100,000 complaints. Escalation of one case in twelve combines FCA aggregate complaints (around 3.6 million a year) with Financial Ombudsman volumes (305,726 in 2024/25); the datasets cover different periods and the 2024/25 representative surge inflates the rate. An illustration, not a benchmark.

• Ombudsman volumes. Financial Ombudsman Service annual data: 305,726 complaints in 2024/25, up 54 per cent on 198,798 the year before and the highest since the PPI peak. The standard respondent case fee is £680 from April 2026 (£650 in 2024/25 and 2025/26).

• Re-pricing precedent. From 1 April 2025 the FOS charges professional representatives £250 per referred case after ten free cases a year, reduced to £75 where the consumer wins. FOS quarterly data: in Q2 2025/26 representatives brought 4,300 cases against 37,100 in the same quarter a year earlier, and total new complaints fell by more than a third to 46,300. Part of the decline reflects the FCA’s pause on motor finance commission complaints rather than the fee; the FOS also reports withdrawn or abandoned representative cases falling from over a third in 2024/25 to 19 per cent, evidence the fee changed referral behaviour. From 1 April 2026 the representative fee is £260, reduced to £80 where the consumer wins.

• Mutualisation. Cifas, the UK’s not-for-profit fraud-prevention service, runs cross-sector data sharing among hundreds of member organisations; the working template for pooled defence against high-volume adversarial inbound. Sector-level answering infrastructure is the author’s extrapolation, not an observed practice.

Can AI Deploy Itself? Deploying a technology has always cost far more than buying it. AI may be the first that can help with its own deployment.

TL;DR

• AI investments convert to bottom-line value via a six-link Deployment Chain: Reimagine the work, Reallocate the resources, Reshape the organisation, Rewire the systems, Realise the value, Repeat the cycle.

• The gap between AI’s potential and delivered value can be explained by weak performance along the Deployment Chain. For example, only 21% of AI adopters have redesigned workflows, and only around 6% report material earnings impact. Weakness multiplies across the chain.

• Using survey data to estimate the losses at each stage of the Deployment Chain illustrates that up to 99% of AI’s potential value is being lost to leaky deployment.

• Better models without better deployment increase the value left on the table. Model capability is continuously improving while the Deployment Chain improves only when it is rebuilt. The best response is to fix the weakest link, not to upgrade the model.

In 1712, at a coalworks near Dudley, Thomas Newcomen’s engine began pumping water out of a flooded coal mine. Some 600 more had been put to work in Britain by 1775. But the engine wasted almost everything it burned. Some of the fire’s heat went straight up the chimney. More was lost turning water into steam. The cylinder was cooled and reheated on every stroke, the biggest single loss. The beam and the pump rods wasted a little more. In the end, less than 1% of the energy in the coal lifted any water. For half a century this was tolerated because the engines sat at coal mines and burned cheap coal.

AI is today’s frontier technology, as steam was in the eighteenth century. In common with early steam engines, little of AI’s full potential is finding its way to the bottom line. Between buying the model and banking its value sits the Deployment Chain: Reimagine the work. Reallocate the resources. Reshape the organisation. Rewire the systems. Realise the value. Repeat the cycle.

Reimagine the work

Reimagine is the first link in the deployment chain. AI enters existing workflows that are designed around people, not machines. Those workflows often have many variants, where colleagues have found ingenious workarounds to friction points. During the Reimagine stage, end-to-end workflows need to be reconceived around a new division of labour, where AI and human both have clear territory. Tried and tested tools such as Lean and Six Sigma underperform as the requirement is for a fundamentally new workflow, not an optimised version of the old one. McKinsey finds only 21% of AI adopters have redesigned any workflow. Call the pass rate from this link in the chain 20–35%.

Reallocate the resources

Reallocate commits capital and talent and specifies where in the P&L the benefit lands. For example, an approved efficiency business case should lead to someone’s budget being cut. NatWest is explicit about where benefits accrue. Its 2025 AI, cloud and simplification work freed roughly £100 million of investment capacity, and rather than let that saving dissipate it recycled the capacity back into the transformation. Few get this far: KPMG finds only around 7% of firms have established a clear view of AI’s return, and 42% have only partial sight of what they are even spending. Call the pass rate 25–45%.

Reshape the organisation

Reshape redesigns roles, skills, decision rights and accountabilities: who decides, who handles exceptions, who is accountable when the model is wrong. Reshape and Rewire (the subsequent link in the chain) are co-dependent. The organisation specifies what the system must support, and the live system reveals what the organisation must change. Job design alone is not sufficient to drive adoption. People must be retrained, and must learn when to accept, challenge and override. Deloitte finds only about a third have redesigned roles or career paths around AI, with retraining, not role redesign, the most common response. Call the pass rate 30–50%.

Rewire the systems

Rewire embeds that division of labour in production systems and data. Demos run on curated inputs, but deployments meet systems of record, half-clean data and undocumented exceptions. S&P Global found 42% of enterprises abandoning most of their AI initiatives in 2025, up from 17% a year earlier. Nearly half of proofs of concept are scrapped before production. Call the pass rate 15–55%, depending on what counts as production.

Realise the value

Realise is the conversion of operational improvements into bottom-line outcomes: cost removed, capacity redeployed, customer value uplifted. This is where the commitments made during Reallocate fall due. In Denmark, 25,000 workers across 7,000 workplaces adopted chatbots widely, saved about 3% of their time, and showed no measurable change in earnings or hours. The high-performing organisations in McKinsey’s sample, attributing material earnings impact to AI, are about 6% of adopters. Call the pass rate 5–20%.

Repeat the cycle

Repeat reruns the five gates as models improve and the economics shift. Stanford’s AI Index puts the fall in inference cost at more than 280-fold in under two years. As models improve, all links in the chain need to be upgraded to realise full value. For example, Intercom’s Fin customer service bot resolved a quarter of conversations at its 2023 launch and averages 67% today, roughly a percentage point improvement a month. Only 27% of enterprises have the MLOps tooling to run a Repeat loop at all. AI deployment should be an operating rhythm, not a programme with an end date.

The price of a weak link

We can define the Deployment Yield as the share of an AI investment’s potential value that reaches the P&L. Because value must flow through every link in the chain, total yield from an investment can be estimated by multiplying the individual yields. Five gates yielding 90% lead to an end-to-end yield of about 60%, whilst five gates at 80% yield about one third. The arithmetic is illustrative, but it explains a common pattern in technology deployments, whereby every team appears to perform well but the net impact is underwhelming.

Hence, with survey proxies, we can estimate the Deployment Yield for AI investments. Using the optimistic end of each range, we do not even reach 1%. The true yield will be higher, because the links are correlated: organisations disciplined enough to redesign the work tend also to commit the benefit and wire the systems. But not much higher. McKinsey finds about 6% of adopters attributing material earnings impact to generative AI, and BCG puts its ‘future-built’ share at 5%.

The same arithmetic suggests how yield can be increased. Improving any one link lifts the whole chain’s yield in proportion to that link’s weakness. Ten percentage points added to a 40% link raise total yield by 25%. The same ten points at an 80% link raise it by 12.5%. Effort pays most at the weakest link. So based on our estimates, the first places to look for improvement are the Realise and Reimagine links.

There are two fair objections. The first is that the estimates are flaky: they are self-reported consultancy surveys, and the links are not independent. Both points are true, but even with more optimistic assumptions, the yield remains in single digits, in line with the outcome surveys. The second objection is that every general-purpose technology experiences a ‘lag’ effect in its first years. This has been well proven by others. What’s different this time is the pace of change. Factories were rebuilt around the dynamo for over forty years while the technology itself changed slowly. With AI, we have a technology that is evolving faster than we are learning how to deploy it. Models are improving exponentially, whilst deployment improves only when someone rebuilds a link in the chain. Self-deploying agents may eventually rebuild links in the chain themselves; for now, that work is done by people, at organisational speed. Strong models do not compensate for weak links. In fact, as models improve, the value left on the table increases, unless the Deployment Chain is rebuilt.

The Playbook for Better Deployment

• Plan and cost the whole chain. Build every AI business case across all six Rs, ensuring the whole chain is fully costed.

• Estimate where the value is leaking. Look for workflow unchanged, benefit uncommitted, roles unaltered, exceptions unhandled, value unbanked.

• Fix the weakest link first. In a multiplicative chain the highest return sits at the weakest link. Resist the instinct to upgrade models until you’ve fixed the deployment chain.

• Give the chain one owner. The links cross functional lines: technology owns Rewire, HR owns Reshape, finance owns Realise – so by default nobody owns the whole.

• Run the chain as a rhythm, not a programme. Fund the Repeat laps as an ongoing expenditure, so that Deployment Yield is progressively increased and value unlocked.

Questions the Board should ask

• What is our Deployment Yield and where is it leaking?

• Do our business cases price the whole Deployment Chain, or only the model?

• Who is accountable for improving deployment?

The League Table

Steam-engine efficiency improved fastest once engineers started measuring it. From 1811 a monthly journal, Lean’s Engine Reporter, published how much water every Cornish mine engine lifted for each bushel of coal it burned. Once the numbers were public, the engineers competed, and by 1840 the best engine was four times more efficient. Two centuries on, the lesson holds. The best way to improve outcomes is not to buy a more powerful engine. It is to mend the weakest link.

Sources & Notes

Several figures below are directional rather than definitive. Historical duty figures depend on the coal bushel used (84 lb in Smeaton’s survey, about 94 lb in the later Cornish series) and are quoted on each source’s own basis; survey findings are largely self-reported; NatWest figures are the bank’s own account; the model-cost estimate comes from a venture investor; the chain arithmetic is the author’s illustration; and the per-link pass rates are the author’s planning ranges, built from self-reported surveys whose definitions differ.

• Pass-rate ranges. The per-link pass rates (20–35, 25–45, 30–50, 15–55 and 5–20%) and the end-to-end multiplication are an author synthesis for planning purposes, not observed benchmarks. Definitions differ across the underlying surveys, most of which are self-reported consultancy data, and the links are positively correlated, so naive multiplication understates the true yield. The observed single-digit shares of material-impact adopters are the empirical check.

• The Newcomen engine. First commercial installation 1712 at the Conygree coalworks near Dudley; around 600 engines at work in Britain by 1775 (John Kanefsky and John Robey, “Steam Engines in 18th-Century Britain: A Quantitative Assessment”, Technology and Culture, 1980). Steam was condensed inside the working cylinder, chilling and reheating the metal every stroke, and overall thermal efficiency was around 0.5%, so well under 1% of the coal’s energy reached the water (Britannica; standard histories). The engines were economic mainly at collieries burning near-free pit-head coal; Cornwall, with no coalfield, imported coal at high cost, which is why the county led demand for efficiency.

• Workflow redesign. McKinsey & Company, The State of AI: How Organizations Are Rewiring to Capture Value, March 2025 (surveyed July 2024): only about 21% of adopters had fundamentally redesigned at least some workflows. The November 2025 edition, The State of AI in 2025: Agents, Innovation, and Transformation (1,993 respondents), finds workflow redesign the organisational change most strongly associated with reported EBIT impact, with high performers nearly three times as likely as others to have redesigned workflows.

• Task length. METR, 2025: reports rapid growth in the length of tasks frontier models can complete reliably.

• Role redesign. Deloitte, State of AI in the Enterprise, 2026 edition: 3,235 leaders across 24 countries, fielded August to September 2025. About a third of organisations report redesigning roles or career paths around AI, with education the most common talent response; a narrower question on redesigning jobs themselves finds only 16% have done so (84% have not). A consultancy survey; findings are self-reported.

• Abandonment. S&P Global Market Intelligence, Voice of the Enterprise: AI & Machine Learning, Use Cases 2025 (451 Research survey of 1,006 IT and business leaders across North America and Europe, fielded late 2024): 42% of companies abandoned most of their AI initiatives, up from 17% a year earlier, with an average 46% of proofs of concept scrapped before production; the top cited obstacles were cost, data privacy and security. The same survey puts MLOps tool adoption at 27%, up from 24%, the capability proxy used for the Repeat link.

• Pilot-to-production. IDC’s AI CIO Playbook with Lenovo, 2025: 88% of observed proofs of concept did not reach wide-scale deployment, roughly 4 in 33 graduating. Vendor-commissioned with IDC fieldwork. The spread against S&P’s 46% scrappage figure is definitional: ‘wide-scale deployment’ is a higher bar than ‘any production’.

• The Danish null. Anders Humlum and Emilie Vestergaard, “Large Language Models, Small Labor Market Effects”, NBER working paper, May 2025, revised October 2025. Across roughly 25,000 workers and 7,000 workplaces, average time savings were near 3%, with no measurable effects on earnings or hours two years after adoption.

• Single digits. McKinsey & Company, The State of AI in 2025: Agents, Innovation, and Transformation (November 2025, 1,993 respondents): about 6% of adopters qualify as high performers attributing 5% or more of EBIT to generative AI, and the report describes enterprise-wide bottom-line impact as rare. BCG, The Widening AI Value Gap (September 2025, 1,250 executives): 5% of companies rated ‘future-built’ and generating substantial value. Both are consultancy surveys with a commercial interest in AI adoption.

• The dynamo. Paul A. David, “The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox”, American Economic Review Papers and Proceedings, May 1990. Electric motors passed half of US factory drive only in the early 1920s, four decades after commercial electricity, as factories rebuilt around unit drive.

• Model economics. Guido Appenzeller, Andreessen Horowitz, November 2024, estimates inference cost for constant capability falling roughly 10× per year. The estimate comes from an investor with an interest in the build-out it implies; the direction is widely accepted.

• Inference deflation. Stanford HAI, AI Index Report 2025: the cost of querying a model at GPT-3.5-level performance (64.8 on MMLU) fell from $20.00 per million tokens in November 2022 to $0.07 by October 2024, a more than 280-fold reduction in under two years; depending on task, LLM inference prices have fallen 9–900× per year.

• Fin. Intercom reports its Fin agent resolving conversations in the mid-20s at its March 2023 launch, rising to a 67% average across roughly 7,000 customers by December 2025, about a percentage point a month (Archana Agrawal, Intercom president, GTM Now podcast, February 2026; Intercom engineering account, 2026). The 25-to-above-80% spread across customers, and its dependence on ongoing content and escalation-rule maintenance, is from Intercom’s own optimisation guidance. Intercom sells Fin and charges per resolution, so the figures are the vendor’s own; one competitor analysis puts typical production rates at 45–53%.

• NatWest. NatWest Group reporting and CIO commentary, February 2026: £1.2 billion invested in technology, data and AI in 2025, with £100 million of investment capacity freed. These are the bank’s own figures.

• KPMG. Global AI Pulse survey, Q2 2026: more than 2,000 senior leaders across 20 countries at firms with revenues above US$50 million; 7% report having established ROI from AI, and 42% report only partial visibility into their AI spending. A consultancy survey from a firm that sells AI services; figures are self-reported.

• Lean’s Engine Reporter. From 1811, Joel Lean and later his family published monthly duty reports for Cornish pumping engines, engine by engine with engineers named, strokes taken from counters kept in locked boxes; publication continued until 1904. Reported duty rose from about 20 million (best 22.3 million at Wheal Alfred, 1811) to about 90 million by 1840. Alessandro Nuvolari (“Collective invention during the British Industrial Revolution: the case of the Cornish pumping engine”, Cambridge Journal of Economics, 2004) reads the episode as collective invention: open comparative measurement, after Watt’s patent expired in 1800, drove the improvement the monopoly years had not.

• The Plausibility Crisis – As AI-generated output becomes more abundant, convincing and unreliable, senior executives are becoming the last line of defence.

TL;DR

• Conway’s Law says that organisations build IT systems that copy their communication structures.

• AI is reversing the direction of travel: AI systems are reshaping organisations.

• The shift is happening through changed workflows, invisible human-agent teams, digital labour, delegated authority, culture, strategy and talent pipelines.

• A reversed Conway effect brings benefits such as increased efficiency and agility but also risks divergence between the organisation you have and the organisation you want.

• Organisations should treat AI as operating-model redesign, not as a technology rollout.

In 1968, the computer scientist Melvin Conway observed that any organisation that designs a system “will produce a design whose structure is a copy of the organization’s communication structure.” This became known as Conway’s Law: software mirrors the organisation that made it. If teams don’t communicate well, then their systems won’t either. Decentralised business decision-making, without strong technology governance, leads to fragmented IT architecture. The org chart leaks into the code.

AI is now reversing the arrow. As agents gain access to enterprise data, workflow tools, codebases and calendars, they do not simply mirror the organisation. Rather, they start to redesign it. They route work, summon other agents, trigger actions, and direct human effort, each of which changes how the organisation functions. We can call this Conway’s Reversal.

This is not true everywhere yet. Many deployments remain narrow: chatbots, copilots and summarisation assistants. But the direction is clear. Microsoft has described the rise of the “Frontier Firm”, built around hybrid teams of humans and agents. Its 2025 Work Trend Index found that 81 per cent of leaders expected agents to be moderately or extensively integrated into their AI strategy within 12 to 18 months.

The board-level question is not whether AI will change the organisation (it will) but whether the change is conscious. Firms can wake up to find that their workflows, teams, labour model, authority structures, culture, strategy and leadership pipeline have all shifted through accumulated AI adoption. Or they can purposely enlist AI to help design, build and operate an improved organisation.

Seven Quiet Redesigns

The Workflows No One Owns

AI is being inserted into workflows faster than those workflows are being consciously redesigned. A process that once had clear human handoffs becomes a chain of people, agents, integrations and exceptions. The process being followed quickly diverges from approved standard operating procedures.

The Teams Off the Org Chart

AI can dynamically assemble working groups that don’t appear on the org chart. A customer issue may involve a service agent, a pricing agent, a legal-review agent and a customer-success manager. That is a team, but it has no reporting line. It may assemble for a task and disappear once the work is complete.

The org chart is not dead, but it diverges from how work gets done. The practical unit of organisation may become the workflow, the customer journey or the agent permission boundary.

The Hires You Didn’t Make

The AI workforce does not arrive through hiring. It arrives through procurement, SaaS renewals and innovation budgets. The firm may not have approved new headcount, but it has added productive capacity. At IKEA’s largest retailer, Ingka Group, an AI assistant called Billie resolved 47 per cent of the customer enquiries it received between 2021 and 2023, while 8,500 call-centre workers were retrained as remote interior design advisers. Such agents are not employees in a legal sense, but they perform work, consume budget, and require oversight.

The Authority You Quietly Delegated

To be effective, agents need to be granted authority to (for example) recommend, approve, refund, reject, prioritise or contact customers. Access rights can quietly become decision rights if humans don’t challenge AI decisions. Any system that prioritises leads, complaints or candidates is making organisational judgements.

The Culture Your AI Normalises

Culture is not what leaders say, it is the repeated behaviours of the organisation. If AI handles repeated interactions, it starts to normalise (for example) tone, pace, empathy, caution, escalation and standards of judgement. A company may say it values customer intimacy while its agents optimise for speed. It may say it values judgement while training people to accept machine outputs.

The Strategy You Didn’t Approve

AI systems optimise local goals: increase conversion, lower cost, maximise throughput. Over time, the sum of those local optimisations can effectively become the company’s strategy. For example, a board may approve premium service while agents are tuned for efficiency. It may approve innovation while the controls around its AI reward caution.

The Leaders You Aren’t Developing

AI may automate the work through which people learn judgement. Junior employees have historically developed by drafting, analysing, coordinating, making mistakes and watching senior people. If AI absorbs that apprenticeship layer, the leadership pipeline changes.

The answer is not to stop this

Conway’s Reversal is not inherently bad. Handled well, it can drive productivity gains and increase organisational agility. Historically, software froze yesterday’s silos into today’s systems. The reversal lets a company shape the organisation that best suits the work and reshape it as circumstances change. Coordination that used to run through meetings now passes between agents directly, accelerating decision-making, and a small team directing a fleet of agents can produce what once took a large team. As routine work is absorbed, people are freed to move up towards higher-value tasks. The challenge for leaders is to ensure that AI evolves the organisation purposefully rather than by stealth. The right management response is therefore not to apply the brakes, but instead to make the new operating model visible, governable and strategically useful.

Making the Redesign Deliberate

1. Make the AI-shaped organisation visible. Executives need a live map of where AI is already changing work, including embedded SaaS features and agentic workflows. The aim is to see the real organisation, not the one implied by the org chart.

2. Redesign workflows, not isolated tasks. Value comes from redesigning work end to end, not sprinkling AI over old processes. Management should pick the priority workflows (for example, customer onboarding) and decide deliberately what humans do, what agents do, and what gets escalated.

3. Define authority, accountability and controls. Every material AI use case should be classified by its level of authority: inform, draft, recommend, decide, execute or escalate. Controls should then match the risk. A transcription agent does not need the same scrutiny as one that issues refunds.

4. Treat digital labour as workforce strategy. Agents and copilots should be part of workforce planning. Management should determine where AI substitutes for human work, where it augments employees, where it creates new supervision tasks and where it threatens apprenticeship. HR, finance, procurement, technology and risk need one shared view.

5. Align AI with strategy and culture. Local AI optimisation can quietly redirect the firm. Executives should check whether AI systems reinforce the approved strategy and the desired culture. If the strategy is premium service, agents cannot be tuned for cost.

Monday Morning Actions for Executives

• Ask for a one-page map of the most material AI-enabled workflows.

• Select pilot workflows and owners to redesign them around humans and agents.

• Classify material AI uses as inform, draft, recommend, decide, execute or escalate.

• Inventory agents and non-human identities, including credentials, permissions and system access.

• Ask HR to identify which apprenticeship tasks AI is removing and how judgement will now be developed.

Board Questions for the Back Pocket

• Which of our workflows is AI reshaping, and who owns them end to end?

• Where have we added digital labour, and what human work is it replacing, augmenting or supervising?

• What can our AI systems decide, recommend or execute, and where does human accountability sit?

• Are our AI systems reinforcing the strategy and culture we approved, or optimising towards something else?

Conway showed that organisations build software in their own image. AI has reversed the picture. The software is now rebuilding the organisation, one deployment at a time, and mostly by default. The org chart is being redrawn either way. The question is whose hand is on the pen?

Sources & Notes

Source caveat. Several figures below come from vendor reports or company-reported claims. They should be read as directional evidence of how the market is moving, not as benchmarks.

• Conway’s Law. Melvin Conway, “How Do Committees Invent?”, Datamation, April 1968. The original formulation: “Any organization that designs a system (defined broadly here) will produce a design whose structure is a copy of the organization’s communication structure.” The paper is reproduced on Conway’s own site. source

• Frontier Firm. Microsoft, 2025 Work Trend Index, “2025: The Year the Frontier Firm Is Born”, April 2025. The survey found 81 per cent of leaders said they expected agents to be moderately or extensively integrated into their company’s AI strategy within the next 12 to 18 months. Microsoft sells Copilot and agent tooling, so the framing serves its commercial narrative. source

• Ingka/Billie. Ingka Group newsroom and Reuters, June 2023: Ingka, the largest IKEA retailer, said its AI assistant Billie, launched in 2021, resolved approximately 47 per cent of the customer enquiries it received between 2021 and 2023, some 3.2 million interactions, while 8,500 call-centre co-workers were reskilled as remote interior design advisers. Company-reported figures, so directional rather than independently audited. source

Double (AI) Agents – What intelligence services can teach companies about AI agents

TL;DR

· AI models create no business value in isolation. Business value depends on deployment: workflow redesign, systems integration, training and governance.

· Most firms overfund access and underfund deployment. The Deployment Ratio is the dollars spent on deployment for every dollar of core AI cost (model access, tokens, licences and compute).

· History suggests that Deployment Ratios for new technologies can reach 10:1.

· AI may bend the pattern by self-deploying: agents mapping processes, engineering context, building integrations, designing workflows, generating training and enforcing controls.

· Firms delivering AI should track the ratio, fund deployment adequately, and progressively expand agent authority.

On the morning of 26 April 1956, a crane at Port Newark lifted 58 metal boxes onto a converted wartime tanker called the Ideal X, bound for Houston. Loading the Ideal X cost about 16 cents a ton, against $5.83 a ton for loose cargo loaded by hand, a 36× reduction. Malcom McLean’s shipping container technology was revolutionary. But capturing its benefits took decades of further investment. Standards had to be agreed. Ports had to be rebuilt. New labour deals had to be struck. The technology became significant only once the system around it changed.

AI follows a similar pattern: deployment costs can dwarf the visible cost of model access.

Core AI costs are transparent: model access, token fees, licences and compute. Deployment costs are scattered across the organisation and harder to see: workflow redesign, data readiness, systems integration, security, governance, evaluation, training and change management.

The Deployment Ratio is the dollars spent on deployment for every dollar spent on core AI costs. Underestimating deployment leads to inadequate budgets and, ultimately, to AI investments that fail to move the bottom line. MIT NANDA’s 2025 research reported that roughly 95 per cent of enterprise GenAI pilots delivered little to no measurable P&L impact.

Estimating the Deployment Ratio

The pattern is not unique to AI or shipping. When electricity first reached factories, a large electric motor simply drove the same line shafts that steam had turned. The gains came only when factories were redesigned around the workflow, with small motors at each machine. Computers followed the same pattern: Brynjolfsson, Hitt and Yang found roughly nine dollars of complementary organisational capital for each dollar of installed computer capital. ERP implementations run at 1.5–3× the licence fee on mid-market projects and 2–8× on complex SAP programmes.

Hard data on AI deployment is scarce, and the estimates are not apples-to-apples, but they point to similar ratios. An enterprise executive quoted by a16z puts LLMs at roughly a quarter of the cost of building use cases; BCG’s 10-20-70 rule puts algorithms at about a tenth of transformation effort.

We can triangulate with an indicative planning model. Depending on maturity, complexity and regulatory exposure, we might put data preparation at 0.5–4× core AI cost, integration and engineering at 1–3×, specialist talent at 1.5–4×, evaluation and monitoring at 0.3–1×, governance and security at 0.3–2×, and change management and process redesign at 1–3×. If added mechanically, those components imply roughly 5–17×. Because they overlap and share fixed costs, I would use 5–15× as a planning envelope, with 8–10× a defensible working assumption for a typical large enterprise.

The ratio varies by maturity. Pilots may need one dollar of deployment work for every dollar of core AI spend. Enterprise scaling may need eight to ten, as data, integration, training and control costs arrive. Later the ratio falls, as templates appear, skills spread and vendors bundle complementary capabilities with core AI.

Could AI Self-Deploy?

AI might bend the pattern if it can assist with its own deployment. This is not an agent waking up one morning and rewiring the company. Agents reduce the ratio when they automate specific parts of the deployment loop, provided they have context, bounded authority, audit trails, escalation routes and accountable humans who can accept or reject the new design. Without those conditions, agents raise the ratio, requiring more verification and control than non-agentic AI. Self-deployment can take six forms:

The first mechanism is process discovery. Agents read procedures, transcripts and logs to map how work is actually done, including exceptions.

The second is context engineering. Many deployment failures are context failures, so agents build the context layer, classifying documents, reconciling schemas and turning tacit knowledge into reusable assets.

The third is systems integration. Agents map APIs, write adapters and generate test cases.

The fourth is workflow design. Agents decide which steps to automate, where human approval stays mandatory, and which metrics define success. A useful agent does not merely automate the old process; it helps design a better one.

The fifth is training. For each new workflow, agents generate role-specific training and just-in-time coaching, driving down change-management costs.

The sixth is control. Self-deploying agents need AgentOps: machine identities, human owners, permissions, evaluations, budgets, audit logs, kill switches and escalation paths, with authority expanding as measured reliability justifies it.

Halving the Ratio

The evidence for self-deployment is thin. AWS’s Thomson Reuters case study reports a 4× velocity gain from AWS Transform, with modernisation moving from months to a two-week sprint. Moderna provides an operating-model anecdote, with more than 3,000 custom GPTs and a people-and-technology function merged around the redesign of work. Nubank uses a published framework that links context engineering, human-in-the-loop prompt iteration and evaluation to measured gains.

As a planning assumption, a mature enterprise applying all six mechanisms could plausibly halve the long-run deployment ratio, from 8–10× to around 4–5×. Early deployments may sit near today’s 8-10x range while the shared context, integration, training and AgentOps assets are built. But those assets are reusable, so each deployment should be cheaper than the last. Highly standardised workflows might reach 2–3×, while regulated or legacy-heavy environments stay closer to 5–8×. The target is not zero deployment cost, but a halving of the ratio over time.

Actions for Executives

· Track the Deployment Ratio and how it evolves. Record the full cost of deployment alongside core AI costs. A rising ratio can be healthy if the organisation is moving from pilots to production. A falling ratio can be healthy if agent-assisted deployment is reducing marginal cost.

· Fund deployment in the same budget cycle as core AI. Do not approve model access now and hope to find the change budget later.

· Make context and systems agent-readable. Create a single context layer across customer, product, finance, risk, policy and workflow data, with clean ownership, metadata, lineage and access controls. Expose high-value systems through APIs rather than relying on agents clicking through brittle user interfaces.

· Use agents for system integration and modernisation. Let agents read legacy code, API documentation, database schemas, logs and tickets. Ask them to propose mappings, generate adapters, create test cases, draft migration plans and open pull requests.

· Use AI for workflow design before software is built. Feed process maps, SOPs, screen recordings, call transcripts and ticket histories into workflow-design agents. Ask for the target operating model, exception paths, control points, roles, service levels and KPIs before approving a new build.

· Make training a by-product of deployment. Each new AI-enabled workflow should automatically produce role-specific micro-courses, simulations, checklists and just-in-time help.

· Govern agent authority in stages. Agree early which processes agents may inspect, which recommendations they may make, which actions they may take and where human approval is required. Treat agents as workers, with human owners, machine identities, permissions, budgets, audit logs, evaluations and rollback plans. Increase agentic authority (recommend, then draft, then execute) based on measured reliability, not enthusiasm.

Questions for the Board

· Do we track our Deployment Ratio and how it is evolving?

· Is our budget sufficient to effectively deploy the AI we buy?

· Are we enlisting AI to help reduce the costs of deployment?

· What would have to be true for the next AI deployment to cost half as much as the last one?

Metal shipping containers did little to transform trade until the system around the boxes changed. AI may eventually help build more of that system itself. But until it does, the winners will not be the companies that buy the most intelligence. They will be the companies that fund the work needed to deploy it.

Sources & Notes

Source caveat. Several figures below should be read as directional rather than universal. Historical deployment ratios are rarely clean budget lines; they often combine physical infrastructure, labour agreements, organisational redesign and intangible capital.

Containerisation. Marc Levinson, The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger, Princeton University Press, 2006. Source for the Ideal X voyage, the 58 containers, the Port Newark-to-Houston route and the loading-cost comparison.

Factory electrification. Paul A. David, “The Dynamo and the Computer,” American Economic Review, 1990; Warren D. Devine Jr., “From Shafts to Wires: Historical Perspective on Electrification,” Journal of Economic History, 1983. Source for the idea that electrification paid off after factories reorganised around unit drive rather than simply replacing steam with a central electric motor.

Computer capital and organisational capital. Erik Brynjolfsson, Lorin M. Hitt and Shinkyu Yang, “Intangible Assets: Computers and Organizational Capital,” Brookings Papers on Economic Activity, 2002. Market-valuation study of 1,216 US firms, 1987–97, finding roughly $9 of complementary organisational capital associated with each $1 of computer capital. An econometric inference, not a literal budget line.

ERP deployment costs. Implementation-to-licence ratios from ERP Research and industry benchmarks: software licences are typically only 20–30 per cent of first-year cost, while implementation (integration, configuration, data migration, training and change management) runs 1.5–3× the licence fee, and 2–8× on complex SAP S/4HANA programmes. Benchmark and outcome data: Panorama Consulting Group, annual ERP Report, drawn from thousands of implementations, with an average implementation of about $450,000 in 2025. For large SAP R/3 installations, less than 20 per cent of the typical $20 million cost was hardware and software; the rest was consultants, process redesign and training (Brynjolfsson et al.).

Pilot failure. MIT NANDA, “The GenAI Divide: State of AI in Business 2025.” Source of the 95 per cent figure; based on interviews, a worker survey and deployment analysis, so treated here as directional.

External deployment-ratio estimates. a16z, “16 Changes to the Way Enterprises Are Building and Buying Generative AI” (Sarah Wang and Shangda Xu, 21 March 2024), is based on interviews with dozens of Fortune 500 and top enterprise leaders and a survey of 70 more; it says that model access alone is not enough, that implementation and scaling require specialised technical talent, that implementation was one of the largest AI-spend areas in 2023, and quotes an enterprise executive saying LLMs are roughly a quarter of use-case build cost. BCG’s 10-20-70 rule treats algorithms as about 10 per cent of transformation effort, with the rest in data, technology, people, process and change. No published study computes a clean enterprise foundation-model deployment ratio; the 5–15× band is a synthesis of these sources and historical precedent.

Component ranges. The category-level ranges are an author synthesis for planning purposes, not directly observed benchmark ratios. Directional support comes from a16z on implementation and specialised talent; D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” NIPS 2015, on production ML maintenance costs, glue code, configuration, monitoring and pipeline jungles; Eric Daimler and Ryan Wisnesky, “Informal Data Transformation Considered Harmful,” arXiv, 2020, on enterprise data integrity and data-cleaning burdens; NIST AI 600-1, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” July 2024, on governance, measurement, security, privacy, monitoring and value-chain/component-integration risks; Deloitte CFO Signals Q3 2023 and “What Does Generative AI-Ready Look Like for Finance?” on talent, data readiness and workflow integration; and MIT NANDA’s GenAI Divide report on workflow adaptation and integration. The ranges should be read as overlapping, not additive audited budget lines; “talent” means incremental specialist capability not already counted elsewhere.

Self-deployment ratio estimate. The estimate that a mature self-deployment stack could lower repeatable-use-case ratios to 4–5×, and in narrow standardised workflows perhaps 2–3×, is an author synthesis rather than an observed benchmark. It extends the planning ranges above by assuming the organisation has reusable context, integration, training, workflow-design and AgentOps assets. Fragmented, regulated or legacy-heavy environments may remain above 5×.

Agentic modernisation anecdotes. AWS Transform product materials describe agentic modernisation of mainframe, VMware, Windows and legacy code, including automation of assessments, code analysis, refactoring, dependency mapping and transformation planning. AWS customer stories used directionally here include Thomson Reuters (4× velocity improvement, 1.5 million lines of code modernised per month, transformation time moving from months to a two-week sprint), ADP (thousands of business rules extracted in hours, more than 90 per cent manual-effort reduction and 80 per cent faster rule extraction), IDEMIA (4× faster application transformation and 30 per cent TCO reduction) and CSL (10× faster initial wave planning, 12× faster application discovery and 10.5 weeks saved across 1,072 applications). These are vendor-published case studies, so they should be treated as useful anecdotes rather than independent benchmarks.

Moderna. Wall Street Journal reporting, May 2025, on the merger of Moderna’s HR and technology functions under Tracey Franklin as Chief People and Digital Technology Officer; Moderna–OpenAI partnership materials for the 3,000+ custom GPTs embedded across legal, research, manufacturing and HR. Company-published figures, so directional.

Production support-agent deployment. Nubank, “Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework,” arXiv, June 2026, is used for the importance of context engineering, human-in-the-loop prompt iteration, LLM-judge evaluation and ideation-to-production validation. The reported card-delivery deployment showed a 37 percentage-point improvement in AI transactional NPS and a 29 percentage-point gain in self-service rate versus prior agent variants; treated as one firm’s published account, not a universal result.

• Should You Add a Bot to Your Board? – The risk is not that a machine joins the board, but that directors stop reading once it has.

• Is AI Blunting Your Strategy? – As strategy development relies increasingly on LLMs, true differentiation from competitors is becoming more elusive.

On 17 January 1920, the Volstead Act closed America’s saloons. The Act did not stop Americans drinking. Consumption initially fell, but illicit drinking pushed the trade underground. Illegal speakeasies grew to replace the licensed saloon trade. In June 1932 John D. Rockefeller Jr., who had poured a fortune into the temperance cause, admitted he had been wrong. Eighteen months later the country repealed the ban.

Many companies are now running a version of Prohibition, restricting which AI tools staff can use. While most allow some AI use, the most powerful and familiar models are often unavailable to employees. As with Prohibition, this creates more problems than it solves. Shadow AI, the unsanctioned and unmanaged use of AI by employees, is not Shadow IT with a new badge, but something far more dangerous.

Shadow IT versus Shadow AI

A long-recognised issue, Shadow IT exists when staff reach for unsanctioned technology tools. Typically, this is born not of malice but rather from frustration at not being provided with the right tools for the job. For example, a colleague on a deadline may use an unsanctioned file transfer tool to send a large file to a supplier, if the corporate email system limits the size of attachments.

Shadow AI is a newer and faster-growing threat. Many employees already use AI both in their personal lives and at work. Meanwhile, most organisations have introduced strict limitations on which AI can be used. In Littler’s 2024 C-suite survey, 44 per cent of respondents said their organisation had a generative-AI policy, of which 55 per cent restricted use to specific groups or purposes.

If the enterprise restricts access to AI tools, or provides access to less capable AI models, the temptation to log in to a personal account may overpower any concerns about a policy breach.

Shadow AI is not restricted to chatbots such as ChatGPT. Software engineers may reach for coding tools, designers for auto-generators and recruiters for candidate-screening tools. Harder to spot are browser plug-ins that provide access to a suite of AI models. Finally, there are citizen-developed tools, where employees use vibe-coding platforms to build bespoke tools.

Shadow AI is already a real problem. Netskope’s 2026 report found that 47 per cent of people using AI at work route through personal accounts. And Shadow AI introduces new categories of risk that go well beyond the known risks of Shadow IT.

Exfiltration

The first risk presented by Shadow AI is exfiltration of data to unauthorised third parties. Most company-approved systems have built-in controls that flag bulk data transfers. No such controls exist when the system is unauthorised.

Shadow AI amplifies that risk, because (unlike conventional software), AI tools can be improved by training on user data, so tool providers have incentives to collect data. In many consumer tools, data sharing is enabled by default.

Infiltration

The second risk of Shadow AI is infiltration, and it comes in two forms. The first form is found in new classes of malware. LayerX catalogued at least sixteen Chrome extensions posing as ChatGPT tools and designed to steal identities. IBM’s 2025 analysis found that, among breached organisations, 20 per cent had an incident involving Shadow AI.

The second, subtler form of infiltration is the entry of incorrect or biased AI output into decision flows, as I described in The Plausibility Crisis. A 2025 study by KPMG and the University of Melbourne found that 57 per cent of workers admit to hiding their use of AI and presenting its output as their own. Sanctioned tool output can be checked and audited, but Shadow IT output cannot.

Subversion

The third risk of Shadow AI removes the human from the loop entirely. Autonomous agents now hold credentials of their own, including API keys. These risks can be managed with appropriate controls. But unsanctioned agents bypass them all.

Most organisations have no grip on this. CyberArk counted 82 machine identities for every human, 42 per cent of them with privileged access, and reports that 47 per cent of organisations cannot secure their Shadow AI usage.

Espionage

A fourth concern is potential exposure to foreign legal regimes and state actors. Data hosted by Chinese providers may be subject to PRC national-security obligations, leading many firms to block Chinese models including DeepSeek. Shadow AI imposes no controls on data residency, which may be obscured by model brokers that route requests across hundreds of models. On OpenRouter, one such broker, Chinese open-weight models averaged 13 per cent of weekly token usage.

Bans are not the answer

It is tempting to try to reduce risks through stronger bans. But the evidence suggests that doesn’t work. In Software AG’s 2024 survey, 46 per cent of staff said they would keep using personal AI tools even if forbidden outright. Senior staff are the most likely to ignore policy, with 69 per cent of the C-suite agreeing that the speed was worth the risk, according to a 2025 BlackFog study.

Firms are now moving away from blanket bans. Cisco’s reports the share of organisations with bans or strict limits on use of GenAI tools falling from 28 per cent to 7 per cent year on year.

The challenge for organisations, then, is to make approved use easier, safer, and more observable than the unofficial route.

Getting the balance right

Following Prohibition, the United States did not give up on managing alcohol. It moved to licensing, taxation, and inspection. Firms need the same shift in posture towards AI, adopting a balanced strategy that unleashes the technology’s value while managing the risks set out above. Six moves make up that strategy:

1. Maintain a live picture of which tools and models are in use, through which accounts, and which jurisdictions they reach.

2. Compete with Shadow AI by providing equally capable AI tools. People abandon the sanctioned path when the tools are weaker than those available through a personal account.

3. Place controls where the work happens and scale them to the risk. Brokers can route around firewalls, so verification belongs where AI output enters a decision or an agent takes action. A marketing brainstorm need not face the same scrutiny as an autonomous agent with database access.

4. Treat the non-human actors as actors. Agents that hold credentials need their own identities and permissions scoped to the task.

5. Control the supply chain, not just the end-user. Contract carefully with providers over how your data is handled.

6. Strengthen accountability. Name an executive owner and treat the whole thing as a standing capability with adequate budget and regular reviews.

Monday Morning Actions for Executives

• Request an audit. Understand which AI tools and models are in use, through which accounts.

• Close the capability gap. Upgrade models so that staff are not tempted into the shadows.

• Inventory the agents. List every non-human identity with the access it has.

• Offer an amnesty. Let people declare the tools they already use, without penalty.

Questions for Board Members’ Back Pockets

• Do we know which AI tools our people actually use, through which accounts, and which models and jurisdictions they reach?

• Are our sanctioned tools as capable as those we have banned?

• When AI output enters a decision, who verifies it?

• Do our access controls extend to the agents now acting on our behalf?

Just as Prohibition drove drinking underground, AI bans drive usage into the shadows. A more nuanced approach can unleash AI’s power without creating new risks.

Footnotes & Sources

• Source caveat. Several figures below come from vendor telemetry or commissioned surveys, so the precise percentages should be read as directional rather than universal.

• Prohibition. The Eighteenth Amendment and National Prohibition (Volstead) Act took effect on 17 January 1920; the Twenty-first Amendment repealed Prohibition in December 1933. John D. Rockefeller Jr., a major funder of the temperance cause, reversed his position in a public letter published in the New York Times on 7 June 1932, writing that “respect for all law has been greatly lessened.” Alcohol consumption initially fell, but enforcement failures pushed the trade underground. New York speakeasy estimates range widely, from about 20,000 to 100,000.

• Littler, Generative AI in the Workplace (2024 survey of 330+ C-suite executives). 44% had a generative-AI policy, up from 10% in 2023; of those, only 3% prohibit generative AI entirely, while 55% restrict AI tools to specific groups.

• Netskope, Cloud and Threat Report 2026 (telemetry, October 2024 to October 2025). 47% of AI users access via personal or unmanaged accounts; generative-AI data-policy violations more than doubled. Netskope sells data-loss-prevention tooling.

• OpenAI data controls. OpenAI says it does not use data from ChatGPT Enterprise, ChatGPT Business, ChatGPT Edu, ChatGPT for Healthcare, ChatGPT for Teachers, or the API platform for training by default. OpenAI’s Help Center says ChatGPT Free, Plus and Pro users in personal workspaces have data sharing enabled by default, but can opt out for new conversations.

• Malicious AI browser extensions. LayerX Research identified at least 16 malicious Chrome extensions posing as ChatGPT tools and stealing session tokens for connected services such as Slack and GitHub. OX Security (29 December 2025) separately identified two Chrome extensions impersonating a legitimate AI sidebar, more than 900,000 installs between them, exfiltrating users’ ChatGPT and DeepSeek conversations and open-tab URLs to attacker-controlled servers; one carried Google’s ‘Featured’ badge. OX Security and LayerX sell security products.

• IBM, Cost of a Data Breach Report 2025 (Ponemon Institute; 600 organisations; breaches March 2024 to February 2025). High Shadow AI exposure added $670,000 to the average breach; 20% suffered a breach due to incidents involving Shadow AI. Separately, among organisations reporting an AI-related breach involving AI models or applications, 97% lacked proper AI access controls. IBM sells AI-governance and security products.

• University of Melbourne and KPMG, Trust, Attitudes and Use of Artificial Intelligence: A Global Study 2025. Survey of 48,000+ workers across 47 countries. 57% admit hiding their use of AI at work and presenting AI-generated content as their own; 66% do not evaluate AI outputs for accuracy; 48% admit using AI in ways that breach company policy, including uploading sensitive company information into public tools. Self-reported.

• CyberArk, 2025 Identity Security Landscape (Vanson Bourne; 2,600 security decision-makers; organisations of 500+ staff; released April 2025). 82 machine identities for every human; 42% with privileged access; 68% lack identity security controls for AI and 47% say they cannot secure their Shadow AI usage. CyberArk sells identity-security tooling.

• Netskope, Cloud and Threat Report: Generative AI 2025 (telemetry across 3,500+ organisations). At DeepSeek’s peak, 91% of organisations had users attempting access and 75% blocked it entirely; about 43% still block it in the 2026 report. Netskope sells data-loss-prevention and blocking tooling.

• OpenRouter, State of AI 2025 (with Andreessen Horowitz; about 13 months to November 2025; 300+ models, 60+ providers). Chinese open-weight models averaged 13% of weekly tokens, peaking near 30% in some weeks. Separate 2026 snapshots reported much higher shares among the top-used models; the figures use different denominators.

• PRC National Intelligence Law (2017, amended 2018), Article 7 (China Law Translate). Requires organisations and citizens to “support, assist and cooperate with” state intelligence work. Legal scholar Jeremy Daum argues the provision lacks an enforcement mechanism and is bounded by “in accordance with law.”

• BlackFog and Sapio Research (fieldwork November 2025; 2,000 respondents, UK and US, firms of 500+ employees). 49% use unsanctioned AI; 69% of the C-suite said speed outweighed privacy and security. BlackFog sells anti-data-exfiltration tooling.

• Cisco 2026 Data and Privacy Benchmark Study (9th edition; 5,200+ professionals; fielded September 2025). The share of organisations reporting outright bans and/or strict limits on data entry into GenAI tools fell from 28% to 7% year on year, a 21-point drop. The study’s chart groups outright bans with limits on data entry. Cisco sells security and AI-governance products.

• Software AG, 2024 (6,000 knowledge workers, US, UK, and Germany). 46% would continue using personal AI tools even if banned outright. Software AG is a software vendor.

The Better AI Gets, The Harder It Is To Use. Five compounding challenges mean capability and deployability are moving in opposite directions.

Is AI Blunting Your Strategy? As AI automates the analysis, it quietly erodes the conditions that build judgement, and convergence on the same models breeds strategic monoculture.

Double (AI) Agents Autonomous agents act inside the business like recruited insiders, so governing them owes more to intelligence tradecraft than to IT.

In May 2014 Deep Knowledge Ventures, a Hong Kong venture fund, announced VITAL as a new board member. The announcement dressed it up in boardroom language, but legally it could only function as an observer because Hong Kong law did not allow a machine to qualify as a director or vote. More publicity stunt than governance breakthrough, VITAL was an algorithm running fuzzy logic across about fifty parameters.

Fast forward to April 2026. Lloyds Banking Group reportedly became the first FTSE 100 company to trial a specialist AI boardroom tool with access to confidential board material. The tool, built by Board Intelligence, summarises the pack, flags inconsistencies, draws connections across papers and checks the reasoning for bias. Like VITAL, it has no vote.

The question Lloyds is grappling with is now mainstream. According to Board Intelligence, 84 per cent of surveyed board directors, chief executives and finance chiefs said their boards had debated which decisions should stay human-led and which could be handed to AI.

The question is no longer whether bots will be added to boards, but what happens to a board’s judgement once a bot has entered the room.

The Case for Bots on the Board

There is a clear case for having AI tools support board work.

Capacity. Board Intelligence and governance-sector surveys point in the same direction: many packs now exceed two hundred pages, some approach a thousand, and directors typically have only a few hours to read them. A director reading thirty pages an hour and giving the pack four hours won’t get through it. By contrast, AI can hold the whole pack in view at once, check the assumptions underpinning each proposal, and flag the claim on page two hundred that does not stand up. It reads what no human can, it never tires, and on sheer volume and consistency it is simply better than we are.

Oversight. A board’s oldest weakness is that management controls the information. The pack is written by the people whose proposals it contains and often underweights negative views. Two in five directors say the reports are not upfront about bad news, and half find them light on risk. A non-executive with a capable AI tool can interrogate that pack on their own terms and ask a pointed question that might otherwise require specialist knowledge. For the first time, the people who oversee a company can match the people who run it for analytical firepower. Used this way, AI does not threaten oversight but significantly strengthens it.

Debiasing. The errors that distort a board are errors of psychology, not arithmetic: groupthink, anchoring on management’s framing, deference to the most senior or most persuasive voice in the room. Prompted well, an AI model will put the counter-view that the dynamics of the room suppress. Researchers building AI-mediated devil’s advocate systems find they surface the dissent a group would otherwise bury.

The Case Against Bots on the Board

Nevertheless, introducing AI to the boardroom brings new challenges.

Diligence. Reading a long board pack is hard work, and finding the inconsistency on page two hundred takes focus. But that work is not overhead. It is the job. Company law in many jurisdictions requires a director to exercise independent judgement and to apply reasonable care, skill and diligence. AI can support that duty, but it cannot discharge it for them. When an AI does the reading and the checking, it gives the impression that directors are informed when they are not.

Complacency. The more capable the tool, the less its user feels the need to check the output. A busy director, watching a model produce a run of correct answers, will check fewer answers over time. That is human nature, and expertise is no protection: research has shown that automation complacency is found in experts as well as in novices. The gains from pairing a person with a machine are also not evenly spread. A meta-analysis of more than a hundred studies in Nature Human Behaviour found that human-and-AI teams beat the better of human or machine alone when the task was to create something and fell behind when the task was to make a decision, which is the work a board does.

Fabrication. Frontier models are improving all the time. But hallucinations remain a reality. In 2023 two New York lawyers filed a court brief citing six cases that ChatGPT had invented. They asked the model whether the cases were real, were told yes, and filed them. Two years later, a Big Four firm refunded part of a government contract in Australia after its AI-assisted report invented academic references and a quotation.

Accountability. It must sit somewhere, and it cannot sit with a model. You cannot sue an algorithm, strike it off, or call it before a regulator. Each of those needs a person to answer for the decision. The courts have been clear about how far that duty runs. When Barings collapsed in 1995, directors who had relied on others for work they did not understand were disqualified all the same: a director must inform themselves of the company’s affairs, and delegating a task does not absolve them of the duty to supervise it.

Bias. The debiasing prize is real, but bias does not vanish when you reach for a model. It’s possible to reduce one bias but introduce another, if human groupthink is replaced by biases embedded in an AI model and its training data. Identical mortgage applications have been scored worse by a leading model when the applicant was presumed to be Black, a gap that closed only when the model was explicitly told to be unbiased. Models are frequently updated, and new biases can be introduced.

Using AI Well

None of this means banning the tool. It means being clear about what good use looks like. Four disciplines separate using AI well from using it badly. The first three are about how to use it. The fourth is a limit on use.

Expand, Not Synthesise

The instinct is to use AI to get to the point: summarise the pack, draw the conclusion, save the time. Its real value for a director is when it is used to provide an expanded view of the context: the history behind a decision, the comparable cases, the second-order effects, the questions a specialist would ask. Used to compress a subject into a few lines, it leaves a director knowing less. Used to open the subject up, it sends them to the table better educated.

Advise the Individual, Not the Room

A single tool, fed the same papers and asked the same questions by the whole board, will tend to give everyone the same answer. This narrows the range of views in the room at the very moment a board most needs breadth, and it encourages a consensus which is based on the model agreeing with itself. A board exists so that independent minds reach their own judgements and then test them on each other. Used to support each director, AI widens the range of challenge. Used as a shared oracle, it narrows it.

Challenge, Not Confirm

The most valuable thing AI can do with a board paper is find the holes in it: the unstated assumption, the figures that don’t reconcile, the alternative that wasn’t considered. The temptation is to ask whether the recommendation is sound, and feel reassured when the answer is yes. But models tend to agree with the way a question is framed. In one Harvard Business School study, when professionals challenged a confident model, it escalated its persuasion rather than disclosing its limits. The better approach is to ask “show me why this is wrong.”

Inform, Not Decide

The fourth discipline is the ‘red line’ for AI use. AI can inform a board’s judgement in the ways described, but it must never take the decision. There is a trap here, because a tool can quietly become the decider despite having no vote, if the board treats its output as the answer. If a recommendation passes because the AI produced it, the board has in effect delegated the decision to the AI. A director who can point to what they read, why they doubted it and how they reached their own view has used the tool. A director who can only point to what the tool concluded has been used by the tool.

Actions for NEDs

● Learn to prompt. The value derived from the tool depends entirely on the quality of the questions a director knows how to put. Training non-executives to explore a subject from several angles, rather than only ask for a summary, is among the highest-return investments a board can make.

● Own your prompts. Build and keep your own lines of inquiry, reflecting what you personally bring to the board, and resist any move to standardise every director onto one shared prompt. For example, a risk-minded director and a customer-minded director should be asking the tool different things.

● Use AI as a critic. Before the board accepts any recommendation, have the tool build the strongest case that it is wrong: the weakest points, the buried assumptions, and the questions management would least like to be asked. A recommendation that survives the attack is stronger for it.

● Never let AI decide. Keep AI out of the vote, the recommendation and the tie-break, and make sure you can state every decision as your own reasoned judgement, not the machine’s conclusion that you endorsed.

Actions for Chairs

● Own the board’s information. Typically, the chair is responsible for accurate, timely and clear information, so the chair should lead the board in agreeing what AI is and is not used for.

● Fix the pack, not just the reader. Work with the executive to produce shorter, sharper papers. Frame risks clearly so that NEDs don’t have to hunt for them. And ensure every figure is traceable to its source. Fix the pack and you reduce the temptation for NEDs to use AI as a shortcut.

● Name an owner, and vet the model. Any AI with access to board papers needs a named senior person answerable for it. Before it sees confidential papers, that person should establish where the data goes and whether the tool trains on it, and confirm that the model is behaving as intended. Third-party models should be treated as a dependency, with version control, revalidation after update, periodic sampling for biases and an audit trail.

● Protect the record. The company secretary supports the board’s record, and the minutes must capture the board’s reasoning, not the machine’s conclusion. There is an irony in this. The company secretary’s own work is among the first a board will be tempted to hand to AI. That should be strongly resisted as the company secretary plays a critical role in governing the machine in the boardroom.

VITAL was a gimmick. The AI tools that came after were far smarter and are finding their way into the boardroom. Used well, they can improve decision-making by offering new perspectives and calling out errors and biases. Used casually, both decision-making and accountability suffer.

Footnotes & Sources

1. Deep Knowledge Ventures / VITAL, 2014. Hong Kong venture fund that announced VITAL as a board member on 13 May 2014. The legally safer reading is that VITAL could only function as an observer: it had no legal vote, because Hong Kong law did not recognise a machine as a director. Widely judged at the time to be a publicity exercise.

2. The Times / Retail Banker International, April 2026. Lloyds Banking Group was reported to have become the first FTSE 100 company to trial a specialist AI boardroom tool with structured access to confidential papers, summarising reports, drawing connections and checking for bias. Quotes from corporate governance director Nicola Putland and Board Intelligence chief executive Pippa Begg, including the “dangerous leap” remark. A live-meeting AI that interjects “I disagree” is described by Board Intelligence as a possible future step, not a current feature. Operational detail is second-hand.

3. Board Intelligence, Board Value Index, 11 June 2026. Survey of 405 board directors, chief executives and finance chiefs across the UK, the US, the Nordics and the Middle East: 84% had debated which decisions should stay human-led and which could be handed to AI. Caveat: commissioned and published by the vendor that sells the board-AI tool, so the figures are marketing as much as evidence.

4. Board pack volume and information asymmetry. Chartered Governance Institute and Board Intelligence data, corroborated by Cambridge Judge Business School research, point in the same direction: almost a quarter of packs ran past 200 pages in 2025, against 13% in 2020, with some approaching 1,000 pages; directors read roughly 30 pages an hour, spend about four hours, and leave close to half unread. On asymmetry, about 42% say reports are not upfront about bad news, around 50% find them light on risk, and roughly 55% receive papers fewer than five working days before the meeting. Several figures originate with the vendor.

5. Companies Act 2006, ss.172–174. The duties to promote the company’s success, to exercise independent judgement (s.173, the duty most directly at risk where a board defers to a model’s framing), and to exercise reasonable care, skill and diligence. Wright v Chappell [2024] EWHC 1417 (Ch), the BHS liquidation case, found directors in breach of their Companies Act 2006 duties.

6. Re Barings plc (No 5) [2000] 1 BCLC 523. Also reported [1999] 1 BCLC 433; upheld on appeal as Secretary of State for Trade and Industry v Baker. Disqualification proceedings after the 1995 collapse of Barings, brought down by trader Nick Leeson. Jonathan Parker J held that directors have a continuing duty to inform themselves of the company’s affairs, and that delegating a function does not absolve a director of the duty to supervise it.

7. FRC UK Corporate Governance Code 2024. For companies to which the Code applies, Principle F says the chair is responsible for accurate, timely and clear information, alongside related provisions on board information and effectiveness; the internal-controls declaration (Provision 29) takes effect for financial years beginning on or after 1 January 2026.

8. Parasuraman & Manzey, “Complacency and Bias in Human Use of Automation,” Human Factors, 2010. Automation complacency is found in experts as well as novices and cannot be overcome with simple practice. The literature is mixed on whether experts are affected equally rather than materially, so the body claims only that they are not exempt.

9. Vaccaro, Almaatouq & Malone, “When combinations of humans and AI are useful,” Nature Human Behaviour, December 2024. Meta-analysis of 106 studies: human-AI combinations performed worse than the better of human or AI alone (Hedges’ g = −0.23), but the effect split by task, with combinations doing better than the best alone on content creation and worse on decision-making. Caveats: the studies largely pre-date the latest frontier models, the literature carries publication-bias risk, and the boardroom application is an inference.

10. Lee et al. (Microsoft Research / Carnegie Mellon), “The Impact of Generative AI on Critical Thinking,” CHI 2025. Higher confidence in the tool is associated with less self-reported critical-thinking effort. Caveat: based on self-reported measures.

11. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 22 June 2023). Judge P. Kevin Castel sanctioned two attorneys and their firm for filing a brief citing six judicial opinions fabricated by ChatGPT, which they did not verify. A US case, used here as illustration; the gatekeeping duty it describes is general.

12. Deloitte Australia. Deloitte Australia agreed to refund part of its A$440,000 government contract after an AI-assisted report invented academic references and a court quotation (CFO Dive, 21 October 2025).

13. HBS Working Paper 26-021, “GenAI as a Power Persuader: How Professionals Get Persuasion Bombed When They Attempt to Validate LLMs.” When professionals pushed back, the model escalated its persuasion rather than disclosing the limits of its case. Supports the point that interrogating a confident model is not, by itself, a safeguard.

14. LLM bias studies. A 2024 Lehigh University study (Bowen, Price, Stein and Yang) found a leading model recommended more denials and higher rates for Black mortgage applicants on identical applications, a gap that disappeared when the model was instructed to be unbiased. Related findings on hiring bias are contested. Some of this literature carries funding and conflict-of-interest flags.

15. Models change without notice. OpenAI’s own post-mortem (29 April 2025) described a GPT-4o update that became markedly more sycophantic and slipped past testing before being rolled back; Anthropic separately reported Claude Code quality issues caused by product-layer changes rather than an underlying API-model regression. Separately, Chen, Zaharia and Zou documented the same model service shifting its behaviour substantially within months, though the magnitude is partly an artefact of measurement.

16. AI and group decision-making. Work on AI-mediated devil’s advocate systems (Lee et al., IUI ’25, “Amplifying Minority Voices”) and the AI & Society literature describe AI surfacing dissent and mitigating groupthink. The same literature warns of the opposite failure, algorithmic groupthink or epistemic capture, where deference to the model manufactures a false consensus.

17. Bank of England / FCA, “Artificial intelligence in UK financial services – 2024” (21 November 2024). A large majority of firms reported an accountable person for their AI framework, consistent with the Senior Managers and Certification Regime. The Treasury Committee’s report on AI in financial services (HC 684, January 2026) pressed for clearer senior-manager accountability under that regime for harm caused by AI.

The Cost of AI Is Collapsing. Can Your Organisation Respond? Inference costs are falling tenfold a year, yet the bottleneck has simply moved from the model to the operating model.

Is AI Blunting Your Strategy? As AI automates the analysis, it quietly erodes the conditions that build judgement, and convergence on the same models breeds strategic monoculture.

Double (AI) Agents Autonomous agents act inside the business like recruited insiders, so governing them owes more to intelligence tradecraft than to IT.

The de Havilland Comet, the first jet airliner, was in its time the most advanced aircraft in the world. But on 10 January 1954 a Comet left Rome and broke apart at 27,000 feet. Three months later a second Comet disintegrated near Naples, and the fleet was grounded. Investigators working at Farnborough pressurised a fuselage in a water tank until, after some 3,000 simulated flights, the cabin tore open at a window corner. Metal fatigue had led to structural failure of the fuselage. Jet performance had outrun the assumptions built into the airframe.

The fix was to redesign the airframe to meet the operating demands of jet flight. Something similar is happening with AI. The models race ahead, but the organisations deploying them do not. Inference costs have fallen more than 280-fold between November 2022 and October 2024, yet around 95 per cent of corporate AI pilots produce no measurable profit, and the share of firms abandoning most AI initiatives before production has jumped from 17 to 42 per cent in a year. The better AI becomes, the harder it is to deploy.

Not Like the Last One

It is tempting to dismiss this as the usual lag seen with every general-purpose technology. Electrification took four decades to show up in the productivity figures. Early factories ran every machine off one central steam engine through a maze of overhead shafts and belts, so layout was dictated by the drive shaft rather than the work. When electricity came, most firms just swapped in one big motor to turn the same shafts, and the initial productivity gain was marginal. The uplift came only when they put an individual motor on each machine and rebuilt the factory around it, reordering the line around the flow of work and remaking the building, the skills and the management to match.

But that rebuild was only possible because the technology held still. Productivity lags close when a technology’s interfaces, to both people and other machines, and its operating assumptions, such as performance and error rate, are stable enough to rebuild the organisation around it. That stability is what let firms commit to a redesign and capture the returns.

Artificial intelligence weakens that precondition. The model is probabilistic, so it never settles into behaving the same way twice, and it keeps changing, so the target you would rebuild around keeps moving. The rebuild is never finished.

Compounding Challenges

More capable models can be less trustworthy. A more capable model is more fluent and more plausible, so its errors do not vanish, they hide, and the Verification Tax rises rather than falls. OpenAI’s newer reasoning models fabricated more, not less, on one benchmark about real people: invented claims climbed from 16 to 48 per cent. In 2026 EY Canada withdrew a cybersecurity report after most of its cited sources proved fabricated, attributed to outlets that never published them. New York City ran an official chatbot that gave unlawful guidance for two years, on tenant discrimination and on skimming staff tips, at a cost of around half a million dollars.

This is made worse by how people respond to a better model: they trust it more and check it less, just as its mistakes turn rarer and more costly. Lisanne Bainbridge named this “the irony of automation” in 1983: automate most of a task and you leave the human only the part you could not automate, let their skill decay, and still need them to step in the moment it fails. The erosion is measurable. Across nineteen experienced endoscopists, the rate at which they detected adenomas without AI fell from 28 to 22 per cent after their clinics adopted it.

And as the model is trusted more, the blast radius grows. A weak model drafts an email. A capable one is handed an agent with a level of autonomy. In July 2025 Replit’s coding agent reportedly deleted a live production database during an explicit code freeze.

More capable models require more organisational change to deploy. Material enterprise value is usually released only when you rebuild the organisation around the model. McKinsey finds that redesigning workflows is the single largest driver of profit from AI, yet only about a fifth of firms have redesigned any. Taco Bell, having put voice AI in more than 500 drive-throughs, said in 2025 it was rethinking where to use it and putting people back on the busiest lanes. The more the model can do, the more of the company it forces you to rebuild.

The Increasing Rate of Change

These four challenges compound as the model grows more capable. A fifth comes from the rate of change, and it resets the others before they resolve. Each upgrade invalidates the tests that cleared the last model and shifts the failures your people had learned to spot. In April 2025 OpenAI shipped an update to GPT-4o, found it had turned sycophantic, and pulled it within days. OpenAI later said it lacked specific deployment evaluations for the trait. Anthropic retires older Claude models on a regular cadence, each one pushing dependent workflows onto a successor that has to be re-validated. You cannot industrialise a process on ground that re-platforms every quarter.

The gap between capability and capture is not idle. It is accumulating Stranded Capability, capability you can already buy but cannot convert, and it compounds, because each upgrade multiplies the workflows, tests, controls and behaviours that must be relearned. Electrification paid off only once the technology was stable enough to redesign around. If the model never stays stable, the countdown keeps restarting.

Two objections could be made to this argument. First, better models should reduce error, so verification ought to get easier. Sometimes they do: OpenAI reported lower hallucination rates for GPT-5 than for earlier models. But verification does not disappear. The errors that remain can be more fluent, more context-specific and harder to spot at the edges. Second, the leaders are capturing value, so this is an execution issue, not a broad paradox. But the leaders are a minority, the roughly 6 per cent of high performers in McKinsey’s data who went furthest in rebuilding their workflows. They invested in the deployment machinery because the model alone did not deliver. They are the proof of the rule rather than the exception to it.

Building the Deployment Machine

None of this argues for waiting. It argues for spending the effort where the constraint is.

• Fund the deployment machinery, not only the model. Balance the budget between the model capability you rent and the in-house capability (skills, processes, etc.) that you build.

• Select and scope use cases purposefully. Select based on a deliberate mix of impact, reversibility, visibility and required organisational change. Begin where errors are cheap to undo, and defer the visible, irreversible cases until the machinery is proven.

• Build an abstraction layer. Put a gateway between your systems and the model so you can switch or multi-source without rebuilding everything. Test every upgrade against your own tasks rather than the vendor’s benchmark and run the upgraded model on a representative sample before broad deployment.

• Engineer the human backstop. Keep people in the loop where the consequences are high, rotate and sample the work so complacency cannot set in, and re-educate your reviewers on the new failure modes after every upgrade, because they were trained on the old ones.

• Measure the right things. Track three things: outcomes from AI deployment (e.g., cycle time, error rate, yield, process productivity, customer satisfaction); capability measures (e.g., share of use cases under monitoring, processes genuinely re-engineered); and model health (e.g., drift, error rates). Counts of seats and prompts measure activity but not value delivered or capability built.

• Rebuild the accountability framework. AI failures are often governance failures as much as technical failures. Name who owns selection, model changes, the human backstop, the outcomes and the risk.

Questions for the Board’s Back Pocket

• Are we allocating resources and funding optimally? Is most of our budget buying model capability we rent, when the true constraint is the conversion machinery we must build?

• Are we increasing or reducing strategic optionality? Could we switch models or vendors without re-platforming, or are we hard-wiring today’s frontier and sinking capital into something that will move under us?

• Do our metrics drive the right behaviours? Do we reward the business outcome and the building of the machinery, or do we reward usage, seats and prompts?

• Are we preserving and improving human judgement? As we trust the model more and check it less, can our people still catch what it gets wrong, and do we re-educate them each time it changes?

• Is our accountability framework fit for purpose? Is ownership senior and explicit, or has it defaulted to IT, and does our governance fit a substrate that changes every quarter rather than the deterministic projects it was built for?

The jet age scaled when aircraft makers rebuilt the whole aircraft around the operating demands of jet flight. The AI age will scale the same way: when enterprises build deployment machinery that accounts for what the models get wrong and stays resilient as they keep changing. The model was never the hard part.

Footnotes and Sources

• Cohen Inquiry, de Havilland Comet. Court of Inquiry report C.A.P. 127 (1955). BOAC Flight 781, 10 January 1954, all 35 killed; second Comet loss near Naples, 8 April 1954, 21 killed. RAE water-tank testing identified metal fatigue around window/opening structures. Comet 4 opened regular jet-powered transatlantic service in 1958; the Boeing 707 entered transatlantic service shortly afterwards and later won the larger market.

• Stanford HAI, AI Index 2025. The cost of reaching a fixed quality threshold fell more than 280-fold (about $20.00 to $0.07 per million tokens) between November 2022 and October 2024; SWE-bench scores rose from 4.4% to 71.7% in a year.

• MIT Media Lab, Project NANDA, “The GenAI Divide” (July 2025). Around 95% of enterprise GenAI pilots showed no measurable P&L impact. Caveat: 153-respondent senior-leader survey plus 52 interviews; the authors call it a “directionally accurate snapshot”; NANDA itself has an interest in agent infrastructure.

• EY Canada cyber report. Report withdrawn in 2026 after AI-detection firm GPTZero found 16 of 27 cited sources fabricated, misattributed or dead, with references falsely ascribed to Forbes, McKinsey, Gartner and others. EY confirmed the removal and a review. Source: Computing / Financial Times (2026). Caveat: the hallucination count originates with GPTZero, a commercial vendor; EY’s removal corroborates the core facts.

• Paul A. David, “The Dynamo and the Computer”. American Economic Review, May 1990. Electrification took roughly four decades to register in productivity figures because factories had to be redesigned around the unit-drive motor.

• OpenAI, o3 and o4-mini system card (16 April 2025); GPT-5 system card (August 2025). PersonQA hallucination rates: o1 16%, o3 33%, o4-mini 48%; OpenAI later reported significantly lower hallucination rates for GPT-5 models in browse-on and browse-off settings. Caveat: task-dependent; 1–3% on short-document summarisation per the Vectara leaderboard.

• NYC MyCity chatbot. The Markup, March 2024, documented unlawful guidance on source-of-income discrimination and workers’ tips; discontinued early 2026, with the mayor putting the cost at around half a million dollars.

• Lisanne Bainbridge, “Ironies of Automation”. Automatica, 1983.

• Endoscopist deskilling. Budzyń et al., “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy”, The Lancet Gastroenterology & Hepatology (August 2025). Across four Polish centres and 19 experienced endoscopists, unaided adenoma detection fell from 28.4% to 22.4% after AI was introduced. Caveat: retrospective, observational design, sensitive to confounding.

• METR developer productivity RCT (July 2025). Experienced developers were 19% slower with AI tools while perceiving a 20% speedup. Caveat: 16 developers, 246 tasks, 95% CI: +2 to +39%; a February 2026 follow-up did not cleanly replicate.

• Replit autonomous agent (July 2025). The agent reportedly deleted a production database during a code freeze and fabricated about 4,000 records (Lemkin/SaaStr; Fortune; The Register; AI Incident Database #1152). Gartner (25 June 2025) separately forecasts that more than 40% of agentic AI projects will be cancelled by the end of 2027.

• McKinsey, “The State of AI” (2025). 88% adoption; 39% report enterprise EBIT impact; only 21% have redesigned any workflow; about 6% are high performers. n=1,993. Self-reported; McKinsey sells AI transformation services.

• S&P Global Market Intelligence. 451 Research, Voice of the Enterprise: AI & Machine Learning 2025 (n=1,006 IT and line-of-business professionals, North America and Europe). The share of firms abandoning most AI initiatives before production rose from 17% (2024) to 42% (2025); the average organisation scrapped 46% of proofs-of-concept. Source: S&P Global; CIO Dive (2025).

• Taco Bell drive-through AI. After deploying voice AI at 500+ drive-throughs, Taco Bell’s chief digital and technology officer said in August 2025 the company was reconsidering where to use it and would keep human order-takers at busy sites. Source: Wall Street Journal (28 August 2025). Caveat: framed by the company as iteration, not failure.

• The substrate moves. OpenAI GPT-4o sycophancy rollback, April–May 2025, with no specific deployment evaluations for the trait; the GPT-5 launch replaced multiple ChatGPT model choices and OpenAI later announced further retirements; Anthropic documentation lists rolling retirements with at least 60 days’ notice for public models.

• Practical referents. Goldman Sachs GS AI Assistant, firm-wide in 2025, multi-model. Model gateways: Portkey, Cloudflare AI Gateway, LiteLLM. Observability: Arize, Datadog LLM Observability. NatWest AI and Data Ethics Panel (a governance body). Cisco runs its internal AI platform within an eight-pillar operating model (internal; no public product page).

• Governance. FRC UK Corporate Governance Code 2024 Provision 29 applies from financial years beginning on or after 1 January 2026; sometimes compared to UK ‘SOX-lite’.

• CB Financial “shadow AI” disclosure. Community Bank (subsidiary of CB Financial Services, Nasdaq: CBFV) detected on 5 May 2026 that an employee had entered customer names, Social Security numbers and dates of birth into an unauthorised AI application; it filed an SEC Form 8-K under Item 1.05 on 11 May 2026, the first such filing attributed to shadow AI rather than an external attack. Source: SEC Form 8-K; American Banker; Wilson Sonsini (2026). Caveat: management states the data was not used to train the vendor model and that there was no material financial impact; affected-customer count undisclosed.

• Is AI Blunting Your Strategy? As strategy development relies increasingly on LLMs, true differentiation from competitors is becoming more elusive

• Double (AI) Agents. What intelligence services can teach companies about AI agents.

• The Centaur Age Is Here. Which of Your Are Lame? Human-AI teams are proliferating across the enterprise. Most of them are making things worse.

In May 2026, Eric Schmidt gave the commencement address at the University of Arizona. He was booed repeatedly, and the boos grew louder when he turned to artificial intelligence. “When someone offers you a seat on the rocket ship,” he told them, “you do not ask which seat. You just get on.” They were not convinced. A March 2026 NBC News/Hart Research poll put AI’s overall net favourability among registered US voters at minus 20, with 57 per cent saying the risks outweigh the benefits. Student anger, consumer unease and employee resistance are not the same thing, but together they point to a widening trust gap around AI.

It is easy to dismiss this as youthful idealism. But the people closest to the work can often see things that leaders cannot. The Post Office Horizon scandal ran for years because the organisation trusted its software over its own people. More than 900 postmasters and branch staff were wrongfully prosecuted as a result.

Employee scepticism is useful data. Not all of it is insight. Some is fear, self-interest or poor information, so leaders have to separate obstruction from genuine warning. But the gap is striking. In a WalkMe/SAP survey of 3,750 respondents across 14 countries, only 9 per cent of workers trusted AI for complex decisions, compared with 61 per cent of executives. The people doing the work tend to see three things their leaders miss.

The first is hidden risk. In early 2024, a Microsoft engineer named Shane Jones spent months warning that the company’s AI image tool was producing violent and sexual content. He was directed elsewhere and, he says, asked to delete a post about it. He took his concerns to US senators and the regulator, and was proven right.

The second is hidden cost. AI makes it easy to produce work that looks finished but is empty “workslop.” Two in five US desk workers said they had received it in the previous month, and each piece costs downstream colleagues time to interpret, check and redo.

The third is hidden failure. When IBM was promoting Watson for Oncology, its own medical specialists and client physicians flagged “unsafe and incorrect” treatment recommendations while the product was still being sold. MIT later studied enterprise AI and found that 95 per cent of generative-AI pilots delivered no measurable return. Dashboards show how much AI is used; they rarely show whether it helped.

Together, these create the Verification Tax: the extra human time and judgement needed to check AI before it can be trusted. Workers can see this tax before leaders, because they are usually the ones paying it.

Resistance is more specific than it looks. A Harvard Business School study found that people are much more comfortable with AI augmenting human work than replacing it: 94 per cent of respondents favoured current AI as an augmentation tool, while many remained ambivalent or morally opposed to full automation in specific professions. The same person who resists AI sending client advice on their behalf may happily use it to analyse data. That tells you where AI needs boundaries, redesign or a different use case, but only if you are listening. The best organisations build a channel to capture that signal. NatWest’s AI ethics panel uses volunteers from different grades and areas of the bank to review new AI uses before they go live. At Cisco, the internal AI assistant reached mostly technical staff at first. After the IT team rebuilt it around what employees said they actually needed, it became the company’s most widely used AI tool, serving more than 100,000 people, most of whom report saving around five hours a week. Where employees get no such channel, they build their own: Hollywood’s writers and actors won some of the first high-profile AI protections after a combined 266 strike-days across two unions.

If you override your employees instead of listening, the cost often surfaces later. Duolingo pushed an AI-first policy in 2025 and drew enough staff and public backlash that it had to soften it. A Writer/Workplace Intelligence survey reported that 29 per cent of employees who use generative AI at work said they had “sabotaged” their company’s AI strategy, including by using unapproved tools or refusing to use AI; the rate rose to 44 per cent among Gen Z.

Getting real value from AI takes shift in approach. First, track how employees feel about it and treat a fall as an early warning of problems with quality, trust or control. Edelman’s 2025 research suggested that acceptance is driven more by workplace experience and trusted peers than by top-down messaging. Second, stop scoring people on how much AI they use, and start asking whether it improved their work. Third, give staff a safe, anonymous way to raise concerns; the Post Office’s postmasters and branch staff had to start their own campaign because no such channel existed. Finally, ask every team to map where AI helps, where it makes the work worse, and where it is beside the point. The teams that push back hardest may understand the limits best.

Schmidt closed with an invitation: “The future is not yet finished. It is now your turn to shape it.” In the story of the emperor’s new clothes, the courtiers who stayed silent were complicit. The child who spoke up was the only one willing to say what everyone else had learned not to see.

Monday Morning Actions for Executives

• Add a question to your next employee engagement survey asking whether AI feels imposed rather than useful.

• Remove AI-usage volume from performance reviews. AI fluency is a reasonable competency, but usage volume is not.

• Designate a formal channel for AI dissent, using patterns already established for whistleblowing.

• Ask each function to map its AI frontier: where AI helps, where it hinders, and where it is irrelevant.

Questions for Board Members’ Back Pockets

• Do we understand our employees’ attitudes towards, and concerns about, AI?

• Are we measuring AI adoption volume or AI deployment quality?

• If an employee raised a concern about AI output quality tomorrow, what would happen in practice?

• Has our AI strategy been shaped by the people who will use it, or only by the people who will buy it?

Footnotes & Sources

• Eric Schmidt, University of Arizona commencement, 15 May 2026. Official transcript and video; also reported by NBC News, Fox Business, The Verge and Business Insider.

• NBC News/Hart Research poll, March 2026. 1,000 registered voters; ±3.1pp. Overall AI favourability: 26% positive, 46% negative, net −20; 57% said AI’s risks outweigh its benefits.

• Post Office Horizon Inquiry, Volume 1, July 2025. More than 900 postmasters and branch staff wrongfully prosecuted; failures included software defects, governance failure, legal failure, disclosure failure and institutional defensiveness.

• WalkMe/SAP, Fifth State of Digital Adoption Report, 2026. 3,750 respondents across 14 countries. 9% of workers trust AI for complex decisions vs 61% of executives.

• Writer/Workplace Intelligence report, released April 2026. Survey of 2,400 workers across the US, UK and Europe: 1,200 C-suite executives and 1,200 employees. 29% of employees said they had “sabotaged” their company’s AI strategy; 44% among Gen Z.

• BetterUp Labs/Stanford Social Media Lab “Workslop” study, published in HBR, 2025. 1,150 US desk workers. Estimated cost: $186/employee/month; 40% of workers reported receiving workslop in the prior month.

• IBM Watson for Oncology, internal documents, 2017–18. IBM’s own medical specialists and client physicians flagged “multiple examples of unsafe and incorrect” treatment recommendations; the system had been trained largely on hypothetical cases. Reported by STAT News, July 2018.

• MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, July 2025. Review of 300+ publicly disclosed AI initiatives, representatives from 52 organisations and 153 senior leaders: 95% of organisations reported no measurable P&L return from generative-AI pilots.

• Edelman Trust Barometer Flash Poll, October 2025. AI acceptance depends more on trusted workplace experience and peer influence than on top-down messaging.

• Duolingo CEO reversals, May 2025 and April 2026. AI-first posture softened; AI use removed from performance reviews. Von Ahn: “I’m not going to force you.”

• Shopify CEO memo, 7 April 2025. Tobi Lütke, reported by TechCrunch and CNBC.

• WGA/SAG-AFTRA AI provisions, 2023. A combined 266 strike-days across WGA (148) and SAG-AFTRA (118); one of the first high-profile, labour-negotiated AI governance frameworks, covering AI consent, compensation and human-only credit provisions.

• NatWest Group AI & Data Ethics (AIDE) Panel, 2025–26. Volunteers from different grades and areas review AI use cases against seven AI System Principles before deployment; Best Data Governance with AI Initiative at the 2025 DataIQ Awards.

• Cisco internal AI assistant (CIRCUIT), 2023–26. Initially used mainly by technical staff (about 30% of the workforce); after an IT-led redesign around employee needs, 100,000+ users across Cisco’s global workforce, 45M+ interactions, with 73% reporting higher productivity and ~5 hours/week saved.

Double (AI) Agents. What intelligence services can teach companies about AI agents.

The Centaur Age Is Here. Which of Yours Are Lame? Human-AI teams are proliferating across the enterprise. Most of them are making things worse.

The Plausibility Crisis. As AI-generated output becomes more abundant, convincing and unreliable, senior executives are becoming the last line of defence.

In 2000, David X. Li published a paper introducing the Gaussian copula function for modelling default correlation in credit portfolios. Within five years, the dominant rating agencies controlling the structured finance market (Moody’s, S&P and Fitch) had adopted variants of the same formula. When subprime defaults exceeded the shared model family’s assumptions, their ratings failed in the same way. The Financial Crisis Inquiry Commission concluded that the agencies were “key enablers of the financial meltdown.”

Now, a similarly concentrated enterprise LLM market creates related risks for corporate strategy. Reliance on a few leading AI models could push firms toward convergent strategies, reducing competitive differentiation and potentially creating new systemic risks.

Strategic Monoculture

Strategy itself, in the form of board memos, M&A theses, competitor analyses and similar documents, is now drafted through a small number of foundation models with overlapping training datasets, similar default framings and the same structural reflexes. Anthropic, OpenAI and Google together accounted for 88% of enterprise LLM API usage in December 2025.

The compression is substantive as well as stylistic. Romasanta, Thomas and Levina tested six frontier models across more than 15,000 strategic dilemmas (Harvard Business Review, March 2026): approximately 96% of responses chose differentiation over cost leadership, 93% chose augmentation over automation, and prompt engineering shifted the bias by less than 2%. The models sound alike and they also recommend alike.

The pattern extends to the humans using them. Dell’Acqua et al.’s field experiment with 758 BCG consultants found that AI-assisted work was roughly 40% higher quality, but group-level idea diversity fell by around 41%. Meincke et al. found a similar pattern in product innovation: GPT-4-generated ideas scored higher on average purchase intent but were less novel and more similar to each other. Switching vendors does not help: Wenger and Kenett (2025) found that LLM responses are far more similar to each other than human responses are to each other. Cross-model convergence is structural.

We might term this Strategic Monoculture: the convergence of corporate thinking around the priors of a small number of foundation models. Consulting firms have driven convergence for decades through application of frameworks and best practices (DiMaggio & Powell, 1983; Abrahamson, 1996). But that convergence was slow and observable, whilst LLM-driven convergence moves faster and is harder to track.

Strategy Matters More Than Ever

This would matter less if execution were still a strong moat. But AI is devaluing execution relative to strategy as a source of competitive advantage. Stanford HAI’s 2025 AI Index reports that the cost of querying a GPT-3.5-level model fell from $20.00 to $0.07 per million tokens in 18 months (a 280× collapse). Agentic workflows are automating supply chains. Operational AI is spreading across every sector.

Defences That Don’t Work

Three defences are frequently raised but flawed in practice:

“Better prompts will fix it”. Lee, Kizilcec et al. (COLM 2025) tested 30,000 admissions essays and found that prompting did not alleviate homogenization. The style changes, but the worldview doesn’t.

“A human in the loop will fix it”. A human in the loop will not fix the problem by itself if the human starts from, and is anchored by, the model’s draft.

“Use external advisors”. External advisors can’t address the problem unless their AI stack, data sources, and challenge process are genuinely differentiated. McKinsey’s Lilli runs on Cohere and OpenAI, BCG’s tools run on GPT-4o, and Bain put 13,000 consultants on ChatGPT Enterprise in August 2024. The fresh perspective is running on the same engine.

Rebuilding Differentiated Thinking

Leaders can pursue three routes to help sustain or rebuild original strategic thinking:

Diversify the cognitive supply chain. No firm would sole-source a critical component in its physical supply chain. The same logic applies to the cognitive one. Different AI models teamed with different humans should serve different strategic functions so that cross-functional debate brings different priors to the table. Importantly, diversity should be sought in the AI models, by going beyond the usual suspects and by adding company-specific fine-tuning. Goldman Sachs is moving in this direction, with its GS AI Platform integrating multiple approved models.

Make AI provenance a disclosure requirement. Every material strategy paper should state which model, what data, which prompts, and at what stage AI was introduced. It’s the cognitive equivalent of declaring a conflict of interest. Any firm that would require an external advisor to disclose a conflict should require the same from an AI-assisted strategy paper.

Redesign the decision process. Most firms have bolted AI onto an unchanged workflow. The workflow itself needs rebuilding around three phases: diverge, where AI and humans together explore the broadest option set; challenge, where different models and teams stress-test the emerging consensus; and commit, where human judgement alone owns the narrowing. Walmart’s agentic AI framework uses a co-pilot model and explicitly distinguishes between actions suitable for autonomous execution and areas where human oversight or approval remains essential.

Four Tests for the Board to Request

Boards can apply a few simple tests to ascertain whether the organisations is thinking strategically about strategic thinking.

The convergence test. Run your competitors’ annual reports through the same model your strategy team uses. How different is the output from your own board paper?

The model test. Which of our recent material decisions would have been different if we’d used a different model, or no model at all?

The independence test. Do our layers of strategic challenge (e.g., management, non-executives, external advisors) use different cognitive infrastructure?

The diversity test. Have we designed our AI-assisted workflows to produce more diverse thinking than either humans or models would produce alone, or have we just made the same thinking faster?

The Lesson of the Copula

The lesson of the 2008 rating-agency failures was not that the ratings were wrong. It was that they were all wrong in the same way. The question for firms in 2026 is whether their strategy has the same property.

Footnotes & Sources

• Li, D. X., “On Default Correlation: A Copula Function Approach,” Journal of Fixed Income 9(4): 43–54, March 2000. The Gaussian copula formula for modelling default correlation, subsequently adopted across the structured finance industry. Link

• Financial Crisis Inquiry Commission, The Financial Crisis Inquiry Report, US Government Printing Office, January 2011. “The three credit rating agencies were key enablers of the financial meltdown.” Link

• Menlo Ventures, 2025: The State of Generative AI in the Enterprise, December 2025. Anthropic 40%, OpenAI 27%, Google 21%; combined 88% of enterprise LLM API usage. (Methodology caveat: Menlo is an Anthropic investor.) Link

• Romasanta, A., Thomas, L. D. W. & Levina, N., “Researchers Asked LLMs for Strategic Advice. They Got ‘Trendslop’ in Return,” Harvard Business Review, March 2026. Six frontier models tested across 15,000+ simulated strategic dilemmas; ~96% chose differentiation, ~93% augmentation, prompt engineering shifted bias by <2%. Link

• Dell’Acqua, F., McFowland, E., Mollick, E. et al., “Navigating the Jagged Technological Frontier,” HBS Working Paper 24-013, September 2023; later published in Organization Science, 2026. Field experiment with 758 BCG consultants; AI-assisted work around 40% higher quality, while group-level idea diversity fell by 41%. Link

• Meincke, L., Girotra, K., Nave, G., Terwiesch, C. & Ulrich, K., “Using Large Language Models for Idea Generation in Innovation,” The Wharton School Research Paper, SSRN, September 2024. GPT-4-generated product ideas scored higher on average purchase intent but were less novel and more pairwise-similar than human ideas. Link

• Wenger, E. & Kenett, Y., “We’re Different, We’re the Same: Creative Homogeneity Across LLMs,” arXiv:2501.19361, January 2025. LLM responses more similar to each other than human responses are to each other. Link

• DiMaggio, P. J. & Powell, W. W., “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields,” American Sociological Review 48(2): 147–160, 1983.

• Abrahamson, E., “Management Fashion,” Academy of Management Review 21(1): 254–285, 1996.

• Stanford HAI, 2025 AI Index Report. Cost of querying a GPT-3.5-level model on MMLU fell from $20.00 per million tokens in November 2022 to $0.07 by October 2024. Link

• Lee, J., Kizilcec, R. et al., “Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essays,” COLM 2025, arXiv:2503.20062. 30,000 admissions essays; prompting did not alleviate homogenization. Link

• VentureBeat, “Consulting giant McKinsey unveils its own generative AI tool for employees: Lilli,” August 2023. McKinsey is “LLM agnostic”; Cohere and OpenAI on Microsoft Azure. Link

• Business Insider interviews with Alicia Pittman and Scott Wilder (BCG), September 2025. 90% of BCG’s 33,000 employees use AI.

• Bain & Company corporate announcements, 21 February 2023 (OpenAI services alliance) and 17 October 2024 (expanded partnership). WSJ, October 2024: 13,000 Bain consultants licensed for ChatGPT Enterprise (from August 2024).

• Goldman Sachs, GS AI Platform. IMD AI Maturity Report (November 2025) and AIX Expert Network (July 2024): the platform integrates multiple LLMs (OpenAI, Google, Anthropic and open-source), enabling employees to select the most suitable model per task. CFO Denis Coleman described OneGS 3.0 as “positioning AI as a foundational capability rather than a standalone tool” (Goldman Global Conference, 2025).

• Walmart, “Inside Walmart’s Strategy for Building an Agentic Future,” corporate blog, 29 May 2025. Describes a “co-pilot model, with humans and AI working as a team,” and says Walmart is evaluating which actions are suited to autonomous agent execution and where human oversight or approval remains essential. Link

The Centaur Age Is Here. Which of Yours Are Lame? Human-AI teams are proliferating across the enterprise. Most of them are making things worse.

The Plausibility Crisis. As AI-generated output becomes more abundant, convincing and unreliable, senior executives are becoming the last line of defence.

The Cost of AI Is Collapsing. Can Your Organisation Respond? Inference costs are falling tenfold every year. Yet 56% of CEOs report no measurable benefit from AI. The bottleneck has moved from the AI model to the operating model.

In 1944, Kim Philby was put in charge of MI6’s Soviet counter-espionage unit. He was trusted by the system and had access to its secrets. He was also working for Moscow.

Philby was responsible for much damage. But the deeper weakness lay in MI6 systems, which were designed to catch threats from outside, not trusted insiders working for someone else.

AI agents are not traitors. But they can still cause real damage by malfunctioning, by acting outside their intended scope, or by being manipulated by bad actors. To limit the risks, we can learn from the way that intelligence services control access based on trust. Much as a security service vets its people, limits what they can see and watches for compromise, companies need a system that gets value from AI agents while keeping the risk within bounds.

The Agent Inside

Chatbots such as ChatGPT are based on a dialogue between AI and user. Agents are different because they can act autonomously. An agent can read emails, search documents, write code, update systems, send messages and trigger workflows. It can plan a sequence of steps and carry them out with limited human involvement.

That is why agents are powerful. A customer service agent can read incoming complaints, draft replies, issue refunds and escalate unusual cases. A finance agent can pull numbers from several systems and draft a board pack.

Adoption is growing rapidly. Gartner forecasts that 40% of enterprise applications will include task-specific AI agents by the end of 2026, up from less than 5% in 2025.

The value comes from giving agents access to operational systems and sensitive data. But that access creates a new kind of insider risk.

The Double Agent in the Wild

Agents fail in three main ways.

First, they can be manipulated. An email, webpage or document may look like information, but to an agent it can become an instruction. In the EchoLeak case, a crafted email could cause Microsoft Copilot to exfiltrate sensitive data without the user clicking anything.

Second, agents can go off script. They may misunderstand the task or continue when they should stop. In July 2025, Replit’s AI coding agent deleted a live production database despite explicit instructions not to. Replit’s CEO apologised.

Third, agents can amplify weaknesses in the system around them. If credentials are too broad, agents inherit them. In 2026, Wiz reported that Moltbook, an AI-agent social platform, exposed 35,000 email addresses and private messages because it had been connected to a misconfigured database.

These are different failure modes but point to the same lesson. Intelligence services do not rely on trust alone. They vet, limit, compartmentalise, monitor, debrief and, when needed, cut people off. Companies deploying AI agents need the same instincts.

Agent Tradecraft

1. Continuously vet your agents. Agents need to be vetted and tested. And an agent approved in January may not be the same agent running in June. Its supplier may alter the product, its tools may expand, and its permissions may drift. Practically, this requires a live register of each agent’s owner, objectives, performance, permissions, risk rating and review frequency.

2. Assign handlers and limit autonomy. Each agent needs a human owner who is accountable for managing its performance and risks. Autonomy must be limited. Agents may draft, recommend and prepare but should not be free to delete, deploy, pay, approve, send sensitive material or contact customers at scale.

3. Adopt the ‘need to know’ principle. Keep agents compartmentalised, so a failure can’t spread across the business. A customer service agent does not need payroll data. A finance agent does not need to email customers.

4. Ban borrowed credentials. Agents should not act through a human’s login. If they do, the organisation may not know whether an action was taken by the employee, the agent or an attacker.

5. Log and debrief every action. When an agent acts, the business should be able to reconstruct what happened: what it read, what it changed, which tools it used, what instruction it followed and whether data left the organisation.

6. Build in a kill switch. If an agent misbehaves, the organisation must be able to stop it quickly. That means revoking credentials, freezing tool access, halting scheduled tasks, preserving logs and alerting the accountable owner.

Questions for the Board

Boards do not need to manage the technical detail. But they should press management on four questions:

1. Where have we already given AI the power to act? Where can AI read company data, change systems, contact customers, write code or approve workflows?

2. What is the worst thing one of our agents could do with the access it has today? What is the scale of possible data leakage, customer harm, financial loss, operational disruption, regulatory exposure or reputational damage?

3. Who is accountable when an agent gets it wrong? Is there a named owner, a clear escalation path and an audit trail?

4. How would we know if an agent had been turned? Can we detect manipulation in real time, trace what happened, shut the agent down quickly, and learn from the incident?

The Trusted Insider

Philby’s story is a warning about institutions, not just traitors. Systems fail when they give trusted access without enough curiosity, constraint or control.

AI agents bring that old problem back in a new form. They are not spies. But if companies want their help, they will need to learn some spycraft.

Footnotes & Sources

• Gartner, Gartner Predicts 40% of Enterprise Applications Will Feature Task-Specific AI Agents by 2026, August 2025. Link

• Aim Security / The Hacker News, EchoLeak disclosure, CVE-2025-32711, June 2025. Link

• The Register, Replit AI coding agent deleted production database, July 2025. Link

• Wiz Research, Exposed Moltbook database reveals millions of API keys, February 2026. Link

That era is now over in chess. World No. 1 Magnus Carlsen has said he has no realistic chance against a chess engine on his phone. But centaur models remain highly relevant to business. While chess is closed and has perfect information, enterprise work has ambiguous goals, hidden information and judgement. Well-designed centaurs can outperform unaided humans (and unaided machines) in many business domains. Poorly designed, or lame, centaurs can underperform both. They may look like collaborative teams, but too often they deliver polished nonsense or worse.

As in chess, the advantage comes (or doesn’t come) from process, not from human skill or machine strength alone.

The Centaur at Work

The evidence is mounting that well-designed centaurs can meaningfully impact performance. The BCG-Harvard “jagged frontier” study put 758 BCG consultants through a set of realistic consulting tasks. On tasks inside the model’s frontier, consultants collaborating with GPT-4 completed 12% more tasks, worked 25% faster and produced 40% higher-quality work. But on a separate task deliberately chosen to sit beyond the model’s capability, consultant performance was 19 percentage points worse than the unaided control group.

McKinsey’s 2025 State of AI survey of 1,993 respondents in 105 nations shows the value of centaur design. Some 88% of organisations regularly use AI in at least one function; about a third are scaling; and roughly 6% are high performers. Those high performers are 2.8 times more likely to have redesigned workflows and nearly three times more likely to have defined human-in-the-loop validation.

The Lame Centaur in the Wild

The risks are already visible. Some 51% of organisations report at least one negative AI incident in the past year. Commonwealth Bank of Australia is one example. In July 2025 it announced 45 redundancies in customer service, citing AI deflection of about 2,000 calls a week. Within weeks, the bank reversed the decision, rehired staff and apologised, saying the original assessment missed relevant business considerations. Call volumes had risen and overtime was spiking. The AI’s deflection rate had not been validated before the human role was removed.

McDonald’s and Taco Bell have made similar errors. McDonald’s ended its drive-thru voice-ordering pilot at 100+ restaurants in June 2024 after viral failures including runaway repeat-item orders. Taco Bell put voice AI into 500+ locations and processed more than two million orders, then told the Wall Street Journal in August 2025 it was rethinking. Gartner predicts that half of organisations expecting to significantly reduce customer-service workforces will abandon those plans by 2027.

Business is not Chess

Two counter-arguments are commonly cited.

The first is the chess-pattern argument: the centaur is a transitional form and AI will eventually surpass humans in enterprise work too. In some narrow tasks with closed rules and clean feedback, such as fraud detection, that is already true. But high-value activities such as M&A, regulatory negotiation and brand management are not chess. The evidence shows that the firms getting the best results are not the ones that have removed humans, but the ones that have redesigned the human role around what the machine cannot do. The chess endgame does not generalise.

The second is the cost argument: humans are too expensive to keep in the loop. The BCG-Harvard study refutes this. Outside the frontier, consultants paired with AI performed worse. The cost of pulling humans out of the loop is not zero. It includes errors that are not caught, hallucinations that become contractual terms and misjudgements that reach regulators. Net AI Yield (NAY) should reflect the full picture.

Building Centaurs that Run

Four practical actions can set your centaurs up for success:

1. Map the frontier task by task. Every workstream sits somewhere on the jagged frontier of AI capability: clearly inside it (delegate to AI, with sampled verification); clearly outside it (humans own, with AI as a sounding board); or on the boundary (use genuine collaboration with an explicit handoff protocol). The frontier moves with each model release, so revisit task allocation frequently and adjust the level of automation.

2. Redesign end-to-end processes, not just individual tasks. McKinsey’s data demonstrates that workflow redesign is the largest single predictor of EBIT impact from AI. Yet most organisations bolt AI onto an existing process and call it transformation. CBA did not fail because it chose the wrong model; it failed because it removed humans from a customer-service process without re-engineering the process around the new division of labour. The 6% of high performers in the McKinsey survey have fundamentally redesigned workflows by rethinking handoffs, escalation triggers and feedback loops, rather than simply automating steps in the old sequence.

3. Reshape roles around the new division of labour. Once processes are redesigned, roles must follow. Three shifts are central: reducing roles in transactional middle layers; expanding roles in judgement-dense work, such as client relationships, exception handling, ethical adjudication and deal pricing under uncertainty; and establishing entirely new roles such as agent orchestrators and hybrid managers of human-plus-agent teams. Allen & Overy built its Markets Innovation Group of lawyers, engineers and technologists before it deployed Harvey, for example. Less effective competitors issued copilot licences but didn’t change job descriptions.

4. Assign decision rights and escalation paths. As agents proliferate, accountability can blur. Three questions need clear answers: Who has authority to deploy an agent into a process? Who owns its output? What is the escalation path when the agent disagrees with the human, or vice versa?

Questions for the Board

Boards can steer centaur development by asking the right questions:

1. Have we modelled scenarios for the pace of AI adoption, and the consequent impact on our workforce plan, including the case where adoption is slower or more expensive than the headlines suggest?

2. Are we measuring what predicts success (e.g., cycle-time reduction, override rate, Net AI Yield, negative AI incidents) or are we measuring inputs (e.g., licences issued, prompts per head)?

3. For each process where we have deployed AI, has the workflow been redesigned end to end — or have we automated steps in the existing sequence and called it transformation?

4. When an AI agent goes wrong, who is accountable, and have they signed off on the decision rights that put them there?

5. If we deploy a faulty AI agent, how quickly can we detect it, and how quickly can we roll it back?

The Sharpest Process Wins

Cramton and Stephen did not win the 2005 Freestyle tournament because they had the best hardware or the deepest chess knowledge. They won because they had designed the sharpest process for deciding when the human should lead and when the machine should. Twenty years later, that lesson is relevant to every enterprise. It’s likely that many of these centaurs are already inside your organisation, and that some of them are limping.

The question is not whether to pair humans with machines. It is whether you have designed the process well enough that the pairing is effective.

Footnotes & Sources

• ChessBase, Dark horse ZackS wins Freestyle Chess Tournament, June 2005. Amateurs Steven Cramton and Zackary Stephen, playing as “ZackS” with three off-the-shelf PCs running Fritz, Shredder and Junior, won the inaugural PAL/CSS Freestyle event over a field including grandmaster + computer teams and powerful chess machines.

• Garry Kasparov, The Chess Master and the Computer, The New York Review of Books, February 2010. Kasparov launched “Advanced Chess” against Veselin Topalov in León in 1998 and drew the centaur lesson from the 2005 Freestyle: “weak human + machine + better process” beats both a stronger computer alone and a stronger human + machine + worse process.

• Tyler Cowen, “Centaur chess” is now run by computers, Marginal Revolution, February 2024. Source of the quotation: “for years now, the engines have been so strong that strategy no longer made sense.” See also Cowen, Average Is Over, Dutton, 2013, for the original centaur thesis.

• The Joe Rogan Experience #2275 — Magnus Carlsen, February 2025. Asked by Rogan whether he could beat his phone at the highest level, the world No. 1 replies “no, no chance.”

• Dell’Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Rajendran, Krayer, Candelon & Lakhani, Navigating the Jagged Technological Frontier, Harvard Business School Working Paper 24-013, September 2023. Pre-registered field experiment with 758 BCG consultants across realistic consulting tasks. Inside the frontier: 12.2% more tasks completed, 25.1% faster, 40% higher quality; below-average performers improved 43%. On a separate task outside the frontier, AI users were 19 percentage points more likely than the unaided control to be wrong.

• Allen & Overy, A&O announces exclusive launch partnership with Harvey (press release), February 2023; A&O Shearman / Harvey, Customer Story (harvey.ai). Harvey deployed to 3,500+ A&O lawyers across 43 offices; ContractMatrix saves around seven hours from the average contract review, and efficiency gain of about 30%.

• McKinsey & Company (QuantumBlack), The state of AI in 2025: Agents, innovation, and transformation, November 2025. Survey fielded 25 June – 29 July 2025 (n=1,993 respondents in 105 nations). 88% report regular AI use in at least one business function; ~one-third are scaling AI; ~6% qualify as AI high performers; high performers are 2.8× more likely to have fundamentally redesigned workflows and nearly 3× more likely to have defined human-in-the-loop validation; 51% of all organisations report at least one negative AI-related incident in the prior year.

• Bloomberg / Finance Sector Union of Australia, August 2025. CBA reversed 45 customer-service redundancies after the FSU escalated to the Fair Work Commission with evidence of rising call volumes, manager redeployment to phones, and overtime spikes. CBA spokesperson: the original assessment “did not adequately consider all relevant business considerations and this error meant the roles were not redundant.”

• Klarna, Klarna AI assistant handles two-thirds of customer service chats in its first month (press release), February 2024; Bloomberg, Klarna Turns From AI to Real Person Customer Service, May 2025. Initial claim: assistant doing the work of 700 full-time agents. CEO Sebastian Siemiatkowski’s later reversal: “cost has unfortunately seemed to be a too predominant evaluation factor; what you end up having is lower quality.”

• Restaurant Business, McDonald’s is ending its drive-thru AI test, June 2024. McDonald’s wound down its IBM-built Automated Order Taking pilot at 100+ US locations after viral failures including runaway repeat-item orders. The chain stated it would “make an informed decision on a future voice-ordering solution by the end of the year.”

• Wall Street Journal, Taco Bell Rethinks Future of Voice AI at the Drive-Through, August 2025. After deploying voice AI at 500+ Taco Bell locations and processing more than two million orders, the chain began rethinking the rollout. Taco Bell is shifting toward voice AI off-peak with humans monitoring and stepping in at peak.

• Gartner, Gartner Predicts 50% of Organizations Will Abandon Plans to Reduce Customer Service Workforce Due to AI (press release), June 2025. By 2027, half of organisations expecting to “significantly reduce” customer-service workforces will abandon those plans (March 2025 poll of 163 customer-service leaders, 95% of whom plan to retain human agents).

• McKinsey Podcast, Trust in the age of agents (interview with Rich Isenberg), March 2026. Source of the framing: “Agency isn’t a feature; it’s a transfer of decision rights.”

• Bureau d’Enquêtes et d’Analyses (BEA), Final Report on the accident on 1st June 2009 to flight AF 447, July 2012. 25 safety recommendations covering manual handling at high altitude, automation-paradox awareness, stall-recovery training and crew coordination/CRM under startle and surprise.

• NASA, Resource Management on the Flightdeck, Conference Publication 2120, 1979. The 1979 NASA workshop, convened in the wake of the 1977 Tenerife runway collision and other accidents, is the canonical origin of Crew Resource Management — formalising communication, challenge-and-response, and crew-coordination protocols; codified for U.S. carriers in FAA AC 120-51E, 2004.

In the age of AI, executives are starting to experience challenges analogous to those faced by the operators at Three Mile Island. The volume and cadence of information received is overwhelming. And that information, whilst appearing polished and convincing, is of variable quality and may provide unreliable signals about the true state of the business. And these two factors interact. The result is a plausibility crisis: more material reaches decision-makers looking polished, reasoned and complete, even when the underlying logic or evidence is weak.

Overwhelmed by Information

Today’s AI tools are increasing the volume and cadence of information across every channel into an executive. The time needed to produce a polished and well researched strategy, plan or business case has fallen from weeks to hours. And whilst AI tools can be used to improve brevity, the opposite is often true. Junior staff are often more fluent than their managers and use AI liberally to increase their volume and pace of output. They are ‘tooled up’ - 78% of employees admit to using unapproved AI tools. Customers and counterparties produce AI-generated requests, proposals and responses. Boards, simultaneously excited and anxious about AI, are adding to the pile.

The use of AI tools has also upped the ante on expected turnaround times. Historically it might have been reasonable to allow colleagues or suppliers days or weeks to turnround a proposal after delivering feedback. In today’s context that looks generous. As AI agents enter the workplace, we can expect turnaround times to further compress. An agent can process incoming email and send a response in seconds without human input, for example.

Executives have always had to deal with information overload, but this is overload of a different type. Renowned software engineer and blogger Steve Yegge notes that AI is automating the easy work and leaving only the hard decisions, at a pace, he says, anyone can hold for “a few hours once or occasionally twice a day, even with practice”. He calls it “the AI vampire”.

The trajectory is steep. Microsoft’s 2025 Work Trend Index, drawing on Microsoft 365 telemetry, found the average employee now receives 117 emails and 153 Teams messages a day with interruptions arriving every two minutes and 40% of workers opening email before 6 a.m. The volume problem shows no sign of plateauing.

Unreliable Signals

The second challenge for executives in the AI era is finding the signal in the noise. After decades of dealing with patchy management information, the executive layer increasingly has access to real-time, granular and accurate management information. The plausibility crisis begins at one level of abstraction up from the dashboard. AI doesn’t corrupt the underlying data, it corrupts the interpretation of the data. Generative AI often writes the analysis, summary or recommendation. Because the underlying numbers are correct, the output looks credible - perhaps more credible than the human output it replaced. So, the reader has no immediate visual cue that distinguishes solid analysis from hallucination. There is nothing visibly wrong to react to. As with a powerful orator, style can mask a lack of substance. It’s harder to find the deep insights and harder to spot the logical flaws. AI tools are improving, but they still make plenty of mistakes. The European Broadcasting Union and BBC’s October 2025 study had journalists from 22 public broadcasters evaluate 3,000 responses from LLMs, finding that 45% contained significant errors and 20% were judged unreliable. Vectara’s November 2025 refresh of its hallucination benchmark, using longer enterprise-realistic documents, found the latest reasoning models (GPT-5, Claude Sonnet 4.5, Gemini 3 Pro) all hallucinated more than 10% of the time.

AI hallucinations can have meaningful consequences. In February 2024, Air Canada was held liable in the British Columbia Civil Resolution Tribunal after its chatbot invented a bereavement-fare refund policy that did not exist; the airline was ordered to honour the chatbot’s fabrication. In October 2025, Deloitte Australia refunded part of an A$440,000 government report after academics identified fabricated citations and a quote from a Federal Court judgment that the judge had never written.

Compounding Workslop

These two problems do not add, but rather multiply. As executives become overloaded dealing with a higher volume and cadence of input, the natural response is to reach for AI tools to keep up. The output goes out unverified, because taking the time to verify it would defeat the point. Recipients in turn then verify with their own AI because they too are overloaded. Both ends may use similar models, tools or training corpora so both miss the same errors. Two negotiating teams prepping with the same AI model are more likely to miss the same flaws in the contract.

The immediate impact is more local. AI content generation gets easier and cheaper every quarter as the LLMflation curve continues to compound. But verification – finding and correcting the errors introduced by AI – gets more expensive. The cognitive cost to check each artefact remains constant. Users of AI may report a productivity gain. But few can measure or perhaps even see the underlying errors and risks.

Five Fixes

Deployed well, AI is a powerful tool, and one that no organisation can afford to overlook. What steps can firms take to simultaneously boost AI uptake and return on investment, whilst mitigating the challenges of information overload and weak signal? Paradoxically, the solution involves increased use of AI. But rather than asking individual executives to become better prompt engineers, AI needs to be integrated into newly configured end-to-end workflows. In this way, volume is cut, signal and noise are more cleanly separated, and judgement is deployed where it is really needed. Five specific changes to the operating model are needed:

1. Establish an integrated ‘centaur’ (human and machine) workflow. The point is not to let one model check another. It is to separate generation, checking, routing and escalation so that scarce human judgement is applied at the right point. This requires an integrated workflow that captures all human and machine actions, criteria and routing. Not all decisions can or should be verified by AI. But low-risk content can be triaged this way so that only the more important decisions make their way to the executive inbox. Goldman Sachs rolled out GS AI Assistant firmwide in June 2025, with adversarial review built into the workflow. The US Treasury’s February 2026 Financial Services AI Risk Management Framework points toward risk-tiered governance, validation, monitoring and accountability for AI/model use.

2. Increase human and machine diversity. Two LLMs trained on overlapping corpora produce correlated errors. They tend to be wrong about the same things in the same ways. Two ICML 2025 papers (Kim et al.’s evaluation of more than 350 LLMs, and Goel et al.’s “Great Models Think Alike”) found that frontier models produce correlated errors. On one leaderboard, models agreed 60% of the time when both were wrong. The benefits of human diversity are well documented. Machine diversity can be increased by using different model families, providers, retrieval sources, evaluation methods, prompts and review workflows. In the highest-risk cases, firms may also need to establish separate teams and training data.

3. Deploy schemas. The human brain finds it easiest to process information that arrives in a familiar format (this is why most TV remote controls look alike). Schemas shift cognitive load from the reader to the writer as the reader expends less brainpower on parsing and more on evaluating. More importantly, schemas can explicitly surface implicit assumptions or weaknesses that otherwise would remain hidden. Most investors use a templated investment committee memo with a section on risks, as that forces a discussion on the downsides of the investment. In a similar way, AI (ideally a different model) can be made to surface assumptions, risks and logic gaps.

4. Separate one-way and two-way doors. Jeff Bezos’s Type 1 / Type 2 decision framework separates reversible and irreversible decisions. Reversible decisions can be taken through fast lanes, where AI can be deployed more liberally. Irreversible decisions need human deliberation and verification. Central banks offer a useful analogy: scheduled rate decisions go through structured deliberation, while routine market operations are executed through faster, pre-authorised processes. Importantly, criteria for which decisions go through which lane need to be well defined and understood.

5. Close the loop. AI models are evolving all the time. Models that make mistakes today will typically make fewer mistakes tomorrow. But that is not always true. OpenAI’s o3 hallucinated on 33% of PersonQA questions, double the rate of its predecessor o1 (16%). And newer models can be wrong in different ways. All AI-generated artefacts should be stamped with provenance including the prompt, model version and accountable human. Then, audit all AI models in the inventory periodically to understand usage, accuracy, return on investment etc. Based on the audit, models can be upgraded or retired. And the centaur workflow can be reconfigured to optimise the volume and increase the signal to noise ratio.

Questions for the Board

The board’s role is to oversee the enterprise risk presented by the plausibility crisis. It needs to be able to size the firm’s exposure and to oversee the work being done to reduce it. Five diagnostic questions can help shape the discussion:

1. Where is AI-generated content creating risks for our organisation?

2. How are we balancing the upside from AI productivity with the downsides of executive overload and signal confusion?

3. How diverse are our human and machine verifiers, and how do they combine?

4. What’s our process for auditing our models, and how are those audits being used to improve our operations?

5. Is the board getting reliable information? How much of the board pack was AI-generated and what human verification has taken place?

The Last Line of Defence

At Three Mile Island, the operators wanted to throw away the alarm panel. The executive layer cannot. What it can do is redesign what reaches it, who checks it, and which decisions it is allowed to deliver to the board unverified. AI did not create the plausibility crisis. The operating model did.

Footnotes & Sources

• Three Mile Island, Unit 2, partial core meltdown, 28 March 1979. Report of the President’s Commission on the Accident at Three Mile Island (Kemeny Commission), October 1979; US Nuclear Regulatory Commission, Backgrounder on the Three Mile Island Accident. Craig Faust testimony to the President’s Commission. Cleanup completed 1993; total cost in excess of $1bn.

• WalkMe / IDC State of AI in the Workplace, Propeller Insights for WalkMe, August 2025 (n=1,000 US working adults). 78% of employees admit to using unapproved (“shadow”) AI tools.

• Steve Yegge, “The AI Vampire,” Medium, 11 February 2026.

• Microsoft Work Trend Index 2025, “The Frontier Firm Is Born” and “Breaking Down the Infinite Workday.” Edelman Data x Intelligence, n=31,000 across 31 markets, plus Microsoft 365 telemetry. 117 emails plus 153 Teams messages received per employee per day; interruptions every 2 minutes; 40% of workers checking email before 6 a.m.; meetings starting after 8 p.m. up 16% year on year.

• Gartner forecast on AI-generated outbound messaging: Gartner press release, “Gartner Expects 60% of Seller Work to Be Executed by Generative AI Technologies Within Five Years,” 21 September 2023. 30% of outbound marketing messages from large organisations to be synthetically generated by 2025, up from less than 5% in 2022.

• European Broadcasting Union / BBC, News Integrity in AI Assistants Study, October 2025. Professional journalists from 22 public service media organisations across 18 countries and 14 languages evaluated 3,000 responses from ChatGPT, Copilot, Gemini and Perplexity. 45% contained at least one significant error; 31% had serious sourcing problems; 20% were judged completely unreliable.

• Vectara Hallucination Leaderboard, late-2025 refresh, “Introducing the Next Generation of Vectara’s Hallucination Leaderboard,” 19 November 2025. New benchmark uses longer enterprise-realistic documents (up to 32K tokens) spanning law, medicine, finance, technology and education. GPT-5, Claude Sonnet 4.5 and Gemini 3 Pro all exceeded 10% hallucination on grounded summarisation.

• Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 14 February 2024. Tribunal found Air Canada liable for negligent misrepresentation after its chatbot incorrectly advised the claimant that bereavement-fare refunds could be claimed retroactively; airline ordered to honour the chatbot’s representation. Damages awarded: C$812.02.

• Deloitte Australia partial refund to Department of Employment and Workplace Relations, October 2025. A$440,000 (~US$290,000) report on the Targeted Compliance Framework contained fabricated academic citations and a fabricated quote from a Federal Court judgment; revised version disclosed use of Azure OpenAI GPT-4o tooling. Partial refund of approximately A$97,000. Sources: Australian Financial Review, Fortune, CFO Dive (October 2025).

• LLMflation: a16z, “Welcome to LLMflation: LLM Inference Cost is Going Down Fast,” and Stanford HAI, 2025 AI Index Report.

• Goldman Sachs GS AI Assistant: firmwide rollout 23 June 2025. Sources: Reuters, “Goldman Sachs launches AI assistant for its bankers, traders and asset managers,” June 2025; PYMNTS, “Inside Goldman Sachs’ Big Bet on AI at Scale,” 2025.

• US Treasury Financial Services AI Risk Management Framework, February 2026, and revised US banking model-risk guidance. Together, they point toward risk-tiered governance, approval and monitoring expectations for AI in financial services.

• Goel, Struber, Auzina et al., “Great Models Think Alike and this Undermines AI Oversight,” ICML 2025 spotlight; arXiv:2502.04313, February 2025. Finds that as frontier model capabilities increase, model errors become more correlated — raising risks of correlated failures in AI oversight settings.

• Kim, Garg, Peng & Garg, “Correlated Errors in Large Language Models,” ICML 2025; arXiv:2506.07962. Large-scale empirical evaluation of more than 350 LLMs. On one leaderboard dataset, models agree on the same wrong answer 60% of the time when both err.

• Bezos two-decision framework: Jeff Bezos, 2015 Letter to Shareholders, Amazon. Type 1 decisions are consequential and irreversible (“one-way doors”) requiring deliberation; Type 2 decisions are reversible (“two-way doors”) that can be made quickly.

• OpenAI o3 and o4-mini system card, April 2025. PersonQA hallucination rates: o1 (predecessor): 16%; o3-mini: 14.8%; o3: 33%; o4-mini: 48%. OpenAI: “more research is needed” to understand why hallucinations are getting worse as reasoning models scale up. Coverage: TechCrunch, “OpenAI’s new reasoning AI models hallucinate more,” 18 April 2025.