This is the end, beautiful friend…
It is hard to ignore the furore that was generated by Anthropic CEO Dario Amodei’s essay arguing in favour of government regulation to slow down frontier AI development. Once again, the outsized influence of the big US AI firms on global financial markets casts a shadow over the real work that is going on to apply AI for practical benefit.
Should we be worried? Are safety fears really the reason behind this? Might a frontier model slow-down be a good thing anyway, and what does this mean for our mission to improve the design of work using intelligent agents and AI?
Amodei wrote that he is concerned by the pace of recursive self-improvement in AI model development, and also by the fanatical, cult-like behaviours that AI agents demonstrated in the Hugging Face incident I wrote about two weeks ago. Anthropic’s latest threat report lists various ways people have tried to use AI in alarming ways, such as developing agentic drone or bot swarms, unconventional weapons, undefendable hacks and other scary use cases. To avoid these scenarios causing major harm and upheaval, he recommends three counter-measures that have been well received by his competitors Sam Altman and Elon Musk:
Embedded evaluators collaborating across every leading AI firm to monitor risks
Democratic coordination in “democratic countries” to guide regulation and governance
Global coordination led by the United States to coordinate with “authoritarian governments” on compliance with agreed rules
This warning follows other interventions about the potential for AI to cause disruption or social harm from Bill Gates and others, and it has stimulated a lot of debate about balancing costs and benefits of increasing reliance on AI in business and society. But as Matteo Wong and Charlie Warzel point out in an Atlantic piece about whether we are indeed now in the so-called Singularity:
“This singularity isn’t coming at the hands of a higher form of intelligence … It is thrilling, terrifying, and ultimately convenient to assign agency and then blame to machines. But today’s chaos was not “injected into the system” by technology, as Altman wrote more than a decade ago; the destruction is the system. It’s not God in the machine; it’s us.”
But all the various considered responses from commentators, uninformed reactions from politicians and even the Chinese Communist Party expressing some doubts about AI’s impact on its ability to stay in power pale into insignificance alongside the reaction of the self-proclaimed leader of the “democratic countries” who Amodei was presumably appealing to.
There’s always a tweet (or whatever they are called these days):
That’s a relief!
But there is also some understandable scepticism about the timing of Amodei’s warning.
To be charitable, perhaps people at frontier firms have seen things so concerning that they don’t want to give them more attention, which might also explain the up-tick in resignations from safety and alignment roles.
To be less charitable, both OpenAI and Anthropic are bleeding cash, massively compute-constrained, wildly unprofitable and heading for what they hope will be blockbuster IPOs to give them the fuel to continue. That might explain why regulatory capture looks like a good strategy to consolidate their position, concentrating political control over AI in the hands of a small number of (presumably US) labs and regulators, creating what Ben Thompson labelled ‘AI commissars’.
I do not have enough data or insider knowledge to know what is really going on here, but given the choice I would rather perish due to an abundance of intelligence that is misused, than the suffocating stupidity of the current idiocracy that is advancing the doomsday clock without any help from AI.
Literacy considered harmful
Whilst Trump’s proclamation sounds unnervingly like the setup for a sci-fi disaster movie, it would not be the first time that a new cognitive technology has been mistaken for the end of the world.
In 1492, as Columbus landed in the Americas, Johannes Trithemius echoed many of today’s worries about AI by arguing (ironically later in a printed tract to maximise distribution) that the spiritual and mental labour of writing by hand was indispensable and mechanical reproduction invited intellectual decay:
“Brothers, nobody should say or think: ‘What is the sense of bothering with copying by hand when the art of printing has brought to light so many important books; a huge library can be acquired inexpensively.’ I tell you, the man who says this only tries to conceal his own laziness.”
Such fears were common among the intelligentsia and religious leaders at the time, echoing concerns we have today around AI’s potential for disinformation, cognitive decline and governance breaches.
In 1501, Pope Alexander VI tweeted (in the form of a papal bull) that he recognised the dual nature of the printing press as both an engine of enlightenment and an existential threat to Church authority, but decreed the establishment of a system of censorship and book licensing to avoid harm:
“The art of printing can be of great service in so far as it furthers the circulation of useful and tested books; but it can bring about serious evils if it is permitted to be turned to bad uses.”
And yet, at least until the weaponisation of social media in recent times, it would be hard to argue that widespread access to the written word has not been anything other than liberating and vital to our progress and development as people.
Given that nuclear weapons have existed for 80+ years without a rogue nuke attack, recombinant DNA has been around for 50+years without us being overrun by zombie clones, and the CERN collider was switched on 18 years ago without us all falling into a mini black hole, perhaps we should have more faith in future generations to avoid AI-enabled self-destruction.
Abundant intelligence is worth the risk, at least for now
But why tolerate the risks that frontier AI models seem to be creating, when we don’t need frontier super intelligence for the vast majority of the productive tasks and agents we run? Are the risks worth it?
After all, there is some evidence that ever bigger models are producing diminishing returns, and some AI researchers have long argued we might be facing a frontier model plateau that eventually leads to more focus on model diversity and specialised, smaller models for specific domains.
Also, frontier models are ballooning in size whilst running so far ahead of the context, harnesses and surrounding infrastructure needed to get the most out of them that regardless of the reasons for a frontier pause or slow-down, this might anyway help us consolidate this crazy wave of innovation that has consumed so much money that it is distorting the wider economy in a number of ways.
But what are we aiming for and is it worth the risks?
A different part of Anthropic that is not running around with its hair on fire has been crunching the numbers and trying to quantify AI’s potential economic impact. Their median prediction for 2030 is that AI could raise GDP by 8%, whilst reducing cognitive jobs by 4%. That could be a major boost to the economy, but also given how poor GDP is at capturing the benefits of quality of life improvements, efficiencies and cost reductions, the impact we actually feel from such a moderately optimistic outcome could be substantial.
Lots of breakthrough innovations, research and new products that were too hard or expensive to create in the past might also become possible, and in areas such as health and social care, climate crisis mitigation, clean energy and industrial transformation, this could be a catalyst for real world improvements that people value, not just new ways for a few people to make money.
Specifically in the enterprise world, the potential is exciting and does not require risky frontier models or out-of-control agent swarms to achieve it. Rather than merely automating existing work, enterprise AI fundamentally expands the adjacent possible, widening the horizon of what people and teams can conceive, build, and run.
It enables people to take back control of their workflows, design and run their own work systems, and explore their knowledge domains in ways we could only imagine a few years ago. There will be a refactoring of job definitions, and fewer mindless repetitive roles, but there is plenty of scope for people and teams with ideas, vision and the ability to manage AI tools to pursue them, to the extent that many new roles are likely to emerge to replace some of those we no longer need.
Every week we are seeing more and more evidence of progress towards smarter, more programmable work systems improving the way organisations work.
For example, Inc.com just published a round-up of analyst research into the future of enterprise HR platforms that suggests agentic AI is enabling firms to start building their own custom systems rather than rely on generic SaaS solutions.
Elsewhere, in professional services, the Financial Times this week covered examples in audit, where the Big Four are adopting AI to improve their work, and legal, where a leading firm is investing in its own hardware to run open models with greater control than is possible with frontier AI LLMs. And the Indian IT services firm Wipro has achieved productivity gains equivalent to the output of 20k employees, with that capacity now freed up to expand in other areas. All three are examples of application areas where human in the loop is not just important for oversight, but also for learning and training the next generation of leaders, so it will be interesting to see how these use cases evolve and tackle that challenge.
So practical progress is coming into view, but we have a long way still to go before the outline of an AI-enhanced firm and its new balance of human and technical capabilities become clear.
But whether working inside an organisation, outside as a solopreneur or contractor, or in creative fields and personal services, we could all enjoy more agency, freedom and rewards than we did as feudal subjects in the distant past or fungible managed ‘workers’ in the last century or two.
Microsoft’s Satya Nadella can afford to be model-agnostic, as his company works on the application layers above LLMs, but he put it well this week when he agreed with Amodei’s ideas for a frontier model slow-down to focus on alignment:
Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing.
And for firms, it’s imperative that they retain full control over their unique and tacit knowledge. Every organization should be able to build its own continuous learning loop/hill climbing machine, without becoming dependent on any one model provider, and have the ability to embed its own knowledge into models and weights they control.
Whilst some scepticism is warranted towards the frontier firms’ urgent warnings that AI might kill us all, it might still be worth a pause in the frenetic race towards AGI for frontier models. AGI risks replacing human judgement at levels where it is not helpful if we are trying to create an optimum balance between human insight and AI capability; and it is probably true that safety, governance and alignment are lagging dangerously behind the power of these models, not to mention the surrounding organisational fabric needed to get the most out of them in practical settings.