Are The Robots Coming For Us?

I’m shamelessly ripping off Inkhaven and doing my own challenge, Egghaven. This is the first post.

What are the right questions to ask?

Many common arguments that advanced AI poses an existential risk include robots in some way. I know nothing about robotics. I suspect that many people who make x-risk arguments also know nothing about robotics, and often have a reasoning step along the lines of “and then the superintelligent AI system uses its unfathomable cognitive powers to produce huge numbers of robots with dramatically improved capabilities in an unprecedentedly short space of time”. There’s some chance this is a perfectly reasonable claim to make, but it’s the kind that makes me suspicious. Due to the prevalence of robotics as a vector for catastrophe, understanding the reality of the field is necessary in order to have any kind of rigorous model of the risks of developing AGI. I want to have such understanding, at least in its minimum viable form.

The common arguments I have in mind include¹:

• Concentration of power achieved through AI-enabled coups or an overwhelmingly dominant military force. Robots replace soldiers or officials who are needed to perpetuate the new authoritarian state. Relatively small numbers of robots might be required if they have the ability to occupy key positions. Alternatively, broader military might could be achieved through dramatic advances in robotics, brought about by superintelligent AI systems. The (extremely undesirable) values of the hegemony are locked in, as it is impossible for other groups to overcome the advantages afforded by the quantity and/or capability of their robots.
• Misaligned takeover whereby AI no longer needs humans to achieve its goals. Robots perform every physical task necessary for the AI’s aims (including potentially killing people). This might include maintaining factories, data centres and energy production. The robots are the ones who actually turn the people into paperclips.
• Gradual disempowerment in which robots remove humanity’s last competitive advantage: competence at physical labour. We have no remaining bargaining power with superintelligent AI systems and are essentially vassals (or worse). Akin to The Matrix, but worse – at least in that movie, the machines have some use for the humans; they can plug them in and extract energy from them. In reality, we might not even have that much value.
• Great power war fought with autonomous weapons that are extremely effective. Essentially a MAD scenario. The war is conducted primarily with UAS, of various shapes and sizes; these include tiny drones that are capable of poisoning people or carrying explosive charges.

All of these rely on something akin to the industrial explosion as described in AI 2027 and further discussed by Daniel Kokotajlo elsewhere. Is such an explosion feasible, and how might it play out? There are multiple relevant aspects of robotics:

• How capable are the robots, and can they generalise to new tasks?
• How reliable are the robots in deployed environments?
• How many robots can be produced per unit time?
• How does the cost per effective hour of automated labour compare to that of human labour?
• Related to all of the above, what form factor do the robots take? Are they drones, arms, humanoids?
• Who builds and controls them? China is far ahead in manufacturing but somewhat behind in AI.

What is the current state of each of these facets of robotics? How might advances in AI contribute in the future?

Capability and generality

Labs like Physical Intelligence make bold claims and produce flashy demos, and do seem to be making some meaningful progress on generality. However, these results seem brittle — this paper shows that a Physical Intelligence vision-language-action model, π0, falls from 94.2% to 15.8% on LIBERO-Plus when its camera angle changes. As with LLMs, there is a split in capabilities between open- and closed-source VLAs, but most frontier VLAs are not benchmarked against those of other companies or against open-source competitors. This makes it hard to meaningfully evaluate what the rate of progress here is; companies are naturally incentivised to inflate their own progress.

That current SotA VLAs don’t seem especially concerning isn’t much of an update — I am much more interested in how they scale with intelligence, and particularly what happens when robots have an LLM as their “brain”. There are surprisingly few benchmarks for this. Andon Labs’ Blueprint Bench is helpful for seeing the trend of spatial reasoning improvements in frontier models, but their (imo much more useful) Butter-Bench hasn’t been updated in over a year.

Fortunately, Artificial Analysis have stepped up with AA-Robotics, a benchmark where frontier LLMs directly operate a robot arm.² This broadly showed that models are extremely slow, incompetent and expensive when compared to humans. This is promising! But I think measuring the rate of progress here will be important. I expect LLMs to continue scaling in a way that VLAs won’t (among other reasons, because it’s much easier to produce data for LLMs than VLAs), and the history of LLMs has taught me that many surprising capabilities can be unlocked simply by having an improved understanding of language.

These results, combined with Epoch’s work evaluating robot capabilities, make me somewhat optimistic. Robots have a long way to go before they can be used in any catastrophic risk scenario. It is entirely possible that improvements in intelligence might unlock robotic capabilities past some particular threshold, or that there could be a new technique leading to an “unhobbling” of these capabilities, or that sufficiently advanced LLMs could substantially speed up robotics R&D. However, these are largely still theoretical; the empirical evidence is early and inconclusive.

Reliability

Clearly, robots can improve significantly, to the point where they are dramatically more reliable than humans in deployed environments — the prototypical example here being Waymo:

Waymo needs to have several nines of reliability in order to be able to deploy its robotaxis. It’s a clear example of a high bar for a robot, which it duly surpasses. Other deployment environments — for example, warehouses — might only require around 99% reliability for their pick-and-place robots. In warzones, drones might need far less reliability at hitting a given target before deploying them becomes acceptable.

Unsurprisingly, AA-Robotics found that not only were LLMs bad at completing tasks with a robotic arm, they were also bad at judging when the task was finished. It seems reasonable to infer that they are also bad at completing tasks reliably and consistently. Generally I find this less positive overall — there are at least some threat models which don’t require robots to be highly reliable for them to be effective, and even if that wasn’t the case, we have clearly shown that we can get robots to be extremely reliable while deployed in complex environments.

Speed of production

There are about 5 million deployed industrial robots, of which about 600,000 were installed last year, per the IFR. This represents a doubling time of around 6 years, although doubling times for risk-relevant categories of robots are much shorter, notably 33 months for drones, 10 months for quadrupeds and 6 months for humanoids³. There are also various physical limits; new factories in China take at least 6 months to build, and in the West they are often 2+ years, and high-precision reducers are a significant bottleneck to dramatically scaling up humanoid production. While factories could be repurposed, historical wartime demand shocks only raised growth rates 1.4-2.2x, although AI-induced demand shocks would presumably be able to squeeze out some additional efficiency.

A high production rate is essential for misaligned takeover and gradual disempowerment, but manufacturing capacity is generally slow to build, and there isn’t a huge volume of it already in place. There are arguments to be made for how advances in AI might be able to yield significant boosts in capacity, but again, these are currently theoretical rather than empirical, and are more speculative in my opinion. The news here is potentially especially promising if you think that the West will reach RSI/AGI first; its manufacturing capacity is significantly behind China’s, and its ability to grow it much slower. For a (much) more thorough discussion on the economics and physical limitations of an industrial explosion, I recommend this series by Damon Binder.

Cost per effective hour

Anthropic found that automated labour is competitively priced against human labour for ~0.3% of US job tasks, and it would take ~40 years to reach 10% if current trends continue. Robots require significant upfront costs that have to be paid back before companies profit from them — Chinese humanoid robots cost around $35k to buy, and Goldman Sachs estimated they cost about 2.8 years of a factory worker’s labour⁴. For a base-model humanoid, assuming it runs at 6000 hours a year for 5 years, that’s ~$1/hour in capital, but VLA-driven robots are 3-10x slower than humans.

Cost is primarily relevant to discussions of gradual disempowerment. The findings here aren’t much of an update; widespread robotic labour isn’t currently economically viable, largely because of issues with capability and reliability. If those things improve significantly, the unit economics would change commensurately.

Form factor

Specialised robots can already scale fast; drone production in Ukraine increased by three orders of magnitude between 2022 and 2025. Amazon deployed its millionth warehouse robot last year. Humanoid production is much lower — only around 16k in 2025 — and robotic arms are still low at around 570k with a comparatively flat growth rate.

Concentration of power and great power war largely require drones, which we are already capable of producing plenty of. Misaligned takeover likely requires significant amounts of dexterity, and gradual disempowerment likely requires even more than that.

Geopolitical control

China already builds >80% of the world’s humanoid robots. It also owns large parts of the supply chain, and has export controls on rare-earth minerals that it has temporarily suspended with the US.

Military control is already a contentious issue; Anthropic declined to remove safeguards that would allow its models to be used in autonomous weaponry, which led to it being designated a supply-chain risk by the US Department of War (as well as presumably costing it lucrative government contracts). The recent UN collective statement on the use of autonomous weaponry was primarily watered down by, ah, Russia and the United States.

LLM-brained robots also inherit all the same issues as their purely textual counterparts, possibly including jailbreaks, prompt injections, trigger phrases etc. It’s possible that robots designed and built by one actor could be vulnerable to being taken over by another actor, including at a large scale.

The current geopolitics of robotics does not fill me with confidence at all. China’s dominance as a manufacturer, combined with the US’ increasing hawkishness, will continue to concern me unless the latter changes at the next election.

In conclusion

I think having a basic understanding of the following is useful in order to understand how advances in robotics might affect catastrophic risks:

• The capabilities of the current SotA VLAs, and of LLMs on robotic benchmarks
• The rate of progress on these two things
• The unit economics of different kinds of robot
• The bottlenecks to scaling up production of different kinds of robot

My broad takeaway here is that the current capabilities are lacklustre, but I am concerned that they could improve rapidly, enough to enable significant risk vectors from developing advanced AI systems — there is not particularly strong evidence here; I am mostly indexing off the rate of progress in other AI domains. Robots right now are extremely expensive, but costs will come down if/when improved capabilities drive increases in demand. There is evidence that global superpowers are willing to use autonomous weaponry on the battlefield, and that they have the capacity to quickly produce huge quantities of drones. I think the current state of robotics shows there is at least a somewhat clear path towards robots enabling catastrophes from concentration of power, misaligned takeover, gradual disempowerment and great power war, but that the first three are primarily contingent on above-trend improvements in capabilities. Catastrophe from great power war seems plausible in the short term even with on-trend improvements.