Superintelligence is destabilizing. If China were on the cusp of building it first, Russia or the US would not sit idly by—they'd potentially threaten cyberattacks to deter its creation. @ericschmidt @alexandr_wang and I propose a new strategy for superintelligence. 🧵

RT @jim_mitre: In a new paper about AGI and preventive war, @RANDCorporation colleagues argue that the probability of war is low in absolut….

The moment I remember most from this series is

War and Peace in the Nuclear Age is a good documentary series: I recommend all the episodes with >=3.8K views.

That said, they still are worse than humans when the puzzles are represented pictorially (image below). The paper: Does Spatial Cognition Emerge in Frontier Models? (ICLR 2025). Thanks to @longphan3110 for running the evaluation

Apple recently published a paper showing that current AI systems lack the ability to solve puzzles that are easy for humans. Humans: 92.7%.GPT-4o: 69.9%. However, they didn't evaluate on any recent reasoning models. If they did, they'd find that.o3 gets 96.5%, beating humans.

Some pointers for the topics that are more helpful:.Deep learning: Law: (week 1 is most helpful).Safety engineering: Complex systems: Game theory: Geopolitics:.

Many fields seem useful for thinking about frontier AI strategically, but most have little to contribute. Surprisingly unhelpful:.* classic machine learning (e.g., SVMs, PGMs),.* statistics,.* machine learning theory,.* algorithms,.* optimization and control theory,.*.

Does AI deterrence require precise redlines?. Nuclear, cyber, and criminal deterrence often have intentional ambiguity. The U.S. maintains a policy of strategic ambiguity on nuclear strikes, keeping open the option of a first strike for undefined conditions. Likewise, the U.S.

This is a strawman. We don't use the phrase "AGI" in the MAIM paper (Superintelligence Strategy). In fact, we discuss how the concept of AGI is too vague to be useful in the appendix. We make it clear that the first thing we want to deter is an intelligence recursion---thousands.

Examples of international AI redlines:. 1. Intelligence explosion redline. AIs might be able to improve AIs all by themselves in the next few years. The US and China should not want anybody to attempt an intelligence explosion where thousands of AIs are autonomously and rapidly.

This depends on the ability to make AIs act as fiduciaries, which requires that AIs be reliably alignable. However, gradual disempowerment is usually posed as a problem that persists even if AIs are alignable.

In Superintelligence Strategy, I made the point that (1) no unleashed AIs, (2) requiring AIs act as fiduciaries, (3) having AIs with forecasting abilities can help avoid erosion of control/evolutionary pressures/gradual disempowerment.

We can prevent gradual disempowerment by AI. We delegate to doctors and lawyers yet can stay in charge because they must earn our informed consent. Requiring AIs to obtain informed consent and have foresight into long-term consequences helps ensure human control isn't eroded.

This riffs on Dennett's Four Competences and Murray Gell-Man's multilevel analysis of adaptation and deception

Levels of AI control: infrastructure, behavior, cognition, and institutions. Infrastructure:.datacenter/chip off switch, thinking time allotment, tools an agent has access to, agent sandboxing, input and output filters.Behavior:.input-output behavior shaped by reinforcement or.

Related post:.

A presentation on evaluating AI models' general capabilities and dual-use capabilities.

Alternative link:

I wrote about why efforts to understand the inner workings of AI keep falling short.