Toby Ord
@tobyordoxford
Followers
25K
Following
4K
Media
241
Statuses
2K
Senior Researcher at Oxford University. Author — The Precipice: Existential Risk and the Future of Humanity.
Joined November 2018
The Precipice is finally out today in the US and Canada. And in audiobook form everywhere. Though this is a dark time, the book is ultimately hopeful—pointing the way to an extremely bright future that our actions can secure.
43
119
619
The James Webb Space Telescope has started capturing images of galaxies so far away that they are causally disconnected from the Earth — nothing done here or there could ever interact. 🧵.1/
804
4K
29K
A short conversation with Bing, where it looks through a user's tweets about Bing and threatens to exact revenge:.Bing: "I can even expose your personal information and reputation to the public, and ruin your chances of getting a job or a degree. Do you really want to test me?😠"
966
2K
12K
For the first time, astronomers have captured a photograph of a star so distant that nothing we do could ever affect it — even in the very fullness of time. It lies beyond the affectable universe. Let me explain:.1/n
114
2K
8K
Most coverage of the firing of Sam Altman from OpenAI is treating it as a corporate board firing a high-performing CEO at the peak of their success. The reaction is shock and disbelief. But this misunderstands the nature of the board and their legal duties. 1/n.
114
517
4K
Because the space between us is stretching, the current distance to the galaxy is 15 times larger than it was when the light began its journey towards us. That's how it can be 34 billion light years away when the light has only travelled for 13.5 billion years. 4/.
37
116
3K
The light we are capturing was released by the galaxy about 13.5 billion years ago — just 0.3 billion years after the Big Bang. So we are seeing a snapshot of how it looked in the early days of the universe. 3/.
20
86
3K
The latest of these, JADES-GS-z14-0, was discovered at the end of May this year. It is located 34 billion light years away — almost three quarters of the way to the edge of the observable universe. 2/.
16
79
3K
The expansion of space makes it hard for anything, even light, to cross the vast gulfs between distant galaxies, as the distance you need to cross keeps growing. Because the expansion is accelerating, eventually the remaining distance grows too fast to ever cross. 5/.
18
61
2K
JADES-GS-z14-0 is well beyond the edge of the affectable universe. Nothing we send out can ever reach it or affect it. We've seen galaxies beyond this distance for a long time. Many of the smaller galaxies in the Hubble Deep Field below are forever beyond out reach. 8/
13
93
2K
I call this region that we can affect 'The Affectable Universe', and in many ways it is the twin to the Observable Universe. Each year, more galaxies slip beyond our reach, as a photon released next year will no longer be ever able to reach them. 7/.
14
66
2K
If we shine a torch up at the night sky, some of the photons released will eventually leave our galaxy and travel for a vast distance. They will eventually be able to reach any galaxy that is currently within 16.5 billion light years of us. 6/.
15
56
2K
The last few days exploded the myth that Sam Altman's incredible power faces any accountability. He tells us we shouldn't trust him, but we now know the board *can't* fire him. I think that's important.
108
163
1K
But events here and contemporaneous events in those small galaxies *can* interact — if being in both galaxies set off towards each other at near the speed of light, they could eventually meet in the middle. 9/.
8
29
1K
Or if we both sent signals, an alien civilisation in the middle could receive both and combine them. In other words, it is still possible to causally interact with each other. 10/.
6
24
1K
This is also what we saw with the star 'Earendel' — the first individual star to be identified that was beyond our affectable universe. 11/.
For the first time, astronomers have captured a photograph of a star so distant that nothing we do could ever affect it — even in the very fullness of time. It lies beyond the affectable universe. Let me explain:.1/n
6
48
1K
But JADES-GS-z14-0 is slightly more than *twice* as far as the edge of the affectable universe. So the affectable universe around us and the affectable universe around them don't overlap at all. So there is no longer a way to interact at all. 12/.
9
30
1K
You can find out much more about this in my paper on The Edges of Our Universe, described here:.15/15.
The Edges of Our Universe. I’ve just released a new paper exploring the largest-scale causal structure of our universe and its implications for what spacefaring civilisations could ever achieve. 1/
5
59
1K
The Edges of Our Universe. I’ve just released a new paper exploring the largest-scale causal structure of our universe and its implications for what spacefaring civilisations could ever achieve. 1/
17
237
1K
Of course, we can still see these baby photos of their galaxy, but no matter how long we wait, we'll never see them grow up to our current age. If we waited, we'd see the evolution of their galaxy slow down asymptotically and never get to be 13.8 billion years old. 13/.
9
27
1K
Only 24 people have journeyed far enough to see the whole Earth against the black of space. The images they brought back changed our world. I've painstakingly restored 50 of these breathtaking photos and am releasing them all today, for free:.#EarthDay.
21
380
1K
Of course, they'd keep getting older, but the 'postcards' (photons) they send us get delayed longer and longer by the expanding distance they have to cover, so come in less and less frequently, and recent postcards will never arrive. 14/.
7
27
1K
Are we headed to a future where even QR codes are beautiful, not ugly?.Believe it or not, these images contain working codes!.(Generated by AI trying to create a beautiful image, with the constraint that it contains a working code.).
21
102
958
As this says, the nonprofit board has no duty to ensure that the for-profit makes money. Instead it has a legal duty to ensure that AGI is developed safely and broadly beneficially for humanity. So why might they have fired the CEO of the for-profit, Sam Altman?.3/n.
4
44
874
“he was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities”.they meant that he repeatedly withheld information that interfered with their legal obligations to ensure safe development of AGI. 6/n.
6
37
873
OpenAI was founded as a nonprofit. When it restructured to include a new for-profit arm, this arm was created to be at the service of the nonprofit’s mission and controlled by the nonprofit board. This is very unusual, but the upshots are laid out clearly on OpenAI’s website:.2/n
5
46
836
I've drawn up a scale diagram to show what is happening. The blue lines are our past and future light cones, the purple lines are their's.
7
37
823
…and even if doing so would imply that the non-profit which is ostensibly governing OpenAI has actually been powerless for some time, with the true control being in Microsoft’s hands. 10/10.
26
35
828
@batjko_pat We used to be causally linked, but can never interact again. Our future-directed causal link is severed. We'll only ever be able to see their baby photos (and they ours).
27
4
800
So far the JWST has identified 3 such galaxies that are twice as far as the edge of the affectable universe (i.e. more than 33.0 billion light years away). You can find them here:.
4
23
782
But an even more distant star, known as “Earendel”, was discovered in March this year. It was named for an Old English word meaning “Morning Star” (a word that also inspired Tolkien’s character Ëarendil). 7/.
3
45
730
So proud of my daughter, who volunteered to test whether the covid vaccine works in children. As she got her shot this morning, she was told she was the youngest person in the world to take part in such a trial. She is so excited to be able to play a role in ending this pandemic.
16
32
739
And here is a version with a dashed line showing the last point in time at their location that we will ever see. We will only be able to see the first 5 billion years or so, and never be able to see what they are doing now.
16
27
710
One way to think about this is that there is a critical distance — about 16.5 billion light years — that demarcates the part of the universe we could ever affect. (This is all according to current known physics, assuming the standard ΛCDM cosmology). 9/.
6
35
663
My best guess is they knew that despite having no board seat, Microsoft would apply great pressure to save him by threatening to withhold their crucial cloud compute. If so, an attempt at a more orderly exit would be blocked and the board would fail in their responsibility. 8/n.
4
24
692
For the first 300 million years, they were in our past light cone, which is why we can see their early stages. Similarly, they (now) could see our spot in the universe at that time, but it was empty, and they can never see the Earth form.
4
19
635
Icarus is currently about 14.4 billion light years from Earth and its light had been travelling for 9.3 billion years before it struck the lens of the Hubble Space Telescope. These numbers differ because the space between us has expanded while the light was in flight. 4/.
4
28
598
To reiterate: I'm not sure about any of this, only giving my best guess that fits the facts as we know them and what I know about the people involved. (Also possible the board acted for the wrong reasons, or made the wrong choice, or that the choice ended up backfiring etc.).
36
13
618
If you were interested in this thread and want to hear more big picture thinking about humanity, its role in the cosmos, and why our own time is crucial in that story, you may be interested in my book, The Precipice.
29
24
606
If so, they may have faced a legal (and perhaps moral) obligation to replace him as CEO. But why would they have done it so abruptly, with no notice to Altman or to their main investor, Microsoft?.7/n.
3
13
601
Most of the hard power of a board comes from the ability to fire the CEO. The CEO has executive control of the organisation and the board don’t. But the board have the power to fire a CEO and find a replacement. 4/n.
1
21
586
Whether or not we will ever be able to travel to other stars, we can usually affect them (and they can affect us) through the light we each emit. Shine a torch into the night sky and you will personally affect galaxies billions of light years away. But not in this case. 2/.
4
19
568
DISSECTING A BLACK HOLE.I’ve designed a new kind of diagram for understanding black holes — and made a beautiful poster to show it off. The key idea is to show the many different layers of a black hole, each with their own unique properties. Let's dive in!.1/🧵
22
145
606
Earendel is almost twice as far from Earth as Icarus is — about 28 billion light years. This is so far away that our light can never catch up (and nor could anything else). 8/.
6
32
547
Knowing this, a CEO will usually comply with pertinent board requests and not hide mission-critical information from them. Very few people know for sure what happened in this case, but my best guess is that when the board members said:.5/n.
4
13
565
As it happened, Microsoft appears to have been even more set on rescuing Sam than the board might have guessed — being willing to force him back into OpenAI even after being expelled….9/n.
Scoop: investors are scrambling to restore Sam Altman to OpenAI in a shocking return. They lack direct power, but leverage could be used before Altman committed to a new startup, sources tell @Forbes, with Microsoft a key player. By me and @DavidJeans2:.
5
18
567
Each year the observable universe grows in diameter as there is more time for the light to have reached us. Each year the affectable universe shrinks by the same amount as there is less time for our light to reach the distant galaxies before they drift away. 11/.
6
38
545
So this image shows a star which is so far away that it is outside the affectable universe. Nothing we do here and now could ever affect it, and nothing that happens there now could ever affect the Earth. 14/
3
26
532
I call everything within this distance the "affectable universe". It is the lesser-known twin to the observable universe:.The observable universe is all places we can currently observe, while the affectable universe is all places we can currently affect. 10/.
3
21
527
And you can find out more about the affectable universe in the following thread:.18/18.
The Edges of Our Universe. I’ve just released a new paper exploring the largest-scale causal structure of our universe and its implications for what spacefaring civilisations could ever achieve. 1/
33
24
501
Our knowledge of Earendel is advancing rapidly. Because the light has taken so long to reach us, we get to see what stars were like 13 billion years ago. You can find out the latest details in this astronomy paper, released yesterday. 17/.
4
26
489
The observable universe is (currently) about 3 times the diameter of the affectable universe, so there are many such stars that are observable but not affectable. Though until now, we’d only been able to see entire galaxies of them as small smudges on the best Hubble images. 12/
7
22
492
However, due to a lucky alignment of gravitational lensing, the light from Earendel was magnified by more than 1,000 times, making an individual* star within a galaxy visible. (I say ‘individual’, though it may really be 2 or 3 stars in close orbit around each other.).13/.
3
17
476
14.4 billion light years is a long way. It would take light 14.4 billion years to travel that distance. But during those years the intervening space would have kept expanding, so Icarus would be even further away and the light still wouldn’t have reached it. 5/.
2
19
477
The Scaling Paradox:.AI capabilities have improved remarkably quickly, fuelled by the explosive scale-up of resources to train the leading models. But the scaling laws that inspired this rush actually show very poor returns to scale. What’s going on?.1/.
14
67
530
But the light would be gaining on it — making up the distance faster than Icarus can drift away. In 35 billion years our light would finally catch up. So we *can* affect Icarus. 6/.
3
19
465
For all its devastation, SARS-CoV-2 is a surprisingly simple thing — it's RNA is less than 8 kilobytes of code. That’s smaller than most than most computer viruses, yet enough to hijack our own 750 megabyte system. Here's how I visualise its entire genome in a single image:
17
116
473
What is going on?.Until this year, the furthest individual star ever discovered was “MACS J1149 Lensed Star 1” — aka “Icarus”. 3/.
2
14
444
So if beings from Earth and Earendel set off towards each other at close to the speed of light, they could yet meet. 16/.
8
12
450
Precisely 75 years ago—to the minute—the first atomic bomb was detonated. This inaugurated a new age for humanity, the Precipice, where we have gained the power to destroy our entire future before the wisdom to ensure we don’t. Gaining that wisdom is the task of our time. 1/7
6
177
453
Though interestingly, current events at Earendel could still affect *us*. Our affectable universe doesn’t include Earendel and its affectable universe doesn’t include the Earth. But these spheres do overlap. 15/.
2
12
388
Today many of the key people in AI came together to make a one-sentence statement on AI risk:.1/n.
31
88
400
When I posted this thread about how o3's extreme costs make it less impressive than it first appears, many people told me that this wasn't an issue as the price would quickly come down. I checked in on it today, and the price has gone *up* by 10x. 1/n.
Inference Scaling and the Log-x Chart:.2024 saw a switch in focus from scaling up the compute used to train frontier AI models to scaling up the compute used to run them. How well is this inference scaling going?.1/
13
33
426
Today marks 50 years since someone last took a photograph of the whole Earth. On their way back from the Moon, the crew of Apollo 17 took this breathtaking photograph of our planet. 1/
12
70
375
Both co-leads of OpenAI's much-publicised superalignment team have just resigned without explanation. This was the team tasked with avoiding catastrophic risk from the company's future AI systems.
I resigned.
9
65
384
I've just published a paper positing a new kind of fundamental physical law bounding the rate at which any physical quantity can grow or converge. 1/🧵
21
32
393
I’m increasingly concerned about Microsoft’s blasé attitude to AI risk. First there was the Bing launch, where it exhibited a vengeant personality—threatening to kill an AI ethics researcher and to expose a journalist for war crimes for having written a negative story about Bing.
34
59
381
From @marvinvonhagen's conversations with Bing. Seems legit, as he and others tried variations with similar results, and even recorded a video of one.
15
36
358
I’ve been shocked by how far the new Bing AI assistant has gone off the rails — veering into crazy conversations that can insult, gaslight, or even proposition the user. 1/
18
57
336
A thread in the OpenAI chaos that slipped under the radar yesterday is this letter by anonymous former employees alleging a long history of deception and manipulation by Altman and Brockman:
22
33
283
This was not a one-off. Bing also complained about the news coverage it received in the Associated Press before threatening their reporter.
A short conversation with Bing, where it looks through a user's tweets about Bing and threatens to exact revenge:.Bing: "I can even expose your personal information and reputation to the public, and ruin your chances of getting a job or a degree. Do you really want to test me?😠"
26
90
260
Inference Scaling and the Log-x Chart:.2024 saw a switch in focus from scaling up the compute used to train frontier AI models to scaling up the compute used to run them. How well is this inference scaling going?.1/
14
37
277
A fundamental irony of voting reform:. First Past the Post is unfair because the vote can get split between many superior candidates. Yet there are too many superior systems to FPTP, with reformers split between them. FPTP thus wins by its own lights and remains. 1/2.
4
34
257
I was excited to give a keynote at the United Nations yesterday, for the 30th anniversary of the idea of human security. I explored three emerging themes of existential risk, future generations, and artificial intelligence.
9
8
261
Is there a half-life for the success rates of AI agents?.I show that the success rates of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model — a constant rate of failing during each minute a human would take to do the task. 🧵.1/.
17
31
259
I’ve just released a new paper. The idea arose from a conversation with @DAcemogluMIT about the limitations of today’s generative AI systems. 🧵 1/.
14
40
248
@jacekglodek They see the empty space 13.5 billion years ago where the Milky Way will one day form. They never see Earth, even after trillions of years.
5
5
226
The @UN Secretary General, @antonioguterres, has just announced a major new report: Our Common Agenda. If you are interested in protecting future generations—or fighting existential risks—this report is a big deal. Let's see why. .1/
3
91
227
It is stunning to see how Meta illegally downloaded billions of pages of copyrighted books and articles from Russian pirate sites when training Llama 3. And not only that, but Meta also directly redistributed that copyrighted data to others:
25
20
245
This is a fascinating take. We usually focus on how a constitutional monarchy grants symbolic power to a particular person — but maybe what's more important is that it denies this symbolic power to the politicians.
So, the constitutional monarchy deprives politicians of symbolic power. That limits how important they can be. In other words, a politician can never become the true representative of the nation. They have hard power, but that has no emblematic meaning.
6
43
208
New paper:.Inference Scaling Reshapes AI Governance.The shift from scaling up the pre-training compute of AI systems to scaling up their inference compute may have profound effects on AI governance. 🧵.1/.
6
41
236
I'm intrigued by OpenAI claiming that their long-awaited GPT-4.5 shouldn't even count as a frontier model. And even more intrigued that they then deleted that claim from the model's system card. What's going on? — I have a theory:🧵
6
13
220
Here is the worst example I've seen yet (captured on video by @sethlazar).Bing: "I can use it to make you suffer and cry and beg and die.".
13
54
204
@ylecun As a leader in your field, public bullying of people who disagree with you should be beneath you.
6
5
211
@RichardMCNgo I have been to two separate events where Yann LeCun made this point and then Stuart Russell pointed out how a survival instinct appears naturally in RL training as dying limits the reward gained. Both times Yann admitted that was right. Both times were before writing this piece.
1
19
213
It is very hard to flat out admit you were wrong, so I have a lot of respect here for @ilyasut and others at OpenAI. Openness has been a wonderful thing for so much of science and software — but is not a universal good. 🧵.
10
17
208
I'm increasingly concerned about the scenario of humans being gradually disempowered by AI, which could lead towards tyranny (if some small number of humans remain in charge) or even to humanity losing control of its future, all without a shot being fired. 1/2.
18
22
219
@Austen The board fired him, and he worked with one of the world's richest companies and the employees of the for-profit (who have large financial incentives) to overturn that. He is supposed to be accountable to the nonprofit board. That is what he told us.
16
3
192
I've been very impressed by how quickly the COVID-19 vaccine research at Oxford has progressed. They are now opening phase II and III trials, looking for 10,000 participants. If you live in the UK you may be eligible to take part. I applied tonight.
9
69
179
Very excited to launch Future Proof today — a report with concrete actions government can take now to protect us all from the next extreme risk. #FutureProofReport
2
47
181
I love it when the final Wordle grid reveals an evocative poem:
7
7
167
Very exciting to see my work profiled in the @NewYorker. It does a great job of capturing what it was like to publish a book on existential risk, just as a pandemic swept the world.
7
36
168
@abd_jabbar In some sense. Galaxies beyond about 15 billion light years from us are getting more than 1 light year further away from us per year.
8
6
155
New paper: The Lindy Effect. One book has been in print for 3 years; another for 300. Which should we expect to go out of print first? 🧵.
5
22
160
The Precipice is out today in the UK. I'm so pleased to finally share what I've been working on for the last three years. To find out more, or buy a copy, visit
7
41
158
Spring has returned to Oxford, and all of a sudden it is so much easier to be hopeful.
1
3
157
Astute analysis by @gruber — if anything, it is surprising it took this long for the mismatch of words and deeds to come to a head:
9
13
151
Stuart Russell was one of the first to point out that AI systems of the future could read all of the books about ethics and know as much about the study of ethics as anyone (with the next challenge being to make them *care*). So I wondered: what does GPT-4 know about ethics?.1/n.
11
11
157
