Joe Carlsmith
@jkcarlsmith
Followers
7K
Following
744
Media
69
Statuses
367
Philosophy, futurism, AI. Senior advisor @open_phil. Opinions my own.
Berkeley, CA
Joined April 2013
I put my report on existential risk from power-seeking AI on arXiv:
arxiv.org
This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that...
7
14
141
In response to a comment from @herbiebradley on my recent talk, I wrote a bit about my backdrop model of the long-term role of human labor in a post-AGI economy.
@herbiebradley I haven't written about this much or thought it through in detail, but here are a few aspects that go into my backdrop model: . (1) especially in the long-term technological limit, I expect human labor to be wildly uncompetitive for basically any task relative to what advanced.
1
0
14
RT @michael_nielsen: Thoughtful discussion of "Can Goodness Compete [with power]?" by @jkcarlsmith (link in next post). It's a really funda….
0
6
0
I recently gave a public talk called “Can goodness compete?”, on long-term equilibria post-AGI. Video here and on YouTube, link to transcript and slides in thread.
I'm giving a public talk Tuesday July 8th, 7:30 pm at Mox in SF. Title: "Can goodness compete?". It's about long-term equilibrium outcomes post-AGI. More info at link in thread.
5
12
130
I'm also aiming to make a recording of some version of the talk publicly available (might be the Vancouver version).
@jkcarlsmith Any chance it will be recorded and made available?.
2
0
22
This is a longer version of the talk I'm giving at this workshop in Vancouver next week:
It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop!. Post-AGI Civilizational Equilibria: Are there any good ones?. Vancouver, July 14th. Featuring: @jkcarlsmith @RichardMCNgo @eshear 🧵
2
0
15
I'm giving a public talk Tuesday July 8th, 7:30 pm at Mox in SF. Title: "Can goodness compete?". It's about long-term equilibrium outcomes post-AGI. More info at link in thread.
4
9
112
RT @DavidDuvenaud: It's hard to plan for AGI without knowing what outcomes are even possible, let alone good. So we’re hosting a workshop!….
0
32
0
RT @zdgroff: 💡Leading researchers and AI companies have raised the possibility that AI models could soon be sentient. I’m worried that to….
0
26
0
To my knowledge, this is the most serious industry-led attempt to investigate the welfare of a frontier AI system in human history. Kudos to Anthropic for leading the way.
🧵For Claude Opus 4, we ran our first pre-launch model welfare assessment. To be clear, we don’t know if Claude has welfare. Or what welfare even is, exactly? 🫠 But, we think this could be important, so we gave it a go. And things got pretty wild….
2
1
83
Slides for the talk here:
docs.google.com
How should we think about AI welfare? Joe Carlsmith Talk at Anthropic, May 2025
0
0
3
And I close with a brief discussion of what AI labs like Anthropic can do.
1
0
5
I also talk about various arguments for the possibility of moral status without consciousness (see slide).
1
0
2
I expect AIs to have many of these for roughly the same high-level reasons we do: namely, that they're useful. So a key question is whether AI minds would accomplish these same useful functions in a way that involves consciousness. I think it's unclear, but plausible.
1
0
1
But of course, AIs are different: produced via a less evolution-like process, already exposed to our discourse about consciousness, trained to give specific takes on consciousness, etc. So we have to focus more on their consciousness-associated capacities.
1
0
1
I also talk about a different intuition pump: namely, imagine that we discover aliens made of silicon, that had independently evolved all our consciousness-associated behaviors, capacities, and philosophical confusions. Should we think they're conscious? I think yes.
1
0
1