Paul Christiano's Barista
I’m shamelessly ripping off Inkhaven and doing my own challenge, Egghaven. This is the sixth post.
When I left my previous job in April of last year, I took a long break from thinking about AI safety. This was nice. Thinking about catastrophic risks all the time wears you down. Imagining oneself as Cassandra, baffled that nobody around you is taking this sufficiently seriously, is not good for the soul1. I was angry at the community that had got me interested in AI risk in the first place. I felt powerless to help, and that I had earned more selfishness anyway.
Of course, avoiding AI altogether wasn’t an option. I was worried that coding would soon be automated, and my most marketable skill would disappear. I wanted to broaden my skillset and position myself well in case I couldn’t go back to a regular software engineering job. I started writing posts about how I was going to found a company, after having been heavily influenced by reading The Intelligence Curse.2 I got really into thinking about the intersection of AI and psychiatry. I started talking about the importance of defensive acceleration and how effective altruists systematically underrate comparative advantage. This was motivated reasoning; I wanted to work on something other than AI safety and not feel guilty about it.3 I figured that all these super smart, status-y types had it under control already.
In April of this year, I had various conversations that led to me feeling excited about being part of the EA community again. I went to EAG London and primarily talked about things other than AI safety.4 I started doing some contract work for an EA-flavoured global health and development charity. I was managing to feel like I was part of the community without getting sucked into AI safety’s gravity well.
I ended up applying for a job at an EA organisation. I managed to get through to the work trial stage, which involved giving people advice. It was noteworthy that diverse groups of people often seemed to need to hear relatively similar advice. I kept repeating something along the lines of:
Build a model of the world. Try to falsify the components of that model. Seriously think through the implications of this model, and then act accordingly.
And then of course something deeply upsetting happened. I realised that I was not following my own advice.
So I went back to basics. Claude and I went back-and-forth over my litany of questions and objections to the idea that advanced AI systems could pose an existential threat. I wrote a post where I attempted to reason through what I thought was likely to happen over the next two years. It started to seem likely that, at the very least, things are going to get weird, soon.
Somewhere during the process it dawned on me that vanishingly few people do this kind of thing; most were LARPing, just like me. When considering something complex, at some point, it’s all too easy to let rigour lapse and hand-wave something away. Thinking rigorously is hard, but in a particularly subtle way that makes this kind of hand-waving easily done without being consciously noticeable.
I felt a sense of clarity. It might be that this clarity is entirely fake and I’m totally wrong, but still, it persists. I was ready to say it with roon:
experts are fake, smart generalists rule the world, everything is designed by people no smarter than you, courage is in shorter supply than genius
I’ve long felt like I’m trapped in a midwit meme, painfully aware of the importance of AI safety but simply lacking the intelligence or skills to actually do anything about it. When I read Scott Alexander’s Parable of the Talents, I imagined myself as Paul Christiano’s personal barista. But I don’t think any of it is true. I’ve sufficient genius. What I was lacking was the courage to actually do the hard work of thinking seriously about things for long enough to reach a conclusion, even when that conclusion is an unpleasant one.
Having sat down and made a good-faith effort to think it through, I do not think we are on track to develop AGI in a safe way. I think I have enough power and resources that there’s some chance I could make a difference. And so I’m going to try.
-
Although Michael Burry seems to be making it work for him. ↩︎
-
I feel weird about this. In some sense, it is true; I am the sole director of a company. The word “found” has various implications that aren’t accurate in my case, but sometimes it is useful to pretend that they are. Sometimes “founding” is quite high-status, and so there’s a temptation to play it up. I think that’s pretty clearly what was happening in this post. The whole thing seems naïve too – although I think that finding success with a startup probably does require some amount of naïveté. ↩︎
-
Despite being motivated, I do think there’s some legitimacy to the claim regardless. ↩︎
-
It was my sixth EAG, and the other five had all been dominated by the topic. ↩︎