Unpluggers, Deflators, and Mantic Pixel Dream Girls, Part 1: Why I am Not a Doomer
Theorycrafting the new meta
First of all, big thanks to everyone who has signed up for this substack. I don’t know about you but I tend to carry around these “vague certainties” in my head - I am confident I have figured something out, that I have some genuine insight into a problem, and it’s only when I struggle to actually articulate what that insight is that I realize how nebulous and unformed it was in my head. It is the struggle to articulate that creates the insight. So I really appreciate you giving me this place to do that and I will try my best not to waste it.
One of the things that makes the current situation so dizzying is the vast chasm between the takes. Which one is right for you? In the next couple posts I want to look at three of the main camps that roughly define the current landscape.
Tune Out, Turn Off, and Unplug
A lot of very smart people are convinced that advanced AI will almost certainly destroy humanity and drain the universe of all value, and we should do everything we can to prevent it. You’re probably already familiar with this “doomer” position, but here’s a good summary.
It’s tempting to view this camp as a kind of wacky, hyper-rationalist death cult, and many people dismiss them derisively as such. But they don’t come off that way to me. They seem thoughtful and reasonable, they are well aware of the best arguments against their position and have sober counterarguments to those. They know that their position is weird, and take great pains to explain that yes, like a normal, totally sane person, they’ve taken this weirdness into account and still remain convinced that they’re correct.
Many people have tried a direct attack on the doomer position, claiming that it isn’t possible that AI will destroy everything, because intelligence isn’t a thing, or intelligence is a thing but doesn’t work that way, or because of some other established principle or law that this would violate. But to my mind, all of these counter-claims require taking, as given, some premise that is even more speculative than the ones that form the foundation of Mt. Doom. I don’t buy these super-confident declarations that the disaster scenario can’t happen. We just don’t know enough about intelligence, artificial or otherwise, to say for sure what it will or won’t do under the circumstances we are creating for it.
Personally, I think we are forced to accept at least a weak version of the doomer claim. Namely, that advanced AI might lead to the total, irreversible destruction of everything we care about. Like, next Tuesday. Which is sobering. And ought to make us sit up and pay attention. Things could go disastrously badly.
And yet I’m not an unplugger. I don’t think we should melt all the GPUs, or make advanced AI illegal, or cultivate an ethical norm against it. And I’m trying to figure out why. I think in a weird way this question itself demonstrates something about the limits of reason. The doomers have laid out a rational argument that I find plausible enough that I think there’s some percentage chance it’s correct. But I don’t feel compelled by that argument to change my beliefs about how to act in the world. And I don’t have a different argument for why this is the case, because I feel like any argument I make will be countered by a fast-talking, hyper-intelligent AI safety advocate. But I do have my reasons.
First, the disaster scenario is a singleton, a one-off, a totally unique event that can only ever happen once. One of the things that makes the AI alignment problem so hard is that we only get one chance. If and when we make a superintelligent AI, if it decides to erase everything then that’s it, game over. We don’t get any second chances, we can’t screw it up and try again, which is how we normally learn how to solve problems. But it’s very hard to reason about singletons. You have to do it from first principles. You have to understand the laws that govern behavior you’ve never seen and create useful theories for phenomena that don’t yet exist.
How are you going to reason about a singleton? Well, you have to extrapolate from things that are sort of like it. But once you start doing that, then why not expect the singleton to have the common feature that all of those other things that are even remotely like it have - that is, being one event among many in the universe, part of the complex causal mesh of other events that come before and after it, that form patterns and share properties and can thereby be reasoned about?
I think, like it or not, we have to encounter problems in order to solve them. And so, if we happen to be in a universe with a magic problem that has to be solved before we encounter it, we’re fucked. No amount of trying really hard is going to work.
And, despite being reasonable and sound, the doomer arguments are not so airtight that we should assume we are definitely in the disasterverse. Consider, for instance, one of the central pillars of the argument - the orthogonality thesis. This thesis proposes that intelligence is unrelated to wisdom - that you can be supremely effective at accomplishing things in the world and yet be motivated by completely arbitrary goals to accomplish utterly trivial things. Again, I think a weak version of this thesis is convincing, and a useful reminder to be on constant alert against destructive, out-of-control, accomplishment loops. (← Don’t click this link, just take a moment to predict where you think it goes. Yes, that’s what I meant.)
But maybe the orthogonality thesis is wrong, or wrong-ish, or incomplete. When I try to predict the text of the smartest person I can think of on this topic it goes something like this:
This thing "intelligence" that we are in the process of understanding/crafting/making more of is just our narrow way of describing a deeper, more fundamental thing - knowledge. Knowledge is an actual force in the universe. DNA encodes knowledge but you wouldn't call it “intelligence”. Knowledge is the force that allows living systems to persist in the world, and it isn’t arbitrary, it reflects true understanding about the world, and that’s as true of ethics as it is of physics.
If you are worried about a world with self-replicating, self-improving systems in it, I’ve got bad news for you: it already happened. We are living in the midst of it, and are indeed the result of it. It’s an explosion, not of intelligence, per se, but of knowledge. From mud puddles to crystals to primitive homeostatic chemical loops to cells to organisms, neurons, people, theories, computers and beyond. And this process is already “out of control”, it’s already feeding on itself and growing exponentially. But the thing that is growing is knowledge. You can’t intelligence your way to infinity.
In the case of DNA, knowledge grows by encoding facts about the environment in more and more efficient ways, through natural selection. In our case knowledge grows through conjecture and refutation, creativity and criticism. Whatever these new processes are that we are creating, to the degree that we succeed at making them "intelligent" (meaning all the things that we're trying to maximize for) they will generate knowledge the same way we do, by creating theories and testing them against each other, and also by having emotions and factoring those in (emotions are just theories we don't have conscious access to), and traditions, and tools, and institutions, and norms, and so on, all of which are flawed but can always be improved. Because that's how knowledge works, and that's the real thing we are noticing here.Or, if they have some other method of creating knowledge, that is as different from ours as ours is different from DNA’s, then all the better. Because it's still knowledge that is being generated. And that can't mean paperclips.1
Now, I recognize that Deutsch’s knowledge explosion is also a singleton. And it’s hard to reason about singletons. But in his story things continue to happen. And it’s only in a universe where things happen that we can reason at all.
So that’s my current plan. Not to reject the doomer argument wholesale, but to see it as a source of important ideas and theories, and as a constant reminder that these things matter. That there is much at stake, and that we should treat this confusing and disorienting time as a challenge to picture how things could go badly and how they could go well and do what we can to achieve the best outcomes possible.
Next up: Part 2: Why I am Not a Deflator.
Apologies to D. Deutsch for my crude attempt to simulate him here. All errors and confusions are obviously mine.