...Mantic Pixel Dream Girls

Step Right Up

Henri Rousseau, Carnival Evening

First of all, let’s agree that nothing is funnier than a joke that needs to be explained, unless it’s one that also sounds vaguely sexist. “Mantic” means prophetic or divinatory, so trust me, it’s hilarious. It’s the linguistic root of “necromancer” and “neuromancer”. So, as my friend Bennett likes to say, haha.

Let me be clear: in this installment of our tour through the camps that have been set up at the perimeter of the AI visitation zone, I am done throwing people under the bus. This is the tent where I want to stop. I’m tired. I can’t think any more. In order to think you have to have a place to sit down. If you are lucky, there will be people there to take care of you. If you are unlucky, you die. It’s called set and setting. For better or worse this is my tent, if they will have me. I don’t know. I don’t know whether I am Griffin Dunne or Rosanna Arquette or Martin Scorsese or Robert “Jumpsteady” Bruce, all I know is it’s late, and I’m tired, and my last 20 dollars just flew out the window. It’s either this or let the wind and sand divide me into a billion pieces. Possibly both.

There have always been nerds trying to figure out what computers were for. Sneaking in after hours to steal time on the big machine. To take the math that was made to pilot bombs and make it dream of caves.

If I knew what I was doing, this would be the post where I link to a bunch of AI-based games and music and creative work that demonstrates how art is going to save us all. I would link to a bunch of cool projects, and some of it would be things I made, or my friends made, or that I otherwise had some piece of. That’s basically how art works. True to form, I can’t even do that.

I mean, there’s definitely stuff I see happening in this tent that I like and find inspiring. Holly Hearndon. Dadabots. Is this Japanese glitch hop video AI art? Whatever it is, it’s in the tent as I see it. Same goes for this shader-based videogame desperately trying to imagine itself into existence. And whatever this is. Look, I don’t know. It’s a big tent, it’s dark in here, it’s loud. I’m just looking for those guys Miyazaki yelled at, I want to buy them a drink. But it’s not like there’s one scene, or movement, or genre I can point to. Just a lot of weirdos watching the machines reconfigure themselves and recklessly rushing in on legs that don’t quite work yet, trying to deliver a kick in the pants to truth, beauty, God… something like that.

But, ironically, it’s how AI illuminates the past that has me most excited. How it’s reactivated the threads of projects from the previous century that now seem strangely prescient. Obviously, the metaphysical fables of Borges and Calvino. But even more so, the generative language-game experiments of George Perec and the rest of the Oulipo group. The cut-up techniques of William S. Burroughs and Brion Gysin. And, while we’re thinking about it, why not Alfred Butts? What a weirdo. Who would spend all day chopping up the New York Times, just to make a game?

Who expected Jacques Derrida to become practically useful for computer de-programming? How can you encounter cyborgism as a technique for spelunking the convoluted weirdness of GPT and not think of Donna Haraway?

Who else is in the VIP section in my version of the tent? I see Conlon Nancarrow. I see David Markson. Everyone who tried to get the pianos to play themselves. The acrobats of latent space, searching for the meaning of meaning. The freaks writing consciousness porn.

In my view, one of the most important things written about what’s currently happening in AI is Peli Grietzer’s A Theory of Vibe, from 2017. In it, Grietzer draws a parallel between the mathematical operations of a neural net that is trained to encode and decode a set of data (like the ones we’re all freaking out about now) and the aesthetic operations by which we sense a genre, style, or mood. How do we recognize something as “Kafkaesque”? We can think of Kafka’s stories as compressing a large cloud of ideas and impressions and observations about the world into a small set of concrete images, situations, and events. But this same operation can be run in reverse to generate new ideas and impressions that fit the same compression scheme.

A reader of Kafka learns to see a kind of Kafkaesque aesthetic at play in the experience of going to the bank, in the experience of being broken-up with, in the experience of waking up in a daze, in the experience of being lost in a foreign city, or in the experience of a police interrogation—in part by learning that surprisingly many of the real life nuances of these experiences can be well-approximated in a literary world whose constructs are all fully bound to the aesthetic rules of Kafkaen construction.

Here we see Chiang’s “blurry jpeg” without the deflationary agenda. This process of encoding and decoding according to a compression scheme which has evolved to best fit a curated set of observations, it shouldn’t be thought of as a diminished or impoverished version of a “clean” transmission of “pure” information. Instead, we can see the compression scheme itself as a form of meaning creation. The concrete details that Kafka chose, and the specific way he arranged them, tell us something about the world not in spite of being artifacts of a compression scheme but by virtue of it.

Grietzer’s framework is a powerful new way of thinking about how artistic creation and interpretation works, one that is grounded in a close technical analysis of the statistical methods by which machine learning transforms raw data into ghostly, multi-dimensional shapes and back again. It also gives us a new software-inspired perspective on how art forges a connection between things and ideas - the concrete stuff of the world and the ghostly shapes of our thoughts and feelings, theories and plans, hopes and fears.

This framework can be usefully applied to any artistic work, but, as Grietzer points out, it feels like an especially good fit for works of modernism - Kafka, Beckett, Pound, Eliot, Joyce, Stein… Even before computers showed up, the various avant gardes of the 20th century were already exploring the structures that enabled and constrained their ability to express ideas, the relationship between mechanism and meaning. They were investigating the limits and possibilities of music, theater, paint, and language as systems. They were making art works that tried to understand how art works.

I bet, at the time, those projects felt vital and thrilling. There must have been a real sense of urgency, an awareness that something crucial was at stake. How do we make sense of a world undergoing massive, tumultuous change, a world in the process of being re-made by facts and machines, a world slipping in and out of the grip of authority, tradition, and myth, a new world full of amazing promise and incomprehensible catastrophes. Sound familiar? Reading Grietzer gives me that same sense of urgency, makes me see the ambitious projects of modernist aesthetics as something ongoing, unfinished, and vitally important.

So that’s why I’m stopping here, this is obviously my tent, the one where Hey kids! let’s put on a show! Where Mad Magazine meets Mad Science and the Mad Professor and we see what these babies can do.

I’ll admit I’m not totally comfortable in this tent. (Big shock, I know.) I just got back from the Game Developers Conference. It must have been four years ago when I was at GDC and I looked around said - why aren’t there any AI projects? There are a million dumb VR games that no one cares about, why aren’t there a million dumb AI games that no one cares about? Well, as Eliezer Yudkowsky likes to say, be careful what you wish for.

The people who were into eSports, who were into mobile, who were into VR, into the metaverse, into serious games, nfts, augmented reality, the blockchain, these people are now all in on AI. And I know this looks like bus under-throwing, but I’m just being honest. This isn’t my regular crew. But I’m here to party, it’s a big tent, and everyone’s invited.

In a tweet thread from last year, Jonathan Lai speculates about how generative AI will transform videogame development, with powerful tools that enable faster, better, cheaper creation. I don’t care about any of that. I assume some version of it will turn out to be true, but I don’t care. I am not interested in AI as a tool. I am interested in AI as an aesthetic.

When they first made synthesizers you might have thought - hey this is good for simulating the sound of any instrument you can think of, or even a whole orchestra! But that turned out not to be the important thing about synthesizers. Kraftwerk figured out the important thing about synthesizers, and Afrika Bambaataa was right there with them. The synthesizer wasn’t a tool, it was an aesthetic. It was a whole new kind of music. Mechanical, but full of soul.

When they first made movie cameras you might have thought - hey you can point this thing at a play and it’s just like being in the audience. But eventually we figured out the important thing about movie cameras was the way film could be chopped up and re-arranged to form a new visual language, the grammar of cinema, out of which emerged a whole new dimension of synthetic dreams.

Games have a job to do, beyond entertain us and make money. Games have always been about figuring out the beauty of logic, the meaning of systems, the purpose of computers. In game design, we search through the possibility space of possibility spaces, looking for those ghostly shapes that connect things to each other, things to us, us to the world, and us to each other. Game design is hard.

I don’t think you can add a drop of machine learning, with its deep, psychedelic weirdness, to the clockwork cathedral of a modern AAA videogame, which is a carefully-arranged puzzle-box of precisely interlocking gears. I think you have to start from scratch, start from the drop of deep, psychedelic weirdness and grow something completely new.