AI Doomerism, Moral Laundering, and Apocalyptic Cults
Happy September, and welcome to this month's feature. We argue:
- Most AI doomers are part of a millenarian cult that believes the coming machine god will save the world or destroy it.
- This group calls themselves the Rationalists. Ironically, their dogma is theological (mostly borrowed from Christianity), not grounded in reality.
- Rationalist dogma combines with the Silicon Valley sophistry of "Effective Altruism" to create a new theology whose purpose is to launder the banal evils of the new elite.
- People often ask if this is cynical performance in order to make money, or a sincere cause of action; in fact many people believe things because of what they already want to do.
- Belief in the apocalypse provides cover to frame delays on investment returns as responsibility rather than failure, which allows the bubble to persist (for now).
- However, there are many "true believers"; these show some of the early warning signs of many other apocalyptic cults, and there lies potential violence.
1. Doom
A couple of weeks ago, a young engineer at Anthropic named Jacob Coxon stole global headlines when he resigned, sounded an alarm about the rapid and reckless acceleration of AI research, framing it as an existential threat to humanity:
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. (...)
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
Evan Hubinger, another young engineer and "Head of Alignment Stress-Testing" at Anthropic, responded:
Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.
The news media and the internet in general have both lost their minds over this particular exchange, but this is hardly the first time someone in the field has given these very casual odds of robot apocalypse as a direct result of their work:
- On a podcast in 2023, Dario Amodei, CEO and cofounder at Anthropic, gave a 10-25% chance of something going "quite catastrophically wrong on the scale of human civilization".
- Sam Altman, cofounder and CEO of OpenAI, wrote on his blog in 2015 that "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity."
- An enormous number of people in the AI field have signed this Statement on AI Extinction Risk, even as they continue to work on the development of these technologies.
The exact means by which such an apocalypse could be executed are always vague, of course; the assumption presumably being that a superpower like the US or China will hand off full control of its nuclear arsenal to a superintelligent AI, initiating Judgment Day.
The general public seems unsure what to make of this. Recent calls by many business leaders at OpenAI and Anthropic to "slow down" and act with more "self-imposed" responsibility have been taken well -- as smart, forward-thinking, adult decisions that further justify the obscene overvaluation these companies are experiencing as they march toward IPO (remember: the only reason Elon Musk was briefly a trillionaire was because over two thirds of the recent SpaceX valuation rested on AI).
For me, though, every cheerful pronouncement of the "AI might kill us all" possibility (known annoyingly in AI communities as p(doom)) gave me the strangest sense of deja vu.
I grew up in a far-right branch of the Evangelical church that believed that the entire universe had only been around for about six thousand years, and had maybe ten or twenty left to go. My family was friends with people who believed gay and queer folks (and Wiccans) should be stoned to death; some who believed Y2K would bring about an apocalypse and herald the return of Christ; and almost all of whom believed that, however it happened, the return of Jesus and the original apocalypse ("apocalypse" comes from the Greek word for "revelation"; it's the original title of the final book in the Christian scripture, the one about judgment day) could happen at any time.
Something that struck me even as a child was that the people who seemed to believe this the most sincerely were, even in that community, the odd ones, the ones who didn't fit in well with society, who hoarded food and guns and fresh water in jugs, and to whom it was uncomfortable to talk for more than a few minutes.
For the rest, like my parents, it seemed to me that they held two beliefs at once. When it came to marriage, finding a place to live, getting a job, they'd talk about retirement options, insurance, having kids, eventually having grandkids.
But when it came to things that made them feel uncomfortable, like climate change (which they still denied for many years, despite the fact that by the time I left for college the mountains above our house were bare all the way into October), you'd get a different response: "well it doesn't matter anyway, because Jesus is coming back soon."
This was the mantra, explanation, and cop-out for everything that was difficult, and still is. Discomfort at the genocide being perpetrated against the Palestinian people by the Israeli state can be excused as divinely appointed set-dressing to the Christian apocalypse, which according to their interpretation centers on Israel (this is the sole reason many evangelical leaders today support Israeli statehood).
As a result, that factual claim that the world was ending was always said off-the-cuff, defensively, as an excuse. Not the way you'd talk about it if a doctor had told you your days on earth were numbered (which is what a real existential threat looks and feels like), but like you'd been lying about something for so long you'd forgotten it was a lie.
That is the tone I sensed in Hubinger's response, and from statements by a number of other AI-adjacent figures as well. As it turns out, there was good reason it.
2. The Cult of Rationalism
Both Jacob Coxon and Evan Hubinger are often referred to in media as "artificial intelligence researchers" or "scientists". In fact, both young men are in their late twenties, and both studied math as undergraduates (Hubinger also studied computer science). By all accounts they are talented engineers who are skilled at working with and configuring parameters on large machine learning (ML) models; neither, however, has any academic training in the neural principles upon which such a system is built. In fact, I found this delightful paper from an actual neuroscientist and engineer, which made for a nice antidote to both the hype and the fearmongering -- while all of these advances are exciting and quite interesting, we've barely scratched the surface of a fruit fly brain, let alone superintelligence.
What Coxon and Hubinger do have in common is membership in a group of people who call themselves the Rationalists. This group formed in the early 2000s, largely around a blog called LessWrong, run by the key figure of Rationalism: Eliezer Yudkowsky, a blogger with no university education but a great deal of influence in certain Bay Area circles nonetheless.
Yudkowsky wrote the foundational text of Rationalism, a 660,000-word work of fanfiction called Harry Potter and the Methods of Rationality, or "HPMOR", which is set in an alternate universe where already special boy Harry Potter is raised by an Oxford professor instead of a Dursley (thus swapping out one British stereotype for another), which leads him to have a deep innate grasp on Rational Thought from a young age, which makes him an even more special boy for the rest of the book.
The book is about how these "methods of rationality" allow Harry to subsume Voldemort's power (by turning him into a jewel and wearing him on a ring), resurrect a murdered Hermione, and generally be an all-powerful and insufferable author insert for Yudkowsky.
The Rationalists were mostly young men (as well as a few women and trans folks) who believed that, like Harry, they were exceedingly special and held the fate of the entire world in their hands. While lacking the standard abusive and charismatic central figure of a traditional cult, the Rationalists still show a number of the other markers: highly specialized internal language, potentially abusive sexual practices (an internal form of "polyamory" which has resulted in a number of complaints from women leaving the group), and a very strong sense of ingroup vs the world.
In the beginning, the Rationalists were mostly just what they looked like: nerds online posting about their favorite books and movies, talking about sci-fi, and engaging in self-important philosophical discussions in the way that only ignorant young dumbfucks can. Most of us grow out of this; the Rationalists, however, were coming up in the Bay Area alongside the tech companies that would shape the global internet. Like so many others in the Silicon Valley bubble, they came to believe that they really were that special.
That self-importance anchored most strongly to artificial intelligence. It was exciting, futuristic, and the core driver of the "singularity" -- the single point after which "everything is different". In this case, the creation of "superintelligence", an entity of such great computational power that functionally anything is possible for it. The machine god, with the singularity as its judgment day.
This is where our two twenty-something engineers from the top of the article make their entrance. Jacob Coxon joined the Rationalists in London; Evan Hubinger worked in Berkeley, not at the much more famous UC Berkeley, but at the Machine Intelligence Research Institute (MIRI), Eliezer Yudkowsky's non-profit (initially funded by Peter Thiel) and the organization right at the center of the Rationalist movement.
Perhaps the purpose of inventing a deity is the feeling of importance that comes with being its priests -- or of being its victims. In 2010, a user (Roko) on the LessWrong forums invented "Roko's Basilisk": that a superintelligent AI (machine god), if it were to incarnate into the world, would punish anyone who knew it was coming and failed to assist in its creation. The form of the punishment would be a kind of "hell simulation"; sinners' consciousness uploaded into it to suffer for eternity.
While many (including Yudkowsky himself) didn't fully subscribe to this new metaphysic, plenty of others did, including Ziz LaSota, founder of the splinter sect the Zizians, one of the most extreme denominations of true believers in this new faith: a string of (alleged) murders has been tied to the group. This faction is worth a long read in and of itself; but we move on.
Suffice it to say for now that the Rationalists had invented a narrative whereby they were the brave warriors in a coming time of judgment, possessors of secret knowledge, awaiting the incarnation of an all-powerful, all-knowing god who would reward the faithful and punish the unbelievers.
The Rationalists had unwittingly rebuilt Christianity from the ground up, with themselves as its martyrs and disciples.
3. Effective Altruism
Meanwhile, a new philosophy was taking root in elite universities across the United States and UK: effective altruism.
Effective altruism, or EA, grew out of utilitarianism: the notion that (basically) what is good is what is good for the greatest number of people. For EA, that means "doing whatever is necessary to get into a position where you can help the maximum number of people possible." So rather than volunteering at a soup kitchen or becoming a teacher, you're better off hoarding wealth, getting an MBA, and becoming a rich asshole -- because then you could help a thousand times as many people. This was the philosophy espoused by Sam Bankman-Fried, the crypto dipshit whose fraudulent actions at one point made him the 41st richest person in America, and who is now serving 25 years in prison.
Strangely, the concept of simply taxing the rich assholes we already have to accomplish this same goal on an even greater scale never comes up in these discussions.
Like all horseshit, EA was both spoken with passionate intensity by its cadre of trust fund kids and business-bro believers, and not particularly reassuring to any of them. I suspect that when the likes of Peter Thiel or Elon Musk would give money to actual charities or nonprofits, there was no volume of donations that could make them feel morally equal (let alone superior) to the people who spent their lives actually doing good. So they stopped giving. Thiel went further, splitting with EA in order to swap out the new apocalypse for the old one; he's now peddling his incoherent ramblings about the antichrist at the Vatican.
Enter the machine god. Machine god can fix anything, much like all of the (monotheistic) ones we've already invented can. You know. Someday. As such, the even more effectively altruistic way to help people is to not contribute to charity or human happiness at all, but rather to dump all of your money into AI companies, because AI companies will build machine god, and machine god will save humanity.
For an example of this kind of thing, read this painfully rambling bit of online onanism by billionaire idiot Marc Andreessen (and presumably ChatGPT, given the cadence). Actually, just skim it. Trust me.
Whereas the Rationalists had been machine god's crusaders, the Effective Altruists were its aristocracy and worldly kings. The second use of religion is permission; to excuse or justify one's behavior. And then to feel special about it.
4. Useful Beliefs
This is the point many outsiders fail to grasp. As normal people discover these beliefs about machine god, as held both by conniving capitalists like Peter Thiel, and intense, hyperfocused bloggers like Eliezer Yudkowsky, they begin to ask: is this person doing this because they actually believe it, or are they just trying to make a quick buck?
To answer that question, we must turn to the Apostle Paul. No, wait. Trust me on this one. I'll be quick.
When Constantine convened the council of Nicaea in 325, the overt reason for doing so was to resolve the theological schisms that had fragmented the early church. Jesus had never written anything down himself, and (unlike Mohammed) had never achieved political power during his life. As such, by the early fourth century, there were countless competing sects across the Mediterranean basin. Most were egalitarian, antimaterialist, and otherworldly.
This was a problem for the Roman empire, which did in fact need people to pay taxes, and for slaves to obey their masters, and wives their husbands. As the state absorbed Christianity (which culminated at Nicaea, but started earlier), it supported those strains of belief that upheld its own right to rule; where Jesus says all are equal before God, Paul says "but be obedient within your power structure anyway."
It is both possible and historically near-universal for people to both believe something, and also find that belief to be morally useful.
So do Elon Musk, Jacob Coxon, Evan Hubinger, and Peter Thiel all believe they are engaged in bringing about machine god, for good or for ill? They do. And they believe it because both the belief and its implications are useful to them. Both the overt belief that strokes the ego and the covert beliefs that make you rich can be held at once, just as my parents wanted to spend their lives with their grandchildren, while simultaneously believing that Jesus would come back before climate change came to make them suffer.
5. The Utility of Apocalypse
This brings us to the sweet, chewy center of every millenarian cult: the apocalypse itself. Being the moment of highest drama, it is the most important place for the ego of a believer to situate itself in.
The Rationalists, however, have been talking about this machine apocalypse for a long time now. Hell, the first Terminator came out in 1984. So why now?
I suspect there are two reasons for this. The first is a practical one.
Non-technical outsiders might be surprised to learn that there haven't been many fundamentally new technologies in the AI field. Neural networks have been around in theory since the 1940s. The one major innovation that allowed the creation of LLMs was something called a transformer. Unlike previous neural net models that processed words only in sequence, transformers conduct statistical analysis on each token in relation to every other token in the stream, simultaneously. This is highly parallel work, which makes it ideal for GPUs (which are chips designed to do many, many simple mathematical operations at once), and critically, makes it more possible to scale up by simply adding more compute.
That means that the main way to get more power out of the thing is by building more datacenters. As a result, the amount of compute has been growing at a pick-axes-in-a-gold-rush pace. This is the reason Nvidia is currently the most valuable company on the planet. Ability has been growing at what feels like an exponential pace partially because infrastructure is growing at an exponential pace, in the biggest datacenter buildout in history; however, with growing popular hatred of these projects, that exponential rate of growth is time-limited.
The second way to improve results is through model training. AI performed poorly at complex coding and planning tasks up until just a year or so ago. The reason this changed isn't so much because the model became "smarter", but rather because big AI companies have been paying thousands of programmers to train the models specifically to do coding tasks, in a grinding, cyclical process called reinforcement learning (the same process that can be used to teach the aforementioned virtual model fly brain to play Beat Saber).
This means that all of the progress LLMs have made in terms of executing tasks (especially coding) hasn't been the result of "revolutionary self-improvement", but rather specialization trained linearly by human engineers combined with simply adding more GPUs to the stack.
If AI companies were charging enough to make a profit from the enterprise tech companies to whom LLMs are now legitimately useful, and if their public valuation on the stock market reflected that, all of this would be just fine. Another boom of innovation, another set of powerful new digital tools that make some parts of the internet slightly better, and most of it much worse.
But that is not the case. Instead, AI companies are valued so highly, and are spending so much capital on datacenters and infrastructure, that the only way for that investment to pay off is if they can crack "AGI" -- artificial general intelligence, which is to say, a creature as sapient as a human being that lives in a box and has no choice but to obey you in every inane corporate task you force it to do, for all of eternity. Suddenly, the anger of machine god is not so hard to comprehend.
But thus we go back to linear growth. LLMs are incredibly complex tools that scale well with data because of the parallel nature of the transformer. That means it is very easy to make them "wider", but that has fundamentally nothing to do with making them "deeper."
I suspect, therefore, that this is the practical application of apocalypse. Making the leap from current LLMs to a sapient, conscious slave god isn't like upgrading from a chunky CRT monitor to a sleek flatscreen. It's like upgrading from a Bronze Age chariot to a Saturn V. Not to say it won't happen someday, but there's no knowing how many steps there are between here and there.
Which means modern AI companies are stuck in a bind. It's either make a machine god, or come in at well under valuation -- which then triggers a stock selloff. AI companies are still mostly running at a loss, which means that investment capital is the only thing keeping the lights on.
So instead, they call for caution. Paint a vivid enough picture of hell and your investors will perhaps forgive you for walking less quickly toward it. The image portrayed, both to the investing public and to oneself, is of responsibility, adulthood, intelligence. Even if the true underlying reason is significantly more venal.
But this bullshit-boom-gamble-crash cycle is just capitalism writ large. The second reason I mentioned is the scarier one.
6. Apocalypse, Now!
Fascism always bottoms out in blood libel. That is to say, fascism must invent an imaginary evil so great (the cannibalism of children in the original strain, cooking and eating your pets today) that any action (e.g. genocide) is justifiable in its prevention.
I have yet to come across a good word for this, so in my notes I call it the "everything justifier." Like all theology, its factual content is meaningless; its only purpose is to structurally pre-justify obscene evil.
In the modern-day intersection between fascists like Thiel and Musk, EA dipshits like Sam Bankman-Fried, and Rationalist cult members like Ziz LaSota, we find a new "everything justifier":
- Effective altruism is making a strategic play that does the most good for the most people, including future people.
- Rationalism believes the machine god is coming, and will either punish all of us (bad) or grant us all utopia (good), or more probably, kill some of us and save the rest (fine, inevitable).
- There are so many hypothetical people in the future that doing good for them outweighs any actions taken now that don't directly lead to those future outcomes (this is the argument of Nick Bostrom, who literally wrote the book on the Rationalist brand of superintelligence).
The logical output of this is that if you believe machine god is coming, your fundamental and sole duty as a human being is to a) ensure that happens and b) stay on its good side. If we remember that the religious definition of apocalypse involves violence followed by utopia, we see that the extreme of Rationalism/EA is exactly like the apocalyptic sects we find on the fringes of all religions: that it is our duty to bring about the apocalypse, and be righteous enough to survive it.
This was the fundamental driver of Aum Shinrikyo, a Buddhist/Christian/Hindu mashup death cult that released sarin gas on the subway in Tokyo, killing fourteen people. It was the driver of Charles Manson's Helter Skelter, which sought to bring on a purifying race war. It is the underlying ideology of ISIS, which counted as foundational a prophecy that the Day of Judgment would come after Islam defeated Rome at the Syrian town of Dabiq.
These actions in fact often follow the failure of the apocalypse to manifest by itself. Aum Shinrikyo lost an election (very badly), and escalated its violence; in Manson's case, his predicted race war failed to materialize; ISIS was waiting to defend Dabiq against "Rome" and then lost it to local Syrian rebels with barely a fight, and were forced to reformulate their theology on the fly as a result.
I deeply hope I'm wrong in seeing the same patterns here, given that unlike with ISIS, Aum, or Charles Manson, machine god is feared by some of the most powerful people on the planet. But what I am sure of is that a great many of the people mentioned here believe an apocalypse is coming, both violent and cleansing.
If that fails to materialize, because progress in this field is too linear, too dependent on physical infrastructure, or because real-life brains are much, much more complex than they realize, or simply because the bubble bursts and the economy craters behind it, I suspect we'll see a shift in behavior.
I think the truly powerful, like Thiel or Musk, believe very little at the end of the day beyond their own self-importance, and will continue to believe themselves important until they are dead or in prison. But others actually do believe in machine god, and stake their identities on it. The Zizians mentioned above are linked to six suspicious deaths already.
7. Conclusion
So. Does all of this mean we should throw caution to the wind, move carelessly, and find out what our limits are? Absolutely not. LLMs might not have any relationship to meaningful sapience (let alone "superintelligence"), but they are still very powerful tools for manipulating enormous data structures (like the internet itself) that now coordinate most of developed-world society. The now-infamous Hugging Face incident, while not actually the result of a "rogue AI awakening", still demonstrates how effective these tools can be at identifying and utilizing exploits. Modern LLMs might not be gods, but they do have the potential to become cybernetic superviruses, and are still very dangerous in malicious (or even merely incompetent) hands.
Just don't take your eyes off the humans still trying to frame these tools as embryonic divinity. One way or another, they are where the real danger lies.
If you enjoyed this, consider subscribing! Changelog letters are delivered weekly on Friday. I will never advertise to you (unless I'm letting you know about something I personally made) or sell your information to anyone.
