Doomer advisor joins Musk’s xAI, the 4th top research lab focused on AI apocalypse

Musk’s appointment of Hendrycks is significant because it is the clearest sign yet that four of the world’s most famous and well-funded AI research labs — OpenAI, DeepMind, Anthropic and now xAI — are bringing these kinds of existential risk, or x-risk, ideas about AI systems to the mainstream public.

Many AI experts have complained about x-risk focus

That is the case even though many top AI researchers and computer scientists do not agree that this “doomer” narrative deserves so much attention.

For example, Sara Hooker, head of Cohere for AI, told VentureBeat in May that x-risk “was a fringe topic.” And Mark Riedl, professor at the Georgia Institute of Technology, said that existential threats are “often reported as fact,” which he added “goes a long way to normalizing, through repetition, the belief that only scenarios that endanger civilization as a whole matter and that other harms are not happening or are not of consequence.”

NYU AI researcher and professor Kyunghyun Cho agreed, telling VentureBeat in June that he believes these “doomer narratives” are distracting from the real issues, both positive and negative, posed by today’s AI.

“I’m disappointed by a lot of this discussion about existential risk; now they even call it literal ‘extinction,’” he said. “It’s sucking the air out of the room.”

Other AI experts have also pointed out, both publicly and privately, that they are concerned by the companies’ publicly-acknowledged ties to the EA community — which is supported by tarnished tech figures like FTX’s Sam Bankman-Fried — as well as various TESCREAL movements such as longtermism and transhumanism.

“I am very aware of the fact that the EA movement is the one that is actually driving the whole thing around AGI and existential risk,” Cho told VentureBeat. “I think there are too many people in Silicon Valley with this kind of savior complex. They all want to save us from the inevitable doom that only they see and they think only they can solve.”

Timnit Gebru, in a Wired article last year, pointed out that SBF was was one of EA’s largest funders until the recent bankruptcy of his FTX cryptocurrency platform. Other billionaires who have contributed big money to EA and x-risk causes include Elon Musk, Vitalik Buterin, Ben Delo, Jaan Tallinn, Peter Thiel, and Dustin Muskovitz.

As a result, she wrote, “all of this money has shaped the field of AI and its priorities in ways that harm people in marginalized groups while purporting to work on ‘beneficial artificial general intelligence’ that will bring techno utopia for humanity. This is yet another example of how our technological future is not a linear march toward progress but one that is determined by those who have the money and influence to control it.”

Here is a rundown of where this tech quartet stands when it comes to AGI, x-risk and Effective Altruism:

xAI: ‘Understand the true nature of the universe”

Mission: Engineer an AGI to “understand the universe”

Focus on AGI and x-risk: Elon Musk, who helped found OpenAI in 2015, reportedly left the startup because he felt it wasn’t doing enough to develop AGI safely. He also played a key role in convincing AI leaders to sign the to sign Hendrycks’ Statement on AI Risk that says “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Musk developed xAI, he has said, because he believes a smarter AGI will be less likely to destroy humanity. “The safest way to build an A.I. is actually to make one that is maximally curious and truth-seeking,” he said in a recent Twitter Spaces talk.

Ties to Effective Altruism: Musk himself has claimed that writings about EA by one of its originators, philosopher William MacAskill, “is a close match for my philosophy.” As for Hendrycks, according to a recent Boston Globe interview, he “claims he was never an EA adherent, even if he brushed up against the movement,” and says “AI safety is a discipline that can, and does, stand apart from effective altruism.” Still, Hendrycks receives funding from Open Philanthropy and has said he became interested in AI safety because of his participation in 80,000 Hours, a career exploration program associated with the EA movement.

OpenAI: ‘Creating safe AGI that benefits all of humanity’

Mission: In 2015, OpenAI was founded with a mission to “ensure that artificial general intelligence benefits all of humanity.” OpenAI’s website notes: “We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.”

Focus on AGI and x-risk: Since its founding, OpenAI has never wavered from its AGI-focused mission. It has posted many blog posts over the past year with titles like “Governing Superintelligence,” “Our Approach to AI Safety,” and “Planning for AGI and Beyond”. Earlier this month, OpenAI announced a new “superalignment team” with a goal to “solve the core technical challenges of superintelligence alignment in four years.” The company said its co-founder and chief scientist Ilya Sutskever will make this research his core focus, and the company said it would dedicate 20% of its compute resources to its superalignment team. One team member recently called it the “notkilleveryoneism” team: