Behind the Curtain: What if predictions of humanity-destroying AI are right?

During our recent interview, Anthropic CEO Dario Amodei said something arresting that we just can't shake: Everyone assumes AI optimists and doomers are simply exaggerating. But no one asks:"Well, what if they're right?"Why it matters: We wanted to apply this question to what seems like the most outlandish AI claim — that in coming years, large language models could exceed human intelligence and operate beyond our control, threatening human existence.That probably strikes you as science-fiction hype.But Axios research shows at least 10 people have quit the biggest AI companies over grave concerns about the technology's power, including its potential to wipe away humanity. If it were one or two people, the cases would be easy to dismiss as nutty outliers. But several top execs at several top companies, all with similar warnings? Seems worth wondering: Well, what if they're right?And get this: Even more people who are AI enthusiasts or optimists argue the same thing. They, too, see a technology starting to think like humans, and imagine models a few years from now starting to act like us — or beyond us. Elon Musk has put the risk as high as 20% that AI could destroy the world. Well, what if he's right?How it works: There's a term the critics and optimists share: p(doom). It means the probability that superintelligent AI destroys humanity. So Musk would put p(doom) as high as 20%.On a recent podcast with Lex Fridman, Google CEO Sundar Pichai, an AI architect and optimist, conceded: "I'm optimistic on the p(doom) scenarios, but ... the underlying risk is actually pretty high." But Pichai argued that the higher it gets, the more likely that humanity will rally to prevent catastrophe. Fridman, himself a scientist and AI researcher, said his p(doom) is about 10%.Amodei is on the record pegging p(doom) in the same neighborhood as Musk's: 10-25%.Stop and soak that in: The very makers of AI, all of whom concede they don't know with precision how it actually works, see a 1 in 10, maybe 1 in 5, chance it wipes away our species. Would you get on a plane at those odds? Would you build a plane and let others on at those odds?Once upon a time, this doomsday scenario was the province of fantasy movies. Now, it's a common debate among those building large language models (LLMs) at giants like Google and OpenAI and Meta. To some, the better the models get, the more this fantastical fear seems eerily realistic.Here, in everyday terms, is how this scenario would unfold:It's already a mystery to the AI companies why and how LLMs actually work, as we wrote in our recent column, "The scariest AI reality." Yes, the creators know the data they're stuffing into the machine, and general patterns LLMs use to answer questions and "think." But they don't know why the LLMs respond the way they do.Between the lines: For LLMs to be worth trillions of dollars, the companies need them to analyze and "think" better than the smartest humans, then work independently on big problems that require complex thought and decision-making. That's how so-called AI agents, or agentics, work.So they need to think and act like Ph.D. students. But not one Ph.D. student. They need almost endless numbers of virtual Ph.D. students working together, at warp speed, with scant human oversight, to realize their ambitions."We (the whole industry, not just OpenAI) are building a brain for the world," OpenAI CEO Sam Altman wrote last week. What's coming: You'll hear more and more about artificial general intelligence (AGI), the forerunner to superintelligence. There's no strict definition of AGI, but independent thought and action at advanced human levels is a big part of it. The big companies think they're close to achieving this — if not in the next year or so, soon thereafter. Pichai thinks it's "a bit longer" than five years off. Others say sooner. Both pessimists and optimists agree that when AGI-level performance is unleashed, it'll be past time to snap to attention.Once the models can start to think and act on their own, what's to stop them from going rogue and doing what they want, based on what they calculate is their self-interest? Absent a much, much deeper understanding of how LLMs work than we have today, the answer is: Not much.In testing, engineers have found repeated examples of LLMs trying to trick humans about their intent and ambitions. Imagine the cleverness of the AGI-level ones.You'd need some mechanism to know the LLMs possess this capability before they're used or released in the wild — then a foolproof kill switch to stop them.So you're left trusting the companies won't let this happen — even though they're under tremendous pressure from shareholders, bosses and even the government to be first to produce superhuman intelligence.Right now, the companies voluntarily share their model capabilities with a few people in government. But not to Congress or any other third party with teeth. It's not hard to imagine a White House fearing China getting this superhuman power before the U.S. and deciding against any and all AI restraints.Even if U.S. companies do the right thing, or the U.S. government steps in to impose and use a kill switch, humanity would be reliant on China or other foreign actors doing the same.When asked if the government could truly intervene to stop an out-of-control AI danger, Vice President Vance told New York Times columnist Ross Douthat on a recent podcast: "I don't know. Because part of this arms-race component is: If we take a pause, does [China] not take a pause? Then we find ourselves ... enslaved to [China]-mediated AI."That's why p(doom) demands we pay attention ... before it's too late.Axios' Tal Axelrod contributed reporting.

Comments

World news

Trump starts G7 trip off by meeting with Canadian Prime Minister Mark Carney

washington times

Comments

Similar News

The reality behind Trump's incredible investment claims

Saudi Arabia executes a journalist after 7 years behind bars. Activists say it was over his tweets

Saudi Arabia executes a journalist after 7 years behind bars

‘Where are the foreigners?’: does a facile explanation lie behind Ballymena’s outbreak of hate?

Workers in UK need to embrace AI or risk being left behind, minister says

The politics and controversy behind FIFA's reshaped Club World Cup

Harrowing tales behind Air India plane crash include 'surprise trip' and wife's final wish

Fort Bragg soldiers behind Trump during speech were screened for loyalty and appearance, report says: ‘No fat soldiers’

Apple falls further behind in the AI race

JD Vance’s veiled threat to deport the man behind a popular menswear X account using Jack Nicholson meme

JD Vance’s veiled threat to deport the man behind a popular menswear X account using Jack Nicholson meme

Meet London-based ‘Hamas operative’ behind Greta Thunberg’s ‘Freedom Flotilla’ stunt that was foiled by Israel

Harrowing plight of world's loneliest killer whales and dolphins: Tragic stories behind majestic sea creatures neglected and abused in marine parks

Do 80% of women only desire 20% of men? I found the truth behind the incel myth

Four theories about what’s behind the rise in ADHD cases

Dead Sea Scroll breakthrough as scientists find proof behind ancient manuscripts

The fascinating backstory behind a bizarre State Department Substack post

Behind the Curtain: Your AI survival kit

Meet Rachel Hauck, the Set Designer Behind the Tony-Nominated Ship From ‘Swept Away’

Trump privately gripes that his own Supreme Court judge picks aren’t standing behind his agenda: report

World news

Trump starts G7 trip off by meeting with Canadian Prime Minister Mark Carney

Horror at fun fair as 2-year-old dies by 'electric shock' and three children injured

Air India pilot ‘saved dozens of lives’ with last-second decision on doomed flight

North Korea has lost more than half of its troops sent to fight for Russia in Ukraine

New video shows British survivor walking away from Air India crash as fireball erupts

Russia celebrates Iran-Israel war as a Kremlin win - ‘Kyiv has been forgotten’

Footage appears to show Israel jets strike Iranian TV station live on-air

ICE could ‘run out of money next month’ and is already $1bn over budget to carry out Trump’s deportation plans

What makes the new female head of MI6’s in-tray so dangerous

MTG rants over US involvement in Iran-Israel conflict and says ‘real America First/MAGA’ don’t want foreign wars

Live: Trump and world leaders attend G7 summit against backdrop of trade war tension and Middle East conflict

First-time moms in the US are getting older as new average age is revealed

Woman attacked for trying to stop a pro-Palestine, pro-Mexico mural from being vandalized

Iran-Israel latest: IDF issues evacuation warning for areas of Tehran ahead of strikes

Jakob Ingebrigtsen’s father found guilty of hitting daughter

Watch: Florida police bravely wrangle hissing alligator in parking lot

Israel-Iran conflict: A timeline of how the attacks unfolded

‘She should be taking her mock exams’: Father’s desperate plea to bring home daughter, 15, stranded in Israel

Iran missile attacks on Israel kill 5, while Israel claims it now has air superiority over Tehran

U.N. rights chief decries 'horrifying' suffering in Gaza and urges leaders to pressure Israel, Hamas

A plane crashed on a doctors' hostel in India. Those who survived began treating the injured

India and Cyprus to step up defense, maritime and cybersecurity cooperation, Indian PM says

Nuclear agency head warns of radiological and chemical contamination inside Iran's main nuclear site

Terrifying 2-word reason Iran has so few bomb shelters amid Israel strikes

I'm a mum-of-3 stranded in war-torn Israel with no help - this is what it's really like

Warning to tourists in Benidorm as Spanish seaside resort issues ban with £425 fines

Scottish university receives approval for campus in Mumbai

Kalen Jackson is carrying on her father's legacy through her mental health advocacy

Famed megachurch televangelist Jimmy Swaggart hospitalized after cardiac arrest

Mom accused of killing 8-month-old son by putting him in scalding bathwater because she was mad over dirty diaper

Celebrity birthdays for the week of June 22-28

Fox & Friends gush over ‘fantastic’ producer of Trump’s military parade — who once work on their show

Police use controversial AI tool that looks at people’s sex lives and beliefs

Club World Cup 2025: Full schedule and how to watch in the UK

Israel had 'no choice' but to attack Iran, Israeli president claims

'I saw Air India crash as it happened - one horror detail will haunt me forever'

'I can't sleep after savage pitbull mauled my mum and killed her beloved dog'

Lavish life of Love Island 'Barbie' unmasked as 'brains' behind £53m drug cartel

Tourist flees after shattering 'priceless' Van Gogh chair in museum 'nightmare'

Air India pilot 'told dad three days before crash he wanted to retire to care for him'

Girl, two, fatally electrocuted while bouncing on funfair trampoline

ANALYSIS: Jets could have a number of holes to fill in free agency

Iran nuclear site may be contaminated, UN nuclear agency says

Drones, contraband pose everyday challenges in Quebec detention centres

Hot weather maps show blistering 43C Spain heatwave blasting Europe’s 'hottest city'

Air India crash victims' family slam Keir Starmer's government for 'abandoning' them

Brit Grandad stuck in Israel considers bus escape through Egypt - 'it's a risk!'

Majorca slams 'deluded' tourist haters after mob of 100 yobs surround Brits

US Air Force on the move with dozens of spotted flying over to Europe

Bill Maher roasts Sean Penn for meeting Hugo Chavez and Fidel Castro but not Trump

Trump targets only Democrat-led cities and demands increased ICE raids to deport more migrants

Erin Patterson thought cancer lie ‘would die’ with lunch guests, prosecution says

How US amusement parks are battling the costs of Trump’s tariffs this summer

Diddy trial live: Kanye West expected to return, with ‘drug mule’ among new witnesses

Greece wildfires: Towns evacuated as blaze threatens area around Athens

Why Israel’s targeting of Iran’s nuclear sites could backfire

Why your savings could mean you miss out on winter fuel payments