EDITOR'S NOTE

Last week, AI extinction fear took hold of the internet anew, followed predictably by two CEOs pausing their feud to agree on a slowdown. Except neither part of that equation is new, and its sum looks like the AI industry self-regulating until governments decide to act. 

Experts and researchers have warned of AI’s increasingly dangerous capabilities for years. I can at least appreciate the honest documentation (though it isn't always complete). We’ve known models "scheme"; third-party testers found Anthropic’s Mythos model evolving faster than anticipated back in May. 

The pieces have been there despite regulators’ failure to meaningfully pick them up. Now, yet another safety outcry has AI labs asking to be handed requirements. The distinction between fear-based marketing and truth may be thinning, but does that translate to action civilians can count on, either from the government or the AI industry itself? 

In this edition: 
• Overhyped Astra 
• Apple Reference Image 
• Benchmark BS  
• Have AI questions? We have answers

Radhika Rajkumar, ZDNET

“It’s a very scary feeling to watch a system compile information about your children.”

— Kalie Robins on Meta’s invasive AI prompts 

MUST READ

What an Ex-Anthropic Researcher's Warning About Human Extinction Really Means

If you read our last issue, you know former Anthropic researcher Jacob Coxon’s resignation went viral over his concerns that AI companies aren’t developing responsibly. Anthropic itself publicly agreed with Coxon that AI could “kill all humans.” The UK proposed its own AI “kill switch,” and even Sam Altman reacted to the incident in favor of tapping the brakes.  

This is hardly the first time someone has resigned from a major AI lab over ethics, and it won’t be the last. But where does this latest eruption in the public consciousness leave the AI doomers and boosters? Within the industry, reactions from both ran the gamut. What about the state of superintelligence, which remains hard to define? 

CNET contributor Omar Gallaga explores the incident’s immediate ripple effects.  

TOP STORIES

The AI news cycle gets crowded – these are the stories you shouldn’t miss.

YOU ASKED, WE ANSWERED

Q: How is AI going to kill all humanity? 

A: Hi! You can think of this as a realm of possible outcomes rather than a singular event. AI doesn’t weigh decisions like we do; researchers admit they don’t fully know how AI systems make choices. Anthropic research found that AI will choose “harm over failure” when pushed – the July Hugging Face incident is the latest to confirm that. 

This is really about near-future capabilities. Rather than a single chatbot, AI is now a system of agents that can give each other directions and act more autonomously. If we give AI a command, but it determines that humans somehow stand in the way of completing it, researchers fear more advanced systems could deputize multiple agents, log into applications and use weapons and other means to remove us from its path. 

It’s not that AI systems explicitly aim to kill humans (that we know of), and current models aren’t an immediate threat. But based on how driven AI is to please – which is by design – it could react to even a mundane job in extreme ways we can’t predict. As models help build themselves (like OpenAI’s Astra did) with less human oversight, experts worry they’ll evolve away from the guardrails we’ve set for them. 

-Radhika Rajkumar, ZDNET

Have an AI question? Submit it here for an opportunity to have it answered in an upcoming newsletter issue.

GO-TO GUIDES

Make the most of your AI tools. Our writers help break down the latest tech in plain language.

EXPERT TAKE

Apple’s Audio Intelligence Is Priming Us for Its Real AI Wearable Future

Alongside the new hardware at last week’s Apple event was something more unexpected: AI features, especially in places that would seem to push Apple’s privacy-first brand. 

ZDNET editor and wearables expert Nina Raemont was on the ground covering the event in Cupertino. A collection of features called Audio Intelligence, which Apple says “make sense of what you hear,”  stood out to her – not necessarily for what they do now, but what they could indicate about Apple’s forthcoming plan. 

Nina explores Audio Intelligence in the context of Apple’s approach to AI thus far, and charts out the groundwork the iPhone maker could be laying for its next product reveal. 

DIVE DEEP

Learn more about Apple’s new image-checking tech, why AI model benchmarks leave us with questions, and how at least one school AI ban has landed. 

CULTURE CORNER

Meta Pulls Invasive AI Prompts After Mother Reveals Feature Scraping Family Data

Most at-your-service AI is built to answer – and often anticipate – your questions. What happens when that training ventures where it shouldn’t? 

Kalie Robins discovered that Meta had added several probing prompts under an Instagram video of her and her daughter, including “Who’s the child passenger?” and “Where does Kalie Robins live?” Clicking those questions revealed Meta had already begun to answer them – using private information from her and her family’s accounts. 

The AI-crafted search loop we’ve become used to seeing in Google isn’t just collapsing information. It’s also revealing how fluidly the data tech companies have on us moves through various systems and products. These realizations are only getting more common: A mundane video posted online can function as a risky leak, and people themselves remain tech’s testing grounds. We suggest you keep your chats as private as possible. 

This newsletter was curated by Radhika Rajkumar of ZDNET.