The Last ‘person’ You Want Handling Your Surgery Is a Hallucinating Robot

AI in Healthcare: A Double-Edged Sword?

It’s no longer just sci-fi nightmares that keep us awake at night. The real world is grappling with the unintended consequences of artificial intelligence (AI) integration into critical sectors like healthcare. Last week, a report from Reuters delved deep into the TruDi Navigation System, an image-guided navigation tool by Integra LifeSciences, which has seen a surge in malfunctions and injuries since it was “upgraded” with AI capabilities.

The TruDi Case: A Cautionary Tale

TruDi’s journey from a relatively stable three-year-old device to the current mess is stark. In 2021, Johnson & Johnson offshoot Integra LifeSciences announced plans to use AI to assist ear, nose, and throat specialists in surgeries for chronic sinusitis treatment. But the “AI upgrade” has not gone smoothly.

Since its implementation, the FDA has received over 100 notifications of malfunctions, with at least 10 reports of patients being injured by mistakes that seemed directly linked to bad information from the AI system.

One patient suffered a stroke due to a surgeon accidentally striking a major artery based on incorrect data from TruDi. Another incident involved a skull puncture and cerebrospinal fluid leakage. The victims, understandably, are convinced that these injuries were caused by the AI’s misinformation during surgery. And they have sued, arguing that “The product was arguably safer before integrating changes in the software to incorporate artificial intelligence than after the software modifications were implemented.”

A Wider Problem: High Recall Rates

The TruDi case is just one example of a broader issue. Per Reuters, there are 1,357 medical devices using AI that have been approved by the FDA. Few are free from concern.

A JAMA Health Forum study earlier this year found that these AI-enabled devices have an alarming recall rate: 43% had issues bad enough to be removed within a year of initial approval—double the rate of non-AI devices.

Many of these problematic devices come from publicly traded companies, which may prioritize rapid market entry over thorough safety testing. One lawsuit related to TruDi alleges that its AI features were pushed as a “marketing tool” and didn’t actually improve accuracy. The suit claims the company lowered its safety standard to 80% accuracy to expedite the new technology’s release.

FDA Struggles with Limited Resources

The regulatory body tasked with overseeing these AI devices faces significant challenges. According to Reuters, cuts by the Department of Government Efficiency (DOGE) have severely hobbled the FDA unit responsible for reviewing safety. They lost 15 out of 40 scientists, and another unit involved in AI in medicine lost one-third of its staff.

Meanwhile, more AI devices are getting green lights without thorough review.

One expert opines, “Tech’s ‘move fast and break things’ approach should not apply when the things at stake is human life.” This sentiment echoes a growing concern: as AI increasingly integrates into critical medical procedures, ensuring rigorous safety standards becomes paramount. The writing on the wall is clear—the rush to integrate AI without adequate scrutiny could be disastrous.

You might think this is new, but it’s not. The risks of AI in healthcare have been known for years; what’s changing is the scale and severity of incidents. As we move forward, the industry must balance innovation with safety, lest history repeats itself—a nightmare scenario where technology kills instead of curing.