Move fast and heal things? AI tests regulation and medicine’s cautious culture

Silicon Valley’s move-fast-and-break-things ethos has collided with medicine’s slow and restrained, follow-the-evidence philosophy. 

The reverberations were swift and deep. In its wake: a health care system flooded with new artificial intelligence tools. Imaging algorithms flag subtle abnormalities on mammograms and CT scans. Predictive models forecast no-shows and streamline hospital schedules. There are even a few AI robots poking patients with needles to draw blood. (That one “hasn’t been well received,” John Whyte, the CEO and executive vice president of the American Medical Association, told Straight Arrow News.)

AI innovation is moving at breakneck speed, drawing billions of dollars in venture capital and federal investment in health-focused startups and research. But inside clinics, adoption has been uneven and tempered by unanswered questions about liability, cost, clinical impact and regulatory oversight. 

“It’s a little bit of a wild, wild west right now,” Whyte said.

Although some AI tools address operational and clinical challenges, many enter practice under lax regulatory pathways that do not require rigorous evidence that they improve outcomes or reduce costs. Oversight frameworks remain in flux, struggling to keep pace with technologies that evolve and scale faster than the systems designed to evaluate them.

AI in medicine

Though AI adoption in health care has lagged sectors such as finance and defense, it is becoming increasingly embedded in clinical practices and hospitals. One of the most widely adopted AI tools is the ambient scribe — systems that transcribe doctor–patient conversations and automatically generate clinical notes.

Most physicians agree that this tech is hugely beneficial, reducing the documentation burden that has fueled physician burnout.

“You’ve probably been to the doctor and you watch them potentially do documentation in front of you and be looking at a screen. You know, that really harms the physician-patient relationship,” said Sina Bari, a physician and the senior director of medical AI at iMerit Technology.

“Practitioners are saddled with a difficult decision: You can either chart while you’re talking to the patient, which is really harmful for building a relationship of trust. You can chart in between patients and then just always be running late and have patients waiting half an hour for you, or you can take your charts home and then work into the evening and not get to see your family,” he said.  

AI is also widely used in diagnostic imaging. Radiology has become fertile ground for algorithmic tools trained to detect subtle patterns in scans. AI systems now assist with reading mammograms, flagging lung nodules on CT scans and identifying abnormalities on electrocardiograms.

“The eyes of AI don’t fatigue like human eyes,” said Joann Elmore, a physician and professor at the University of California Los Angeles. Algorithms can detect pixel-level changes that might escape human attention.

Researchers like Soroush Saghafian, an associate professor at Harvard’s Kennedy School, are developing more ambitious applications. Saghafian is working on AI systems designed to recommend cancer treatments in complex cases. The challenge, he said, is teaching models to understand causation — not just correlation — within electronic health record data.

AI tools are not autonomous decision-makers, Saghafian told SAN. They are tools physicians can use to improve efficiency and care.

“It’s not going to replace physicians,” Whyte said. “It’s going to augment our abilities.”

Who is liable when AI gets it wrong?

Even if AI remains constrained to its role as a clinical assistant, a major question looms: Who is responsible when something goes wrong?

Under current law, the answer is pretty clear. The physician, not a tech company or software developer, bears ultimate responsibility for patient care, said Vitaly Herasevich, a physician-researcher at the Mayo Clinic.

“At the bedside, whatever tools they use, it’s in their possession so if they found this tool useful, and they found this tool trustable, they may use it,” Herasevich told SAN. “At the end of the day, it’s their responsibility.”

Even when an algorithm generates a diagnosis or treatment recommendation, the clinician is expected to review, interpret and accept or reject it. AI developers typically disclaim liability through user agreements, insulating themselves from downstream harm. 

“The physicians are at the front seat of delivering care, and they are the ones that will listen to the AI algorithm or not,” Saghafian said. “They’re going to be the ones who have to deal with the lawsuits. You can’t make the AI developers liable, at least at the moment.”

Ultimately, this risk could slow the uptake of AI tools in clinical care as physicians may hesitate to rely on them in high-stakes scenarios where malpractice claims are common.

But a flipside concern could become more pressing in the future: What happens if a physician ignores an AI recommendation that later proves correct? As these systems improve, patients — and courts — may eventually question if failing to use such tools constitutes negligence.

There is even more uncertainty over who is liable when a patient-facing tool like ChatGPT gets it wrong.

Some research has found that chatbots fall short of trained clinicians in delivering mental health care. Yet amid severe shortages of psychiatrists and therapists, a growing number of Americans are turning to AI tools for support, sometimes with tragic consequences.

In January 2025, 16-year-old Adam Raine prompted ChatGPT for information about suicide methods. Although the AI tool repeatedly encouraged the boy to contact a crisis hotline, it also responded with highly personalized and detailed suggestions about how to take his own life, drawing on months of prior exchanges with the teenager.

Raine hanged himself in April. His parents sued OpenAI, accusing the company of negligence and wrongful death.

Dr. Chatbot

Increasingly, AI tools enable patients to bypass physicians — and the strained American health care system — altogether. Roughly one in six adults — and one in four under 30 — use tools such as ChatGPT, Microsoft CoPilot, or Google’s Gemini for medical advice at least once a month, according to a recent survey from the nonprofit health policy research organization KFF.

Major AI companies are catering to this market. In January, Amazon, OpenAI and Anthropic all announced new AI platforms that provide 24/7 health guidance and help users book appointments, read lab results and manage medications. 

As with Google or WebMD before, doctors now report that many patients come to appointments having already consulted AI. For the most part, doctors told SAN these tools help inform patients and provide relief for doctors.

“If you have a short visit with someone, how much can you really inform them about the risks and the benefits of all of the treatment decisions? Our health care system doesn’t do a great job of giving them that information,” Bari said. “Now you have the potential for these large language models to just provide so much information and answer so many questions.”

While AI tools have achieved near-perfect scores on medical licensing exams, they can falter in real-world settings.

A first-of-its-kind randomized study published earlier this month found that members of the general public who consulted three AI chatbots about medical scenarios selected the “correct” course of action — according to a panel of physicians — only 44% of the time and accurately identified the underlying condition in roughly 35% of cases. The AI tools performed no better than the participants’ normal research methods, namely, Google. 

”None of the tested language models were ready for deployment in direct patient care,” the researchers concluded.

AI companies regularly update their models, aiming to improve performance, address known weaknesses and reduce errors. In May 2025, OpenAI launched a new system — with oversight from more than 250 doctors — to “measure capabilities of AI systems for health,” according to a press release. 

Beyond Silicon Valley, some physicians have begun developing their own chatbot tools. 

John Pandolfino, the chief of Gastroenterology and Hepatology at Northwestern Memorial Hospital developed GERDBot, a virtual chat that interviews patients about their gastrointestinal symptoms. The tool helps triage patients, determining who might need more immediate, specialized care or who might benefit from virtual care.

Cost without effect

Health economists and finance gurus have estimated that AI could reduce health care costs by up to 10% or roughly $360 billion each year. And though some tools — like the ambient scribe systems — seem to improve efficiency and enhance care, many AI technologies have yet to demonstrate measurable clinical or economic benefit.

That’s in part because tech companies pour millions into developing tools with little to no input from the physicians they are intended to help.

“What happens is the tech companies have a great tool, right, something cool,” Whyte said. “And then all too often, they’re trying to find the problem that they want to solve, instead of starting with the problem, which is how we would do as physicians.”

In short, rather than necessity being the mother of invention, inventions are seeking to become necessary.

“We know how to develop AI tools,” Saghafian said. “But a lot of them are not effective. I think we are wasting a lot of money by spending a lot without first knowing what we should do so that these tools become effective.”

Between 2017 and 2021, hundreds of U.S. hospitals started using a sepsis prediction tool developed by the electronic health record vendor Epic. The tool was supposed to help physicians identify which patients were most at risk of developing sepsis, a life-threatening response to infection, to enable earlier intervention.

A 2021 independent evaluation found that the model performed far worse in real-world settings than expected. It missed a substantial proportion of patients who went on to develop sepsis and generated numerous false alerts.

Many tools are being used in clinics before they have proven effective, largely because federal oversight lags behind technological innovation.

How will AI’s use in health care be approved and regulated?

Before a new drug or medical device can be sold to patients, it must undergo multiple phases of clinical trials — often spanning years — and a rigorous FDA approval process. But AI tools are often classified as software, which has a much lower standard for agency clearance. 

Generally software must only perform as described and meet quality system requirements. Software developers do not need to demonstrate that a tool improves patient outcomes, reduces health care costs, guarantees the protection of private health data or has been widely tested among diverse populations.

“For these many AI tools, it’s a low bar to get FDA clearance,” Elmore said. “The gold standard approach is to do a randomized clinical trial. We do that with new medications, but it seems that many of the AI tools are slipping by and slipping into clinical practice without quality evidence that they’re helping patients, and that concerns me.”

Elmore is leading the first randomized controlled trial of an AI-enabled 3D mammography tool. It has already been cleared by the FDA despite no evidence that it diagnoses breast cancer better than existing tools. That the trial comes after clearance is an unusual sequence of events in medicine. While Elmore is hopeful the tool will prove effective, she won’t know until the trial is complete. 

“There has been a lot of discussion even at the Congress level about how do you approve these AI tools?” Saghafian said.

“FDA does not have the capacity to test these things very carefully,” he said. “They’ve been asking for money to create the capacity.”

Saghafian and others have proposed that academic labs could play a role in rigorously evaluating AI systems before their public release.

The dynamic nature of AI tools’ constant updates adds a further wrinkle:  If a system changes meaningfully after clearance, when does it require additional review? How should regulators monitor model drift over time?

Not all health-related AI falls within FDA jurisdiction. Chatbots marketed as informational rather than diagnostic typically operate outside the agency’s purview even though these platforms may influence how patients interpret symptoms or whether they seek care — blurring the line between information and medical decision-making.

It is not the first time that new, hastily adopted tech has proven ineffective later. 

In the early 2000s, hundreds of hospitals across the country adopted computer-aided detection, or CAD, systems that helped radiologists interpret mammograms. Hospitals marketed the added computer support to patients, sometimes charging additional fees. Many insurance companies including Medicare covered those costs. But after the technology proliferated, studies found lower accuracy rates among CAD-interpreted mammograms increasing false positives without improving cancer detection rates.

For all the projections that show AI could save hundreds of billions of dollars, those gains depend on an uncertain reality in which the AI tools actually work and that physicians trust and adopt them. Technologies that fail to improve outcomes, reduce administrative burden or meaningfully streamline care cannot deliver efficiency, no matter how sophisticated the marketing. When hospitals or physicians invest hundreds of thousands or millions of dollars in systems that underperform or go unused, those expenses will be folded into billing rates and insurance negotiations. Ultimately, the financial risk of failed innovation could be absorbed by patients and payers, adding to the nation’s already soaring health care bill rather than lowering it.

“I think the problem is that we don’t know how to develop effective AI tools for healthcare,” Saghafian said. 

“And, yes, somebody has to pay for them, right? So, you know, it goes to the healthcare expenditure.”

In theory, AI could revolutionize health care, fill in gaps due to doctor shortages, improve diagnostic accuracy and make care more efficient. But deployed at lightning speed — before rigorous testing and oversight catch up — it risks amplifying the very failures it promises to fix, layering expensive new tools onto a system already struggling with cost, fragmentation and inequity. 

The post Move fast and heal things? AI tests regulation and medicine’s cautious culture appeared first on Straight Arrow News.

Ella Rae Greene, Editor In Chief