Paging Dr. Algorithm: AI steps into medicine, but not without concerns
Utah has become the first state to let an artificial intelligence system renew certain prescriptions without human oversight, a move officials say could relieve pressure on strained clinicians, but medical groups warn may erode critical safeguards in patient care.
In January, Utah announced a first‑of‑its‑kind partnership with Doctronic, a New York–based startup whose autonomous AI platform can assess patients and prescribe routine medication refills. State leaders say the move is intended to ease pressure on clinicians as health care costs rise and doctor shortages persist, particularly in rural areas.
Health care expenses keep climbing, and clinicians — especially in rural areas — are stretched thin, Margaret Busse, executive director of the Utah Department of Commerce, which oversees the program, told Politico. She said automating routine prescription renewals could help lower costs while giving providers more time to focus on complex cases.
Busse said the state also views the project as a way to encourage innovation among companies navigating emerging technologies.
“It’s also a way to provide a pathway to innovation for entrepreneurs who are using AI in creative ways that may be bumping up against regulation,” she said.
Doctors’ groups urge caution
Physicians’ organizations, however, have raised concerns that removing doctors from prescribing decisions could introduce new safety risks.
In a statement to Politico, Dr. John Whyte, CEO and executive vice president of the American Medical Association, warned that autonomous AI tools could miss important clinical details.
“While AI has limitless opportunity to transform medicine for the better, without physician input, it also poses serious risks to patients and physicians alike,” Whyte said.
Physician organizations say prescription decisions involve more than confirming an existing diagnosis, arguing that even routine renewals can reveal warning signs that require clinical judgment. Without that human review, they say, subtle but important details could be missed.
“There are drugs that carry serious risks,” Dr. Daniel G. Aaron, a University of Utah physician and legal scholar told Straight Arrow News. “When patients come in for a renewal and mention a headache or a rash, a clinician evaluates whether that’s related to the medication or something else entirely. Asking AI to make that judgment raises serious concerns.”
A broader AI moment in medicine
The debate over AI prescribing is unfolding amid a rapid expansion of artificial intelligence tools across health care, a trend documented by SAN health reporter Jess Craig in a recent examination of medicine’s increasingly uneasy relationship with Silicon Valley–style innovation.
AI systems are already widely used to transcribe doctor‑patient conversations, flag abnormalities in medical imaging and streamline hospital scheduling. One of the fastest‑spreading tools is the “ambient scribe,” software that listens during clinical visits and automatically produces medical notes. Many physicians say the technology reduces burnout and paperwork, but insurers and regulators worry it may also inflate billing by documenting more diagnoses and higher‑complexity care.
AI tools are being adopted so quickly that regulation hasn’t been able to keep up. As a result, both regulators and insurers are now scrambling to figure out where to draw the line between tools that support clinical decisions and those that take on actual clinical authority. While such tools are meant to assist clinicians, experts caution that fully autonomous systems are something entirely different, particularly when they move beyond decision support and replace physicians altogether.
Regulatory questions
Utah allowed the Doctronic program through its regulatory sandbox, which temporarily waives certain state requirements for technologies deemed potentially beneficial to consumers. Federal law, however, generally requires prescription drugs to be dispensed by licensed practitioners, and medical devices are typically subject to U.S. Food and Drug Administration review.
Because the system has not been reviewed by the Food and Drug Administration, experts say it occupies uncertain territory between consumer software and regulated medical technology.
State officials say patient safety remains a priority. Busse acknowledged the initiative carries risk but said Utah is closely monitoring its rollout.
“We want it to be done in such a way that people will trust that Utah is looking at this carefully and is not being cavalier about how we granted this regulatory mitigation,” she said to Politico. “In a way it’s a risk for us as we do this.”
Rapid growth, unanswered questions
Founded in 2023, Doctronic promotes its platform as an around‑the‑clock digital clinician and offers optional physician consultations at an added cost, positioning the service as both automation and access expansion.
The company says its system could improve access to care and reduce medication nonadherence, but independent experts note that evidence supporting autonomous prescribing remains limited and largely unpublished.
For critics, Utah’s experiment highlights a central challenge in governing emerging technologies: how to balance innovation against safety before systems become widespread and difficult to regulate.
“This is hardly going to be the last example of a physician’s work being outsourced to AI,” Aaron said. “The real question is whether we are moving fast enough to innovate — or too fast to protect patients.”
Ella Rae Greene, Editor In Chief
