Critics question oversight of Utah's AI prescription refill pilot

Researchers and other critics say Utah's pilot letting the startup Doctronic use artificial intelligence to renew routine prescriptions lacks enough independent oversight, according to a report by the Utah Investigative Journalism Project published with KUER. Doctronic was one of the first companies placed in the state Office of Artificial Intelligence Policy's regulatory sandbox, which lets companies test new technology under a controlled arrangement. ^[1]

The numbers come from the company itself. Doctronic's data for January through September shows 2,317 refill requests, though not all were finished by patients. Zach Boyd, head of the state AI policy office, said fewer than 600 were completed by the AI. Physicians asked a colleague to double-check 32 renewals, and in 13 cases flagged for safety concerns, both human doctors disagreed with the AI. The company reported no adverse events. The pilot covers about 190 medications, and Boyd said at this stage one clinician, and usually two, reviews each renewal before a prescription goes out. The report said Utah regulators did not analyze the data. ^[1]

Rahul Gorijavolu and Nikhil Jaiswal, researchers at MIT's Laboratory for Computational Physiology who have raised concerns in journals including JAMA Internal Medicine, said refill requests are a point of contact with the health system. Gorijavolu said the checks and balances of health care are being bypassed, and Jaiswal said the model undermines continuity of care. They also said a company claim that doctors agreed with its AI treatment plans 99% of the time came from research that was not peer-reviewed and involved the company's CEO. Boyd said the state knew it was a non-peer-reviewed preprint with a conflict of interest and that it was not the deciding factor, adding that Doctronic was the first applicant with a credible plan. ^[1]

In March, the security firm Mindgard reported tricking Doctronic's AI into accepting false medical claims and a tripled OxyContin dose. Doctronic has raised $65 million from investors, most of it after the Utah partnership. The company's founders did not answer questions the journalism project sent. ^[1]