Ontario tested 20 government-approved AI doctor scribes. All 20 got something wrong.
Ontario auditor general finds all 20 approved AI medical scribes produce errors or hallucinations
The auditor general of Ontario reviewed 20 AI scribe vendors that the provincial government had approved and pre-qualified for purchase by healthcare providers — tools that automatically summarize patient conversations into structured medical notes. All 20 showed some issue with accuracy or completeness in at least one of two simple test scenarios. Nine hallucinated patient information. Twelve recorded information incorrectly. Seventeen missed key details about mental health issues discussed in the test conversations.
Among the specific mistakes: AI scribes hallucinating nonexistent referrals for blood tests or therapy, incorrectly transcribing the names of prescription medications, and missing "key details" of mental health discussions. The average scribe scored 12 out of 20 on the "accuracy of medical notes generated" metric. That metric accounted for about 4 percent of a vendor's total approval score. A vendor could score zero on accuracy and still meet the minimum threshold for government approval.
One metric worth 30 percent of the overall score: "domestic presence in Ontario." The auditor general's conclusion was that these AI scribes "were not evaluated adequately," which, given the above, lands as one of the more restrained sentences published this year.