The accuracy of digital symptom checkers for both triage and diagnosis is generally low, which could pose a risk to patient safety, according to a review of studies published in npj Digital Medicine. 

Researchers compiled 10 studies that evaluated a total of 48 online symptom checkers. Half of the studies used real patients, while the other half studied simulated cases. The review found primacy diagnostic accuracy, or listing the correct diagnosis first, ranged from 19% to 38%. Top three diagnostic accuracy, which was evaluated in seven studies, ranged from 33% to 58%.

Triage accuracy, or sending users to the right level of care, was higher, ranging from 49% to 90%. Three studies examined cases by emergency status as well and two found symptom checkers were more accurate at triaging emergent cases. However, one study found triage accuracy for ophthalmic emergencies was lower than non-urgent cases, at 39% compared with 88%. 

"Our review highlighted that both diagnostic and triage accuracies were generally low. Moreover, there is considerable variation across symptom checkers despite being presented with uniform symptom parameters. We also note that the diagnostic and triage accuracies of symptom checkers, as well as the variation in performance, were greatly dependent on the acuity of the condition assessed," the review's authors wrote. 

"As a whole, these issues raise multiple concerns about the use of symptom checkers as patient-facing tools, especially given their increasingly endorsed role within health systems as triage services that direct patients towards appropriate treatment pathways."

WHY IT MATTERS

Researchers noted some limitations to the review, including the use of simulated patient cases and the studies' focus on high-income countries like the U.S. and the UK. They also said bias could be introduced in retrospective studies where information is collected after the event, variation in performance could be attributed to the difference between studies, and AI-enabled symptom checkers may improve over time. 

However, the results raise concerns about using symptom checkers, particularly when they're promoted by health systems to send patients to the correct care site. The study's authors said there are also issues with transparency surrounding how symptom checkers are built and what context, patient demographics and clinical information are used. 

"Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards," they wrote. "Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required."

THE LARGER TREND

There are a number of companies that offer digital symptom checkers. Digital health firm Babylon, which until recently was embedded with multiple NHS Trusts, offers an AI-backed triage chatbot. The company has been dialing back those NHS contracts as it grows its value-based care segment in the U.S. 

Japanese startup Ubie launched its AI-backed symptom checker in the U.S. earlier this year. It recently raised $26.2 million in Series C funding. Other companies in the space include Ada Health, K Health and Infermedica.