While health and medical apps may be helpful in making diagnoses, it seems they still haven’t caught up to doctors. In a new study published in JAMA Internal Medicine, a head-to-head competition between doctors and algorithmic symptom-checking apps, the real human doctors came out on top by a margin of more than two to one.

Researchers from Harvard, Brigham & Women’s Hospital and the Human Diagnosis Project in Washington, D.C. put 234 physicians to the test against 23 symptom-checking apps, asking them to look at a selection of 45 clinical vignettes featuring hypothetical patients. The doctors were mainly trained in internal medicine, but there were also family practitioners and pediatricians, and about half of them were in residency or fellowship.

The idea was to see if digital health tools could help reduce physician diagnostic error, which the JAMA study mentioned was a common problem.

“Given advancements in computer science, computers may be able to independently make accurate clinical diagnoses,” the authors wrote, adding that while studies have looked at computers versus doctors for reading EKGs, diagnostic accuracy was unknown.
The vignettes were generated from Human Dx, the Human Diagnosis Project’s web and app-based platform, and covered conditions ranging from high, medium and low acuity (15 each) and 19 uncommon condition vignettes. The vignettes included the patient’s medical history, but no physical exam or test results.

The real doctors outperformed computer algorithms in diagnostic accuracy, with 84 percent of doctors making the correct diagnosis on the first try versus 51 percent of symptom-checking apps.

Flesh and blood doctors look especially more reliable than the apps when it comes more severe illnesses. In cases where acuity was low, doctors outperformed algorithms by 65 percent to 41 percent. When conditions were categorized as high acuity, doctors were right 79 percent of the time versus the app’s 24 percent accuracy.

That’s not to say diagnostic error is simply a computer glitch – physicians make mistakes quite often in the real world, and they made incorrect diagnoses in 15 percent of the cases for this study. Another point to note is the symptom checkers are not the only digital diagnostic tool out there, and the researchers conceded “other tools may have superior performance.”

So, rather than simply having man and machine face off, the study indicated that doctors and computers could work together to improve diagnoses.

“While this project compared diagnostic performance, future work should test whether computer algorithms can augment physician diagnostic accuracy,” the researchers conclude. 

Although it may not have rivaled the human doctors, the 51 percent first try accuracy of the symptom checker was a big improvement over the 34 percent boasted by a similar study in the British Medical Journal in July 2015.