Sharecare's AskMD.

Online and mobile symptom checkers like WebMD, AskMD, iTriage, and even offerings from the Mayo Clinic and the UK's NHS, come up with an accurate diagnosis only about a third of the time, according to a study by Harvard researchers recently published in the British Medical Journal.

Researchers looked at 23 free English-language symptom checkers that either offered patients a diagnosis; triaged a condition into emergency care, non-emergency care, or home-care; or did both. Researchers found that 11 symptom checkers offered both diagnoses and triage advice, eight were diagnoses only, and four provided triage advice only. They ran 45 standardized patient vignettes through each symptom checker and evaluated whether the correct diagnosis was presented as the top choice, in the top three, or in the top 20. They also evaluated whether they triaged the conditions correctly.

On average, symptom checkers listed the correct diagnosis first just 34 percent of the time. They had it listed in the top three 51 percent of the time and listed it in the top 20 just 58 percent of the time. Conditions were triaged correctly 57 percent of the time on average, but the symptom checkers erred on the side of sending patients to the emergency room: 80 percent of vignettes requiring emergency visits were triaged correctly, compared to 55 percent requiring non-emergency care and 33 percent requiring self-care.

"Whether this level of performance for diagnosis and triage we observed is acceptable depends on the standard for comparison," researchers wrote. "If symptom checkers are seen as a replacement for seeing a physician, they are likely an inferior alternative. It is believed that physicians have a diagnostic accuracy rate of 85 to 90 percent, though in some studies using clinical vignettes, performance was lower. However, in-person physician visits might be the wrong comparison because patients are likely not using symptom checkers to obtain a definitive diagnosis but for quick and accessible guidance."

However, researchers noted, symptom checkers still outperform a simple Google search and offer a diagnostic accuracy about on par with telephone nurse lines, according to some recent studies. So they may have some worth even with a surprisingly low diagnostic accuracy.

Sites owned by government groups, payers, providers, and private companies all performed similarly on the diagnostic side, but provider tools were a little more accurate at triage (68 percent accuracy on average, compared to 59 percent from private companies and 43 percent from payer/government groups).

As for the individual tools, some of the more well-known symptom checkers were top performers. WebMD was just a little above average, listing the correct diagnosis in the top 20 options 62 percent of the time. AskMD managed 75, Mayo Clinic posted 76, and iTriage got 77 percent in the top 20. The best performing symptom checker was a tool called Isabel, that managed 84 percent. DocResponse also did well, getting the correct diagnosis on the first try 50 percent of the time and in the top 20 choices 72 percent of the time.

"Physicians should be aware that an increasing number of their patients are using new internet based tools such as symptom checkers and that the diagnosis and triage advice patients receive may often be inaccurate," researchers wrote. "For patients, our results imply that in many cases symptom checkers can give the user a sense of possible diagnoses but also provide a note of caution, as the tools are frequently wrong and the triage advice overly cautious. Symptom checkers may, however, be of value if the alternative is not seeking any advice or simply using an internet search engine. Further evaluations and monitoring of symptom checkers will be important to assess whether they help people learn more and make better decisions about their health."