A hybrid system combining artificial intelligence with live human specialists was able to more effective diagnose pneumonia from chest X-rays than a previously validated automatic diagnostic algorithm, according to data presented this week at the 2018 SIIM Conference on Machine Intelligence in Medical Imaging.
The study, conducted by researchers from Stanford University School of Medicine and Unanimous AI, investigated the Swarm AI system, a technical emulation of the “swarm intelligence” phenomenon observed in schools of fish or swarms of bees. Much like how these creatures can appear to act with a combined intelligence, the system brought eight radiologists together as a real-time, collaborative network whose collective diagnoses decisions were moderated by an AI.
"Diagnosing pathologies like pneumonia from chest X-rays is extremely difficult, making it an ideal target for AI technologies," Dr. Matthew Lungren, assistant professor of radiology at Stanford University and a collaborator on the study, said in a statement. "The results of this study are very exciting as they point towards a future where doctors and AI algorithms can work together in real-time, rather than human practitioners being replaced by automated algorithms."
To test the Swarm AI system, researchers compared diagnoses reached by the human-AI hybrid to those of CheXNet, an automated, 121-layer convolutional neural network that was previously shown to outperform practicing radiologists. Each system was presented with 50 chest X-rays and tasked with generating the probability that each patient had an active case of pneumonia.
The performance of these two approaches was assessed via three statistical methods. When taking a binary classification approach with a 50 percent probability cutoff, CheXNet and Swarm AI demonstrated 60 percent and 82 percent accuracy, respectively. Mean absolute errors analyses also found the swarm of radiologists to be significantly more accurate than the algorithm (p < .001), while an ROC analysis revealed an 0.906 area under curve for the former, and a 0.708 area under curve for the latter.
While the Swarm AI system generally performed well over the machine learning system in this investigation, the researchers noted that this trend may not persist across all case types. Rather, they wrote, additional research should look to identify the scenarios in which each approach performs best, so that each might someday be applied to where it will perform best.
"We've seen Swarm AI amplify intelligence across many fields, from financial forecasting to business decision-making, but medical applications like the ones we're exploring with Stanford may be the most exciting," Louis Rosenberg, CEO and chief scientist of Unanimous AI, said in a statement. "We fundamentally believe that human wisdom, knowledge, and experience should never be fully replaced from critical decisions. This study helps demonstrate how 'humans-in-the-loop' add real value, even in the world of AI.”
An expert-AI hybrid system like Swarm AI may so far be rare in healthcare, but purely algorithm-powered diagnostic tools are beginning to gain steam. Just a few weeks back, for instance, npj Digital Medicine published pivotal trial data that led to April’s de novo clearance of the first AI-based diagnostic system to not require clinician interpretation.
