Pivotal trial data that led to April’s de novo clearance of the first AI-based diagnostic system to not require clinician interpretation is now available to the public. Published today in npj Digital Medicine, this 900-patient investigation showed that the IDx-Dr system for autonomous detection of diabetic retinopathy “robustly exceeded the pre-specified primary endpoint goals” and is able to realistically be deployed and used by non-specialist providers.
“The AI system's primary role is to identify those people with diabetes who are likely to have diabetic retinopathy that requires further evaluation by an eye-care provider,” Dr. Michael Abràmoff, founder and president of IDx and the study’s principal investigator, said in a statement. “The study results demonstrate the safety of autonomous AI systems to bring specialty-level diagnostics to a primary care setting, with the potential to increase access and lower cost.”
While these results were enough for the FDA to grant its approval, an accompanying editorial raised a few warning flags over the researcher's robustness claims. Penned by Dr. Pearse A. Keane, a clinician scientist at the National Institute for Health Research (NIHR), and Dr. Eric J. Topol, director of the Scripps Translational Science Institute and EVP at Scripps Research Institute, it highlighted sampling limitations, protocol decisions, and specific clinical limitations that will need to be addressed in future validation studies. However, they still wrote that the study is “undoubtedly a milestone, and an important benchmark for future research.”
IDx-DR’s algorithm analyzes images taken with the Topcon NW400 retinal camera and uploaded to a cloud server. Within minutes, the software provides doctors with a binary result, either indicating that more than mild diabetic retinopathy is present and that the patient should be referred to an eye care professional, or that the screen is negative and should be repeated in 12 months. Notably, since the system’s results do not require clinician interpretation, it can be used by providers who aren’t normally involved with patients’ eye care, according to the agency.
To test the system, Abràmoff and colleagues collected the retinal images of 900 diabetes patients from 10 US sites. These images, which were taken by AI-assisted primary care staff, were fed into IDx-DR’s algorithm for diagnosis. The researchers compared the algorithm’s conclusions to those reached by experienced, Wisconsin Fundus Photograph Reading Center (FPRC)-certified retinal photographers who took their own images and evaluated each case using the FPRC’s gold standard grading system.
Among the 900 participants, 892 completed all of the study procedures, 852 received an FPRC diagnosis, and 819 received both an FPRC and IDx-DR diagnosis. Among the lattermost group, 198 patients received a mild diabetic retinopathy diagnosis using the gold standard, while IDx-DR’s algorithm diagnosed 173. Of the 621 determined by FPRC imaging not to have the disease, the AI corroborated this conclusion on 556 patients.
These two rates translate to 87.2 percent sensitivity, and 90.7 percent specificity. The system also demonstrated an imageability rate of 96.1 percent, and according to the researchers was well integrated into the primary care setting.
“Early detection of retinopathy is an important part of managing care for the millions of people with diabetes, yet many patients with diabetes are not adequately screened for diabetic retinopathy since about 50 percent of them do not see their eye doctor on a yearly basis,” Dr. Malvina Eydelman, director of the Division of Ophthalmic and Ear, Nose, and Throat Devices at the FDA's Center for Devices and Radiological Health, said in an April statement announcing the agency's de novo approval. “Today’s decision permits the marketing of a novel artificial intelligence technology that can be used in a primary care doctor’s office. The FDA will continue to facilitate the availability of safe and effective digital health devices that may improve patient access to needed health care.”
Peers express skepticism in accompanying editorial
However groundbreaking the technology at hand may appear to be, the investigation of its validity wasn’t above critique.
In their accompanying editorial, Keane and Topol stressed that Abràmoff and colleagues’ patient sample was relatively small for a diagnostic accuracy study, and instituted preferential recruitment to bolster its numbers. Along with concerns that 40 participants excluded from the analysis could have a significant impact on the study’s final findings, they wondered how the tool’s sole capacity for diabetic neuropathy diagnosis would realistically fit scenarios where patients could present with a number of other, more severe retinal conditions.
“There is also the question as to whether the now-approved device will have significant uptake in the clinic. Besides the expense, it remains to be determined how and where it would be implemented,” they wrote. “Will primary care clinics incorporate retinal screening into their practice? This is not really an ‘autonomous system’ since someone needs to acquire the image — who will perform that?”
Keane and Topol did note that the FDA’s De Novo premarket review pathway, by necessity, allows a bit more leeway than the standard 510(k) clearance. Keeping in mind that future investigations are likely to face more stringent inquiry prior to regulatory approval, they applauded the study as a much-need trailblazer for what’s to come in AI diagnosis technology.
“While it is always easy to be critical of studies that forge new ground, it is important to applaud the authors for this pivotal work. Although deep learning will not be a panacea, it has huge potential in many clinical areas where high dimensional data is mapped to a simple classification and for which datasets are potentially stable over extended periods. As such, it will be incumbent on healthcare professionals to become more familiar with this and other AI technologies in the coming years to ensure that they are used appropriately. This study represents an important first step in that direction,” they concluded.
Of note, Keane reports a consulting relationship with DeepMind, which earlier this month released its own eye disease diagnosis study. Topol holds an advisory position at Verily Life Science.
