In May, a pair of studies were published in the journals Autism Research and the Journal of the American Medical Informatics Association (JAMIA) that evaluated Cognoa’s artificial intelligence-based tool for early identification of autism in children. While both investigations suggest that the tool’s performance significantly exceeded those of other standardized screeners, comparing the two studies shows a sizable difference in the reported accuracy of the company’s offering.
However, this disparity in performance doesn’t necessarily describe an inconsistency, heads from the AI company told MobiHealthNews. While the two Cognoa-funded studies were unveiled during the same month, they feature different iterations of the software due to the varying amount of time it takes to publish a work in peer-reviewed journals. For the company, these data are emblematic of the advancements that come with algorithm iteration, and how the “learning” part of machine learning can increasingly improve care.
“When you do [machine learning] and you do it right, this is what it looks like: the ability to expand datasets; to have a very rapid improvement from version ’n’ to version ’n-plus-one;’ and to show that as your AI and machine learning gets better and continually improves, it can ultimately lift and change the standard of care,” Brent Vaughan, CEO of Cognoa, told MobiHealthNews. “I think that’s the exciting feature — machine learning changes, and there can be a very predictable path toward showing improvement.”
By analyzing parent-provided information and videos of a child’s natural behavior, Cognoa’s app uses machine learning to provide an assessment of whether that child is developing at the right pace, as well as to evaluate their behavioral health. The platform received categorization from the FDA as a diagnostic medical device for autism in February, and according to Cognoa can help clinicians identify autism among children as young as 18 months.
“The whole goal of this is really to empower pediatricians and patients to get earlier diagnosis so they can get earlier intervention,” Dr. Sharief Taraman, chief medical officer at Cognoa and practicing pediatric neurologist, told MobiHealthNews. “In my current clinical practice … getting kids evaluated for autism by a specialist is hard because there is a clinician shortage. As clinicians, we’re doing a disservice to these children to make them wait so long and not get services because they don’t have a diagnosis. And if you don’t have a diagnosis, you can’t get the right treatment. We’re trying to help fix that problem.”
In the first study, published in Autism Research, researchers evaluated the performance of Cognoa’s platform against four standardized autism screening tools — the Modified Checklist for Autism in Toddlers Revised with Follow-up, the Social Responsiveness Scale (second edition), the Social Communication Questionnaire, and the Child Behavior Checklist. Among a sample of 230 children aged 18 to 72 months who were referred to one of three autism spectrum disorder-specializing tertiary care centers, researchers found that Cognoa’s tool performed comparatively or significantly better than the standardized screeners, and unlike the standard screeners was applicable across all included age groups. Overall, the tool achieved a sensitivity and specificity of 75 percent and 62 percent, respectively.
“But that version was basically the device from 2015, 2016,” Taraman said. “What we’ve been able to do with every new set of learning and new set of information that comes in is really fine-tune the algorithm. … We’re at the point where we have large amounts of data that allows us to extract accurately predictive features from these patient samples, and the ability of the algorithm to identify children with and without autism more accurately, and more efficiently, happens with every [annual] reiteration of the algorithm.”
To support this claim, Taraman and Vaughan pointed to the JAMIA study, which they said features the second iteration of the company’s algorithm. Here, the revamped tool was evaluated in 162 additional children recruited from the same centers and, again, demonstrated a significant accuracy improvement following comparison. While the researchers did note some data sparsity and “several important confounding factors” resulting from how the algorithm was trained versus how it was deployed, they did note the continued improvement in its performance (sensitivity, 90 percent; specificity, 60 percent) and the potential for additional “significant” improvements with additional data and refinement.
“With each version of the software, after we’ve completed a clinical study, we have the ability to take the data from that study and our learnings, fold it into our training set, and refine our algorithms and come up with a next version,” Vaughan said. “And what we’ve shown now through multiple iterations since we started is the ability is to consistently improve the accuracy in terms of both sensitivity and specificity of the algorithm and the diagnostic prediction.
Vaughan and Taraman noted that publication of another study incorporating a third version of the algorithm is underway, and that plans for yet another iteration including more inputs are also in the works. Further, Vaughan said that discussions between Cognoa and the FDA have been open for over a year, and that the company is hoping for clearance as a Class 2 device sometime in 2019.
