Pharma companies are increasingly embracing the use of wearables and other connected devices to streamline and improve data collection in clinical trials. But for all the advantages these devices might have, there also challenges — especially when it comes to data standardization.

This week at BIO 2018, Dr. Daniel Karlin, Pfizer’s head of clinical, informatics, regulatory strategy, digital medicine, and the Pfizer Innovation Research Lab, led a panel discussion delving into the many aspects of the thorny problem.

“We’re trying to take the real world and encode it in a way that makes sense. And we’ve hit a point in that effort where there’s a recognition … that without some efforts at standardization, and interoperability, and research that’s reproducible from one site to another and one device to another, it’s going to be really, really difficult to have the observations made by digital devices take on real meaning,” he said.

Step data, the example that panelists focused on for much of the discussion, is much more complicated than it seems on the surface. The accuracy of a device is impacted by the sensor hardware itself as well as the algorithm within the device — which can change without warning when an invisible firmware update is pushed out to a device.

“Arguably the algorithm itself has more impact on the data than the device that does the measuring,” Greg Plante, digital health and technology principal at IQVIA, said. “It varies, but the truth is from one device to the next there are more similarities than there used to be. So you get to the point where, if I don’t know the source and I don’t know the interpretation, I just don’t understand the data. In some of these where they’ve at least done clinical validation of that structure — hardware to algorithm to data generated — you have some assurance, but even there, location of manufacturer, quality measures, the precision of that data can vary significantly.”

Even measures that seem on the surface like they would be consistent across devices often aren’t.

“Ironically a lot of these devices we’re talking about that are built into watches are actually awful at keeping time,” Dr. Stephen Wicks, principal data scientist at Clarivate Analytics, said “In all of the studies that I’ve been involved with, you’ll get 10 to 15 seconds of drift between two devices that are not synchronized to the same source. And this gets back to the algorithms and the firmware, because how it recalibrates to some external clock varies quite a lot.”

The problem is compounded by the fact that the metadata researchers need about how the devices actually work is often a closely-held trade secret.

“Look, I’m a pharma company,” Karlin said. “I’m not trying to sell devices or algorithms, I’m trying to figure out what medicines help with what conditions and sell those. So it’s easy for me to look at a device manufacturer and say ‘You need to classify your algorithm or standardize your algorithm and then we’ll use you.’ And they say ‘Our entire business model is that algorithm. I can’t do that and I won’t do that.’”

For that data to be useful, companies would have to share it in a way that would ultimately become public, meaning even NDAs are not a solution.

“Part of when you submit an application to the FDA, what you’re measuring becomes part of that submission,” Dr. Mahesh Fuldeore, head of patient centered outcomes and digital health at AbbVie, said. “So you have to demonstrate the data, and the data comes from that device company. If you just take that data and do nothing with it, it’s probably feasible. But if you want to use it in that package, it becomes part of the label and becomes public knowledge, based on which the clinicians will make the decision on how to treat the patient. I think that’s where the device companies get concerned in terms of just it’s not between the two companies anymore.”

Panelists didn’t just diagnose the problem, they suggested a number of possible or partial solutions. In the short term, larger sample sizes can mitigate the problem somewhat.

“Having the context is great, [as is] being able to build a large dataset and collect data for a very long time,” Dr. Zachary Beattie, associate professor of neurology at Oregon Health and Science University, said. “We’re developing standards, we have schemas for what device was used, what software it was, what version it was, what firmware version we were using. We get as much data around it [as possible] so that as we do the analysis we have the context. We can look at someone who’s had these sensors for 10 years now, and see if you can come to the same conclusion with sensor A and sensor B and seeing what combinations will work. If you have enough data you might be able to statistically tease it out.”

Researchers can make the data valuable by using it in a comparative way.

“Most of what we’re interested in with most of these sensors is change over time, the longitudinal analysis of the data,” Plante said. “So if we have consistency in collection, that’s ok. But it does require us to look to a different standard because in this case, we’re only looking at change over time that matters.”

But ultimately a systemic solution will involve coming up with a shared, recognized set of standards for data collection, and then getting device companies onboard with them. Several groups are working on this from different angles, but it’s a delicate balancing act.

“We need to remember that we’re only about a decade into deployment of these devices in any stages,” Wicks said. “It’s a young garden, we’ve just planted it, and I hesitate to impose too much structure on the medical device manufacturers to tell them what they need to capture and how they need to capture it. There is still a lot of innovation occurring in this field, there are still new sensors being deployed, there are still ways to examine new [aspects of] the environment. So I think we need to have a lot of caution when we talk about standardization.”