Since it was first announced, fans and critics of the FDA’s Pre-Cert program for medical software developers have mused on the benefits and burdens that might come with participation in the experimental program.

What to expect from Pre-Cert

The question has only become more pertinent since the agency released its 1.0 Working Model in January and, just a couple weeks ago, began seeking new De Novo and 510(k) volunteers to give its program a test run.

“Regarding new companies that might want to join, some of the benefits are you’re not going to get a faster, streamlined review at this point, but you will get a teaser for once precertification is opened up more broadly,” Lesley Maloney, head of US regulatory policy and a digital strategy lead at Pre-Cert participant Roche Diagnostics, said during a panel at BIO 2019 in Philadelphia. “What does that mean? [It’s] what types of questions the FDA is going to ask; what are the types of elements, the dashboard I need to create in order to share this data and demonstrate I’ve met the five principles they’re looking for. So pros and cons if you wanted to volunteer, but I understand FDA’s goal to want more data points, especially as others are looking very closely — other regulators globally — at this program to see, at the end of the day, does this work?”

Maloney and Diane Johnson, senior director of strategic regulatory at Johnson & Johnson’s Medical Devices and Diagnostics Businesses, provided attendees with an update on the FDA program’s progress and some firsthand examples of what exactly the agency is searching for when it’s conducting an “excellence appraisal.”

First and foremost, both presenters acknowledged the extra strain these “audits on steroids” might entail, but wanted to dispel the notion that startups or other small developers will have a harder time preparing for and passing an appraisal.

“Our staff at Roche is currently undergoing a week-long discussion about when the FDA comes in — how do we demonstrate this dashboard of all the ways that we can demonstrate that we are a trusted developer, that we know what we’re doing, that we’re able to commit to real-world performance of that software once it’s on the market?” Maloney said. “It is a heavy lift, I will say, but we’ve also heard comments externally [that] precertification is something that’s easier for a large company to do. And we’ve pushed back on that, saying in many ways, to try and disassemble the Roche quality management system could be more difficult than a smaller developer, who gets to build it from the start and can tailor it and look at the precertification working model and say ‘How do I develop software? How does what I do internally in terms of data points meet what the FDA is looking for?’”

So far, many of the qualities and metrics in question have not been what the pharmaceutical giants originally anticipated when they entered the pilot program, Johnson said. Rather than reviewing specific standard operating procedures, the regulator has so far been more interested in examining whether or not a participant has established good manufacturing practices (GMPs) that are flexible and and accountable.

“So for example, trying to put together a design history file on a software, with the whole idea of developing software is you start with this barely sketched out user story and you iterate it all the time — that just does not bode well for a paper design history file on software,” Johnson said. “But the thing is, they’re saying you don’t need a design history file. You need to meet the requirements. So if you can show traceability to your user story at the front end, if it changes 6,000 times along the way but you can still maintain the traceability to what comes out the back end, that’s what they’re looking for, because you would do that anyway. A good software developer would do that anyway.”

Similarly, Johnson said that the program’s emphasis on real-world product use analytics and other post-market data collection will likely require a shift in thinking for longtime medical product manufacturers, but not one that would be out of place for those accustomed to developing and supporting live software.

"We’re used to, in kind of the classic medical device world, you put a product out there and you sit there and wait for the phone to ring to have someone tell you there’s a problem, from ‘I don’t like it,’ to ‘I’m hurt.’ The beauty of software is you can instrument software to tell you that right away,” Johnson said. “That comes down to not only the information around what could be an adverse event — what we’re all pretty classically used to thinking about in the regulated space — but do they like it? That was another bit of a shift in thinking for us, because what does it matter if they like it? It’s not an [adverse event], what’s the difference? … But if you’re an excellent software developer, you should care whether or not they like it. If it crashes every time they go to launch it, that is not good. So, your real-world performance is not just your adverse event complaint stuff; it’s also how you use that information to provide ongoing presentations on how you are a good software developer.”

Intersection of software, drugs, AI is fraught with unknowns

Beyond the question of precertification, yesterday’s panel also dipped into a handful of the other efforts US regulators have made to define medical software, its applications and its review processes.

Speaking on the ambiguity of regulatory guidelines and pathways for digital products associated with drugs, Bradley Merrill Thompson, a partner at Epstein Becker Green who specializes in FDA law, described a recent survey his group conducted among health product manufacturers. When polled on the clarity of drug and device requirements for digital health, 42% of respondents felt that device rules were understandable for devices while 19% felt similarly for drugs.

“What they’re telling us is they go into FDA, and they’ve hired people from a device regulatory expertise in order to put them on the team for a combination product … and they’ve typically followed the [Center for Devices and Radiological Health (CDRH)] guidance on devices, on how to do a clinical trial or whatever the issue might be,” Thompson said later on during the panel. “They go in and they meet with FDA’s [Center for Drug Evaluation and Research (CDER)] responds ‘well, that’s a CDRH guidance. It doesn’t apply to what we’re talking about.’ And when pressed why, CDER would typically respond ‘Well, when you get specific about use of a piece of software with a particular drug, that’s different than the generalized approach that CDRH takes that’s not specific to a drug.’

“And I get that. It makes sense … but the frustration comes with saying ‘Well, what are the rules?’ There simply aren’t written FDA CDER guidance documents on how to deal with the software issues when the software is used in tandem with a drug. And an equal frustration is sometimes we get a sense that the CDER folks are taking the opportunity to take a fresh look at it. They’re not even grounding it in the CDRH position … our perception is that it’s being ignored.”

Dr. Leonard Sacks, acting deputy director of medical policy at CDER, was tight lipped about the disconnect between the agency’s centers on this issue in his response.

"If a device is necessary for the labeling of a drug, obviously we see the need for these things to be approved together,” he said after noting to the audience that he is not personally involved with this area. “But again, I think we’re in our infancy, and we’ll probably have to wait until the rest of the world has a chance to sort it out.”

Similarly, Johnson and Maloney highlighted the agency’s April release of an exploratory whitepaper on artificial intelligence and machine learning algorithms, which they described as “a great start” despite more than 100 pages of comments from industry. But while daunting issues involving ethics, system training, data ownership, global harmonization and more are still need to be addressed, Johnson said that she doesn’t want any of these regulatory uncertainties to hamper developers and manufacturers from embracing what digital products have to offer.

“The difference between now and back in the 90s … [is that] we seem to have hit the tipping point where the regulators have gone from skeptical to, I’d even say, wildly enthusiastic,” she said. “If you’re a person sitting around your desk making the decision ‘Am I a device or not, and if I am what does that mean?’ it’s a really exciting time, have fun with it. But what I’m trying to encourage my folks [to do] is don’t run from it; don’t take that functionality out of the product that makes you a device. Give the patient what they need. Give your surgeons what they need to get the right patient outcomes.”