Innovation, especially digital innovation, isn’t what it used to be. A word that once referred primarily to medical research and rare breakthroughs is now a dynamic, continuous process at hospitals that entails constantly interrogating the status quo, and improving it by use of technology.

As innovation becomes a constant process in the enterprise, it’s more important than ever to track and manage, to make sure resources are being used as effectively as they can be.

“What we’ve tried to do is we said ‘it’s great that there are lots of pilots out there, it’s great that our clinicians want to do pilots, but how do we make it so those pilots are successful or can be killed quickly?’” said Josie Elias, program manager for the Digital Health Innovation Guide within the Brigham Digital Innovation Hub. “And how do we make sure we know who they are so we don’t find out a year down the line we’ve got somebody who has a server that’s offsite sharing PHI to a company we’ve never heard of? How do we make sure we’ve created the guard rails to keep projects inline as well as create the superhighway to make sure that it happens, to see we can actually get projects done.”

That’s where the concept of an innovation dashboard can come in handy. MobiHealthNews spoke to a number of innovation experts to ask what they track when running innovation projects or groups and when they track it.
 

Never too early to track If the goal is investing wisely, hospitals should start asking key questions before an innovation process even begins.

“I have a pretty high threshold if you want to build something from scratch, because I say ‘Well that’s awesome, but have you looked at the market? It’s innovating in digital health like crazy,’” said Julia Jackson, Managing Director of the Healthcare Transformation Lab at Massachusetts General Hospital. “And if you can tell me beyond a shadow of a doubt that this is really something the MGH, whose core competency is not computer science, should be building, great. Let’s go for it.”

One important question to ask, with internal or external innovation: Is there an easier way to get or do this?

“I see so many examples of people … that because they happen to know Bill and Bill has a new startup that they end up doing a pilot with Bill and they don’t stop to ask ‘Who else is doing this? Who’s Bill’s competition? Is there a mainstream vendor already offering the same exact thing?’” said Dr. Joseph Kvedar, vice president of connected health at Partners Healthcare. “In my opinion, we don’t have enough discipline around that.”

Another question to ask right off the bat is whether the project is really needed, as well as whether it’s going to generate more work for hospital staff.

“We may get excited about new opportunities, but if it’s not being driven by people on the clinical side that need it, it puts the project at massive disadvantage,” said Boston Children’s Hospital Chief Innovation Officer John Brownstein. “We’re not going to take a process on and push it unless A) it has to be driven by individuals and B) it has to reduce the level of overall workload. We’re not going to throw something in that’s going to be a lot of extra work to improve things, it’s got to be by definition less work.”

Brownstein said Boston Children’s looks at potential projects through the lens of whether new tools are additive as measured by taking more time to use, or ideally less.

Not one dashboard fits all

Once a project gets going, it’s key to have a plan to measure it as you go, rather than just planning to check in again at the end of the pilot.

“I like to focus on how do you engage stakeholders really early on? How do you make sure this works for them? How do you bring in the voice of the end user, the patient or the provider, into the solution design early on? And how do you validate as you go through the construction of a program, rather than developing the whole thing on your own and then validating it?” said Kamal Jethwani, senior director of connected health innovation at Partners HealthCare.

To that end, Jethwani recommended forming advisory panels with clinicians, patients and any other necessary stakeholders very early in a project’s design phases to ensure all the right perspectives are included.

It’s not easy to give a comprehensive list of what things to track, because every hospital, every project, and every pilot is different. But it’s important to have a plan.

“We craft a project timeline around the innovator and the project,” Brownstein explained. “We create very specific deliverables and timelines around product development, execution, and piloting.”

Those timelines also identify funding and create  a roadmap that informs Brownstein and colleagues as to whether the project is on track or not.

That can get tricky because not all metrics apply the same to every project.

“It’s going to vary based on what it is and those are huge variations,” Brownstein said. “We’re going to have startup companies we’re working with and there’s a specific set of things we’re trying to get done, or there’s much earlier stage projects within the hospital that are requiring a lot more intense development. So it’s quite broad.”

What you measure will depend on what your goal is. Broadly speaking, most innovations will be trying to either save money, improve outcomes, enhance patient experience, streamline processes to free up staff time, or some combination of the above.

“For any particular innovation project, there should be a set of goals or metrics to determine if the innovation itself is actually successful,” said Naomi Fried, who previously led innovation at Biogen and Boston Children’s and now heads up consultancy firm Health Innovation Strategies. “Sometimes the metrics are around if the patient will use the solution. Sometimes the metrics are around saving time for providers. Sometimes you are trying to produce a cheaper way to accomplish something. For any innovation it is important to focus on what you are trying to achieve, to have a hypothesis, and to be able to track your success metrics. It is important in the pilot phase to have clearly defined metrics, and to avoid ‘moving goal posts’ — that is, changing how you define success.”

Usability, cost and quality, and scalability

Kvedar said that Partners asks three key questions of an innovation project and generally ends up asking them one at a time.

“The first level of scrutiny is on usability,” he said. “Will people use it? If they will use it, then we get it out to enough of a sample where we can determine if it makes a difference in cost and quality. And then the third is ‘is it of commercial interest to anyone and can we get it adopted internally at scale?’ So that’s how we at Connected Health measure our success.”

Brownstein said Boston Children’s uses the same general pattern.

“In pilot form, we’re going to be looking at of course clinical outcomes, but we’re also going to be looking at usability among the clinical staff, and obviously adoption is a big part of it — is the tool being used? All of that is going to factor into our assessment of how well the project is going and whether it has real legs to get to market,” Brownstein said.

Usability breaks down into both patient usability and staff usability. For patients, having a patient advisory board involved early on is a good first step. One-on-one user interviews and testing sessions are useful early on, and building a survey into an app is a good way to continue to iterate and perfect usability, Brownstein added.

Once a tool is found to be usable, the next question is whether it’s useful. And in the enterprise, that question tends to be very similar to the question of return on investment.

“We push all of our projects to have a financial ROI,” Elias said. “Apple pie in the sky is wonderful, but I can’t sell a project on that and our budgets are so tight I need to show a measurable impact.”

If that ROI can be demonstrated before the project starts, so much the better. But Kvedar said to take that data with a grain of salt.

“We can prove a concept,” Kvedar said of specific . “But if the catcher’s mitt at the system level isn’t there right now today, then there’s no way it’s going to move forward.”

The other I is ROI: Innovation

Hospitals should also look beyond the traditional way of thinking about ROI because many different types of metrics can gauge the success or failure of innovation initiatives.

“Which metrics you use very much depends on what your objective is," Fried said. "If your goal is saving money, then you will track financial ROI. But I like to think of ‘ROI’ more broadly as ‘return on innovation’ not just return on investment.”

The distinction is more than just a nuance. Return on innovation, for instance, can come in many forms.  

“Return on innovation could be financial, but it could also be time savings, quality enhancement, a better patient experience or a better provider experience,” Fried said. “What is really important is to have clarity around your purpose and  to define your objectives and the related metrics of success.”

Innovation checklist and dashboard

So it may be the case that other hospitals can’t build your innovation dashboard for you. But they can point the way to building your own. Here’s a short summary of the advice above.

Before the project (checklist)

• Is this startup or team the best option for this project? What other options are out there?
• Do we have the skills and competency to do this project?
• Does this project have committed clinical and executive champions?
•  Will this project save time for staff, or at least not add additional time?
• What are our goals for this project? (Better clinical outcomes, improved patient experience, cost savings, or time savings)
• How will we monitor those goals?
• What’s the timeline for the project and when will we check in on it?
• What’s the plan for scaling and expanding this project if it works?

Once the project gets rolling (dashboard)

• Monitor provider/administrator user experience
• Monitor provider/administrator time savings
• Monitor patient user experience and engagement
• Monitor clinical outcomes, if applicable
• Monitor financial outcomes, if applicable