It’s been six years since IBM first alluded to plans to build artificially intelligent physician’s assistant. Even as that mission became more concrete with a formal launch of the health-focused Watson and years of near-constant announcements and partnerships, it still hasn’t been clear just exactly what impact the cognitive computing program will have on healthcare. But newly shared data could offer some insight on where IBM Watson can immediately start to make its mark in care.
The data, which was publicly shared during the American Society of Clinical Oncology meeting this week, gives a progress report on how Watson is performing in treatment decision-making and clinical trial matching. Watson for Oncology, which was trained by Memorial Sloan Kettering beginning in 2013, was involved in pilots at hospitals in India, Thailand, South Korea and Mexico, and showed high concordance rates in multiple types of cancer. In Manipal Comprehensive Cancer Center in Bangalore, Watson had concordance rates of 96 and 93 percent for lung and rectal cancer cases, respectively, and 81 percent for lung cancer. In Thailand, Watson hit 83 percent concordance for multiple cancer types, and was at 73 percent for colon cancer in South Korea.
That’s in-line with what IBM Watson was hoping for, said the company’s deputy chief health officer Dr. Andrew Norden, and is representative of what happens with human doctors, wherein a group of physicians will have differing opinions on a treatment plan.
“This is really exciting for us, of course, because what it means is this is really the beginning of showing really robust evidence of our solutions are working in practice,” Norden told MobiHealthNews. “Those numbers are where we want them. If concordance rate was 100 percent, that means every single time a physician makes a treatment, Watson has the same advice, so it would be easy to argue right there that it wouldn’t be offering any value at all. If it were way lower or even zero percent, that would point to a big problem too.”
What Watson’s performance shows, Norden said, is that is can be a useful and efficient way of ensuring that the most robust and relevant knowledge is coming to the table, but it stops short of claiming to be the best answer.
To test how Watson performed in clinical trial matching, Novartis sponsored a 16-week pilot at Highlands Oncology Group (a community cancer clinic in rural Arkansas). Watson processed data from 2,620 lung and breast cancer patients and was able to cut the time needed to screen for clinical trials by nearly 80 percent.
The company also announced several new collaborators, including a pilot with Hackensack Meridian Health oncologists and a collaboration with health tech company Cota. Altogether, the company counts partnerships with 55 hospitals and health organizations around the world.
However, that doesn’t necessarily mean all partners are actually using the system: in March, Jupiter Medical Center in South Florida became the first US regional medical center to adopt Watson, but in some cases, the partnerships could mean pilots in various stages of completion, training collaborations, or actual use. The financial agreements vary in each case as well, and its a different economic undertaking depending on who the client is. For instance, before the Novartis-sponsored pilot could begin at Highlands, the cancer clinic had to pay for some of their own technological upgrades in order to get the necessary data. A spokesperson said that, most of the time, Watson collaborations begin with a paid Proof of Concept by the client, followed by a training period. Then, it moves to the IBM cloud with a SaaS model on one of four models: pay-per-insight, subscription, shared value, or licensing.
In fact, the sheer number and diversity of partnerships and integrations Watson has announced have far outpaced the real-world examples of how the AI system is delivering on its ambitious promise. The feasibility of Watson has been debated since IBM introduced the health-focused venture at HIMSS 2015, and didn’t stop when the company opened their Cambridge, Massachusetts headquarters a few months later amid a flurry of announcements and partnerships. From joining forces with major health systems and research centers to integrations with genetic sequencing technology and population health management companies, Watson has almost aggressively inserted itself into every corner. Along with that has been skepticism and criticism. In February, MD Anderson broke off its engagement with Watson citing high costs and low output.
But Watson is still a work in progress, as the company’s vice president and chief strategy officer Shiva Kumar reminded the audience at the Light Forum event last month.
“We tend to use word cognitive computing, because it goes beyond machine learning and deep learning. Being able to derive insights, being able to integrate, and learn,” Kumar said at the time. “Healthcare is unique; it’s highly regulated, and has a ton of data it can’t use. And there are many silos,” he said. “So it’s a place where a lot of technology can improve it. But at the end of day, success is determined by practitioners.”
In the meantime, Norden said studying Watson in use around the globe will improve the supercomputer’s performance and eventually lead to more and more sophisticated applications, such figuring out how to actually prolong life and improve outcomes. As of the end of this year, Watson for Oncology should be able to support at least 12 cancer types, which makes up 80 percent of the global incidence of cancer.
“These concordance and efficiency studies we are showing at ASCO this year are the beginning of a much deeper look at the way Watson can impact cancer care,” Norden said. “We’ll start to see the potential impact it can have on a given geography – there are important differences moving from one place to another, given the local guidelines and regulations. Then, we can start to study the impact on patient outcomes. Does it increase compliance? Does it make tumors shrink? We’re just in the planning stage now.”
