One large healthcare system in the Pacific Northwest is moving machine learning technology from theory to practice. MultiCare Health System is using machine learning to develop a predictive model for reducing heart failure readmissions. Starting with 88 predictive variables applied to data from 69,000 heart failure patient encounters, the machine learning team has been able to quickly develop and refine a predictive model.
The output from the model has guided resource allocation efforts and pre-discharge decision making to significantly improve patient care management activities. And the data has engendered trust among clinicians who rely on it the most for clinical decision making.
This inside look at the application of advanced technology offers lessons for any healthcare system planning to ramp up its machine learning and predictive analytics efforts.
Download
What if a technology could accurately predict the likelihood of heart failure readmissions? Or the likelihood that a patient with heart failure would not take his medications or would miss his appointments?
Heart failure readmissions are one of healthcare’s biggest blocks to providing value-based care. Heart failure consistently ranks as one of the top five principle diagnoses causing readmissions within 30 days. And it’s expensive for hospitals, which pick up almost 70 percent of the $110,000 incurred by each patient with heart failure over a lifetime.
Although healthcare systems across the United States already use readmission risk assessment tools, these tools can be unreliable. Some tools, such as the LACE index, require slow, manual processes that can produce inaccurate results. Critics have questioned the validity of the LACE index in its applicability to broad patient populations. These tools are also only available at a particular point in the patient journey. For example, LACE uses data that is only available at the time of discharge. But if predictions are going to guide decisions throughout the continuum of care, they need to be readily available well before discharge. With stricter initiatives calling for reduced readmissions, many systems are pursuing more accurate prediction tools.
MultiCare Health System, working with Health Catalyst, learned about the potential for machine learning in healthcare to more accurately make predictions. As a bonus, MultiCare could use machine learning to automate the prediction process and reduce the documentation burden on clinical staff. Through the approach outlined below, the health system began exploring machine learning’s ability to predict, and ultimately lower, heart failure readmissions.
Any improvement initiative should begin with buy-in from stakeholders across the system. It’s no different when that initiative includes machine learning tools. The collaboration should include frontline clinicians, data scientists, quality directors, and program managers. Developing a machine learning program—in MultiCare’s case, a predictive model to reduce readmissions across the entire organization—requires knowledge and expertise from multiple disciplines.
When developing a predictive model, the team must gather the relevant historical data. The team also needs to identify input variables (or features) that could influence or predict the target outcome (i.e., likelihood of readmission). Broadly, the role of machine learning here is to learn the relationships between patient attributes and subsequent outcomes. While many types of events could be predicted, the aforementioned business priorities focused this project on readmission risk. Initially, the dataset will include a large number of input variables that the machine learning algorithm will analyze and pare to a smaller set of the most important outcome drivers.
For MultiCare’s predictive model, data scientists wanted to be able to predict 30-day heart failure readmissions in particular and worked with clinicians to identify 88 input variables thought to be drivers of readmissions. This data was gathered from 69,000 heart failure-related encounters over a six-year period.
Data scientists then ran the data through a variety of machine learning algorithms to evaluate the 88 input variables against the 30-day readmission outcome. Each experiment generated a predictive model and measured the accuracy against a special subset of the 69,000 input records that was not included in the experiment. Of the original 88 input variables explored, the final model used 24. The data scientists then selected the most accurate model for guiding readmission interventions. The final model was shown to accurately predict readmissions with a c-statistic of 0.84 (AUROC = 0.84). This is an improvement over the best models in the literature that show an accuracy of 0.78. Note: using this metric, a perfect model is 1.00; random predictions would yield a score of 0.50.
MultiCare now makes around 150 predictions every day on currently admitted patients. The system further improves usability of the model by categorizing individual patients into five different risk levels. Patients are classified by their individual readmission score (predicted probability). Each risk group’s overall readmission rate is also reported in parenthesis:
Machine learning automation leads to more efficient resource allocation and more appropriate interventions. Automation means the machine learning tool should run and update itself every day – or even more often. For example, when a patient admits, within 24 hours the model should show the percentage chance of readmitting and the top three drivers indicating why. Then the team can allocate resources appropriately, ensuring that the patients receive interventions consistent with their risk level. The algorithm runs again the next day and produces a new score, either higher or lower, which tells the team if the care the patient received the previous day decreased or increased the chance of readmission. The team can use a chart that shows the trend and drivers on any given day.
This type of machine learning-based decision support can go beyond inpatient care to also inform post-discharge interventions—especially when the team is trying to reduce readmissions. If, for example, one of the risk drivers is a socioeconomic issue, such as transportation to an appointment or help paying for medication, then the tool will suggest social worker involvement. Depending on the issue, the care management team could include a physician, nurse practitioner, social worker, care manager, pharmacist, cardiologist, or other clinical specialist. The entire system should be simple, automated, near-real time, and updated daily—qualities that are vital to care team adoption.
MultiCare learned a few valuable lessons while developing its machine learning program:
Trust in the data being used to develop the predictive model is critical to machine learning’s successful rollout. Developing trust among clinicians will generate strong buy-in and adoption of the suggested interventions. But creating data transparency can be challenging, especially if the data source is unknown. For example, when analyzing readmissions, is the data just for heart failure readmissions or does it include all-cause readmissions? Always ask questions about data—where does it come from; has it been filtered in any way—to build trust.
There’s also safety in numbers, meaning bigger datasets produce more reliable results. If the model is missing pieces of data, it erodes trust in the predictive model. However, an appropriately large dataset (e.g., the special subset of data from the 69,000 input records described earlier) is enough to reliably train the model and provides a much tighter probability that the model will be accurate.
With the right stakeholders on board (e.g., clinicians, administrators, IT, domain managers from across the organization), the lifecycle for implementing machine learning can be relatively rapid. The right team is needed to guide the model development by suggesting input features as well as validate the results. Having the right stakeholders from the beginning will also ensure that the model is adopted. If clinical leadership is involved from the beginning, bringing the model into the clinical workflows can be planned and implemented much earlier.
Implementing a machine learning model to influence decisions requires a thoughtful user experience. The data that is presented to clinicians must be concise, but still convey enough context for the clinician to know how to use the data to make meaningful decisions. Simply putting a risk score in front of a clinician may result in a clinician mentally asking, ‘Now what?’. However, presenting that same risk score with additional context like the specific factors that are driving that patient’s score can significantly enhance the clinician’s ability to translate the data into meaningful action. Providing the right context is a balance: too much information can quickly overwhelm busy front-line staff. This careful balance is best navigated by data visualization specialists who understand clinical workflow and visualization techniques.
Machine learning can help remedy the problematic, time-consuming, and inaccurate predictive risk models most healthcare organizations currently use. Organizations should take input from stakeholders across the entire organization, including clinicians and care managers, when creating and refining a machine learning tool. This will increase adoption and the chances that interventions suggested by the tool are carefully considered. Using historical data to produce accurate models is an iterative process of refinement. It’s important to use trusted data that, when coupled with buy-in from the right stakeholders, can help organizations see results from machine learning tools very quickly.
Daily machine learning predictions are now directly fed into MultiCare’s EMR, which helps make it an integral part of the clinician workflow. Considering the accuracy and workflow integration, this new decision support tool shows great promise toward achieving the goal of reduced heart failure readmissions. The model’s ability to accurately predict these readmissions (at 0.84 AUC) is unprecedented in the literature, and will go a long way to optimizing care for many patients.
