Today’s healthcare leaders want their organizations to be data driven. However, many leaders incorrectly interpret data and analytic visualizations. Without accurate interpretation of analytics, healthcare leaders don’t have real data-driven insight to know how to strategically improve and measure wins and losses.
Concerningly, only 4% of analysts, department leaders, executives, and board members can correctly identify the following four pivotal pieces of information in a line chart:
1. Baseline performance.
2. Extreme historical points.
3. Persistent shifts in performance.
4. Forecasted performance in six months.
Is your organization’s leadership part of the 4% that can correctly identify these four points in data presented over time? If not, your data and analytics may be doing more harm than good.
Healthcare has focused its development of artificial intelligence or augmented intelligence (AI) on narrow, transactional use cases, such as readmission risk, deterioration, sepsis detection, and smoking cessation likelihood. These important, transactional use cases happen one patient and provider at a time.
Just as importantly, health systems must focus on broader use cases like management and strategic analyses. An error in interpreting performance at the management or strategic level impacts many patients and providers, not just one patient and their care team as in the case of narrow, transactional use cases.
Surprisingly, only 4% of analysts, department leaders, and even board members can correctly identify all four of the following points in a line chart: baseline, extreme points, persistent shift, and forecast. Regrettably, these four points remain a secret to most analysts and leaders, even though these points unlock the meaning of the data. In response to these findings, Health Catalyst urges health systems to focus significantly more development of AI in healthcare on empowering broader, strategic use cases for leaders to reduce the risk of incorrect analytic interpretation.
Is your organization’s leadership part of the 4% that can correctly identify these four points in data presented over time? If not, your data and analytics may be doing more harm than good. Take this interactive assessment now to find out where you stand.
Today’s health system leaders strive to be data driven. They understand that across industries, organizations that use data to make decisions are two times more likely to hit their business goals and grow at an average annual rate of 30%.
To become data-driven, healthcare organizations must uncover meaningful insights from their data. Correct analysis and interpretation will reveal valuable insights or secrets hidden in the data health systems already capture. For example, leaders should analyze data collected over time to determine whether performance is increasing, decreasing, or remaining steady. Specifically, leaders should attempt to uncover four valuable secrets in that data—a baseline, extreme points, and persistent shifts in performance, and use that information to forecast future performance.
Leaders can use the four points shown in Figure 1 to make data-driven decisions:
Data is used to improve performance throughout healthcare organizations. Chief quality officers (CQOs) and quality directors want to use data to make decisions about care delivery for septic patients. They could use the four data points in a myriad of ways:
Health systems invest millions of dollars into data and analytics—this includes investments in people, processes, and technologies—to become data-driven organizations. Even with those investments, business and clinical leaders can still draw incorrect conclusions from correct data and analytics. Worse, business and clinical leaders don’t know how often they draw incorrect conclusions from analytic visualizations.
Consider a data visualization of sepsis mortality like that in Figure 2:
Note that the four points shown in Figure 1 are more difficult to determine in Figure 2. This is the line chart Health Catalyst uses extensively to measure the accuracy of analytic interpretations. Surprisingly, only 4% of analysts, department leaders, and even board members correctly determine all four points: the baseline, extreme points, persistent shifts, and forecast. Unfortunately, misidentifying these four data points can lead to incorrect conclusions and, therefore, negative consequences. Let’s look at each.
Identifying baselines is not easy. In the sepsis mortality line chart, the baseline could be interpreted as 9%, 13%, or somewhere in between.
If leaders incorrectly identify the baseline as better than it really is, they may not set a much-needed goal to improve that measure’s performance. Alternatively, SMART (Specific, Measurable, Attainable, Relevant, Timebound) goals can be misdirected without a clear baseline.
Use case – Without a clear baseline, quality leaders may accidentally set a goal with a target rate for sepsis mortality that falls within the range of normal variation. While that goal will certainly be attainable, it is not meaningful. Worse, leaders may falsely conclude that the “improvement” is due to targeted interventions that, in reality, did nothing to improve performance.
Identifying extreme points is complicated. Notice in the line chart below that some extreme points are obvious, while others are not. The Month End Date 2016-02-29 is likely an extreme point. But what about 2018-02-28?
Misidentifying extreme points wastes time and opportunity.
Even worse, should leaders misidentify an extreme point of positive performance, conduct an RCA, and replicate the cause, all the effort to replicate the cause is worthless because the performance was within normal variation.
Use case – Without clear identification of extreme points, quality leaders may not recognize their sepsis mortality rate significantly worsened during a period with high rates of respiratory illness. They will not realize high rates of respiratory illness stretched their emergency department (ED) staff and disrupted sepsis care delivery. Without that knowledge, they will not set a goal to mitigate the impact of surges in the ED.
Identifying persistent shifts in performance is also challenging. Notice in the sepsis mortality chart below that the presence or absence of a persistent shift is unclear.
More unclear is whether multiple persistent shifts have occurred:
The time and opportunity costs of misidentifying persistent shifts are similar to the cost of misidentifying extreme points.
Even worse, should leaders misidentify variation as a persistent shift, conduct an RCA, and intervene, all the effort to reverse the cause is worthless.
Use case – Without clearly identifying persistent shifts in performance, quality leaders will not know whether the start of a labor shortage in critical care staff worsened sepsis mortality rates.
Performance data does not typically include a forecast for future performance. However, to aid in decision making about interventions for improving sepsis mortality, leaders must consider what the performance rate will be in the future if nothing changes.
Forecasting performance isn’t easy. In the sepsis mortality line chart, performance ranged from 11 to 19% from August 31, 2020, to March 31, 2021. Further, performance declined three months in a row (October through December) but improved during the last two months (February and March).
Consequently, predicting sepsis mortality rates during the next six months won’t be easy. Questions include the following:
A leader must correctly predict performance to know if they must intervene to improve performance. Even more difficult, a group of stakeholders must agree on the predicted performance to know whether they must intervene to improve performance. When stakeholders can’t agree on predicted performance, they can’t agree on whether to intervene to improve performance. One leader may confidently declare that performance will improve only to find that performance worsens.
Use case – A quality leader may incorrectly decide to shift their focus to performance in other areas based on an incorrect forecast that sepsis mortality rates will continue to improve.
Given the level of complexity in data interpretation—even when using visualizations—it’s no wonder only 4% of healthcare analytics users can correctly interpret the four key points. Fortunately, augmented intelligence (AI) can help healthcare leaders optimize their analytics for a myriad of strategic use cases, including the interpretation of performance data over time.
The use of AI in healthcare to strengthen traditional analytics has been termed augmented analytics. Gartner defines augmented analytics as, “the use of enabling technologies such as machine learning and AI to assist with data preparation, insight generation and insight explanation to augment how people explore and analyze data in analytics and BI platforms.”
Organizations must develop AI in healthcare that automatically and correctly applies advanced statistical methods for use with performance measurements over time (time series data). Accordingly, data science teams must develop AI that applies methods like statistical process control (SPC), changepoint detection, and forecasting:
Organizations must develop AI in healthcare that applies advanced statistical methods automatically because data science resources are limited. Most data science teams do not have time to manually apply advanced statistical methods for all charts that present performance over time—not with the volume of charts consumed every day across healthcare organizations by clinical, operational, financial, and population health leaders. Additionally, most business intelligence analysts do not have the knowledge base to apply advanced statistical techniques. Consequently, data science teams must create AI that automatically applies advanced statistical methods that guide leaders when interpreting performance over time.
To develop AI in healthcare that automatically applies advanced statistical methods for time series data, data science teams must develop AI that correctly applies SPC charts (with changepoint detection), and time series forecasting based on the variables below. Specifically, the AI must determine the appropriate techniques for different types of healthcare measures, including the values in each measure. Incorrect configuration may lead to incorrect conclusions and flawed decision making.
AI in healthcare must automatically and correctly apply SPC charts. To apply SPC charts correctly, the AI must be able to identify the measure type and distribution of the time series data first. Then, the AI must correctly apply the appropriate chart type, center method, and confidence interval calculation. Below, Table 1 illustrates the common mapping between some measure types, chart types, and confidence interval calculations.
AI in healthcare must automatically and correctly apply changepoint methods. The correct changepoint detection method is based on data type (closely related to measure type), length (number of data points), and period (e.g., daily, biweekly, monthly data). Consequently, the AI must automatically identify type, length, and period, then correctly apply changepoint methods such as Strucchange, BCP, E.Agglo, and E.Divisive.
AI in healthcare must automatically and correctly apply forecasting methods. The correct forecast method is based on length (number of data points) and period (e.g., daily, biweekly, monthly data). Consequently, the AI must automatically identify the length and period of the data and then correctly apply forecasting methods such as exponential smoothing, Tbats, and regression.
The scope of work described above is significant. Regrettably, most data science teams won’t have the time required to build algorithms that automatically and correctly apply advanced statistical methods for time series data.
However, without advanced statistical techniques, analysts, department leaders, and even board members can do more harm than good when they incorrectly interpret time series data. Fortunately for healthcare leaders, Health Catalyst has already built the algorithms described above that automatically apply SPC, changepoint detection, and forecasting.
Health Catalyst built its Healthcare.AI product suite to automatically apply advanced statistical methods into analytic outputs and visualizations. In other words, the Healthcare.AI product suite provides augmented analytics for analysts and leaders.
The Healthcare.AI product suite automatically, correctly, and transparently (not a black box) applies advanced statistical and machine learning methods in common analytic tools such as Power BI, Qlik, Tableau, as well as analytics applications provided by Health Catalyst. The Healthcare.AI product suite applies advanced statistical methods such as SPC+, Forecast, Power, Time Series+, Forest Plot+, Risk Adjustment, and Peer Finding.
The Time Series+ capabilities in Healthcare.AI apply to time series data. The “+” in “Time Series+” refers to the addition of SPC (with changepoint detection) and forecasting. The next two versions of the line chart (from Figure 2, above) compare the visualization of sepsis mortality rates with and without the Time Series+ capabilities in the Healthcare.AI product suite.
Without the Time Series+ Capabilities in Healthcare.AI
With the Time Series+ Capabilities in Healthcare.AI
Notice in the line chart above that the Time Series+ capabilities in Healthcare.AI clearly identify the baseline, extreme points, persistent shift, and forecast:
Using the Time Series+ capabilities in Healthcare.AI, analysts, leaders, and board members correctly identify baselines, extreme points, persistent shifts, and forecasted performance. Correct identification of those four points empowers leaders to accomplish the following:
In addition to the decisions listed above, analysts and department leaders use the Healthcare.AI product suite to make many more data-driven decisions. That’s the value of developing AI for strategic use cases in addition to transactional use cases like readmission risk, deterioration, sepsis detection, and smoking cessation. Advanced statistical techniques and AI ensure leaders do not do more harm than good with their data and analytics.
The process of transforming data into measurable value has many potential points of failure. Interpreting common reports and analytics is just one potential point of failure. Because there are many points of failure in transforming data into measurable value, Health Catalyst provides solutions for interpreting analytic outputs as well as solutions that eliminate other points of failure across the complete healthcare data value chain. The healthcare data value chain illustrates the building blocks required to transform raw data into measurable value (Figure 3).
While this article focuses on analytics as a common point of failure in transforming data into measurable value, AI depends on good data. Consequently, Health Catalyst urges health systems to think upstream and downstream from the point of analytics in the data value chain:
Upstream, health systems must think about points of failure such as data governance, data correctness, and data completeness—significant points of failure in the process of transforming data into measurable value. Downstream, health systems must think about points of failure such as interventions and change management. As you work to transform more data into more measurable value, remember these points of failure. And remember that Health Catalyst is the only company exclusively focused on healthcare that provides solutions proven to integrate and optimize the complete data value chain.
Healthcare leaders committed to making their organizations data driven by optimizing the complete healthcare data value chain can start by taking the analytics interpretation online assessment to understand their capabilities around interpreting analytics outputs. They can also access more knowledge through Health Catalyst thought leadership on AI. Finally, healthcare leaders can contact a Health Catalyst representative to discuss how to optimize the healthcare data value chain to transform more data into more measurable value.
Would you like to learn more about this topic? Here are some articles we suggest:
