3: Analyzing Behavioral Health Surveillance Data

Section 2 of this toolkit provided an overview of several data sources that could be used to conduct BH surveillance in the PHE context and introduced several potential indicators to consider monitoring in your local jurisdiction.

This section discusses, at a high level, the methodologic approaches that you can consider for BH surveillance, depending on the question(s) you want to explore or the indicator(s) you want to monitor or surveil; the type(s) and volume of data you have for analysis; the timing in relation to a PHE; and the capacity (e.g., time, competing priorities) and technical proficiency that you and your colleagues bring to this work.

We will walk through the basics of analyzing your BH surveillance data. These consist of the following:

• defining the questions you want to explore about BH in your jurisdiction
• understanding your data
• describing your data
• using smoothing to better visualize your data
• plotting a time series.

This section of the toolkit also briefly discusses ARIMA (autoregressive integrated moving average) modeling, although we recommend consulting other resources for step-by-step instructions on performing this type of analysis if you are new to it or need a detailed refresher. Finally, it covers the importance (and current limitations) of examining your data by subgroup, when possible, to understand variation in the BH indicator by sociodemographic characteristics and potential inequities in BH in the PHE context.

If you already are familiar with the basics and want to explore more-advanced methods, including setting alert thresholds, using anomaly detection algorithms, and conducting statistical testing with BH surveillance data, click on the button below.

Explore more-advanced methods if you're comfortable with the basics.

Define the Question(s) About Behavioral Health That You Want to Explore in Your Jurisdiction

A good starting point is to clearly define the questions you want to explore through your BH surveillance efforts.

Let’s say your jurisdiction experienced a recent PHE that caused widespread destruction and loss of life. You are hearing from partners in the health care system that they are observing an uptick in ED visits for SI among adolescents, or you are getting queries from high school principals and the media about whether this anecdotal trend is “real” and, if so, what to do about it.

Using the example BH indicator of proportion of all visits to the ED that were for SI, Summary 3.1 lists questions that you may want to answer about BH impacts on your community from a recent (or anticipated) PHE and a series of suggested steps, typically performed in the order shown, to answer them.

Questions to Explore About the Behavioral Health Impacts of a Public Health Emergency in Your Jurisdiction and Approach to Answering Those Questions

Questions About BH Impacts of a PHE		Suggested Approach
To what extent is the BH indicator (e.g., proportion of ED visits that were for SI) in my jurisdiction changing after a PHE?		Understand your data Describe your data Smooth your data Visualize your data with a time series plot Use modeling to compare observed values with what you would expect based on past trends Implement anomaly detection algorithms
When after the PHE did I start to see a change in the BH indicator (e.g., proportion of ED visits)?
Does the change in the BH indicator (e.g., proportion of ED visits) vary across population subgroups?		Stratify by subgroup of interest, if data are available

Understand Your Data

Key Characteristics of Data Sources for Behavioral Health Surveillance

Data Are a Univariate Time Series

Existing data sources that could be used for BH surveillance in the PHE context have a key characteristic in common: They are a univariate time series, meaning that they consist of data on a single BH indicator collected from the same unit (such as zip code or call center) over time. The length of your time series (e.g., whether you have multiple data points from before, during, and after the PHE) will affect the types of analytic methods that you can apply to your data to answer the questions you have about BH impacts of PHEs.

Behavioral Health Outcomes Are Reported as Counts but Need Transformation

The BH outcome of interest is reported as a count (e.g., number of EMS calls for overdose). For analytic purposes, a count often needs to be converted to a

• proportion (e.g., share of total ED visits that were for SI)
• rate (e.g., share of total ED visits that were for SI per month)
• relative incidence (e.g., post-PHE incidence of a BH indicator relative to its incidence in the same month pre-PHE).

Spatio-Temporal Granularity Will Vary

Existing data sources can have variable spatio-temporal granularity (i.e., level of detail in your data).

Data sources with high spatial granularity include data that are disaggregated to small geographic units, such as zip code, Census tract level, or county (compared with, for example, a region of the country or a state).

Data sources with high temporal granularity include data that are reported on a frequent basis, such as daily or weekly (as opposed to quarterly, annually, or more than annually). Temporal granularity is especially important in the context of fast-moving PHEs.

The data sources we use in our exploratory analyses demonstrate varying degrees of spatio-temporal granularity. If your data have low spatial granularity, it may not be possible to target PH interventions to a particular location. If they have low temporal granularity, it may be more difficult to detect an anomaly or there may be a delay in detection.

Data Sources May Show Autocorrelation

For time series data, it is important to look for autocorrelation—the correlation of a value in a data set with its own previous values. Doing so quantifies the similarity between longitudinal observations as a function of time separation (i.e., lag) between them. If your data are autocorrelated, you’ll notice a cyclical pattern that may repeat on a weekly or monthly basis. For instance, call volume to a
2-1-1 call center is consistently lower on certain days of the week and higher on others.

If your dataset is strongly autocorrelated, you will want to account for this feature in your analytic model selection, which allows you to more accurately detect potential anomalies in the BH indicator. Click the button below to see an example of detecting and addressing autocorrelation in 2-1-1 call data.

See an example of addressing autocorrelation in 2‑1‑1 call data.

Describe Your Data

As with most datasets that you are beginning to analyze, you’ll want to calculate descriptive statistics to understand the following data characteristics:

• central tendency, using the mean or median statistic
• variation, or the dispersion of the data values around the measure of central tendency.

For time series data that you are using for BH monitoring and surveillance, there are a few additional considerations.

Consider Examining Different Geographic Aggregations

If your analysis covers multiple jurisdictions, you may want to calculate descriptive statistics at different geographic aggregations. For instance, if data are available, you can compare statistics at the Census tract, zip code, city, county, and state levels to uncover interesting patterns you may want to explore further.

If You Plan to Conduct Modeling, Examine Your Time Series for Non-Stationarity

If these descriptive statistics vary over time (e.g., the variance increases), your time series data are considered nonstationary and will need to be transformed (often by differencing) before modeling. Non-stationarity can be assessed through statistical testing. See the later section on ARIMA modeling.

Consider Smoothing Your Data

Analysis of time series data requires calculating descriptive statistics over time. You may need to examine summary statistics for discrete periods, such as before, during, and after the PHE; by day of the week; or by month. These summary statistics, when plotted over time, smooth your data and allow the important patterns to stand out (see Analysis 3.1).

Moving averages, which calculate averages across a moving time window, are a common approach to preparing to plot a “noisy” time series (i.e., one that is reported daily and thus has many data points) (Table 3.1). It is useful to set the length of the time window to match cyclical patterns in the data. For instance, if your data show a strong dependence on the day of the week (i.e., they show autocorrelation), then calculating a seven-day moving average may help smooth your data, as demonstrated in the section on time series plots (Analysis 3.1). Alternatively, using your examination of the data, you could instead take a 14- or 28-day moving average. Note that the smoothness increases with the length of the time window.

In the case of a sparser time series, you can use an exponentially weighted moving average (EWMA) to help you identify shifts in the data (Table 3.1).

Approaches to Smooth Your Time Series Data to More Easily Visualize Patterns and Anomalies

Approach	Description
Moving average	The moving average is a good approach if your data are noisy, which is often the case when the data are granular and available on a daily basis. You average data values over a time window to eliminate noise and produce a smoothed trend. If you plot the individual data points (e.g., daily counts) and the smoothed trend (e.g., seven-day average), you will notice that a shift is easier to see with smoothed data (see Analysis 3.1, showing a shift in 2-1-1 calls in Broward County, Florida, after Hurricane Maria).
EWMA	The EWMA is a good approach for sparser time series. The statistic for time t is computed recursively as a weighted linear combination of the current value and the previous day’s moving average. This approach requires you to decide how much relative weight should be given to recent observations. You can explore different weights to decide which are appropriate.

Visualize Your Data with a Time Series Plot

After calculating summary statistics on your data, looking for autocorrelation (if relevant), and smoothing your data (if appropriate), you will next visualize your data using a time series plot, which will allow you to identify obvious shifts in the data.

By convention, the outcome is plotted on the y-axis and time is plotted on the x-axis. The appropriate unit of time will depend on how frequently the data are collected, the duration of the PHE, and the time frame of desired surveillance of its impacts (e.g., short, medium, or long term). Analysis 3.1 shows a time series plot of daily (raw) and smoothed (with a seven-day average) counts of 2-1-1 calls in Broward County, Florida, in 2016 and 2017 (the year Hurricane Maria affected this county). The increase in 2-1-1 calls after Hurricane Maria in 2017 (yellow curve) is more readily visualized in the bottom panel.

The raw data from 2-1-1 calls made in Broward County, Florida, from 2016 to 2017, the time period before and after Hurricane Maria, show a series of peaks and valleys from about 50 calls per day to more than 500 calls per day. The smoothed data make it move obvious that calls peaked immediately after the hurricane, which occurred mid-September 2017.

The raw data from 2-1-1 calls made in Broward County, Florida, from 2016 to 2017, the time period before and after Hurricane Maria, show a series of peaks and valleys from about 50 calls per day to more than 500 calls per day. The smoothed data make it move obvious that calls peaked immediately after the hurricane, which occurred mid-September 2017.

Conduct Statistical Testing

After creating time series plots of your data, you may want to compare observed values of the BH indicator of interest with what you would expect based on past trends. For those with capacity for more-complex analyses, either in your agency or through external partnerships, the next step in conducting ongoing, real-time monitoring and surveillance of BH indicators in the PHE context is to build a model and apply it to your time series.

Autoregressive Integrated Moving Average Modeling

The most common type of model you will likely use to examine many of the data sources discussed in this toolkit is the ARIMA model, which predicts the evolving values of the BH indicator based on a moving average of its past values. ARIMA has three components (i.e., parameters):

• p (autoregressive) tells the model how many prior values to use (based on autocorrelations within the time series);
• q (moving average) is different from the moving averages under smoothing models and refers to the number of error terms on which one should regress the current value; in a moving average model, past errors are propagated into future values;
• d (differencing) tells the model whether to take first or second difference of values to allow for the time series to become stationary before the autoregressive or moving average modeling (this is the integrated part).

When any of these three inputs are set to 0, the ARIMA model is simplified. When p and q are large, the models get more complex. A good rule of thumb is to identify parsimonious models (i.e., models with small p and q values that fit the data well and avoid differencing, if it is not needed to attain stationarity).

These models provide estimates of forecast uncertainty in the form of standardized residuals. A standardized residual is the difference between the model-predicted value and the observed value, divided by the standard deviation of the residuals. More simply, standardized residuals provide a measure of the degree to which the model accurately predicts the value of the BH indicator over time. Smaller residuals indicate better model fit; larger residuals indicate poorer model fit; large residuals may indicate the presence of a disruption associated with a PHE.

Access sample code in R to generate an ARIMA plot with your data.

Access a sample analysis showing the use of a moving average to set an alert threshold for 2-1-1 call volume in Broward County, Florida, after a hurricane and the start of COVID-19.

Apply Anomaly Detection Algorithms

After building your model (e.g., ARIMA) and applying it to your time series, you will get a set of residuals. Then, you can create one or more alert rules to detect anomalies in your data. Click on the button to explore a few of the potential approaches for anomaly detection.

Learn more about approaches to detecting anomalies in your surveillance data.

Stratify by Sociodemographic Characteristics, If Available in Your Data

Ideally, data sources would provide sufficient sociodemographic information so that you could monitor inequities in BH needs and the impacts of PHEs by race or ethnicity, age, gender, disability status, language status, many other key sociodemographic characteristics, and their intersections. However, the collection of this level of detail on individual characteristics is complex and resource intensive. A few of the most-promising data sources for conducting BH surveillance allow you to examine variation in trends in BH indicators by subgroup, as shown in Summary 3.2.

Availability of Age, Gender, and Race or Ethnicity Variables in the Most-Promising Data Sources for Behavioral Health Surveillance

	Age	Gender	Race/Ethnicity
UI claims
2-1-1 calls
PCC calls
Prescription medication fills
OTC sleep aid sales
Hotline calls
ED visits
EMS activations

Note: A green checkmark denotes available and complete data. A yellow question mark indicates that data missingness may be a concern. A red X indicates that this information is not available. Information on race or ethnicity in prescription medication data depends on the source. Claims data (e.g., Medicaid) include this variable; IQVIA data do not. For EMS activations, while age, gender, and race are required data fields, there is variation in the degree of missingness in these variables.

To illustrate the importance of examining BH indicators across subgroups, we provide an analysis of initial UI claims in Los Angeles County in the context of the COVID-19 pandemic declaration, stratified by gender (Analysis 3.2). Initial UI claims were one of the few promising data sources for BH surveillance that had complete information on age, gender, and race, allowing monitoring of this upstream indicator of BH risk at the intersection of these key characteristics.

The weekly counts of initial UI claims in Los Angeles County show that claims submitted by women peaked around the 12th week of 2020, with over 140,000 claims and that claims submitted by men peaked around the 35th week of 2020, with just under 140,000 claims.

A Note About Health Equity

Age, gender, and race or ethnicity can be useful to begin exploring how PHEs affect subpopulations at the intersections of race, gender, and age. However, better understanding how to disaggregate, interpret, represent, and act on data in meaningful ways requires the inclusion of, and active participation by, the individuals affected by PH decisionmaking in the context of a PHE (Chandra et al., 2022). For more information on using data to advance equity and community health, see this research brief (Gaddy and Scott, 2020) published by the Urban Institute (undated) as part of its Elevate Data for Equity project.

Learn more about addressing equity in BH surveillance.

What Do These Analyses Look Like in Practice?

To illustrate the methods described in this part of the toolkit and the more technical modeling described elsewhere, we conducted a series of proof-of-concept, exploratory analyses with five of the eight most-promising data sources and identified real-world examples of the use of data on two more of the most-promising data sources: EMS activations and ED visits. The eighth promising data source, the 9-8-8 Suicide & Crisis Lifeline, went live as this toolkit was going into production, so data were not available for exploratory analyses. We used data from one or more U.S. counties that were affected by a variety of PHEs: COVID-19, wildfires, a winter storm leading to statewide power outages, and hurricanes.

Some of the methods discussed here will be applicable to a variety of data sources; others may be relevant only for a particular data source or may be possible only with sufficient pre-PHE data. These proofs of concept are intended to show some potential ways to apply the analytic methods, depending on your needs, the specific questions you want to answer, the data you have available or can obtain, the data science capacity at your agency, and your relationships with other institutions that can provide data science support. The exploratory analyses start with the most-basic methods for examining your data and progress to more-advanced analyses. Click on the buttons below for results of exploratory analyses conducted with each of the most-promising data sources or real-world examples of their use.

Upstream

UI Claims

Midstream

2-1-1 calls PCC calls Prescription medication fills OTC sleep aid sales

Downstream

ED visits EMS activations

Find More Examples of the Data Sources and Methods Covered Here

Within each of the data source overviews (Appendix D) and in the exploratory analyses using these sources (Appendix E), you’ll see a button that looks like this:

Access selected publications that have used this data source for monitoring and surveillance.

Use these buttons to navigate to a list of peer-reviewed publications that use that particular data source and a brief synthesis of available literature. These studies serve as useful examples of how the analytic methods described here can be used for monitoring and surveillance of BH and/or in the PHE context.

Consider Ways to Build Data Science Capacity

If there are data sources, analytic methods, or visualization techniques that your PH agency is not able to use on its own and that you think you may want to use during a PHE, consider ways to build capacity well before the PHE occurs, as well as after, when the acute response phase is over. Summary 3.4 provides three suggestions and briefly describes capacity-building in action.

Build Capacity

1. Partner with other institutions for data science support
2. Obtain funding and other resources related to data science
3. Upskill staff through targeted workforce development and/or request technical assistance

Capacity in Action

• The Kentucky Injury Prevention and Research Center, a partnership between the Kentucky Department for Public Health and the University of Kentucky’s College of Public Health, conducted analysis of multiple data sources to both monitor prescription drugs and track drug overdoses as part of its Overdose Data to Action surveillance strategy.
• The Pandemic-Ready Interoperability Modernization Effort (PRIME), a collaboration between CDC—as part of its Data Modernization Initiative—and the U.S. Digital Service, aims to strengthen data quality and information technology systems in state and local health departments.
• The Mental Health Research Network is a consortium of 14 research centers that collect data and conduct research on pressing mental health issues and rapidly disseminate the results.
• SAMHSA’s Disaster Technical Assistance Center can help you plan for and respond to mental health and substance use–related needs after a disaster, and the Tribal Training and Technical Assistance Center offers training and technical assistance on mental health and substance use disorders. These are two of several resources in SAMHSA’s practitioner training network.