DoD Influenza Forecasting Challenge 2019-2020

DoD Influenza Challenge: Seasonal Influenza Forecasting for U.S. Military Treatment Facilities

Currently military leaders and health providers are informed about past influenza activity through standard reports. Public messaging is used to inform service members how they can protect themselves and their family from the effects of influenza. However, when and where influenza will increase, how large the impact of the influenza season will be, and when the flu season will peak varies from season to season, making the preparation for and response difficult. Influenza forecasting can change that by offering the possibility to better plan ahead and potentially reduce the impact and burden of influenza illnesses among service members and their families.

To help support the development of the science of influenza forecasting and its application for public health preparedness the Defense Health Agency (DHA) in collaboration with CDC has initiated the U.S. Department of Defense (DoD) Military Treatment Facility (MTF) Influenza Forecasting Challenge to describe the timing, intensity, and short-term activity of an influenza season at the MTF level. If your organization is interested in participating in the DoD MTF Influenza Forecasting Challenge please email for more information:

dha.ncr.health-surv.mbx.dodflucontest@mail.mil

Submitted Forecasts

Use the interactive tool below to explore submitted forecasts for the current influenza season. Click throughout the season to examine forecasts received during a given week. To see the most recent forecasts, click the forecast week immediately preceding the dotted "Today" line. Peak week and intensity predictions are visualized by the stand-alone dots with confidence intervals, and week-ahead forecasts are visualized as the connected dots with confidence bands. A specific Military Treatment Facility (MTF) can be selected using the dropdown menu on the right side of the graph. Please note that forecasts will not display on Internet Explorer.

Data from all MTFs (with a sufficient number of encounters to calculate baseline) are made available to forecasting teams. Only forecasts for 26 MTFs will be submitted for evaluation (see Appendix A of the DoD Influenza Forecasting Challenge Guidance document).

Location: MADIGAN AMC-JB LEWIS-MCCHORD Disclaimer: Electronic health records for this site transitioned to the MHS GENESIS system in 2017, and therefore data is currently unavailable.

Location: 60TH MED GRP-TRAVIS AFB Disclaimer: Electronic health records for this site transitioned to the MHS GENESIS system in 2019 and therefore data is currently unavailable.

Forecast Targets

For each week during the season, participants will be asked to provide MTF-level probabilistic forecasts for the entire influenza season (seasonal targets) and for the next four weeks (four-week ahead targets). The seasonal targets are the peak week and the peak intensity of the influenza season for each MTF being forecast. The four-week ahead targets are the percent of outpatient visits experiencing influenza-like illness (ILI) one week, two weeks, three weeks, and four weeks ahead from the week of the most recently published data.

Season Onset Week

The onset of the season is defined as the Morbidity and Mortality Weekly Report (MMWR) surveillance week when the percentage of outpatient encounters (visits) for ILI at a MTF (rounded to the nearest 0.1) reaches or exceeds the baseline value for three consecutive weeks. Forecasted onset week values should be for the first week of that three week period. 2019–2020 ILI baseline values for each MTF will be provided to participants by AFHSB on Tuesday, October 15, 2019.

Seasonal Peak Week

Definition: The peak week will be defined as the MMWR surveillance week that the percentage of ILI encounters is the highest for a given MTF for the influenza season.

Motivation: Accurate and timely forecasts for the peak week can be useful for planning and promoting activities to increase influenza vaccination prior to the bulk of influenza illness. For healthcare, pharmacy, and public health authorities, a forecast for the peak week can guide efficient staff and resource allocation.

Seasonal Peak Intensity

Definition: The intensity will be defined as the highest numeric value, rounded to one decimal place that the percentage of ILI encounters reaches during the influenza season.

Motivation: Accurate and timely forecasts for the peak week and intensity of the influenza season can be useful for influenza prevention and control, including the planning and promotion of activities to increase influenza vaccination prior to the bulk of influenza illness. For healthcare, pharmacy, and public health authorities, a forecast for the peak week and intensity can help with appropriate staff and resource allocation since a surge of patients with influenza illness can be expected to seek care and receive treatment in the weeks surrounding the peak.

Short Term Forecasts

Definition: One- to four-week ahead forecasts will be defined as the percentage of ILI encounters for the target week, rounded to one decimal place.

Motivation: Forecasts capable of providing reliable estimates of influenza activity over the next month are critical because they allow healthcare and public health officials to prepare for and respond to near-term changes in influenza activity and bridge the gap between reported incidence data and long-term seasonal forecasts.

Military Treatment Facility ILI Encounter and Laboratory Data

Influenza-like illness (ILI) encounter data, aggregated total encounter data and laboratory result data is reported for individual military treatment facilities (MTFs). These data will be provided weekly to challenge participants via DoD SAFE. This data may also be used for model development for FluSight’s state, regional, and national forecasts.

Additional Data

Teams are welcome to use data sources for model development beyond the provided data.

Forecast Evaluation

All forecasts will be evaluated using the percentage of ILI encounters pulled from individual Military Treatment Facilities (MTFs), and the logarithmic score will be used to measure the accuracy of the probability distribution of a forecast. Logarithmic scores will be averaged across different time periods, the seasonal targets, the four-week ahead targets, and MTF locations to provide measures of model accuracy.

Logarithmic Score:

All forecasts will be evaluated using the observations pulled from the DoD surveillance system during week 28 of 2020, and the logarithmic scoring rule will be used to measure the accuracy of the probability distribution of a forecast. If p is the set of probabilities for a given forecast, and pi is the probability assigned to the observed outcome i, the logarithmic score is:

$$S(\mathbf{p},i) = \text{ln}(p_i)$$

For onset and peak week, the probability assigned to the single bin containing the observed outcome (based on the percentage of ILI encounters value). If onset is never reached during the season, only the probability assigned to the bin for “none” will be scored. In the case of multiple peak weeks, the probability assigned to the bins containing each peak week will be summed.

Undefined natural logs (which occur when the probability assigned to the observed outcome is 0) will be assigned a value of -10. Forecasts which are not submitted (e.g., if a week is missed) or that are incomplete (e.g., sum of probabilities greater than 1.1) will also be assigned a value of -10. Example: A forecast predicts, for a given MTF, there is a probability of 0.3 (i.e., a 30% chance) that the flu season starts on week 45, with the remaining 0.7 probability distributed across other weeks according to the forecast. Once the flu season has started, the prediction can be evaluated, and the ILI data show that true onset was on week 45. The probability assigned to week 45, 0.3, would be derived, and the forecast would receive a score of log(0.3) = -1.20. If the season started on another week, the score would be calculated on the probability assigned to that week.

References:

• Gneiting T and AE Raftery. (2007) Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association. 102(477):359-378. Available at: https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf

• Rosenfeld R, J Grefenstette, and D Burke. (2012) A Proposal for Standardized Evaluation of Epidemiological Models. Available at: http://delphi.midas.cs.cmu.edu/files/StandardizedEvaluationRevised12-11-09.pdf.

FluSight_DoD Package

The FluSight_DoD R package contains functions to help create and format forecasts, read and verify forecast CSVs, and score forecasts. These are the functions that will be used at AFHSB-IB to verify and score submitted forecasts. Teams are welcome to use these tools to ensure their forecasts fit the required template and score their forecasts prior to receiving official scores from AFHSB-IB.

Install and load package.

Download 'FluSight_DoD.zip', the folder containing the edited master files, from the link provided by AFHSB-IB. Open the “FluSight_DoD.R” R program in the main directory. 

Use the annotated code in the FluSightDoD.R script to load all functions in the 'R' subfolder of the FluSightDoD main folder.

sourceEntireFolder("INSERT LOCAL FILEPATH/FluSight_DoD/R/")

Read in MTF forecast entry CSV file

entry <- read_entry("your_csv.csv", challenge = "state_ili")

Read in MTF past_baselines CSV file provided by AFHSB-IB

past_baselines <- read.csv("past_baselines.csv", header=T, sep=',')

Create file of observed truth

truth <- create_truth(fluview = FALSE, year = 2019, challenge = "state_ili")

Score a weekly entry against the observed truth

exact_scores <- score_entry(entry, truth, challenge = "state_ili")

Note: function names from the FluSightDoD package are identical to the original functions from the CDC-produced FluSight R package. To distinguish between packages, the convention "FluSight::FUNCTIONNAME()" should be used when calling the original (not from the FluSight_DoD folder) version of a given function.

Guidance Documents

Guidance for the DoD MTF Influenza Forecasting Challenge 2019–2020 is available here

An empty copy of the official submission template is available here.

The instructions for registering your model and submitting forecasts can be found here.

Model Questionnaire

Please submit one copy of this form for each model you submit. Please send an updated form if you update your model during the season.