Modeling and Surveillance of Reporting Delays of Mosquitoes and Humans Infected With West Nile Virus and Associations With Accuracy of West Nile Virus Forecasts

Key Points español 中文 (chinese)

Question  What are the operational challenges limiting effective implementation of a real-time West Nile virus (WNV) forecasting system?

Findings  In this modeling study of historical and real-time mosquito WNV assay results and human medical records, delays in data reporting for both infected mosquitoes and human WNV cases were associated with a reduction in average WNV forecast accuracy.

Meaning  For public health departments and mosquito abatement districts to integrate forecasting effectively into operational decision making, the relaying of real-time health and environmental surveillance data should be prioritized.

Importance  West Nile virus (WNV) is the leading cause of domestically acquired arboviral disease.

Objective  To develop real-time WNV forecasts of infected mosquitoes and human cases.

Design, Setting, and Participants  Real-time forecasts of WNV in 4 geographically dispersed locations in the United States were generated using a WNV model-inference forecasting system previously validated with retrospective data. Analysis was performed to evaluate how observational reporting delays of mosquito WNV assay results and human medical records were associated with real-time forecast accuracy.

Exposures  Mosquitoes positive for WNV and human cases.

Main Outcomes and Measures  Delays in reporting mosquito WNV assay results and human medical records and the association of these delays with real-time WNV forecast accuracy.

Results  Substantial delays in data reporting exist for both infected mosquitoes and human WNV cases. For human cases, confirmed data (n = 37) lagged behind the onset of illness by a mean (SD) of 5.5 (2.3) weeks (range, 2-14 weeks). These human case reporting lags reduced mean forecast accuracy for the total number of human cases over the season in 110 simulated outbreaks for 2 forecasting systems by 26% and 14%, from 2 weeks before to 3 weeks after the predicted peak of infected mosquitoes. This period is the time span during which 47% of human cases are reported. Of 7064 mosquito pools, 500 (7%) tested positive; the reporting lag for these data associated with viral testing at a state laboratory was a mean (SD) of 6.6 (2.6) days (range, 4-11 days). This reporting lag was associated with decreased mean forecast accuracy for the 3 mosquito infection indicators, timing, magnitude, and season, by approximately 5% for both forecasting systems.

Conclusions and Relevance  Delays in reporting human WNV disease and infected mosquito information are associated with difficulties in outbreak surveillance and decreased real-time forecast accuracy. Infected mosquito lags were short enough that skillful forecasts could still be generated for mosquito infection indicators, but the human WNV case lags were too great to support accurate forecasting in real time. Forecasting WNV is potentially an important evidence-based decision support tool for public health officials and mosquito abatement districts; however, to operationalize real-time forecasting, more resources are needed to reduce human case reporting lags between illness onset and case confirmation.

During the past 30 years, the Western Hemisphere witnessed the arrival of a number of unexpected and important arthropod-borne viruses (arboviruses): dengue trickled in with small outbreaks that became more aggressive in the 1990s,¹ West Nile virus (WNV) emerged in 1999,² chikungunya in 2013,³ and in 2015, Zika virus became a global health emergency.⁴ Arboviruses are viruses transmitted by arthropods, predominantly mosquitoes and ticks, and are often maintained in complex zoonotic transmission cycles.⁵ Arboviruses are normally maintained in wild animal reservoirs, such as nonhuman primates or birds, making elimination difficult and prediction of spillover to humans challenging.^6,7 As a consequence, effective allocation of public health resources is complicated and often reactive. Quantitative decision support tools that accurately and rapidly forecast the progression of arbovirus outbreaks and future human case numbers have the potential to help public health officials better prevent, identify, and treat these emerging infectious diseases.

In the continental United States, the arbovirus most monitored is WNV (family Flaviviridae, genus Flavivirus). West Nile virus is the leading cause of domestically acquired arboviral disease and has produced the 3 largest arboviral neuroinvasive disease outbreaks ever recorded.⁸ In response to the emergence of WNV in the Western Hemisphere, the US Congress provided funding to establish ArboNET, an electronic surveillance system for arboviral diseases in the United States, and to support surveillance activities in affected states and large cities. Surveillance consists of 2 distinct but complementary measures: (1) a network monitoring WNV activity in mosquito vectors and nonhuman vertebrates and (2) a passive system that records WNV human disease, which can be used to quantify disease burden and identify seasonal, geographic, and demographic patterns of morbidity and mortality in humans.⁹

The principal objective of WNV surveillance is to provide a robust data stream for public health officials and community-based mosquito management programs that quantifies the intensity of virus transmission and can be used as an evidence-based decision support tool.^10,11 Ideally, this surveillance should also support accurate real-time WNV forecasting. Such a forecasting system would provide public health officials and mosquito control programs with expected likelihoods describing how outbreaks may progress. These forecasts would, in turn, support an evidence-based decision-making approach for targeting control of infected mosquito populations (ie, larviciding [insecticide applications targeting mosquito larvae] and adulticiding [insecticide applications targeting adult mosquitoes]), alerting the public to future periods of elevated WNV spillover transmission risk, and identifying when to intensify blood donor screening. However, a forecasting tool must be tested in real time to determine whether the forecasting apparatus and observational network are sufficient to support accurate operational prediction.

In a recent study,^12,13 we showed that accurate and reliable forecasts of WNV outbreaks can be made using WNV surveillance data and a mathematical model representing the transmission dynamics of WNV among mosquitoes and birds, as well as spillover to humans. This model system was able to retrospectively forecast mosquito infection rates prior to the week of mosquito peak infection and to forecast accurately the seasonal total number of human WNV cases prior to when the majority of cases were reported. However, for this forecasting system to operate effectively in real time, timely data streams are needed. Mosquito monitoring practices vary around the country and are influenced by local socioeconomic factors, the tax base, and the public and political will to budget for mosquito surveillance.^6,14-16 In addition, the reporting of human WNV disease has long lag times that influence our understanding of how an outbreak is progressing and challenge our ability to generate accurate real-time forecasts.

We report the results from real-time forecasting during the 2017 WNV season in 4 mosquito vector control districts (Chicago, Illinois¹⁷; Coachella Valley, California¹⁸; Suffolk County, New York¹⁹; and St Tammany Parish, Louisiana²⁰) (eFigure 1 in the Supplement) as well as operational challenges associated with these efforts. Each week during the 2017 season, observations of human WNV cases and mosquito infection rates were assimilated using a data assimilation method,¹⁷ and a probabilistic forecast was generated in real time using 2 separate prediction models.^12,13 We report the accuracy of these real-time forecasts and quantify the association between reporting lags and forecast accuracy.

This analysis used 2 forecasting systems, a baseline model with no temperature forcing¹³ and a model that includes environmental forcing (average daily temperature [ie, climatology] for the region) by accounting for temperature modulation of the extrinsic incubation period for mosquitos.¹² Both models used a standard susceptible-infected-recovered epidemiological construct and were optimized using a data assimilation method and 2 observed data streams: mosquito infection rates and reported human WNV cases. The methodological approach and its application in real-time infectious diseases forecasting are briefly described later in this section. For further details on the validation of the WNV model-inference forecasting systems, see the articles by DeFelice et al.^12,13 The remainder of the Methods section describes how we used the observed reporting lags from the 2017 WNV season to evaluate the associations of reporting delays with forecast accuracy for 110 historical outbreaks.¹² We followed the Consolidated Health Economic Evaluation Reporting Standards (CHEERS)²¹ reporting guideline to report the difference in likelihood of an accurate forecast given current reporting lags relative to ideal reporting of all data in real time. This research received institutional review board approval from Columbia University, which waived requirements for participant informed consent because the study used deidentified data.

Abatement districts carried out weekly mosquito surveillance subject to budgetary constraints, the severity of WNV, mosquito nuisance problems, and weather. This surveillance was conducted using gravid, carbon dioxide–baited, light, and sentinel traps, depending on the mosquito abatement district and the species being trapped. The total number of traps set each week within a season and mosquito abatement district varied. For more details on the testing protocol, differences among abatement district collections, mosquito types, and how data were transferred for forecasting, see the eAppendix and eTable in the Supplement.

All mosquito pools reported at the time of forecast were used for model optimization. A maximum likelihood approach was used to estimate the weekly prevalence of infected mosquitoes²² given the number of mosquitoes tested in each pool and detection of virus. The prevalence of infected mosquitoes represents both noninfectious mosquitoes, for which the virus is only in the midgut, and infectious mosquitoes, for which the virus has spread to the salivary glands. The eTable in the Supplement presents statistics on mosquito collections by abatement district, including the primary vector, trap type, variability in traps per week, variability in mosquito pools per week, weeks forecasted (weeks with active monitoring after WNV virus was identified), the estimated peak mosquito infection rate (peak magnitude), the week of peak infection rates (peak timing), the number of Culex pools assayed for WNV, and the number of WNV-positive Culex pools.

In the United States, WNV is a nationally notifiable disease. Medical professionals are required to report a positive case of WNV to their local health department within 7 days of diagnosis; California requires reporting within 1 day.²³ Once a case is reported, either of neuroinvasive or nonneuroinvasive (West Nile fever) disease (eAppendix in the Supplement), the local health department opens an investigation and confirms the case. On confirmation, the case is reported to the local mosquito abatement district. At that point, the human case record is available for assimilation into the model and can be backdated using the onset-of-illness date. For details on the differences in human case reporting among districts, as well as reporting lags between onset of illness and reporting to health departments and confirmation of the cases, see the eAppendix in the Supplement.

To better understand human case reporting lags, historical information from the city of Chicago²⁴ was obtained describing the lag time between the date of illness onset and the date a suspected WNV case was reported to the local health department. Unfortunately, similar historical information was not available describing the typical period for determining whether a suspected case is probable or confirmed. Only observations from the 4 study locations forecast during the 2017 outbreak were used to estimate the reporting lag from disease onset date to when the investigation of a human WNV case was closed.

The forecast system relies on 3 components: (1) a core compartmental epidemiological model representing the transmission dynamics of WNV among mosquitoes and birds, as well as spillover to humans^12,13; (2) WNV surveillance data, ie, vector mosquito WNV infection rates and reported human WNV cases; and (3) a data assimilation method (here, the ensemble adjustment Kalman filter²⁵). The data assimilation method uses the surveillance data to recursively inform and optimize an ensemble of model simulations and, in so doing, provide an improved, posterior estimate of the true state as well as estimates of unobserved state variables and parameters. The forecasting is then generated in 2 successive steps. First, an ensemble of model simulations is recursively optimized using the ensemble adjustment Kalman filter and weekly observations of mosquito infection rates and human WNV cases until the week at which a forecast is to be initiated (in real time, this is the current week). Through the recursive ensemble adjustment Kalman filter optimization, model variables and parameters are better aligned with the local dynamics of the outbreak as thus far observed. Next, a forecast is generated by integrating the optimized ensemble of model simulations through to the end of the season. For further details on the WNV model-inference forecasting systems, see the eAppendix in the Supplement or the articles by DeFelice et al.^12,13

In real time, the future progression of an outbreak is unknown; thus, a reference point for the forecasts is needed. Here, the forecast reference is defined as forecast lead time, which is the number of weeks before or after the predicted peak of infected mosquitoes. This predicted peak is defined as the maximum weekly mosquito infection rate from the mean of the ensemble of simulations. This point can occur either in the posterior (the period for which observations have been assimilated) or in the mean ensemble forecast, depending on the trajectory of the forecast. Forecast lead time was calculated by subtracting the week of predicted peak mosquito infection from the week of forecast generation (in real time, the current week).²⁶

Expected accuracies were derived from historical forecast performance for the expected forecast lead time. A forecast was deemed accurate if (1) it peaked within 1 week of the observed peak of infected mosquitoes; (2) the maximum mosquito infection rate was within 25% of the observed peak infection rate; (3) the total number of human cases over the entire season was within 25% or 1 case of the total number of reported cases, whichever was larger; and (4) total infected mosquitoes were within 25% of the observed.

To better understand the factors associated with forecast accuracy, retrospective forecasts for 110 historical outbreaks¹² were run with mock reporting delays generated from the 2017 real-time forecasting results (Table). The 110 historical outbreaks were from 12 geographically diverse US counties. Two of the real-time forecast counties (Suffolk County and St Tammany Parish) were part of this analysis. For more details on the 110 retrospective forecast sites and years, see the article by DeFelice et al.¹²

The generated reporting lags represent different reporting scenarios. For mosquito reporting lags, we used a 1-week lag to represent the delay between mosquito trapping and viral testing similar to the reporting lag seen when mosquito abatement districts send samples to the state for viral testing (IM₁). We compare these lags with mosquito abatement districts that do in-house viral testing and have test results within the same week (IM₀). To capture the variability in human case reporting, we compared 3 scenarios: (1) all human cases were reported within the week of illness (HC no lag) and all cases are compliant with the national notifiable disease guidelines, (2) human case reporting represented observed lags associated with case reporting to the health department (HC reported), and (3) human case reporting represented the lags associated with health department confirmation (HC confirmed). For these retrospective forecasts, human case reporting lags were randomly selected from a Poisson distribution.

Real-time forecasts were generated each week during the 2017 season following assimilation of the most recent available information. For the 4 locations, 37 human WNV cases were reported, and 212 179 collected mosquitoes were tested for WNV in 7064 pools (500 pools [7%] were positive). The accuracy of each real-time forecast by calendar week of forecast is shown in eFigure 2 through eFigure 5 in the Supplement. Overall, the real-time forecasts were able to estimate accurately the timing and magnitude of the peak of infected mosquitoes and the total number of infected mosquitoes for the season in real time prior to the peak timing of infected mosquitoes; however, forecasts of human WNV cases showed little skill (the term skill is used to describe a model’s measure of prediction accuracy to the observation being predicted). The breakdown in forecasting human cases occurred 1 to 2 weeks prior to the infected mosquito peak, a critical time when spillover events begin. During this period, the forecasts began to underestimate the total number of human WNV cases for the season.

Real-time forecast accuracy for the 4 locations was then compared with the accuracy previously derived during retrospective forecast^12,13 (Figure 1). Forecast accuracy for the peak week of mosquito infection rates was close to that generated for retrospective outbreaks. The temperature-forced forecasts oscillated around the historical probability of accurately forecasting each lead week, whereas the baseline forecasts were biased somewhat below the retrospective forecasts. The real-time peak magnitude forecasts did better than the retrospective predictions at 4 and 0 weeks prior to the predicted peak for the baseline and temperature-forced forecasts, respectively. The seasonal total number of infected mosquitoes was accurately forecast beginning 1 week past the predicted peak. Human case forecasts showed some skill 6 to 3 weeks prior to the predicted peak of infected mosquitoes but were not able to accurately forecast cases from 2 weeks prior to 3 weeks after the predicted peak (lead weeks −2 to 3), a forecast period on which we will now focus. As we will show, some of this forecast error was due to human case reporting lags (Figure 2).

Using 110 historical outbreaks from multiple locations around the country, we evaluated how reporting lags of human cases are associated with forecast accuracy under the scenario of no mosquito testing lag (an onsite laboratory) and different human case reporting lags (Figure 3, Figure 4, and the Table). There was a mean (SD) lag of 2.1 (1.4) weeks (range, 0-8 weeks) between disease onset and when the local health department received the data. This lag reduced average forecast accuracy for total human cases in the temperature-forced and baseline forecasts by 10% and 2%, respectively, for lead weeks −2 to 3, the period when 47% of human cases were reported. Once the health department receives a potential case of WNV, it needs to be confirmed. The combined delay due to notification of the health department and confirmation of the case produced a mean (SD) lag of 5.5 (2.3) weeks (range, 2-14 weeks) between onset of illness and data release to the public during 2017 in the 4 study regions (Figure 2). During lead weeks −2 to 3, this longer combined lag time was associated with reduced average forecast accuracy of 26% and 14% for the temperature-forced and baseline forecasts, respectively.

In addition to accounting for the human case reporting lags, we also evaluated the associations between the reporting lag for mosquito infection rates and processing occurring at a state laboratory. Suffolk County¹⁹ and St Tammany Parish²⁰ used a state laboratory, which subsidized testing but delayed processing. St Tammany Parish and Suffolk County had lags of 4 to 10 days and 5 to 11 days, respectively, between mosquito trapping and state provision of test results. The mosquito reporting lag associated with processing occurring at a state laboratory was a mean (SD) of 6.6 (2.6) days; thus, we represented this lag as a 1-week delay of mosquito infection rate reporting (Figure 3, Figure 4, and the Table). This 1-week lag had minimal association with human case forecasting accuracy; however, it was associated with reduced forecasting accuracy for the 3 mosquito outcomes. For both the baseline and temperature-forced models, forecast accuracy during lead weeks −2 to 3 decreased by approximately 5% for all 3 indicators.

This study highlights some of the current challenges impeding operational implementation of real-time arbovirus infectious disease forecasts in the United States. Specifically, we found that current delays in reporting of confirmed human WNV cases and infectious mosquitoes were associated with decreased real-time forecasting accuracy and utility as an evidence-based decision support tool.

Currently, human case reporting has significant lags related to communication from medical professionals to local health departments, laboratory confirmation, and the interview process used to determine where and when the infection was acquired. In addition to these lags, sharing these data (ie, location and date of disease onset) with local mosquito abatement districts may be limited owing to confidentiality constraints. In the future, accelerating and standardizing the appropriate transfer of timely human case information is needed.

Mosquito lags are shorter and associated with less reduction in accuracy than human case reporting lags but could also be shortened. The mosquito-testing lags were 5 to 8 days longer in state-run laboratories than onsite laboratories. The cost of this mosquito-testing lag was seen prior to the peak of mosquito infection rates, a critical period during which a better understanding of viral prevalence may provide more scientific guidance on decisions directing specific mosquito control practices. Evidence suggests that the timing of 2 key mosquito control interventions, larviciding and adulticiding, is critical for reducing the impact of WNV on humans.^16,27,28 The costs of switching from state-run to onsite mosquito infection assessment varies. For example, for St Tammany Parish, the cost would be $15 per test for onsite testing compared with $1.50 per test for tests run by the state. In contrast, Suffolk County is fully subsidized by the state and would likely need to budget $15 to $30 per sample for onsite testing. A full assessment of the costs of switching to onsite testing across different abatement districts is beyond the scope of the present study and is left for future work.

For surveillance data and forecasting to help support more targeted and cost-effective interventions, they must have minimal reporting lags, be analyzed quickly, and be presented in easy-to-understand visualizations depicting the current state of circulating virus with probabilities of the likely progression to spillover in a given area. Thus, it is important to develop and standardize a way to communicate risk among mosquito abatement districts, public health departments, and medical professionals to prevent the spread of arbovirus infections. All 3 stakeholders play a unique role in preventing an arbovirus outbreak, and the information each group collects is valuable to the others. Providing medical professionals with a better understanding of entomological risk will give them another data point to help guide clinical differential diagnosis. In return, medical professionals could provide public health departments and mosquito abatement districts with more rapid diagnostic information on spillover events and the current status of an outbreak. High-quality outbreak data reported in a timely fashion are a needed component for forecasting systems, which will help enable better-targeted and more cost-effective interventions.

This study has limitations. In 2017, real-time forecasting of WNV was conducted in 4 unique geographical locations across the United States. Although these sites capture a variety of governing structures between health department and vector control district, they represent only a small fraction of the vector control districts across the United States. Thus, the data used for confirmed human case reporting lags may underrepresent or overrepresent the time it takes to confirm each case. In addition to the time lag between onset and confirmation, human case reporting lags could only be derived from historical outbreaks in the city of Chicago. The city of Chicago reports human WNV cases electronically, which may reduce the reporting lag relative to other health departments. Chicago has also experienced multiple large outbreaks; as a consequence, clinicians may be more aware of reporting practices than in other parts of the country, and reporting lags may underrepresent the rest of the country (even though WNV is a national notifiable disease for which a positive case must be reported to local health department within 7 days of diagnosis). In addition, the study is biased toward mosquito abatement districts and health departments where WNV is endemic and resources are devoted toward surveillance.

Delays in reporting human WNV cases and infectious mosquito information hinder outbreak surveillance and corrupt real-time forecasting accuracy. During 2017 at 4 studied sites, infectious mosquito reporting lags were short enough that skillful forecasts could still be generated for mosquito indicators; however, human WNV case lags were too great to support accurate forecasting. WNV forecasting is potentially an important evidence-based decision support tool; however, for this tool to support more targeted and cost-effective interventions, more resources are needed to reduce data reporting lags.

Accepted for Publication: March 13, 2019.

Published: April 26, 2019. doi:10.1001/jamanetworkopen.2019.3175

Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 DeFelice NB et al. JAMA Network Open.

Corresponding Author: Nicholas B. DeFelice, PhD, Icahn School of Medicine at Mount Sinai, Environmental Medicine & Public Health, 17 E 102nd St, CAM Bldg, 2 West D2-141, One Gustave L. Levy Place, Box 1057, New York, NY 10029 (nicholas.defelice@mssm.edu).

Author Contributions: Dr N. B. DeFelice had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: N. B. DeFelice, Shaman.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: N. B. DeFelice, Henke, Wittie, Cole, Shaman.

Critical revision of the manuscript for important intellectual content: N. B. DeFelice, Birger, N. DeFelice, Gagner, Campbell, Romano, Santoriello, Kaiser, Shaman.

Statistical analysis: N. B. DeFelice.

Obtained funding: Shaman.

Administrative, technical, or material support: N. B. DeFelice, Birger, N. DeFelice, Gagner, Campbell, Romano, Santoriello, Henke, Wittie, Cole, Kaiser.

Supervision: N. B. DeFelice, Shaman.

Conflict of Interest Disclosures: Dr Shaman reported partial ownership along with Columbia University of SK Analytics. No other disclosures were reported.

Funding/Support: This work was supported by National Institutes of Health grants GM110748 and ES009089, National Institute of Environmental Health Sciences training grant T32ES023770, and Defense Threat Reduction Agency contract HDTRA1-15-C-0018.

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Additional Contributions: We thank the staff of the Suffolk County Arthropod-Borne Disease Laboratory, the Division of Vector Control, and the New York State Department of Health Arbovirus Laboratory for assistance in mosquito and arboviral surveillance efforts and viral analysis of the mosquito samples; the staff of the city of Chicago’s arboviral surveillance efforts; the staff of the Coachella Valley’s arboviral surveillance efforts and Christopher Barker, PhD, and Jody Simpson, BS, who helped maintain the mosquito data; Kevin Caillouet, PhD, and the staff of St Tammany Parish Mosquito Abatement District; Sasikiran Kandula, MS, and Jeff Turmelle, MS, for help maintaining the website; and Richard DeFelice, MD, for discussions on WNV case diagnostics and reporting protocols. None of these individuals were compensated for their contributions.