Data

Throughout, we restrict our attention to nowcasting state-level hospitalizations. This is because the medical claims signals that we describe below are not generally available at each US county. That said, in principle, the same ideas we describe in what follows could be applied to nowcast hospital reports in large counties (subject to enough claims data being available in order to form robust signals). We also restrict our attention to nowcasting daily reported hospitalizations between April 1, 2021 and August 1, 2023, though we use data back until November 1, 2020 for training models.

State-level, daily COVID-19 hospitalization reports are obtained from the HHS, accessed via the Delphi Epidata API [14, 38]. We use to denote the 7-day trailing average of finalized reported new COVID-19 hospitalization counts corresponding to location ℓ and time s. Averaging over 7-day trailing windows is mainly used as a smoother, and to account for weekday-weekend differences. Reported hospitalization counts are subject to revision (these are typically minor in comparison to revisions for claims signals), hence we also introduce notation to work with versioned data henceforth: we denote by the 7-day trailing average of hospitalization counts for location ℓ and time s, but whose data version is as of time t. In this context, we often refer to s as the reference date and t the issue date. We use analogous notation and nomenclature for all versioned data in this paper.

De-identified, aggregated medical insurance claims data are provided to us by Change Healthcare, which covers around 25% of all commercial medical insurance claims in the US. Moreover, based on comparing total counts of COVID-19 hospitalizations from inpatient claims and those reported to the HHS (over the full period of our analysis), we find that there is a broad range: at the maximum end, for some US states, over 55% of hospitalizations are reflected in the claims data; at the minimum end, for other states, less than 20% appear in the claims data. (A more detailed analysis is given in Sect A of S1 Appendix.) Nevertheless, we have enough claims data per state in order for us to be able to derive meaningful signals which reflect COVID-associated outpatient and inpatient activity. Specifically, in this work, we consider the following two signals:

• : the finalized percentage of outpatient claims in a 7-day trailing window with a confirmed COVID diagnostic code, corresponding to location ℓ and time s.
• : the finalized percentage of inpatient claims in a 7-day trailing window with a COVID-associated diagnostic code, corresponding to location ℓ and time s.

As before, we use and to denote the versioned outpatient and inpatient claims signals, respectively, corresponding to issue date t. The use of versioned data is extremely important when working with claims-based signals because medical insurance claims are often submitted and/or processed late, many days (and even months) after a given date of service. This process is generally referred to as backfill, and it has quite a pronounced effect on both outpatient and inpatient signals. We can look back at Fig 1 for an example of the inpatient signal in California. What is labeled as the “real-time inpatient signal” in the figure is in our notation introduced here, for s ranging over the time values along the x-axis (and ℓ =  California). What is labeled as the “finalized inpatient signal” is actually ; note that 30 is just a large number chosen for simplicity, and it is not necessarily the case that all claims would have been filed after 30 days.

Sometimes backfill is so severe that no claims for reference date s are filed at all until a later issue date t, which we refer to as latency. If the latency is large enough, in particular, if no claims are available until a full 7 days after a given reference date, then the real-time claims signal value will be missing. This happens at a few dates in Fig 1 in May and June of 2021.

In defining the inpatient and outpatient claims signals, the use of a ratio of COVID-associated claims to all claims, instead of a count of COVID-associated claims, is important for two reasons. First, it adjusts for the unknown market share (unknown to us) of Change Healthcare in each given location. Second, this ratio tends to be more robust to backfill than a pure count, as both the numerator and denominator get updated as new claims are filed (whereas a count would only be revised upward, and as we will see later in Fig 2, especially for the inpatient signal, only a small fraction of the total volume of claims are available in the first few days after a given date of service).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Analysis to help guide lag selection for inpatient and outpatient features in the working model (1).

The rows correspond to different features, and the columns to different metrics, as explained precisely in the main text. The shaded regions show  ± 1 standard error bands for each metric, over the state averages.

https://doi.org/10.1371/journal.pcbi.1012717.g002

Like the HHS hospitalization reports, various signals derived from medical insurance claims are available in the Delphi Epidata API. The precise diagnostic codes used in defining the outpatient and inpatient signals are given in the API documentation: https://cmu-delphi.github.io/delphi-epidata. For convenience, we have relayed these definitions in Sect A of S1 Appendix; we have also made all data (including properly-versioned data) used in our analysis available for download at: https://github.com/cmu-delphi/hhs-nowcasting.

Hypothetical scenarios

We consider two hypothetical scenarios, described next. Recall that our nowcasting evaluation spans April 1, 2021 to August 1, 2023, and we have training data all the way back until November 1, 2020.

Scenario 1: monthly updates.

Our first hypothetical scenario examines nowcasting from April 1, 2021 to November 30, 2021. In this scenario, on the first day of each month during this period, we receive daily hospitalization counts for the previous month, and we nowcast and backcast the (unobserved) hospitalization counts for each following day in the given month, using the outpatient and inpatient claims signals described above. Note that the period in this scenario covers the Delta wave in the US. The start of this hypothetical scenario is chosen to be April 1, 2021 so that we have a large enough “burn-in set” (initial training set), which extends back to November 1, 2020. (This includes the winter wave of 2020, which helps the initial models capture relationships during a time of dynamic change.)

To make this all more precise, let us introduce some notation. We drop reference to the location ℓ here and in what follows, whenever convenient (whenever it is not needed for the given explanation). Let be a date that marks the start of a month during the period of April 1, 2021 to November 30, 2021. Then on each day , where marks the first day of the next month, we have access to:

• , hospitalization counts through day , with versions as of day ; and
• , outpatient and inpatient signals through day t, with versions as of day t.

On each such day t, we use the data we have to train a regression model, call it , to predict hospitalizations from claims signals. We use this to make nowcasts:

and lag-k backcasts:

for each k = 1 , … , 10. (Clearly, a nowcast is equivalent to a lag-0 backcast.) This is repeated for each of the 9 months in the period spanned by this monthly-update scenario. The details of how regression models are trained and evaluated will be given in the next subsection.

Regression model

As a basic working model, we predict hospitalizations using a linear combination of a set of lags of the inpatient signal and a set of lags of the outpatient signal,

(1)

where the coefficients , , and are estimated, i.e., the model is fit, by training on historical data available at time t, either separately for each location ℓ (recall, the notational dependence on the location has been dropped for now), or in a way that pools data across locations. Details will be given below. First, we describe how we select the lag sets , for the inpatient and outpatient features.

Training the regression model.

At each nowcast date t, we fit the coefficients , , in (1) by solving the following (weighted) least squares optimization problem:

(2)

where we use exponentially decaying observation weights:

(3)

for a tuning parameter γ ≥ 0. The index in (2) marks the latest observation boundary for hospitalization reports before time t: this is either the start of the month in scenario 1 (where we receive monthly updates), or December 1, 2021 in scenario 2 (where we receive no further updates). Hence, in other words, we fit the coefficients in (2) by minimizing the weighted mean squared error of our working regression model, over all dates at which response values (reported hospitalization counts) are available. Note carefully that in (2) we only use properly-versioned data that would have been available at t. If any such data, response or feature values, are missing at time t then the corresponding summand is simply omitted from (2).

As a default, we fit the model by solving (2) separately for each location ℓ (recall, we have suppressed the dependence on ℓ in the notation for simplicity). We call this the state-level model. Later, we discuss schemes for pooling training data across locations. Next, we focus on the decay parameter γ in (2), (3).

Selecting the decay parameter by cross-validation.

Before describing how we select γ in the exponential weights (3) that are used in the weighted least squares problem (2), we pause to discuss the question: why is it useful to use decaying observation weights in the first place? The reason is because the the features—lagged versions of the inpatient and outpatient medical insurance claims signals, and response—reported hospitalizations, need not be jointly stationary, i.e., their relationship may be changing over time.

Looking back at Fig 1, note that we observe evidence of nonstationarity in the relationship between the real-time inpatient signal and reported hospitalizations. If we were to regress reported hospitalizations on the inpatient signal alone, then the regression coefficient that would be appropriate for the period April–July 2021 would be too small for the first hospitalization wave starting in December 2020 (and to a lesser extent, also too small for the second wave starting in August 2021).

By allowing γ itself change over time, we can adapt to the degree of nonstationarity at any point in time, i.e., we can adapt to the amount of past training data that is relevant for the current prediction. Indeed, we will select γ in a dynamic, time-varying fashion using cross-validation (CV). Given the sequential nature of our prediction problem, we use a version of CV that is purely forward-looking, and is sometimes referred to as time series cross-validation in the literature [39]. This works as follows. Let denote the index that marks the most recent observation boundary for hospitalization reports, and let denote the indices that mark the previous two observation boundaries before . For each γ in a grid Γ of tuning parameter values, we carry out the following procedure:

• for each :
  • fit a regression model by solving:
  • produce backcasts , ;
• for each :
  • fit a regression model by solving:
  •  -  produce backcasts , ;
• compute the mean absolute error (MAE) of all backcasts made at all times , against reported hospitalization counts , .

In other words, this procedure uses the last 2 months of data as a validation set for tuning γ. Ultimately, we choose γ ∈ Γ that minimizes the MAE computed in the last step above. To be clear, after choosing γ in this way, we then fit the model by solving (2) and use this to make backcasts at each time , where is the observation boundary after .

Toward defining the tuning parameter set Γ, we first compute , the value of γ such that the effective sample size of the weight sequence equals 30, i.e., it solves

This is chosen as a heuristic upper bound on the “reasonable” range of γ values, motivated by the idea that restricting to at least (effectively) 1 month of training data helps to avoid fitted models which are too volatile. We then set Γ to contain 25 evenly-spaced values between 0 and .

Stabilizing predictions by geo-pooling.

To borrow strength across locations, we consider fitting the regression model at each time t by pooling data across locations, which we refer to as the geo-pooled model. To be precise, instead of solving (2) per location ℓ, here we instead solve:

(4)

Note that in (4), the training set includes data from all locations ℓ. Importantly, in (4), the response is the hospitalization rate in location ℓ at reference date s, as of time t, which is defined as the hospitalization count per 100,000 people, i.e.,

where is the population of location ℓ. The use of rates rather than counts in the pooled regression (4) is critical, because otherwise the coefficients would have different meanings in different locations, and pooling across locations would not make sense. After solving (4), in order to use the fitted model to make predictions of hospitalization counts at a given location ℓ, we would then need to rescale the predictions by .

Finally, we also consider a mixed model, which linearly combines the state-level and geo-pooled predictions, via

(5)

for a mixing parameter λ ∈ [ 0 , 1 ] . We select λ, separately per state, using the same cross-validation strategy described previously, where we tune λ over a grid of 50 evenly-spaced values between 0 and 1. (We tune over γ separately for each of the state-level and geo-pooled models, and then tune over λ.)

Prediction intervals

In addition to considering point predictions, from the models described above, we use residuals from these models and quantile tracking [40] to generate prediction intervals. Quantile tracking is a method from the online conformal literature, and produces nonparametric intervals which are guaranteed to attain long-run coverage over an arbitrary bounded data sequence (including nonstationary time series). Abstractly, given a predicted value of some unobserved target value at time t, we begin by specifiying a score function φ (assumed to be negatively-oriented, so that lower values correspond to better accuracy) and we construct a prediction set at time t via

Here denotes a parameter that is output by the quantile tracking algorithm. Given a nominal coverage level 1 - α, the parameter is updated using an online gradient descent step with respect to an optimization problem defined by summing the level 1 - α quantile losses of over the sequence t = 1 , 2 , 3 , ….

In our problem, we do not receive new target values after we make each prediction, unlike the standard online learning setup used in [40], so we adapt their method so that is adjusted only at the observation boundaries in scenario 1. Further, to allow our intervals to exhibit varying-width between observation boundaries, we use a scaled score that divides the residual by the value of the prediction. Finally, in order to allow our intervals to be asymmetric around the predicted value, we define a separate lower and upper score, and , respectively:

and effectively run two instances of quantile tracking, with parameters and , respectively, each at the level 1 - α ∕ 2. Combining the results from these two procedures then gives us the lower and upper endpoints of the prediction intervals.

We are now ready to describe our adaptation of quantile tracking (under batched updates). In the same notation as that used to describe fitting the regression models above, let denote the latest observation boundary, and denote the next observation boundary. Suppose we are making backcasts at lag k (with k = 0 representing nowcasts). For each , we:

• produce a point prediction using one of the regression models described in previous subsections;
• produce a prediction interval via

Then, at , we:

• look back and compute one-sided coverage errors,
• update each of , by taking appropriate gradient descent steps,
  where η > 0 is a small fixed learning rate.

To prevent crossing of quantiles, we enforce and (by clipping them at zero if a gradient descent update were to make them negative).

Scenario 1: backcast error analysis

Fig 3 shows the MAE of all backcasts, as a function of lag k = 0 , … , 10, from the state-level, geo-pooled, and mixed models over the monthly-update period (which, recall spans April 1, 2021 to November 30, 2021). To be clear, for each model, this is computed by averaging the absolute error of backcasts made to finalized hospitalization counts, per state. The left panel displays the average of these state MAE values, and as well as standard error bands. The right panel is similar but normalizes the state MAE values by state population times , thus considering MAE on the scale of hospitalization rates (rather than counts).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. MAE as a function of backcast lag for the state-level, geo-pooled, and mixed models in scenario 1, the monthly-update period.

The shaded regions show  ± 1 standard error bands, over the state MAE values.

https://doi.org/10.1371/journal.pcbi.1012717.g003

The geo-pooled model is clearly worse in terms of MAE than the mixed and state-level models. On the counts scale (left panel), the state-level model has slightly better MAE than the mixed model; on the rates scale (right panel), the opposite is true. This is because the mixed model generally performs better for smaller states, where predictions tend to be more volatile and shrinking toward the geo-pooled model helps to reduce variance. This is not reflected in the MAE plot on the left panel, since poor backcast performance on small states contributes little when measured in terms of counts.

In absolute terms, the backcasts made by the state-level and mixed models are generally quite accurate. For example, the nowcasts (backcasts at lag 0) made by the state-level and mixed models have MAEs of 0.428 and 0.405, respectively, on the scale of hospitalization rates. These correspond to proportions of variance explained (PVEs) of 71.6% and 75.4%, respectively. Even the geo-pooled model, which is relatively quite a bit worse, produces nowcasts with an MAE of 0.524 on the scale of hospitalization rates, which corresponds to a still respectable PVE of 61.1%.

Scenario 1: illustrative nowcast examples

We examine nowcasts made by the state-level and mixed models during the monthly-update period, in four states: California (CA), Kentucky (KY), Vermont (VT), and New York (NY). The first three are chosen to demonstrate the qualitative behavior of nowcasts in states of different sizes, with CA having the largest population, KY having roughly median population, and VT having the smallest. NY is chosen because it represents somewhat of failure case for the robustness of nowcasts from the state-level model.

Fig 4 displays state-level and mixed model nowcasts for CA, KY, and VT. To be clear, in each panel of the figure, the nowcasts use real-time inpatient and claims signals as predictive features, and the models behind these nowcasts are trained using hospitalization data up to the latest observation boundary (marked by dotted vertical lines). The shaded bars in the figure display finalized reported hospitalizations, the target of ultimate interest. In CA and KY, where the Delta wave is prominent (roughly July to November), we can see that both the state-level and mixed model nowcasts track the dynamics of the Delta wave. For example, looking at July 1, 2021 in CA, the hospitalization reports from the previous months (which is all that would have been available at that time) show no indication of an upswing to come. Still, the nowcasts during July present a clear upward trend, which means that policy-makers would know from these nowcasts that a wave is underway. As we see it, evaluating nowcasts for their qualitative shape (in comparison to hospitalization waves) is important—just as much as (if not more so than) numerical error analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Nowcasts from the state-level and mixed models in scenario 1, the monthly-update period, for CA, KY, and VT.

The dotted vertical lines mark the observation boundaries—when hospitalization reports for the previous month are received, and models are retrained.

https://doi.org/10.1371/journal.pcbi.1012717.g004

Being much smaller, VT has on the order of ten reported COVID-19 hospitalizations daily, not several hundreds, as in KY or CA. Accordingly, we can see in Fig 4 that the (finalized) reported hospitalizations curve in VT is much noisier—we do not really see a clear Delta wave (instead, an increasing but quite noisy trend from August through December). The claims signals are also more noisy in a small state like VT, since they are ratios of small counts. Therefore, both the response and features are more volatile in the prediction problem for VT, and not surprisingly, the nowcasts here appear qualitatively worse. That said, the nowcasts still roughly track the gross trends in reported hospitalizations: higher in April and May, lower in June and July, and growing again in August onward.

Generally, the discussion above translates to the rest of the US: nowcasts tend to be in good qualitative agreement with hospitalization waves in larger states, and less so in smaller states. Sect C of Appendix S1 provides a full set of nowcast and backcast plots in all 50 states, from the mixed model, under scenario 1. Occasionally, we find that the trend in the predictions disagrees with that in hospitalization reports in a given month, but this gets corrected at the next observation boundary, once new training data is available and the regression model is refit. We also find that backcasts at lag 5 or 10 tend to be smoother than nowcasts.

Fig 5 shows state-level and mixed model nowcasts for NY. This is a notable example of a failure in robustness of the state-level model: its nowcasts for a good part of the month of June are actually negative (and we truncate them at zero for visualization purposes). This is due to a large, systematic revision which occurred on June 8 in the outpatient signal, where its values for reference dates in May were revised upward. Moreover, when the state-level model is fit to data through the end of May, the regression coefficient on the largest outpatient feature lag ends up being negative, and consequently the upward revision of past feature values on June 8 introduces a strong downward bias in the subsequent predictions. Supporting analysis for this explanation is given in Sect B of S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Nowcasts from the state-level and mixed models in scenario 1, the monthly-update period, for NY.

The dotted vertical lines mark the observation boundaries, as in Fig 4.

https://doi.org/10.1371/journal.pcbi.1012717.g005

As we can see in the figure, the mixed model produces nowcasts that are much more stable in the month of June. However, the mixed model and even the geo-pooled model are themselves not immune to large and erratic revisions in the input features at prediction time. These revisions can result in erratic nowcasts and backcasts. To mitigate this, we could turn to models that fuse information across a wider variety of auxiliary signals (where ideally, some of these signals would have less severe backfill than claims-based signals), an idea that we return to in the discussion section.

Scenario 2: backcast error analysis

Fig 6 shows the MAE of all backcasts, as a function of lag k = 0 , … , 10, from the state-level, geo-pooled, and mixed models over the no-update period (spanning December 1, 2021 to August 31, 2023). The format is as in Fig 3. We see two notable differences compared to the results in the monthly-update period. First, as expected, each model performs worse than it did in Fig 6. On the scale of hospitalization rates (right panel), the range of MAEs has jumped from (roughly) 0.4–0.53 in the monthly-update period to 0.62–0.72 in the no-update period, and correspondingly, the PVEs have dropped from (roughly) 61–75% to 10–37%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. MAE as a function of backcast lag for the state-level, geo-pooled, and mixed models in scenario 2, the no-update period.

The shaded regions show  ± 1 standard error bands, over the state MAE values.

https://doi.org/10.1371/journal.pcbi.1012717.g006

Second (and perhaps a bit more surprising), we see in the no-update period that the state-level model performs clearly worse than the geo-pooled and mixed models when MAE is measured either on the counts or rates scale. The mixed model MAE is somewhat better than the geo-pooled model MAE with respect to counts, while the two are basically the same with respect to rates. It is encouraging to see that the mixed model performs competitively across both monthly-update and no-update scenarios: this model—equipped with the CV procedure for tuning λ—is able to effectively adapt the strengths of local and global training approaches (underlying the state-level and geo-pooled models) to the task at hand, in order to yield accurate predictions in an average-case sense.

Scenario 2: illustrative nowcast examples

We examine nowcasts made by the geo-pooled and mixed models for CA, KY, and VT. NY, which served as failure case in the monthly-update period, is not shown here, because KY itself provides such an example: as we will see, the mixed model lacks robustness over a part of the Omicron wave (meanwhile, the mixed model performs fine for NY throughout the no-update period—as shown in the Sect B of S1 Appendix).

Fig 7 displays the nowcasts for CA, KY, and VT, with the same general format as in Fig 4. CA is a clear success case: both geo-pooled and mixed models capture the dynamics of the Omicron wave faithfully, and also track the summer 2023 and winter 2023 waves, despite (recall) receiving no reported hospitalizations past December 1, 2022. VT, as in the monthly-update period, is a challenging case because it corresponds to much noisier prediction problem, and the nowcasts here look qualitatively worse overall. However, the mixed model nowcasts still pick up the Omicron wave. KY represents a failure case: the mixed model nowcasts for all of February are negative and truncated at zero for the visualization. What this is actually demonstrating is a failure of the state-level model, and simultaneously, a failure of tuning of the mixing parameter λ. The mixed model here ends up placing a large weight on state-level predictions (not shown), which are themselves volatile for reasons similar to what happens in NY during the monthly-update period—large revisions to the input features at prediction time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Nowcasts from the geo-pooled and mixed models in scenario 2, the no-update period, for CA, KY, and VT.

The dotted vertical line (on the left side of the plot) marks the sole observation boundary in this period—hospitalization reports for all dates prior to this are available, but no reports are received after that.

https://doi.org/10.1371/journal.pcbi.1012717.g007

The volatility of the mixed model nowcasts in KY during Omicron provides an important perspective: in the MAE sense, we found in Fig 6 that the mixed model (with its CV tuning for λ) successfully navigates the strengths and weaknesses of the geo-pooled and state-level models; and yet for specific states and periods of time, the mixed model can still lack robustness. To be clear, such fragility is not limited to the no-update period, and it could have happened in NY in the monthly-update period, had the CV tuning procedure for λ not downweighted the state-level predictions so heavily. Improving robustness is a direction for future work, and we revisit this topic in the discussion section.

Lastly, in Sect D of S1 Appendix, we again provide a full set of nowcast and backcast plots in all 50 states, from the mixed model, under scenario 2. For larger states, we find that the nowcasts generally trace out the Omicron wave, but for smaller states, this happens less consistently and the nowcasts look more noisy. Backcasts at lag 5 or 10 tend to smooth out the nowcasts, though not dramatically.

Ablation study

To examine the importance of some of our modeling choices (as described in the methods section), we carry out an ablation study in which we remove a particular component of the model, use a simpler alternative in its place, and evaluate the result in terms of MAE. In particular, we consider four ablated models:

1. Unweighted, all past: we fit the regression model (2) without observation weights, on all past reported hospitalization data available (considering summands s < t in (2)).
2. Unweighted, two months: we fit the regression model (2) without observation weights, on the reported hospitalization data available for the latest two months (considering summands in (2)).
3. Weighted, inpatient only: we fit the regression model (2) using only the lags of the inpatient signal.
4. Weighted, outpatient only: we fit the regression model (2) using only the lags of the outpatient signal.

Tables 1 and 2 compare the performance of the state-level model to these ablated models, for scenarios 1 and 2, respectively. In each scenario, for each model, we compute its MAE over the range of time values in the scenario, per backcast lag k = 0 , … , 10 and state, then we report the average and standard error of these MAE values across all backcast lags and states. The state-level and inpatient-only models perform the best overall, and make up the two best models in either scenario. Meanwhile, the all-past model is slightly worse and the two-month model is significantly worse, which emphasizes the importance of decaying weights (and CV tuning) as part of the original model design. The outpatient-only model is competitive in scenario 1 but not in scenario 2. This could be a reflection of distribution-drift in the relationship between the outpatient signal and hospitalization reports post Omicron, which in scenario 2 is compounded by our inability to retrain the regression model in order to account for such drift.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. MAE results from the state-level and ablated models, averaged over all backcast lags and all states, in scenario 1.

https://doi.org/10.1371/journal.pone.0313772.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. MAE results from the state-level and ablated models, averaged over all backcast lags and all states, in scenario 2.

https://doi.org/10.1371/journal.pone.0313772.t002

Prediction interval analysis

We now examine the performance of the quantile tracker, applied to the state-level model in scenario 1. To gain an understanding of how quantile tracking performs against off-the-shelf alternatives, we consider:

• parametric intervals: we apply the standard method for obtaining prediction intervals associated with a well-specified linear regression model with independent Gaussian errors;
• sample quantiles: we obtain intervals by simply computing and as sample quantiles of all past scores (in lieu of quantile tracking), where the empirical distribution of past scores is weighted with the same decaying weights that are used to train the regression model.

We and evaluate all methods in terms of both coverage of their prediction intervals, and interval score (a metric that combines coverage and sharpness), which for a predicted interval  [ l , u ]  at the nominal level 1 - α, and target value y, is defined as

Fig 8 reports interval score and coverage averaged over all dates in scenario 1, at the nominal coverage levels 0.6 and 0.8. (The setup is the same as that in Fig 3, except with either interval score or coverage replacing absolute error.) Quantile tracking clearly dominates the other methods, and it is the only method achieving anywhere close to the nominal coverage level.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Interval score and coverage as a function of backcast lag for quantile tracking, parametric (linear regression) modeling, and sample quantiles in scenario 1, when the nominal coverage level is 0.6 (first row) and 0.8 (second row).

The shaded regions show  ± 1 standard error bands, over the state values.

https://doi.org/10.1371/journal.pcbi.1012717.g008

Fig 9 visualizes the intervals formed by quantile tracking and the two alternative methods in CA, KY, VT. The nominal coverage level is 0.8. We can see that quantile tracking generally accounts for uncertainty much better than the other methods, avoiding exceedingly narrow intervals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Nowcasts and prediction intervals from quantile tracking, parametric (linear regression) modeling, and sample quantiles in scenario 1, for CA, KY, and VT.

https://doi.org/10.1371/journal.pcbi.1012717.g009