# BACK2FUTURE: LEVERAGING BACKFILL DYNAMICS
## FOR IMPROVING REAL-TIME PREDICTIONS IN FUTURE

**Harshavardhan Kamarthi, Alexander Rodríguez, B. Aditya Prakash**
College of Computing
Georgia Institute of Technology
{harsha.pk,arodriguezc,badityap}@gatech.edu

ABSTRACT

For real-time forecasting in domains like public health and macroeconomics, data
collection is a non-trivial and demanding task. Often after being initially released,
it undergoes several revisions later (maybe due to human or technical constraints)

-  as a result, it may take weeks until the data reaches a stable value. This socalled ‘backfill’ phenomenon and its effect on model performance have been barely
addressed in the prior literature. In this paper, we introduce the multi-variate
backfill problem using COVID-19 as the motivating example. We construct a
detailed dataset composed of relevant signals over the past year of the pandemic.
We then systematically characterize several patterns in backfill dynamics and
leverage our observations for formulating a novel problem and neural framework,
Back2Future, that aims to refines a given model’s predictions in real-time. Our
extensive experiments demonstrate that our method refines the performance of
diverse set of top models for COVID-19 forecasting and GDP growth forecasting.
Specifically, we show that Back2Future refined top COVID-19 models by 6.65%
to 11.24% and yield 18% improvement over non-trivial baselines. In addition,
we show that our model improves model evaluation too; hence policy-makers can
better understand the true accuracy of forecasting models in real-time.

1 INTRODUCTION

The current COVID-19 pandemic has challenged our response capabilities to large disruptive events,
affecting the health and economy of millions of people. A major tool in our response has been
forecasting epidemic trajectories which enabled policymakers to plan interventions (Holmdahl &
Buckee, 2020). Broadly two classes of approaches have been devised: traditional mechanistic
epidemiological models (Shaman & Karspeck, 2012; Zhang et al., 2017), and the fairly newer
statistical approaches (Brooks et al., 2018; Adhikari et al., 2019; Osthus et al., 2019b) including deep
learning models (Adhikari et al., 2019; Panagopoulos et al., 2021; Rodríguez et al., 2021a), which
have become among the top-performing ones for multiple forecasting tasks (Reich et al., 2019). These
models also use newer digital indicators like search queries (Ginsberg et al., 2009; Yang et al., 2015)
and social media (Culotta, 2010). Epidemic forecasting is still a challenging enterprise (Metcalf &
Lessler, 2017; Biggerstaff et al., 2018) because it is affected by weather, mobility, strains, and others.

However, real-time forecasting also brings new challenges. As noted in multiple CDC real-time
forecasting initiatives for diseases like flu (Osthus et al., 2019a) and COVID-19 (Cramer et al., 2021),
as well as in macroeconomics (Clements & Galvão, 2019; Aguiar, 2015) the initially released public
health data is revised many times after and is known as the ’backfill’ phenomenon.The various factors
that affect backfill are multiple and complex, ranging from surveillance resources to human factors
like coordination between health institutes and government organizations within and across regions
(Chakraborty et al., 2018; Reich et al., 2019; Altieri et al., 2021; Stierholz, 2017).

While previous works have addressed anomalies (Liu et al., 2017), missing data (Yin et al., 2020),
and data delays (Žliobaite, 2010) in general time-series problems, the backfill problem has not been
addressed. In contrast, the topic of revisions has not received as much attention, with few exceptions.
For example in epidemic forecasting, a few papers have either (a) mentioned about the ‘backfill
problem’ and its effects on performance (Chakraborty et al., 2018; Rodríguez et al., 2021b; Altieri


-----

et al., 2021; Rangarajan et al., 2019) and evaluation (Reich et al., 2019); or (b) proposed to address the
problem via simple models like linear regression (Chakraborty et al., 2014) or ’backcasting’ (Brooks
et al., 2018) the observed targets. The related problem of nowcasting involves prediction of revised
stable value of current week’s target from sequence of right-truncated past values [Aditya: cite one
paper ]. Prior works have used data assimilation and sensor fusion from a readily available stable
set of features to refine unrevised features for accurate nowcasting (Farrow, 2016; Osthus et al.,
2019a). However, most methods focus only on revisions in the target and typically study in the
context of influenza forecasting, which is substantially less noisy and more regular than the novel
COVID-19 pandemic or assume access to stable values for some features which is not the case for
COVID-19. In economics, Clements & Galvão (2019) surveys several domain-specific (Carriero et al.,
2015) or essentially linear techniques for data revision/correction behavior of several macroeconomic
indicators (Croushore, 2011).

Motivated from above, we study the challenging problem of multi-variate backfill for both features
and targets. We go further beyond prior work and also show how to leverage our insights towards a
general neural framework to improve model predictions and performance evaluation (i.e. rectification
of current target from the evaluator’s perspective). Our specific contributions are the following:

_• Multi-variate backfill problem: We introduce the multi-variate backfill problem using real-time_
epidemiological forecasting as the primary motivating example. In this challenging setting, which
generalizes (the limited) prior work, the forecast targets, as well as exogenous features, are subject to
retrospective revision. Using a carefully collected diverse dataset for COVID-19 forecasting for the
past year, we discover several patterns in backfill dynamics, show that there is a significant difference
in real-time and revised feature measurements, and highlight the negative effects of using unrevised
features for incidence forecasting in different models both for model performance and evaluation.
Building on our empirical observations, we formulate the problem BFRP, which aims to ‘correct’
given model predictions to achieve better performance on eventual fully revised data.

_• Spatial and Feature level backfill modeling to refine model predictions: Motivated by the_
patterns in revision and observations from our empirical study, we propose a deep-learning model
Back2Future (B2F) to model backfill revision patterns and derive latent encodings for features.
B2F combines Graph Convolutional Networks that capture sparse, cross-feature, and cross-regional
backfill dynamics similarity and deep sequential models that capture temporal dynamics of each
features’ backfill dynamics across time. The latent representation of all features is used along with the
history of the model’s predictions to improve diverse classes of models trained on real-time targets,
to predict targets closer to revised ground truth values. Our technique can be used as a ‘wrapper’ to
improve model performance of any forecasting model (mechanistic/statistical).

_• Refined top models’ predictions and improved model evaluation: We perform an extensive_
empirical evaluation to show that incorporating backfill dynamics through B2F consistently improves
the performance of diverse classes of top-performing COVID-19 forecasting models (from the
CDC COVID-19 Forecast Hub, including the top-performing official ensemble) significantly. B2F
also enables forecast evaluators and policy-makers better evaluate the ‘eventual’ true accuracy of
participating models (against revised ground truth). This allows the model evaluators to quickly
estimate models that perform better w.r.t revised stable targets instead of potentially misleading
current targets. Our methodology can also be further adapted for nowcasting and other general
time-series forecasting problems. We also show the generalizability of our framework and model
B2F to other domains by significantly improving predictions of non-trivial baselines for US National
GDP forecasting (Marcellino, 2008).
2 NATURE OF BACKFILL DYNAMICS
In this section, we study important properties of the revision dynamics of our signals. We introduce
some concepts and definitions to aid in the understanding of our empirical observations and method.

**Real-time forecasting. We are given a set of signals F = Reg × Feat, where Reg is the set of all**
regions (where we want to forecast) and set Feat contains our features and forecasting target(s) for
each region. At prediction week t, x[(]i,[t]1:[)] _t_ [is a time series from 1 to][ t][ for feature][ i][, and the set of all]

signals results in the multi-variate time series X1:[(][t]t[)]1. Similarly, Y1:[(][t]t[)] [is the forecasting target(s) time]

1In practice, delays are possible too, i.e, at week t, we have data for some feature i only until t − _δi. All our_
results incorporate these situations. We defer the minor needed notational extensions to Appendix for clarity.


-----

series. Further, let’s call all data available at time t, D1:[(][t]t[)] [=][ {X][ (]1:[t]t[)][,][ Y]1:[(][t]t[)][}][ as][ real-time sequence][. For]

clarity we refer to ‘signal’ i ∈F as a sequence of either a feature or a target, and denote it as d[(]i,[t]1:[)] _t[.]_

Thus, at prediction week t, the real-time forecasting problem is: Given D1:[(][t]t[)][, predict next][ k][ values of]
_forecasting target(s), i.e. ˆyt+1:t+k. Typically for CDC settings and this paper, our time unit is week,_
_k = 4 (up to 4 weeks ahead) and our target is COVID-19 mortality incidence (Deaths)._

**Revisions. Data revisions (‘backfill’) are common. At prediction week t + 1, the real-time sequence**
_D1:[(][t]t[+1)]+1_ [is available. In addition to the length of the sequences increasing by one (new data point),]
values ofstudied backfill limited to D1:[(][t]t[+1)]+1 [already in] Y[ D][(][t][)]1:, while we address it in both[(][t]t[)] [may be revised i.e.,][ D]1:[(][t]t[)] X[̸][=][(][ D][t][)] 1:[(]and[t]t[+1)] Y. Note that previous work has[(][t][)]. Also, note that the data in
the backfill is the same used for real-time forecasting, but just seen from a different perspective.

**Backfill sequences: Another useful way we propose to look at backfill is by focusing on revisions of**
a single value. Let’s focus on value of signal i at an observation week t[′]. For this observation week,
the value of the signal can be revised at any t > t[′], which induces a sequence of revisions. We refer
to revision week r ≥ 0 as the relative amount of time that has passed since the observation week t[′].

**Defn. 1. (Backfill Sequence BSEQ) For signal i and observation week t[′], its backfill sequence is**
BSEQ(i, t[′]) = ⟨d[(]i,t[t][′][′][)][, d][(]i,t[t][′][′][+1)], . . ., d[(]i,t[∞][′][ ⟩][)] _[, where][ d][(]i,t[t][′][′][)][ is the][ initial value][ of the signal and][ d][(]i,t[∞][′][ is the][)]_
final/stable value of the signal.

**Defn. 2. (Backfill Error BERR) For revision week r of a backfill sequence, the backfill error is**
BERR(r, i, t[′]) = |d[(]i,t[t][′][′][+][r][)] _−_ _d[(]i,t[∞][′][ |][)]_ _[ /][ |][d][(]i,t[∞][′][ |][)]_ _[.]_

**Defn. 3. (Stability time STIME) of a backfill sequence BSEQ is the revision week r[∗]** _that is the_
_minimum r for which the backfill error BERR < ϵ for all r > r[∗], i.e., the time when BSEQ stabilizes._

_Note: We ensured that BSEQ length is at least 7, and found that in our dataset most signals stabilize_
before r = 20. For d[(]i,t[∞][′][, we use][)] _[ d][(]i,t[t][f][′][ )][, at the final week][ t][f][ in our revisions dataset. In case we do not]_
find BERR < ϵ in any BSEQ, we set STIME to the length of that BSEQ. We use ϵ = 0.05. Example:
For BSEQ {223, 236, 236, 404, . . ., 404}, BERR for third week is _[|][236]404[−][404][|]_ = 0.41 and STIME is 4.

2.1 DATASET DESCRIPTION
Table 1: List of features in

We collected and pre-processed important publicly available signals our CoVDS
from a variety of trusted sources that are relevant to COVID-19
forecasting to form the COVID-19 Surveillance Dataset (CoVDS ). **Type** **Features**
See Table 1 for the list of 20 features ( Feat = 21, including
_|_ _|_
Deaths). We collected revised features every week from April 2020 +veInc, HospInc
to July 2021. Our analysis covers 30 observation weeks from June Recovered,
2020 to December 2020 (to ensure all our backfill sequences are of onVentilator,
length at least 7) for all Reg = 50 US states. The rest of the unseen
_|_ _|_
data from Jan 2021 to July 2021 is used strictly for evaluation. -veInc,

**Patient line-list: traditional surveillance signals used in epidemiologi-** Mobility RetailRec,
cal models (Chakraborty et al., 2014; Brooks et al., 2018) derived from Grocery,
line-list records e.g. hospitalizations from CDC (CDC, 2020), posi- Parks, Transit
tive cases, ICU admissions from COVID Tracking (COVID-Tracking, WorkSpace,
2020). Testing: measure changes in testing from CDC and COVID- AppleMob
Tracking used by Rodríguez et al. (2021b). Mobility: quantify change Exposure DexA
in people’s movement to several point of interests (POIs); derived from Social Sur- FbCLI, FbWiLi
mobility reports released by Google (2020); Apple (2020). Exposure: vey

|Type|Features|
|---|---|
|Patient Line-List|ERVisits, HospRate, +veInc, HospInc, Recovered, onVentilator, inICU|
|Testing|TestResultsInc, -veInc, Facilities|
|Mobility|RetailRec, Grocery, Parks, Transit, WorkSpace, Resident, AppleMob|
|Exposure|DexA|
|Social Sur- vey|FbCLI, FbWiLi|

digital signal measuring closeness between people at POIs,(Chevalier
et al., 2021) Social Survey: used by (Wang et al., 2020; Rodríguez et al., 2021b) CMU/Facebook
Symptom Survey Data contains self-reported responses about COVID-19 symptoms.

2.2 OBSERVATIONS
We first study different facets of the significance of backfill in CoVDS . Using our definitions, we
generate a backfill sequence for every combination of signal, observation week, and region (not all
signals are available for all regions). In total, we generate more than 30, 000 backfill sequences.


-----

**Backfill error BERR is significant. We computed BERR for the initial values, i.e., BERR(r =**
0, i, t[′]), for all signals i and observation weeks t[′].
**Obs. 1. (BERR across signals and regions) Compute the average BERR for each signal; the median**
_of all these averages is 32%, i.e. at least half of all signals are corrected by 32% of their initial value._
_Similarly in at least half of the regions the signal corrections are 280% of their initial value._
We also found large variation of BERR. For features (Figure 1a), compare avg. BERR = 1743% of
five most corrected features with 1.6% of the five least corrected features. Also, in contrast to related
work that focuses on traditional surveillance data Yang et al. (2015), perhaps unexpectedly, we found
that digital indicators also have a significant BERR (average of 108%). For regions (see Figure 1b),
compare 1594% of the five most corrected regions with 38% of the five least corrected regions.


10[1]

10[0]

Backfill Error10 1

Line List TestingFeature typeMobility ExposureSocial Survey


4.0

3.5

3.0

2.5

2.0

1.5

Stability Time1.00.5

0.0 Line List TestingFeature typeMobility ExposureSocial Survey


1

0.5

0

-0.5

-1


0.4

0.2

0

-0.2

-0.4


(a) BERR per feat. type (b) BERR per region (c) STIME per feat. type (d) STIME per region

Figure 1: BERR and STIME across feature type and regions, heat maps are log scaled.


**Stability time STIME is significant. A similar analysis for STIME found significant variation across**
signals (from 1 weeks for to 21 weeks for COVID-19, see Figure 1c for STIME across feature
types) and regions (from 1.55 weeks for GA to 3.83 weeks for TX, see Figure 1d). This also impacts
our target, thus, actual accuracy is not readily available which undermines real-time evaluation and
decision making.
**Obs. 2. (STIME of features and target) Compute the average STIME for each signal; the average of**
_all these averages for features is around 4 weeks and for our target Deaths is around 3 weeks, i.e._
_on average, it takes over 3 weeks to reach the stable values of features._
**Backfill sequence BSEQ patterns. There is significant similarity among BSEQs. We cluster BSEQs**
via K-means using Dynamic Time Warping (DTW) as pair-wise distance (as DTW can handle
sequences of varying magnitude and length). We found five canonical categories of behaviors (see
Figure 2), each of size roughly 11.58% of all BSEQs. Also, each cluster is not defined only by signal







Early Decline Early Increase Steady/Spike Late rise Mid Decrease
1.0

0.8

0.6

0.4

0.2

**BSeq value (0-1 scaled)0.0**

0 10 20 30 0 10 20 30 0 **Revision week10** 20 30 0 10 20 30 0 10 20 30

Figure 2: Centroid BSEQ of each cluster, scaled between [0, 1], showing canonical backfill behaviors

nor region. Hence there is a non-trivial similarity across both signals and regions.
**Obs. 3. (BSEQ similarity and variety) Five canonical behaviors were observed in our backfill**
_sequences (Figure 2). No cluster has over 21% of BSEQs from the same region, and no cluster has_
_over 14% of BSEQs from the same signal._ 100

**Model performance vs BERR. To study the rela-** 0 50
tionship between model performance (via Mean Abso- 100 0
lute Error MAE of a prediction) and BERR, we use
REVDIFFMAE: the difference between MAE com- 300400 100150
puted againststable target value. We analyze the top-performing real- real-time target value and one against the Real-time MAE - Stable MAE 0.0 0.2 Backfill Error0.4 0.6 0.8 1.0 Real-time MAE - Stable MAE 0.0 0.2 Backfill Error0.4 0.6 0.8 1.0

100

0

100

200

300

400

Real-time MAE - Stable MAE

0.0 0.2 Backfill Error0.4 0.6 0.8 1.0

100

50

0

50

100

150

Real-time MAE - Stable MAE 0.0 0.2 0.4 0.6 0.8 1.0

Backfill Error

(a) GT-DC (b) YYG

time forecasting models as per the comprehensive evalu
Figure 3: BERR vs model REVDIFFMAE.

ation of all models in COVID-19 Forecast Hub (Cramer
et al., 2021). YYG and UMASS-MB are mechanistic while CMU-TS and GT-DC are statistical
models. The top performing ENSEMBLE is composed of all contributing models to the hub. We expect
a well-trained real-time model will have higher REVDIFFMAE with larger BERR in its target (Reich
et al., 2019). However, we found that higher BERR does not necessarily mean worse performance.
See Figure 3—YYG has even better performance with more revisions. This may be due to the more
complex backfill activity/dependencies in COVID in comparison to the more regular seasonal flu.
**Obs. 4. (Model performance and backfill) Relation between BERR and REVDIFFMAE can be**
_non-monotonous and positively or negatively correlated depending on model and signal._


-----

**Real-time target values to measure model performance: Since targets undergo revisions (5%**
BERR on average), we study how this BERR affects the real-time evaluation of models. From
Figure 4, we see that the scores are not similar with real-time scores over-estimating model accuracy.
The average difference in scores is positive which implies that evaluators would overestimate models’
forecasting ability.

**Obs. 5. MAE evaluated at real-time overestimates model performance by 9.6 on average, with the**
_maximum for TX at 22.63._

3 REFINING THE FUTURE VIA B2F

Our observations naturally motivate improving training and evaluation 17501500 GT-DCYYGEnsemble
aspects of real-time forecasting by leveraging revision information. Thus, 1000 CMU
we propose the following two problems. Let predictions of model M for
week t + k be y(M, k)t. Since models are trained on real-time targets, 250
_y(M, k)t is the model’s estimate of target yt[(]+[t][+]k[k][)][.]_ Real-time MAE

_k-week ahead Backfill Refinement Problem, BFRPk(M_ ): At predic- Figure 4: Real-time vs sta
17501500 GT-DCYYGEnsemble

1250 UMass-MB

1000 CMU

750

Stable MAE 500

250

0

0 250 500Real-time750 1000 MAE1250 1500 1750

_ble MAE._

tion week t, we are given a revision dataset {D1:[(][t]t[′][)][′] _[}][t][′][≤][t][, which includes]_
our target Y1:[(][t]t[)][. For a model][ M][ trained on][ real-time][ targets, given history of model’s predictions till]
last week ⟨y(M, k)1, . . . y(M, k)t−1⟩ and prediction for current week y(M, k)t, our goal is to refine
_y(M, k)t to better estimate the stable target yt[(]+[t][f]k[ )][, i.e. the ’future’ of our target value at][ t][ +][ k][.]_

**Leaderboad Refinement problem, LBRP: At each week t, evaluators are given a current estimate**
of our target yt[(][t][)] and forecasts of models submitted on week t _k. Our goal is to refine yt[(][t][)]_ to
_−_
_yˆt, a better estimate of yt[(][t][f][ )], so that using ˆyt as a surrogate for yt[(][t][f][ )]_ to evaluate predictions of
models provides a better indicator of their actual performance (i.e., we obtain a refined leaderboard
of models). Since LBRP aims to refine the current estimate of the target, it is closely related to the
nowcasting problem. LBRP is also a special case of BFRP: Assume a hypothetical model Meval
whose predictions are real-time ground truth, i.e. y(Meval, 0)t = yt[(][t][)][,][ ∀][t][. Then, refining][ M][eval] [is]
equivalent to refining yt[(][t][)] to better estimate yt[(][t][f][ )] which leads to solving LBRP.

**Overview: We leverage observations from Section 2 to derive Back2Future (B2F), a deep-learning**
model that uses revision information from BSEQ to refine predictions. Obs. 1 and 2 show that realtime values of signals are poor estimates of stable values. Therefore, we leverage patterns in BSEQ of
past signals and exploit cross-signal similarities (Obs. 3) to extract information from BSEQs. We also
consider that the relation of models’ forecasts to BERR of targets is complex (Obs. 4 and 5) to refine
their predictions. B2F combines these ideas through its four modules: • GRAPHGEN: Generates
a signal graph (where each node maps to a signal in Reg × Feat) whose edges are based on BSEQ
similarities. • BSEQENC: Leverages the signal graph as well as temporal dynamics of BSEQs to learn
a latent representation of BSEQs using a Recurrent Graph Neural Network. • MODELPREDENC:
Encodes the history of the model’s predictions, the real-time value of the target, and past revisions of
the target through a recurrent neural network. • REFINER: Combines encodings from BSEQENC and
MODELPREDENC to predict the correction to model’s real-time prediction.

In contrast to previous works that studies target BERR (Reich et al., 2019), we simultaneously
model all BSEQ available till current week t using spatial and signal similarities in the temporal
dynamics of BSEQ. Recent works that attempt to model spatial relations for COVID19 forecasting
need explicitly structural data (like cross-region mobility) (Panagopoulos et al., 2020) to generate
a graph or use attention over temporal patterns of regions’ death trends. B2F, in contrast, directly
models the structural information of signal graph (containing features from each region) using
BSEQ similarities. Thus, we first generate useful latent representations for each signal based on
BSEQ revision information of that feature as well as features that have shown similar revision
patterns in the past. Due to the large number of signals that cover all regions, we cannot model
the relations between every pair using fully connected modules or attention similar to (Jin et al.,
2020). Therefore, we first construct a sparse graph between signals based on past BSEQ similarities. Then we inject this similarity information using Graph Convolutional Networks (GCNs)
and combine it with deep sequential models to model temporal dynamics of BSEQ of each signal
while combining information from BSEQ s of signals in the neighborhood of the graph. Further,


-----

we use these latent representations and leverage the history of a model M ’s predictions to refine its prediction. Thus, B2F solves BFRPk(M ) assuming M is a black box, accessing only its
past forecasts. Our training process, that involves pre-training on model-agnostic auxiliary task,
greatly improves training time for refining any given model M . The full pipeline of B2F is also
shown in Figure 5. Next, we describe each of the components of B2F in detail. For the rest of
this section, we will assume that we are forecasting k weeks ahead given data till current week t.

RNN **GRAPHGEN generates an**

RNN GConv undirectedGt = (V, Esignalt) whosegraph

RNN edges represent similar
SignalGraph ity in BSEQs between

signals, where vertices
_V = F = Reg × Feat. We_
measure similarity using
DTW distance due to reasons described in Section 2.
GRAPHGEN leverages the
similarities across BSEQ

Figure 5: B2F pipeline with all components patterns irrespective of the

RNN

RNN GConv

RNN

Signal
Graph

exact nature of canonical behaviors which may vary across domains. We compute the sum of DTW
distances of BSEQs for each pair of nodes summed over t[′] _∈{1, 2, . . ., t −_ 5}. We threshold t[′] till
_t −_ 5 to make the BSEQs to be of reasonable length (at least 5) to capture temporal similarity without
discounting too many BSEQs. Top τ node pairs with lowest total DTW distance are assigned an edge.

**BSEQENC. While we can model backfill sequences for each signal independently using a recurrent**
neural network, this doesn’t capture the behavioral similarity of BSEQ across signals. Using a
fully-connected recurrent neural network that considers all possible interactions between signals also
may not learn from the similarity information due to the sheer number of signals (50 × 21 = 1050)
while greatly increasing the parameters of the model. Thus, we utilize the structural prior of
graph Gt generated by GRAPHGEN and train an autoregressive model BSEQENC which consists
of graph recurrent neural network to encode a latent representation for each of backfill sequence in
_Bt = {BSEQ(i, t) : i ∈F}. At week t, BSEQENC is first pre-trained and then it is fine-tuned for a_
specific model M (more details later in this section).

Our encoding process is in Figure 5. Let BSEQt′+r(i, t[′]) be first r + 1 values of BSEQ(i, t[′]) (till

_r[)]_
week t[′] + r). For a past week t[′] and revision week r, we denote h[(]i,t[t][′][′][ ∈] [R][m][ to be the latent encoding]

of BSEQt′r [(][i, t][′][)][ where][ t]r[′] [=][ t][′][ +][ r][ and][ t][′][ ≤] _[t]r[′]_ _[≤]_ _[t][. We initialize][ h]i,t[(0)][′][ for any observation week]_

_t[′]_ to be a learnable parameter h[(0)]i _∈_ R[m] specific to signal i. For each week t[′]r [we combine latent]

_r[−][1)]_ _r[)]_
encoding h[(]i,t[t][′][′] and signal value d[(]i,t[t][′][′][ using a GRU (Gated Recurrent Unit) (Cho et al., 2014) cell to]

get the intermediate embedding vi,t[(][t][′][)][. Then, we leverage the signal graph][ G][t][ and pass the embeddings]

_r[)]_ _r[)]_
_{vi,t[(][t][′][′][ :][ i][ ∈F}][ through a Graph Convolutional layer (Kipf & Welling, 2016) to get][ h][(]i,t[t][′][′][ :]_

_vi,t(t[′]r[′][ = GRU][)]_ [BE][(][d](i,tt[′]r[′][)][, h](i,tt[′]r[′][−][1)]), _{h(i,tt[′]r[′][)][ }][i][∈F][ = GConv(][G][t][,][ {][v]i,t(t[′]r[′][ }][)]_ _[i][∈F]_ [)][.] (1)

_r[)]_ _r[)]_
Thus, h[(]i,t[t][′][′][ contains information from][ BS][EQ][t][′]r [(][i, t][′][)][ and structural priors from][ G][t][. Using][ h][(]i,t[t][′][′][,]

_r[+1)]_
BSEQENC predicts the value d[(]i,t[t][′][′] by passing through a 2-layer feed-forward network FFNi:
_dˆ[(]i,t[t]r[′][′][+1)]_ = FFNi(h[(]i,t[t]r[′][′][)][ )][. During inference, we only have access to real-time values of signals for]

the current week. We autoregressively predict h[(]i,t[t][+][l][)] for each signal by initially passing _d[(]i,t[t][)]_
_{_ _[}][i][∈F]_
through BSEQENC and using the output {d[ˆ][(]i,t[t][+1)]}i∈F as input for BSEQENC. Iterating this l times

we get {h[(]i,t[t][+][l][)]}i∈F along with {d[ˆ][(]i,t[t][+][l][)]}i∈F where l is a hyperparameter.

**MODELPREDENC. To learn from history of a model’s predictions and its relation to target revisions,**
MODELPREDENC encodes the history of model’s predictions, previous real-time targets, and revised
(up to current week) targets using a Recurrent Neural Network. Given a model M, for each observation week t[′] 1, 2, . . ., t 1 _k_, we concatenate the model’s predictions y(M, k)t′, real-time target
_∈{_ _−_ _−_ _}_


-----

_yt[(][′][t]+[′][+]k[k][)]_ as seen on observation week t[′] and revised target yt[(][′][t]+[)] _k_ [as][ C]t[t][′][ =][ y][(][M, k][)][t][′][ ⊕] _[y]t[(][′][t]+[′][+]k[k][)]_ _⊕_ _yt[(][′][t]+[)]_ _k[,]_

where ⊕ is the concatenation operator. A GRU is used to encode the sequence {C1[(][t][)][, . . ., C]t[(]−[t][)]1−k[}][:]

_{z1[(][t][)][, . . ., z]t[(]−[t][)]1−k[}][ = GRU][ME][(][{][C]1[(][t][)][, . . ., C]t[(]−[t][)]1−k[}][)]_ (2)

**REFINER. It leverages the information from above three modules of B2F to refine model M** ’s prediction for current week y(M, k)t. Specifically, it receives the latent encodings of signals {h[(]i,t[t][+][l][)]}i∈F
from BSEQENC, zt[(][t][)]k 1 [from M][ODEL][P][RED][E][NC][, and the model’s prediction][ y][(][M, k][)][t][ for week][ t][.]
_−_ _−_

BSEQ encoding from different signals may have variable impact on refining the signal since a few
signals may not very useful for current week’s forecast (e.g., small revisions in mobility signals may
not be important in some weeks). Moreover, because different models use signals from CoVDS
differently, we may need to focus on some signals over others to refine its prediction. Therefore,
we first take attention over BSEQ encodings from all signals {h[(]i,t[t][+][l][)]}i∈F w.r.t y(M, k)t. We use
multiplicative attention mechanism with parameter w ∈ R[m] based on Vaswani et al. (2017):

_αi = softmaxi(y(M, k)twh[T]_ _[h]i,t[t][+][l][)][,]_ _h¯_ [(][t][)] = _αih[t]i,t[+][l]_ _._ (3)

_iX∈F_  

Finally we combine _h[¯][(][t][)]_ and zt[(][t][)]k 1 [through a two layer feed-forward layer][ FNN][RF][ which]
_−_ _−_
outputs a 1-dim value followed by tanh activation to get the correction γt ∈ [−1, 1] i.e., γt =
tanh(FFNRF(h[¯][(][t][)] _⊕_ _zt[(]−[t][)]k−1[))][. Finally, the refined prediction is][ y][∗][(][M, K][)][t][ = (][γ][ + 1)][y][(][M, K][)][t][.]_
Note that we limit the correction by B2F by at most the magnitude of model’s prediction because the
average BERR of targets is 4.9% and less than 0.6% of them have BERR over 1. Therefore, we limit
the refinement of prediction to this range.

**Training: There are two steps of training involved for B2F: 1) model agnostic autoregressive BSEQ**
prediction task to pre-train BSEQENC; 2) model-specific training for BFRP.

_Autoregressive BSEQ prediction: Pre-training on auxiliary tasks to improve the quality of latent_
embedding is a well-known technique for deep learning methods (Devlin et al., 2019; Radford et al.,
_r[+1)]_
2018). We pre-train BSEQENC to predict the next values of backfill sequences {x[(]t[′][t],i[′] _}i∈F_ . Note
that we only use BSEQ sequences BSEQt(t[′], i) _i_ _,t′<t available till current week t for training_
_{_ _}_ _∈F_
BSEQENC. The training procedure in itself is similar to Seq2Seq prediction problems (Sutskever
et al., 2014) where for initial epochs we use the ground truth inputs at each step (teacher forcing) and
then transition to using output predictions of previous time step by the recurrent module as input to
next time step. Once we pre-train BSEQENC, we can use it for BFRP as well as LBRP for current
week t for any model M . Fine-tuning usually takes less than half the epochs required for pre-training
enabling quick refinement of multiple models in parallel.

_Model specific end-to-end training:_ Given the pre-trained BSEQENC, we train, end-to-end,
the parameters of all modules of B2F. The training set consists of past model predictions
_⟨y(M, k)1, y(M, k)2, . . . y(M, k)t−1⟩_ and backfill sequences {BSEQt(t[′], i)}i∈F _,t′≤t. For datapoint_
_y(M, k)t[′] of week t[′]_ _< t_ _k, we input backfill sequences of signals whose observation week is_
_−_
_t[′]_ into BSEQENC to get latent encodings {h[(]t[′][t],i[)] _[}][i][∈F]_ [. We also derive][ z]t[(][′][t][ from M][)] [ODEL][P][RED][E][NC]

and finally REFINER ingests zt[(][′][t][,][)] _[ {][h]t[t][′],i[}][i][∈F][ and][ y][(][M, k][)][t][′][ to get][ γ][t][′]_ [. Overall, we optimize the loss]

2
function: = _i=1_ _γt′_ _y(M, k)t′_ _yt[(][′][t]+[)]_ _k_ . Following real-time forecasting, we train B2F
_L[(][t][)]_ _−_
each week from scratch (including pre-training). Throughout training and forecasting for week  _t, we_
use Gt as input to B[P][t]SEQ[−][k][−]E[1]NC since it captures average similarities in BSEQs till current week t.

4 BACK2FUTURE EXPERIMENTAL RESULTS

In this section, we describe a detailed empirical study to evaluate the effectiveness of our framework
B2F. All experiments were run in an Intel i7 4.8 GHz CPU with Nvidia Tesla A4 GPU. The model
typically takes around 1 hour to train for all regions. The appendix contains additional details (all
hyperparameters and results for June-Dec 2020 and k = 1, 3 and GDP forecasting). We also release
the code and datasets at www.github.com/AdityaLab/Back2Future.

**Setup: We perform real-time forecasting of COVID-19 related mortality (Deaths) for 50 US states.**
We leveraged observations (Section 2) from BSEQ for period June 2020 - Dec. 2020 to design B2F.


-----

We tuned the model hyperparameters using data from June 2020 to Aug. 2020 and tested it on the rest
of dataset including completely unseen data from Jan. 2021 to June 2021. For each week t, we train
the model using the CoVDS dataset available till the current week t (including BSEQs for all signals
revised till t) for training. As described in Section 3, for each week, we first pre-train BSEQENC on
BSEQ data and then train all components of B2F for each model we aim to refine. Then, we predict
the forecasts Deaths y[∗](M, k)t for each model M . Similarly we also evaluated B2F for real-time
GDP forecasting task with detailed results in Appendix. We observed that setting hyperparameter
_τ = c|F| where c ∈{2, 3, 4, 5} provided best results. Note that τ influences the sparsity of the graph_
as well as the efficiency of the model since sparser graphs lead to fast inference across GConv layers.
We also found setting l = 5 provided the best performance.

**Evaluation: Refined prediction y[∗](M, K)t are evaluated against the most revised version of the**
target yt[(][′][t]+[f][ )]k[, where][ t][f][ is second week of Feb 2021. For evaluation, we use standard metrics in this do-]

main Reich et al. (2019); Adhikari et al. (2019). Let absolute error of prediction e(M, k)t = _yt[(]+[t][f]k[ )]_

_ey([∗]M, k(M, K)t |)t and (b) Mean Absolute Percentage Error| for a week t and model M_ . We use (a) Mean Absolute Error MAPE = _T1[′]_ _Ti=1[′]_ _[e][(] MAE([M, k][)][t][ /]M[ |]) =[ y]t[(]+[t][f]Tk[ )]1[′] |[|]P[.]_ _iT=1[′]_ _[−][|]_

**Candidate models: We focus on refining/rectifying the top models from the COVID-19 Forecast**

P

Hub described in Section 2; these represent different variety of statistical and mechanistic models.

Table 2: B2F consistently refines all models. % improvements in MAE and **Baselines:** Due to
_MAPE scores averaged over all regions from Jan 2021 to June 2021_ the novel problem,

|Col1|Col2|k=2|Col4|k=4|Col6|
|---|---|---|---|---|---|
|Cand. Model|Refining Model|MAE|MAPE|MAE|MAPE|
|ENSEMBLE|FFN|-0.35 ± 0.11|-0.12 ± 0.22|0.87 ± 0.64|0.77 ± 0.14|
||B2F-MB|-2.23 ± 0.82|-1.57 ± 0.65|-2.19 ± 0.35|-2.85 ± 0.53|
||B2F-NOGRAPH|-1.45 ± 0.14|-2.73 ± 0.35|-5.72 ± 0.21|-6.72 ± 0.82|
||B2F-BSEQ|1.42 ± 0.60|0.37 ± 0.75|0.74 ± 0.36|0.44 ± 0.07|
||Back2Future|5.25 ± 0.13|4.39 ± 0.62|4.41 ± 0.73|3.15 ± 0.57|
|GT-DC|FFN|-2.42 ± 0.22|-1.51 ± 0.90|-1.54 ± 0.57|-0.48 ± 0.43|
||B2F-MB|-3.02 ± 0.40|-3.41 ± 0.16|-2.91 ± 0.29|-3.22 ± 0.74|
||B2F-NOGRAPH|2.24 ± 0.37|3.51 ± 0.21|1.93 ± 0.39|0.78 ± 0.37|
||B2F-BSEQ|2.13 ± 0.12|3.84 ± 0.78|1.08 ± 0.23|2.33 ± 0.97|
||Back2Future|10.33 ± 0.19|11.84 ± 0.18|9.92 ± 0.98|11.27 ± 0.88|
|YYG|FFN|-2.08 ± 0.39|-1.34 ± 0.12|-2.64 ± 0.13|-3.36 ± 0.18|
||B2F-MB|-3.84 ± 0.08|-6.99 ± 0.56|-8.84 ± 0.96|-5.61 ± 0.27|
||B2F-NOGRAPH|-1.25 ± 0.70|-0.7 ± 0.90|-6.13 ± 0.08|-5.31 ± 0.06|
||B2F-BSEQ|-1.78 ± 0.74|-2.26 ± 0.83|-0.79 ± 0.21|-0.62 ± 0.27|
||Back2Future|8.93 ± 0.26|6.32 ± 0.44|7.32 ± 0.42|5.73 ± 0.66|
|UMASS-MB|FFN|-3.25 ± 0.38|-5.74 ± 0.75|-1.01 ± 0.18|-5.28 ± 0.07|
||B2F-MB|-8.2 ± 0.61|-7.54 ± 0.26|-6.49 ± 0.29|-7.56 ± 0.30|
||B2F-NOGRAPH|-2.16 ± 0.22|-1.88 ± 0.67|-2.15 ± 0.24|-2.87 ± 0.34|
||B2F-BSEQ|1.58 ± 0.49|0.86 ± 0.17|0.36 ± 0.06|0.96 ± 0.82|
||Back2Future|5.43 ± 0.51|4.66 ± 0.63|3.32 ± 0.76|3.11 ± 0.29|
|CMU-TS|FFN|-5.24 ± 0.57|-4.93 ± 0.39|-3.12 ± 0.71|-0.65 ± 0.81|
||B2F-MB|-8.17 ± 0.34|-8.21 ± 0.24|-3.72 ± 0.32|-6.11 ± 0.84|
||B2F-NOGRAPH|-0.67 ± 0.69|-0.57 ± 0.32|-0.46 ± 0.07|-1.77 ± 0.79|
||B2F-BSEQ|1.46 ± 0.33|1.05 ± 0.16|2.38 ± 0.43|2.26 ± 0.02|
||Back2Future|7.5 ± 0.60|8.04 ± 0.58|5.73 ± 0.19|6.22 ± 0.58|



from MODELPREDENC and model’s prediction (c) B2F-BSEQ: exploits only BSEQ without
model bias and only uses BSEQENC architecture and append a linear layer that takes encodings
from BSEQENC and model’s prediction (d) B2F-NOGRAPH: does not use BSEQ similarity from
GRAPHGEN by removing graph convolutional layers and retain only RNNs.

**Refining real-time model-predictions: We compare the mean percentage improvement (decrease)**
in scores of B2F refined predictions of diverse set of top models w.r.t stable targets over 50 US
states.We observe that B2F is the only method, compared to baselines, that improves scores for all
candidate models consistently (Table 2). The poor scores of baselines also shows the necessity of
incorporating both backfill information (unlike FFN and B2F-MB) and model prediction history
(unlike B2F-BSEQ and B2F-NOGRAPH).

We achieve substantial avg. improvements of 6.93% and 6.79% in MAE and MAPE respectively
with low standard deviation across 29 test weeks which shows that the improvements were consistent
across time and not just over few weeks that experienced large revision anomalies. Candidate models
refined by B2F show improvement of over 10% in over 25 states and over 15% in 5 states (NJ, LA,
GA, CT, MD). The improved predictions of CMU-TS and GT-DC (ranked 3rd and 4th in COVID-19


-----

Forecast Hub) due to B2F, outperform all the models in the hub (except for ENSEMBLE) with 7.17%
and 4.13% improvements in MAE respectively. UMASS-MB, ranked 2nd, is improved by 11.24%.
B2F also improves ENSEMBLE, the current best-performing model of the hub, by 3.6% - 5.18% with
over 5% improvement in 38 states, and with IL and TX experiencing over 15% improvement.

65 0.60.5 **Rectifying real-time model-evaluation: We**

3214 % improv. in MAE0.40.30.20.10.0 CMUYYG UMass-MBEnsembleGT-DC evaluate the efficacy of B2F in rectifying thereal-time evaluation scores for the Llem. We noted in Obs 5 that real-time MAEBRP prob
0 Week No. was lower than stable MAE by 9.6 on average.

(a) Average % (b) % improve. in The difference between B2F rectified estimates

6
5

4
3

2

1

0

0.7

0.6

0.5

0.4

0.3

% improv. in MAE0.20.1 YYG UMass-MBEnsemble

0.0 CMU GT-DC

0 5 10 Week No.15 20 25 30

decrease of MAE MAE for each week MAE stable MAE was reduced to 4.4, a 51.7%

Figure 6: (a) B2F refines ENSEMBLE predictions decrease (Figure 7). This results in increased

MAE scores across most regions towards stable

_significantly for most states. (b) efficacy of B2F_

estimates. Eg: We reduce the MAE difference

_ramps up within 6 weeks of revision data._

in large highly populated states such as GA by
26.1% (from 22.52 to 16.64) and TX by 90% (from 10.8 to 1.04) causing an increase in MAE scores
from real-time estimates by 5.88 and 9.4 respectively.

**Refinement as a function of data availability: During the initial weeks of the pandemic, we have**
access to very little revision data both in terms of length and number of BSEQ. So we evaluate the
mean performance improvement for each week across all regions (Figure 6b). B2F’s performance
ramps up and quickly stabilize in just 6 weeks. Since signals need around 4 weeks (Obs 2) to stabilize,
this ramp-up time using small amount of revision data is impressive.


20001750 GT-DCEnsemble **B2F adapts to anomalies During real-time forecasting, models need**

1500 UMass-MBYYG to be also robust to rare events of large anomalous data revisions, like

1000 during the initial stages of the pandemic when data collection was

Stable MAE 500 not fully streamlined. Consider observation week 5 where there was

0 an abnormally large revision to deaths nationwide (Figure 8a) when

0 250 500 Rectified MAE750 1000 1250 1500 1750 2000 the BERR was 48%. B2F still provided significant improvements of

Figure 7: B2F rectified MAE up to 74.2% for most model predictions (Figure 8b).

2000 Ensemble

1750 GT-DC

1500 UMass-MBYYG

1250 CMU

1000

750

Stable MAE 500

250

0

0 250 500 Rectified MAE750 1000 1250 1500 1750 2000

_are closer to stable MAE_

**B2F refines GDP forecasts To evaluate the extensibility of B2F to**
other domains that encounter the problem of backfill, we tested on the task of forecasting US National
GDP using 25 macroeconomic indicators from past and their revision history for years 2000-2021. We
found that B2F improves predictions of candidate models by 6%-15% and significantly outperforms
baselines. The details of the dataset and results are found in Appendix Sections B, E.2.


5 CONCLUSION 750070006500 StableReal-time 8060

We introduced the important and chal- 60005500 4020
lenging multi-variate backfill problem Deaths5000 0
using COVID-19 and GDP forecast- 4000 40

released the comprehensive CoVDS Models

75007000 StableReal-time

6500

6000

5500

Deaths5000

4500

4000

3500

2 4 No. Weeks6 8 10

80

60

40

20

0

20

40

% improvement MAE 60

Ensemble GT-DC YYG UMass-MB CMU-TS

dataset to study revision patterns in (a) Revision of US deaths (b) % decrease in MAE due to B2F
features and targets as well as aid in on week 5 refinement on week 5 targets
COVID-19 forecasting. We presented Figure 8: B2F adapts to abnormally high revisions
Back2Future (B2F), the novel deep-learning method to model this phenomenon, which exploits our
observations of cross-signal similarities using Graph Recurrent Neural Networks to refine predictions
and rectify evaluations for a wide range of models. Our extensive experiments showed that leveraging
similarity among backfill patterns as well as model bias via our proposed method leads to significant
6 - 11% improvements in all the top models.

As future work, our work can potentially help improve data collection and alleviate systematic
differences in reporting capabilities across regions. For example, B2F provides significant gains
consistently across time including when there are large anomalies in data. Therefore, our revision
modelling approach can be helpful for anomaly detection (Homayouni et al., 2021). We can also study
how backfill can affect uncertainty calibration in time-series analysis (Yoon et al., 2020). Adapting to
situations where data revisions can occur at different frequencies is another research direction.


-----

6 ETHICS STATEMENT

The features used in the CoVDS dataset and for GDP forecasting are publicly available and
anonymized without any sensitive information. Our backfill refinement framework and B2F is
generalizable to any domain that deals with real-time prediction tasks with feature revisions. Due to
the relevance of our dataset to public health and macroeconomics, prospects for misuse should not be
discounted. The disparities in data collection across features and regions can also have implications
on equity of prediction performance and is an interesting direction of research.

7 REPRODUCIBLITY STATEMENT

As described in Section 4, we evaluated our model over 5 runs with different random seeds to
show the statistical significance of our method. We also provide a more extensive description of
hyperparameters and data pre-processing in the Appendix. The code for B2F and the CoVDS dataset
[is publicly available at https://github.com/AdityaLab/Back2Future.](https://github.com/AdityaLab/Back2Future)

8 ACKNOWLEDGEMENTS

This work was supported in part by the NSF (Expeditions CCF-1918770, CAREER IIS-2028586,
RAPID IIS-2027862, Medium IIS-1955883, Medium IIS-2106961, CCF-2115126), CDC MInD
program, ORNL, and faculty research awards from Facebook, funds/computing resources from
Georgia Tech.


-----

REFERENCES

Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan, and B Aditya Prakash. Epideep: Exploiting
embeddings for epidemic forecasting. In Proceedings of the 25th ACM SIGKDD International
_Conference on Knowledge Discovery & Data Mining, pp. 577–586, 2019._

Angel Aguiar. Macroeconomic data. 2015.

Nick Altieri, Rebecca L Barter, James Duncan, Raaz Dwivedi, Karl Kumbier, Xiao Li, Robert
Netzorg, Briton Park, Chandan Singh, Yan Shuo Tan, Tiffany Tang, Yu Wang, Chao Zhang, and
Bin Yu. Curating a covid-19 data repository and forecasting county-level death counts in the united
[states. Harvard Data Science Review, 2 2021. doi: 10.1162/99608f92.1d4e0dae. URL https://](https://hdsr.mitpress.mit.edu/pub/p6isyf0g)
[hdsr.mitpress.mit.edu/pub/p6isyf0g. https://hdsr.mitpress.mit.edu/pub/p6isyf0g.](https://hdsr.mitpress.mit.edu/pub/p6isyf0g)

Apple. Apple mobility trends reports., 2020. URL www.apple.com/covid19/mobility.

Alberto Baffigi, Roberto Golinelli, and Giuseppe Parigi. Bridge models to forecast the euro area gdp.
_International Journal of forecasting, 20(3):447–460, 2004._

Matthew Biggerstaff, Michael Johansson, David Alper, Logan C Brooks, Prithwish Chakraborty,
David C Farrow, Sangwon Hyun, Sasikiran Kandula, Craig McGowan, Naren Ramakrishnan, et al.
Results from the second year of a collaborative effort to forecast influenza seasons in the united
states. Epidemics, 24:26–33, 2018.

Logan C. Brooks, David C. Farrow, Sangwon Hyun, Ryan J. Tibshirani, and Roni Rosenfeld.
Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions. PLOS
_Computational Biology, 14(6):e1006134, June 2018. ISSN 1553-7358. doi: 10.1371/journal.pcbi._
[1006134. URL https://dx.plos.org/10.1371/journal.pcbi.1006134.](https://dx.plos.org/10.1371/journal.pcbi.1006134)

Andrea Carriero, Michael P Clements, and Ana Beatriz Galvão. Forecasting with bayesian multivariate vintage-based vars. International Journal of Forecasting, 31(3):757–768, 2015.

CDC. _Coronavirus Disease 2019 (COVID-19), 2020._ [URL https://www.cdc.gov/](https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/)
[coronavirus/2019-ncov/cases-updates/.](https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/)

Prithwish Chakraborty, Pejman Khadivi, Bryan Lewis, Aravindan Mahendiran, Jiangzhuo Chen,
Patrick Butler, Elaine O Nsoesie, Sumiko R Mekaru, John S Brownstein, Madhav V Marathe, et al.
Forecasting a moving target: Ensemble models for ili case count predictions. In Proceedings of the
_2014 SIAM international conference on data mining, pp. 262–270. SIAM, 2014._

Prithwish Chakraborty, Bryan Lewis, Stephen Eubank, John S. Brownstein, Madhav Marathe, and
Naren Ramakrishnan. What to know before forecasting the flu. PLoS Computational Biology,
[14(10), October 2018. ISSN 1553-734X. doi: 10.1371/journal.pcbi.1005964. URL https:](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6193572/)
[//www.ncbi.nlm.nih.gov/pmc/articles/PMC6193572/.](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6193572/)

Judith A Chevalier, Jason L Schwartz, Yihua Su, Kevin R Williams, et al. Measuring movement and
social contact with smartphone data: A real-time application to covid-19. Technical report, Cowles
Foundation for Research in Economics, Yale University, 2021.

Kyunghyun Cho, B. V. Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of
neural machine translation: Encoder-decoder approaches. ArXiv, abs/1409.1259, 2014.

Michael P Clements and Ana Beatriz Galvão. Data revisions and real-time forecasting. In Oxford
_Research Encyclopedia of Economics and Finance. 2019._

[COVID-Tracking. The covid tracking project., 2020. URL https://covidtracking.com.](https://covidtracking.com)

Estee Y Cramer, Velma K Lopez, Jarad Niemi, Glover E George, Jeffrey C Cegan, Ian D Dettwiller,
William P England, Matthew W Farthing, Robert H Hunter, Brandon Lafferty, et al. Evaluation of
individual and ensemble probabilistic forecasts of covid-19 mortality in the us. medRxiv, 2021.

Dean Croushore. Forecasting with real-time data vintages. The Oxford handbook of economic
_forecasting, pp. 247–267, 2011._


-----

Aron Culotta. Towards detecting influenza epidemics by analyzing twitter messages. In Proceedings
_of the first workshop on social media analytics, pp. 115–122, 2010._

Songgaojun Deng, Shusen Wang, Huzefa Rangwala, Lijing Wang, and Yue Ning. Cola-gnn: Crosslocation attention based graph neural networks for long-term ili prediction. In Proceedings of the
_29th ACM International Conference on Information & Knowledge Management, pp. 245–254,_
2020.

J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In NAACL-HLT, 2019.

David Farrow. Modeling the past, present, and future of influenza. Private Communication, Farrow’s
_work will be part of the published Phd thesis at Carnegie Mellon University, 2016._

Jeremy Ginsberg, Matthew H Mohebbi, Rajan S Patel, Lynnette Brammer, Mark S Smolinski, and
Larry Brilliant. Detecting influenza epidemics using search engine query data. Nature, 457(7232):
1012–1014, 2009.

[Google. Google covid-19 community mobility reports., 2020. URL https://www.google.](https://www.google.com/covid19/mobility/)
[com/covid19/mobility/.](https://www.google.com/covid19/mobility/)

Iwona Hawryluk, Henrique Hoeltgebaum, Swapnil Mishra, Xenia Miscouridou, Ricardo P Schnekenberg, Charles Whittaker, Michaela Vollmer, Seth Flaxman, Samir Bhatt, and Thomas A Mellan. Gaussian process nowcasting: Application to covid-19 mortality reporting. arXiv preprint
_arXiv:2102.11249, 2021._

Inga Holmdahl and Caroline Buckee. Wrong but Useful — What Covid-19 Epidemiologic Models
Can and Cannot Tell Us. NEJM, 383(4):303–305, 2020.

Hajar Homayouni, Indrakshi Ray, Sudipto Ghosh, Shlok Gondalia, and Michael G Kahn. Anomaly
detection in covid-19 time-series data. SN Computer Science, 2(4):1–17, 2021.

Xiaoyong Jin, Yu-Xiang Wang, and Xifeng Yan. Inter-series attention model for covid-19 forecasting.
_ArXiv, abs/2010.13006, 2020._

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
_arXiv preprint arXiv:1609.02907, 2016._

Vasileios Lampos, Tijl De Bie, and Nello Cristianini. Flu detector-tracking epidemics on twitter.
In Joint European conference on machine learning and knowledge discovery in databases, pp.
599–602. Springer, 2010.

Vasileios Lampos, Andrew C Miller, Steve Crossan, and Christian Stefansen. Advances in nowcasting
influenza-like illness rates using search query logs. Scientific reports, 5(1):1–10, 2015.

JF Lawless. Adjustments for reporting delays and the prediction of occurred but not reported events.
_Canadian Journal of Statistics, 22(1):15–31, 1994._

Shenghua Liu, Bryan Hooi, and Christos Faloutsos. Holoscope: Topology-and-spike aware fraud
detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge
_Management, pp. 1539–1548, 2017._

Massimiliano Marcellino. A linear benchmark for forecasting gdp growth and inflation? Journal of
_Forecasting, 27(4):305–340, 2008._

C Jessica E Metcalf and Justin Lessler. Opportunities and challenges in modeling emerging infectious
diseases. Science, 357(6347):149–152, 2017.

Baltazar Nunes, Isabel Natário, and M Lucília Carvalho. Nowcasting influenza epidemics using
non-homogeneous hidden markov models. Statistics in Medicine, 32(15):2643–2660, 2013.

Dave Osthus, Ashlynn R Daughton, and Reid Priedhorsky. Even a good influenza forecasting model
can benefit from internet-based nowcasts, but those benefits are limited. PLoS computational
_biology, 15(2):e1006599, 2019a._


-----

Dave Osthus, James Gattiker, Reid Priedhorsky, Sara Y Del Valle, et al. Dynamic bayesian influenza
forecasting in the united states with hierarchical discrepancy (with discussion). Bayesian Analysis,
14(1):261–312, 2019b.

G. Panagopoulos, Giannis Nikolentzos, and M. Vazirgiannis. United we stand: Transfer graph neural
networks for pandemic forecasting. ArXiv, abs/2009.08388, 2020.

George Panagopoulos, Giannis Nikolentzos, and Michalis Vazirgiannis. Transfer graph neural networks for pandemic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence,
2021.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. 2018.

Prashant Rangarajan, Sandeep K Mody, and Madhav Marathe. Forecasting dengue and influenza
incidences using a sparse representation of google trends, electronic health records, and time series
data. PLoS computational biology, 15(11):e1007518, 2019.

Nicholas G. Reich, Logan C. Brooks, Spencer J. Fox, Sasikiran Kandula, Craig J. McGowan, Evan
Moore, Dave Osthus, Evan L. Ray, Abhinav Tushar, Teresa K. Yamana, Matthew Biggerstaff,
Michael A. Johansson, Roni Rosenfeld, and Jeffrey Shaman. A collaborative multiyear, multimodel
assessment of seasonal influenza forecasting in the United States. Proceedings of the National
_Academy of Sciences of the United States of America, 116(8):3146–3154, 2019. ISSN 1091-6490._
doi: 10.1073/pnas.1812594116.

John C Robertson and Ellis W Tallman. Vector autoregressions: forecasting and reality. Economic
_Review-Federal Reserve Bank of Atlanta, 84(1):4, 1999._

Alexander Rodríguez, Nikhil Muralidhar, Bijaya Adhikari, Anika Tabassum, Naren Ramakrishnan,
and B Aditya Prakash. Steering a historical disease forecasting model under a pandemic: Case of
flu and COVID-19. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021a.

Alexander Rodríguez, Anika Tabassum, Jiaming Cui, Jiajia Xie, Javen Ho, Pulak Agarwal, Bijaya
Adhikari, and B. Aditya Prakash. DeepCOVID: An Operational Deep Learning-Driven Framework
for Explainable Real-Time COVID-19 Forecasting. In Proceedings of the AAAI Conference on
_Artificial Intelligence, 2021b._

Gerhard Rünstler and Franck Sédillot. Short-term estimates of euro area real gdp by means of
monthly data. Technical report, ECB working paper, 2003.

Jeffrey Shaman and Alicia Karspeck. Forecasting seasonal outbreaks of influenza. Proceedings of
_the National Academy of Sciences, 109(50):20425–20430, 2012._

Katrina Stierholz. Economic data revisions: What they are and where to find them. DttP, 45:4, 2017.

Oliver Stoner and Theo Economou. Multivariate hierarchical frameworks for modeling delayed
reporting in count data. Biometrics, 76(3):789–798, 2020.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks.
In NIPS, 2014.

Greg Tkacz and Sarah Hu. Forecasting gdp growth using artificial neural networks. Technical report,
Bank of Canada, 1999.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.

Peng-Wei Wang, Wei-Hsin Lu, Nai-Ying Ko, Yi-Lung Chen, Dian-Jeng Li, Yu-Ping Chang, and
Cheng-Fang Yen. Covid-19-related information sources and the relationship with confidence
in people coping with covid-19: Facebook survey study in taiwan. Journal of medical Internet
_research, 22(6):e20021, 2020._

Shihao Yang, Mauricio Santillana, and Samuel C Kou. Accurate estimation of influenza epidemics
using google search data via argo. Proceedings of the National Academy of Sciences, 112(47):
14473–14478, 2015.


-----

Changchang Yin, Ruoqi Liu, Dongdong Zhang, and Ping Zhang. Identifying sepsis subphenotypes
via time-aware multi-modal auto-encoder. In Proceedings of the 26th ACM SIGKDD international
_conference on knowledge discovery & data mining, pp. 862–872, 2020._

Jaesik Yoon, Gautam Singh, and Sungjin Ahn. Robustifying sequential neural processes. In
Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine
_Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10861–10870. PMLR,_
[13–18 Jul 2020. URL http://proceedings.mlr.press/v119/yoon20c.html.](http://proceedings.mlr.press/v119/yoon20c.html)

Qian Zhang, Nicola Perra, Daniela Perrotta, Michele Tizzoni, Daniela Paolotti, and Alessandro
Vespignani. Forecasting seasonal influenza fusing digital indicators and a mechanistic disease
model. In Proceedings of the 26th International Conference on World Wide Web, pp. 311–319.
International World Wide Web Conferences Steering Committee, 2017.

Indre Žliobaite. Change with delayed labeling: When is it detectable? In 2010 IEEE International
_Conference on Data Mining Workshops, pp. 843–850. IEEE, 2010._


-----

