Under review as a conference paper at ICLR 2022
CONTEXT-INVARIANT, MULTI-VARIATE TIME SERIES
REPRESENTATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern time series corpora, in particular those coming from sensor-based data,
exhibit characteristics that have so far not been adequately addressed in the litera-
ture on representation learning for time series. In particular, such corpora often al-
low to distinguish between exogenous signals that describe a context which inﬂu-
ences a given appliance and endogenous signals that describe the internal state of
the appliance. We propose a temporal convolution network based embedding that
improves on the state-of-the-art by incorporating recent advances in contrastive
learning to the time series domain and by adopting a multi-resolution approach.
Employing techniques borrowed from domain-adversarial learning, we achieve an
invariance of the embeddings with respect to the context provided by the exoge-
nous signal. To show the effectiveness of our approach, we contribute new data
sets to the research community and use both new as well as existing data sets to
empirically verify that we can separate normal from abnormal internal appliance
behaviour independent of the external signals in data sets from IoT and DevOps.
1
INTRODUCTION
Many modern applications in the physical and virtual world are equipped with sensors that measure
the state of the application, its sub-components, and the environment. Examples can be found in
the Internet-of-Things (IoT) or in the DevOps/AIOPs space like monitoring wind turbines or cloud-
based applications (Lu et al., 2009; Lohrmann & Kao, 2011; Nedelkoski et al., 2019; Li et al., 2020;
Krupitzer et al., 2020). Leveraging such time series to identify abnormal appliance behaviour is ap-
pealing (see Figures 1b,1c for an overview of time series anomaly types), yet certain characteristics
of these time series make them difﬁcult to model with existing representation learning techniques.
First, time series corpora are often highly multi-variate as illustrated in Figure 1a. Each appliance
has several sensors1 associated with it that measure both exogenous signals from the environment
as well as endogenous signals from the internal state of the appliance. Examples for exogenous
variables include user behaviour/trafﬁc in a web-based application or physical measurements such
as temperature in an IoT context. Conversely, endogenous variables could include the CPU usage or
the vibrations of a machine. Increased (application-internal) network trafﬁc is expected with higher
user load, and higher ambient temperatures naturally result in elevated temperature of a wind turbine.
It is however important to understand when an application deviates from such expected patterns and
exhibits unexpected behaviour relative to its environment. We call such effects contextual anomalies.
In addition, a deﬁning characteristic of such time series corpora is the sparsity and noisiness of their
associated labels. A label could indicate time spans when an application was in an a-typical state.
This sparsity may be due to diverse reasons ranging from practical (e.g., data integration or cost of
labelling) to fundamental (internal system failures may be exceedingly rare). Noisiness stems from
the fact that failures are often subjective and human insight is needed or alarms come from rule-
based systems that are themselves overly noisy (Bogatinovski et al., 2021; Wu & Keogh, 2021).
Hence, unsupervised or self-supervised representations of time series are needed that take the char-
acteristics of such modern time series corpora into account. However, while the ﬁeld of representa-
tion learning for sequential data has received considerable attention in domains s.a. natural language
processing (NLP) (Lan et al., 2020; Mikolov et al., 2013; Fang et al., 2020; Jaiswal et al., 2021),
1Each of these sensors may also measure multiple statistics of the signal (e.g., min, max, avg, std).
1
Under review as a conference paper at ICLR 2022
Server app
Exogenous
Endogenous
Request rate
Net trafﬁc
CPU usage
RAM usage
avg
std
min
max
(a) DevOps signal hierarchy ex-
ample. Each entity contains ex-
ogenous and endogenous sensors
and each sensor is comprised of
different summary statistics.
0
5
0.5
1
endog.
exog.
(b) Pattern anomaly: The time se-
ries is anomalous w.r.t its typical
behaviour.
The depency structure
between exogenous and endogenous
variables is preserved.
0
5
0.2
0.4
0.6
0.8
1
pred.
(c) Contextual anomaly: The time
series is anomalous w.r.t context
from another time series.
The de-
pendency structure between exoge-
nous and endogenous variables is
changed.
Figure 1: Structure of time series corpora (left) and different types of anomaly (middle and right).
similar work in the numerical time series domain remains rare. Speciﬁcally, rich, multi-purpose rep-
resentations facilitating down-stream applications are common in NLP. Instead, feature extraction
methods mostly dominate for time series (Lubba et al., 2019; Christ et al., 2017) with Franceschi
et al. (2019) providing a notable exception based on temporal convolution networks (TCN). The
main contribution of our paper is the extension of this TCN-based approach to cater for the afore-
mentioned complications. We summarize our contributions as follows:
1. We propose context-invariant embeddings that allow to identify representations of time se-
ries that are invariant to the exogenous variables. We achieve this by adapting domain adver-
sity (Ganin et al., 2016) to the time series domain.
2. We extend the TCN (Franceschi et al., 2019) model with (i) modern contrastive losses that we lift
to the time series domain for the ﬁrst time, (ii) data augmentation techniques, and (iii) considering
time series simultaneously at multiple resolutions.2
3. We conduct an empirical study in which we show the effectiveness of our approach. We provide
a semi-synthetic DevOps data set that we contribute to the research community and consider an
under-explored wind turbine dataset (apart from classical synthetic and physical datasets).
Our quantitative results show that context-invariant embeddings indeed represent time series data
such that contextual anomalies can be identiﬁed in a label-effective way. The qualitative results
show that the embeddings allow us to navigate complex data sets in an explorative manner (e.g.,
considering nearest/farthest neighbours of interesting time series snippets).
2
REPRESENTATION LEARNING WITH CONTEXT INVARIANCE
To motivate our approach, consider a simpliﬁed system where under normal operation a single
endogenous variable y depends instantaneously on a single exogenous signal x via a function
y = g(x) + ε, where ε is a noise term. The ideal signal to detect contextual anomalies (those
that break this relation) is the residual δ := y −g(x). Under normal operation this signal carries
no information about the exogenous variable and thresholding the magnitude of this residual signal
can detect anomalies. Our approach is motivated by this setup but extends it to more complex situ-
ations where (i) exogenous & endogenous variables can be multivariate, (ii) the relation stochastic
& highly non-linear, and (iii) may depend on the history of the system state. In this case we cannot
simply compute a ”residual signal”, but instead we can try to learn unsupervised representations that
are invariant to the exogenous variable. This means the embeddings should be independent of the
driving signal as long as the endogenous variables respond in a typical manner, which captures some
aspect of the residual signal of the toy example. In the following sections we formalize this intuition
further and show that context invariance indeed helps detect such anomalies.
Let Z = {zi ∈DT }N
i=1 be a set of N equally spaced time series zi of length at most T ∈N where
D is a domain of numerical values. We do not assume time series to be of equal length. We assume
2Indeed, time series corpora typically consist of equally-spaced time series (e.g., time series with measure-
ments in 1-min, 5-min or 10-min intervals). This allows us to reason at multiple resolutions.
2
Under review as a conference paper at ICLR 2022
a decomposition Z = X ∪Y such that time series in X allow to predict time series in Y . We call
X the set of environmental/exogenous time series and Y the set of internal/endogenous time series.
We assume that it is possible to predict Y from X, but we make no assumption on causality.
The goal of this paper is to map sub-series of Z into a high-dimensional embedding space RM which
preserves loosely deﬁned properties such as: “normal” time series are close to each other and far
away from “abnormal” states. This facilitates down-stream tasks such as time series classiﬁcation
or anomaly detection in a label sparse setting. In particular, our deﬁnition of “normal” should be
context-invariant, that is, only changes in the dependency structure between Y and X should result
in large distances in the embedding space. For these tasks, a limited number of labels is available
that allows to identify a time span of abnormal behaviour. Typically, the amount of labels is such
that a supervised approach is prohibitive and even evaluation may be a challenge.
Our representation learning approach consists of two main components: a predictor network g that
ties the endogenous and exogenous time series together (either by predicting endogenous from ex-
ogenous variables, or vice-versa), and, an embedding network f which learns embeddings using
contrastive losses. We can combine both in multiple ways. One extreme is a two-step approach
where we learn embeddings on the residuals of the predictor network. The other extreme is an
end-to-end approach, where we learn embeddings such that the distance between (multivariate) time
series is adjusted based on the exogenous variables in a domain-adversarial way. Figure 2 depicts the
main components of our approach. For the predictor network, we mainly resort to standard models,
so we focus our exposition on the main (novel) components in the following.
2.1
CONTRASTIVE, SELF-SUPERVISED, LEARNING OF MULTI-RESOLUTION TCN NETWORK
The basic building block of our embedding network architecture (Franceschi et al., 2019) consists
of stacked temporal dilated causal convolutions (Bai et al., 2018). We have multiple such networks,
one per time resolution. We illustrate in Figure 4 the effect of aggregation on the input time series.
To obtain a consolidated representation, the concatenated representations are mapped through a
neural network. These multi-resolution representations allow the network to encode patterns that
are more pronounced in the higher resolutions of the time series in a way that is more effective
than an encoder which only operates on a single resolution. We choose resolutions manually as
the natural granularities corresponding to the base frequency of the time series we consider in our
empirical studies (e.g., seconds, minutes, hours).
Similar to (Franceschi et al., 2019), we rely on a contrastive, self-supervised learning approach to
train the embedding network. This crucially relies on a loss function and a careful selection of
positive (a, b)p ∈(X, Y ), reference (c, d)c ∈(X, Y ) and negative (x, y)n ∈(X, Y ) time series
snippets on which to compute the loss terms (depicted in Figure 2). Similar time series should be
close to each other and dissimilar time series distant from each other in the embedding space. For
an embedding network fW with parameters W, the loss function takes the following general form:
min
W dist(fW ((c, d)c), fW ((a, b)p)) + max
W dist(fW ((x, y)n, (a, b)p))
(1)
We choose (a, b)p, (c, d)c such that (c, d)c ⊇(a, b)p while (x, y)n is such that x ∩c ≈∅, y ∩d ≈∅
(e.g., time snippets at different times and from different elements in the batch). Note further that
(x, y)n is constructed to explicitly break the dependency structure in Z by choosing x to be the
exogenous variables at a different time than y. During training, we further augment the examples
randomly before feeding them to the TCN network. In particular we apply random jittering, scaling,
ﬂipping direction, 2d rotation around a center, permuting random segments, magnitude or time
warping (Um et al., 2017) and window slicing or wrapping (Guennec et al., 2016).
Equation (1) is designed to support a variety of contrastive loss functions. Apart from the loss
discussed in Franceschi et al. (2019), we rely on other, more recent losses which we describe in the
following brieﬂy. These losses, in particular the latter two, aim to avoid collapse of the embeddings
while taking practical consideration (e.g., the size of the batch) into account.
The SimCLR (Chen et al., 2020) takes two random windows zA and zB of a time series and encodes
it to get two representations hA and hB. It then maximizes the similarity between these two repre-
sentations from the same time series and dissimilarity between others representations in the batch
using the Normalized Temperature-Scaled Cross-Entropy loss (Sohn, 2016) as the distance in (1).
3
Under review as a conference paper at ICLR 2022
Embedding
Contrastive
component
Domain
adversarial
component
Series 1 exog.
Series 1 endog.
Series 2 exog.
Series 2 endog.
fW
fW
fW
gU
gU
c
d
a
b
x
y
c
d
a
b
x
b
x
a
x
x
x
Figure 2: Context-invariant embeddings with contrastive and domain adversarial learning. We
select negative samples such that the correlation structure between exogenous and endogenous sig-
nals is explicitly broken. For the other samples, we augment them randomly, denoted by the squiggly
arrows. In addition to the contrastive component, we further add a domain adversarial component
that, given a positive embedded sample, tries to reconstruct the context of the positive sample as
badly as possible while reconstructing the context of the negative sample as well as possible.
Formally, for temperature parameter τ and mini-batch size of N for each pair we deﬁne the loss as:
ℓA,B = −log
exp(dot(hA, hB)/τ)
P2N
n=1,n̸=A exp(dot(hA, hn)/τ)
,
(2)
where dot is the dot-product between ℓ2-normalized vectors. This contrastive loss beneﬁts from
larger N which might not be feasible in the time series setting and thus we also explore other losses.
Barlow Twins (Zbontar et al., 2021) is a loss operating on two batches of different windows from
the same respective time series embeddings, ZA and ZB. It computes the cross-correlation matrix
along the batch dimension and stores the result in a square matrix C. The ﬁnal loss then encourages
the diagonal terms in this matrix to be close to 1 and the off-diagonal terms be close to 0. Formally,
ℓ=
X
i
(1 −Cii)2 + λ
X
i
X
i̸=j
C2
ij,
(3)
where λ > 0 trades off the contribution of the ﬁrst and second term in the loss. Intuitively, this
decorrelation reduces the redundancy between the output embeddings forcing them to contain non-
redundant information about the time series.
Unlike SimCLR, MoCo (He et al., 2020) uses two encoders to obtain representations for the two
random windows from the same time series. The representations through the 2nd momentum en-
coder are preserved in a queue. During training, positive pairs in (2) are constructed from the current
batch while negative pairs (denominator of (2) are constructed from the queue of embeddings. The
2nd encoder is updated by linear interpolation of the two encoder with a momentum-based moving
average of their weights during training. By using a queue with a slowly changing encoder, this loss
attempts to construct large and consistent embeddings which better samples the continuous high
dimensional space, independent of the batch size.
2.2
COMBINATIONS OF PREDICTOR AND EMBEDDING NETWORK: FROM TRIVIAL TO
CONTEXT-INVARIANT
We can combine the components described above to obtain context-invariant embeddings in a num-
ber of ways. The most trivial way is to ignore the predictor network gU and learn representations of
4
Under review as a conference paper at ICLR 2022
the entire set Z (or similarly, only for the endogenous part Y ). We denote this idea as BasicEmb.
Next, a two-step approach for context-invariance consists of training the predictor network gU ﬁrst,
then learn embeddings gU(Y ) −Y on top of the residual signals in a second step. We refer to this
approach as ResEmbRegr. Finally, we describe how to combine both networks into an end-to-end
model which we call ContInvEmb. This approach is the most ﬂexible in the sense that it allows to
adjust the strength of the context-invariance depending on the need of the application.
Our goal is to construct embeddings that are invariant to the corresponding (exogenous) context. Put
differently, the embeddings should not contain information that allows prediction of the correspond-
ing exogenous context. This in turn requires to regress the exogenous signals against the embedding
but instead of minimizing the regression error, we attempt to maximize the regression error. More
formally, we have
R(W) = min
U
X
(a,b)p,(x,y)n
Lr(W, U, a, b, x, y) + λ max
U
X
(a,b)p
Lr(W, U, a, b) ,
(4)
where Lc(W, U, a, b, x, y) is a regression loss, W are the parameters of the encoder network fW ,
and U are the free parameters of the prediction network gU.
The loss in Equation (4) aims to reconstruct the exogenous variables from embeddings as badly as
possible, thereby pushing embeddings to invariance wrt the exogenous context. The ﬁrst loss term
leads to good predictions of the exogenous variables shifted with respect to the embedded signals.
The second term leads to bad predictions of exogenous variables a from embeddings of positive
examples (a, b)p. Gradient reversal handles the adversarial learning aspect (Ganin et al., 2016).
Instead of using to a regression loss in (4), we resort to a multi-class classiﬁcation problem by
discretizing the input space. This has attractive properties in related tasks (Rabanser et al., 2020), but
importantly for this particular application avoids explicitly handling trivial predictions (like g(·) =
±∞) and domains are naturally bounded in the practical applications we consider. Combining (4)
with a contrastive loss, we arrive at the following overall loss:
min
W (L(W) + λR(W)) ,
(5)
where L(W) is a contrastive loss as discussed in Section 2.1, and R(W) is weighted by λ (a hyper-
paramater) and acts as a regularization term. For λ > 0, we obtain context-invariant embeddings
and for λ = 0, we recover (Franceschi et al., 2019) (modulo our extensions).
3
RELATED WORK
Computer vision (CV) and NLP (Chen et al., 2020; van den Oord et al., 2019; He et al., 2020; Fang
et al., 2020; Jaiswal et al., 2021) have embraced self-supervised representations. Most relevant to
us, Sohn et al. (2021) learn representations with a contrastive loss to enable anomaly detection in
CV. In contrast, time series analysis has not seen a similar adoption of self-supervised techniques
for learning general-purpose representations. Franceschi et al. (2019) provide a notable exception
by proposing a TCN based embedding (Bai et al., 2018) learnt with a contrastive loss.
The approach of Franceschi et al. (2019) departs from a rich ﬁeld of feature extraction from time
series (Lubba et al., 2019; Christ et al., 2017). While these approaches indeed classify time series
well in practice, they mostly focus on the uni-variate case. Their extensions to the multi-variate case
are out-performed by Franceschi et al. (2019); Bagnall et al. (2018). Furthermore, the versatility
of the features learned by classical approaches is limited by the fact that distances in the induced
embeddings are not properly learnt. We extend Franceschi et al. (2019) in the following directions:
(i) we adopt it to be multi-resolution; (ii) we equip it with more recent contrastive loss functions and
(iii) we turn it into a context-invariant embedding via domain-adversarial learning.
For (i), we note that the de facto choice in a multi-resolution context would be to sub-sample the
time series. This is done in classical Wavelet analysis (Mallat, 1989). Instead, we draw inspiration
from temporal hierarchical time series analysis (Athanasopoulos et al., 2017) and opt to aggregate
time series along the time dimension leading to vectors with a ﬁxed dimension. For (ii), we adopt
loss functions (He et al., 2020; Zbontar et al., 2021; Chen et al., 2020) recently proposed for con-
trastive and self-supervised learning and transfer them to the time series domain. For (iii), we draw
5
Under review as a conference paper at ICLR 2022
on domain-adversarial representation learning, primarily Ganin et al. (2016). While Ganin et al.
(2016) learns embeddings with respect to a speciﬁc label classiﬁcation task, we instead adopt an
unsupervised approach via contrastive learning. Further, we replace the domain classiﬁer with an
exogenous context regressor whose loss we seek to maximize. Such deep prediction networks can be
sophisticated, as is the case in particular in the forecasting literature (see e.g., Benidis et al. (2020)
for an overview). Our approach readily extends to these, but we restrict ourselves to standard neural
regression models as they sufﬁce in the scenarios that we consider for our empirical studies. Other
approaches such as hypernetworks (Ha et al., 2016) are conceivable. Yet, they suffer from a lack of
computational efﬁciency and robustness which inhibits their practical applicability.
4
EMPIRICAL EVALUATION
Our empirical evaluation consists of two parts. We ﬁrst focus on dissecting the improvements on
embedding learning through multi-resolution handling, contrastive losses and data-augmentation.
Second, we examine the context-invariant representations towards their anomaly detection potential.
4.1
IMPROVEMENTS TO EMBEDDING
For this base experiment, we aim to show the versatility of the embeddings through a down-stream
forecasting and classiﬁcation task each of which we evaluate on two datasets representing the easy
and the hard spectrum of the task. We do not aim for comprehensiveness in these ﬁrst set of ex-
periments but rather for an assessment of the relative improvements through our extensions. As a
reference point to gauge the absolute accuracy better, we include a classical feature extraction base-
line (Lubba et al., 2019), Catch22. In our experiments, we address the downstream classiﬁcation
and forecasting task via simple linear models.
In classiﬁcation, we consider a synthetically generated data set for which we know the labels from
the data generation process and the M5 forecasting competition dataset (Theodorou et al., 2021).
The latter data set is a retail demand forecasting data set that has product categories associated with
the time series. Using a (multinomial) logistic classiﬁer, we aim to predict the labels in both data
sets. The synthetic data set is designed such that a high classiﬁcation accuracy can be and the labels
are ”objective”. In contrast, for the M5 data set, a high classiﬁcation accuracy cannot be expected.
Apart from the amount of product categories available (≈3k), time series associated to different
products may have similar characteristics and hence, from a time series classiﬁcation point of view,
labels do not represent a ground truth, a common scenario in practice.
In forecasting, we predict electricity3 and M5 (Makridakis & Spiliotis, 2021) using a shared linear
forecaster. The forecaster takes the context embeddings from a large time series window together
with a smaller context window of the actual time series to predict for the dataset’s target horizon.
This horizon is 24 time steps ahead for electricity and 28 days for M5 and compare the metrics with
the corresponding test splits.
Discussion. Table 1 summarizes our ﬁndings. The more recent proposals for contrastive losses,
MoCo and Barlow Twins, are superior, but there is no clear indication which of both losses is
superior overall. While present, we remark that the relative improvements in accuracy are small
when compared to the performance wins reported in the original domains for which these losses
were designed. We speculate that this may be more attributable to the datasets in the respective
domains (and the objectivity of the associated labels) than the loss functions themselves.
Note that including multi-resolution leads to such overwhelming improvements in the classiﬁcation
task that we show only multi-resolution embeddings (for the base resolution, factor 60 and 300)
for the classiﬁcation tasks as these are almost strictly superior (> 10%). Similarly, we report re-
sults in the classiﬁcation task with data augmentation, although improvements are not consistent
for data augmentation (Appendix B.2 contains results for an ablation study). The effectiveness of
multi-resolution may be surprising as higher capacity models in general and convolutions with a
higher dilation in particular should in theory be able to model similar effects. However, adding
multiple resolutions offers an inductive bias akin to lagged values in RNN-based forecasting models
(e.g., Salinas et al. (2019)) which have been shown to lead to superior practical results.
3https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
6
Under review as a conference paper at ICLR 2022
Loss
Classiﬁcation ↑: Syn
Classiﬁcation ↑: M5
Forecasting ↓: Electricity
Forecasting ↓: M5
SimCLR
97.00
1.34
0.097
0.70
Barlow Twins
97.25
5.93
0.091
0.699
MoCo
99.50
3.08
0.089
0.694
Catch22
64.50
0.03
0.11
0.713
Linear
-
-
0.092
0.698
DeepAR
-
-
0.073
0.90
Table 1: Classiﬁcation and forecasting accuracy using embeddings learnt with different losses. For
forecasting, the accuracy metric shown is P50 loss. Results are averages over 5 runs. The last three
lines are baselines using classic time series features, Catch22, and replacing embeddings with
more historic values for the forecasting task, Linear and a pure forecasting method, DeepAR.
For forecasting, we note that past historic values instead of embeddings (Linear in Table 1) is
a competitive approach, in particular for forecasting the M5 data set. Nevertheless, higher quality
embeddings coincide with better forecasting accuracy (comparing Catch22 with our embeddings).
Compared with the Electricity dataset, the M5 dataset offers less overall structure so historic values
offer a strong signal. Consequently, forecasting accuracy wins are more pronounced in the Electric-
ity dataset.
4.2
CONTEXT INVARIANT EMBEDDINGS
In our main experiments, we consider the effect of domain-adversarially learnt context-invariant
embeddings in both qualitative and quantitative experiments. The task consists of the identiﬁcation
of contextual anomalies. Perhaps surprisingly, qualitative evaluation is almost more meaningful
for the anomaly detection task in general, and for contextual anomalies in particular, given the
subjectivity and noisiness of the labels. Note that most publicly available anomaly detection data
sets4 are not suitable for the task, so we do not consider them. Instead, we evaluate on 4 different
data sets, each of which allows for a separation into exogenous and endogenous signals. We discuss
these data sets ﬁrst, then the evaluation approach, the models under consideration, and ﬁnally discuss
the results. Appendix B.3 contains further details.
Datasets
The selection of the evaluation datasets aims to balance physical and virtual appliances
as well as synthetic and real-world data. Note that synthetic data come with perfect labels, while
real-world data typically does not. While lamentable, we believe that this subjectivity and noisiness
must be embraced as fundamental in the task. Appendix A contains further details on the data sets.
Synthetic data. We generate a total of 4 × 360 time series of length 700, based on simple generative
models. We generate two exogenous signals, as well as two endogenous signals. We inject two types
of anomalies into the data: (i) pattern anomalies, i.e., anomalies in the exogenous which are also
instantly reﬂected in the endogenous variables and (ii) contextual anomalies only in the endogenous
variables. We aim to detect contextual anomalies.
Pendulum. We consider the case of a swinging pendulum with added control signals, where we
control the dampening of the acceleration from the outside as an exogenous signal (towards which
we want to be invariant) and consider as contextual anomamlies those where we inject an anomalies
as a change in the length of the chord which we capture as part of the endogenous signal. Our aim
with this data set is to understand how well our models handle cases where the dependency structure
between X and Y is more complex.
DevOps. This is a new, semi-synthetic data set5 that we generated for the purpose of this publication
to resemble commonly observed data sets behind corporate ﬁrewalls. The object under considera-
tion is a popular cloud-based microservice demo application,6 which is commonly used in an AIOPs
context (Wu et al., 2020). As exogenous signal, we record user interaction approximated by the net-
4Wu & Keogh (2021) convincingly argue that many of these datasets should be abandoned.
5Jointly with this publication, we open-source both the raw data as well as the set-up to produce the data.
To the best of our knowledge, we are the ﬁrst to extract a data set from this set-up that allows the machine
learning community to interact with this area without deep engineering knowledge which is more present in the
system’s community where the data generation framework is typically considered.
6https://github.com/microservices-demo/microservices-demo
7
Under review as a conference paper at ICLR 2022
work outbound trafﬁc of an application that induces synthetic load on the application and which
we fully control. The endogenous signals consists of metrics like CPU, memory and others of the
actual microservice application. Each recorded metric has multiple statistics available. We inject
anomalies both in the user behavior (leading to pattern anomalies) and in the internal state (contex-
tual anomalies). We want to ignore the former and ﬁnd the latter. The appendix contains illustrative
plots similar to Figure 6. Note that although we have almost total control of the application and its
anomalies, the labelling of anomalies is still not perfect, thereby adding further to the complications
of public anomaly detection benchmarks (Wu & Keogh, 2021).
Turbine. We consider a wind turbine data set open-sourced7 by Energias de Portugal. The time
series panel can be separated into exogenous and endogenous signals. The former consists of wind
speed/direction, ambient temperature and, the pitch angle of the blades (this is controlled from
the outside and including it improves the quality of the predictive model). The latter consists of
rotation speeds for the turbine and generators, internal temperature on different components, as
well as the power output each. All series are available for 4 distinct turbines, are sampled at a 10
minute frequency and are available for 2016. Note that this data set contains only few (43) labelled
anomalies and visual inspections of the data reveals inconsistencies with these labels, e.g., some
time series appear to be mislabeled (see Sec. 4.2 for an example). The providence of these labels is
from automated alarming systems which are often threshold based. Hence, quantitative evaluations
cannot be taken at face value. However, given the rich structure of the data and the fuzziness of the
task stemming from the labels, the versatility of our approach can be illustrated qualitatively.
Models & Evaluation
We evaluate the following model conﬁgurations for their suitability for
the contextual anomaly detection task: BasicEmb, the modiﬁed (Franceschi et al., 2019) (with
multi-resolution) learnt ignoring the structure imposed by endogenous and exogenous signals;
ResEmbRegr, embeddings on the residuals of a predictive models (a feed-forward neural network);
and ContInvEmb, context-invariant embeddings with a simple linear model for the prediction task
(the simplicity ensure that the embeddings are adjusted enough). Moreover, we also provide results
for three baseline approaches: ResTresh, which computes residuals as in ResEmbRegr com-
bined with a simple thresholding mechanism; and two instances of (Lubba et al., 2019): Catch22
and and ResCatch22 which compute a feature vector per original and residual time series re-
specitvely similar to ResEmbRegr. The only hyperparameter tuning that we perform is on the
Synthetic data set as we do not have enough labels available otherwise.
For all methods, we report standard AUROC scores on the contextual anomaly detection task. For
the ResThresh method, we compute the AUROC score based on the maximum residual value over
the full residual series. For the embedding-based approaches, we use a k-nearest-neighbor classiﬁer
in the embedding space to determine a discrete anomaly label.
Quantiative Results
Tables 2 summarizes the quantiative results. First, we note that the difﬁ-
culty of contextual anomaly detection differs widely with the data sets. For example, contextual
anomaly detection seems relatively easier on the Turbine data set compared with the DevOps data
set despite the latter being semi-synthetically generated with controlled labels. Second, we note that
ContInvEmb leads to overall superior results when comparing the embedding based approaches
(the ﬁrst three columns in Table 2) and is overall competitive results in many cases, but not always.
For the DevOps and Turbine data sets ResTresh is overall best by a margin. One explanation is
that ResTresh best resembles the label generation process by the automated alarming systems on
the turbines. For Pendulum data set it is worthwhile to note that ContInvEmb delivers an overall
superior approach despite the complex non-linear interaction between the exogenous and endoge-
nous signals and only a linear context-predictor component.
Qualitative Results: a case study on wind turbines
The embeddings allow to navigate the time
series corpus via distances which can be helpful in exploring the data set and uncovering data or
label issues. In Figure 3 we show this in qualitative results for context-invariant embeddings. The
ﬁrst two columns show sanity checks: as expected, reference time series (in the top row) labelled as
normal or abnormal have as their nearest neighbors (in the same column as the reference time series
below it) normal and abnormal time series respectively. Despite the multi-variate nature of the data,
visual inspection conﬁrms that the nearest neighbors are plausible. The last column in Figure 3
7https://opendata.edp.com/pages/homepage/
8
Under review as a conference paper at ICLR 2022
0
20
40
60
80
100 120 140
(a) Clean reference series (correctly
labelled as clean).
0
20
40
60
80
100 120 140
(b) Anomalous reference series
(correctly labelled as anomalous).
0
20
40
60
80
100 120 140
(c) Anomalous reference series (in-
correctly labelled as clean).
0
20
40
60
80
100 120 140
0
20
40
60
80
100 120 140
0
20
40
60
80
100 120 140
Figure 3: Nearest-neighbor analysis of context-invariant approach on the Turbine dataset.
Each panel shows exogenous signals on the top and endogenous signals at the bottom. The ﬁrst
row shows (a) a normal reference series; (b) an abnormal reference series (anomaly between time
steps 20 and 70); and (c) an anomalous reference series that was mistakenly labeled as clean. Below
each of these series we show 3 nearest neighbors in the learned embedding space. For the left and
center column, both the correct label and our predicted label match. For series (c), the reference
time series is labelled as clean, but its nearest neighbors are abnormal. It is apparent that the erratic
spiking patterns in either exogenous/endogenous signals are usually not reﬂected in the other series.
BasicEmb
ResEmbRegr
ContInvEmb
ResTresh
Catch22
ResCatch22
Synthetic
0.512 (± 0.022)
1.000 (± 0.000)
0.999 (± 0.002)
1.000 (± 0.000)
0.494 (± 0.008)
1.000 (± 0.000)
Pendulum
0.969 (± 0.013)
0.951 (± 0.015)
0.980 (± 0.002)
0.510 (± 0.000)
0.904 (± 0.000)
0.891 (± 0.000)
DevOps
0.535 (± 0.041)
0.532 (± 0.036)
0.587 (± 0.007)
0.619 (± 0.000)
0.573 (± 0.000)
0.573 (± 0.000)
Turbine
0.632 (± 0.015)
0.725 (± 0.018)
0.736 (± 0.022)
0.845 (± 0.000)
0.512 (± 0.000)
0.680 (± 0.000)
Table 2: AUROC results on the anomaly detection task with 5 seeds. Larger values are better.
shows an example where a reference time series is labelled as normal, but its nearest neighbors
consist of time series that are labeled as abnormal. This could point to an issues with the labels. In
this case, without further domain knowledge, it may make sense to re-label the reference time series
in column (c) as abnormal for the abnormal stretches 0-40 and 100-130.
5
CONCLUSION
In this work, we presented self-supervised learning of time series embeddings that are invariant with
respect to a known and fully-observed context. While the architectures that we presented here lean
on techniques invented for computer vision, we make non-trivial contributions to adapt them to the
time series domain. For example, we equip our embeddings to consider multiple resolutions of the
original sensor signals simultaneously. We observe that the learned embeddings are sensitive to
changes of the dependency structure between exogenous and endogenous variables. As conﬁrmed
in our evaluation, this allows our approach to learn embeddings that separate dependency-breaking
anomalies in the state of the appliance which is the object of interest.
Potential future works could explore techniques from causal discovery (Haufe et al., 2009; Qiu et al.,
2020; 2012) to automatically derive an exogenous/endogenous decomposition of the multi-variate
time series panel and extensions to causal representation learning (Sch¨
olkopf et al., 2021).
9
Under review as a conference paper at ICLR 2022
6
REPRODUCIBILITY
We use a combination of public datasets and synthetic data sets generated using publicy available
code. Method implementations will be open sourced as part of the review process. Finally, we will
share the data sets, the models to create the synthetic data and the code with notebooks that allows
to reproduce all our results with the ﬁnal version of this paper.
REFERENCES
George Athanasopoulos, Rob J. Hyndman, Nikolaos Kourentzes, and Fotios Petropoulos. Fore-
casting with temporal hierarchies.
European Journal of Operational Research, 262(1):60–
74, 2017.
ISSN 0377-2217.
doi: https://doi.org/10.1016/j.ejor.2017.02.046.
URL https:
//www.sciencedirect.com/science/article/pii/S0377221717301911.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul
Southam, and Eamonn Keogh. The uea multivariate time series classiﬁcation archive, 2018, 2018.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling, 2018.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernie Wang, Danielle Mad-
dix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella,
Laurent Callot, and Tim Januschowski. Neural forecasting: Introduction and literature overview,
2020.
Jasmin Bogatinovski, Sasho Nedelkoski, Alexander Acker, Florian Schmidt, Thorsten Wittkopp,
Soeren Becker, Jorge Cardoso, and Odej Kao. Artiﬁcial intelligence for it operations (aiops)
workshop white paper, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning,
pp. 1597–1607. PMLR, 2020.
Maximilian Christ, Andreas W. Kempa-Liehr, and Michael Feindt. Distributed and parallel time
series feature extraction for industrial big data applications, 2017.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive
self-supervised learning for language understanding, 2020.
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation
learning for multivariate time series. arXiv preprint arXiv:1901.10738, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc
¸ois
Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net-
works. The journal of machine learning research, 17(1):2096–2030, 2016.
Arthur Le Guennec, Simon Malinowski, and Romain Tavenard.
Data augmentation for time
series classiﬁcation using convolutional neural networks, 2016.
URL https://halshs.
archives-ouvertes.fr/halshs-01357973/.
David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016.
Stefan Haufe, Guido Nolte, Klaus-Robert Mueller, and Nicole Kraemer. Sparse causal discovery in
multivariate time series, 2009.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for
unsupervised visual representation learning, 2020.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia
Makedon. A survey on contrastive self-supervised learning, 2021.
Christian Krupitzer, Tim Wagenhals, Marwin Z¨
uﬂe, Veronika Lesch, Dominik Sch¨
afer, Amin
Mozaffarin, Janick Edinger, Christian Becker, and Samuel Kounev. A survey on predictive main-
tenance for industry 4.0. arXiv preprint arXiv:2002.08224, 2020.
10
Under review as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori-
cut. Albert: A lite bert for self-supervised learning of language representations, 2020.
Yangguang Li, Zhen Ming (Jack) Jiang, Heng Li, Ahmed E. Hassan, Cheng He, Ruirui Huang,
Zhengda Zeng, Mian Wang, and Pinan Chen. Predicting node failures in an ultra-large-scale cloud
computing platform: An aiops solution. ACM Trans. Softw. Eng. Methodol., 29(2), April 2020.
ISSN 1049-331X. doi: 10.1145/3385187. URL https://doi.org/10.1145/3385187.
Bj¨
orn Lohrmann and Odej Kao. Processing smart meter data streams in the cloud. In 2011 2nd
IEEE PES International Conference and Exhibition on Innovative Smart Grid Technologies, pp.
1–8, 2011. doi: 10.1109/ISGTEurope.2011.6162747.
Bin Lu, Yaoyu Li, Xin Wu, and Zhongzhou Yang. A review of recent advances in wind turbine
condition monitoring and fault diagnosis. In 2009 IEEE Power Electronics and Machines in Wind
Applications, pp. 1–7, 2009. doi: 10.1109/PEMWA.2009.5208325.
Carl H Lubba, Sarab S Sethi, Philip Knaute, Simon R Schultz, Ben D Fulcher, and Nick S Jones.
catch22: Canonical time-series characteristics. Data Mining and Knowledge Discovery, 33(6):
1821–1852, 2019.
Spyros Makridakis and Evangelos Spiliotis. The m5 competition and the future of human expertise
in forecasting. Foresight: The International Journal of Applied Forecasting, (60):33–37, 2021.
S.G. Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 11(7):674–693, 1989. doi: 10.1109/
34.192463.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa-
tions of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling,
Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Sys-
tems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.
cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf.
Sasho Nedelkoski, Jorge Cardoso, and Odej Kao. Anomaly detection from system tracing data using
multimodal deep learning. In 2019 IEEE 12th International Conference on Cloud Computing
(CLOUD), pp. 179–186, 2019. doi: 10.1109/CLOUD.2019.00038.
Huida Qiu, Yan Liu, Niranjan A. Subrahmanya, and Weichang Li. Granger causality for time-series
anomaly detection.
In Mohammed Javeed Zaki, Arno Siebes, Jeffrey Xu Yu, Bart Goethals,
Geoffrey I. Webb, and Xindong Wu (eds.), 12th IEEE International Conference on Data Mining,
ICDM 2012, Brussels, Belgium, December 10-13, 2012, pp. 1074–1079. IEEE Computer Society,
2012. doi: 10.1109/ICDM.2012.73. URL https://doi.org/10.1109/ICDM.2012.73.
Juan Qiu, Qingfeng Du, Kanglin Yin, Shuang-Li Zhang, and Chongshu Qian. A causality mining
and knowledge graph based method of root cause diagnosis for performance anomaly in cloud
applications. Applied Sciences, 10(6), 2020. ISSN 2076-3417. doi: 10.3390/app10062166. URL
https://www.mdpi.com/2076-3417/10/6/2166.
Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. The
effectiveness of discretization in forecasting: An empirical study on neural time series models,
2020.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic
forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2019.
Bernhard Sch¨
olkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner,
Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning, 2021.
Kihyuk
Sohn.
Improved
deep
metric
learning
with
multi-class
n-pair
loss
objec-
tive.
In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Ad-
vances
in
Neural
Information
Processing
Systems,
volume
29.
Curran
Associates,
Inc.,
2016.
URL
https://proceedings.neurips.cc/paper/2016/file/
6b180037abbebea991d8b1232f8a8ca9-Paper.pdf.
11
Under review as a conference paper at ICLR 2022
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pﬁster. Learning and evaluating
representations for deep one-class classiﬁcation, 2021.
Evangelos Theodorou, Shengjie Wang, Yanfei Kang, Evangelos Spiliotis, Spyros Makridakis, and
Vassilios Assimakopoulos. Exploring the representativeness of the m5 competition data, 2021.
Terry T. Um, Franz M. J. Pﬁster, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban
Fietzek, and Dana Kuli´
c. Data augmentation of wearable sensor data for parkinson’s disease
monitoring using convolutional neural networks. Proceedings of the 19th ACM International
Conference on Multimodal Interaction, Nov 2017. doi: 10.1145/3136755.3136817. URL http:
//dx.doi.org/10.1145/3136755.3136817.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic-
tive coding, 2019.
Li Wu, Jasmin Bogatinovski, Sasho Nedelkoski, Johan Tordsson, and Odej Kao. Performance Di-
agnosis in Cloud Microservices using Deep Learning. In AIOPS 2020 - International Workshop
on Artiﬁcial Intelligence for IT Operations, Dubai, United Arab Emirates, December 2020. URL
https://hal.inria.fr/hal-02948735.
Renjie Wu and Eamonn Keogh. Current time series anomaly detection benchmarks are ﬂawed and
are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering,
pp. 1–1, 2021. ISSN 2326-3865. doi: 10.1109/tkde.2021.3112126. URL http://dx.doi.
org/10.1109/TKDE.2021.3112126.
Jure Zbontar, Li Jing, Ishan Misra, Yann Lecun, and Stephane Deny.
Barlow twins: Self-
supervised learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Pro-
ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceed-
ings of Machine Learning Research, pp. 12310–12320. PMLR, 18–24 Jul 2021. URL https:
//proceedings.mlr.press/v139/zbontar21a.html.
12
