# REINFORCEMENT LEARNING STATE ESTIMATION
## FOR HIGH-DIMENSIONAL NONLINEAR SYSTEMS

**Anonymous authors**
Paper under double-blind review

ABSTRACT

In high-dimensional nonlinear systems such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) of the
dynamics. However, ROMs are prone to large errors, which negatively affects
the performance of the estimator. Here, we introduce the reinforcement learning
reduced-order estimator (RL-ROE), a ROM-based estimator in which the data assimilation feedback term is given by a nonlinear stochastic policy trained through
reinforcement learning. The flexibility of the nonlinear policy enables the RLROE to compensate for errors of the ROM, while still taking advantage of the
imperfect knowledge of the dynamics. We show that the trained RL-ROE is able
to outperform a Kalman filter designed using the same ROM, and displays robust
estimation performance with respect to different reference trajectories and initial
state estimates.

1 INTRODUCTION

Active control of turbulent flows has the potential to cut down emissions across a range of industries
through drag reduction in aircrafts and ships or improved efficiency of heating and air-conditioning
systems, among many other examples (Brunton & Noack, 2015). But real-time feedback control
requires inferring the state of the system from sparse measurements using an algorithm called a state
estimator, which typically relies on a model for the underlying dynamics (Simon, 2006). Among
state estimators, the Kalman filter is by far the most well-known thanks to its optimality for linear
systems, which has led to its widespread use in numerous applications (Kalman, 1960; Zarchan &
Musoff, 2015). However, continuous systems such as fluid flows are governed by partial differential equations (PDEs) which, when discretized, yield high-dimensional and oftentimes nonlinear
dynamical models with hundreds or thousands of state variables. These high-dimensional models
are too expensive to integrate with common state estimation techniques, including the Kalman filter
or its numerous extensions. Thus, state estimators are instead designed based on a reduced-order
model (ROM) of the underlying dynamics (Barbagallo et al., 2009; Rowley & Dawson, 2017).

A big challenge is that ROMs provide a simplified and imperfect description of the dynamics, which
negatively affects the performance of the state estimator. One potential solution is to improve the
accuracy of the ROM itself through the inclusion of additional closure terms (Ahmed et al., 2021).
In this paper, we leave the ROM untouched and instead propose a new design paradigm for the
estimator itself, which we call a reinforcement-learning reduced-order estimator (RL-ROE). The
RL-ROE is constructed from the ROM in an analogous way to a Kalman filter, with the crucial
difference that the linear filter gain function is replaced by a nonlinear stochastic policy trained
through reinforcement learning (RL). The flexibility of the nonlinear policy enables the RL-ROE to
compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the
dynamics. We describe how we frame the problem as a stationary Markov decision process in order
to enable RL training, which is non-trivial since the RL-ROE must be able to estimate time-varying
states. Finally, we show that the trained RL-ROE is able to outperform a Kalman filter designed
using the same ROM, and displays robust estimation performance with respect to different reference
trajectories and initial state estimates. The RL-ROE is the first application of reinforcement learning
to state estimation for high-dimensional systems.


-----

2 PROBLEM FORMULATION

2.1 SETUP

Consider the discrete-time nonlinear system given by

**_zk+1 = f_** (zk), (1a)
**_yk = Czk,_** (1b)

whereis a time-invariant nonlinear map from current to next state, and zk ∈ R[n] and yk ∈ R[p] are respectively the state and measurement at time C ∈ R[p][×][n] is a linear map from k, f : R[n] _→_ R[n]
state to measurement. In this study, we assume that the dynamics given in (1) are obtained from the
numerical discretization of a nonlinear partial differential equation (PDE), which typically requires
a large number n of state dimensions. Note that we do not account for exogenous control inputs to
the system, which will be studied in future extensions of the present work.

2.2 REDUCED-ORDER MODEL

Because the high dimensionality of (1) makes online prediction and control impractical, it is instead
customary to formulate a reduced-order model (ROM) of the dynamics (Rowley & Dawson, 2017).
defining anFirst, one chooses a suitable linearly independent set of modes r-dimensional subspace of R[n] in which most of the dynamics is assumed to take place. {u1, . . ., ur}, where ui ∈ R[n],
Stacking these modes as columns of a matrixthe reduced-order statea ROM for the dynamics of xk ∈ xRk, which is vastly cheaper to evolve than (1) when[r] represents the coordinates of U ∈ R[n][×][r], one can then express zk in the subspace. Finally, one finds r zk ≃nUx. _k, where_
_≪_

There exist various ways to find an appropriate set of modes U and corresponding ROM for the
dynamics of xk (Taira et al., 2017). In this work, we employ the Dynamic Mode Decomposition
(DMD), a purely data-driven algorithm that has found wide applications in fields ranging from fluid
dynamics to neuroscience (Schmid, 2010; Kutz et al., 2016). Starting with a collection of snapshots
**_Z =_** **_z0, . . ., zm_** collected along a trajectory of (1a), the DMD seeks a best-fit linear model of the
_{_ _}_
the orthogonality ofthedynamics in the form of a matrix r leading principal component analysis (PCA) modes of U then yield a linear discrete-time ROM of the form A ∈ R[n][×][n] such that zk+1 ≃ Z. The transformationAzk, and computes the modes zk ≃ **_Ux Uk and as_**

**_xk+1 = Arxk + wk,_** (2a)
**_yk = Crxk + vk,_** (2b)

where Ar = U [T]AU ∈ R[r][×][r] and Cr = CU ∈ R[p][×][r] are the reduced-order state-transition and observation models, respectively. In order to account for the neglected PCA modes of Z as well as the
unmodeled dynamics incurred by the linear approximation zk+1 **_Azk, we add (unknown) non-_**
Gaussian process noise wk and observation noise vk. Additional details regarding the calculation ≃
of Ar and U are provided in Appendix A.

2.3 REDUCED-ORDER ESTIMATOR

This paper uses reinforcement learning (RL) to solve the following estimation problem: given a sequence of measurements **_y0,_** _, yk_ from a reference trajectory **_z0,_** _, zk_ of (1) and knowing
_{_ _· · ·_ _}_ _{_ _· · ·_ _}_
the ROM (2) defined by Ar, Cr and U, we want to estimate the high-dimensional state zk at current
time k. To this effect, we design a reduced-order estimator (ROE) of the form

**_xˆk = Ar ˆxk_** 1 + ak, (3a)
_−_
**_ak ∼_** **_πθ( · |yk, ˆxk−1),_** (3b)

wherestochastic policy ˆxk is an estimate of the reduced-order state πθ which depends on the current measurement xk, and ak ∈ ykR and the previous state estimate[r] is an action sampled from a
**_xˆk−1. The subscript θ denotes the set of parameters that defines the stochastic policy, whose goal_**
is to minimize the mean square errorreduced-order state estimates. Here, ˆz Ek =[z Uk − ˆxzkˆ denotes the high-dimensional state estimate recon-k] over a range of reference trajectories and initial
structed from ˆxk. A Kalman filter is a special case of such an estimator, for which the action in (3b)
is given by
**_ak = Kk(yk_** **_CrAr ˆxk_** 1), (4)
_−_ _−_


-----

withtransition and observation models are known exactly, its performance suffers in the presence of Kk ∈ R[r][×][p] the optimal Kalman gain. Although the Kalman filter is optimal when the stateunmodeled dynamics. In our case, such model errors are unavoidable due to the ROM (2) being
an inherent approximation of the high-dimensional dynamics (1), which motivates our adoption of
the more general form (3b). This form retains the dependence of ak on yk and ˆxk 1 but is more
_−_
flexible thanks to the nonlinearity of the stochastic policy πθ, which we train with deep RL in an
offline stage. The stochasticity of πθ forces the RL algorithm to explore different actions during
the training process, in order to find eventually an optimalfor various reference trajectories and initial estimates. We call the estimator constructed and trained θ[∗] such that E[zk − **_zˆk] is minimized_**
through this process an RL-trained ROE, or RL-ROE for short.

Thus, the methodology we propose consists of two steps. In a first offline stage, a ROM of the form
(2) is obtained using high-dimensional snapshots zk from a single trajectory of (1). The RL-ROE (3)
is then constructed based on this ROM, and its policy πθ is trained using high-dimensional snapshots
**_zk from multiple reference trajectories of (1). Finally, the trained RL-ROE is deployed online to_**
track a reference trajectory of (1). In the online stage, the RL-ROE only requires measurements yk
from the reference trajectory, and gives an estimate ˆzk = U ˆxk for the high-dimensional state.

In summary, our contributions in this paper are two-fold:

1. We propose RL-ROE, a reduced-order state estimator for high-dimensional nonlinear systems. The RL-ROE takes the form (3), which combines two unique features: first, the state
transition model Ar is a ROM of the high-dimensional dynamics; second, the term ak that
assimilates measurements is sampled from a stochastic policy πθ trained with RL. The
training procedure for πθ, which involves a non-trivial reformulation of the time-varying
tracking problem as a stationary Markov decision process, is described in Section 4.

2. The performance of the RL-ROE is compared in Section 5 with that of KF-ROE, a Kalman
filter constructed from the same ROM. The comparison is performed in the context of the
Burgers equation using a range of reference trajectories and initial state estimates.

3 RELATED WORK

Previous studies have already proposed designing state estimators using policies trained through
reinforcement learning. Morimoto & Doya (2007) introduced an estimator of the form ˆxk =
**_fthe state-dependent filter gain matrix(ˆxk−1) + L(ˆxk−1)(yk−1 −_** **_C ˆxk− L1)(ˆ, wherexk−1) is defined using Gaussian basis functions whose pa- f_** (·) is the state-transition model of the system, and
rameters are learned through a variant of vanilla policy gradient. Their reward function, however,
was calculated using the measurement error instead of the state estimate error, potentially limiting the performance of the trained estimator. Hu et al. (2020) proposed an estimator of the form
**_xˆk = f_** (ˆxk 1) + L(xk **_xˆk)(yk_** **_Cf_** (ˆxk 1)), where L(xk **_xˆk) is approximated by neural_**
networks trained with a modified Soft-Actor Critic algorithm (Haarnoja et al., 2018). Although they− _−_ _−_ _−_ _−_
derived convergence properties for the estimate error, the dependence of the filter gain L(xk **_xˆk)_**
on the reference state xk limits its practical application. _−_

A major difference between these past studies and our work is that they do not construct a ROM of
the dynamics and only consider low-dimensional systems with four state variables at most, in comparison with the hundred or more state dimensions that our RL-ROE can handle. Therefore, RL-ROE
represents the first application of reinforcement learning to state estimation for high-dimensional
systems, which makes it applicable to systems governed by PDEs such as fluid flows.

4 TRAINING METHODOLOGY

In this section, we describe the offline training process for the policy πθ in the RL-ROE (3). In
order to train πθ with reinforcement learning, we need to formulate the problem as a stationary
Markov decision process (MDP). However, this is no trivial task given that the aim of the policy is
to minimize the error between the state estimate ˆzk = U ˆxk and a time-dependent reference state
**_zk. At first sight, such trajectory tracking problem requires a time-dependent reward function and,_**
therefore, a time-varying MDP.


-----

To be able to use off-the-shelf RL algorithms, we introduce a trick to translate this time-varying
MDP to an equivalent, extended stationary MDP. Indeed, we show hereafter that the problem can
be framed as a stationary MDP by including zk into our definition of the MDP’s state. Letting
**_sk = (zk, ˆxk_** 1) R[n][+][r] denote an augmented state at time k, we can define an MDP consisting of
_−_ _∈_
the tuple (S, A, P, R), where S = R[n][+][r] is the augmented state space, A ⊂ R[r] is the action space,
( **_sk, ak) is a transition probability, and_** (sk, ak, sk+1) is a reward function. At each time step
_P_ _·|_ _R_
expressed ask, the agent selects an action ak ∈A according to the policy πθ defined in (3b), which can be
**_ak_** **_πθ(_** **_ok),_** (5)
_∼_ _· |_
where ok = (yk, ˆxk 1) = (Czk, ˆxk 1) is a partial observation of the current state sk. The state
_−_ _−_
**_sk+1 = (zk+1, ˆxk) at the next time step is then obtained from equations (1a) and (3a) as_**

**_sk+1 = (f_** (zk), Ar ˆxk 1 + ak), (6)
_−_

which defines the transition model sk+1 ( **_sk, ak). Finally, the agent receives the reward_**
_∼P_ _·|_

_rk = R(sk, ak, sk+1) = −(zk −_ **_U ˆxk)[T]Q(zk −_** **_U ˆxk) −_** **_a[T]k_** **_[Ra][k][,]_** (7)

where Q ∈ R[n][×][n] and R ∈ R[r][×][r] are positive semidefinite and positive definite matrices, respectively. The first term in the reward function R penalizes the difference between the high-dimensional
state estimate ˆzk = U ˆxk and the reference zk, which is only partially observed by the agent. The
second term favors smaller values for the action ak; such regularization leads to more robust estimation performance in the presence of noise during online deployment of the RL-ROE, as we will see
later. Unless indicated otherwise, we will consider Q = I and R = I. Thanks to the incorporation
of zk into sk, the reward function (7) has no explicit time dependence and the MDP is therefore
stationary.

The goal of the RL training process is then to find the optimal policy parameters

**_θ[∗]_** = arg max E (8)
**_θ_** _τ_ **_πθ[[][R][(][τ]_** [)]][,]
_∼_

where the expectation is over trajectories τ = (s1, a1, s2, a2, . . . ), and R(τ ) is the finite-horizon
undiscounted return


_rk,_ (9)

_k=1_

X


_R(τ_ ) =


with the integer K denoting the length of each training trajectory. Contrary to conventional RL
notation, trajectories here start at time k = 1. Indeed, the environment is initialized at time k = 0
according to the distributions

**_z0_** **_pz0_** ( ), (10a)
_∼_ _·_
**_xˆ0 ∼_** **_pxˆ0_** [(][·][)][,] (10b)

from which the augmented state s1 = (z1, ˆx0) = (f (z0), ˆx0) follows immediately. Thus, s1
constitutes the start of the trajectory of agent-environment interactions. This sequence of operations
mirrors that of a Kalman filter: the calculation of the current state estimate uses the previous state
estimate as well as the current measurement, and therefore begins from the second time step, which
is k = 1 in our case.

To find the optimal policy parameters θ[∗], we employ the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017), which belongs to the class of policy gradient methods (Sutton et al.,
2000). PPO alternates between sampling data by computing a set of trajectories _τ1, τ2, τ3, . . ._ us_{_ _}_
ing the most recent version of the policy, and updating the policy parameters θ in a way that increases
the probability of actions that led to higher rewards during the sampling phase. The policy πθ encodes a diagonal Gaussian distribution described by a neural network that maps from observation
to mean action, µθ′ (ok), together with a vector of standard deviations σ, so that θ = **_θ[′], σ_** . We
_{_ _}_
utilize the Stable Baselines3 (SB3) implementation of PPO (Raffin et al., 2019) and define our MDP
as a custom environment in OpenAI Gym (Brockman et al., 2016). Besides the discount factor γ,
all results to follow are obtained with the default PPO hyperparameters in SB3, which demonstrates
the robustness of our approach with respect to the RL hyperparameters.

_Remark 1. The offline RL training phase described in this section (to find the optimal parameters_
**_θ[∗]) requires knowledge of the high-dimensional state zk from several reference trajectories of (1)._**


-----

During online deployment, however, the RL-ROE (3) only relies on measurements yk, since the
trained policy πθ∗ is conditioned on yk and the previous reduced state estimate ˆxk 1.
_−_

_Remark 2. Similar to any learning method, the trained RL-ROE is expected to perform well for_
reference trajectories and initial estimates sampled from the same distributions used during the offline training phase. Thus, the choice of the distributions for z0 and ˆx0 in (10) is critical since it
defines the range of initial reference states and initial estimates that the trained RL-ROE will be able
to handle during online deployment. This is similar to the state of the art in model-based observer
theory, which only guarantees convergence of nonlinear observers locally, i.e., near the true initial
states (Benosman & Borggaard, 2021).

5 RESULTS

We evaluate our proposed RL-ROE using simulations of the Burgers equation, a prototypical nonlinear hyperbolic PDE which takes the form

_∂u_

(11)

_∂t_ [+][ u∂u]∂x _[−]_ _[ν ∂]∂x[2][u][2][ =][ f]_ [(][x, t][)][,]

where u(x, t) is the velocity at position x ∈ [0, L] and time t ∈ [0, T ], f (x, t) is a distributed
time-dependent forcing, and the scalar parameter ν acts like a viscosity. The boundary conditions
are periodic and the initial condition u(x, 0) = u0(x) will be specified later. We choose L = 1,
_T = 10, and ν = 0.01._

We discretize the Burgers equation (11) with a spectral method using n = 256 Fourier modes, and
integrate forward in time using a fifth-order Runge-Kutta method. Solution snapshots are saved
every ∆t = 0.05 time unit, yielding the discrete high-dimensional state

**_zk = [u(x0, k∆t), u(x1, k∆t), . . ., u(xn_** 1, k∆t)][T] R[n], (12)
_−_ _∈_

where xi = iL/n are the collocation points, and k = 0, . . ., T/∆t is the time index. Further, we
place p = 8 sensors at equidistant locations, providing the sparse measurement vector

**_yk = [u(¯x0, k∆t), u(¯x1, k∆t), . . ., u(¯xp_** 1, k∆t)][T] R[p], (13)
_−_ _∈_

where ¯xi = iL/p are the sensor locations. Thus, the state zk and measurement yk are governed by
a high-dimensional discrete-time nonlinear system of the form given in equation (1).

We then follow the procedure outlined in Section 2 to construct offline a ROM and corresponding
ROE, which we train offline using PPO following the methodology presented in Section 4. Finally,
the trained RL-ROE is deployed online and compared against a Kalman filter under various initial conditions for the reference trajectory as well as the state estimate. Specifically, we adopt the
following steps:

1. Construction of the ROM. Starting from the pulse-shaped initial condition

1
_u0(x) =_ (14)

cosh(20(x _L/2))_ _[,]_
_−_

we calculate one solution trajectory of (1) for t ∈ [0, T/2] = [0, 5], and we denote as
**_Z_** [DMD] = **_z0[DMD], . . ., zm[DMD]_** the resulting solution snapshots at times k = 0, . . ., m =
_{_ _}_
_T/2∆t. The DMD is then applied to these snapshots, yielding a ROM of the form (2)_
defined by matrices Ar, Cr and U . The ROM governs the evolution of a reduced-order
statesubspace xk ≃ (which corresponds in this case to 99.99% of the energy of the snapshotsU [T]zk ∈ R[r]. We pick r = 15 for the dimensionality of the reduced-order Z [DMD]
being included in the modes U ), giving the ROM a significant computational advantage
compared with the high-dimensional system (1) of size n = 256.


2. Training of the RL-ROE. We train the stochastic policy πθ of the RL-ROE (3) using PPO,
as described in Section 4. In order for the resulting estimator to perform well under various
reference trajectories and initial estimates, we initialize each trajectory of the MDP during
the offline training process with

**_z0 = αz0[DMD],_** (15a)

**_xˆ0 = U_** [T]z0[DMD] + β, (15b)


-----

|(b)|Col2|
|---|---|
|||
|||
|M ining ta||
|||


|-shaped initial c where α ∼ in (10). In or scaled do|ondition (14). U(0.5, 2) ∈R and β ∼N(0, 0.1I) ∈Rr, other words, the reference trajectories are wn version of the pulse defined in (14), a|which defines the distri initialized as a random nd the reduced-order s|butions given ly scaled up tate estimate|
|---|---|---|---|


and feed measurements yk to the RL-ROE and KF-ROE. Their tracking performance of

(a) (b)

ROM
training
data

Figure 1: (a) Training trajectory of the unforced Burgers equation used for the construction of the(a) (b) (c)
ROM with DMD. The discrete-time snapshots in t ∈ [0, 5] constitute the training data fed to the
DMD. The tracking performance of the RL-ROE will be evaluated for t ∈ [0, 10] on trajectories
with similar dynamical behavior as this one. (b) Zooming into t ∈ [0, 1] reveals the fast decay of the
pulse-shaped initial condition (14).

where α ∼U(0.5, 2) ∈ R and β ∼N (0, 0.1I) ∈ R[r], which defines the distributions given
in (10). In other words, the reference trajectories are initialized as a randomly scaled up
or scaled down version of the pulse defined in (14), and the reduced-order state estimate
is initialized as a reduced-order projection of that pulse, polluted with additive Gaussian

(d) (e) (f)

noise. During training, we limit each trajectory to the same time window t ∈ [0, T/2] that
was used in constructing the ROM – that is, we pick K = T/2∆t in the finite-horizon
return (9). We end the training when the return no longer increases on average.

3. Evaluation of the RL-ROE. We evaluate the trained RL-ROE against a time-dependent
Kalman filter constructed from the same ROM, which we refer to as KF-ROE. The KFROE is given by equations (3a) and (4), with the calculation of the time-varying Kalman
gain detailed in Appendix B. The RL-ROE and KF-ROE are compared online based on
three specific reference trajectories initialized from (15a) using α = 0.5, 1, and 2. For each
reference trajectory, we consider 20 different initial state estimates sampled from (15b),
and feed measurements

the reference state zk is then evaluated and compared over the full time window t ∈ [0, T ].

We carry out the above procedure for two different choices of the forcing f (x, t), each leading
to qualitatively different dynamics: first, the unforced case f (x, t) = 0, and second, a sinusoidal
forcing of the form f (x, t) = sin(ωt − _kx) with ω = π and k = 2π/L. We emphasize that the_
RL-ROE is not (yet) designed to account for different exogenous control inputs to the system. Here,
the forcing f (x, t) is simply considered as a way to generate more complex solutions by altering
the dynamics of the PDE. Thus, the two choices for f (x, t) will be treated separately in the results
below, and we will construct and train a different RL-ROE in each case.

5.1 UNFORCED CASE

Beginning with the unforced case f (x, t) = 0, we first construct the ROM on which the RL-ROE
and KF-ROE will be based. Figure 1 depicts the solution of the discretized Burgers equation to the
initial condition (14). The solution snapshots belonging to the time window t ∈ [0, T/2] = [0, 5]
constitute the training data Z [DMD] fed to the DMD algorithm, which yields the ROM (2). Further,
this figure illustrates the type of dynamics exhibited by the Burgers equation in the unforced regime:
a quickly decaying and widening pulse for t ≤ 2, followed by a slow convergence to a uniform
steady state for t > 2.

Before training and evaluating the RL-ROE, we quantify the accuracy of the ROM itself – that is,
given knowledge of the true initial condition. For this purpose, we consider three initial conditions
defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding reference solution (using
the discretized Burgers equation) as well as the ROM solution. The results, reported in Appendix
C, show that the model error is very low for the initial condition α = 1 and time window t ∈ [0, 5],


-----

(a) (b) (c)

(d) (e) (f)

Figure 2: Accuracy of the RL-ROE for the unforced case. (a,b,c) L2 error of the RL-ROE and KFROE with respect to specific reference trajectories of the Burgers equation, initialized from (15a)
using α = 0.5, 1, 2. The RL-ROE and KF-ROE are evaluated using 20 different initial estimates
sampled from (15b). The curves and shaded area indicate the mean and standard deviation of the
error, respectively. (Appendix E shows the same data for t ∈ [0, 2].) (d,e,f) Phase space trajectories
of the RL-ROE and KF-ROE predictions for 5 initial estimates, and the reference solution.

since this case corresponds to the same solution trajectory used for constructing the ROM. On the
other hand, the model error increases for larger times or other values of α.

The RL-ROE is then trained using PPO following the methodology outlined in Section 4, together
with the distributions (15) for trajectory initialization. The RL hyperparameters and learning curve
displaying the performance improvement of the RL-ROE during the training process are reported in
Appendix D. The trained RL-ROE is now compared with the KF-ROE, a Kalman filter constructed
from the same ROM. Figures 2(a,b,c) show the L2 error of the RL-ROE and KF-ROE with respect
to specific reference trajectories of the Burgers equation, initialized from (15a) using α = 0.5, 1,
2. For each reference trajectory, we consider 20 different initial state estimates sampled from (15b).
The curves and shaded area reported in Figures 2(a,b,c) indicate the mean and standard deviation of
the error, defined at time stepRL-ROE or KF-ROE, and zk k is the high-dimensional reference solution. (Appendix E shows the by |U ˆxk − **_zk|, where ˆxk is the reduced-order estimate given by the_**
same data for t ∈ [0, 2].) Figures 2(d,e,f) show the trajectories of the reference, RL-ROE and KFROE solutions in a two-dimensional slice of phase space spanned by the second and third columns[1]
of U – in other words, the time history of the second and third components of U [T]zk and ˆxk.

A few important observations emerge from Figure 2. First, when the ROM suffers from large model
errors due to initial conditions deviations, as is the case for α = 0.5 and α = 2, the RL-ROE is able
to outperform the KF-ROE. In the time window t ≤ 2 during which most of the transient dynamics
take place, the RL-ROE displays up to an order of magnitude lower error than the KF-ROE. Second,
when the ROM is very accurate, as is the case for α = 1, the KF-ROE gives lower error for most of
the time duration. Even then, however, Figure 2(e) shows that the RL-ROE converges faster to the
reference trajectory. Last, the RL-ROE manages to keep the error at a low level in the time window
_t ∈_ [5, 10], despite the fact that it was trained using trajectories that end at t = 5.

1Since the columns of U approximate the PCA modes of the training snapshots Z DMD without centering, the
first column will be dominated by the mean of the data. Thus, we display the trajectory coordinates associated
with the second and third columns, which capture the largest amount of variance within the data.


-----

|h similar dynamica|al behavior as this on|ne. (b) Zoomi|∈ ing into t ∈[0, 1] reveals the|fast decay of th|he|
|---|---|---|---|---|---|
|se-shaped initial c e Burgers equation y both the RL-RO|ondition (14). converges to a uni|form steady s|∈ tate in this unforced case, w|hich can explai|n -|
||E and KF-ROE display low erro||rs at larger times. We now investigate a fun|||


transient phase with for t ≤ 2 during which the pulse changes shape into a skewed wave, followed

(a) (b)

ROM
training
data

Figure 3: (a) Training trajectory of the forced Burgers equation used for the construction of the(a) (b) (c)
ROM with DMD. The discrete-time snapshots in t ∈ [0, 5] constitute the training data fed to the
DMD. The tracking performance of the RL-ROE will be evaluated for t ∈ [0, 10] on trajectories
with similar dynamical behavior as this one. (b) Zooming into t ∈ [0, 1] reveals the fast decay of the
pulse-shaped initial condition (14).

The Burgers equation converges to a uniform steady state in this unforced case, which can explain
why both the RL-ROE and KF-ROE display low errors at larger times. We now investigate a fundamentally different dynamical regime in the next section by adding nonzero forcing to the Burgers
equation, which prevents the solution from settling on a steady state.(d) (e) (f)

5.2 FORCED CASE

We now consider the forced case f (x, t) = sin(ωt − _kx), with ω = π and k = 2π/L. Repeating_
the steps of the previous section, we first construct the ROM on which the RL-ROE and KF-ROE
will be based. Figure 3 depicts the solution of the discretized Burgers equation with forcing to the
initial condition (14). The solution snapshots belonging to the time window t ∈ [0, T/2] = [0, 5]
constitute the training data Z [DMD] fed to the DMD algorithm for constructing the ROM (2). This
figure illustrates the type of dynamics exhibited by the Burgers equation in the forced regime: a
transient phase with for during which the pulse changes shape into a skewed wave, followed

by a limit cycle regime with the wave traveling at constant velocity for t > 2.

As in the unforced case, we first quantify the accuracy of the ROM itself. For this purpose, we consider three initial conditions defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding
reference solution (using the discretized Burgers equation with forcing) as well as the ROM solution. This time, the results reported in Appendix C show that the model error is higher for all cases,
which reflects the more complicated nature of the dynamics in this forced regime.

The RL-ROE is then trained using PPO following the methodology outlined in Section 4, together
with the distributions (15) for trajectory initialization. The RL hyperparameters and learning curve
displaying the performance improvement of the RL-ROE during the training process are reported in
Appendix D. The trained RL-ROE is now compared with the KF-ROE, a Kalman filter constructed
from the same ROM. Figures 4(a,b,c) show the L2 error of the RL-ROE and KF-ROE with respect
to specific reference trajectories of the Burgers equation, initialized from (15a) using α = 0.5, 1,
2. As before, we consider 20 different initial state estimates sampled from (15b) for each reference
trajectory. The curves and the surrounding shade reported in Figures 4(a,b,c) indicate the mean
and standard deviation of the resulting error, respectively. (Appendix E shows the same data for
_t ∈_ [0, 2].) Figures 4(d,e,f) show the trajectories of the reference, RL-ROE and KF-ROE solutions
in a two-dimensional slice of phase space spanned by the second and third columns of U – in other
words, the time history of the second and third components of U [T]zk and ˆxk.

The RL-ROE outperforms the KF-ROE in both the initial transient phase as well as the later limit
cycle regime, for all three reference trajectories. This is consistent with our previous observation
that the RL-ROE has an advantage over the KF-ROE when the ROM suffers from large model
errors. Moreover, the estimation performance of the RL-ROE remains stable in the time window
_t ∈_ [5, 10], even though trajectories stop at t = 5 during the training process. All together, these


-----

(a) (b) (c)

(d) (e) (f)

Figure 4: Accuracy of the RL-ROE for the forced case. (a,b,c) L2 error of the RL-ROE and KF-ROE
with respect to specific reference trajectories of the Burgers equation, initialized from (15a) using
_α = 0.5, 1, 2. The RL-ROE and KF-ROE are evaluated using 20 different initial estimates sampled_
from (15b). The curves and the surrounding shade indicate the mean and standard deviation of the
error, respectively. (Appendix E shows the same data for t ∈ [0, 2].) (d,e,f) Phase space trajectories
of the RL-ROE and KF-ROE predictions for 5 initial estimates, and the reference solution.

results demonstrate the estimation performance of the RL-ROE, and its robustness with respect to
model errors.

We include a number of additional results as appendices. In Appendix F, we show that the regularization term in the reward function (7) leads to better estimation performance in the presence of
observation noise. In Appendix G, we compare our RL-ROE approach with two alternative datadriven estimators formulated as standard supervised learning problems, and we observe that the
RL-ROE is more robust to observation noise and gives smoother predictions.

6 CONCLUSIONS

In this paper, we have introduced the reinforcement learning reduced-order estimator (RL-ROE), a
new methodology for estimating the state of a high-dimensional nonlinear dynamical system. Our
approach follows the standard practice of constructing a computationally inexpensive reduced-order
model (ROM) to approximate the dynamics of the system. The novelty of our contribution lies in the
design, based on this ROM, of a reduced-order estimator (ROE) in which the feedback correction
term is given by a nonlinear stochastic policy trained through reinforcement learning. To be able to
use off-the-shelf RL algorithms, we introduce a trick to translate this trajectory tracking problem,
i.e., time-varying MDP, to an equivalent stationary MDP based on an augmented state. We show
using simulations of the Burgers equation in two very different dynamical regimes that the trained
RL-ROE is able to outperform a Kalman filter designed using the same ROM and displays robust
estimation performance with respect to different reference trajectories and initial state estimates.

This work opens the door to a number of potential future directions. A logical next step is to evaluate
the performance of the RL-ROE on other physical systems, for instance the Navier-Stokes equations
governing the motion of fluid flows. Different types of ROM could also be considered, potentially
leading to improved performance of the RL-ROE.


-----

REFERENCES

Shady E Ahmed, Suraj Pawar, Omer San, Adil Rasheed, Traian Iliescu, and Bernd R Noack. On
closures for reduced order models—a spectrum of first-principle to machine-learned avenues.
_Physics of Fluids, 33(9):091301, 2021._

Alexandre Barbagallo, Denis Sipp, and Peter J Schmid. Closed-loop control of an open cavity flow
using reduced-order models. Journal of Fluid Mechanics, 641:1–50, 2009.

Mouhacine Benosman and Jeff Borggaard. Robust nonlinear state estimation for a class of infinitedimensional systems using reduced-order models. International Journal of Control, 94(5), 2021.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.

Steven L Brunton and Bernd R Noack. Closed-loop turbulence control: Progress and challenges.
_Applied Mechanics Reviews, 67(5), 2015._

Petr Chaupa, Jakub Nov´ak, and Peter Januˇska. Model predictive control using different state
observers,. In Recent Advances in Automatic Control, Information and Communications,
_pages=191–196, year=2013._

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy
maximum entropy deep reinforcement learning with a stochastic actor. In International confer_ence on machine learning, pp. 1861–1870. PMLR, 2018._

Liang Hu, Chengwei Wu, and Wei Pan. Lyapunov-based reinforcement learning state estimator.
_arXiv preprint arXiv:2010.13529, 2020._

R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic
_Engineering, 82(1):35–45, 1960._

J Nathan Kutz, Steven L Brunton, Bingni W Brunton, and Joshua L Proctor. Dynamic mode decom_position: data-driven modeling of complex systems. SIAM, 2016._

Jun Morimoto and Kenji Doya. Reinforcement learning state estimator. Neural computation, 19(3):
730–756, 2007.

Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor[mann. Stable baselines3. https://github.com/DLR-RM/stable-baselines3, 2019.](https://github.com/DLR-RM/stable-baselines3)

Clarence W Rowley and Scott TM Dawson. Model reduction for flow analysis and control. Annual
_Review of Fluid Mechanics, 49:387–417, 2017._

Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid
_mechanics, 656:5–28, 2010._

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Dan Simon. Optimal state estimation: Kalman, H infinity, and nonlinear approaches. John Wiley
& Sons, 2006.

Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient
methods for reinforcement learning with function approximation. In Advances in neural informa_tion processing systems, pp. 1057–1063, 2000._

Kunihiko Taira, Steven L Brunton, Scott TM Dawson, Clarence W Rowley, Tim Colonius, Beverley J McKeon, Oliver T Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence S
Ukeiley. Modal analysis of fluid flows: An overview. Aiaa Journal, 55(12):4013–4041, 2017.

Jonathan H Tu, Clarence W Rowley, Dirk M Luchtenburg, Steven L Brunton, and J Nathan Kutz.
On dynamic mode decomposition: theory and applications. Journal of Computational Dynamics,
1(2):391–421, 2014.

Paul Zarchan and Howard Musoff. Fundamentals of Kalman filtering: a practical approach. Aiaa,
2015.


-----

A DYNAMIC MODE DECOMPOSITION

In this appendix, we describe the DMD algorithm (Schmid, 2010; Tu et al., 2014), which is a popular
data-driven method to extract spatial modes and low-dimensional dynamics from a dataset of highdimensional snapshots. Here, we use the DMD to construct a ROM of the form (2) given a snapshot
sequence Z = **_z0, . . ., zm_** collected along a trajectory of (1a) and an observation model C.
_{_ _}_

Fundamentally, the DMD seeks a best-fit linear model of the dynamics in the form of a matrix
**_A ∈_** R[n][×][n] such that zk+1 ≃ **_Azk. Arranging the snapshots into two time-shifted matrices_**

**_X = {z0, . . ., zm−1},_** **_Y = {z1, . . ., zm},_** (16)

the best-fit linear model is given by A = Y X _[†], where X_ _[†]_ is the pseudoinverse of X. The ROM
is then obtained by projecting the matrices A and C onto a basis U consisting of the r leading
left singular vectors of X, which approximate the r leading PCA modes of Z. Using the truncated
singular value decomposition

**_X = U_** **ΣV** [T] (17)

where U _, V ∈_ R[n][×][r] and Σ ∈ R[r][×][r], the resulting reduced-order state-transition and observation
models are given by

**_Ar = U_** [T]AU = U [T]Y V Σ[−][1], (18a)
**_Cr = CU_** _._ (18b)

Conveniently, the ROM matrices Ar and Cr can be calculated directly from the truncated SVD of
**_X, which avoids forming the large n × n matrix A._**

B KALMAN FILTER

The time-dependent Kalman filter that we use as a benchmark in this paper, KF-ROE, is based on the
same ROM (2) as the RL-ROE, with identical matrices Ar, Cr and U . Similarly to the RL-ROE,
the reduced-order estimate ˆxk is given by equation (3a), from which the high-dimensional estimate
is reconstructed as ˆzk = U ˆxk. However, the KF-ROE differs from the RL-ROE in its definition of
the action ak in (3a), which is instead given by the linear feedback term (4). The calculation of the
optimal Kalman gain Kk in (4) requires the following operations at each time step:

**_Pk[−]_** [=][ A][r][P][k][−][1][A]r[T] [+][ Q][k][,] (19)

**_Sk = CrPk[−][C]r[T]_** [+][ R][k][,] (20)

**_Kk = Pk[−][C]r[T][S]k[−][1][,]_** (21)

**_Pk = (I −_** **_KkCr)Pk[−][,]_** (22)

where Pk[−] [and][ P][k][ are respectively the a priori and a posteriori estimate covariance matrices,][ S][k][ is]
the innovation covariance, and Qk and Rk are respectively the covariance matrices of the process
noise wk and observation noise vk in the ROM (2). Since these noise covariance matrices are
unknown, we choose Qk = Rk = I for all k after verifying empirically that these values yield
the best possible results. At time step k = 0, the a posteriori estimate covariance is initialized
asdistributions (15). P0 = cov(U [T]z0 − **_xˆ0), which can be calculated from the initial reference and estimated state_**

_Remark 3. We are seeking a Kalman-type observer, which in ‘filters domain’ is also known as an_
infinite impulse response filter (IIR). These observers are to be contrasted with the finite impulse
filters (FIR). Indeed, the later are well known to be based on a mapping between n previous samples
of input/output and the desired observed state at the current instant (e.g., Chaupa et al., equations
2,3, and 12), and lead to exact convergence in finite-time, in the noiseless setting. On the other
hand the IIR observer is well known to be based on the last measurement of the output/input only,
e.g., (Chaupa et al., equation 8), and lead to an average asymptotic performance, e.g.,maximum
likelihood estimate in the our case.


-----

(b) (c)

(e) (f)

Figure 5: Accuracy of the ROM for the unforced case. (a,b,c) L2 error of the ROM prediction with
respect to a reference numerical solution of the Burgers equation, both initialized from (15a) using
_α = 0.5, 1, 2. The case α = 1 corresponds to the solution trajectory whose snapshots in t ∈_ [0, 5]
were used to construct the ROM. (d,e,f) Phase space trajectories of the ROM prediction and the
reference solution.

C MODEL ERROR

In this appendix, we quantify the accuracy of the two ROMs utilized in Sections 5.1 and 5.2 for
the unforced and forced Burgers equations, respectively. For this purpose, we consider three initial
conditions defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding reference solution
(using the discretized Burgers equation) as well as the ROM solution. Results pertaining to the
unforced and forced cases are shown in Figures 5 and 6, respectively.

Figures 5(a,b,c) and 6(a,b,c) show the resulting L2 error of the ROM solution, defined at each time
dimensional reference solution. Figures 5(d,e,f) and 6(d,e,f) show the trajectories of the referencestep k by |Uxk − **_zk|, where xk is the reduced-order state given by the ROM and zk is the high-_**
and ROM solutions in a two-dimensional slice of phase space spanned by the second and third
columns of U – in other words, the time history of the second and third components of U [T]zk and
**_xk. The L2 error curves and ROM trajectory curves are colored according to the value of time using_**
the same color scheme, which confirms that most of the dynamics take place during the first two
time units, as already observed in Figure 1 for α = 1.

D TRAINING HYPERPARAMETERS AND LEARNING CURVES

The stochastic policy πθ is trained with PPO using the default hyperparameters from Stable Baselines3, except for the discount factor γ which we choose as 0.75. The mean output of the stochastic
policy and the value function are approximated by two neural networks, each containing two hidden
layers with 64 neurons and tanh activation functions. The training process alternates between sampling data for 20 trajectories (of length 100 timesteps each) and updating the policy. Each policy
update consists of multiple gradients steps through the most recent data using 10 epochs, a minibatch
size of 64 and a learning rate of 0.0003. The policy is trained for a total of one million timesteps,
corresponding to 10000 trajectories. Figure 7 reports the learning curves corresponding to the unforced and forced cases. During training, the policy is tested (with stochasticity switched off) after
each update using 10 separate test trajectories, and is saved if it outperforms the previous best policy.


-----

(b) (c)

(e) (f)

Figure 6: Accuracy of the ROM for the forced case. (a,b,c) L2 error of the ROM prediction with
respect to a reference numerical solution of the Burgers equation, both initialized from (15a) using
_α = 0.5, 1, 2. The case α = 1 corresponds to the solution trajectory whose snapshots in t ∈_ [0, 5]
were used to construct the ROM. (d,e,f) Phase space trajectories of the ROM prediction and the
reference solution.

Finally, the RL-ROE is defined using the latest saved policy upon ending of the training process, and
the stochasticity of the policy is switched off during subsequent evaluation of the RL-ROE.

Figure 7: Learning curves for the stochastic policy in the (a,b) unforced and (c,d) forced cases. The
line and shaded area show the mean and standard deviation of the results over 10 runs, each one
smoothed with a moving average of size 100 episodes.

E ESTIMATION ERROR FOR SHORT TIMES

Figures 8 and 9 show the same data as the first row in Figures 2 and 4, but for t ∈ [0, 2]. Focusing on
this initial transient phase, it becomes clear that the variance of the RL-ROE estimate decays much
faster than the KF-ROE estimate.


-----

(a) (b) (c)

(a) (b)

ROM
training
data

Figure 8: Same as the first row of Figure 2, but shown for t ∈ [0, 2].

(a) (b) (c)

Figure 9: Same as the first row of Figure 4, but shown for t ∈ [0, 2].


F EFFECT OF PENALIZING THE ACTION MAGNITUDE IN THE REWARD

We investigate the effect of penalizing the magnitude of the action ak in the reward function (7). To
this effect, we repeat the experiments of Sections 5.1 and 5.2, this time training the RL-ROE using
different magnitudes R for the matrix R = RI. When evaluating the performance of the trained
RL-ROE, we consider different amounts of Gaussian observation noise added to the measurements
**_yk. The results for the unforced case are shown in Figures 10, 11, and 12 for observation noise of_**
standard deviation σ = 0, 0.1, and 0.3, respectively. The results for the forced case are shown in
Figures 13, 14, and 15 for observation noise of standard deviation σ = 0, 0.1, and 0.3, respectively.
In absence of noise, the highest estimation accuracy is obtained for R = 0, and decreases as R
increases. However, in the presence of noise, Figures 11, 12, and 14, 15, the estimation accuracy
is generally highest for R = 10. This confirms that penalizing the magnitude of the action ak
in the reward function acts as a regularization that allows the RL-ROE to perform better on noisy
measurement data.

G COMPARISON WITH TWO SUPERVISED LEARNING APPROACHES

In this appendix, we compare our proposed RL-ROE with two alternative estimation techniques
trained offline in a supervised learning setting.

The first approach is purely data-driven. Here, the estimator takes the form

**_zˆk = U_** _fDNN(yk, . . ., yk_ _q),_ (23)
_−_

where the nonlinear function fDNN : R[p][×][q] _→_ R[r] is a feed-forward deep neural network (DNN)
taking a given number q of past observations as input. The DNN is then trained by minimizing the
loss given for each data point by the squared estimation errortruth high-dimensional state. The entire data set consists of the same high-dimensional reference ||zˆk − **_zk||[2], where zk is the ground-_**
trajectories used for training the RL-ROE – that is, they are initialized from the distribution (15a)
and are limited to the time window t ∈ [0, T/2] = [0, 5]. This results in a total of 14646 data points
for all trajectories and time indices. The DNN consists of 4 hidden layers of 50 neurons each, and


-----

= 0.5, estimation error

2 4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10

10


10

10


10


10


10


10


= 0.5, trajectories

2 1

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

2

x2


= 2, trajectories

4 2 0

x2


0.0

0.5

1.0

1.5

2.0

2.5


Figure 10: Effect of R on the accuracy of the RL-ROE for the unforced case. Same as Figure 2, but
comparing several RL-ROEs trained using different magnitudes R for the matrix R = RI in the
reward function (7).


= 0.5, estimation error

2 4 6 8

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


10

10


10

10


10

10


10


10


10


= 0.5, trajectories

2 0

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

2 0

x2


= 2, trajectories

2 0

x2


Figure 11: Effect of R on the accuracy of the RL-ROE for the unforced case with moderate noise.
Same as Figure 2, but comparing several RL-ROEs trained using different magnitudes R for the
matrix R = RI in the reward function (7), and with independent Gaussian observation noise of
standard deviation σ = 0.1 added to the measurements yk.


-----

= 0.5, estimation error

2 4 6 8

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


10

10


10


10


10


10


10


= 0.5, trajectories

2 0

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

2 0

x2


= 2, trajectories

5.0 2.5 0.0 2.5

x2


Figure 12: Effect of R on the accuracy of the RL-ROE for the unforced case with large noise. Same
as Figure 2, but comparing several RL-ROEs trained using different magnitudes R for the matrix
**_R = RI in the reward function (7), and with independent Gaussian observation noise of standard_**
deviation σ = 0.3 added to the measurements yk.


= 0.5, estimation error

2 4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10

10


10


10 2

4.0

3.5

3.0

3 2.5

2.0

1.5

1.0


10


10


10


= 0.5, trajectories

5.0 2.5 0.0 2.5

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

0

x2


= 2, trajectories

5 0

x2


2.5

2.0

1.5

1.0

0.5

0.0


Figure 13: Effect of R on the accuracy of the RL-ROE for the forced case. Same as Figure 4, but
comparing several RL-ROEs trained using different magnitudes R for the matrix R = RI in the
reward function (7).


-----

= 0.5, estimation error

2 4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


= 1, estimation error

4 6 8

time


= 2, estimation error

2 4 6 8

time


6 × 10

4 × 10

3 × 10

2 × 10


10


10


10


10


10


= 0.5, trajectories

5.0 2.5 0.0 2.5 5.0

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

0

x2


= 2, trajectories

5 0

x2


10


Figure 14: Effect of R on the accuracy of the RL-ROE for the forced case with moderate noise.
Same as Figure 4, but comparing several RL-ROEs trained using different magnitudes R for the
matrix R = RI in the reward function (7), and with independent Gaussian observation noise of
standard deviation σ = 0.1 added to the measurements yk.


= 0.5, estimation error

2 4 6 8

time


= 1, estimation error

2 4 6 8

time


= 2, estimation error

2 4 6 8

RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

time


2 × 10

10


3 × 10

2 × 10

10


10


6 × 10

4 × 10

3 × 10


6 × 10


10


10


10


= 0.5, trajectories

0

reference
RL-ROE, R=0
RL-ROE, R=1
RL-ROE, R=10

x2


= 1, trajectories

5 0

x2


= 2, trajectories

5 0

x2


10


Figure 15: Effect of R on the accuracy of the RL-ROE for the forced case with large noise. Same
as Figure 4, but comparing several RL-ROEs trained using different magnitudes R for the matrix
**_R = RI in the reward function (7), and with independent Gaussian observation noise of standard_**
deviation σ = 0.3 added to the measurements yk.


-----

is trained using the Adam optimizer over 1000 epochs with minibatch size of 256. The resulting
estimator is denoted DNN in the following results.

The second approach is a hybrid ROM-based and data-driven approach, similarly to our proposed
RL-ROE. Here, the estimator takes the form
**_zˆk = U_** (ˆx[ROM]k + fDNN(ˆx[ROM]k _, yk)),_ (24)
where ˆx[ROM]k is the reduced state generated by the raw ROM (in our case, the DMD model), and
the nonlinear function fDNN : R[r][+][p] _→_ R[r] is a DNN acting as a nonlinear correction to the ROM
prediction. The DNN is then trained by minimizing the loss given for each data point by the squared
set consists of pairwise combinations of high-dimensional reference trajectories and reduced ROMestimation error ||zˆk − **_zk||[2], where zk is the ground-truth high-dimensional state. The entire data_**
trajectories. Similar to the training process for the RL-ROE, the reference and ROM trajectories are
initialized from the distributions (15a) and (15b), and are limited to the time window t ∈ [0, T/2] =

[0, 5]. This results in a total of 305020 data points for all pairs of trajectories and time indices.
The DNN consists of 4 hidden layers of 80 neurons each, and is trained using the Adam optimizer
over 500 epochs with minibatch size of 256. The resulting estimator is denoted ROM+DNN in the
following results.

We compare in Figures 16 and 17 the estimation performance of the RL-ROE with that of the
DNN estimator (23) using q = 3. Since the DNN estimator essentially acts as an interpolation
function, it is more interesting to compare them in the presence of observation noise, which we
model as Gaussian noise of standard deviation σ = 0.1 added to the measurements yk. The RLROE produces more accurate estimates with smoother trajectories; such robustness to measurement
noise is a consequence of the stochasticity inherent to the RL training process. Furthermore, a clear
benefit of the RL-ROE design philosophy is that it yields an accurate dynamic model, since the
imperfect ROM dynamics (in our case given by DMD) are corrected by the RL-trained nonlinear
policy ak **_π, unlike the DNN estimator design which has no inherent dynamics. This will later_**
allow the RL-ROE to be paired with model-based controllers, opening the door to a wide class of ∼
control strategies that are commonly used with Kalman filters.


= 0.5, estimation error

2 4 6 8

RL-ROE
DNN

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10

10


10


10

10


10


10


10


= 0.5, trajectories

2 0

reference
RL-ROE
DNN

x2


= 1, trajectories

2 0

x2


= 2, trajectories

2 0

x2


Figure 16: Accuracy of the DNN estimator for the unforced case with noise. Same as Figure 2, but
comparing the RL-ROE with the DNN estimator, and with independent Gaussian observation noise
of standard deviation σ = 0.1 added to the measurements yk.

We compare in Figures 18 and 19 the estimation performance of the RL-ROE with that of the
ROM+DNN estimator (24). The comparison is also performed in the presence of Gaussian observation noise of standard deviation σ = 0.1 added to the measurements yk. Once more, the RL-ROE


-----

= 0.5, estimation error

2 4 6 8

RL-ROE
DNN

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10


10


10


10


10


10


= 0.5, trajectories

5.0 2.5 0.0 2.5 5.0

reference
RL-ROE
DNN

x2


= 1, trajectories

0

x2


= 2, trajectories

5 0

x2


10


Figure 17: Accuracy of the DNN estimator for the forced case with noise. Same as Figure 4, but
comparing the RL-ROE with the DNN estimator, and with independent Gaussian observation noise
of standard deviation σ = 0.1 added to the measurements yk.

produces more accurate estimates with smoother trajectories, especially in the forced case. Finally,
we show in Figure 20 that in the absence of observation noise, the RL-ROE has better generalization
abilities once the time goes beyond the window t ∈ [0, 5] seen during training. These comparisons
suggest that the RL-ROE design, in which the policy directly corrects the ROM dynamics and is
trained in an RL setting that accounts for errors compounding over time, is more robust to data not
seen during training and to measurement noise.


-----

= 0.5, estimation error

2 4 6 8

RL-ROE
ROM+DNN

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10

10


10


10

10


10


10


10


= 0.5, trajectories

2 0

reference
RL-ROE
ROM+DNN

x2


= 1, trajectories

2 0

x2


= 2, trajectories

5.0 2.5 0.0 2.5

x2


Figure 18: Accuracy of the ROM+DNN estimator for the unforced case with noise. Same as Figure 2, but comparing the RL-ROE with the ROM+DNN estimator, and with independent Gaussian
observation noise of standard deviation σ = 0.1 added to the measurements yk.


= 0.5, estimation error

2 4 6 8

RL-ROE
ROM+DNN

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10


10


10


10


10


10


= 0.5, trajectories

5.0 2.5 0.0 2.5 5.0

reference
RL-ROE
ROM+DNN

x2


= 1, trajectories

0

x2


= 2, trajectories

5 0

x2


10


Figure 19: Accuracy of the ROM+DNN estimator for the forced case with noise. Same as Figure 4, but comparing the RL-ROE with the ROM+DNN estimator, and with independent Gaussian
observation noise of standard deviation σ = 0.1 added to the measurements yk.


-----

= 0.5, estimation error

2 4 6 8

RL-ROE
ROM+DNN

time


= 1, estimation error

4 6 8

time


= 2, estimation error

4 6 8

time


10

10


10

10


10 2

3.5

3.0

3 2.5

2.0

1.5

1.0


10


10


10


= 0.5, trajectories

5.0 2.5 0.0 2.5

reference
RL-ROE
ROM+DNN

x2


= 1, trajectories

0

x2


= 2, trajectories

0

x2


2.5

2.0

1.5

1.0

0.5

0.0


Figure 20: Accuracy of the ROM+DNN estimator for the forced case. Same as Figure 4, but comparing the RL-ROE with the ROM+DNN estimator.


-----

