# AARL: AUTOMATED AUXILIARY LOSS FOR REINFORCEMENT LEARNING

**Anonymous authors**
Paper under double-blind review

ABSTRACT

A good state representation is crucial to reinforcement learning (RL) while an
ideal representation is hard to learn only with signals from the RL objective. Thus,
many recent works manually design auxiliary losses to improve sample efficiency
and decision performance. However, handcrafted auxiliary losses rely heavily on
expert knowledge, and therefore lack scalability and can be suboptimal for boosting RL performance. In this work, we introduce Automated Auxiliary loss for Reinforcement Learning (AARL), a principled approach that automatically searches
the optimal auxiliary loss function for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size 4.6×10[19] and explore
the space with an efficient evolutionary search strategy. We evaluate AARL on
the DeepMind Control Suite and show that the searched auxiliary losses have significantly improved RL performance in both pixel-based and state-based settings,
with the largest performance gain observed in the most challenging tasks. AARL
greatly outperforms state-of-the-art methods and demonstrates strong generalization ability in unseen domains and tasks. We further conduct extensive studies to
shed light on the effectiveness of auxiliary losses in RL.

1 INTRODUCTION

Reinforcement learning (RL) has been a hot research topic and has shown significant progress in
many fields (Mnih et al., 2013; Silver et al., 2016; Gu et al., 2017; Vinyals et al., 2019). Recent
RL research focuses on obtaining a good representation of states as it is shown to be the key to
improve sample efficiency of RL (Laskin et al., 2020). This is because in high-dimensional environments, applying RL directly on the complex state inputs is incredibly sample-inefficient (Lake et al.,
2017; Kaiser et al., 2019), making RL hard to scale to real-world tasks where interacting with the
environment is costly (Dulac-Arnold et al., 2019). Standard RL paradigm learns the representation
from critic loss (value prediction) and / or actor loss (maximizing cumulative reward), which hardly
extracts informative representations in challenging environments like pixel-based RL and complex
robotics systems (Lake et al., 2017; Kober et al., 2013).

This motivates adding auxiliary losses in support of learning better latent representations of states.
The usage of auxiliary loss encodes prior knowledge of RL and its environments, and puts regularization on training (Shelhamer et al., 2016). Typical auxiliary losses are manually designed by
human experts, including observation reconstruction (Yarats et al., 2019), reward prediction (Jaderberg et al., 2017) and environment dynamics prediction (Shelhamer et al., 2016; De Bruin et al.,
2018; Ota et al., 2020). However, such auxiliary loss designs and choices rely heavily on human
knowledge of what might be helpful for RL training, which requires extensive human efforts to scale
to new tasks and can be suboptimal for improving RL performance (Yang & Nachum, 2021).

In this paper, we rigorously treat auxiliary loss functions for RL as a first-class problem to explicitly
address the question: what is the universal approach to find a good auxiliary loss function for RL?
Considering that the automated machine learning (AutoML) community has shown promising results with automated loss search in computer vision tasks (Li et al., 2019), we propose to automate
the process of designing auxiliary loss functions of RL.

Specifically, we formulate the task as a bi-level optimization, where we try to find the best auxiliary
loss function, which, to the most extent, helps train a good RL agent. The inner-level problem is
a typical RL training problem, while the outer-level can be seen as a search problem. To tackle


-----

Table 1: Existing auxiliary losses in our search space.

|Loss inputs Auxiliary loss Loss operator Horizon Source Target|Col2|
|---|---|
|Forward dynamics Inverse dynamics Reward prediction Action inference CURL (Laskin et al., 2020) ATC (Stooke et al., 2021) SPR (Schwarzer et al., 2020)|MSE 1 {st, at} {st+1} MSE 1 {at, st+1} {st} MSE 1 {st, at} {rt} MSE 1 {st, st+1} {at} Bilinear 1 {st} {st} Bilinear k {st} {st+1, · · ·, st+k} N-MSE k {st, at, at+1, · · ·, at+k−1} {st+1, · · ·, st+k}|



this, we first design a novel search space of auxiliary loss functions, which is a combination of
two components: loss input and loss operator. As shown in Table 1, the search space covers many
existing handcrafted losses, and it is also substantially large (as large as 4.6 × 10[19]). We then
propose an efficient search strategy to explore the space. This is done in a two-step manner. We first
finalize the loss operator specification and then use an evolutionary strategy that performs mutations
on configurations of loss input, to identify the top-performing loss inputs quickly.

We evaluate our search framework on both pixel-based and state-based environments in the DeepMind Control suite (Tassa et al., 2018). Extensive experiments show that the searched auxiliary loss
functions greatly outperform state-of-the-art methods and can easily transfer to unseen environments
(that are never used during the search), showing that our proposed method is robust and effective.

We highlight the main contributions of this paper below:

-  We introduce a principled and universal approach for auxiliary loss design in RL. To the best of
our knowledge, we are the first to derive the optimal auxiliary loss function with an automatic
process. Our framework can be easily applied to arbitrary RL tasks to search for the best auxiliary
loss function.

-  We demonstrate that AARL significantly outperforms state-of-the-art methods in both pixel-based
and state-based environments. The searched auxiliary loss functions show strong generalization
ability to transfer to unseen environments.

-  We analyze the derived auxiliary loss functions and deliver some insightful discussions that we
hope will deepen the understanding of auxiliary losses in RL. We will also open source our codebase to facilitate future research.

2 METHODOLOGY

We consider the standard Markov Decision Process (MDP) setting where we specify the state,
action and reward at time step t as (st, at, rt). The sequence of interaction data is denoted as
(st, at, rt, _, st+k), where k represents the horizon of this sequence. Suppose we have an RL_
_· · ·_
agent Mω parameterized by ω and R(Mω; E) is the agent performance (i.e., cumulative discounted
reward) in environment E. The goal of AARL is to find the optimal auxiliary loss function L,
such that, when ω is optimized under an arbitrary RL objective function LRL (e.g., actor-critic loss)
together with L, the agent achieves the best performance. Formally, we have

max _R(Mω∗(L); E),_
_L_ (1)

s.t. _ω[∗](L) = arg minω_ (LRL(ω; E) + λL(ω; E)),

where λ is a hyper-parameter that controls the relative weight of the auxiliary task. We solve this
bi-level optimization problem with AutoML techniques. The inner optimization is a standard RL
training procedure. For the outer one, we define a finite and discrete search space (Section 2.1), and
use a variation of evolution strategy to explore the space (Section 2.2). We will explain the details
in the rest of this section.

2.1 SEARCH SPACE

An auxiliary loss function L can be viewed as a combination of two components: 1) loss input I and
2) loss operator f . We define a search space as shown in Figure 1, where I is a pair of binary masks


-----

|1|1|0|
|---|---|---|

|0|0|0|1|
|---|---|---|---|


**Loss Input** **Auxiliary Loss**

**Search Space**

**Loss Operator** **Loss Operator**

**Example of forward dynamics prediction (horizon k=1)** **Predictor** **momentum update** **stop-grad**

**Encoder** **Target Encoder**

**Binary Mask** **Binary Mask**

**1** **1** **0** **0** **0** **0** **0** **1**

**Loss Input**


Figure 1: Computation graph and search space of auxiliary loss functions.

( ˆm, m) that selects from interaction data to compute ˆy and y, and f is an operator that aggregates
the selected inputs into an expression of the loss function with scalar output.

**Loss Input** Unlike supervised machine learning tasks with an explicit prediction and ground truth
for the loss function, auxiliary losses in RL have no ground truths beforehand. Instead, they are
generated accordingly upon interaction with the environment itself in a self-supervised manner. As
shown in Figure 1, we take some tokens from the sequence (st, at, rt, _, st+k) with binary mask_
_· · ·_
vector ˆm and feed them into the encoder _θ[ˆ] which maps states to latent representations. The predictor_
module then tries to predict targets (encoded with a momentum encoder θ, as is typically done in
recent works), which are another few tokens we select from the sequence with another binary mask
vector m. Details about momentum encoder are given in Appendix A.1.2. In other words, the search
space for loss input is a pair of binary masks ( ˆm, m), each of which is up to length (3k + 1) if the
length of an interaction data sequence, i.e., horizon, is limited to k steps. In our case, we set the
maximum horizon length kmax = 10.

**Loss Operator** We optimize ˆy to be as similar as possible to y. Therefore, we make loss operator f
cover commonly-used similarity measures, including inner product (Inner) (He et al., 2020; Stooke
et al., 2021), bilinear inner product (Bilinear) (Laskin et al., 2020), cosine similarity (Cosine) (Chen
et al., 2020), mean squared error (MSE) (Ota et al., 2020; De Bruin et al., 2018) and normalized mean
squared error (N-MSE) (Schwarzer et al., 2020). Some existing works, e.g., contrastive objectives
like InfoNCE loss (Oord et al., 2018), also incorporate the trick to sample un-paired predictions
and targets as negative samples and maximize the distances between them. We find this technique
applicable to all the loss operators mentioned above and thus incorporate the discriminative version
of these operators in our search space.

**Search Space Complexity** As shown in Table 1, many existing manually designed auxiliary losses
can naturally fit into our loss space thus are our special cases, which proves that this loss space is
reasonable, flexible, and general. However, this is also at the cost that the space is substantially
large. In total, the size of the entire space is 10 × _i=1_ [2][6][i][+2][ ≈] [4][.][6][ ×][ 10][19][. The detailed derivation]
can be found in Appendix C.

[P][10]

2.2 SEARCH STRATEGY

**Search Space Pruning** Considering that the loss space is extremely large, an effective optimization strategy is inevitably required. Directly grid-searching over the whole space is infeasible because of unacceptable computational cost. Thus some advanced techniques such as space pruning
and an elaborate search strategy are necessary. Our search space can be seen as a combination of
the space for the input I and the space for the operator f . Inspired by AutoML works (Dai et al.,
2020; Ying et al., 2019) that search for hyper-parameters first and then neural architectures, we approximate the joint search of input and operator in Equation (1) in a two-step manner. The optimal


-----

Table 2: Normalized episodic rewards (mean & standard deviation for 5 seeds) of 3 environments
used in evolution on pixel-based DMControl500K with different loss operators.

|Loss operator and discrimination|Inner Bilinear Cosine MSE N-MSE|
|---|---|
|w/ negative samples w/o negative samples|0.979 ± 0.344 0.953 ± 0.329 0.872 ± 0.412 0.124 ± 0.125 0.933 ± 0.360 0.669 ± 0.311 0.707 ± 0.299 0.959 ± 0.225 1.000 ± 0.223 0.993 ± 0.229|



auxiliary loss {I _[∗], f_ _[∗]} can be optimized as:_

max _R(Mω∗(L); E) = max,f_ _R(Mω∗(I,f ∗); E)_
_L_ _I_ _[R][(][M][ω][∗][(][I][,f]_ [)][;][ E][)][ ≈] [max]I (2)

where _f_ _[∗]_ _≈_ arg maxf EI[R(Mω∗(I,f ); E)]

To decide the best loss operator, for every f in the loss operator space, we estimate
EI[R(Mω∗(I,f ); E)] with a random sampling strategy. We run 15 trials for each loss operator and
calculate the average of their normalized return on three environments. More details are available
in Appendix A.1.3. Surprisingly, as summarized in Table 2, the simplest MSE without negative
samples outperforms all other loss operators with complex designs. Therefore, this loss operator is
chosen for the rest of this paper.

**Evolution** Even with the search space pruning, the rest
of the space still remains large. To efficiently identify
top candidates from the search space, we adopt an evo- **Population (P=100)**
lutionary algorithm (Real et al., 2019). The pipeline of **Binary Mask** **Binary Mask**
the evolution process is illustrated in Figure 2. In eachstage, we first train and evaluate a population of candi- **Binary Mask** **Binary Mask**
dates with size P = 100, where each candidate corre- **Binary Mask** **Binary Mask** **Loss-Rejection**
sponds to a pair of binary masks ( ˆm, m). The candidates **Protocol**
are then sorted by the approximated area under learn- **Reject**
_ing curve (AULC) (Ghiassian et al., 2020; Stadie et al.,_ **Training**
2015), which well captures both convergence speed and **Mutation**
final performance (Viering & Loog, 2021) and reduces **Top 25% Candidates**
the variance of RL evaluation. **Binary Mask** **Binary Mask**

The top-25% candidates are selected after each training **Binary Mask** **Binary Mask**
stage, and we perform four types of mutations on the selected candidates to form a new population for the next **Binary Mask** **Binary Mask**
stage: replacement (50% of the population); crossover
(20%); horizon decrease and horizon increase (10%), as

**Population (P=100)**

**Binary Mask** **Binary Mask**

**Binary Mask** **Binary Mask**

**Binary Mask** **Binary Mask** **Loss-Rejection**

**Protocol**

**Reject**
**Training**

**Mutation**

**Top 25% Candidates**

**Binary Mask** **Binary Mask**

**Binary Mask** **Binary Mask**

**Binary Mask** **Binary Mask**

shown in Figure 3. Moreover, the last 20% of the new Figure 2: Overview of the evolution
population is from random generation. As for each bit pipeline.
of masks, replacement operation flips the given bit with
probability p = 2 (3k1+1) [, where][ k][ is the horizon length. Crossover generates a new candidate by]

_·_
randomly combining the mask bits of two candidates with the same horizon length in the population.
We also incorporate the knowledge from the existing auxiliary loss designs by bootstrapping the initial population with a prior distribution (Co-Reyes et al., 2020), so that we can quickly converge to
a reasonable loss function. More implementation details are available in Appendix A.3.

**Mask before mutation** **Mask After mutation**

|Replacement|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|


**Selected** **Not Selected**

**Horizon decrease**

**Changed by Muation**

**Horizon increase**

**Crossover**


Figure 3: Mutation operations of auxiliary loss functions.


-----

3 EXPERIMENTS

We evaluate our method and the obtained auxiliary loss functions on DMControl suite (Tassa et al.,
2018). DMControl suite is powered by MuJoCo physics engine (Todorov et al., 2012) with various
challenging tasks. In both pixel-based (images) and state-based (proprioceptive features) settings,
we choose Soft Actor-Critic (SAC) (Haarnoja et al., 2018) as our base RL algorithm since it is the
state-of-the-art model-free RL algorithm. Implementation details are given in Appendix A.

3.1 PIXEL-BASED RL

**Experiment Settings** We follow the same network architecture as CURL with a 4-layer convolutional encoder for image input. In the search phase, we compare our method AARL to SAC, SAC

(no aug), and CURL with the same hyper-parameters reported in Appendix A.2.1. All methods
randomly crop images from 100 × 100 to 84 × 84 as data augmentation apart from SAC (no aug).
After searching, we test the generalization ability of the obtained auxiliary losses in other unseen
environments in comparison with state-of-the-art model-free and model-based methods.

**Cheetah-Run (Pixel)** **Reacher-Easy (Pixel)** **Walker-Walk (Pixel)**

Figure 4: Evolution process in pixel-based environments. Every white dot represents a loss candidate, and the score of y-axis shows its corresponding approximated AULC score (Ghiassian et al.,
2020; Stadie et al., 2015). The horizontal lines show the scores of baselines. The AULC score is
approximated with the average evaluation score at 100k, 200k, 300k, 400k, 500k time steps.

**Search Results** We apply our search algorithm on Cheetah-Run, Reacher-Easy, and Walker-Walk,


which we call “training environments”. We choose these environments because they are more challenging and there might be more room for improvement with auxiliary losses. For each environment,
we set the total budget of each experiment to 16k GPU hours (on NVIDIA P100) and terminate the
experiment when the resource is exhausted. The evolution process is demonstrated in Figure 4,
where the y-axis shows the AULC of candidates. The results show that, during the evolution process, we can easily find candidates surpassing baselines in the first stage. In the subsequent stages,
the overall population score continues to grow, and most candidates can outperform baselines. This
demonstrates the effectiveness of our search space and the efficiency of our search strategy.

**Generalize to Other Environments** To reduce the risk that auxiliary losses overfit to one particular environment and avoid selecting candidates that perform well only due to randomness, we
run cross-validation on the three searched environments to decide the best candidate. The details
of this process can be found in the Appendix A.4. After cross-validation, we finalize an auxiliary
loss function, which we call “AARL-Pixel” (all the top candidates during evolution are reported in
Appendix D), and we train agents with the obtained auxiliary loss to compare with state-of-the-art
model-free and model-based methods on DMControl100k and DMControl500k. We select these
environments because they are common benchmarks for state-of-the-art algorithms of pixel-based
RL. (Laskin et al., 2020). Note these comparisons are additionally made on three unseen environments, that are entirely not accessible during evolution, to test the generalizability of our method and
to ensure our comparisons are fair. The results are summarized in Table 3, where AARL greatly
outperforms state-of-the-art methods on 11 out of 12 benchmark settings. Note that Finger-Spin,
Reacher-Easy and Ball in cup-Catch are environments that are never used during the search, while
AARL still performs very well on them. This result implies that AARL-Pixel is a robust and effective auxiliary loss that is potentially helpful to RL under various settings. The performance gain


-----

Table 3: Episodic rewards (mean & standard deviation for 10 seeds) on DMControl100K and DMControl500K with pixel inputs. Note that the optimal score of DMControl is 1000 for all environments. The baselines methods are PlaNet (Hafner et al., 2019b), Dreamer (Hafner et al., 2019a),
SAC+AE (Yarats et al., 2019), SLAC (Lee et al., 2019), pixel-based SAC (Haarnoja et al., 2018).
Performance values of all baselines are referred to Laskin et al. (2020), except for Pixel SAC. We
also show learning curves of all 12 DMC environments in Appendix B.1.

|500K Steps Scores|AARL-Pixel CURL§ PlaNet§ Dreamer§ SAC+AE§ SLACv1§ Pixel SAC|
|---|---|
|Finger-Spin∗ Cartpole-Swingup∗ Reacher-Easy† Cheetah-Run† Walker-Walk† Ball in cup-Catch∗|983 ± 4 926 ± 45 561 ± 284 796 ± 183 884 ± 128 673 ± 92 282 ± 102 864 ± 19 841 ± 45 475 ± 71 762 ± 27 735 ± 63 - 344 ± 104 938 ± 46 929 ± 44 210 ± 390 793 ± 164 627 ± 58 - 312 ± 132 613 ± 39 518 ± 28 305 ± 131 570 ± 253 550 ± 34 640 ± 19 99 ± 28 917 ± 18 902 ± 43 351 ± 58 897 ± 49 847 ± 48 842 ± 51 76 ± 44 970 ± 8 959 ± 27 460 ± 380 897 ± 87 794 ± 58 852 ± 71 200 ± 114|
|100K Steps Scores||
|Finger-Spin∗ Cartpole-Swingup∗ Reacher-Easy† Cheetah-Run† Walker-Walk† Ball in cup-Catch∗|872 ± 27 767 ± 56 136 ± 216 341 ± 70 740 ± 64 693 ± 141 160 ± 138 815 ± 66 582 ± 146 297 ± 39 326 ± 27 311 ± 11 - 243 ± 19 778 ± 164 538 ± 223 20 ± 50 314 ± 155 274 ± 14 - 277 ± 69 449 ± 34 299 ± 48 138 ± 88 235 ± 137 267 ± 24 319 ± 56 128 ± 12 510 ± 151 403 ± 24 224 ± 48 277 ± 12 394 ± 22 361 ± 73 127 ± 28 862 ± 167 769 ± 43 0 ± 0 246 ± 174 391 ± 82 512 ± 110 100 ± 90|



_†: Training environments. ∗: Unseen environments. §: Results reported in Laskin et al. (2020)._

is obvious in DMControl100K, where AARL-Pixel (pixel-based input) has shown great sample
efficiency.

3.2 STATE-BASED RL


**Experiment Settings** As for state-based RL, since the state is low-dimensional,
we simply use a 1-layer densely connected MLP as the state encoder as shown in
Figure 5. So, for this setting, we focus on this simple encoder structure. Additional
ablations on state encoder architectures are given in Section 3.4. In the search
phase, we compare AARL to SAC-Identity, SAC-DenseMLP, CURL-DenseMLP.
To ensure a fair comparison, all SAC related hyper-parameters are the same as
those reported in the CURL paper. Details can be found in Appendix A.2.2.
SAC-Identity is vanilla SAC with no state encoder, while the other three methods
(AARL, SAC-DenseMLP, CURL-DenseMLP) use the same encoder architecture.
Different from the pixel-based setting, there is no data augmentation in the statebased setting. Note that many environments that are challenging in pixel-based
settings become easy to tackle with state-based inputs. Therefore we apply our
search framework to more challenging environments for state-based RL, including
Cheetah-Run, Hopper-Hop and Quadruped-Run.

|Co|ncaFt|
|---|---|


**State Embedding**


**ConcatF**

**MLP Encoder**


Figure 5: Network architecture of 1-layer
DenseMLP state
encoder.


**Search Results** Similar to pixel-based settings, we approximate AULC with the average score
agents achieve at 300k, 600k, 900k, 1200k and 1500k time steps[1]. For each environment, we early
stop the experiment when the budget of 1,500 GPU hours is exhausted. The evolution process is
shown in Figure 6, where we find a large portion of candidates outperform baselines (horizontal
dashed lines). The performance improvement is especially significant on Cheetah-Run, where almost all candidates in the population greatly outperform all baselines by the end of the first stage.

**Generalize to Other Environments** Similar to pixel-based settings, we also use cross-validation
to select the best loss function, which we call “AARL-State” here (all the top candidates during
evolution are reported in Appendix D), and run a thorough evaluation on all 18 environments. The
results are summarized in Table 4. AARL-State again brings significant performance gain, achieving much stronger sample efficiency than SAC. It is noteworthy that AARL-State is able to outperform baseline methods in 16 out of 18 environments (with 15 unseen environments). These results
show that even for tasks with lower-dimensional state space, there is still a huge potential for improvement with better auxiliary objectives. Moreover, this performance gain is especially significant

1As for Cheetah-Run, we still use average score agents achieve at 100k, 200k, 300k, 400K and 500k time
steps since agents converge close to optimal score within 500k time steps.


-----

**Cheetah-Run (State)** **Hopper-Hop (State)** **Quadruped-Run (State)**

Figure 6: Evolution process in state-based environments. Every white dot represents a loss candidate, and the score of y-axis shows its corresponding approximated AULC score. The horizontal
lines show the scores of baselines. The AULC score is approximated with the average evaluation
score at 300k, 600k, 900k, 1200k, 1500k time steps (Cheetah-Run at 100k, 200k, 300k, 400K).
Table 4: Episodic rewards (mean & standard deviation for 10 seeds) on DMControl100K (easy
tasks) and DMControl1000K (difficult tasks) with state inputs. SAC-Identity has no state encoder
while AARL, SAC and CURL use the same state encoder architecture. All four variants here use
the same hyper-parameters.


|100K Steps Scores|AARL-State SAC-Identity SAC CURL|
|---|---|
|Finger-Spin∗ Finger-Turn hard∗ Cartpole-Swingup∗ Cartpole-Swingup sparse∗ Reacher-Easy∗ Cheetah-Run† Walker-Stand∗ Walker-Walk∗ Walker-Run∗ Ball in cup-Catch∗ Fish-Upright∗ Hopper-Stand∗|837 ± 52 805 ± 32 785 ± 106 712 ± 83 218 ± 117 347 ± 150 174 ± 94 43 ± 42 877 ± 5 873 ± 10 866 ± 7 854 ± 17 695 ± 147 455 ± 359 627 ± 307 446 ± 196 934 ± 38 697 ± 192 874 ± 87 749 ± 183 472 ± 30 237 ± 27 172 ± 29 190 ± 32 948 ± 7 940 ± 10 862 ± 196 767 ± 104 906 ± 78 873 ± 89 925 ± 22 852 ± 64 564 ± 45 559 ± 34 403 ± 43 289 ± 61 965 ± 7 954 ± 12 962 ± 13 941 ± 32 498 ± 88 471 ± 62 400 ± 62 295 ± 117 311 ± 177 14 ± 16 26 ± 40 6 ± 3|


_†: Training environments. ∗: Unseen environments._


|1000K Steps Scor|es AARL-State SAC-Identity SAC CU|
|---|---|
|Quadruped-Run Pendulum-Swingu Hopper-Hop† Humanoid-Stand Humanoid-Walk Humanoid-Run∗|† 838 ± 58 345 ± 157 707 ± 148 497 p∗ 579 ± 410 506 ± 374 379 ± 391 363 278 ± 106 121 ± 51 134 ± 93 60 ∗ 286 ± 15 9 ± 2 7 ± 1 7 ∗ 299 ± 55 16 ± 28 2 ± 0 2 88 ± 2 1 ± 0 1 ± 0 1|


_†: Training environments. ∗: Unseen environments._


in complex environments like Humanoid, where SAC barely learns anything at 1000K time steps,
while AARL-State is able to achieve much better performance.

3.3 ANALYSIS OF AUXILIARY LOSS FUNCTIONS

We have collected a large number of data from the evolution process so far. To make the best use
of these data, we now conduct a comprehensive analysis to investigate statistical relations between
patterns of auxiliary loss functions and RL performance.

**Typical Patterns** We first analyze how RL performance is affected by typical patterns of auxiliary loss functions. Define the set of the input to the encoder that is being trained as “source”, and
the set of prediction targets as “target”. We say an auxiliary loss candidate has a certain pattern
if the pattern’s source is a subset of the candidate’s source, and the pattern’s target is a subset of
the candidate’s target. For instance, a loss candidate of **_st, at_** **_st+1, st+2_** has the pattern
**_{_** **_} →{_** **_}_**
_st, at_ _st+1_, and does not have the pattern _at, st+1_ _st_ . The patterns we consider
_{_ _} →{_ _}_ _{_ _} →{_ _}_
include forward dynamics _st, at_ _st+1_, inverse dynamics _at, st+1_ _st_, reward pre_{_ _} →{_ _}_ _{_ _} →{_ _}_
diction _st, at_ _rt_, action inference _st, st+1_ _at_ and state reconstruction in the latent
_{_ _} →{_ _}_ _{_ _} →{_ _}_
space _st_ _st_ . For each of these patterns, we divide auxiliary loss candidates into two classes:
_{_ _} →{_ _}_
1) with this pattern or 2) without this pattern. We then calculate the average performance of these
two classes and compare their performances to conclude whether this pattern is helpful for RL. The
results are summarized in Table 5. Here a positive number indicates that this pattern is beneficial,
and if the performance gain is statistically significant, the number is marked with the asterisk, indicating it is very likely to be helpful. A negative number indicates that this pattern is detrimental
and correlated to worse performance. We highlight some interesting observations as follows. 1) Forward dynamics is helpful in most environments, and improves the RL performance on Reacher-Easy
(Pixel) and Cheetah-Run (State) significantly (i.e., p-value<0.05). 2) State reconstruction in the latent space is able to improve RL performance in pixel-based environments while it has significant
negative effects on state-based environments. We hypothesize that this is because in the pixel-based
setting, data augmentation encourages the encoder to learn augmentation-invariant representations,
while in the state-based setting, no augmentation is used, and thus the encoder learns no useful rep

-----

resentations. This also explains why CURL is underperforming in state-based experiments. Note
that not all typical patterns bring performance gains. Some can even be very detrimental, and this
shows that commonly-used auxiliary loss patterns do not work well in the state-based setting, further
highlighting the fact that great research potential lies in this setting.

Table 5: Statistical analysis on auxiliary loss functions. The number reported is the difference of
the mean score of two classes of auxiliary losses during evolution, and we report its corresponding
p-value from the t-test.

|Col1|Typical patterns (w/ - w/o)|
|---|---|
||Forward dynamics Inverse dynamics Reward prediction Action inference State reconstruction|
|Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hopper-Hop (State) Quadruped-Run (State)|+1.28 −3.51 −31.16∗∗ −75.95∗∗ +42.44∗∗ +28.25∗ +8.36 +37.80∗∗ +3.35 +70.72∗∗ +22.20 −48.59∗∗ −8.11 +29.86∗ +13.93 +94.18∗∗ −23.66∗∗ −33.28∗∗ −109.33∗∗ −50.15∗∗ +15.50∗∗ −16.47∗∗ −11.30∗ −32.10∗∗ −25.67∗∗ −28.07 −18.19 −114.23∗∗ −105.37∗∗ −82.06∗∗|


|∗: p-value < 0.05. ∗∗: p-v|value < 0.01|
|---|---|
||Source or target|
||State, ntarget > nsource Action, ntarget > nsource Reward, ntarget > nsource|
|Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hopper-Hop (State) Quadruped-Run (State)|+80.09∗∗ +13.62 +3.33 +1.98 −12.72 +65.66∗∗ +73.56∗∗ +42.22∗ −41.90∗ +188.06∗∗ −102.62∗∗ −93.94∗∗ +19.80∗∗ −29.70∗∗ −5.03 +75.17∗∗ −4.31 −46.60∗|


_∗: p-value < 0.05. ∗∗: p-value < 0.01_

**Number of Sources and Targets** We then investigate whether it is more beneficial to use a small
number of sources to predict a large number of targets (ntarget > nsource, e.g., use st to predict
_st+1, st+2, st+3), or the other way around (ntarget < nsource, e.g., use st, st+1, st+2 to predict_
_st+3). The statistical results are shown in Table 5, where we find that auxiliary losses with more_
states on the target side have a significant advantage over losses with more states on the source
side. This result echoes recent works (Stooke et al., 2021; Schwarzer et al., 2020) which show that
predicting more states leads to strong performance gains.

3.4 ADDITIONAL ABLATION STUDY

**Search Space Pruning** As introduced in Section 2.2, we decompose the full search space into loss
operator and loss inputs. Here we try to directly apply evolution strategy to the whole space without
the pruning step. The comparison results are shown in Figure 7. We can see that pruning improves
the evolution process, making it easier to find good candidates.


Table 6: Normalized episodic rewards
of AARL (mean & standard deviation
for 5 seeds of 6 environments) on statebased DMControl100K with different
encoder architectures.


**Encoder Architecture for State-based RL** As shown
in Figure 5, we choose a 1-layer densely connected MLP
as the state encoder for state-based RL. Here, we conduct an ablation study on different encoder architectures
in the state-based setting. The results are summarized
in Table 6, where AARL with 4-layer encoders consistently perform worse than 1-layer encoders. We also note
that dense connection is helpful in the state-based setting
compared with naive MLP encoders.

|AARL-MLP (1-layer) 0.919 ± 0.217|AARL-MLP (4-layer) 0.544 ± 0.360|
|---|---|
|AARL-DenseMLP (1-layer) 1.000 ± 0.129|AARL-DenseMLP (4-layer) 0.813 ± 0.218|


**Stage 1** **Stage 2** **Stage 3**

Figure 7: Comparison of evolution with and without pruning by performance histogram.


-----

4 RELATED WORK

4.1 REINFORCEMENT LEARNING WITH AUXILIARY LOSSES

Using auxiliary tasks to improve sample efficiency of RL, especially on pixel-based control tasks,
has been explored in many recent works. A number of manually designed auxiliary objectives are
shown to boost RL performance, including observation reconstruction (Yarats et al., 2019), reward
prediction (Jaderberg et al., 2017), dynamics prediction (De Bruin et al., 2018) and contrastive learning objectives (Laskin et al., 2020; Schwarzer et al., 2020; Stooke et al., 2021). It is noteworthy that
these works mainly focus on pixel-based settings, while only a limited number of works study the
state-based setting (Munk et al., 2016; Ota et al., 2020). Although it might seem that the state-based
setting benefits less from auxiliary tasks due to their lower-dimensional state space, we show that
there is in fact a huge potential of improving state-based RL performance with auxiliary objectives.

Compared to the previous works, we point out two major advantages of our approach. 1) Instead
of handcrafting an auxiliary loss with expert knowledge, AARL automatically searches for the best
auxiliary loss, relieving researchers from such tedious work. 2) AARL is a principled approach that
can be used in arbitrary RL settings. In both pixel-based and the rarely studied state-based settings,
we discover good auxiliary losses that bring significant performance improvement.

4.2 AUTOMATED REINFORCEMENT LEARNING

It is well known that RL training is sensitive to hyper-parameters and environment changes (Henderson et al., 2018). Thus many works have attempted to use techniques in AutoML to alleviate
human intervention. Exiting works focus on hyper-parameter optimization (Espeholt et al., 2018;
Paul et al., 2019; Xu et al., 2020; Zahavy et al., 2020), reward search (Faust et al., 2019; Veeriah
et al., 2019) and network architecture search (Runge et al., 2019; Franke et al., 2021). In contrast,
our method aims to search for auxiliary loss functions that generalize across different environments.

4.3 AUTOMATED LOSS DESIGN

A few recent works in the AutoML community have been automating the design of good loss functions that outperform traditionally handcrafted ones. Specifically, AM-LFS (Li et al., 2019) defines
the loss function search space as a parameterized probability distribution of softmax loss hyperparameters. AutoLoss-Zero (Li et al., 2021) proposes to search loss functions with primitive mathematical operators. These methodologies are proposed specifically for computer vision tasks.

For RL, existing works focus on searching for a better RL objective, EPG (Houthooft et al., 2018)
and MetaGenRL (Kirsch et al., 2020) define the search space of loss functions as parameters of a low
complexity neural network. Co-Reyes et al. (2020) defines the search space of RL loss functions as
a directed acyclic graph and discovers two DQN-like regularized RL losses. Note that none of these
works investigate auxiliary loss functions, which are crucial to facilitate representation learning in
RL and to make RL successful in highly complex environments. To the best of our knowledge, our
work is the first attempt to search for auxiliary loss functions and greatly improve RL performance.

5 CONCLUSION AND FUTURE WORK

We present AARL, a principled and universal approach for automated auxiliary loss design for
RL. With this framework, we discover highly performant auxiliary loss functions that generalize
beyond the training environments for both pixel-based and state-based settings. We present extensive
experimental results that provide strong empirical evidence for the effectiveness of our method, and
also conduct an in-depth investigation of the statistical relations between auxiliary loss patterns
and RL performance. We hope our studies provide insights that will deepen the understanding
of auxiliary losses in RL, and shed light on how to make RL more efficient and practical. One
interesting future work direction is to further investigate the relationship between auxiliary loss
patterns and RL performance, and to better understand how and why auxiliary losses can help to
improve RL performance. Another direction is to study how to combine the RL objective and the
auxiliary loss functions in a more effective manner.


-----

REPRODUCIBILITY STATEMENT

All implementation details are introduced in the main text and appendix (Appendix A). We will open
source our codebase to facilitate future research.

REFERENCES

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning,
pp. 1597–1607. PMLR, 2020.

John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Quoc V Le, Sergey Levine, Honglak
Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Con_ference on Learning Representations, 2020._

Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong
Tian, Matthew Yu, Peter Vajda, and Joseph E. Gonzalez. Fbnetv3: Joint architecture-recipe search
using predictor pretraining, 2020.

Tim De Bruin, Jens Kober, Karl Tuyls, and Robert Babuˇska. Integrating state representation learning
into deep reinforcement learning. IEEE Robotics and Automation Letters, 3(3):1394–1401, 2018.

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement
learning. arXiv preprint arXiv:1904.12901, 2019.

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam
Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning,
pp. 1407–1416. PMLR, 2018.

Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reinforcement
learning. arXiv preprint arXiv:1905.07628, 2019.

J¨org K. H. Franke, Gregor K¨ohler, Andr´e Biedenkapp, and Frank Hutter. Sample-efficient automated deep reinforcement learning. In 9th International Conference on Learning Repre_sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL_
[https://openreview.net/forum?id=hSjxQ3B7GWq.](https://openreview.net/forum?id=hSjxQ3B7GWq)

Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance
in reinforcement learning by breaking generalization in neural networks. _arXiv preprint_
_arXiv:2003.07417, 2020._

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for
robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer_ence on robotics and automation (ICRA), pp. 3389–3396. IEEE, 2017._

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash
Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning
behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019a.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James
Davidson. Learning latent dynamics for planning from pixels. In International Conference on
_Machine Learning, pp. 2555–2565. PMLR, 2019b._

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on
_Computer Vision and Pattern Recognition, pp. 9729–9738, 2020._


-----

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.
Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial
_intelligence, volume 32, 2018._

Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho,
and Pieter Abbeel. Evolved policy gradients. In Samy Bengio, Hanna M. Wallach,
Hugo Larochelle, Kristen Grauman, Nicol`o Cesa-Bianchi, and Roman Garnett (eds.), Ad_vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa-_
_tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr´eal, Canada, pp._
[5405–5414, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/](https://proceedings.neurips.cc/paper/2018/hash/7876acb66640bad41f1e1371ef30c180-Abstract.html)
[7876acb66640bad41f1e1371ef30c180-Abstract.html.](https://proceedings.neurips.cc/paper/2018/hash/7876acb66640bad41f1e1371ef30c180-Abstract.html)

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David
Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th
_International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26,_
_[2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.](https://openreview.net/forum?id=SJ6yPD5xg)_
[net/forum?id=SJ6yPD5xg.](https://openreview.net/forum?id=SJ6yPD5xg)

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad
Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based
reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019.

Louis Kirsch, Sjoerd van Steenkiste, and J¨urgen Schmidhuber. Improving generalization in meta
reinforcement learning using learned objectives. In 8th International Conference on Learning
_Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020._
[URL https://openreview.net/forum?id=S1evHerYPr.](https://openreview.net/forum?id=S1evHerYPr)

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The
_International Journal of Robotics Research, 32(11):1238–1274, 2013._

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building
machines that learn and think like people. Behavioral and brain sciences, 40, 2017.

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639–
5650. PMLR, 2020.

Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:
Deep reinforcement learning with a latent variable model. _arXiv preprint arXiv:1907.00953,_
2019.

Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs:
Automl for loss function search. In Proceedings of the IEEE/CVF International Conference on
_Computer Vision, pp. 8410–8419, 2019._

Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. Autoloss-zero:
Searching loss functions from scratch for generic tasks. arXiv preprint arXiv:2103.14026, 2021.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. _arXiv preprint_
_arXiv:1312.5602, 2013._

Jelle Munk, Jens Kober, and Robert Babuˇska. Learning state representation for deep actor-critic
control. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 4667–4673. IEEE,
2016.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

Kei Ota, Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing
input dimensionality improve deep reinforcement learning? In International Conference on Ma_chine Learning, pp. 7424–7433. PMLR, 2020._


-----

Supratik Paul, Vitaly Kurin, and Shimon Whiteson. Fast efficient hyperparameter tuning for policy
gradients. arXiv preprint arXiv:1902.06583, 2019.

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image
classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, 33:
4780–4789, Jul 2019. ISSN 2159-5399. doi: 10.1609/aaai.v33i01.33014780. [URL http:](http://dx.doi.org/10.1609/aaai.v33i01.33014780)
[//dx.doi.org/10.1609/aaai.v33i01.33014780.](http://dx.doi.org/10.1609/aaai.v33i01.33014780)

Frederic Runge, Danny Stoll, Stefan Falkner, and Frank Hutter. Learning to design RNA. In
_7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA,_
_May 6-9, 2019. OpenReview.net, 2019._ [URL https://openreview.net/forum?id=](https://openreview.net/forum?id=ByfyHh05tQ)
[ByfyHh05tQ.](https://openreview.net/forum?id=ByfyHh05tQ)

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International
_Conference on Learning Representations, 2020._

Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Selfsupervision for reinforcement learning. arXiv preprint arXiv:1612.07307, 2016.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,
Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering
the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.

Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement
learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015.

Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning
from reinforcement learning. In International Conference on Machine Learning, pp. 9870–9879.
PMLR, 2021.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv
_preprint arXiv:1801.00690, 2018._

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033.
IEEE, 2012.

Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful
questions as auxiliary tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alch´e-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu_ral Information Processing Systems 32:_ _Annual Conference on Neural Information Pro-_
_cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp._
[9306–9317, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/](https://proceedings.neurips.cc/paper/2019/hash/10ff0b5e85e5b85cc3095d431d8c08b4-Abstract.html)
[10ff0b5e85e5b85cc3095d431d8c08b4-Abstract.html.](https://proceedings.neurips.cc/paper/2019/hash/10ff0b5e85e5b85cc3095d431d8c08b4-Abstract.html)

Tom Viering and Marco Loog. The shape of learning curves: a review. _arXiv preprint_
_arXiv:2103.10948, 2021._

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha¨el Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster
level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.

Zhongwen Xu, Hado Philip van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and
David Silver. Meta-gradient reinforcement learning with an objective discovered online. In
Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin (eds.), Advances in Neural Information Processing Systems 33: _Annual Con-_
_ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12,_
_[2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/](https://proceedings.neurips.cc/paper/2020/hash/ae3d525daf92cee0003a7f2d92c34ea3-Abstract.html)_
[ae3d525daf92cee0003a7f2d92c34ea3-Abstract.html.](https://proceedings.neurips.cc/paper/2020/hash/ae3d525daf92cee0003a7f2d92c34ea3-Abstract.html)


-----

Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815, 2021.

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. _arXiv preprint_
_arXiv:1910.01741, 2019._

Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nasbench-101: Towards reproducible neural architecture search, 2019.

Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien
Lin (eds.), Advances in Neural Information Processing Systems 33: _Annual Conference_
_on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020,_
_virtual,_ 2020. [URL https://proceedings.neurips.cc/paper/2020/hash/](https://proceedings.neurips.cc/paper/2020/hash/f02208a057804ee16ac72ff4d3cec53b-Abstract.html)
[f02208a057804ee16ac72ff4d3cec53b-Abstract.html.](https://proceedings.neurips.cc/paper/2020/hash/f02208a057804ee16ac72ff4d3cec53b-Abstract.html)


-----

A IMPLEMENTATION DETAILS

A.1 ARCHITECTURE AND SEARCH SPACE

A.1.1 STATE ENCODER ARCHITECTURES

We demonstrate in Figure 8 the overall architecture where our auxiliary loss is used. The architecture
is generally identical to those adopted in CURL (Laskin et al., 2020). “Pixel-based” and “Statebased” are the architectures we used in our experiments. “MLP” and “4-layer DenseMLP” are for
ablations.

|Col1|Co|ncat|
|---|---|---|
|MLP E||ncoder|


**SAC Loss**

**Actor** **Critic**

**State Embedding** **Concat**

**Faltten**

**Conv Encoder**

**Auxiliary Loss**

**Loss Input**

**Loss Operator**


**SAC Loss**

**Actor** **Critic**

**State Embedding** **Concat**

**MLP Encoder**

**Auxiliary Loss**

**Loss Input**

**Loss Operator**


**SAC Loss**

**Actor** **Critic**

**State Embedding** **Concat**

**Concat**

**MLP Encoder**

**Auxiliary Loss**

**Loss Input**

**Loss Operator**


**State Embedding**

**Concat**

**MLP Encoder**

**Concat**

**MLP Encoder**

**Concat**

**MLP Encoder**

**Concat**

**MLP Encoder**


**Pixel-Based** **State-Based (MLP)**


**State-Based (1-Layer DenseMLP)** **State-Based (4-Layer DenseMLP)**


Figure 8: Network structures of pixel-based RL and state-based RL with auxiliary losses.

A.1.2 SIAMESE NETWORK

For a fair comparison with baseline methods, we follow the same Siamese networks structure for
representation learning as CURL (Laskin et al., 2020). As shown in Figure 1, when computing
targets y for auxiliary losses, we map states to state embeddings with a target encoder. We stop
gradients from target encoder θ and update θ in the exponential moving averaged (EMA) manner
where θ[′] = τ _θ[ˆ] + (1 −_ _τ_ )θ. This step, i.e., to freeze the gradients of the target encoder, is necessary
when the loss is computed without negative samples. Otherwise, encoders will collapse to generating
the same representation for any input. We have verified this in our early experiments.

A.1.3 LOSS OPERATORS

**Instance Discrimination** Our implementation is based on InfoNCE loss (Oord et al., 2018):

exp(φ(ˆy, y+))
_L = log_ (3)

exp(φ(ˆy, y+)) + _i=0_ [exp(][φ][(ˆ]y, yi))

The instance discrimination loss can be interpreted as the log-loss of a K-way softmax classifier

[P][K][−][1]

whose label is y+. The difference between discrimination based loss operators lies in the discrimination objective φ used to measure agreement between (ˆy, y) pairs. Inner employs inner product
_φ(ˆy, y) = ˆy[⊤]y while Bilinear employs bilinear product φ(ˆy, y) = ˆyWy, where W is a learnable_
parameter matrix. Cosine uses cosine distance φ(ˆy, y) = _∥yˆyˆ∥·∥[⊤]yy∥_ [for further matrix calculation. As]

for cross entropy based loss without negative samples, we only take diagonal elements for matrix
_M where Mi,j = φ( ˆyi, yj) for cross entropy calculation._

**Mean Squared Error** The implementation of MSE-based loss operators are straightforward.
MSE loss operator = (ˆy _y+)[2]_ while normalized MSE = ( _yyˆˆ_ _yy++_
_−_ _∥_ _∥_ _[−]_ _∥_ _∥_ [)][2][. When combined with]

negative samples, MSE loss operator (with negative pairs) =ized MSE (with negative pairs) = ( _∥yyˆˆ∥_ _[−]_ _∥yy++∥_ [)][2][ −] [(] _∥yyˆˆ∥_ _[−]_ _∥ (ˆyyy−− −∥_ [)][2]y[.]+)[2] _−_ (ˆy − _y−)[2]_ while normal

-----

**Evaluation of Performance Expectation** To reduce the large search space consists of loss input
_I and loss operator f_, we run 15 trials for each loss operator to get an estimation of performance
expectation. For each of 10 possible f in the search space (5 operators with optional negative
samples), we run 5 trials on each of the 3 pixel-based environments (used in evolution) with the same
loss inputs _st, at_ _st+1_, as we found that forward dynamics is a reasonable representative of
_{_ _} →{_ _}_
our search space with highly competitive performance.

A.2 TRAINING DETAILS

A.2.1 HYPER-PARAMETERS IN THE PIXEL-BASED SETTING

We use the same hyper-parameters for AARL, SAC (no aug), SAC and CURL during the search
phase, to ensure a perfectly fair comparison. When evaluating the searched auxiliary loss, we use a
slightly larger setting (e.g., larger batch size) to train RL agents sufficiently. A full list is shown in
Table 7.

Table 7: Hyper-parameters used in pixel-based environments.

|Hyperparameter|During Evolution|Final Evaluation of AARL-Pixel|
|---|---|---|
|Random crop Observation rendering Observation downsampling Replay buffer size Initial steps Stacked frames Actoin repeat Hidden units (MLP) Hidden units (Predictor MLP) Evaluation episodes Optimizer (β1, β2) for actor/critic/encoder (β1, β2) for entropy α Learning rate for actor/critic Learning rate for encoder Learning for α Batch size for RL loss Batch size for auxiliary loss Q function EMA τ Critic target update freq Convolutional layers Number of filters Non-linearity Encoder EMA τ Latent dimension Discount γ Initial temperature|False for SAC (no aug); True for others (84, 84) for SAC (no aug); (100, 100) for others (84, 84) 100000 1000 3 4 (Cheetah-Run, Reacher-Easy) 2 (Walker-Walk); 1024 256 10 Adam (.9, .999) (.5, .999) 1e-3 1e-3 1e-4 128 128 0.01 2 4 32 ReLU 0.05 50 .99 0.1|True (100, 100) (84, 84) 100000 1000 3 8 (Cartpole-Swingup); 4 (Others) 2 (Walker-Walk, Finger-Spin) 1024 256 10 Adam (.9, .999) (.5, .999) 2e-4 (Cheetah-Run); 1e-3 (Others) 3e-3 (Cheetah-Run, Finger-Spin, Walker-Walk); 1e-3 (Others) 1e-4 512 128 (Walker-Walk) 256 (Cheetah-Run, Finger-Spin) 512 (Others); 0.01 2 4 32 ReLU 0.05 50 .99 0.1|



A.2.2 HYPER-PARAMETERS IN THE STATE-BASED SETTING

We use the same hyper-parameters for AARL, SAC-Identity, SAC-DenseMLP and CURLDenseMLP, shown in Table 8. As training in state-based environments is substantially faster than
pixel-based environments, there is no need to balance training cost and agent performance. We use
this setting for both the search phase and the final evaluation phase.

A.3 EVOLUTION STRATEGY

**Horizon-changing Mutations** There are two kinds of mutations that are able to change horizon length. One is to decrease horizon length. Specifically, we remove the last time step, i.e.,


-----

Table 8: Hyper-parameters used in state-based environments.

|Replay buffer size Initial steps Action repeat Hidden units (MLP) Hidden units (Predictor MLP) Evaluation episodes Optimizer (β, β ) for actor/critic/encoder 1 2 (β, β ) for entropy α 1 2 Learning rate for actor/critic/encoder Learning for α Batch size Q function EMA τ Critic target update freq DenseMLP Layers Non-linearity Encoder EMA τ Latent dimension of DenseMLP Discount γ Initial temperature|100000 1000 4 1024 256 10 Adam (.9, .999) (.5, .999) 2e-4 (Cheetah-Run); 1e-3 (Others) 1e-4 512 0.01 2 1 ReLU 0.05 40 .99 0.1|
|---|---|



(at+k, rt+k, st+k+1) if the target horizon length is k. The other is to increase horizon length, in
which we append three randomly generated bits to the given masks at the end. We do not shorten
the horizon when it becomes too small (less than 1), or lengthen the horizon when it is too long
(exceeding 10).

**Mutating Source and Target Masks** When mutating a candidate, the mutation on the source and
the target masks are independent to each other except for horizon change mutation where two masks
could either both increase horizon or decrease horizon.

**Loss-rejection Protocol** The loss-rejection protocol makes sure that invalid loss functions do not
go into the expensive computation of RL training. Concretely, the following conditions must be
satisfied to make a valid loss function: 1) having at least one state embedding in ˆm to make sure the
gradient of auxiliary loss backward propagates to the state encoder, and 2) target m is not empty. If
a loss is rejected, we repeat the mutation to fill up the population.

**Initialization** At each initialization, we randomly generate 100 auxiliary loss functions (every bit
of masks are generated from Bernoulli(p) where p = 0.5.) and generate 25 auxiliary loss functions
with prior probability, which makes the auxiliary loss have some features like forward dynamics
prediction or reward prediction. The prior probability for generating forward dynamics pattern is:
1) every bit of states from target is generated from Bernoulli(p) where p = 0.2; 2) every bit of
actions from source is generated from Bernoulli(p) where p = 0.8; 3) every bit of states from target
is generated by flipping the states of source; 4) The other bits are generated from Bernoulli(p) where
_p = 0.5. The prior probability for generating reward prediction pattern is: 1) every bit of rewards_
from target is generated from Bernoulli(p) where p = 0.8; 2) Every bit of states and actions from
target are 0; 3) The other bits are generated from Bernoulli(p) where p = 0.5.

A.4 CROSS VALIDATION

After the evolutionary search is done, we choose top-5 candidates for cross-validation across environments. This step is to avoid overfitting to one single environment. For each top-5 loss candidate,
we run 3 trials on each of these 3 training environments (45 runs in total). Then we sort these top-5
candidates by cross-validation overall performance (mean AULC score on three environments) and
retrieve the top-1 candidate as our final searched loss. We do this in pixel-based environments and
state-based environments separately. For pixel-based RL, the top-5 candidates are the top candidates


-----

from stage 5 of Cheetah-Run (Pixel). For state-based RL, the top-5 are the top candidates from stage
4 and 5 of Cheetah-Run (State).

A.5 BASELINES IMPLEMENTATION

**Pixel-based Setting** CURL (Laskin et al., 2020) is the main baseline to compare with in the
pixel-based setting, which is considered to be the state-of-the-art pixel-based RL algorithm. CURL
learns state representations with a contrastive auxiliary loss. PlaNet (Hafner et al., 2019b) and
Dreamer (Hafner et al., 2019a) are model-based methods that generate synthetic rollouts with a
learned world model. SAC+AE (Yarats et al., 2019) uses a reconstruction auxiliary loss of images
to boost RL training. SLAC (Lee et al., 2019) leverages forward dynamics to construct a latent space
for RL agents. Note that there are two versions of SLAC with different gradient Updates per agent
step: SLACv1 (1:1) and SLACv2(3:1). We adopt SLACv1 for comparison since all methods only
make one gradient update per agent step. Pixel SAC are just vanilla SAC (Haarnoja et al., 2018)
agents with images a as inputs respectively.

**State-based Setting** As for the state-based setting, we compare AARL-State with SAC-Identity,
SAC and CURL. SAC-Identity is the vanilla state-based SAC where states are directly fed to actor/critic networks. SAC and CURL use the same architecture of 1-layer densely connected MLP as
a state encoder. Note that both AARL and baseline methods use the same hyper-parameter reported
in Table 8 without additional hyper-parameter tuning.

B FURTHER EXPERIMENT RESULTS

B.1 LEARNING CURVES FOR AARL ON PIXEL-BASED DMCONTROL

We benchmark the performance of AARL to the best-performing pixel-based baseline (CURL). As
shown in Figure 9, the sample efficiency of AARL outperforms CURL on 10 out of 12 environments.
Note that the learning curves of CURL may not match to the data in Table 3, this is because we use
the data reported in CURL paper for tabular while we rerun CURL for learning curves plotting,


where we find the performance of our rerunning CURL is sightly below the CURL paper.

Ball in cup-Catch Cartpole-Swingup Cartpole-Swingup sparse Cheetah-Run

Finger-Spin Finger-Spin hard Hopper-Hop Hopper-Stand

Pendulum-Swingup Reacher-Easy Walker-Stand Walker-Walk

Figure 9: Learning curves of AARL-Pixel and CURL on 12 DMC environments. Shadow represents
the standard deviation over five random seeds. The curves are uniformly smoothed for visual display.
The y-axis represents episodic reward and x-axis represents interaction steps.


-----

B.2 EFFECTIVENESS OF AULC SCORES

To illustrate intuitively why we use the area under learning curve instead of other metrics, we select
top-10 candidates with different evolution metrics. Figure 10 demonstrates the usage of AULC score
could well balance both sample efficiency and final performance. The learning curves of the top-10
candidates selected by AULC score look better than the other two metrics (that select top candidates
simply with 100k step score or 500k step score).


500

400

300

200

100


100k 200k 300k 400k 500k

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||||
|||||100k 500k|
|||||AULC|


100k
500k
AULC

Steps


Figure 10: Learning curves of top-10 loss candidates selected with different metrics.

B.3 HISTOGRAM OF AUXILIARY LOSS ANALYSIS


The histogram of each pattern analysis is shown in Figure 11.

C SEARCH SPACE COMPLEXITY ANALYSIS


The search space size is the size of loss input space multiplied by the size of loss operator space.

For loss input, we calculate separately for each possible horizon length k. When length is k, the
interaction sequence length (st, at, rt, _, st+k) has length (3k + 1). For binary mask ˆm, there are_
_· · ·_
2[3][k][+1] different options. There are also 2[3][k][+1] distinct binary mask m to select targets. Therefore,
there are 2[6][k][+2] combinations when horizon length is fixed to k. As our maximum horizon is 10, we
enumerate k from 1 to 10, resulting in _i=1_ [2][6][i][+2][.]

For loss operator, we can learn intuitively from Table 2 that there are 5 different similarity measures
with or without negative samples, resulting in[P][10] 5 × 2 = 10 different loss operators.

In total, the size of the entire space is


10

2[6][i][+2] _≈_ 4.6 × 10[19].
_i=1_

X


10 ×


-----

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(a) Forward dynamics

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(b) Inverse dynamics

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(c) Reward prediction

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(d) Action inference

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(e) State reconstruction

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(f) State source & target

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(g) Action source & target

Cheetah-Run (Pixel) Reacher-Easy (Pixel) Walker-Walk (Pixel) Cheetah-Run (State) Hooper-Hop (State) Quadruped-Run (State)

(h) Reward source & target

Figure 11: Histogram of statistical analysis of auxiliary loss candidates in 6 evolution process. The
x-axis represents approximated AULC score while the y-axis represents the percentage of corre-
sponding bin of population. Best viewed in color.


-----

D TOP-PERFORMING AUXILIARY LOSSES

D.1 AARL-PIXEL AND AARL-STATE

The auxiliary loss of AARL-Pixel is:

_st+1, at+1, at+2, at+3_ _rt, rt+1, st+2, st+3_ (4)
_{_ _} →{_ _}_

which is the third-best candidate of stage 4 in Cheetah-Run (Pixel).

The auxiliary loss of AARL-State is:

_st, at, at+1, st+2, at+2, at+3, rt+3, at+4, rt+4, at+5, at+7, st+8, at+8, rt+8_
_{_ _}_ (5)
_st+1, st+3, at+4, st+6, st+9_
_→{_ _}_

which is the fourth-best candidate of stage 4 in Cheetah-Run (State).

These two losses are chosen because they are the best-performing loss functions during crossvalidation (Appendix A.4).

D.2 DURING EVOLUTION

We report all the top-5 auxiliary loss candidates during evolution in this section.

Table 9: Top-5 candidates of each stage in Cheetah-Run (Pixel) evolution process


|Col1|Cheetah-Run (Pixel)|
|---|---|
|Stage-1|{rt, st+1, at+1, rt+1, at+2, rt+2, at+3, rt+3} →{st, at, st+2, st+3, st+4} {st, at, rt} →{st+1} {st, at, at+1, rt+2} →{st, at, st+1, at+1, rt+1, st+2, rt+2, st+3} {st, rt, at+1, at+2, at+3, rt+3, rt+4, at+5, rt+5, st+6, st+7} →{st, at, st+1, st+2, rt+2, rt+3, st+4, rt+5, st+6, at+6, st+7} {st, at, st+1, at+1, st+2, rt+2} →{st, st+1, rt+1, st+2, rt+2, st+3}|
|Stage-2|{st, at+1, rt+2, st+4, rt+4} →{st+2, at+3, rt+3, at+4, st+5} {st, at, at+2, rt+2} →{st, rt, st+1, st+2, rt+2} {at, rt, st+1, rt+1, st+2, at+2, rt+2, at+3, at+4} →{st+1, st+2, st+3, at+3, st+4} {st, at, rt, at+1, rt+1, at+2, rt+2, at+3, at+4} →{st, st+1, st+2, st+4, st+5} {rt, st+1, rt+1} →{st, at, at+1, st+2}|
|Stage-3|{st, at, at+2, rt+2} →{st, st+1, st+2, rt+2} {st, rt, at+1, at+3, rt+3, rt+4, at+5, rt+5, st+6, at+6, st+7} →{st, at, st+1, st+2, rt+2, rt+3, st+4, rt+5, st+6, st+7, at+7} {st, at, at+1, rt+1, at+2, rt+2, at+3, st+4, at+4} →{st+1, st+2, st+4, st+5} {st, at, at+1, rt+1, rt+2, st+3, at+3, rt+4} →{st+1, st+2, rt+3, st+4, at+4, st+5} {rt, st+1} →{st, at, at+1, st+2}|
|Stage-4|{st, st+1, at+2, rt+2, st+3, st+4} →{at+1, st+2, rt+2, st+4, at+4, st+5} {st, at, at+1, rt+1, rt+2, st+3, at+3} →{st+1, st+2, rt+3, st+4} {st} →{st, rt, st+1, rt+1, st+2, at+2, rt+2} {st, rt, at+1, st+2, at+2, rt+2, at+3, rt+3, at+4} →{st, at, st+1, rt+1, rt+3, st+4, at+4, st+5} {rt, st+1, at+1} →{st, at, at+1, st+2}|
|Stage-5∗|{at, rt, st+1, at+1, rt+1, at+2, rt+2, at+3, rt+3} →{rt, st+1, at+1, st+2, st+4} {st, at+1, rt+2, at+3, st+4, rt+4} →{st+1, st+2, at+3, st+4, at+4, st+5} †{st+1, at+1, at+2, at+3} →{rt, rt+1, st+2, st+3} {st} →{st, rt, st+1, rt+1, st+2} {st} →{st, rt, st+1, rt+1, st+2, rt+2}|
|Stage-6|{at+1, rt+1, st+2, rt+2, at+3, at+4} →{rt, st+1, rt+1, rt+3, st+4, at+4, st+5} {st, at+1, at+3, st+4, rt+4} →{st+1, st+2, at+3, st+4, at+4, st+5} {st, at+1, rt+1, st+2, at+2, rt+2, st+3, at+3, rt+3} →{at, st+2, rt+2, at+3, st+4, at+4, st+5} {st, at+1, rt+2, at+3, st+4, rt+4} →{st+1, st+2, at+3, st+4, at+4, st+5, at+5} {st, at+1, at+2, rt+2, st+3, at+3, st+4, rt+4} →{st+1, at+1, st+2, rt+2, st+4, at+4, rt+4, st+5}|
|Stage-7|{st, at, rt, at+1, rt+2, at+3, st+4} →{at, rt, st+2, rt+3, st+4, at+4, rt+4, st+5} {st, rt+2, at+3, st+4} →{at, st+1, st+2, rt+2, at+3, rt+3, st+4, at+4, rt+4, st+5} {st+1, at+2, rt+2, st+3, at+3, rt+3, st+4} →{rt, at+1, rt+1, st+2, rt+2, st+4, at+4, st+5} {st, at+1, st+2, at+2, rt+2, st+3, at+3, rt+3, at+4} →{at, st+1, st+2, rt+2, st+3, rt+3, st+4, at+4, st+5} {st, at+1, rt+2, at+3, st+4, rt+4} →{st+1, st+2, rt+2, at+3, st+4, at+4, st+5, at+5}|


_∗: Used for cross-validation. †: AARL-Pixel._


-----

Table 10: Top-5 candidates of each stage in Reacher-Easy (Pixel) evolution process

|Col1|Reacher-Easy (Pixel)|
|---|---|
|Stage-1|{st+1, at+1} →{rt, rt+1} {rt, st+1, at+1, st+2, at+2, rt+2, at+3, at+4, rt+4, st+5, at+5, rt+5, st+6, at+6, rt+6, rt+7, st+8, at+8, st+9} →{at+1, rt+1, st+2, st+3, rt+3, st+4, at+4, at+5, at+6, rt+6, st+7, at+7, rt+7, at+8, rt+8, st+10} {st, rt, st+1, at+2, st+4, at+4, rt+5, st+6, at+6, at+7, st+9, st+10} →{st, st+1, rt+1, st+2, rt+2, rt+3, st+4, at+4, rt+4, st+5, st+6, at+6, rt+7, rt+8, st+10} {st, rt, st+1, at+1, st+2, st+3, at+3, at+4, st+5, rt+5, at+6, rt+6, st+7} →{at+1, st+2, at+2, rt+2, st+3, at+3, rt+3, st+4, at+4, rt+4, st+6, at+6} {st+1, rt+1, at+2, rt+2, rt+3, st+4, at+4, rt+4, st+5, st+6, rt+6} →{rt, st+1, st+2, at+2, rt+2, st+4, st+5, rt+5, st+6, at+6, st+7, st+8}|
|Stage-2|{st, st+1, at+1} →{rt, rt+1} {st, rt, at+2, at+3, st+4, at+4, rt+4, at+5, rt+5, st+6, at+7, at+8} →{rt, at+1, st+2, rt+2, st+3, at+3, rt+3, at+4, rt+4, at+5, rt+5, at+6, st+7, st+9} {st, at, rt, st+1, at+2, at+3, st+5} →{at, st+2, at+2, rt+2, st+3, at+3, at+4, rt+4} {st, at, st+1, at+3, st+4, at+4, at+5, st+6, at+6, at+7, rt+7, st+9, at+9} →{st, at, rt, st+1, rt+1, st+2, rt+3, st+4, rt+4, st+6, at+6, rt+7, at+8, st+10} {st, at, rt, at+1, at+3, rt+3, st+4, at+4, st+5, at+5, rt+5, at+6, rt+6, at+7} →{st+1, st+2, at+2, st+3, st+6, at+6, st+7, st+8}|
|Stage-3|{at, st+1, at+1, st+2, at+2} →{rt, at+1, rt+1, rt+2} {st, at, st+1, st+2, at+2, at+3, st+4, rt+4, at+5, rt+5, rt+6, at+7} →{st, at, rt, st+2, st+3, at+3, st+4, at+4, at+5, rt+5, st+6, st+7, st+8} {st, at, st+1, at+1, rt+1, st+2, at+2, at+3, st+4, rt+6, at+7} →{rt, at+2, rt+2, at+3, rt+3, st+4, at+4, at+5, st+6, at+6, st+7, rt+7, st+8} {st, at, st+1, rt+1, at+2, at+3, st+4, rt+4, st+6, rt+6, at+7} →{st, rt, rt+1, at+2, rt+2, at+3, rt+3, st+4, at+4, at+5, rt+5, st+6, st+7, st+8} {st, at, rt, at+1, rt+1, at+2, st+6, st+7, at+7, st+8} →{st, rt, st+2, at+2, rt+2, at+3, rt+3, st+8, at+8}|
|Stage-4|{st, at, rt, at+1, at+3, rt+3, st+4, st+5, at+5, rt+5, at+6, rt+6, at+7} →{st+1, st+2, st+3, rt+4, rt+5, rt+7, st+8} {at, st+1, at+1, st+2} →{rt, rt+1, at+2, rt+2} {st, at, rt, at+1, rt+3, st+4, st+5, st+6, rt+6, at+7} →{st+1, st+2, at+3, st+4, rt+4, st+5, st+6, st+7, rt+7, st+8} {st, at, rt+1, at+2, at+3, st+4, rt+4, rt+5, st+6, rt+6, at+7} →{st, rt, rt+1, at+2, rt+2, at+3, rt+3, st+4, at+4, at+5, rt+5, st+6, st+7, st+8} {at, st+1, at+1, st+2, at+2} →{rt, rt+1, rt+2}|
|Stage-5|{st, at, at+1, at+2, at+3, st+4, st+5, at+6, rt+6, at+7, st+8} →{rt, at+2, rt+2, at+3, st+4, st+5, at+5, st+6, at+6, st+7, rt+7, st+8} {st, at, st+1, at+1, rt+1} →{rt, st+2} {st, at, rt, at+1, rt+3, st+4, st+5, st+6, rt+6, at+7} →{st+1, st+2, at+3, st+4, rt+4, st+5, rt+6, st+7, rt+7, st+8} {st, at, rt, at+1, rt+1, st+2, at+2, at+3, st+4, st+5, at+5, rt+6, at+7} →{st, rt, at+1, rt+1, at+2, rt+2, st+3, at+3, st+4, st+6, at+6, rt+6, st+7} {st, at, st+1, rt+1, at+2, st+3, st+4, rt+6, at+7, st+8} →{st, rt, rt+1, at+2, rt+2, at+3, st+4, at+4, at+5, st+6, rt+6, st+7, st+8}|
|Stage-6|{st, at, rt, at+1, at+3, rt+3, st+4, at+4, st+5, at+5, rt+5, st+6, at+6, rt+6} →{st+1, st+2, at+2, st+3, rt+4, st+6} {st, at, rt, st+1, at+1, rt+1, st+2, at+2, st+3, at+4, rt+5, rt+6, st+7, st+8} →{st, rt, rt+2, at+3, rt+3, st+4, rt+4, at+5, rt+5, st+6, at+6, rt+6, st+7, at+7, rt+7, st+8} {st, at, rt, rt+1, at+2, at+3, st+4, st+5, rt+5} →{st, rt, rt+1, at+2, at+3, rt+3, at+4, rt+4, at+5, st+6, at+6, rt+6, st+7, st+8} {st, at, st+1, at+3, at+4, at+5, st+6, at+6, st+7, at+7, rt+7, st+9, at+9} →{st, at, rt, st+1, rt+1, st+2, st+3, rt+3, st+4, at+4, rt+4, st+6, at+6, rt+7, at+8, at+9, st+10} {st, at, rt, at+1, rt+1, at+2, at+3, st+4, st+5, at+5, rt+6, at+7} →{rt, st+1, rt+1, at+2, rt+2, st+3, st+4, st+6, at+6, rt+6, st+7, st+8}|
|Stage-7|{st, at, st+1, at+1, rt+1, st+2, at+2, rt+5, rt+6, at+7, st+8} →{rt, at+2, rt+2, rt+3, st+4, at+4, at+5, at+6, rt+6, st+7, st+8} {st, at, at+1, st+2, at+2, at+3, st+4, at+5, rt+5, rt+6, at+7} →{st+1, rt+2, at+3, rt+3, st+4, at+5, rt+5, st+6, at+6, st+7, rt+7} {st, at, rt, at+2, at+3, st+4, st+5, rt+5, at+6} →{st, rt, rt+1, st+2, at+2, at+3, rt+3, rt+4, st+6, st+8} {st, at, rt, at+1, at+3, rt+3, st+4, at+4, st+5, at+5, rt+5, st+6, at+6, rt+6} →{rt, st+1, st+2, at+2, st+3, rt+4, st+6} {st, at, rt, at+1, rt+1, at+3, rt+3, st+4, at+4, st+5, rt+5, st+6, at+7, rt+7} →{st, rt, st+1, rt+1, at+2, st+3, at+3, rt+4, st+6, st+8}|



Table 11: Top-5 candidates of each stage in Walker-Walk (Pixel) evolution process

|Col1|Walker-Walk (Pixel)|
|---|---|
|Stage-1|{st, at, st+2, at+2, rt+2, st+3, at+4, at+5, st+6, at+6, at+7, st+8, rt+8} →{st, st+1, rt+1, st+2, rt+2, rt+3, at+4, rt+4, at+5, rt+5, at+6, rt+6, rt+7, st+8, at+8, st+9} {st, at, at+1, rt+1} →{at, st+1, rt+1} {rt, at+1, st+2, at+2, rt+3, st+5, at+5, at+6, rt+7, at+8} →{st, at, st+1, st+3, at+3, st+4, at+4, st+6, st+7, at+7, st+8, st+9, at+9, st+10} {st, rt, st+1, at+1, st+2, at+2, rt+2, st+3, at+4, rt+4, st+5, at+5, at+6} →{st, at, rt, st+1, st+2, st+3, st+4, rt+4, at+5, rt+5, st+6, st+7} {st, rt, at+1, st+2, at+2, st+3, at+3, at+4, st+5, at+5, rt+5} →{st, at, rt, at+1, rt+1, st+2, st+3, at+3, st+5, at+5, rt+5, st+6}|
|Stage-2|{st, rt, st+1, at+1, rt+1, st+3, at+3, at+4} →{at, rt, st+1, st+2, st+3, rt+3, at+4, st+5} {st, rt, st+2, at+2, rt+2, st+3, rt+3} →{at, st+1, at+1, st+2, at+2, rt+2, rt+3, st+4} {rt, at+1, st+2, rt+2, st+3, rt+3} →{at, st+1, at+1, st+2, at+2, rt+2, at+3, rt+3, st+4} {st, rt, st+2, st+3, rt+3, st+4} →{st, rt, st+1, at+2, rt+2, st+3, st+4, at+4, st+5} {st, rt, st+1, st+2, st+4, at+4, at+5, st+6, rt+6, st+7} →{st, rt, at+1, rt+1, st+2, rt+2, st+3, st+4, at+4, rt+5, st+6, at+7, rt+7, st+8}|
|Stage-3|{st, at, st+1, at+1, rt+1, st+2, rt+2, st+3, at+3, rt+3} →{st+2, at+2, st+3, rt+3, st+4} {st, rt, st+1, at+1, rt+1, st+2, rt+2, at+3, st+4, at+4} →{at, rt, st+1, st+2, at+2, st+3, rt+3, at+4, st+5} {st, at, at+1, rt+1} →{st+2} {st, rt, st+1, at+1, rt+1, st+3, at+3, at+4, rt+4} →{st, at, rt, st+1, st+2, at+2, st+3, rt+3, at+4, st+5} {st, rt, st+2, at+2, rt+2, st+3, rt+3} →{at, st+1, at+1, st+2, at+2, rt+2, rt+3, st+4}|
|Stage-4|{st, at, at+1} →{st+1, at+1, st+2} {st, rt, rt+1, st+2, st+3, rt+3, rt+4} →{st, at, rt, st+1, rt+1, st+2, at+2, st+3, rt+3, st+4, at+4, st+5} {st, st+2, st+3, at+3, rt+3, st+4, at+4} →{st, at, rt, at+2, rt+2, at+4, st+5} {st, rt, st+1, at+1, rt+1, st+2, at+3, st+4, at+4} →{at, rt, st+1, st+2, at+2, rt+3, st+5} {st, rt, st+1, at+1, rt+1, rt+2, st+3, at+3, st+4, at+4} →{at, rt, st+1, st+2, at+2, st+3, rt+3, at+4, st+5}|


-----

Table 12: Top-5 candidates of each stage in Cheetah-Run (State) evolution process

|Col1|Cheetah-Run (Raw)|
|---|---|
|Stage-1|{st, at, rt, at+1, rt+1} →{st+1, st+2} {at, rt, st+2, at+2, at+3, rt+3} →{st, st+1, at+1, st+3, st+4} {at, at+1, st+2, at+2, rt+2, at+3, rt+3, st+4, rt+4, at+5, rt+5, at+7, rt+7, st+8, at+8, rt+8} →{st, st+1, st+3, at+4, st+5, st+6, at+6, st+7, st+9} {at+1, at+2, st+3, at+3, at+4, at+5, rt+5, at+6, rt+7} →{st, at, st+1, st+2, st+4, st+5, st+6, st+7, at+7, st+8} {st, at, at+1, at+2, rt+3, at+4, rt+4, st+5, at+5, at+6, st+7, at+7, st+8, at+8, rt+8} →{st+1, st+2, st+3, at+3, st+4, st+6, st+9}|
|Stage-2|{st, at, rt, at+1, rt+1} →{st+1, st+2} {st, at, rt, at+1, rt+1} →{st+1, st+2} {st, at, at+1, rt+1, at+2, rt+2, at+3, rt+3, at+4, rt+4, st+5, at+5, rt+5, at+6, at+7, at+8, rt+8} →{at, st+1, st+2, at+2, st+3, st+4, st+6, st+9} {st, at, at+1, st+2, at+2, at+3, rt+3, at+4, rt+4, at+5, at+7, st+8, at+8, rt+8} →{st+1, st+3, at+4, st+6, st+9} {st, at, rt} →{rt, st+1}|
|Stage-3|{st, at, rt, at+1, rt+1} →{st+1} {st, at} →{rt, st+1} {st, at, rt, at+1} →{st+2} {st, at, rt} →{st+1} {st, at, at+1, rt+1} →{st, st+1, st+2}|
|Stage-4∗|{st, at, rt, at+1} →{st+1, st+2} {st, at, at+1, rt+1, at+2, rt+2, rt+3, rt+4, st+5, at+5, rt+5, at+6, at+7, at+8, rt+8} →{at, st+1, st+2, at+2, st+3, st+4, st+6, st+9} {st, at, at+1, at+2, rt+2, rt+3, at+4, rt+4, at+5, rt+5, at+6, at+7, at+8, rt+8} →{at, st+1, st+2, at+2, st+3, st+4, st+6, at+8, st+9} †{st, at, at+1, st+2, at+2, at+3, rt+3, at+4, rt+4, at+5, at+7, st+8, at+8, rt+8} →{st+1, st+3, at+4, st+6, st+9} {st, at, at+1, at+2, rt+3, at+4, rt+4, st+5, at+5, st+7, at+7, st+8, at+8, rt+8} →{st+1, st+2, st+3, at+3, st+4, st+6, at+8, rt+8, st+9}|
|Stage-5∗|{st, at, rt, at+1} →{st+1, rt+1} {st, at, at+1, rt+1} →{st+1, at+1, st+2, at+2, rt+2, st+3} {st, at, rt, at+1, rt+1} →{st+1} {st, at, at+1, at+2, rt+2, rt+3, at+4, rt+4, st+5, at+5, rt+5, at+6, at+7, at+8, rt+8} →{st+1, st+2, st+3, st+4, st+6, at+8, st+9} {st, at, rt, at+1} →{st+1, st+2}|
|Stage-6|{st, at, at+1, at+2, rt+2, at+3, rt+3, rt+4, st+5, at+5, rt+5, at+6, at+7, at+8, rt+8} →{st, at, st+1, st+2, at+2, st+3, at+3, at+4, st+5, st+6, at+8, st+9} {st, at, at+1} →{rt, st+1, rt+1, st+2} {st, at, at+1, at+2, rt+2, at+3, rt+3, at+4, rt+4, st+5, at+5, at+6, at+7, st+8, at+8, rt+8} →{at, st+1, st+2, st+3, st+6, at+8, rt+8, st+9} {st, at, rt, at+1} →{st+1, st+2} {st, at, rt, at+1} →{st+1, st+2}|
|Stage-7|{st, at, rt, at+1, rt+1} →{st+1, st+2} {st, at, rt, at+1, rt+1} →{st+1, rt+1, st+2} {st, at, rt, at+1} →{rt, st+1, st+2} {st, at, at+1} →{st+1, at+1, st+2} {st, at, at+1, at+2, rt+2, at+3, rt+3, at+4, rt+4, st+5, at+5, at+6, at+7, at+8, rt+8} →{st+1, st+2, st+3, at+3, at+4, st+6, st+8, st+9}|



_∗: Used for cross-validation. †: AARL-State._

Table 13: Top-5 candidates of each stage in Hopper-Hop (State) evolution process

|Col1|Hopper-Hop (Raw)|
|---|---|
|Stage-1|{st, at} →{rt, st+1} {at, rt, st+2, at+2, rt+2, st+3, at+3, rt+3, st+5, at+5, rt+5, at+6, at+7, rt+7, at+8, rt+8} →{st, st+1, at+1, st+4, at+4, st+6, st+7, st+8, st+9} {st, at, st+2, at+3, rt+4, at+5, rt+5, st+6, at+6, rt+7, rt+8} →{st, rt, st+1, at+1, rt+1, st+2, st+3, st+4, at+6, st+7, at+7, rt+7, at+8, st+9} {st, at, st+2, at+3, rt+3, at+5} →{st, at+1, st+2, st+3, rt+3, st+4, rt+4, rt+5, st+6} {st, rt} →{st, rt, st+1}|
|Stage-2|{st, at, st+1, at+1} →{st+1, st+2} {st, at, rt, st+2, rt+2} →{rt+1, st+2, at+2, st+3} {st, at, st+1, at+1, at+4, st+5, at+5, st+6, at+6} →{st+2, rt+2, rt+3, rt+4, st+5, rt+5, st+6, at+6, rt+6} {rt, at+1, rt+1, st+2, at+2, st+3, at+3, at+4, rt+4} →{st, at, rt, st+1, st+2, st+3, st+4, rt+4} {st, at+1, st+2, at+2, rt+2} →{st+1, rt+1, at+2, st+3}|
|Stage-3|{st, at, st+1, at+1} →{st, st+2} {st} →{st, at, rt, st+1} {st, at, rt, st+2, at+2, rt+2} →{st, st+1, rt+1, st+2, at+2, rt+2, st+3} {st, at, at+1} →{st+1, st+2} {st, at, at+1} →{st, st+1, st+2}|
|Stage-4|{st, rt, st+2, at+2, rt+2} →{rt+1, at+2, st+3} {st, at, st+1, at+1} →{st+2} {st, rt, st+1, at+1, rt+1, st+2, at+2, rt+2} →{st, rt+1, at+2, st+3} {st, at+1, st+2, at+2, rt+2} →{st+1, rt+1, at+2, st+3} {st, at, st+1, at+1} →{st+1, st+2}|
|Stage-5|{st, at, rt, at+1, st+2, at+2, rt+2} →{st, rt+1, at+2, st+3} {st, at+1, rt+1, st+2, at+2, rt+2} →{st, rt+1, st+2, at+2, st+3} {st, at, rt+1} →{st, st+1, st+2} {st, rt, at+1, st+2, rt+2} →{st+1, at+2, st+3} {st, at, st+1, at+1} →{rt+1, st+2}|
|Stage-6|{st, at, at+1} →{st, st+1, st+2} {st, rt, st+2, at+2, rt+2} →{st, st+1, st+2, st+3} {st, at, at+1} →{st, st+1, st+2} {st, at, rt, st+1, at+1, rt+1} →{rt, st+1, st+2} {st, at, rt, at+1, st+2} →{st, st+1, rt+1}|


-----

Table 14: Top-5 candidates of each stage in Quadruped-Run (State) evolution process

|Col1|Quadruped-Run (Raw)|
|---|---|
|Stage-1|{at, rt, st+1, st+2, at+2} →{st, at+1, st+3} {rt, st+1, st+3, rt+3} →{st, at, rt, st+1, at+1, rt+1, st+2, rt+2, st+3, at+3, st+4} {at, at+1, rt+1, st+2, rt+2, st+3, at+3, rt+3} →{st, st+1, at+2, st+4} {st, at, rt+1, at+2, st+3, at+3, rt+3, st+4, st+5} →{at, at+1, rt+1, at+2, rt+3, at+4} {st, at, rt, st+1, at+1, st+3} →{rt+1, rt+2}|
|Stage-2|{at, rt, at+2, rt+2, st+3, at+3, rt+3, at+4} →{st, st+1, at+1, st+2, st+4, st+5} {st, at, at+1, rt+1, rt+2, at+3, at+4, rt+4, at+5, rt+5, st+6, rt+6, at+7, at+8, rt+8, st+9} →{st+1, st+2, at+2, st+3, st+4, st+5, st+7, st+8} {at+1, rt+1, st+2, at+3, rt+3} →{st, at, rt, at+1, at+3, st+4} {at, at+1, st+2, at+2, at+3, at+4, at+5, rt+5, at+6, rt+6, at+7, st+8, at+8} →{st, st+1, st+3, st+4, st+5, st+6, st+7, st+8, st+9} {st, st+1, at+1, st+2, at+2, st+3, st+4} →{st, at, st+1, st+2, at+2}|
|Stage-3|{at, at+1, at+3, rt+3, rt+4, at+5, at+7, rt+7, st+8, at+8} →{st, st+1, st+2, st+3, st+4, at+4, st+5, at+5, rt+5, st+6, at+6, rt+6, st+7, st+9} {at, at+1, at+3, rt+3, at+5, at+7, st+8, at+8} →{st, at, st+1, at+2, st+3, st+4, at+4, st+5, st+6, at+6, st+7, st+9} {at, rt, rt+2, st+3, at+3, rt+3, rt+4} →{st, st+1, at+1, st+2, at+2, at+3, st+4, st+5} {at, at+1, rt+3, at+4, rt+4, at+5, at+7, rt+7, st+8} →{st, rt, st+1, st+3, st+4, st+5, st+6, at+6, st+7, st+8, st+9} {st, at, rt, rt+1, at+2, rt+2, st+3, at+3, rt+3, st+4, st+5} →{at, at+1, rt+1, at+2, rt+3}|
|Stage-4|{rt, rt+1, at+2, rt+2, st+3, at+3, rt+3, st+4, st+5} →{at+2, rt+3, at+4} {st, at, rt+1, at+2, rt+2, st+3, at+3, rt+3, st+4, st+5} →{at, at+1, at+2, rt+3, at+4} {at, at+1, at+3, rt+3, st+4} →{st+1, rt+1, st+2, at+2, rt+2, st+3} {st, at, rt, rt+1, at+2, st+3, at+3, rt+3, st+4, st+5} →{at, at+2, rt+3, at+4} {st+2, at+2, at+3} →{st, at, at+2, st+3, at+3, st+4}|
|Stage-5|{at+1, rt+1, st+2, at+2, rt+2, at+3, rt+3} →{st, at, rt, at+1, rt+1, at+2, st+3, at+3} {at, rt, rt+1, at+2, rt+2, st+3, at+3, rt+3, rt+4} →{st, st+1, at+1, st+2, at+2, at+3, st+4, st+5} {at, at+1, at+3, rt+3, at+4, rt+4, at+5, at+7, rt+7, st+8, at+8} →{st, rt, st+1, st+2, st+3, st+4, st+5, at+5, st+6, at+6, rt+6, st+7, st+8} {at, at+1, st+2, at+3, rt+3, at+4, at+5, st+6, at+7, st+8, at+8} →{st, st+1, st+2, at+2, st+3, st+4, at+4, st+5, st+6, st+7, rt+7, rt+8} {st, at, rt, rt+1, at+2, st+3, at+3, rt+3, st+5} →{at, at+2, rt+3, at+4}|


-----

