# DO WHAT NATURE DID TO US: EVOLVING PLASTIC RECURRENT NEURAL NETWORKS FOR TASK GENER## ALIZATION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

While artificial neural networks (ANNs) have been widely adopted in machine
learning, the gaps between ANNs and biological neural networks (BNNs) are
receiving increasing concern. In this paper, we propose a framework named as
_Evolutionary Plastic Recurrent Neural Networks (EPRNN). Inspired by BNN,_
EPRNN composes Evolution Strategies, Plasticity Rules, and Recursion-based
Learning in one meta-learning framework for generalization to different tasks.
More specifically, EPRNN incorporates nested loops for meta-learning — an outer
loop searches for optimal initial parameters of the neural network and learning
rules; an inner loop adapts to specific tasks. In the inner loop of EPRNN, we
effectively attain both long-term and short-term memory by forging plasticity with
recursion-based learning mechanisms, both of which are believed to be responsible
for memristance in BNNs. The inner-loop setting closely simulates BNNs, which
neither use gradient-based optimization nor require the exact forms of learning
objectives. To evaluate the performance of EPRNN, we carry out extensive experiments in two groups of tasks: Sequence Predicting, and Wheeled Robot Navigating.
The experiment results demonstrate the unique advantage of EPRNN compared to
state-of-the-arts based on plasticity and recursion while yielding comparably good
performance against deep learning-based approaches in the tasks. The experiment
results suggest the potential of EPRNN to generalize to a variety of tasks and
encourage more efforts in plasticity and recursion-based learning mechanisms.

1 INTRODUCTION

ANNs have achieved great success in handling machine learning tasks. Despite being initially
inspired by Biological Neural Networks (BNNs), there are apparent gaps between ANNs and BNNs.
Mainstream ANNs use gradient-based optimizers to minimize learning objectives. Shreds of evidence
show that BNNs learn through plasticity (Gerstner et al., 1993) without explicit learning objectives,
among which Hebb’s rule (Hebb, 1949) is most well known. Though gradient descent methods are
the most efficient optimizers for ANNs, their side effects are also noticed, including the problems
of catastrophic forgetting, over-consumption of data, and the requirement for manual efforts in
designing objective functions. Those challenges are becoming an essential impedance to the further
development of machine intelligence.

Recent studies show the learning mechanisms of BNNs, such as plasticity (Soltoggio et al., 2008;
Najarro & Risi, 2020) and model-based learning (Santoro et al., 2016; Mishra et al., 2018), under
appropriate meta-parameter optimization, can be effective alternative for task generalization in
ANNs. Unlike gradient-based methods, these mechanisms simulate the learning behaviors of BNNs
and don’t require any explicit-form learning objectives. More recently, authors in (Miconi et al.,
2019) proposed a plastic recurrent neural network for lifelong learning of ANNs, where implements
Hebbian plasticity with differentiable objectives and gradient-based optimization. Though the above
studies have investigated learning of ANNs using the two mechanisms derived from BNNs with
gradient-based methods (Miconi et al., 2019) optionally, in this work, we aim at further verify the
path of discovering those rules evolutionarily, simulating that of BNNs.


-----

**Backgrounds. Though learning in BNNs has not been fully understood, some of the learning**
mechanisms and rules, such as plasticity (Gerstner et al., 1993) and recursion (Pollen, 2003), have
been observed in brains and adopted by ANNs. Typically, Model-based learning employs recurrent
neural networks (RNN), LSTM (Hochreiter & Schmidhuber, 1997), and self-attention (Mishra et al.,
2018; Chen et al., 2021) layers as learners. Learning is based on memories within the feed-forward
pass. The information is updated in the hidden states instead of the parameters. Model-based learners
are found to be sample efficient in generalized supervised tasks (Santoro et al., 2016), zero-shot
generalization in language (Brown et al., 2020), and reinforcement learning (Mishra et al., 2018;
Chen et al., 2021) when compared with various type of gradient descent methods. So far, among
model-based learners, though self-attention-based learners such as Transformers have state-of-the-art
performance, the O(T [2]) (where T is the sequence length) makes them only available to relatively
short sequences. On the other hand, recurrent learners such as RNN and LSTM have O(T ) inference
costs but suffer from poor asymptotic performances. That is, when sequences are getting longer,
performances no longer improve or even deteriorate. It is partly due to the limitation of the memory
spaces. For instance, an recurrent neural network of hidden size n has a memory of O(n). In contrast,
its parameters scale with O(n[2]), making parameter-updating more powerful as learning mechanisms
than recursion-only.

Inner-Loop Decision &

Learning Action

Mechanism

Genomes

Observation
& Feedback

**…** Fitness

**Natural Evolution (Outer Loop)** **Individual Lifetime (Inner Loop)**


Figure 1: An illustration of the natural evolution: The evolution takes place in the outer loop, where
the genomes are mutated and selected, and the population either thrive or become extinct based on the
_Fitness. Lifetime of each individual composes the inner loop. At the beginning, the genomes decide_
the learning mechanisms and initial neural configurations of the brain in the new born life. As the
neural networks interact with the environment through actions and observations, its connections and
hidden neuron states are further updated to better adapt to the environment. Plasticity are believed
to be important part of the learning mechanisms. The fitness depends on the learning and adapting
capability of each individual.

In addition to recursion-based learning, evolving plasticity (Soltoggio et al., 2008; 2018; Lindsey
& Litwin-Kumar, 2020; Yaman et al., 2021) has been proposed to reproduce the natural evolution
and plasticity in simulation, as shown in Figure 1. Implementing plasticity is not straightforward;
unlike gradient-based learning methods, plastic rules are not universal but have to be optimized
beforehand, which is not possible without a further outer-loop optimizer over the inner-loop learning.
Evolutionary algorithms (Zhang et al., 2011; Salimans et al., 2017a) are typically applied in the outer
loop to search for meta-parameters shaping the learning rules, which can be regarded as information
carried by genomes during evolution. Those optimized plasticity rules are then applied in the inner
loop to further tune NN’s parameter for better adaptions to the environment. Another line of works
tries to bring gradient-based learning algorithms to plasticity rule optimization (Miconi et al., 2018;
2019). It is found that evolution can be more efficient in cases of long-horizon in reinforcement
learning (Salimans et al., 2017b; Stanley, 2019).

**Our Works. Inspired by the previous works (Cabessa & Siegelmann, 2014; Miconi et al., 2018;**
2019) that improve recursive neural networks using plastic rules for capacity of learning, we propose


-----

|Methods|Inner Loop|Outer Loop|
|---|---|---|
|Memory Augmented NN, (Santoro et al., 2016) MAML, (Finn et al., 2017) Conditional Neural Processes, (Garnelo et al., 2018) SNAIL, (Mishra et al., 2018) ES-MAML, (Song et al., 2019) EPMLP, (Najarro & Risi, 2020) Differential Platicity, (Miconi et al., 2018) Backpropamine, (Miconi et al., 2019) EPRNN (ours)|Recursion Gradient Average Attention Gradient, Hill Climbing Plasticity Plasticity & Recursion Plasticity & Recursion Plasticity & Recursion|Gradient Gradient Gradient Gradient Evolution Evolution Gradient Gradient Evolution|


Table 1: A brief review of meta-learning methods

a novel meta-learning framework namely Evolutionary Plastic Recurrent Neural Networks (EPRNN)
for task generalization. Specifically, this work makes contributions as follows.

_• We study the potential of learning plasticity and recursion rules through the natural evolution_
in task generation. We show that recursion and plasticity-based rules can surpass gradientbased methods as inner-loop learners.

_• We present investigations and analyses on the learned rules and parameters, showing that the_
learning framework discovers plasticity rules that effectively update the connection weights
according to the learning tasks. The differences between the transformation of hidden states
and parameters are also shown, verifying the efficacy of combining recursion with plasticity.

The most relevant works to our study are (Miconi et al., 2018; 2019; Lindsey & Litwin-Kumar, 2020;
Yaman et al., 2021). Compared to (Miconi et al., 2018) that leverage gradient oracles to efficiently
search plastic rules, the proposed EPRNN could work well in gradient-free settings. Compared
to (Lindsey & Litwin-Kumar, 2020; Yaman et al., 2021) that use evolutionary strategies to learn
plastic rules, EPRNN also incorporates an RNN-based inner loop for recursion-based learning. The
work (Miconi et al., 2019) also uses recursion (i.e., RNN) and differentiable plasticity in nested loops
to train self-modifying neural networks, EPRNN replaces the outer loop with evolutionary strategies
to generalize tasks with non-differentiable objectives. Though EPRNN is not as competitive as
gradient-based methods, which can optimize advanced neural networks with large datasets, our work
still demonstrates the potential of using plasticity and recursion for meta-learning through natural
evolution.

2 RELATED WORKS

**Meta-Learning. Meta-learning aims at building learning machines that gain experience using task-**
specific data over the distribution of tasks. Inspired by human and animal brains that are born with
both embedded skills and the capability of acquiring new skills, meta-learning implements two nested
learning loops: The outer learning loops optimize the meta-parameters that typically involves initial
parameters (Finn et al., 2017; Song et al., 2019), learning rules (Zoph & Le, 2017; Najarro & Risi,
2020; Pedersen & Risi, 2021), model structures (Soltoggio et al., 2008; Li & Malik, 2016) and all
of three (Real et al., 2020) over distribution of tasks; The inner learning loops adapt the model to
specific tasks by utilizing those meta-parameters. According to different inner-loop optimizers, we
roughly classify the methods into model-based and parameter-updating methods. The model-based
methods do not update the parameters in the inner-loop, where only hidden states is updated (e.g.,
recursion); The parameter-updating methods re-modify the connection weights in the inner-loop
(e.g., gradient descent (MAML), plasticity). From this point of view, our method can be classified
into both groups. A brief review of the typical meta-learning paradigms is presented in Table 1.

**Plasticity-based Learning. The proposal of the learning mechanism of BNNs is initially raised by**
Hebb’s rule (Hebb, 1949), the most prominent part of which is “neurons fire together wire together”.
It is further polished by Spike Time-Dependent Plasticity (STDP) (Gerstner et al., 1993) indicating
that the signal of learning is dependent on the temporal patterns of the presynaptic spike and postsynaptic spike. Learning could also appear in inhibitory connections, also known as anti-Hebbian


-----

(Barlow, 1989). Also, relationships between STDP and memory are investigated (Linares-Barranco
& Serrano-Gotarredona, 2009). Since many of those rules are related to spiking neural networks
(Ghosh-Dastidar & Adeli, 2009), to apply them to ANNs, simplified rules are proposed (Soltoggio
et al., 2008) instead: given the pre-synaptic neuron state X and post-synaptic neuron state Y, the
connections between X and Y are updated by

_δW = m[A · XY + B · X + C · Y + D],_ (1)

_m is the output from neuron modulators that adjust the learning rates of plasticity. Most of the_
existing rules are sub-classes of Equation 1. For instance, some works neglect the neural modulator
_m (Najarro & Risi, 2020; Miconi et al., 2018), others have set B, C, and D to 0 (Miconi et al., 2018;_
2019). The learned rule A, B, C, D will inevitably be dependent on initial parameter of W, however,
learning plastic rules that is not dependent on the initial parameters was also investigated (Najarro &
Risi, 2020; Yaman et al., 2021).

3 ALGORITHMS

**Problem Settings. We consider an agent (learner) that is dependent on meta-parameter θ. It has the**
capability of adapting itself to a distribution of tasks Tj by interacting with the environment
_Tj through observation it and action at. In K-shot learning, the agent is allowed to first observe ∈T_
samples of length K (this stage can be referred as meta-training-training, see Beaulieu et al. (2020)),
then its fitness is calculated in meta-training-testing rollouts. In Generalized Supervised Learning
tasks (GSL), the observations typically include features (xt) and labels (yt) in meta-training-training
stage (it = {xt, yt}), and the labels are left out for predicting in the meta-training-testing stage
(Santoro et al., 2016; Garnelo et al., 2018). In Generalized Reinforcement Learning tasks (GRL), the
observations typically include states (st), actions (at 1), and feedbacks (rt 1) (it = _st, at_ 1, rt 1,
_−_ _−_ _{_ _−_ _−_ _}_
sometimes rt 1 can not be observed) (Mishra et al., 2018). The goal of meta-training is to optimize
_−_
_θ such that the agent achieves higher fitness in meta-training-testing. In meta-testing, similarly, the_
learned parameters are given meta-testing-training and meta-testing-testing in order, the performances
in meta-testing-testing are evaluated.

**Plastic Recurrent Neural Networks (PRNN). Given a sequence of observations i1, ..., it, ..., we**
first consider an recurrent neural network (RNN) that propagates forward and yields sequence of
outputs at following:

_ht+1 = σ(Wt · ht + Wi · it + b),_ (2)
_at = f_ (ht+1) (3)

where ht is the hidden states at step t. In PRNN, we kept Wi stationary, but we set Wt to be plastic,
so that we add a subscript t to mark different Wt at different steps. Regarding ht as pre-synaptic
neuron states, and ht+1 as post-synaptic neuron states, by applying Equation 1, we update Wt with:

_Wt+1 = Wt + δWt_ (4)

_δWt = WA_ (h[ˆ]t+1 _h[T]t_ [) +][ W][B] _t_ [) +][ W][C] _ht+1_ **1[T]) + WD** _mt_ (5)
_⊙_ _·_ _[⊙]_ [(][m][t] _[·][ h][T]_ _[⊙]_ [(ˆ] _·_ _·_
_hˆt+1 = mt ⊙_ _ht+1,_ (6)

where we use ⊙ and · to represent “element-wise multiplication” and “matrix multiplication” respectively. h and 1 are column vectors. WA, WB, WC, WD are collection of plastic rules of A, B, C, D
from Equation 1, which has the same shape as Wt. mt is the neural modulators that adjusts the
learning rates of plasticity. We calculate mt by applying a neuron modulating layer denoted with:

_mt = σ(Wh[(][m][)]_ _ht + Wi[(][m][)]_ _it + b[(][m][)])._ (7)

_·_ _·_

A sketch of PRNN is presented in Figure 2. The main difference between PRNN and naive RNN is
that PRNN updates both the hidden states and the connection weights during the forward pass.

**Evolving PRNNtraining and meta-training-testing. Given task Tj ∈T, the fitness is eventually dependent on the initial parameters,, by continuously applying Equation 2 to 7 over meta-training-**
learning rules, and the sampled task T, which is denoted as:

_Fit(θ, T_ ) = Fitness(iK+1, aK+1, iK+2, aK+2, ..., ) (8)

_Wi, W0, WA, WB, WC, WD, Wh[(][m][)], Wi[(][m][)], b, b[(][m][)]_ _θ_ (9)
_∈_


-----

Neuron Modulator (𝑚!)

Plasticity rule: 𝑊$[, 𝑊]% [, 𝑊]&[,𝑊]'

ℎ!(# ℎ! ℎ!"# ℎ!")

𝑊!(# 𝑊! 𝑊!"#

Observations (𝑖!)


Figure 2: A sketch of the information flow in plastic recurrent neural networks (PRNN). The red
connections are plastic, the black connections are static.

Following Evolution Strategies (ES) (Salimans et al., 2017a), in kth outer-loop iteration, we sample
different tasks from T, and meta-parameters θk,i (i ∈ [1, n]) from the neighbourhoods of θk. We
evaluate the fitness of sampled meta-parameters, and update the meta-parameters by applying:


_θk+1 = θk + α_ [1]


_Fit(θk,i, Tk)(θk,i_ _θk)_ (10)
_i=1_ _−_

X


**Why Recurrent Neural Networks? As stated in Equation 1, plasticity in feed-forward-only NNs**
allows NNs to gain experiences from single-frame observation only. In cases of non-sequential GSL,
the plasticity has chances to tune the connection weights to the specific task by relying on observing
one single frame of data (it = {xt, yt}), since its information of the feature and the supervision is
complete. However, in general cases, learning can be effective without summarizing sequences of
observations. For instances, a human driver getting used to a new car through continuously interacting
and observing. Moreover, in GRL, there are time lag between the observation of states and feed-backs.
Recursion helps to summarize historical observations to give the correct update for the connection
weights.

Although, compared with naive RNN, there are obviously bunches of more sophisticated neural

_Wheeled Robot Navigating task._


structure such as GRU and LSTM, we believe it is more desirable to start from simplest recurrent
structure to study the potential of combining recursion and plasticity.

2

1 1.0

0.5

0.0

0

0.5

1.0

1

1.5

1.0

0.5

2 0 10 20 30 40 50 2 1 0 1 2 1.51.00.50.0

(a) (b) (c)

Figure 3: Demonstration of the tasks.(a) Two tasks sampled from Sequence Predicting
(l=1,K=25,N=25), the red lines are training sets and the green lines are testing sets. (b) One task
sampled from Sequence Predicting (l=3,K=25,N=25). (c) A trajectory generated by an agent in a


-----

4 EXPERIMENTS

4.1 TASKS FOR EVALUATION

In generalized tasks, we have each task Tj dependent on some configuration parameters that are
hidden from the agent. Below we introduce two groups of generalized tasks that we experiment on.

_Sequence Predicting (Generalized Supervised Learning tasks). We randomly generate sequences of_
vectors y(t) = (y1(t), ..., yl(t)), where yi(t) = Aisin( [2]n[πt]i [+][ φ][i][)][,][ t][ = 1][,][ 2][,][ 3][, ...][ and][ A][i][ ∼U][(1][,][ 3)][,]

_ni_ (10, 100), φi (0, 2π), and (a, b) represents the uniform distribution between a and
_b. A ∼U, n and φ are hidden from the agent. The front part of the sequence ∼U_ _U_ **_y(1), y(2), ..., y(K) is_**
exposed to the agent, and the left part y(K + 1), y(K + 2), ..., y(N ) is to be predicted. The fitness
is the opposite of mean square error (MSE) between the predicted sequence and the ground truth.
We test the methods for comparison in four groups of tasks including (l = 1, K = 10, N = 20),
(l = 1, K = 25, N = 50), (l = 3, K = 10, N = 20), and (l = 3, K = 25, N = 50) (see
Figure 3(a)(b)).

_Wheeled Robot Navigating (Generalized Reinforcement Learning tasks). The agent is to navigate a_
two-wheeled robot to a randomly generated goal in 2-D space g = (gx, gy). We assume that there is
a signal transmitter on the goal and a receiver on the robot. The robot observes the signal intensity
decided byFriis (1946)), where At = A0 − dt is the current distance between the robot and the goal,k · log(dt/d0) + ϵ (inspired by the attenuation for electromagnetic wave, see ϵ ∼N (0, σ) is the
white noise in the observation. gx, gy, A0, k are environment related configurations that is hidden
from the agent. For each task, we sample configurations by gx, gy ( 0.5, 0.5), A0 (0.5, 2.0),
and k ∼U(0.1, 0.5). The action is the rotation speed of its two wheels that controls the orientation ∼U _−_ _∼U_
and velocity of the robot. The reward at each step is rt = −dt, an episode terminates when the
robot approaches the goal or steps reaches the maximum of 100. We also hide its own position and
orientation from the agent, such that the agent relies on recording its own action the signal strength
_At to locate itself. We investigate three types of navigating circumstances with different level of_
noises in the observed signal: Low Noise (σ = 0.01), Median Noise (σ = 0.05), and High Noise
(σ = 0.2) (see Figure 3(c)).

4.2 EXPERIMENT SETTINGS

We add the following methods into comparison, the methods share exactly the same outer loop and
differ in the inner loop.

_• ES-MAML (Song et al., 2019) : We use four gradient descent steps in the inner loop, the_
learning rate of each step is treated as meta-parameters which is to be optimized by the outer
loop. As MAML can not utilize instant observation in zero-shot case, we show results of
both ES-MAML (zero-shot) and ES-MAML (one-rollout) in Wheeled Robot Navigating.
Except for ES-MAML, the other methods are measured with zero-shot meta-testing score
only in Wheeled Robot Navigating.

_• ES-RNN: Vanilla RNN as the inner loop learner._

_• ES-LSTM: Long Short Term Memory (Hochreiter & Schmidhuber, 1997) as the inner loop_
learner.

_• EPMLP (Soltoggio et al., 2018): Multi-Layer Perceptrons (MLP) with plasticity rules_
implemented.

_• EPMLP (Random) (Najarro & Risi, 2020): The main difference with EPMLP is randomly_
setting the parameters of the plastic layers at the beginning of each inner loop instead of
using fixed trainable initial parameters.

_• EPRNN (w/o m): Removing neuron modulator (mt) from the plasticity rule in PRNN._

We add additional non-plastic fully connected layers before and after the plastic layers to increase the
representation capability of the models. For fairness, we kept those layers identical for all compared
methods. For every 100 outer loops in meta training, we add an extra meta testing epoch, evaluating
the average fitness of current meta-parameters on testing tasks. Each run includes 15000 outer


-----

loops and 150 meta-testing epochs. Each result is concluded from independent 3 runs. Our code[1]
relies on PARL[2]for parallelization. We leave the detailed illustration of the model structures and
hyper-parameters in the Appendices.

4.3 RESULTS

We present the experiment results in Figure 4 (In Table 2, and Table 3, we also list the summarized
performance by averaging the Top-3 meta-testing scores in the latest 10 meta-testing epochs of each
run over 3 independent runs). Generally, we can conclude that PRNN performs substantially better
when compared with naive RNN. In some cases, it even produces better results compared with LSTM,
despite the simpler model architecture. It is also interesting to notice that the gap between RNN and
PRNN are smaller in shorter sequences or low-noise environments, but larger in more challenging
tasks with longer sequence or higher noise (In Wheeled Robot Navigating tasks, higher noise pushes
the agent to maintain a longer memory in order to filter the noise and figure out the way to goal). This
phenomenon reaffirms the lack of long-term memories in RNN, and shows that PRNN significantly
improves this drawback. Comparing EPRNN (w/o m) with RNN and EPRNN clearly demonstrates
that simple ABCD rule (without the neural modulator) may also work to some extent, but introducing
the neuron modulator mt can further benefit the learner.

Table 2: Summarized performances comparison in Sequence Predicting tasks

|Methods|l=1,K=10,N=20|l=1,K=25,N=50|l=3,K=10,N=20|l=3,K=25,N=50|
|---|---|---|---|---|


|ES-RNN ES-LSTM ES-MAML EPMLP EPMLP- (Random) EPRNN EPRNN (w/o m)|0.385 0.060 − ± 0.165 0.014 − ± 1.452 0.292 − ± 0.732 0.031 − ± 1.233 0.115 − ± −0.114±0.012 0.135 0.026 − ±|1.228 0.191 − ± 0.283 0.013 − ± 1.747 0.064 − ± 1.185 0.045 − ± 1.319 0.035 − ± −0.208±0.025 0.351 0.058 − ±|1.273 0.009 − ± 1.229 0.010 − ± 1.218 0.013 − ± 1.339 0.004 − ± 1.489 0.023 − ± −1.107±0.015 1.128 0.014 − ±|1.811 0.015 − ± 1.475 0.016 − ± 1.796 0.013 − ± 1.735 0.013 − ± 1.788 0.003 − ± −1.430±0.019 1.520 0.041 − ±|
|---|---|---|---|---|



Table 3: Summarized performances comparison in Wheeled Robot Navigating tasks




|Methods|Low Noise (σ = 0.01)|Median Noise (σ = 0.05)|High Noise (σ = 0.2)|
|---|---|---|---|


|ES-RNN ES-LSTM ES-MAML (zero-shot) ES-MAML (1 rollout) EPMLP EPMLP (Random) EPRNN EPRNN (w/o m)|16.90 1.30 − ± 14.04 0.08 − ± 37.18 0.45 − ± 23.10 0.46 − ± 21.24 6.08 − ± 13.02 0.19 − ± −12.71 ± 0.75 14.93 0.49 − ±|18.99 0.14 − ± 15.50 0.58 − ± 37.91 0.80 − ± 29.21 0.46 − ± 16.65 0.39 − ± 18.25 0.23 − ± −15.07 ± 0.03 16.72 0.33 − ±|31.23 3.95 − ± −22.53 ± 0.45 37.90 0.05 − ± 37.12 0.12 − ± 23.75 2.02 − ± 28.85 1.1 − ± 23.79 0.55 − ± 24.11 0.34 − ±|
|---|---|---|---|


Among plasticity based methods, we show that recursion is more advantageous than MLP in evaluated
tasks. It is also worth noticing that EPMLP and EPMLP (Random) perform steadily beyond the
gradient-based learner (ES-MAML). In ES-MAML, the gradient can only be calculated after an
episode is completed, while EPMLP is able to perform sequential learning even though no feedback
is available during its life time. This demonstrates the possibility of surpassing human-designed
gradient-based learning rules with automatically learned unsupervised rules. Moreover, comparison
between EPMLP and EPMLP (Random) validate the proposal of Najarro & Risi (2020), implying
the possibility of discovering global plastic learning rules instead of rules coupled with the initial

1https://github.com/WorldEditors/EvolvingPRNN
2https://github.com/PaddlePaddle/PARL


-----

parameters. Yet we see optimization of the initial parameters is still advantageous, which can also be
validated by evidences in nature that the newborn lives already have certain embedded skills (e.g.,
Newborn human babies have reflexes of suckling and grasping; Foals can stand shortly after being
born).


0.0

1.25

0.5

1.50

1.0

1.75

1.5

2.00

2.0 EPRNNEPRNN (w\o m) 2.25 EPRNNEPRNN (w\o m)

ES_RNN ES_RNN

2.5 ES_LSTMEPMLP 2.50 ES_LSTMEPMLP

EPMLP (RANDOM) EPMLP (RANDOM)

3.0 ES_MAML 2.75 ES_MAML

0 2000 4000 6000 8000 10000 12000 14000 0 2000 4000 6000 8000 10000 12000 14000


(a) Sequence Predicting
(l=1,K=10,N=20)

1.4

0.5 1.6

1.8

1.0

2.0

1.5 2.2

2.0 EPRNN 2.4 EPRNN

2.5 EPRNN (w\o m)ES_RNNES_LSTM 2.6 EPRNN (w\o m)ES_RNNES_LSTM

EPMLP 2.8 EPMLP
EPMLP (RANDOM) EPMLP (RANDOM)

3.0 ES_MAML 3.0 ES_MAML

0 2000 4000 6000 8000 10000 12000 14000 0 2000 4000 6000 8000 10000 12000 14000


(c) Sequence Predicting
(l=1,K=25,N=50))


(b) Sequence Predicting
(l=3,K=10,N=20)

EPRNN
EPRNN (w\o m)
ES_RNN
ES_LSTM
EPMLP

ES_MAML

2000 4000 6000 8000 10000 12000

(d) Sequence Predicting
(l=3,K=25,N=50)


10 20

15

25

20 20

30

30 25

35

30

40

EPRNN 35 EPRNN 40 EPRNN
EPRNN (w\o m) EPRNN (w\o m) EPRNN (w\o m)

50 ES_RNNES_LSTM 40 ES_RNNES_LSTM 45 ES_RNNES_LSTM

EPMLP EPMLP EPMLP

60 EPMLP (RANDOM)ES_MAML (1 roullout) 45 EPMLP (RANDOM)ES_MAML (1 roullout) 50 EPMLP (RANDOM)ES_MAML (1 roullout)

0 2000 4000 6000 8000 10000 12000 14000 0 2000 4000 6000 8000 10000 12000 14000 0 2000 4000 6000 8000 10000 12000 14000


(e) Wheeled Robot Navigating
(Low Noise)


(f) Wheeled Robot Navigating
(Median Noise)


(g) Wheeled Robot Navigating
(High Noise)


Figure 4: Plotting meta-testing scores against meta-training iterations.

4.4 ANALYSIS AND DISCUSSION

To investigate whether plasticity rules update the connection weights as expected, we test the trained
model with different tasks and record the updating trajectories of plastic connection weights Wt
and hidden states ht. We run t-SNE visualization to map those tensors (Wts and hts) to 2-D space
and show their trajectories in Figure 5. The connection weights Wt typically start from the same
position and gradually move in different directions depending on the task configurations. The final
weights effectively capture environmental configuration that was hidden from the agent. Particularly,
in Figure 5(d) for Wheeled Robot Navigating tasks, we can see that Wt captures only the signal
transmission patterns (A0, k), but neglects the position of the goal (gx, gy). We guess that A0, k
are important stationary patterns that helps the agent to interpret the observed signal strength (At),
while the absolute position of goal is less important as its relative position to the robot is changing
continuously. This demonstrates that plasticity has performed meaningful updates on the weights of
the connections depending on the tasks. On the other hand, the hidden states (Figure 5(c)(d)(e)(h)) are
noisier and less distinguishable among different tasks. The small repeated circles in the trajectory of
hidden states might correspond to short-term patterns such as the periodicity of Sequence Predicting
task and the circular movements in Wheeled Robot Navigation tasks. (see Appendices A.2)


-----

10 A=1.0,n=50,phi=0.0*piA=1.5,n=50,phi=0.0*pi 10 A=2.0,n=20,phi=0.0*piA=2.0,n=30,phi=0.0*pi 7.5
A=2.0,n=50,phi=0.0*pi A=2.0,n=40,phi=0.0*pi

5 A=2.5,n=50,phi=0.0*piA=3.0,n=50,phi=0.0*pi 5 A=2.0,n=50,phi=0.0*piA=2.0,n=60,phi=0.0*piA=2.0,n=70,phi=0.0*pi 5.0

A=2.0,n=80,phi=0.0*pi 2.5

0

0 0.0

5 2.5

5 10 5.0 A=2.0,n=50,phi=0.0*piA=2.0,n=50,phi=0.2*piA=2.0,n=50,phi=0.4*pi

7.5 A=2.0,n=50,phi=0.6*pi

10 A=2.0,n=50,phi=0.8*pi

15 10.0 A=2.0,n=50,phi=1.0*pi

2 0 2 4 6 10 5 0 5 10 12 10 8 6 4 2 0 2 4


(a) Wt for different A

A=1.0, n=50, phi=0.0*PI 20 10 A=2.0, n=50, phi=0.2*PI

20 A=1.5, n=50, phi=0.0*PI A=2.0, n=50, phi=0.4*PI

A=2.0, n=50, phi=0.0*PIA=2.5, n=50, phi=0.0*PIA=3.0, n=50, phi=0.0*PI 15 5 A=2.0, n=50, phi=0.6*PIA=2.0, n=50, phi=0.8*PIA=2.0, n=50, phi=1.0*PI

10 10

A=2.0, n=20, phi=0.0*PI

5 A=2.0, n=30, phi=0.0*PIA=2.0, n=40, phi=0.0*PI 0

0 0 A=2.0, n=50, phi=0.0*PIA=2.0, n=60, phi=0.0*PI

A=2.0, n=70, phi=0.0*PI

5 A=2.0, n=80, phi=0.0*PI 5

10 10

10

15

20

15 10 5 0 5 10 15 20 15 10 5 0 5 10 15 20 20 10 0 10 20


(d) ht for different A


(b) Wt for different n

A=2.0, n=20, phi=0.0*PI
A=2.0, n=30, phi=0.0*PI
A=2.0, n=40, phi=0.0*PI
A=2.0, n=50, phi=0.0*PI
A=2.0, n=60, phi=0.0*PI
A=2.0, n=70, phi=0.0*PI
A=2.0, n=80, phi=0.0*PI

15 10 5 0 5 10 15

(e) ht for different n


(c) Wt for different φ

A=2.0, n=50, phi=0.2*PI
A=2.0, n=50, phi=0.4*PI
A=2.0, n=50, phi=0.6*PI
A=2.0, n=50, phi=0.8*PI
A=2.0, n=50, phi=1.0*PI

20 10 0 10

(f) ht for different φ


200 g=(0.3, 0.3), A=0.5, k=0.1
g=(0.3, 0.3), A=2.0, k=0.1g=(0.3, 0.3), A=0.5, k=0.5 30

150 g=(0.3, 0.3), A=2.0, k=0.5

100 g=(-0.3, -0.3), A=0.5, k=0.1g=(-0.3, -0.3), A=2.0, k=0.1g=(-0.3, -0.3), A=0.5, k=0.5 20

g=(-0.3, -0.3), A=2.0, k=0.5 10

50

0

0 10 g=(0.3, 0.3), A=0.5, k=0.1g=(0.3, 0.3), A=2.0, k=0.1

50 g=(0.3, 0.3), A=0.5, k=0.5

20 g=(0.3, 0.3), A=2.0, k=0.5g=(-0.3, -0.3), A=0.5, k=0.1

100 g=(-0.3, -0.3), A=2.0, k=0.1

150 30 g=(-0.3, -0.3), A=0.5, k=0.5g=(-0.3, -0.3), A=2.0, k=0.5

300 200 100 0 100 200 300 30 20 10 0 10 20 30


(g) Wt for different A0, k, and
goals


(h) ht for different A, k, and
goals


Figure 5: t-SNE visualization of the transformation of the connection weights (Wt) and hidden states
(ht) in the inner loop. (a)(b)(c)(d)(e)(f) are from Sequence Predicting (l=1,K=25,N=50) task, (g)(h)
are from Wheeled Robot Navigating (Low Noise) task.

Note that we did not compare EPRNN with many gradient-based solutions such as differentiable
plasticity (Miconi et al., 2018; 2019), since the classic Hebbian plasticity has been observed and
confirmed from neurons for more than half century (Hebb, 1949) — there is not evidences that
the nature do differentiation or backpropagation (Lillicrap et al., 2020) to update plastic rules. The
purpose of our work is to reveal the potential of natural evolution of plasticity, and we don’t hope to
claim that EPRNN could outperform gradient-based solutions on machine learning tasks.

5 CONCLUSIONS

In this paper we present EPRNN a nature-inspired learning framework composed of Evolution
Strategies, Plastic rules, and Recursion. Experiment results show that plasticity can be effectively
forged with recursion to enhance the learning capability. The proposed framework can achieve
equivalent or even better performances compared with more sophisticated neural structures, by
applying the simplest recurrent neural structures. Moreover, we also show that under proper meta
parameters, plasticity has a chance to surpass gradient descent methods.

We believe the learning framework of Figure 1 can be extended to more sophisticated plastic rules
and model structures, uncovering better learners in the future. Also, it would be more interesting if
such framework can be validated in more complex environments such as natural language processing
tasks and vision-related tasks. Finally, we are looking forward that this work can shed light to new
paradigm of building intelligent machines and inspire more efforts in this line of research.


-----

REFERENCES

Horace B Barlow. Adaptation and decorrelation in the cortex. The computing neuron, 1989.

Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O Stanley, Jeff Clune, and Nick
Cheney. Learning to continually learn. In ECAI 2020, pp. 992–1001. IOS Press, 2020.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

Jeremie Cabessa and Hava T Siegelmann. The super-turing computational power of plastic recurrent
neural networks. International journal of neural systems, 24(08):1450029, 2014.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel,
Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence
modeling. arXiv preprint arXiv:2106.01345, 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of
deep networks. In International Conference on Machine Learning, pp. 1126–1135. PMLR, 2017.

Harald T Friis. A note on a simple transmission formula. Proceedings of the IRE, 34(5):254–256,
1946.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray
Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In
_International Conference on Machine Learning, pp. 1704–1713. PMLR, 2018._

Wulfram Gerstner, Raphael Ritz, and J Leo Van Hemmen. Why spikes? hebbian learning and retrieval
of time-resolved excitation patterns. Biological cybernetics, 69(5):503–515, 1993.

Samanwoy Ghosh-Dastidar and Hojjat Adeli. Spiking neural networks. International journal of
_neural systems, 19(04):295–308, 2009._

Donald Olding Hebb. The organization of behavior; a neuropsycholocigal theory. A Wiley Book in
_Clinical Psychology, 62:78, 1949._

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735–1780, 1997.

Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885, 2016.

Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335–346, 2020.

Bernabé Linares-Barranco and Teresa Serrano-Gotarredona. Memristance can explain spike-timedependent-plasticity in neural synapses. Nature precedings, pp. 1–1, 2009.

Jack Lindsey and Ashok Litwin-Kumar. Learning to learn with feedback and local plasticity. arXiv
_preprint arXiv:2006.09549, 2020._

Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural
networks with backpropagation. In International Conference on Machine Learning, pp. 3559–3568.
PMLR, 2018.

Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training
self-modifying neural networks with differentiable neuromodulated plasticity. In 7th International
_Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019._
[OpenReview.net, 2019. URL https://openreview.net/forum?id=r1lrAiA5Ym.](https://openreview.net/forum?id=r1lrAiA5Ym)

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive metalearner. In International Conference on Learning Representations, 2018.

Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks.
_arXiv preprint arXiv:2007.02686, 2020._


-----

Joachim Winther Pedersen and Sebastian Risi. Evolving and merging hebbian learning rules:
Increasing generalization by decreasing the number of rules. arXiv preprint arXiv:2104.07959,
2021.

Daniel A Pollen. Explicit neural representations, recursive neural networks and conscious visual
perception. Cerebral Cortex, 13(8):807–814, 2003.

Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning
algorithms from scratch. In International Conference on Machine Learning, pp. 8007–8019.
PMLR, 2020.

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a
scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017a.

Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative
[to reinforcement learning. CoRR, abs/1703.03864, 2017b. URL http://arxiv.org/abs/](http://arxiv.org/abs/1703.03864)
[1703.03864.](http://arxiv.org/abs/1703.03864)

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Metalearning with memory-augmented neural networks. In International conference on machine
_learning, pp. 1842–1850. PMLR, 2016._

Andrea Soltoggio, John A Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Floreano. Evolutionary
advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In Proceedings of
_the 11th international conference on artificial life (Alife XI), number CONF, pp. 569–576. MIT_
Press, 2008.

Andrea Soltoggio, Kenneth O Stanley, and Sebastian Risi. Born to learn: the inspiration, progress,
and future of evolved plastic artificial neural networks. Neural Networks, 108:48–67, 2018.

Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao
Tang. Es-maml: Simple hessian-free meta learning. In International Conference on Learning
_Representations, 2019._

Kenneth O Stanley. Why open-endedness matters. Artificial life, 25(3):232–235, 2019.

Anil Yaman, Giovanni Iacca, Decebal Constantin Mocanu, Matt Coler, George Fletcher, and Mykola
Pechenizkiy. Evolving plasticity for autonomous learning under changing environmental conditions.
_Evolutionary computation, 29(3):391–414, 2021._

Jun Zhang, Zhi-hui Zhan, Ying Lin, Ni Chen, Yue-jiao Gong, Jing-hui Zhong, Henry SH Chung,
Yun Li, and Yu-hui Shi. Evolutionary computation meets machine learning: A survey. IEEE
_Computational Intelligence Magazine, 6(4):68–75, 2011._

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th
_International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26,_
_2017, Conference Track Proceedings, 2017._


-----

A APPENDICES

A.1 MODEL ARCHITECTURE AND TRAINING DETAILS

To maintain fairness for comparison, the model architectures of the compared methods are similar,
where the differences lie in only one of their layers (Figure 6). We use 3 hidden layers for Sequence
_Predicting tasks and 4 hidden layers for Wheeled Robot Navigating tasks. For ES-MAML, we_
replace the PRNN layer with fully connected layer; For ES-RNN and ES-LSTM, we replace it with
_RNN and LSTM respectively; For EPMLP, we replace it with plastic fully connected layer, where_
the plasticity rule is stated by Equation 1, and the neural modulator m is calculated by an additional
dense layer with sigmoid activation. The hidden sizes of all the hidden layers are 64 (for LSTM it is
64 hidden states and 64 cell states).


2

output

64

Fully Connected + RELU Fully Connected + RELU

64

**PRNN** RNN

64

Fully Connected + RELU LSTM

64

Fully Connected + RELU Plastic Fully Connected

3

input


𝑙

output

64

Fully Connected + RELU Fully Connected + RELU

64

**PRNN** RNN

64

LSTM

Fully Connected + RELU

𝑙 Plastic Fully Connected

input


(a) Sequence Predicting


(b) Wheeled Robot Navigating


Figure 6: A sketch of the model architectures for evaluated tasks

For Sequence Predicting tasks, the input observation and the output action has the dimension of l.
In the meta-training-training and meta-testing-train stages, we use the ground truth yt 1 as input
_−_
observation; In the the meta-training-test and meta-testing-test stages, its previous action (at−1) are
taken as inputs. For Wheeled Robot Navigating tasks, the output action is the control command of its
two wheels (a length 2 vector), the input observation is the concatenation of its previous action and
the current observed signal intensity (At).

- Running Inner-Loop

- 𝐹𝑖𝑡𝜃#,( = "! [∑𝐹𝑖𝑡(𝜃]) [#,(][, 𝑇])[)]

…

Worker 𝑖−1 Worker 𝑖 Worker 𝑖+ 1 Worker 𝑛

𝐹𝑖𝑡𝜃!,#

{𝑇![, …, 𝑇]"[}][,][ 𝜃]#,(&!

- Sampling Tasks {𝑇![, …, 𝑇]"[}]

- In the Neighborhoods of 𝜃# Sample Meta-Parameter 𝜃#,!, 𝜃#,%, …,

Server - Update with: 𝜃#&! = 𝜃# + 𝛼 '! [∑(𝜃]( [#,(][ −𝜃][#][)] 𝐹𝑖𝑡𝜃#,(


Figure 7: A sketch of the parallel training framework.

The training process is accelerated by utilizing the paralleling mechanism of PARL. We employ
400 workers (400 Intel(R) Xeon(R) CPU E5-2650) running inner-loops for 400 off-springs in
each generation of the evolution, and additional 1 CPU to perform evolutionary update (shown in
Figure 7(b)). It takes 3 to 12 hours for each run depending on the length of the inner loop and
model architectures. Following the previous work (Salimans et al., 2017a), we rank normalize the
fitness among 400 workers. The learning rate α is set to 0.2. The mutation is performed by adding
independent Gaussian noises to each parameter. During the meta-training, we sample B different
tasks and 400 meta-parameters. Each worker run B meta-training-training and B meta-training_testing given the assigned meta-parameter, then the fitness is averaged. We set B = 16 for Sequence_


-----

_Predicting tasks, and B = 4 for Wheeled Robot Navigating tasks. For each meta-testing epoch, we_
evaluated the current meta parameters in newly sampled 1600 tasks.

A.2 SUPPLEMENTARY EXPERIMENTAL RESULTS AND ANALYSES

We show more t-SNE visualization of the connection weights and hidden states in Figure 8 which can
be compared with Figure 5. The transformation of hidden states in ES-RNN (Figure 8(a)(b)(c) and (e))
are even less distinguishable among different tasks compared with that of EPRNN (Figure 5(d)(e)(f)
and (h)), showing naive RNNs fail to capture task configurations. The weights of EPMLP can also
capture some hidden configurations (Figure 8(d)) but are less accurate compared with that of EPRNN
(Figure 5(g)). For instance, the gold and grey line share the same hidden configuration of A = 0.5
and k = 0.5. The connection weights effectively captures this information and the two lines overlap
with each other in EPRNN, but it does not happen in EPMLP. Similar phenomenon can be observed
for the red and the green lines (A = 0.5 and k = 0.1).

|A=1.0, n=50, phi=0.0*PI A=1.5, n=50, phi=0.0*PI A=2.0, n=50, phi=0.0*PI A=2.5, n=50, phi=0.0*PI A=3.0, n=50, phi=0.0*PI|Col2|
|---|---|
|A=1.0, n=50, phi=0.0 A=1.5, n=50, phi=0.0 A=2.0, n=50, phi=0.0 A=2.5, n=50, phi=0.0 A=3.0, n=50, phi=0.0||
|||


A=1.0, n=50, phi=0.0*PI
A=1.5, n=50, phi=0.0*PI
A=2.0, n=50, phi=0.0*PI
A=2.5, n=50, phi=0.0*PI
A=3.0, n=50, phi=0.0*PI


15 A=2.0, n=20, phi=0.0*PI 15 A=2.0, n=50, phi=0.2*PI
A=2.0, n=30, phi=0.0*PI A=2.0, n=50, phi=0.4*PI
A=2.0, n=40, phi=0.0*PI A=2.0, n=50, phi=0.6*PI

10 A=2.0, n=50, phi=0.0*PIA=2.0, n=60, phi=0.0*PI 10 A=2.0, n=50, phi=0.8*PIA=2.0, n=50, phi=1.0*PI

A=2.0, n=70, phi=0.0*PI
A=2.0, n=80, phi=0.0*PI 5

5

0

0 5

10

5

15

30 20 10 0 10 20 30 15 10 5 0 5 10 15


(a) ht for different A in ES-RNN
agent


(b) ht for different n in ES-RNN
agent


(c) ht for different φ in ES-RNN
agent


g=(0.3, 0.3), A=0.5, k=0.1

30 g=(0.3, 0.3), A=2.0, k=0.1

g=(0.3, 0.3), A=0.5, k=0.5

400 g=(0.3, 0.3), A=2.0, k=0.5

20 g=(-0.3, -0.3), A=0.5, k=0.1

g=(-0.3, -0.3), A=2.0, k=0.1

g=(0.3, 0.3), A=0.5, k=0.1 g=(-0.3, -0.3), A=0.5, k=0.5

200 g=(0.3, 0.3), A=2.0, k=0.1g=(0.3, 0.3), A=0.5, k=0.5 10 g=(-0.3, -0.3), A=2.0, k=0.5

g=(0.3, 0.3), A=2.0, k=0.5
g=(-0.3, -0.3), A=0.5, k=0.1 0

0 g=(-0.3, -0.3), A=2.0, k=0.1g=(-0.3, -0.3), A=0.5, k=0.5g=(-0.3, -0.3), A=2.0, k=0.5 10

200 20

30

400

600 400 200 0 200 400 600 40 30 20 10 0 10 20 30 40


(d) Wt for different A0, k, and goals
in EPMLP agent


(e) ht for different A0, k, and goals
in ES-RNN agent


Figure 8: Supplementary t-SNE visualization of the transformation of the connection weights (Wt)
and hidden states (ht) in the inner loop. (a)(b)(c) are from Sequence Predicting (l=1,K=25,N=50)
task, (d)(e) are from Wheeled Robot Navigating (Low Noise) task.

We also plot the trajectories of the robot in Wheeled Robot Navigation tasks. All the agents struggle
to find the goal by continuously shifting their directions, which is more frequent in cases of higher
observation noise. A reasonable guess is that in higher noise cases, the agents are less confident in
their estimations. They have to more frequently adjust their directions to acquire diverse observations
for more effective denoising.


-----

(a) EPRNN (Low Noise) (b) EPRNN (Median Noise) (c) EPRNN (High Noise)


(d) EPRNN (w/o m)
(Median Noise)


(e) EPMLP (Median Noise) (f) ES-RNN (Median
Noise)


(g) ES-LSTM (Median
Noise)


(h) Random Controller


Figure 9: Trajectories of the agents from Wheeled Robot Navigating task with the task configuration
of (A = 0.5, k = 0.2, (gx, gy) = ( 0.3, 0.3)). We show the trajectories of the wheeled robot in
_−_ _−_
black and red lines and the goal in green circles.


-----

