# TRGP: TRUST REGION GRADIENT PROJECTION FOR CONTINUAL LEARNING

**Sen Lin[1], Li Yang[1], Deliang Fan[1], Junshan Zhang[1,2]**

1School of ECEE, Arizona State University, 2Department of ECE, University of California, Davis
_{slin70, lyang166, dfan}@asu.edu, jazh@ucdavis.edu_

ABSTRACT

Catastrophic forgetting is one of the major challenges in continual learning. To address this issue, some existing methods put restrictive constraints on the optimization space of the new task for minimizing the interference to old tasks. However,
this may lead to unsatisfactory performance for the new task, especially when the
new task is strongly correlated with old tasks. To tackle this challenge, we propose
Trust Region Gradient Projection (TRGP) for continual learning to facilitate the
forward knowledge transfer based on an efficient characterization of task correlation. Particularly, we introduce a notion of ‘trust region’ to select the most related
old tasks for the new task in a layer-wise and single-shot manner, using the norm
of gradient projection onto the subspace spanned by task inputs. Then, a scaled
weight projection is proposed to cleverly reuse the frozen weights of the selected
old tasks in the trust region through a layer-wise scaling matrix. By jointly optimizing the scaling matrices and the model, where the model is updated along the
directions orthogonal to the subspaces of old tasks, TRGP can effectively prompt
knowledge transfer without forgetting. Extensive experiments show that our approach achieves significant improvement over related state-of-the-art methods.

1 INTRODUCTION

Human beings can continuously learn different new tasks without forgetting the learnt knowledge of
old tasks in their lifespan. Aiming to achieve this remarkable capability for the deep neural networks
(DNNs), continual learning (CL) (Chen & Liu, 2018) has garnered much attention in recent years.
Nevertheless, many existing CL methods still leave the DNN vulnerable to forget the knowledge
of old tasks when learning new tasks. Such a phenomenon is known as ‘Catastrophic Forgetting’
(McCloskey & Cohen, 1989), which has become one of the major challenges for CL.

Many approaches (e.g., (Rusu et al., 2016; Li & Hoiem, 2017; Dhar et al., 2019; Guo et al., 2020;
Zeng et al., 2019)) have been proposed to address the forgetting issue, which can be generally
divided into two classes depending on the network architecture, i.e., expansion methods and nonexpansion methods. In order to understand the fundamental limit of a fixed capacity neural network,
we focus on non-expansion methods in this work. The basic idea for non-expansion methods is
to constrain the gradient update either explicitly or implicitly when learning the new task, so as to
minimize the introduced interference to old tasks. For example, the regularization-based methods
(e.g., (Kirkpatrick et al., 2017; Serra et al., 2018)) penalize the modification on the most important
weights of old tasks through model regularizations; experience-replay based methods (e.g., (Shin
et al., 2017; Chaudhry et al., 2019)) constrain the gradient directions by replaying the data of old
tasks during learning of new tasks, in the format of either real data or synthetic data from generative
models; and orthogonal-projection based methods (e.g., (Farajtabar et al., 2020; Saha et al., 2021))
update the model with gradients in the orthogonal directions of old tasks, without the access to old
task data. In particular, the recently proposed Gradient Projection Memory (GPM) (Saha et al.,
2021) has demonstrated superior performance compared to other approaches.

To sufficiently minimize the interference to old tasks, most existing non-expansion methods (particularly the orthogonal-projection based methods), often put restrictive constraints on the optimization
space of the new task, which may throttle the learning performance for the new task. A plausible
conjecture is that such a scenario is likely to occur when the new task is strongly correlated with


-----

old tasks, and in this study we provide evidence to support this conjecture. The underlying rationale
is as follows: The weights that are important to the new task are also important to the old tasks
strongly correlated with the new task, which are often frozen to address the forgetting in the existing
methods; however, they should be updated in the learning of the new task.

To tackle this challenge, a key insight is that for a new task that is strongly correlated with old tasks,
although the model optimization space could be more restrictive, there should be better forward
_knowledge transfer from the correlated old tasks to the new task. With this insight, we propose an_
innovate continual learning approach to facilitate the forward knowledge transfer without forgetting.
The main contributions can be summarized as follows:

(1) Inspired by (Schulman et al., 2015), we introduce a novel notion of ‘trust region’ based on the
norm of gradient projection onto the subspace spanned by task inputs, which selects the old tasks
strongly correlated to the new task in a layer-wise and single-shot manner. Intuitively, the new task
and the selected old tasks in the trust region have similar input features for the corresponding layer.

(2) We propose a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the trust region through a scaled weight projection. Particularly, a scaling matrix
is learnt in each layer for the new task to scale the weight projection onto the subspace of old tasks
in the trust region, in order to reuse the frozen weights of old tasks without modifying the model.

(3) Building on the introduced trust region, scaled weight projection, and a module to construct task
input subspace, we develop a continual learning approach, trust region gradient projection (TRGP),
that jointly optimizes the scaling matrices and the model for the new task. To mitigate the forgetting
issue further, the model is updated along the directions orthogonal to the subspaces of old tasks.

(4) We evaluate TRGP on standard CL benchmarks using various network architectures. Compared
to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all
benchmarks, and demonstrates universal improvement on all tasks. The superior performance indicates that TRGP can effectively promote the forward knowledge transfer while alleviating forgetting.

2 RELATED WORK

**Expansion-based methods. Expansion-based methods (e.g., (Rusu et al., 2016; Li & Hoiem, 2017;**
Rosenfeld & Tsotsos, 2018; Hung et al., 2019; Yoon et al., 2017; Li et al., 2019; Veniat et al., 2020))
dynamically expand the network capacity to reduce the interference between the new tasks and the
old ones. Progressive Neural Network (PNN) (Rusu et al., 2016) expands the network architecture
for new tasks and preserves the weights of old tasks. Learning Without Forgetting (LWF) (Li &
Hoiem, 2017) splits the model layers into two parts, i.e., the shared part co-used by all tasks, and
the task-specific part which grows for new tasks. Dynamic-Expansion Net (DEN) (Yoon et al.,
2017) and Compacting-Picking-Growing (CPG) (Hung et al., 2019) combine the strategies of model
compression/pruning, weight selection and model expansion. In order to find the optimal structure
for each of the sequential tasks, Reinforced Continual Learning (RCL) (Xu & Zhu, 2018) leverages
reinforcement learning and (Li et al., 2019) adapts architecture search. APD (Yoon et al., 2020) adds
additional task-specific parameters for each task and selectively learns the task-shared parameters.

**Regularization-based methods. This category of methods (e.g., (Kirkpatrick et al., 2017; Lee et al.,**
2017; Chaudhry et al., 2018a; Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al.,
2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model
change on their important weights. Notably, to determine the weight importance, Elastic Weight
Consolidation (EWC) (Kirkpatrick et al., 2017) leverages Fisher information matrix, HAT (Serra
et al., 2018) learns hard attention masks. MAS (Aljundi et al., 2018) evaluates the model outputs
sensitivity to the inputs in an unsupervised manner.

**Memory-based methods. Depending on if data of old tasks is utilized when learning new tasks,**
memory-based methods can be further divided into the following two categories. 1) Experience_replay based methods. This class of methods replays the old tasks data along with the current task_
data to mitigate catastrophic forgetting. Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato,
2017) and Averaged GEM (A-GEM) (Chaudhry et al., 2018b) alter the current gradient based on
the gradient computed with data in the memory. A unified view of episodic memory based methods
is proposed in (Guo et al., 2020), based on new approaches are developed to balance between old
tasks and the new task. Tiny episodic memory is considered in (Chaudhry et al., 2019) and meta

-----

learning is leveraged in (Riemer et al., 2018). 2) Orthogonal-projection based method. To eliminate
the need of storing data of old tasks, recently a series work (Zeng et al., 2019; Farajtabar et al.,
2020; Saha et al., 2021) updates the model in the orthogonal direction of old tasks, and has shown
remarkable performance. Particularly, Orthogonal Weight Modulation (OWM) (Zeng et al., 2019)
learns a projector matrix to multiply with the new gradients. Orthogonal Gradient Descent (OGD)
(Farajtabar et al., 2020) stores the gradient directions of old tasks and projects the new gradients on
the directions orthogonal to the subspace spanned by the old gradients. Gradient Projection Memory
(GPM) (Saha et al., 2021) stores the bases of the subspaces spanned by old task data and projects
the new gradients on the directions orthogonal to these subspaces.

3 PROBLEM FORMULATION

**Continual learning. Consider the setting where a sequence of tasks T = {t}t[T]=1** [arrives sequentially.]
Each task t has a dataset Dt = {(xt,i, yt,i)}i[N]=1[t] [with][ N][t][ sample pairs, where][ x][t,i][ is the input vector]
and yt,i is the label vector. Consider a fixed capacity neural network with L layers, and the set of
weights is denoted as W = {W _[l]}l[L]=1[, where][ W][ l][ is the layer-wise weight for layer][ l][. Given the data]_
input xt,i for task t, denote x[l]t,i [as the input of layer][ l][ and][ x]t,i[1] [=][ x][t,i][. The output][ x]t,i[l][+1] [for layer][ l][ is]
computed as x[l]t,i[+1] = f (W _[l], x[l]t,i[)][, where][ f][ is the operation of the network layer. Following (Saha]_
et al., 2021), we denote x[l]t,i [as the][ representations][ of][ x][t,i][ at layer][ l][. When learning task][ t][, we only]
have access to dataset Dt. Let L(W, {(xt,i, yt,i)}) = Lt(W) denote the loss function for training,
e.g., mean squared and cross-entropy loss, and Wt denote the model after learning task t.

**Orthogonal-projection based methods. To minimize the interference to old tasks, recently a series**
of studies (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) has been carried out to update
the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks.
In what follows, we briefly introduce the main ideas through a basic case with two tasks 1 and 2.

Denote the subspace spanned by the inputs of task 1 for layer l as S1[l] [and the learnt model for task][ 1]
as W1 = {W1[l][}][L]l=1[. It is clear that][ x]1[l] _,i_ _[∈]_ _[S]1[l]_ [. When learning task 2, the model][ W][ l]1 [will be modified]
in the direction orthogonal to S1[l] [, by either multiplying the gradient][ ∇]W _[l]_ _[L][2]_ [with a projector matrix]
(e.g, (Zeng et al., 2019)), or projecting the gradient ∇W l _L2 onto the orthogonal direction to S1[l]_ [(e.g.,]
(Saha et al., 2021)). Let ∆W1[l] [denote the model change after learning task 2. It follows immediately]
that ∆W1[l][x][l]1,i [= 0][, and the model][ W]2[ l] [for task 2 is][ W][ l]2 [=][ W][ l]1 [+ ∆][W][ l]1[. Therefore, for task 1:]

**_W2[l][x][l]1,i_** [= (][W]1[ l] [+ ∆][W]1[ l][)][x][l]1,i [=][ W]1[ l][x][l]1,i [+ ∆][W]1[ l][x][l]1,i [=][ W]1[ l][x][l]1,i[,] (1)
which indicates that no interference is introduced to task 1 after learning task 2, thereby addressing
the forgetting issue. Such an analysis can be generalized to a sequence of tasks.

**When would orthogonal projection hinder the learning of a new task? Orthogonal projection**
provides a promising solution to address the forgetting in continual learning. However, by modifying
the model only in the orthogonal direction to the input space of old tasks, the optimization space of
learning the new task could be more restrictive, resulting in compromised performance of the new
task. To get a more concrete sense, consider the following basic examples with two tasks 1 and 2.

_(Toy example 1) Suppose task 1 has dataset D1 = {(xi, yi)}i[N]=1_ [and task 2 has dataset][ D][2][ =]
_{(−xi, yi)}i[N]=1[, where only the sign is changed for the input vectors. Consider the case where two]_
tasks share the same classifier (Saha et al., 2021). It is clear that for the l-th layer, the subspace
spanned by {x[l]i[}][N]i=1 [of task 1 is same with the subspace spanned by][ {−][x]i[l][}][N]i=1 [of task 2, i.e.,]
_S1[l]_ [=][ S]2[l] [, given the learnt model][ W]1[ l] [for task 1. Based on the fact that stochastic gradient descent]
updates lie in the subspace spanned by the data input (Zhang et al., 2021; Saha et al., 2021), it follows
that the gradientthe orthogonal direction to ∇W l _L2 ∈ SS21[l][l][, such that][is 0, which means that the model][ ∇]W_ _[l]_ _[L][2]_ _[∈]_ _[S]1[l]_ [. Therefore, the projection of][ W]1[ l] [will not be updated when][ ∇]W _[l]_ _[L][2]_ [onto]
learning task 2, i.e., W2[l] [=][ W]1[ l][. However, the optimal model for task 2 should be][ W][ l]2 [=][ −][W][ l]1[,]
because W1[l][x][l]i [achieves the minimum loss for the label][ y][i][ after learning task 1.]

_(Toy example 2)follows that the projection of Suppose the input subspace of task 1 is orthogonal to that of task 2, i.e., ∇W l_ _L2 onto the orthogonal direction to S1[l]_ [is indeed equal to] S1[l][ ∇][⊥]W[S][l]2[l][L][. It][2][.]
Consequently, updating the model for task 2 based on orthogonal projection will not only introduce
no interference to task 1, but also move along the direction of steepest descent for task 2.


-----

Figure 1: Layer-wise task correlation for the case where the subspace spanned by the representations
is a two-dimensional plane. The subspaces are weakly correlated if they are nearly orthogonal and
strongly correlated if they are nearly parallel.

Motivated by these examples, a plausible conjecture is that naive orthogonal projection could possi_bly compromise the learning performance of the new task that is strongly correlated with old tasks,_
_especially when the correlation is “negative” as in the toy example 1. In this study, we advocate to_
characterize the task correlation through the correlation between the input subspaces for two tasks.
As illustrated in Fig. 1, when the subspace is 2-dimensional, two tasks are weakly correlated if their
input subspaces are nearly orthogonal, and strongly correlated if their subspaces are nearly parallel.

4 TRUST REGION GRADIENT PROJECTION FOR CONTINUAL LEARNING

To tackle these challenges, a key insight is that for a new task that is strongly correlated with old
_tasks, although the model optimization space could be more restrictive, there should be better for-_
_ward knowledge transfer from the correlated old tasks to the new task. With this insight, we propose_
a novel approach to prompt forward knowledge transfer without forgetting, by 1) introducing a novel
notion of trust region to select the most related old tasks in a single-shot manner and 2) cleverly
reusing the frozen weights of the selected tasks in the trust region with a scaled weight projection.

4.1 TRUST REGION

To facilitate forward knowledge transfer from the correlated old tasks to the new task, the first
question is how to efficiently select the most correlated old tasks. Towards this end, we characterize
the correlation between the input subspaces for two tasks, through the lens of gradient projection.

Specifically, denote Sj[l] [=][ span][{][B]j[l] _[}][ as the subspace spanned by the task][ j][ data for layer][ l][, where]_
**_Bj[l]_** [= [][u]j,[l] 1[, ...,][ u][l]j,Mj,l []][ is the bases for][ S]j[l] [(totally][ M][j,l][ bases extracted from the input). For any]
matrix A with a suitable dimension, denote its projection onto the subspace Sj[l] [as:]

ProjSjl [(][A][) =][ AB]j[l] [(][B]j[l] [)][′] (2)

where (·)[′] is the matrix transpose. We next define a layer-wise trust region for a new task as a set of
its most related old tasks, based on the norm of projected gradient onto the subspaces of old tasks.
**Definition 1 (Layer-Wise Trust Region). For any new task t ≥** 2 and layer l, we define a layer-wise
_trust region T Rt[l]_ [=][ {][j][}][ for][ j][ ∈] [[1][, t][ −] [1]][, where for any task][ j][ ∈T R]t[l] _[the following holds:]_
_∥ProjSjl_ [(][∇][W][ l] _[L][t][(][W][t][−][1][))][∥][2][ ≥]_ _[ϵ][l][∥∇][W][ l]_ _[L][t][(][W][t][−][1][)][∥][2][,]_ (3)

_where ϵ[l]_ [0, 1] and Wt 1 is the model after learning task t 1.
_∈_ _−_ _−_


Intuitively, for the new task t, the norm
of its gradient projection onto the subspace of an old task j serves as a surrogate for characterizing the correlation
between input subspaces for these two
tasks, due to the fact that the gradient
lies in the span of its input. When the
condition Eq. (3) is satisfied, the gradient **_W l_** _t(Wt_ 1) has a large projec_∇_ _L_ _−_
tion onto the subspace of an old task j,
which implies that the subspace St[l] [for]
task t and the subspace Sj[l] [for task][ j][ may]
have sufficient common bases for layer
_l. In this case, we trust that the old task_


Figure 2: Trust region for a 2-dimensional subspace can
be interpreted as: if the angleProjSjl [(][∇][W][ l] _[L][t][)][ is less than][ θ] θth[l]_ between[(larger projection on] ∇W l _Lt and_

_Sj[l]_ [), old task][ j][ is selected to][ T R]t[l][; otherwise not.][ θ]th[l]
can be set as a large value, and we can pick tasks with
top-K smallest θ to T Rt[l][.]


-----

_j is strongly correlated with the new task t in layer l, and put it into task t’s trust region T Rt[l][. A]_
simple illustration of trust region is shown in Figure 2. Note that the notion of trust region can also
be generalized to a task-wise definition, where the most correlated old tasks will be selected based
on the projection of the entire gradient ∇WLt(Wt−1). However, the layer-wise trust region could
select different tasks for different layers, which provides a more fine-resolution characterization of
task correlations in terms of layer-level features.

**Practical implementation. Besides the valuable functionality provided by the trust region for se-**
lecting most correlated old tasks, another significant benefit is the simplicity of its practical implementation. Consider the implementation for learning a new task t.

(1) Single-shot manner. Given the learnt model Wt 1, we select a sample batch from dataset Dt, and
_−_
compute the gradient **_W l_** _t(Wt_ 1) in one forward-backward pass for all layers at once. Given the
_∇_ _L_ _−_
subspace Sj[l] [for an old task][ j][, the condition Eq. (3) can be immediately evaluated for all old tasks.]

(2) Top-K correlated tasks. It is clear that the choice of ϵ[l] has a nontrivial impact on the selection
of the most correlated old tasks. To reduce the sensitivity of the performance on ϵ[l], we can set
a relatively small value of ϵ[l], and pick the top-K old tasks with the largest gradient projection
norm ∥ProjSjl [(][∇][W][ l] _[L][t][(][W][t][−][1][))][∥][2][ from the tasks satisfying Eq. (3). As demonstrated later in our]_
experiments, setting K = 1 is enough to achieve a significant performance improvement.

4.2 SCALED WEIGHT PROJECTION

Given the layer-wise trust region T Rt[l] [for the new task][ t][, the next key question is how to efficiently]
leverage the knowledge of the most correlated old tasks in T Rt[l] [for learning task][ t][. To this end, we]
propose a novel approach to reuse the frozen weights of the selected old tasks in T Rt[l] [through a]
scaled weight projection with a scaling matrix.

At the outset, it is of interest to understand what knowledge is preserved for old tasks during continual learning with orthogonal projection. Based on Eq. (1) for the simple case with two learning
tasks 1 and 2 as mentioned earlier, it can be shown that
ProjS1l [(][W]2[ l][) = Proj]S1[l] [(][W]1[ l] [+ ∆][W]1[ l][) = Proj]S1[l] [(][W]1[ l][) + Proj]S1[l] [(∆][W]1[ l][) = Proj]S1[l] [(][W]1[ l][)] (4)

where the last equation holds because the model W1[l] [is updated in the direction orthogonal to][ S]1[l]
when learning task 2. By generalizing Eq. (4) to the case with a sequence of tasks, we can have that
for the model Wt 1 after learning task t 1 and any old task j < t:
_−_ _−_
ProjSjl [(][W]t[ l]−1[) = Proj]Sj[l] [(][W]j[ l][)][,] (5)
which indicates that the model weight projection on the subspace of old tasks is actually “frozen”
_during continual learning so as to overcome forgetting of the old tasks._

On the other hand, because the trust region T Rt[l] [is constructed]
in a way that the subspace St[l] [of task][ t][ is strongly correlated]
with the subspace Sj[l] [for any old task][ j][ ∈T R]t[l][, the bases][ B]j[l]
of Sj[l] [is very likely to contain important bases for task][ t][. As a]
result, the weight projection ProjSjl [(][W]t[ l] 1[)][ is][ important for the]
_−_
_new task t and should be modified accordingly in order to guar-_
antee the learning performance of task t, which however has to
be frozen to protect task j. To find an efficient way to lever- Figure 3: [u[l]j,1[,][ u][l]j,2[]][ is the bases]
age ProjSjl [(][W][ l]t−1[)][ without modifying it, note that the projec-] of subspace Sj[l] [, and][ [][c][l]j,1[, c][l]j,2[]][ is]

tion ProjSjl [(][W][ l]t−1[)][ is indeed a linear combination of the pro-] the coordinate of Projslj [(][W]t[ l] 1[)][.]

_−_

jection of Wt[l]−1 [onto each basis in][ B]j[l] [, and every point in][ S]j[l] Any point ProjSjl [(][W][ l][)][ in][ S]j[l] [can]
can be obtained by scaling the coordinates of ProjSjl [(][W][ l]t−1[)][.] be obtained by scaling the coorFigure 3 shows a simple example for two-dimensional sub- dinate [c[l]j,1[, c][l]j,2[]][ with some scalar]
space. Therefore, we propose a scaled weight projection to _s1 and s2._
find the best point for task t in Sj[l] [by leveraging the projection]
ProjSjl [(][W][ l]t 1[)][ through a square scaling matrix][ Q][l]j,t[:]
_−_

Proj[Q]Sj[l] [(][W]t[ l]−1[) =][ W]t[ l]−1[B]j[l] **_[Q][l]j,t[(][B]j[l]_** [)][′][.] (6)


-----

The dimension of Q[l]j,t [depends on the number of bases in][ B]j[l] [(dimension of][ S]j[l] [), which is usually]
small for each task. In this way, we explicitly transfer the knowledge of the selected old tasks in the
trust region T Rt[l] [to the new task][ t][ through a scaling matrix][ Q]j,t[l] [.]

4.3 TASK SUBSPACE CONSTRUCTION

To successfully leverage the trust region, a missing ingredient is the construction of input subspaces
of old tasks. We next show how the subspace Sj[l] [can be constructed for task][ j][ at layer][ l][.]

**For task j = 1. As in (Saha et al., 2021), we obtain the bases B1[l]** [after learning task 1 using Singular]
Value Decomposition (SVD) on the representations. Specifically, given the model W1 after learning
task 1, we construct a representation matrix R1[l] [= [][x]1[l] _,1[, ...,][ x][l]1,n[]][ ∈]_ [R][m][×][n][ with][ n][ samples, where]
each x[l]1,i

_[∈]_ [R][m][, is the representation at layer][ l][ by forwarding the sample][ x][1][,i][ through the network.]
Then, we apply SVD to the matrix R1[l] [, i.e.,][ R]1[l] [=][ U][ l]1[Σ]1[l] [(][V][ l]1 [)][′][, where][ U][ l]1 [= [][u]1[l] _,1[, ...,][ u]1[l]_ _,m[]][ ∈]_
R[m][×][m] is an orthogonal matrix with left singular vector u[l]1,i _[∈]_ [R][m][,][ V][ l]1 [= [][v]1[l] _,1[, ...,][ v]1[l]_ _,n[]][ ∈]_ [R][n][×][n]

is an orthogonal matrix with right singular vector v1[l] _,i_ 1

_[∈]_ [R][n][, and][ Σ][l] _[∈]_ **_[R][m][×][n][ is a rectangular]_**

diagonal matrix with non-negative singular values _σ1[l]_ _,i[}]i[min]=1[{][m,n][}]_ on the diagonal in a descending
_{_
order. To obtain the bases for subspace S1[l] [, we use][ k]1[l] [-rank matrix approximation to pick the first][ k]1[l]
left singular vectors in U1[l][, such that the following condition is satisfied for a threshold][ η]th[l]

_[∈]_ [(0][,][ 1)][:]
_∥(R1[l]_ [)]k1[l] _[∥]F[2]_ _[≥]_ _[ϵ]th[l]_ _[∥][R]1[l]_ _[∥]F[2]_ (7)

where (R1[l] [)]k1[l] [=][ P]i[k]=11[l] _[σ]1[l]_ _,i[u][l]1,i[(][v]1[l]_ _,i[)][′][ is a][ k]1[l]_ [-rank (][k]1[l]

_[≤]_ _[r][) approximation of the representation]_
matrix R1[l] [with rank][ r][ ≤] [min][{][m, n][}][, and][ ∥· ∥][F] [is the Frobenius norm. Then the bases][ B]1[l] [for]
subspace S1[l] [can be constructed as][ B]1[l] [= [][u]1[l] _,1[, ...,][ u][l]1,k1[l]_ []][.]

**For task j ∈** [2, T ]. We construct the bases Bj[l] [after learning task][ j][ given the learnt model][ W][j][.]
A representation matrix Rj[l] [will be first obtained in the same manner as][ R]1[l] [. Note that the bases]
_{Bi[l][}][j]i=1[−][1]_ [learnt for old tasks may include important bases for task][ j][. Therefore, we learn the bases]
**_Bj[l]_** [by selecting the most important bases from both bases of old tasks and newly constructed bases.]
Specifically, (1) (old bases) we first concatenate the bases {Bi[l][}][j]i=1[−][1] [of old tasks together in][ M]j[ l] [and]
eliminate the common bases. For each basis u[l]i _[∈]_ **_[M]j[ l][, we compute the corresponding eigenvalue of]_**
**_Rj[l]_** [(][R]j[l] [)][′][, i.e.,][ δ]i[l] [= (][u]i[l][)][′][R]j[l] [(][R]j[l] [)][′][u][l]i[, which is the square of the singular value of][ R]j[l] [with respect to]
**_u[l]i[. (2) (][new bases][) We perform SVD on][ ˆ]Rj[l]_** [=][ R]j[l] _[−]_ **_[R]j[l]_** **_[M]j[ l][(][M]j[ l][)][′][ to generate new bases beyond]_**
**_Mj[l][, i.e.,][ ˆ]Rj[l]_** [= ˆ]Uj[l]Σ[ˆ] _[l]j[( ˆ]Vj[l][)][′][ with singular values][ {]σ[ˆ]j,h[l]_ _[}][h][. (3) (][select the most important bases]_
**from both old and new bases) Next we concatenate {δi[l][}][i][ and][ {][(ˆ]σj,h[l]** [)][2][}][h][ together in a vector][ δ][,]
and sort them in a descending order. We perform kj[l] [-rank matrix approximation of][ R]j[l] [, such that the]
summation of the first kj[l] [elements in][ δ][ is greater than][ ϵ]th[l] _[∥][R]j[l]_ _[∥][2]F_ [. Then][ B]j[l] [can be constructed by]
selecting the bases corresponding to the first kj[l] [elements in][ δ][.]

4.4 CONTINUAL LEARNING WITH TRUST REGION GRADIENT PROJECTION

Building on the three modules proposed earlier, i.e., task subspace construction, trust region and
scaled weight projection, we next present our approach TRGP for continual learning that efficiently
facilitate forward knowledge transfer without forgetting the old tasks.

**Learning task 1. The first task is learnt using standard gradient descent. The subspace {S1[l]** _[}][L]l=1_ [is]
constructed by following Section 4.3.

**Learning task 2, ..., T.correlated old tasks selected for layer For task t ∈ l. The optimization problem for task[2, T** ], we first determine the trust region t is as follows: T Rt[l] [with top-][K]
min ( **_Weff[l]_** _[}]l[,][ D]t[)][,]_ (8)
_{W_ _[l]}l,{Q[l]j,ts.t[}]l,j∈T Rt[l]_ _LW{eff[l]_ [=][ W][ l][ +] _j∈T Rt[l]_ [Proj[Q]Sj[l] [(][W][ l][)][ −] [Proj][S]j[l] [(][W][ l][)]][,] (9)

where the gradient for updating W _[l]_ is ∇W l _L =X ∇W l_ _L −_ (∇W l _L)Mt[l][(][M]t[ l][)][′][ and][ M]t[ l]_ [is the bases]
of all old tasks as in Section 4.3. The subspace {St[l][}][L]l=1 [is next obtained by following Section 4.3.]


-----

Table 1: The averaged accuracy (ACC) and backward transfer (BWT) over all the tasks on different
datasets. Note that, Multitask jointly learns all tasks only once in a single network by using the
whole dataset, which does not adhere to CL setup.

PMNIST CIFAR-100 Split 5-Dataset MiniImageNet
Method

ACC(%) BWT(%) ACC(%) BWT(%) ACC(%) BWT(%) ACC(%) BWT(%)

Multitask 96.70 -  79.58 -  91.54 -  69.46 - 

OWM 90.71 -1 50.94 -30 -  -  -  - 
EWC 89.97 -4 68.80 -2 88.64 -4 52.01 -12
HAT -  -  72.06 0 91.32 -1 59.78 -3
A-GEM 83.56 -14 63.98 -15 84.04 -12 57.24 -12
ER Res 87.24 -11 71.73 -6 88.31 -4 58.94 -7
GPM 93.91 -3 72.48 -0.9 91.22 -1 60.41 -0.7

Ours (TRGP) **96.34** **-0.8** **74.46** **-0.9** **93.56** **-0.04** **61.78** **-0.5**

5 EXPERIMENTAL RESULTS

5.1 EXPERIMENTAL SETUP

**Datasets and training details. We evaluate our method on multiple datasets against state-of-the-art**
CL methods. 1) PMNIST. Following (Lopez-Paz & Ranzato, 2017; Saha et al., 2021), we create
10 sequential tasks using different permutations where each task has 10 classes. We use a 3-layer
fully-connected network. 2) CIFAR-100 Split. We split the classes of CIFAR-100 (Krizhevsky
et al., 2009) into 10 group, and consider 10-way multi-class classification in each group as a single
task. Similar with (Serra et al., 2018; Saha et al., 2021), we use a version of 5-layer AlexNet. 3)
**CIFAR-100 Sup. We divide the CIFAR-100 dataset into 20 tasks where each task has 5 classes.**
We use a modified version of LeNet-5. 4) 5-Datasets. We use a sequence of 5-Datasets which
includes CIFAR-10, MNIST, SVHN (Netzer et al., 2011), not-MNIST (Bulatov, 2011) and Fashion
MNIST (Xiao et al., 2017), where each dataset is set to be a task. We adapt a reduced ResNet18
network that is used in (Lopez-Paz & Ranzato, 2017). 5) MiniImageNet Split. We split the 100
classes of MiniImageNet (Vinyals et al., 2016) into 20 sequential tasks where each task has 5 classes,
and consider a reduced ResNet18 network. In addition, for all the experiments, the threshold ϵ[l] is
set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We use the same threshold ϵ[l]th [as]
GPM (Saha et al., 2021) for subspace construction. More details are in the appendix.

**Methods for comparison. To test the efficacy of our method, we compare it with state-of-the-art**
approaches in three categories: 1) Memory-based methods. We compare with Experience Replay
with reservoir sampling (ER Res) (Chaudhry et al., 2019), Averaged GEM (A-GEM) (Chaudhry
et al., 2018b), Orthogonal Weight Modulation (OWM) (Zeng et al., 2019) and Gradient Projection
Memory (GPM) (Saha et al., 2021). 2) Regularization-based methods. We compare with state-ofthe-art HAT (Serra et al., 2018) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017).
**3) Expansion-based methods. We further compare with Progressive Neural Network (PNN) (Rusu**
et al., 2016), Learning Without Forgetting (LWF) (Li & Hoiem, 2017), Dynamic-Expansion Net
(DEN) (Yoon et al., 2017), and APD (Yoon et al., 2020), by using CIFAR-100 Sup dataset.

**Metrics. Following GPM (Saha et al., 2021), two metrics are used to evaluate the performance:**
Accuracy (ACC), the average final accuracy over all tasks, and Backward Transfer (BWT), which
measures the forgetting of old tasks when learning new tasks. ACC and BWT are defined as:

_T_ 1 _T −1_

_ACC = [1]_ (10)

_T_ _i=1_ _[A][T,i][, BWT][ =]_ _T −_ 1 _i=1_ _[A][T,i][ −]_ _[A][i,i]_

where T is the number of tasks, AXT,i is the accuracy of the model onX _i-th task after learning the T_ -th
task sequentially.

5.2 MAIN RESULTS

**ACC and BWT comparison. As shown in Table 1, TRGP achieves significantly accuracy improve-**
ment compared with prior works on all datasets. For example, in contrast to the best prior results,
TRGP achieve the accuracy gain of 2.43%, 1.98% and 1.37% over GPM on PMNIST, CIFAR-100
Split and MiniImageNet, respectively, and 2.34% over HAT on 5-Dataset. Surprisingly, we could
even achieve better accuracy than Multitask on 5-Datasets, which usually serves as an upper bound
for CL benchmarks. This superior performance of TRGP clearly shows its capability to effectively
facilitate forward knowledge transfer. In addition, TRGP also demonstrates strong performance with
the lowest BWT, reducing 0.2% than OWM and 0.6% than GPM, even with 5.63% and 2.34% ac

-----

Figure 4: The final accuracy for all tasks on three datasets (GPM VS Ours).


Final accuracy for PMNIST Final accuracy for CIFAR-100 100 Final accuracy for 5-Datasets

96.5 77.5

96.0 75.0 90

95.5 72.5

95.0 Ours 70.0 Ours 80 Ours

Test Accuracy (%) GPM Test Accuracy (%) GPM Test Accuracy (%) GPM

94.5 67.5

0 2 4 6 8 0 2 4 6 8 0 1 2 3 4

Number of tasks Number of tasks Number of tasks


Table 2: The performance for CIFAR-100 Sup dataset. Note that Single-task learning (STL) trains
a separate network for each task, which does not adhere to CL setup.

Methods
Metric

STL PNN DEN RCL APD GPM Ours (TRGP)


ACC(%) 61.00 50.76 51.10 51.99 56.81 57.72 **58.25**
Capacity(%) 2000 271 191 184 130 100 100

curacy improvement on PMNIST and 5-Dataset, respectively. Compared with HAT on CIFAR-100
Split, TRGP has marginally worse BWT, but achieves 2.4% accuracy gain.

Moreover, TRGP exhibits an universal dominance over GPM about the final accuracy of all tasks on
all the three datasets. According to the apple to apple comparison with GPM in Fig. 4, one interesting
phenomenon is observed: TRGP has the similar accuracy on “easy” tasks, but significantly improves
the accuracy on the “difficult” tasks. For example, in the 5-Dataset setting, both TRGP and GPM
achieve good accuracy on Task 1 (MNIST) and 3 (Fashion MNIST), which can be easily trained
well, but TRGP significantly outperforms GPM on the rest three more difficult Tasks (CIFAR-10,
SVHN and NotMNIST). In the end, as shown in Table 2, we further compare with the expansionbased methods by using CIFAR-100 Sup setting. It can be seen that TRGP outperforms all other CL
methods, with a fixed capacity network.

**Discussion. We next show the accuracy evolution of specific tasks during the training of all tasks**
sequentially. We randomly select three tasks for each dataset to compare with GPM (we only show
results on PMNIST in Fig. 5 and relegate the rest to the appendix). There are two main obervations:
1) TRGP completely outperforms GPM during training for all the sequential tasks on the three
datasets; 2) For the PMNIST and 5-Dataset settings, TRGP could significantly reduce forgetting. To
understand why, consider the case where GPM and TRGP learns a new task t given the same model
Wt 1, and denote **_Mt[l]_** 1[}][l] [as the bases of all old tasks. Then we can have]
_−_ _{_ _−_

_For GPM, the effective weight for layer l is_
**_Weff[l]_** [= Proj]Mt[l] 1 [(][W][ l][) + Proj][⊥][M]t[l] 1 [(][W][ l][)] (11)
_−_ _−_

where the weight projection on Mt[l] 1 [is frozen to protect old tasks, and only the weight projection]
_−_
orthogonal to Mt[l] 1 [can be updated for learning task][ t][.]
_−_

_For TRGP, the effective weight for layer l is_
**_Weff[l]_** [= Proj]{Sj[l] _[}]j /∈T Rt[l]_ (W _[l]) + Proj[Q]{Sj[l]_ _[}]j∈T Rt[l]_ (W _[l]) + Proj⊥Mtl−1_ [(][W][ l][)] (12)

where only the first term, i.e., weight projection on subspaces of old tasks that are not in the trust
region T Rt[l][, is frozen for task][ t][. In contrast to GPM, an additional and also important part of]
weights, i.e., the scaled weight projection on subspaces of related old tasks in T Rt[l][, can be learnt]
in a favorable way for task t. As a result, TRGP can achieve better forward knowledge transfer
by explicitly and cleverly reusing the important knowledge of strongly correlated old tasks in the
trust region. More interestingly, benefiting from the task-unique information captured by the scaled
weight projection, the backward transfer can also be reduced.





Figure 5: Accuracy evolution for different tasks on PMNIST setting.


Accuracy envolution for Task 2 Accuracy envolution for Task 4 Accuracy envolution for Task 6

Ours 97.0 Ours

97 GPM 97.0 GPM

96.5 96.5

96

96.0 Ours 96.0

Test Accuracy (%)95 Test Accuracy (%)95.5 GPM Test Accuracy (%)

2 4 6 8 4 6 8 5 6 7 8 9

Number of tasks Number of tasks Number of tasks


-----

Table 3: Ablation study on CIFAR-100 Split and 5-Datasets settings.

Impact of threshold ϵ[l] Layer-wise VS Task-wise Number of selected tasks
Datasets

0.2 0.5 0.7 Layer-wise Task-wise Top-1 Top-2


CIFAR-100 74.52 74.46 74.30 74.46 73.25 74.00 74.46
5-Datasets 93.28 93.56 93.43 93.56 92.85 92.94 93.56

5.3 ABLATION STUDY AND ANALYSIS


**Impact of the threshold ϵ[l]. To understand the impact of the threshold ϵ[l], we evaluate the learning**
performance for three different values of ϵ[l] (i.e., 0.2, 0.5, 0.7) as shown in Table 3. The results show
that the accuracy is very stable across the three threshold values, with ignoble accuracy difference on
both CIFAR-100 Split and 5-Dataset settings. The reason behind is because we only select top-2 old
tasks with largest gradient projection norm into the trust region, among all tasks satisfying condition
Eq. (3). Therefore, for a wide range of ϵ[l], the selected tasks in the trust region are actually fixed.
The small accuracy fluctuation is because with some possibility only one old task satisfies Eq. (3)
and is selected for some layers when ϵ[l] increases. Overall, TRGP is very robust to the value of ϵ[l].

**Layer-wise vs. Task-wise trust region. To show the efficacy of** Ours Layer-wise GPM
layer-wise trust region, we compare it with the task-wise variant 77.5 Ours Task-wise
which shares a fixed trust region across all layers for each task. 75.0
First, as shown in Table 3, layer-wise could achieve 1.21% accu- 72.5
racy gain over task-wise on CIFAR-100 Split. Furthermore, we 70.0
illustrate the final accuracy of all tasks for layer-wise and task- 67.5
wise of the proposed TRGP, and GPM on CIFAR-100 Split set- 0 2 Number of tasks4 6 8
ting in Fig. 6. First, the performance of layer-wise is better than

Figure 6: The final accuracy for

Ours Layer-wise GPM

77.5 Ours Task-wise

75.0

72.5

70.0

Test Accuracy (%)

67.5

0 2 4 6 8

Number of tasks

or comparable to task-wise for all tasks, because layer-wise pro
all tasks of Task-wise VS Layer
vides a much finer characterization of task correlations in terms

wise on CIFAR-100 Split.

of layer-level features. Then, it is interesting to see that the learning behavior for the three cases follows the same trend. This observation further corroborates that
TRGP can improve the accuracy and mitigate forgetting on both “easy” and “difficult” tasks.

**Impact of selected tasks in trust region. We first evaluate the accuracy of Top-1 and Top-2 selected**
tasks as shown in Table 3. It shows that selecting the top-2 most correlated tasks could achieve better
accuracy on both CIFAR-100 Split and 5-Dataset settings. Note that good performance can also be
achieved even with the Top-1 case. Moreover, we illustrate the detailed task selection in the trust
region for both layer-wise and task-wise on 5-Dataset setting in Fig. 7. For the task-wise, current task
always selects the two adjacent previous tasks for all layers. Differently, the task selection varies for
layer-wise, leading to more accurate selection of related tasks for each layer. For example, the layer
wise trust region for Task 4 (Fashion MNIST) selects Task 1 (MNIST) or Task 3 (not-MNIST) as the
most related tasks almost for all layers, over Task 0 (CIFAR-10) and Task 2 (SVHN), which clearly
makes sense because Fashion MNIST shares more common features with MNIST and not-MNIST.




Task-wise Layer-wise for Task 3 Layer-wise for Task 4

2 3

3 1st task

2ed task

2 2

1

1 1

Top-2 Seleted tasks0 Top-2 Seleted tasks0 0

0 2 4 0 3 6 9 12 15 18 0 3 6 9 12 15 18

Task Index Layer Index Layer Index


Figure 7: The detailed selected tasks on 5-Datasets setting.


6 CONCLUSION

In this work, we propose trust region gradient projection for continual learning to facilitate forward
knowledge transfer with forgetting, based on an efficient characterization of task correlation. Particularly, our approach is built on two key blocks, i.e., the layer-wise trust region which effectively
select the old tasks strongly correlated to the new task in a single-shot manner, and scaled weight
projection which cleverly reuses the frozen weights of old tasks in the trust region without modifying
the model. Extensive experiments show that our approach significantly improves over the related
state-of-the-art methods.


-----

ACKNOWLEDGEMENT

This work is supported in part by NSF Grants CNS-2003081, CNS-2203239, CPS-1739344, and
CCSS-2121222.

REPRODUCIBILITY STATEMENT

For the experimental results presented in the main text, we include the code in the supplemental
material, and specify all the training details in Section 5.1 and Appendix A. For the datasets used in
the main text, we also give a clear explanation in Section 5.1.

REFERENCES

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.
Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer_ence on Computer Vision (ECCV), pp. 139–154, 2018._

Yaroslav Bulatov. Notmnist dataset. _Google (Books/OCR), Tech. Rep.[Online]. Available:_
_http://yaroslavvb. blogspot. it/2011/09/notmnist-dataset. html, 2, 2011._

Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian
walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the
_European Conference on Computer Vision (ECCV), pp. 532–547, 2018a._

Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient
lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018b.

Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K
Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. 2019.

Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence
_and Machine Learning, 12(3):1–207, 2018._

Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF Conference on Computer Vision and
_Pattern Recognition, pp. 5138–5146, 2019._

Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762–3773.
PMLR, 2020.

Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory based lifelong learning algorithm. In Conference on Neural Information Processing Systems,
2020.

Steven CY Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song
Chen. Compacting, picking and growing for unforgetting continual learning. arXiv preprint
_arXiv:1910.06562, 2019._

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences,
114(13):3521–3526, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.

Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint arXiv:1703.08475,
2017.


-----

Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual
structure learning framework for overcoming catastrophic forgetting. In International Conference
_on Machine Learning, pp. 3925–3934. PMLR, 2019._

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis
_and machine intelligence, 40(12):2935–2947, 2017._

David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning.
_Advances in neural information processing systems, 30:6467–6476, 2017._

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The
sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109–165.
Elsevier, 1989.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading
digits in natural images with unsupervised feature learning. 2011.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald
Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018.

Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for
overcoming catastrophic forgetting. arXiv preprint arXiv:1805.07810, 2018.

Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE transac_tions on pattern analysis and machine intelligence, 42(3):651–663, 2018._

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray
Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint
_arXiv:1606.04671, 2016._

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning.
_arXiv preprint arXiv:2103.09762, 2021._

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference on machine learning, pp. 1889–1897. PMLR,
2015.

Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye
Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pp. 4528–4537. PMLR, 2018.

Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic
forgetting with hard attention to the task. In International Conference on Machine Learning, pp.
4548–4557. PMLR, 2018.

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative
replay. arXiv preprint arXiv:1705.08690, 2017.

Tom Veniat, Ludovic Denoyer, and Marc’Aurelio Ranzato. Efficient continual learning with modular
networks and task-driven priors. arXiv preprint arXiv:2012.12631, 2020.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one
shot learning. Advances in neural information processing systems, 29:3630–3638, 2016.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.

Ju Xu and Zhanxing Zhu. Reinforced continual learning. arXiv preprint arXiv:1805.12369, 2018.

Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically
expandable networks. arXiv preprint arXiv:1708.01547, 2017.


-----

Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual
learning with additive parameter decomposition. In Eighth International Conference on Learning
_Representations, ICLR 2020. ICLR, 2020._

Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364–372, 2019.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence.
In International Conference on Machine Learning, pp. 3987–3995. PMLR, 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding
deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–
115, 2021.


-----

EXPERIMENT SETUPS


**Training hyper-parameters. We evaluate our method on multiple datasets against state-of-the-art**
continual learning methods. 1) PMNIST. We use a 3-layer fully-connected network. with two
hidden layer of 100 units. and train the network for 5 epochs with batch size of 10 for each task. 2)
**CIFAR-100 Split. CIFAR-100 (Krizhevsky et al., 2009) consists of images from 100 generic object**
classes. We use a version of 5-layer AlexNet and train each task for maximum of 200 epochs with the
early termination strategy based on the validation loss value. The batch size is set to 64. 3) CIFAR**100 Sup. We use a modified version of LeNet-5 with 20-50-800-500 neurons and train 50 epochs for**
each task sequentially. The batch size is set to 64. 4) 5-Datasets. We train each task for maximum
of 200 epochs with the early termination strategy. The batch size is set to 64. 5) MiniImageNet
**Split. Following GPM (Saha et al., 2021), we use the reduced ResNet18 architecture, where the**
covolution with stride 2 in the first layer. We train each task for maximum of 100 epochs with
the early termination strategy with 0.1 initial learning rate and 64 batchsize. In addition, for all the
experiments, the threshold ϵ[l] is set to 0.5, and we select top-2 tasks that satisfy condition Eq. (3). We
use the same threshold ϵ[l]th [as GPM (Saha et al., 2021) for subspace construction. We initialize the]
scaling matrix with the identity matrix and train all models with plain stochastic gradient descent.

B MORE EXPERIMENTAL RESULTS


B.1 ACCURACY EVOLUTION




Accuracy envolution for Task 2 Accuracy envolution for Task 4 Accuracy envolution for Task 6

76

74

69 75

68 72 Ours 74 Ours

GPM GPM

67 Ours 70 73

Test Accuracy (%) GPM Test Accuracy (%) Test Accuracy (%)72

66

2 4 6 8 4 6 8 5 6 7 8 9

Number of tasks Number of tasks Number of tasks


(a) CIFAR-100 Split

Accuracy envolution for Task 1 Accuracy envolution for Task 2 Accuracy envolution for Task 3

99.4

80 92 Ours

GPM

99.2

78 Ours 90

GPM

99.0

Test Accuracy (%)76 Test Accuracy (%) OursGPM Test Accuracy (%)88

98.8

0 1 2 3 4 1 2 3 4 2.0 2.5 3.0 3.5 4.0

Number of tasks Number of tasks Number of tasks


(b) 5-Datasets


Accuracy envolution for Task 1


Figure 8: Accuracy evolution for different tasks on CIFAR-100 Split and 5-Datasets settings.

B.2 STANDARD DEVIATION


We have summarized the results on the standard deviation for the averaged accuracy and backward
transfer over 5 different runs on all datasets in Table 4.

Table 4: The averaged accuracy (ACC) and backward transfer (BWT) with the standard deviation
values over 5 different runs on different datasets.

PMNIST CIFAR-100 Split 5-Dataset MiniImageNet
Method


ACC(%) BWT(%) ACC(%) BWT(%) ACC(%) BWT(%) ACC(%) BWT(%)

Multitask 96.70 ± 0.02 -  79.58 ± 0.54 -  91.54 ± 0.28 -  69.46 ± 0.62 - 

A-GEMER ResOWMEWCGPMHAT 9089838793.....7197245691 ± ± ± ± ±- 0 0 0 0 0.....1157165316 _−−−−−1114143 ± ± ±- ± ± 0 1 0 1 1_ 725068727163......948073480698 ± ± ± ± ± ± 0 0 0 0 0 1......608850226340 _−−−−−00301526. ±9 ± ± ± ± ± 0 1 1 1 2 0_ 9184888891.....3204643122 ± ± ± ± ±- 0 0 0 0 0.....2618332220 _−−−−−124141 ± ± ± ±- ± 1 0 0 0 1_ 5752595860.....0178249441 ± ± ± ± ±- 2 0 0 0 0.....5357728561 _−−−0−−.1212377 ± ± ±- ± ± 0 1 0 3 1.4_

Ours (TRGP) 969696...343434 ± ± ± 0 0 0...111111 _−−−000...888 ± ± ± 0 0 0...111_ 747474...464646 ± ± ± 0 0 0...323232 _−−−000...999 ± ± ± 0 0 0...010101_ 939393...565656 ± ± ± 0 0 0...101010 _−−−000...040404 ± ± ± 0 0 0...010101_ 616161...787878 ± ± ± 0 0 0...606060 _−−−000...555 ± ± ± 0 0 0...666_


-----

B.3 FORWARD TRANSFER

To evaluate the forward transfer, we follow the metric used in (Veniat et al., 2020) and consider the
accuracy of the model on i-th task after learning the i-th task sequentially, i.e., Ai,i as defined in
Eq. (10). Tables 5 - 8 summarize the comparison of Ai,i for each task i between GPM and TRGP on
PMNIST, CIFAR-100 Split and 5-Dataset, respectively. As the same baseline (e.g., the accuracy of
the model learnt from scratch using the task’s own data) for each task will be used when evaluating
the forward transfer for GPM and TRGP, we can infer that TRGP achieves the forward transfer
gain of 0.17%, 2.01%, 2.00% and 2.36% over GPM on PMNIST, CIFAR-100 Split, 5-Datasets and
MiniImageNet respectively.

Table 5: The accuracy Ai,i of the model on i-th task after learning the i-th task sequentially on
PMNIST 10 tasks.

|Methods|1|2|3|4|5|6|7|8|9|10|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|


|GPM|97.5|97.5|97.3|97.1|97.0|96.9|96.8|96.4|96.5|96.5|96.95|
|---|---|---|---|---|---|---|---|---|---|---|---|


|Ours (TRGP)|97.5|97.5|97.5|97.3|97.1|97.1|96.9|96.7|96.9|96.7|97.12|
|---|---|---|---|---|---|---|---|---|---|---|---|



Table 6: The accuracy Ai,i of the model on i-th task after learning the i-th task sequentially on
CIFAR-100 Split 10 tasks.

|Methods|1|2|3|4|5|6|7|8|9|10|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|


|GPM|76.8|68.5|72.4|69.9|74.8|72.3|70.3|71.9|73.2|75.1|72.52|
|---|---|---|---|---|---|---|---|---|---|---|---|


|Ours (TRGP)|76.9|69.5|75.1|74.1|75.3|75.8|72.8|73.8|73.9|78.1|74.53|
|---|---|---|---|---|---|---|---|---|---|---|---|



Table 7: The accuracy Ai,i of the model on i-th task after learning the i-th task sequentially on
5-Dataset 5 tasks.

|Methods|1|2|3|4|5|Avg|
|---|---|---|---|---|---|---|


|GPM|78.3|99.1|87.1|99.1|94.1|91.54|
|---|---|---|---|---|---|---|


|Ours (TRGP)|80.9|99.3|92.8|99.4|95.3|93.54|
|---|---|---|---|---|---|---|



Table 8: The accuracy Ai,i of the model on i-th task after learning the i-th task sequentially on
MiniImageNet Split 20 tasks.

|Methods|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|Avg|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|GPM|58.6|63.6|57.2|59.0|53.6|78.0|63.0|66.0|74.0|83.8|43.0|60.4|55.6|57.8|59.6|53.0|56.0|47.6|66.0|56.8|60.63|
|Ours (TRGP)|58.7|66.1|59.2|59.3|57.1|81.4|67.3|70.1|75.7|85.2|43.2|61.8|58.0|60.1|60.0|54.8|61.4|48.4|69.8|62.2|62.99|



B.4 COMPUTATIONAL COMPLEXITY

Memory: In terms of the memory, the major difference between TRGP and GPM is that TRGP
requires additional memory to store the scaling matrices for each task. However, since the dimension
of the scaling matrix is the same with the number of the extracted bases for the input subspace,
which is usually small and controllable by the matrix approximation accuracy ϵth in Eq. (7), the
memory increase is marginal and controllable. Particularly, the memory usage of TRGP can be
further reduced by only learning the scaling matrices for the convolutional layers.

Training time: We compare the training time between TRGP and other baselines on relatively complex task sequences. As shown in Table 9, for CIFAR-100 Split, TRGP takes around 65% more
time than GPM, is comparable with HAT and ER Res, and takes less time than OWM and EWC; for
5-Datasets, TRGP takes around 21% more time than GPM, but is much faster than other baselines
including EWC, HAT, A-GEM and ER Res; for MiniImageNet, TRGP tasks around 34% more time
than GPM, is comparable with EWC, but is much faster than A-GEM.


-----

Table 9: Training time comparison on CIFAR-100 Split, 5-Datasets and MiniImageNet. Here the
training time is normalized with respect to the value of GPM. Please refer (Saha et al., 2021) for
more specific time.

Methods
Dataset

OWM EWC HAT A-GEM ER Res GPM Ours (TRGP)


CIFAR-100 2.41 1.76 1.62 3.48 1.49 1 1.65
5-Datasets -  1.52 1.47 2.41 1.40 1 1.21
MiniImageNet -  1.22 0.91 1.79 0.82 1 1.34

B.5 ACCURACY VS LEARNING EPOCHS


The learning dynamics for each task are shown in Figure 9 and 10. Clearly, our approach can
perform significantly better than GPM on some tasks, especially for the tasks in the tail of the task
sequence. This is because in GPM, with more tasks being learnt, the optimization space for new
tasks becomes more restrictive, leading to limited performance for new tasks. Note that the y-axis
is the validation accuracy with a split validate dataset that used during training, by following the
setup in (Saha et al., 2021). The validation accuracy varies because the size of the validate dataset
is relatively small (See (Saha et al., 2021) for the specific size). For the testing accuracy in all the
tables, we evaluate the accuracy with the testing dataset after training.


Task-1


Task-2



70

65

60

55

75

70

65

60


70

60

50

70

60

50

|Col1|GPM Ours|
|---|---|

|Col1|GPM Ours|
|---|---|


20 40 60 80

GPM
Ours

Number of Epochs

Task-4


20 40 60 80

GPM
Ours

Number of Epochs

Task-5


|Col1|GPM Ours|
|---|---|


20 40 60 80 100

GPM
Ours

Number of Epochs

Task-7


|Col1|GPM Ours|
|---|---|


20 40 60 80

GPM
Ours

Number of Epochs

Task-8


70

65

60

55


70

65

60

80

75

70

65


|Col1|GPM Ours|
|---|---|


20 40 60 80

GPM
Ours

Number of Epochs


|Col1|GPM Ours|
|---|---|


20 40 60 80

GPM
Ours

Number of Epochs

Task-10


Task-3

70

60 GPM

Valid Accuracy (%) Ours

0 20 40 60 80

Number of Epochs

Task-6

80

70

60 GPM

Valid Accuracy (%) Ours

0 20 40 60 80 100

Number of Epochs

Task-9

75

70

65

60 GPM

Valid Accuracy (%) Ours

0 20 40 60 80

Number of Epochs


|Col1|GPM Ours|
|---|---|


0 20 40 60 80

GPM
Ours

Number of Epochs

Figure 9: Accuracy vs learning epochs for different tasks on CIFAR-100 Split.


-----

Task-1 CIFAR10 Task-2 MNIST Task-3 SVHN

80 99.2 92.5

70 99.0 90.0

98.8 87.5

60

GPM 98.6 GPM 85.0 GPM

Valid Accuracy (%)50 Ours Valid Accuracy (%)98.4 Ours Valid Accuracy (%)82.5 Ours

0 10 20 30 0 10 20 30 0 10 20

Number of Epochs Number of Epochs Number of Epochs

Task-4 Fashion-mnist Task-5 Notmnist

99.4 96

99.2

94

99.0

98.8 92

GPM GPM

Valid Accuracy (%)98.6 Ours Valid Accuracy (%)90 Ours

0 10 20 30 0 10 20 30

Number of Epochs Number of Epochs


Figure 10: Accuracy vs learning epochs for five tasks on 5-Dataset.


-----

