# TARGET PROPAGATION VIA REGULARIZED INVERSION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Target Propagation (TP) algorithms compute targets instead of gradients along
neural networks and propagate them backward in a way that is similar yet different than gradient back-propagation (BP). The idea was first presented as a perturbative alternative to back-propagation that may improve gradient evaluation accuracy when training multi-layer neural networks (Le Cun et al., 1989). However,
TP may have remained more of a template algorithm with many variations than
a well-identified algorithm. Revisiting insights of Le Cun et al. (1989) and more
recently of Lee et al. (2015), we present a simple version of target propagation
based on a regularized inversion of network layers, easily implementable in a differentiable programming framework. We compare its computational complexity
to the one of BP and delineate the regimes in which TP can be attractive compared
to BP. We show how our TP can be used to train recurrent neural networks with
long sequences on various sequence modeling problems. The experimental results
underscore the importance of regularization in TP in practice.

1 INTRODUCTION

Target propagation algorithms can be seen as perturbative learning alternatives to the gradient backpropagation algorithm, where virtual targets are propagated backward instead of gradients (Le Cun,
1986; Le Cun et al., 1989; Rohwer, 1990; Mirowski & LeCun, 2009; Bengio, 2014; Goodfellow
et al., 2016). A high-level summary is presented in Fig. 1: while gradient back-propagation considers storing intermediate gradients in a forward pass, target propagation algorithms proceed by
computing and storing approximate inverses. The approximate inverses are then passed on backward along the graph of computations to finally yield a weight update for stochastic learning.

Target propagation aims to take advantage of the availability of approximate inverses to compute
better descent directions for the objective at hand. Bengio et al. (2013); Bengio (2020) argued that
the approach could be relevant for problems involving multiple compositions such as the training of
Recurrent Neural Networks (RNNs), which generally suffer from the phenomenon of exploding or
vanishing gradients (Hochreiter, 1998; Bengio et al., 1994; ?). Recently, empirical results indeed
showed the potential advantages of target propagation over classical gradient back-propagation for
training RNNs on several tasks (Manchev & Spratling, 2020). However, these recent investigations
remain built on multiple approximations, which hinder the analysis of the core idea of TP, i.e., using
layer inverses.

On the theoretical side, difference target propagation, a modern variant of target propagation, was
related to an approximate Gauss-Newton method, suggesting interesting venues to explain the benefits of target propagation (Bengio, 2020; Meulemans et al., 2020). Previous works have considered
approximating inverses by adding multiple reverse layers (Manchev & Spratling, 2020; Meulemans
et al., 2020; Bengio, 2020). However, it is unclear whether such reverse layers actually learn layer
inverses during the training process. Even if they were, the additional cost of computational complexity of learning approximate inverses should be carefully accounted for.

In this work, we propose a simple target propagation approach, revisiting the original insights
of Le Cun et al. (1989) on the critical importance of the good conditioning of layer inverses. We
define regularized inverses through a variational formulation and we obtain approximate inverses
via these regularized inverses. In this spirit, we can also interpret the difference target propagation
formula (Lee et al., 2015) as a finite difference approximation of a linearized regularized inverse.
We propose a smoother formula that can directly be integrated into a differentiable programming
framework.


-----

GUadienW Back-PUoSagaWion TaUgeW PUoSagaWion

FoUZaUd

PaVV

BackZaUd

PaVV


Fig. 1: Our implementation of target propagation uses linearization of gradient inverses instead of gradients in
a backward pass akin to gradient back-propagation.

We detail the computational complexity of the proposed target propagation and compare it to the
one of gradient back-propagation, showing that the additional cost of computing inverses can be
effectively amortized for very long sequences. Following the benchmark of Manchev & Spratling
(2020), we observe that the proposed target propagation can perform better than classical gradientbased methods on several tasks involving RNNs.

**Related work.** Many variations of back-propagation algorithms have been explored; see Werbos

(1994); Goodfellow et al. (2016) for an extensive bibliography. Closer to target propagation, penalized formulations of the training problem have been considered to decouple the optimization of
the weights in a distributed way or using an ADMM approach (Carreira-Perpinan & Wang, 2014;
Taylor et al., 2016; Gotmare et al., 2018). Rather than modifying the backward operations in the
layers, one can also modify the weight updates for deep forward networks by using a regularized
inverse (Frerix et al., 2018). Wiseman et al. (2017) recast target propagation as an ADMM-like algorithm for language modeling and reported disappointing experimental results. Recently, in a careful
experimental benchmark evaluation, Manchev & Spratling (2020) explored further target propagation to train RNNs, mapping a sequence to a single final output, in an attempt to understand the benefits of target propagation to capture long-range dependencies, and obtained promising experimental
results. Another line of research has considered synthetic gradients that approximate gradients using an additional layer instead of using back-propagated gradients (Jaderberg et al., 2017; Czarnecki
et al., 2017) to speed up the training of deep neural networks. Recently, Ahmad et al. (2020); Dalm
et al. (2021) considered using analytical inverses to implement target propagation and blend it with
what they called a gradient-adjusted incremental formula. Yet, an additional orthogonality penalty
is critical for their approach to work. Recently, Meulemans et al. (2020) considered using as many
reverse layers as forwarding operations. We focus here on the optimization gains of using target
propagation that cannot be obtained by adding a prohibitive number of reverse layers. Finally, we
do not discuss the biological plausibility of TP since we are unable to comment on this. We refer
the interested reader to, e.g., (Bengio, 2020).


**Notations.** For f : R[p] _⇥_ R[q] _! R[d], we denote @xf_ (x, y) =


_@f_ _[j](x, y)/@xi_


_i,j_

_[2][ R][d][⇥][p][.]_


2 TARGET PROPAGATION WITH LINEARIZED REGULARIZED INVERSES

While target propagation was initially developed for multi-layer neural networks, we focus on its
implementation for recurrent neural networks, as we shall follow the benchmark of Manchev &
Spratling (2020) in the experiments. Recurrent Neural Networks (RNNs) are also a canonical family
of neural networks in which interesting phenomena arise in back-propagation algorithms.

**Problem setting.** A simple RNN parameterized by ✓ = (Whh, Wxh, bh, Why, by) maps a se
quence of inputscorresponding to the inputs x1:⌧ = (x1 x, . . ., xt. _⌧_ ) to an output ˆy = g✓(x1:⌧ ) by computing hidden states ht 2 R[p]

Formally, the output ˆy and the hidden states ht are computed as an output operation following

transition operations defined as

_yˆ = c✓(h⌧_ ) := s(Whyh⌧ + by),

_ht = f✓,t(ht_ 1) := a(Wxhxt + Whhht 1 + bh) for t 1, . . ., ⌧ _,_
_−_ _−_ _2 {_ _}_


-----

where s is, e.g., the soft-max function for classification tasks, a is a non-linear operation such as
the hyperbolic tangent function, and the initial hidden state is generally fixed as h0 = 0. Given
samples of sequence-output pairs (x1:⌧ _, y), the RNN is trained to minimize the error `(y, g✓(x1:⌧_ ))
of predicting ˆy = g✓(x1:⌧ ) instead of y.

As one considers longer sequences, RNNs face the challenge of exploding/vanishing gradients
_@g✓(x1:⌧_ )/@ht (Bengio & Frasconi, 1995); see Appendix A for more discussion. We acknowledge that specific parameterization-based strategies have been proposed to address this issue of exploding/vanishing gradients, such as orthonormal parameterizations of the weights (Arjovsky et al.,
2016; Helfrich et al., 2018; Lezcano-Casado & Martınez-Rubio, 2019). The focus here is to simplify
and understand target propagation as a backpropagation-type algorithm using RNNs as a workbench.
Indeed, training RNNs is an optimization problem involving multiple compositions for which approximate inverses can easily be available. The framework could also be potentially applied to, e.g.,
time-series or control models (Roulet et al., 2019).

Given the parameters Whh, Wxh, bh of the transition operations, we can get approximate inverses
of f✓,t(ht−1) for all t 2 {1, . . ., ⌧ _}, that yield optimization surrogates that can be better perform-_
ing than the ones corresponding to regular gradients. We present below a simple version of target
propagation based on regularized inverses and inverse linearizations.

**Back-propagating targets.** The idea of target propagation is to compute virtual targets vt for each

layer t = ⌧, . . ., 1 such that if the layers were able to match their corresponding target at time t, i.e.,
_f✓,t(ht−1) ⇡_ _vt, the objective would decrease. The final target v⌧_ is computed as a gradient step on
the loss w.r.t. h⌧ . The targets are then back-propagated using an approximate inverse[1] _f✓,t[−][1]_ [of][ f][✓,t]

at each time step.

Formally, consider an RNN that computed ⌧ states h1, . . ., h⌧ from a sequence x1, . . ., x⌧ with
associated output y. For a given stepsize γh > 0, we propose to back-propagate targets by computing

_v⌧_ = h⌧ _γh@h`(y, c✓(h⌧_ )), (1)
_−_

_vt−1 = ht−1 + @hf✓,t[−][1][(][h][t][)]>(vt −_ _ht),_ for t 2 {⌧, . . ., 1}. (2)

The update rule (2) blends two ideas: i) regularized inversion; ii) linear approximation. We shall
describe below that our update (2) allows us to interpret the “magic formula” of difference target
propagation in Eq. 15 of Lee et al. (2015) as 0th-order finite difference approximation, while ours is
a 1st-order linear approximation. We shall also show that (2) puts in practice an insight from Bengio
(2020) suggesting to use the inverse of the gradients in the spirit of a Gauss-Newton method.

Once all targets are computed, the parameters of the transition operations are updated such that the
outputs of f✓,t at each time step move closer to the given target. Formally, the update consists of a
gradient step with stepsize γ✓ on the squared error between the targets and the current outputs, i.e.,
for ✓h _Whh, Wxh, bh_,
_2 {_ _}_


_✓h[next]_


_@✓h_ _f✓,t(ht_ 1) _vt_ 2[/][2][.] (3)
_k_ _−_ _−_ _k[2]_


= ✓h _γ✓_
_−_


_t=1_


As for the parameters ✓y = (Why, by) of the output operation, they are updated by a simple gradient
step on the loss with a stepsize γ✓.

2.1 REGULARIZED INVERSION

To explore further the original idea of Le Cun et al. (1989), we consider using the variational definition of the inverse,

_f✓,t[−][1][(][v][t][) = argmin]_ 2 [= argmin] 2[.] (4)

_vt−12R[p][ k][f][✓,t][(][v][t][−][1][)][ −]_ _[v][t][k][2]_ _vt−12R[p][ k][a][(][W][xh][x][t][ +][ W][hh][v][t][−][1][ +][ b][h][)][ −]_ _[v][t][k][2]_

As long as vt belongs to the image f✓,t(R[p]) of f✓,t, this definition recovers exactly the inverse
of vt by f✓,t. More generally, if vt 62 f✓,t(R[p]), Eq. (4) computes the best approximation of the

1In the following, to ease the presentation, we abuse notations and denote approximate inverses by f −✓,t1[.]


-----

inverse in the sense of the Euclidean projection. When one considers an activation function a and
_✓h = (Whh, Wxh, bh), the solution of (4) can easily be computed._

Formally, for the sigmoid, the hyperbolic tangent or the ReLU, their inverse can be obtained analytically for any vt 2 a(R[p]). So for vt 2 a(R[p]) and Whh full rank, we get

_f✓,t[−][1][(][v][t][) = (][W][ >]hh[W][hh][)][−][1][W][ >]hh[(][a][−][1][(][v][t][)][ −]_ _[W][xh][x][t]_ _[−]_ _[b][h][)][.]_

Ifthe linear operation. To account for non-invertible matrices vt 62 a(R[p]), the minimizer of (4) is obtained by first projecting Whh, we also add a regularization in vt onto a(R[p]), before inverting
the computation of the inverse. Overall we consider approximating the inverse of the layer by a
regularized inverse of the form

_f✓,t[−][1][(][v][t][) = (][W][ >]hh[W][hh]_ [+][ r][ I)][−][1][W][ >]hh[(][a][−][1][(][⇡][(][v][t][))][ −] _[W][xh][x][t]_ _[−]_ _[b][h][)][,]_

with r > 0 and ⇡ a projection onto a(R[p]).


**Regularized inversion vs.** **parameterized inversion.** Bengio (2014); Manchev & Spratling

(2020) parameterize the inverse as a reverse layer such that

_f✓,t[−][1][(][v][t][) =]_ _[✓][0][,t][(][v][t][) :=][ a][(][W][xh][x][t][ +][ V v][t][ +][ c][)][,]_

and learn the parameters ✓[0] = (V, c) for this reverse layer to approximate the inverse of the forward
computations. The parameterized layer needs to be learned to get a good approximation which involves numerically solving an optimization problem for each layer. These optimization problems
come with a computational cost that can be better controlled by using regularized inversions presented earlier.

However, the approach based on parameterized inverses may lack theoretical grounding, as pointed
out by Bengio (2020), as we do not know how close the learned inverse is to the actual inverse
throughout the training process. In contrast, the regularized inversion (4) is less ad hoc and clearly
defined and, as we shall show in the experiments, leads to competitive performance on real datasets.

In any case, the analytic formulation of the inverse gives simple insights on an approach with parameterized inverses. Namely, the analytical formula suggests parameterizing the reverse layer s.t.
(i) the reverse activation is defined as the inverse of the activation and not any activation, (ii) the
layer uses a non-linear operation followed by a linear one instead of the usual scheme, i.e., a linear
operation followed by a non-linear one.

2.2 LINEARIZED INVERSION

Earlier instances of target propagation used direct inverses of the network layers such that the target
propagation update formula would read vt−1 = f✓,t[−][1][(][v][t][)][ in (][2][). Yet, we are unaware of a success-]

ful implementation of TP using directly the inverses. To circumvent this issue, Lee et al. (2015)
proposed the difference target propagation formula that back-propagates the targets as

_vt−1 = ht−1 + f✓,t[−][1][(][v][t][)][ −]_ _[f][ −]✓,t[1][(][h][t][)][.]_

If the inverses were exact, the difference target propagation formula would reduce to vt 1 =
_−_
_f✓,t[−][1][(][v][t][)][.][ Lee et al.][ (][2015][) introduced the difference target propagation formula to mitigate the]_

approximation error of the inverses by parameterized layers. In practice, difference-type target propagation appears to be the only known successful implementation of target propagation we are aware
of. The difference target propagation formula can naturally be interpreted as an approximation of
the linearization used in (2), as

_f✓,t[−][1][(][v][t][)][ −]_ _[f][ −]✓,t[1][(][h][t][) =][ @][h][f][ −]✓,t[1][(][h][t][)]>(vt −_ _ht) + O(kvt −_ _htk22[)][,]_

where @hf✓,t[−][1][(][h][t][)]> denotes the Jacobian of the inverse of the layer at ht.


We show in Appendix D that the first-order approximation we propose (2) leads to slightly better
training curves than the finite-difference approximation. Moreover, our interpretation illuminates
the “mystery” of this formula, which appeared to be critical to the success of target propagation.


-----

... ...

... ...

... ...


Fig. 2: The graph of computations of target propagation is the same as the one of gradient back-propagation
except that f needs to be computed and Jacobian of the inverses, @hf are used instead of gradients @hf

_[−][1]_ _[−][1][ >]_
in the transition operations.

3 GRADIENT BACK-PROPAGATION VERSUS TARGET PROPAGATION

**Graph of computations.** Gradient back-propagation and target propagation both compute a de
scent direction for the objective at hand. The difference lies in the oracles computed and stored
in the forward pass, while the graph of computations remains the same. To clarify this view, we
reformulate target propagation in terms of displacementsread _λt := vt −_ _ht such that Eq. (1), (2) and (3)_


_λ⌧_ = −γh@h`(y, c✓(h⌧ )), _λt−1 = @hf✓,t[−][1][(][h][t][)]>λt,_ for t 2 {⌧, . . ., 1},


_@✓h_ _f✓,t(ht−1)λt,_ _✓h[next]_


_d✓h =_


= ✓h + γhd✓h _._


_t=1_


Target propagation amounts then to computing a descent direction d✓h for the parameters ✓h with
a graph of computations, illustrated in Fig. 2, analogous to that of gradient-back-propagation illustrated in Appendix A. The difference lies in the use of the Jacobian of the inverse

_@hf✓,t[−][1][(][h][t][)]>_ instead of _@hf✓,t(ht−1)._

The implementation of TP with the formula (2) can be done in a differentiable programming framework, where, rather than computing the gradient of the layer, one evaluates the inverse and keep the
Jacobian of the inverse. With the precise graph of computation of TP and BP, we can compare their
computational complexity explicitly and bound the difference of the directions they output.

**Arithmetic complexity.** Clearly, the space complexities of gradient back-propagation (BP) and

our implementation of target propagation (TP) are the same since the Jacobians of the inverse, and
the original gradients have the same size. In terms of time complexity, TP appears at first glance
to introduce an important overhead since it requires the computation of some inverses. However, a
close inspection of the formula of the regularized inverse reveals that a matrix inversion needs to be
computed only once for all time steps. Therefore the cost of the inversion may be amortized if the
length of the sequence is particularly long.

Formally, the time complexity of the forward-backward pass of gradient back-propagation is essentially driven by matrix-vector products, i.e.,


BP = (f✓,t) + (@hf✓,t) + (@✓h _f✓,t)_ + (@hf✓,t(ht 1)) + (@✓f✓,t(ht 1))
_T_ 2T _T_ _T_ _T_ _−_ _T_ _−_ 3

Xt=1 Forward Backward

4 5

_⌧_ (dp + p[2] + pq) + ⌧ (p[2] + pq),
_⇡_ | {z } | {z }

where d is the dimension of the input xt, q is the dimension of the parameters ✓h, for a function
_f we denote by T (f_ ) the time complexity to evaluate f and we consider e.g. @✓f✓,t(ht−1)) as the
linear function λ ! @✓f✓,t(ht−1))λ.


-----

On the other hand, the time complexity of target propagation is

_⌧_

_TTP =_ 2T (f✓,t)+T (f[ −]✓,t[1][)+][T][ (][@][✓]h _[f][✓,t][) +][T][ (][@][h][f][ −]✓,t[1][)]_ + T (@hf✓,t[−][1][)(][h][t][)][>][)+][T][ (][@][✓][f][✓,t][(][h][t][−][1][))]3

_t=1_

X Forward Backward

6 7
4 5

+ P(f|✓,t[−][1][)][,] {z } | {z }

where P(f✓,t[−][1][)][ is the cost of encoding the inverse, which, in our case, amounts to the cost of encoding]

_g✓_ : z ! (Whh[>] _[W][hh][ +][ r][ I)][−][1][W][ >]hh[,][ such that our regularized inverse can be computed as][ f][ −]✓,t[1][(][v][t][) =]_

_g✓(a[−][1](vt)_ _Wxhxt+bh). Encoding g comes at the cost of inverting one matrix of size p. Therefore,_
_−_
the time-complexity of target propagation can be estimated as

_TTP ⇡_ _p[3]_ + ⌧ (dp + p[2] + pq) + ⌧ (p[2] + pq) ⇡TBP if ⌧ _≥_ _p,_

i.e., for long sequences whose length is larger than the dimension of the hidden states, the cost of TP
with regularized inverses is approximately the same as the cost of BP. If a parameterized inverse was
used rather than a regularized inverse, the cost of encoding the inverse would correspond to the cost
of updating the reverse layers by, e.g., a stochastic gradient descent. This update has a cost similar
to BP. However, it is unclear whether these updates get us close to the actual inverses.


**Bounding the difference between target propagation and gradient back-propagation.** As the

computational graphs of BP and TP are the same, we can bound the difference between the oracles
returned by both methods. First, note that the updates of the parameters of the output functions are
the same since, in TP, gradients steps of the loss are used to update these parameters. The difference
between TP and BP lies in the updates with respect to the parameters of the transition operations.
For BP, the updates are computed by chain rule as

_⌧_

_@✓h_ _` (y, g✓(x1:⌧_ )) = _@✓h_ _f✓,t(ht−1)_ _[@h]@h[⌧]t_ _@h`(y, c✓(h⌧_ )),

_t=1_

X

where the term @h⌧ _/@ht decomposes along the time steps as @h⌧_ _/@ht =_ _s=t+1_ _[@][h][f][✓,s][(][h][s][−][1][)][.]_

The direction computed by TP has the same structure, namely it can be decomposed for γh = 1 as

_⌧_ _@hˆ_ _⌧_ [Q][⌧]

_d✓_ = _t=1_ _@✓h_ _f✓,t(ht−1)_ _@hˆ_ _t_ _@h`(y, c✓(h⌧_ )),

X

where _@h[ˆ]_ _⌧_ _/@h[ˆ]_ _t =_ _s=t+1_ _[@][h][f][ −]✓,s[1][(][h][s][)]>. We can then bound the difference between the directions_

given by BP or TP as, for any matrix norm k · k as formally stated in the following lemma.

[Q][⌧]

**Lemma 3.1.** _The difference between the oracle returned by gradient back-propagation_

_@✓h_ _` (y, g✓(x1:⌧_ )) and the oracle returned by target propagation can be bounded as


_k@hf✓,t(ht−1) −_ _@hf✓,t[−][1][(][h][t][)]>k,_


_@✓h_ _` (y, g✓(x1:⌧_ )) _d✓_ _c_ sup
_k_ _−_ _k _ _t=1,...,⌧_


_where_ _c_ = _⌧t=1_ _ts−=01_ _[a][s][b][t][−][1][−][s]_ _with_ _a_ = supt=1,...⌧ _k@hf✓,t(ht−1)k, b_ =

supt=1,...⌧ _k@hf✓,t[−][1][(][h]P[t][)]>k.P_

_For regularized inverses, we have, denoting ut = Wxhxt + Whhht_ 1 + bh,
_−_

_k@hf✓,t(ht−1) −_ _@hf✓,t[−][1][(][h][t][)]>k kW >hh[k]_ _kra(ut) −ra(ut)[−][1]k + k I −(Whh[>]_ _[W][hh]_ [+][ r][ I)][−][1][kkr][a][(][u][t][)][−][1][k]

⇣

For the two oracles to be close, we then need the preactivation ut = Wxhxt + Whhht 1 + bh to
_−_
lie in the region of the activation function that is close to being linear s.t. _a(ut)_ I. We also
_r_ _⇡_
need (Whh[>] _[W][hh][ +][ r][ I)][−][1][ to be close to the identity which can be the case if, e.g.,][ r][ = 0][ and the]_

weight matrices Wh were orthonormal. By initializing the weight matrices as orthonormal matrices,
the differences between the two oracles can be closer. However, in the long term, target propagation
appears to give better oracles, as shown in the experiments below.


-----

|TP BP|Col2|
|---|---|
|||





TP BP

4000 4000 100

Train Loss2000 2000 50

0 0 0

0 2 4 0 2 4 0 1 2

Iterations _⇥10[4]_ Iterations _⇥10[4]_ Iterations _⇥10[4]_

100 100 100

75 75 75

50 50 50

Accuracy

25 25 25

0 0 0

0 2 4 0 2 4 0 1 2

Iterations _⇥10[4]_ Iterations _⇥10[4]_ Iterations _⇥10[4]_


Fig. 3: Temporal order problem T = 60, Temporal Problem T = 120, Adding problem T = 30.

**Target propagation as a Gauss-Newton method?** Recently target propagation has been inter
preted as an approximate Gauss-Newton method, by considering that the difference target propagation formula approximates the linearization of the inverse, which itself is a priori equal to the
inverse of the gradients (Bengio, 2020; Meulemans et al., 2020; 2021). Namely, provided that
_f✓,t[−][1][(][f][✓,t][(][h][t][−][1][))][ ⇡]_ _[h][t][−][1][ such that][ @][h][f][✓,t][(][h][t][−][1][)][@][h][f][ −]✓,t[1][(][h][t][)][ ⇡]_ [I][, we have]

_@hf✓,t[−][1][(][h][t][)][ ⇡]_ [(][@][h][f][✓,t][(][h][t][−][1][))][−][1][ .]

By composing the inverses of the gradients, we get an update similar to the one of Gauss-Newton
(GN) method. Namely, recall that if n invertible functions f1, . . ., fn were composed to solve a
least square problem of the form kfn ◦ _. . . ◦_ _f1(x) −_ _yk2[2][, a Gauss-Newton update would take the]_

form x[(][k][+1)] = x[(][k][)] _−_ _@x0_ _f1(x0)[−>]@x1_ _f2(x1)[−>]_ _. . . @xn−1_ _f_ (xn−1)[−>](xn − _y) where xt is defined_
iteratively as x0 = x[(][k][)], xt+1 = ft(xt). In other words, GN and TP share the idea of composing the
inverse of gradients. However, numerous differences remain: (i) in e.g. RNNs, the gradients w.r.t.
to the weights are defined as a sum of the composition of the gradients and the inverse of this sum
is a priori not the sum of the inverses, (ii) if the real rationale of GN was used, the gradients w.r.t.
the loss should also be inverted, and the gradients w.r.t. to the weights should also be inverted which
is not the case in the usual implementation of TP Lee et al. (2015); Bengio (2020). Even if TP was
approximating GN, it is unclear whether GN updates are adapted to stochastic problems.


4 EXPERIMENTS

In the following, we compare our simple target propagation approach, which we shall refer to as TP,
to gradient Back-Propagation referred to as BP. We follow the experimental benchmark of Manchev
& Spratling (2020) to which we add results on RNNs on CIFAR and GRUs on FashionMNIST.
Additional experiments, details on the initialization and the hyper-parameter selection can be found
in Appendix D.

**Data.** We consider two synthetic datasets generated to present training difficulties for RNNs and

several real datasets consisting of scanning images pixel by pixel to classify them (Hochreiter &
Schmidhuber, 1997; Le et al., 2015; Manchev & Spratling, 2020).

_Temporal order problem. A sequence of length T is generated using a set of randomly chosen_
symbols _a, b, c, d_ . Two additional symbols X and Y are added at positions t1 [T/10, 2T/10]
and t2 _{ [4T/10, 5}T/10]. The network must predict the correct order of appearance of 2_ _X and Y_
out of four possible choices 2 _{XX, XY, Y X, Y Y }._


-----

|TP BP|Col2|Col3|Col4|
|---|---|---|---|
||7200 7100 7000|||






TP BP

8000 8000

7200 8000

6000 6000

Train Loss4000 4000 7100 7000

2000 2000 7000 6000

0 2 4 0 2 4 0.0 2.5 5.0 7.5 0.0 2.5 5.0 7.5

100 Iterations _⇥10[4]_ 100 Iterations _⇥10[4]_ Iterations _⇥10[3]_ 40 Iterations _⇥10[3]_

75 75 20 30

50 50 20

Accuracy 10

25 25 10

0 0 2 4 0 0 2 4 00.0 2.5 5.0 7.5 00.0 2.5 5.0 7.5

Iterations _⇥10[4]_ Iterations _⇥10[4]_ Iterations _⇥10[3]_ Iterations _⇥10[3]_


Fig. 4: Image classification pixel by pixel. From left to right: MNIST, MNIST with permuted images, CIFAR10, FashionMNIST with GRU.

_Adding problem. The input consists of two sequences: one is made of randomly chosen numbers_
from [0, 1], and the other one is a binary sequence full of zeros except at positions t1 [1, T/10]
andthe network is to output the mean of the two random numbers of the first sequence t2 2 [T/10, T/2]. The second position acts as a marker for the time steps t1 and ( tX2t 2. The goal of1 + Xt2 )/2.

_Image classification pixel by pixel._ The inputs are images of (i) grayscale handwritten digits

given in the database MNIST (LeCun & Cortes, 1998), (ii) colored objects from the database
CIFAR10 (Krizhevsky, 2009) or (iii) grayscale images of clothes from the database FashionMNIST (Xiao et al., 2017). The images are scanned pixel by pixel and channel by channel for CIFAR10, and fed to a sequential network such as a simple RNN or a GRU network (Cho et al.,
2014). The inputs are then sequences of 28 ⇥ 28 = 784 pixels for MNIST or FashionMNIST and
32 ⇥ 32 ⇥ 3 = 3072 pixels for CIFAR with a very long-range dependency problem. We also consider permuting the images of MNIST by a fixed permutation before feeding them into the network,
which gives potentially longer dependencies in the sequential data.

**Model.** In both synthetic settings, we consider randomly generated mini-batches of size 20, a sim
ple RNN with hidden states of dimension 100, and hyperbolic tangent activation. For the temporal
order problem, the last layer uses a soft-max function on top of a linear operation, and the loss is
the cross-entropy. For the adding problem, the last layer is linear, the loss is the mean-squared error,
and a sample is considered to be accurately predicted if the mean squared error is less than 0.04 as
done by (Manchev & Spratling, 2020).

For the classification of images with sequential networks, we consider mini-batches of size 16 and a
cross-entropy loss. For MNIST and CIFAR, we consider a simple RNN with hidden states of dimension 100, hyperbolic tangent activation, and a softmax output. For FashionMNIST, we consider a
GRU network and adapted our implementation of target propagation to that case while using hidden
states of dimension 100 and a softmax output.


**Target propagation can tackle long sequences better than gradient back-propagation.** In

Fig. 3, we observe that TP performs better than BP on the temporal ordering problem: it is able
to reach 100% accuracy in fewer iterations than BP for sequences of length 60 and, for sequences
of length 120, it is still able to reach 100% accuracy in fewer than 40 000 iterations while BP is
not. On the other hand, for the adding problem, TP performs less well than BP. The contrast in
performance between the two synthetic tasks was also observed by (Manchev & Spratling, 2020)
using difference target propagation with parameterized inverses. The main difference between

these tasks is the different nature of the outputs, which are binary for the temporal problem and
continuous for the adding problem.

In Fig. 4, we observe that TP generally performs better than BP for image classification tasks. For
the MNIST dataset, it reaches around 74% accuracy after 4 _·_ 10[4] iterations. This phenomenon is also


-----

Region of parameters

Zith conYergence


appear to reach a significant accuracy, though TP is still faster. On the FashionMNIST dataset,

512

256 TP

Width12864

32 BP

16

4 14 49 196 392 784

Length

(6b) Perf. vs width & length.

observed with permuted images, where the optimization appears smoother, and TP obtains around
iterations and is still faster than BP. On the CIFAR dataset, no algorithms

where a GRU network is used, our implementation of TP performs on par with BP, which shows
that our approach can be generalized to more complex networks than a simple RNN.

**Target propagation requires a non-zero regularization term.** As mentioned in Sec. 3, by using

an analytical formula to compute the inverse of the layers, we can question the interpretation of TP
as a Gauss-Newton method, which would amount to TP without regularization. To understand the
effect of the regularization term, we computed the area under the training loss curve of TP for 400
iterations on a log10 grid of varying step-sizes γ✓ and regularizations r for a fixed γh = 10[−][3]. The
results are presented in Fig. 6a, where the smaller the area, the brighter the point and the absence
of dots in the grid mean that the algorithm diverged. Fig. 6a shows that without regularization we
were not able to obtain convergence of the algorithm. Simply using the gradients of the inverse as in
a Gauss-Newton method may not directly work for RNNs. Additional modifications of the method
could be added to make target propagation closer to Gauss-Newton, such as inverting the layers with
respect to their parameters as proposed by Bengio (2020). For now, the regularization appears to
successfully handle the rationale of target propagation.

**Target propagation is adapted for long sequences.** In Fig. 6b, we compare the performance of

BP and TP in terms of accuracy after 400 iterations on the MNIST problem for various widths determined by the size of the hidden states and various lengths determined by the size of the inputs (i.e.,
we feed the RNN with k pixels at a time, which gives a length 784/k). Fig 6b shows that TP is generally appropriate for long sequences, while BP remains more efficient for short sequences. TP can
then be seen as an interesting alternative for dynamical problems which involve many discretization
steps as in RNNs and related architectures.

CONCLUSION

We proposed a simple target propagation approach grounded in two important computational components, regularized inversion, and linearized propagation. The proposed approach also sheds light
on previous insights and successful rules for target propagation. We will publicly release our code
to facilitate the reproduction of the results. We have used target propagation within a stochastic gradient outer loop to train neural networks for a fair comparison to stochastic gradient using gradient
backpropagation. Developing adaptive stochastic gradient algorithms in the spirit of Adam that lead
to boosts in performance when using target propagation instead of gradient backpropagation is an
interesting avenue for future work. Continuous counterparts of target propagation in a neural ODE
spirit is also an interesting avenue for future work.

REFERENCES

Nasir Ahmad, Marcel A van Gerven, and Luca Ambrogioni. Gait-prop: A biologically plausible

learning rule derived from backpropagation of error. Advances in Neural Information Processing
_Systems, 33, 2020._


-----

Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In

_Proceedings of the 33rd International Conference on Machine Learning, 2016._

Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target

propagation. arXiv preprint arXiv:1407.7906, 2014.

Yoshua Bengio. Deriving differential target propagation from iterating approximate inverses. arXiv

_preprint arXiv:2007.15139, 2020._

Yoshua Bengio and Paolo Frasconi. Diffusion of context and credit information in markovian mod
els. Journal of Artificial Intelligence Research, 3:249–270, 1995.

Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient

descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients

through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.

Miguel Carreira-Perpinan and Weiran Wang. Distributed optimization of deeply nested systems. In

_Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, 2014._

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol
ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder
for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.

Wojciech Marian Czarnecki, Grzegorz Swirszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals,[´]

and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In
_Proceedings of the 34th International Conference on Machine Learning, 2017._

Sander Dalm, Nasir Ahmad, Luca Ambrogioni, and Marcel van Gerven. Scaling up learning with

gait-prop. arXiv preprint arXiv:2102.11598, 2021.

Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex

optimization with inexact oracle. Mathematical Programming, 146(1-2):37–75, 2014.

Thomas Frerix, Thomas Möllenhoff, Michael Moeller, and Daniel Cremers. Proximal backpropa
gation. In Proceedings of the 6th International Conference on Learning Representations, 2018.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. The MIT Press, 2016.

Akhilesh Gotmare, Valentin Thomas, Johanni Brea, and Martin Jaggi. Decoupling backpropagation

using constrained optimization methods. In Credit Assignment in Deep Learning and Reinforce_ment Learning Workshop (ICML 2018 ECA), 2018._

Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled

cayley transform. In Proceedings of the 35th International Conference on Machine Learning,
2018.

Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem

solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):
107–116, 1998.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):

1735–1780, 1997.

Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David

Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Pro_ceedings of the 34th International Conference on Machine Learning, 2017._

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University

of Toronto, 2009.

Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks

of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.


-----

Yann Le Cun. Learning process in an asymmetric threshold network. In Disordered systems and

_biological organization. Springer, 1986._

Yann Le Cun, Conrad C Galland, and Geoffrey E Hinton. GEMINI: gradient estimation through

matrix inversion after noise injection. In Advances in neural information processing systems, pp.
141–148, 1989.

Yann LeCun and Corinna Cortes. MNIST handwritten digit database.

http://yann.lecun.com/exdb/mnist/, 1998.

Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation.

In Machine Learning and Knowledge Discovery in Databases. Springer, 2015.

Mario Lezcano-Casado and David Martınez-Rubio. Cheap orthogonal constraints in neural net
works: A simple parametrization of the orthogonal and unitary group. In Proceedings of the 36th
_International Conference on Machine Learning, 2019._

Nikolay Manchev and Michael Spratling. Target propagation in recurrent neural networks. Journal

_of Machine Learning Research, 21(7):1–33, 2020._

Alexander Meulemans, Francesco Carzaniga, Johan Suykens, João Sacramento, and Benjamin F.

Grewe. A theoretical framework for target propagation. In Advances in Neural Information

_Processing Systems 33, 2020._

Alexander Meulemans, Matilde Tristany Farinha, Javier García Ordóñez, Pau Vilimelis Aceituno,

João Sacramento, and Benjamin F Grewe. Credit assignment in neural networks through deep
feedback control. arXiv preprint arXiv:2106.07887, 2021.

Piotr Mirowski and Yann LeCun. Dynamic factor graphs for time series modeling. In Machine

_Learning and Knowledge Discovery in Databases. Springer, 2009._

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob
lem. arXiv preprint arXiv:1211.5063, 2012.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor

Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner,
Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep
learning library. In Advances in Neural Information Processing Systems 32. 2019.

Richard Rohwer. The “moving targets" training algorithm. In Advances in neural information

_processing systems 3, 1990._

Vincent Roulet, Siddhartha Srinivasa, Dmitriy Drusvyatskiy, and Zaid Harchaoui. Iterative lin
earized control: stable algorithms and complexity guarantees. In International Conference on
_Machine Learning, 2019._

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning Internal Representations by Error

_Propagation. MIT Press, Cambridge, MA, USA, 1986._

Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net
works. In Proceedings of the 28th International Conference on Machine Learning, 2011.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial
ization and momentum in deep learning. In Proceedings of the 30th International conference on
_machine learning, 2013._

Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Training

neural networks without gradients: A scalable ADMM approach. In Proceedings of the 33rd
_International Conference on Machine Learning, 2016._

Paul Werbos. The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and

_Political Forecasting. Wiley-Interscience, 1994._


-----

Sam Wiseman, Sumit Chopra, Marc-Aurelio Ranzato, Arthur Szlam, Ruoyu Sun, Soumith Chin
tala, and Nicolas Vasilache. Training language models using target-propagation. arXiv preprint
_arXiv:1702.04770, 2017._

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset

for benchmarking machine learning algorithms, 2017. URL https://github.com/

zalandoresearch/fashion-mnist.


-----

