# HOW DOES SIMSIAM AVOID COLLAPSE WITHOUT NEGATIVE SAMPLES? A UNIFIED UNDERSTANDING
## WITH SELF-SUPERVISED CONTRASTIVE LEARNING

**Chaoning Zhang[∗]& Kang Zhang[∗]** **& Chenshuang Zhang & Trung X. Pham**
**Chang D. Yoo & In So Kweon**
Korea Advanced Institute of Science and Technology (KAIST), South Korea
chaoningzhang1990@gmail.com & zhangkang@kaist.ac.kr

ABSTRACT

To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely
used but often requires a large number of negative samples. Without negative
samples yet achieving competitive performance, a recent work (Chen & He, 2021)
has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse
without negative samples remains not fully clear and our investigation starts by
revisiting the explanatory claims in the original SimSiam. After refuting their
claims, we introduce vector decomposition for analyzing the collapse based on
the gradient analysis of the l2-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a
unified perspective comes timely for understanding the recent progress in SSL.

1 INTRODUCTION

Beyond the success of NLP (Lan et al., 2020; Radford et al., 2019; Devlin et al., 2019; Su et al.,
2020; Nie et al., 2020), self-supervised learning (SSL) has also shown its potential in the field of
vision tasks (Li et al., 2021; Chen et al., 2021; El-Nouby et al., 2021). Without the ground-truth
label, the core of most SSL methods lies in learning an encoder with augmentation-invariant representation (Bachman et al., 2019; He et al., 2020; Chen et al., 2020a; Caron et al., 2020; Grill et al.,
2020). Specifically, they often minimize the representation distance between two positive samples,
_i.e. two augmented views of the same image, based on a Siamese network architecture (Bromley_
et al., 1993). It is widely known that for such Siamese networks there exists a degenerate solution,
_i.e. all outputs “collapsing” to an undesired constant (Chen et al., 2020a; Chen & He, 2021). Early_
works have attributed the collapse to lacking a repulsive component in the optimization goal and
adopted contrastive learning (CL) with negative samples, i.e. views of different samples, to alleviate
this problem. Introducing momentum into the target encoder, BYOL shows that Siamese architectures can be trained with only positive pairs. More recently, SimSiam (Chen & He, 2021) has caught
great attention by further simplifying BYOL by removing the momentum encoder, which has been
seen as a major milestone achievement in SSL for providing a minimalist method for achieving
competitive performance. However, more investigation is required for the following question:
**How does SimSiam avoid collapse without negative samples?**

Our investigation starts with revisiting the explanatory claims in the original SimSiam paper (Chen
& He, 2021). Notably, two components, i.e. stop gradient and predictor, are essential for the success
of SimSiam (Chen & He, 2021). The reason has been mainly attributed to the stop gradient (Chen
& He, 2021) by hypothesizing that it implicitly involves two sets of variables and SimSiam behaves
like alternating between optimizing each set. Chen & He argue that the predictor h is helpful in
SimSiam because h fills the gap to approximate expectation over augmentations (EOA).

Unfortunately, the above explanatory claims are found to be flawed due to reversing the two paths
with and without gradient (see Sec. 2.2). This motivates us to find an alternative explanation, for
which we introduce a simple yet intuitive framework for facilitating the analysis of collapse in SSL.

_∗equal contribution_


-----

Specifically, we propose to decompose a representation vector into center and residual components.
This decomposition facilitates understanding which gradient component is beneficial for avoiding
collapse. Under this framework, we show that a basic Siamese architecture cannot prevent collapse,
for which an extra gradient component needs to be introduced. With SimSiam interpreted as processing the optimization target with an inverse predictor, the analysis of its extra gradient shows that
(a) its center vector helps prevent collapse via the de-centering effect; (b) its residual vector achieves
dimensional de-correlation which also alleviates collapse.

Moreover, under the same gradient decomposition, we find that the extra gradient caused by negative samples in InfoNCE (He et al., 2019; Chen et al., 2020b;a; Tian et al., 2019; Khosla et al.,
2020) also achieves de-centering and de-correlation in the same manner. It contributes to a unified
understanding on various frameworks in SSL, which also inspires the investigation of hardnessawareness Wang & Liu (2021) from the inter-anchor perspective Zhang et al. (2022) for further
bridging the gap between CL and non-CL frameworks in SSL. Finally, simplifying the predictor for
more explainable SimSiam, we show that a single bias layer is sufficient for preventing collapse.

The basic experimental settings for our analysis are detailed in Appendix A.1 with a more specific
setup discussed in the context. Overall, our work is the first attempt for performing a comprehensive
study on how SimSiam avoids collapse without negative samples. Several works, however, have
attempted to demystify the success of BYOL (Grill et al., 2020), a close variant of SimSiam. A
technical report (Fetterman & Albrecht, 2020) has suggested the importance of batch normalization
(BN) in BYOL for its success, however, a recent work (Richemond et al., 2020) refutes their claim
by showing BYOL works without BN, which is discussed in Appendix B.

2 REVISITING SIMSIAM AND ITS EXPLANATORY CLAIMS

_l2-normalized vector and optimization goal. SSL trains an encoder f for learning discriminative_
representation and we denote such representation as a vector z, i.e. f (x) = z where x is a certain
input. For the augmentation-invariant representation, a straightforward goal is to minimize the distance between the representations of two positive samples, i.e. augmented views of the same image,
for which mean squared error (MSE) is a default choice. To avoid scale ambiguity, the vectors are
often l2-normalized, i.e. Z = z/ **_z_** (Chen & He, 2021), before calculating the MSE:
_||_ _||_

_LMSE = (Za −_ **_Zb)[2]/2 −_** 1 = −Za · Zb = Lcosine, (1)

which shows the equivalence of a normalized MSE loss to the cosine loss (Grill et al., 2020).

**Collapse in SSL and solution of SimSiam. Based on a Siamese architecture, the loss in Eq 1**
causes the collapse, i.e. f always outputs a constant regardless of the input variance. We refer to
this Siamese architecture with loss Eq 1 as Naive Siamese in the remainder of paper. Contrastive
loss with negative samples is a widely used solution (Chen et al., 2020a). Without using negative
samples, SimSiam solves the collapse problem via predictor and stop gradient, based on which the
encoder is optimized with a symmetric loss:

_LSimSiam = −(Pa · sg(Zb) + Pb · sg(Za)),_ (2)

where sg(·) is stop gradient and P is the output of predictor h, i.e. p = h(z) and P = p/||p||.

2.1 REVISING EXPLANATORY CLAIMS IN SIMSIAM

**Interpreting stop gradient as AO. Chen & He hypothesize that the stop gradient in Eq 2 is an**
implementation of Alternating between the Optimization of two sub-problems, which is denoted
as AO. Specifically, with the loss considered as (θ, η) = Ex, _θ(_ (x)) _ηx_, the op_L_ _T_ _F_ _T_ _−_

timization objective minθ,η (θ, η) can be solved by alternating ηh
[t] arg minη (θ[t], η) and
_θ[t][+1]_ arg minθ (θ, η[t]) L. It is acknowledged that this hypothesis does not fully explain why← _L[2][i]_
the collapse is prevented (Chen & He, 2021). Nonetheless, they mainly attribute SimSiam success← _L_
to the stop gradient with the interpretation that AO might make it difficult to approach a constant ∀x.

**Interpreting predictor as EOA. The AO problem (Chen & He, 2021) is formulated independent of**
predictor h, for which they believe that the usage of predictor h is related to approximating EOA for
filling the gap of ignoring E [ ] in a sub-problem of AO. The approximation of E [ ] is summarized
_T_ _·_ _T_ _·_


-----

in Appendix A.2. Chen & He support their interpretation by proof-of-concept experiments. Specifcan help prevent collapse without predictor (see Fig. 1 (b)). Given that the training completely failsically, they show that updating ηx with a moving-average ηx[t] _[←]_ _[m][ ∗]_ _[η]x[t]_ [+ (1][ −] _[m][)][ ∗F][θ][t]_ [(][T][ ′][(][x][))]
when the predictor and moving average are both removed, at first sight, their reasoning seems valid.

Moving average 2 _×_ 0.0108 46.57

2.2 DOES THE PREDICTOR FILL THE GAP TO APPROXIMATE EOA?

New Figure 1

**Reasoning flaw. Considering the stop gradient, we divide the framework into two sub-models with**
different paths and term them Gradient Path (GP) and Stop Gradient Path (SGP). For SimSiam, only
the sub-model with GP includes the predictor (see Fig. 1 (a)). We point out that their reasoning flaw
of predictor analysis lies in the reverse of GP and SGP. By default, the moving-average sub-model,
as shown in Fig. 1 (b), is on the same side as SGP. Note that Fig. 1 (b) is conceptually similar to
Fig. 1 (c) instead of Fig. 1 (a). It is worth mentioning that the Mirror SimSiam in Fig. 1 (c) is what
stop gradient in the original SimSiam avoids. Therefore, it is problematic to perceive h as EOA.

**GP** **SGP** **GP** **GP**

pa similarity similarity similarity pb

predictor h moving predictor h

average

za zb za zb **SGP** za zb **SGP**

encoder f encoder f encoder f encoder f encoder f encoder f

image x image x image x

(a) SimSiam (b) Moving Average (c) Mirror SimSiam

Figure 1: Reasoning Flaw in SimSiam. (a) Standard SimSiam architecture. (b) Moving-Average
Model proposed in the proof-of-concept experiment (Chen & He, 2021). (c) Mirror SimSiam, which
has the same model architecture as SimSiam but with the reverse of GP and SGP.

**GP** **SGP** **GP** **SGP** **GP**

similarity similarity similarity

inverse

predictor h[-1]

**SGP**

pa pb pa pb

predictor h predictor h predictor h predictor h

za zb za zb za zb

encoder f encoder f encoder f encoder f encoder f encoder f

image x image x image x

(a) Naive Siamese (b) SimSiam with Symmetric Predictor (c) SimSiam with Inverse Predictor

Figure 2: Different architectures of Siamese model. When it is trained experimentally, the inverse
predictor in (c) has the same architecture as predictor h.

Method # aug Collapse Std Top-1 (%)

Moving average 2 0.0108 46.57

Same batch 10 ✓ 0 1
Same batch 25 ✓ 0 1

Table 1: Influence of Explicit EOA. Detailed setup is reported in Appendix A.3

**Explicit EOA does not prevent collapse. (Chen & He, 2021) points out that “in practice, it would**
be unrealistic to actually compute the expectation E [ ]. But it may be possible for a neural net_T_ _·_
work (e.g., the preditor h) to learn to predict the expectation, while the sampling of T is implicitly
distributed across multiple epochs.” If implicitly sampling across multiple epochs is beneficial, explicitly sampling sufficient large N augmentations in a batch with the latest model would be more
beneficial to approximate E [ ]. However, Table 1 shows that the collapse still occurs and suggests
_T_ _·_
that the equivalence between predictor and EOA does not hold.


-----

2.3 ASYMMETRIC INTERPRETATION OF PREDICTOR WITH STOP GRADIENT IN SIMSIAM

**Symmetric Predictor does not prevent collapse. The difference between Naive Siamese and Sim-**
siam lies in whether the gradient in backward propagation flows through a predictor, however, we
show that this propagation helps avoid collapse only when the predictor is not included in the SGP
path. With h being trained the same as Eq 2, we optimize the encoder f through replacing the Z
in Eq 2 with P . The results in Table. 2 show that it still leads to collapse. Actually, this is well
expected by perceiving h to be part of the new encoder F, i.e. p = F (x) = h(f (x)). In other
words, the symmetric architectures with and without predictor h both lead to collapse.

**Predictor with stop gradient is asymmetric. Clearly, how SimSiam avoids collapse lies in its**
asymmetric architecture, i.e. one path with h and the other without h. Under this asymmetric architecture, the role of stop gradient is to only allow the path with predictor to be optimized with the
encoder output as the target, not vice versa. In other words, the SimSiam avoids collapse by excluding Mirror SimSiam (Fig. 1 (c)) which has a loss (mirror-like Eq 2) as LMirror = −(Pa _·Zb_ +Pb _·Za),_
where stop gradient is put on the input of h, i.e. pa = h(sg[za]) and pb = h(sg[zb]).

**Predictor vs. inverse predictor. We interpret h as a**
function mapping from z to p, and introduce a conceptual inverse mapping h[−][1], i.e. z = h[−][1](p). Here, Mirror SimSiamSimsiam ✓× 66.621
as shown in Table 2, SimSiam with symmetric predic- Naive Siamese ✓ 1
tor (Fig. 2 (b)) leads to collapse, while SimSiam (Fig. 1 Symmetric Predictor ✓ 1

|Method|Collapse Top-1 (%)|
|---|---|
|Simsiam Mirror SimSi Naive Siame Symmetric Pred|× 66.62 am ✓ 1 se ✓ 1 ictor ✓ 1|

(a)) avoids collapse. With the conceptual h[−][1], we in
Table 2: Results of various Siamese archi
terpret Fig. 1 (a) the same as Fig. 2 (c) which differs

tectures. Detailed trend and setup are re
from Fig. 2 (b) via changing the optimization target

ported in Appendix A.4

from pb to zb, i.e. zb = h[−][1](pb). This interpretation
suggests that the collapse can be avoided by processing the optimization target with h[−][1]. By contrast, Fig. 1 (c) and Fig. 2 (a) both lead to collapse, suggesting that processing the optimization
target with h is not beneficial for preventing collapse. Overall, asymmetry alone does not guarantee
collapse avoidance, which requires the optimization target to be processed by h[−][1] not h.

70

**Trainable inverse predictor and its implication on**
**EOA. In the above, we propose a conceptual inverse** 60
predictor h[−][1] in Fig. 2 (c), however, it remains yet 50
unknown whether such an inverse predictor is experi- 40
_mentally trainable. A detailed setup for this investiga-_ op-1 (%)30

T

tion is reported in Appendix A.5. The results in Fig. 3

20

show that a learnable h[−][1] leads to slightly inferior (I) SimSiam
performance, which is expected because h[−][1] cannot
make the trainable inverse predictor output zb[∗] [com-] 0 200 400 600 800 1000

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
||(I) SimSiam|||
||(II) Inverse|Predictor||


(I) SimSiam
(II) Inverse Predictor

lent to SimSiam ifpletely the same as z zb[∗]b. Note that it would be equiva-[=][ z][b][. Despite a slight perfor-] Figure 3: Comparison of original SimSiamEpoch
mance drop, the results confirm that h[−][1] is trainable. and SimSiam with Inverse Predictor.
The fact that h[−][1] is trainable provides additional evidence that the role h plays in SimSiam is not EOA
because theoretically h[−][1] cannot restore a random augmentation T _[′]_ from an expectation p, where
**_p = h(z) = E_** _θt_ ( (x)) .
_T_ _F_ _T_
h i


3 VECTOR DECOMPOSITION FOR UNDERSTANDING COLLAPSE

By default, InfoNCE (Chen et al., 2020a) and SimSiam (Chen & He, 2021) both adopt l2normalization in their loss for avoiding scale ambiguity. We treat the l2-normalized vector, i.e. Z,
as the encoder output, which significantly simplifies gradient derivation and the following analysis.

**Vector decomposition. For the purpose of analysis, we propose to decompose Z into two parts,**
**_Z = o + r, where o, r denote center vector and residual vector respectively. Specifically, the_**
center vector o is defined as an average of Z over the whole representation space oz = E[Z].
However, we approximate it with all vectors in current mini-batch, i.e. oz = _M1_ _Mm=1_ **_[Z][m][, where]_**

_M is the mini-batch size. We define the residual vector r as the residual part of Z, i.e. r = Z_ **_oz._**
P _−_


-----

3.1 COLLAPSE FROM THE VECTOR PERSPECTIVE

**Collapse: from result to cause. A Naive Siamese is well expected to collapse since the loss is**
designed to minimize the distance between positive samples, for which a constant constitutes an
optimal solution to minimize such loss. When the collapse occurs, ∀i, Zi = _M1_ _Mm=1_ **_[Z][m][ =][ o][z][,]_**

where i denotes a random sample index, which shows the constant vector is oz in this case. This
interpretation only suggests a possibility that a dominant o can be one of the viable solutions, whileP
the optimization, such as SimSiam, might still lead to a non-collapse solution. This merely describes
**_o as the consequence of the collapse, and our work investigates the cause of such collapse through_**
analyzing the influence of individual gradient components, i.e. o and r during training.

**Competition between o and r. Complementary to the Standard Deviation (Std) (Chen & He,**
2021), for indicating collapse, we introduce the ratio of o in z, i.e. mo = ||o||/||z||, where || ∗|| is
the L2 norm. Similarly, the ratio of r in z is defined as mr = ||r||/||z||. When collapse happens,
_i.e. all vectors Z are close to the center vector o, mo approaches 1 and mr approaches 0, which_
is not desirable for SSL. A desirable case would be a relatively small mo and a relatively large
_mr, suggesting a relatively small (large) contribution of o (r) in each Z. We interpret the cause of_
collapse as a competition between o and r where o dominates over r, i.e. mo _mr. For Eq 1, the_
derived negative gradient on Za (ignoring Zb for simplicity due to symmetry) is shown as: ≫

_cosine =_ = Zb **_Za_** = Zb, (3)
_G_ _−_ _[∂][L]∂[MSE]Za_ _−_ _⇐⇒−_ _[∂][L]∂[cosine]Za_

having zero gradient on the encoderwhere the gradient component Za is a f . dummy term because the loss −Za · Za = −1 is a constant


**Conjecture1. With Za = oz + ra, we conjecture**
that the gradient component of oz is expected to update the encoder to boost the center vector thus increase mo, while the gradient component of ra is expected to behave in the opposite direction to increase
_mr. A random gradient component is expected to_
have a relatively small influence.


oz


random

To verify the above conjecture, we revisit thegradient termZa sg(Za Zao. We design lossz) to show the influence of gradi- −Za · sg(o dummyz) and ra 0 1000 2000Iteration Step3000 4000 5000 6000
_−ent component ·_ _− o and ra respectively. The results in_ Figure 4: Influence of various gradient comFig. 4 show that the gradient component oz has the ponents on mr and mo.
effect of increasing mo while decreasing mr. On the
contrary, ra helps increase mr while decreasing mo. Overall, the results verify Conjecture1.


3.2 EXTRA GRADIENT COMPONENT FOR ALLEVIATING COLLAPSE

**Revisit collapse in a symmetric architecture. Based on Conjecture1, here, we provide an intuitive**
interpretation on why a symmetric Siamese architecture, such as Fig. 2 (a) and (b), cannot be trained
without collapse. Take Fig. 2 (a) as example, the gradient in Eq 3 can be interpreted as two equivalent
forms, from which we choose Zb _−Za = (oz +rb)−(oz +ra) = rb_ _−ra. Since rb comes from the_
same positive sample as ra, it is expected that rb also increases mr, however, this effect is expected
to be smaller than that of ra, thus causing collapse.

**Basic gradient and Extra gradient components. The negative gradient on Za in Fig. 2 (a) is**
derived as Zb, while that on Pa in Fig. 2 (b) is derived as Pb. We perceive Zb and Pb in these
basic Siamese architectures as the Basic Gradient. Our above interpretation shows that such basic
components cannot prevent collapse, for which an Extra Gradient component, denoted as Ge, needs
to be introduced to break the symmetry. As the term suggests, Ge is defined as a gradient term that
is relative to the basic gradient in a basic Siamese architecture. For example, negative samples can
be introduced to Naive Siamese (Fig. 2 (a)) for preventing collapse, where the extra gradient caused
by negative samples can thus be perceived as Ge with Zb as the basic gradient. Similarly, we can
also disentangle the negative gradient on Pa in SimSiam (Fig. 1 (a)), i.e. Zb, into a basic gradient
(which isprevents collapse via studying the independent roles of its center vector Pb) and Ge which is derived as Zb − **_Pb (note that Zb = Pb + o Ge and residual vectore). We analyze how r Ge._** _e_


-----

3.3 A TOY EXAMPLE EXPERIMENT WITH NEGATIVE SAMPLE

**Which repulsive component helps avoid collapse? Existing works often attribute the collapse in**
Naive Siamese to lacking a repulsive part during the optimization. This explanation has motivated
previous works to adopt contrastive learning, i.e. attracting the positive samples while repulsing the
whereindicates the representation of anegative samples. We experiment with a simple triplet loss Zb is the basic gradient component and thus Negative sample. The derived negative gradient on Ge = −[1]Z, Ln in this setup. For a sample represen-tri = −Za·sg(Zb − ZZan is), where Zb − **_ZZnn,_**
tation, what determines it as a positive sample for attracting or a negative sample for repulsing is the
residual component, thus it might be tempting to interpret that re is the key component of repulsive
_part that avoids the collapse. However, the results in Table 3 show that the component beneficial for_
preventing collapse inside Ge is oe instead of re. Specifically, to explore the individual influence
of oe and re in the Ge, we design two experiments by removing one component while keeping the
other one. In the first experiment, we remove the re in Ge while keeping the oe. By contrast, the
**_oe is removed while keeping the re in the second experiment. In contrast to what existing explana-_**
tions may expect, we find that the residual component oe prevents collapses. With Conjecture1, a
gradient component alleviates collapse if it has negative center vector. In this setup, oe = −oz, thus
**_oe has the de-centering role for preventing collapse. On the contrary, re does not prevent collapse_**
and keeping re even decreases the performance (36.21% < 47.41%). Since the negative sample is
randomly chosen, re just behaves like a random noise on the optimization to decrease performance.

Method _triplet_ Std _mo_ _mr_ Collapse Top-1 (%)
_L_

RemovingRemovingBaseline o ree _−−−ZZZaaa · · · sg sg sg(((ZZZbbb + + + G o reee)))_ 0.020050.0200 0.0260.0261 0.990.990 ✓×× 36.2147.411

Table 3: Gradient component analysis with a random negative sample.

3.4 DECOMPOSED GRADIENT ANALYSIS IN SIMSIAM


It is challenging to derive the gradient on the encoder output in SimSiam due to a nonlinear MLP
module in h. The negative gradient on Pa for LSimSiam in Eq 2 can be derived as

_SimSiam =_ = Zb = Pb + (Zb **_Pb) = Pb + Ge,_** (4)
_G_ _−_ _[∂][L][SimSiam]∂Pa_ _−_

where Ge indicates the aforementioned extra gradient
component. To investigate the influence of oe and re on **_oe_** **_re_** Collapse Top-1 (%)
the collapse, similar to the analysis with the toy exam- ✓ ✓ _×_ 66.62
ple experiment in Sec. 3.3, we design the experiment by ✓ _×_ _×_ 48.08
removing one component while keeping the other. The _×_ ✓ _×_ 66.15
results are reported in Table 4. As expected, the model _×_ _×_ ✓ 1
collapses when both components in Ge are removed and
the best performance is achieved when both components Table 4: Gradient component analysis
are kept. Interestingly, the model does not collapse when for SimSiam.
either oe or re is kept. To start, we analyze how oe affects the collapse based on Conjecture1.

**How oe alleviates collapse in SimSiam. Here, op is used to denote the center vector of P to**
_expected thatthus the residual gradient component is derived to bedifferentiate from the above introduced oe helps prevent collapse if o oze for denoting that of contains negative oe = o ozp − Z since the analyzed vector is. In this setupop. With Conjecture1, it is well Ge = Zb P −a. ToPb,_
determine the amount of component of op existing in oe, we measure the cosine similarity between
**_ois zero whene −_** _ηpop and ηp o is aroundp for a wide range of −0.5, suggesting ηp. The results in Fig. 5 (a) show that their cosine similarity oe has ≈−0.5op. With Conjecture1, this negative ηp_
explains why SimSiam avoids collapse from the perspective of de-centering.

**How oe causes collapse in Mirror SimSiam. As mentioned above, the collapse occurs in Mirror**
evaluate the amount of componentSimSiam, which can also be explained by analyzing its oz existing in oe via reporting the similarity between oe. Here, oe = op − **_oz, for which we oe −_** _ηzoz_

1Note that the triplet loss here does not have clipping form as in Schroff et al. (2015) for simplicity.


-----

and oz. The results in Fig. 5 (a) show that their cosine similarity is zero when ηz is set to around
0.2. This positive ηz explains why Fig. 1(c) causes collapse from the perspective of de-centering.

Overall, we find that processing the optimization target with h[−][1], as in Fig. 2 (c), alleviates collapse
(ηp 0.5), while processing it with h, as in Fig. 1(c), actually strengthens the collapse (ηz 0.2).
In other words, via the analysis of ≈− **_oe, our results help explain how SimSiam avoids collapse as well ≈_**
as how Mirror SimSiam causes collapse from a straightforward de-centering perspective.


1.00

0.75

0.50

0.25

1.00

0.75

0.50

0.25

0.00


0.8

0.6

0.4

0.2

0.0


0.5

0.0

0.5

1.0


1000 2000 3000

Iteration Step

(b)


|Col1|Col2|Col3|Col4|Col5|oe re|Col7|
|---|---|---|---|---|---|---|
||||||||
||||||||
||||||||


0.0 0.2 0.4 0.6 0.8 1.0

oe
re

Tempture

(c)


|Col1|Col2|Col3|p|Col5|
|---|---|---|---|---|
||||z||
||||||
||||||
||||||
||||||


(a)


Figure 5: (a) Investigating the amount ofop **_oz. (b) Normally train the model as SimSiam for 5 epochs, then using collapsing loss for 1 op existing in oz −_** **_op and the amount of oz existing in_**
epoch to reduce − _mr, followed by a correlation regularization loss. (c) Cosine similarity between re_
(oe) and gradient on Za induced by a correlation regularization loss.

**Relation to prior works. Motivated from preventing the collapse to a constant, multiple prior**
works, such as W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar et al., 2021), DINO (Caron
et al., 2021), explicitly adopt de-centering to prevent collapse. Despite various motivations, we find
that they all implicitly introduce an oe that contains a negative center vector. The success of their
approaches aligns well with our Conjecture1 as well as our above empirical results. Based on our
findings, we argue that the effect of de-centering can be perceived as oe having a negative center
vector. With this interpretation, we are the first to demonstrate that how SimSiam with predictor and
stop gradient avoids collapse can be explained from the perspective of de-centering.

**Beyond de-centering for avoiding collapse. In the toy example experiment in Sec. 3.3, re is**
found to be not beneficial for preventing collapse and keeping re even decreases the performance.
Interestingly, as shown in Table 4, we find that re alone is sufficient for preventing collapse and
achieves comparable performance as Ge. This can be explained from the perspective of dimensional
de-correlation, which will be discussed in Sec. 3.5.

3.5 DIMENSIONAL DE-CORRELATION HELPS PREVENT COLLAPSE


**Conjecture2 and motivation. We conjecture that dimensional de-correlation increases mr for pre-**
venting collapse. The motivation is straightforward as follows. The dimensional correlation would
be minimum if only a single dimension has a very high value for every individual class and the
dimension changes for different classes. In another extreme case, when all the dimensions have the
same values, equivalent to having a single dimension, which already collapses by itself in the sense
of losing representation capacity. Conceptually, re has no direct influence on the center vector, thus
we interpret that re prevents collapse through increasing mr.

To verify the above conjecture, we train SimSiam normally with the loss in Eq 2 and train for several
epochs with the loss in Eq 1 for intentionally decreasing the mr to close to zero. Then, we train the
loss with only a correlation regularization term, which is detailed in Appendix A.6. The results in
Fig. 5 (b) show that this regularization term increases mr at a very fast rate.

**Dimensional de-correlation in SimSiam. Assuming h only has a single FC layer to exclude the**
influence of oe, the weights in FC are expected to learn the correlation between different dimensions
for the encoder output. This interpretation echos well with the finding that the eigenspace of h weight
aligns well with that of correlation matrix (Tian et al., 2021). In essence, the h is trained to minimize
the cosine similarity between h(za) and I(zb), where I is identity mapping. Thus, h that learns the
correlation is optimized close to I, which is conceptually equivalent to optimizing with the goal of
de-correlation for Z. As shown in Table 4, for SimSiam, re alone also prevents collapse, which


-----

1.0

0.8

0.6

0.4

0.2

0.0


1.0

0.8

0.6

0.4

0.2

0.0


0.004

0.003

0.002

0.001

0.000


40

20


50 100 150 200

SimSiam
InfoNCE
re removal

Epoch


50 100 150 200

Epoch


50 100 150 200

Epoch


50 100 150 200

Epoch


Figure 6: Influence of various gradient components on mr and mo.

is attributed to the de-correlation effect since re has no de-centering effect. We observe from Fig.
6 that except in the first few epochs, SimSiam decreases the covariance during the whole training.
Fig. 6 also reports the results for InfoNCE which will be discussed in Sec. 4.


4 TOWARDS A UNIFIED UNDERSTANDING OF RECENT PROGRESS IN SSL

**De-centering and de-correlation in InfoNCE. InfoNCE loss is a default choice in multiple seminal**
contrastive learning frameworks (Sohn, 2016; Wu et al., 2018; Oord et al., 2018; Wang & Liu, 2021).
The derived negative gradient of InfoNCE on Za is proportional to Zb + _i=0_
exp(Za **_Zi/τ_** ) _[−][λ][i][Z][i][, where]_
_λi =_ _N_ _·_
_i=0_ [exp(][Z][a][·][Z][i][/τ] [)] [, and][ Z][0][ =][ Z][b][ for notation simplicity. See Appendix A.7 for the detailed]

[P][N]

derivation. The extra gradient componentP **_Ge =_** _i=0_ _[−][λ][i][Z][i][ =][ −][o][z][ −]_ [P]i[N]=0 _[λ][i][r][i][, for which]_
**_oe = −oz and re = −_** [P]i[N]=0 _[λ][i][r][i][. Clearly,][ o][e][ contains negative][ o][z][ as de-centering for avoiding]_
collapse, which is equivalent to the toy example in Sec. 3.3 when the[P][N] **_re is removed. Regarding_**
**_re, the main difference between Ltri in the toy example and InfoNCE is that the latter exploits a_**
batch of negative samples instead of a random one. λi is proportional to exp(Za·Zi), indicating
that a large weight is put on the negative sample when it is more similar to the anchor Za, for which,
intuitively, its dimensional values tend to have a high correlation with Za. Thus, re containing
such negative representation with a high weight tends to decrease dimensional correlation. To verify
this intuition, we measure the cosine similarity between re and the gradient on Za induced by a
correlation regularization loss. The results in Fig. 5 (c) show that their gradient similarity is high for
a wide range of temperature values, especially when τ is around 0.1 or 0.2, suggesting re achieves
similar role as an explicit regularization loss for performing de-correlation. Replacing re with oe
leads to a low cosine similarity, which is expected because oe has no de-correlation effect.

The results of InfoNCE in Fig. 6 resembles that of SimSiam in terms of the overall trend. For example, InfoNCE also decreases the covariance value during training. Moreover, we also report the
results of InfoNCE where re is removed for excluding the de-correlation effect. Removing re from
the InfoNCE loss leads to a high covariance value during the whole training. Removing re also
leads to a significant performance drop, which echos with the finding in (Bardes et al., 2021) that
dimensional de-correlation is essential for competitive performance. Regarding how re in InfoNCE
achieves de-correlation, formally, we hypothesize that the de-correlation effect in InfoNCE arises
_from the biased weights (λi) on negative samples. This hypothesis is corroborated by the tempera-_
ture analysis in Fig. 7. We find that a higher temperature makes the weight distribution of λi more
balanced indicated a higher entropy of λi, which echos with the finding in (Wang & Liu, 2021).
Moreover, we observe that a higher temperature also tends to increase the covariance value. Overall, with temperature as the control variable, we find that more balanced weights among negative
samples decrease the de-correlation effect, which constitutes an evidence for our hypothesis.

**Unifying SimSiam and InfoNCE. At first sight, there is no conceptual similarity between SimSiam**
and InfoNCE, and this is why the community is intrigued by the success of SimSiam without negative samples. Through decomposing the Ge into oe and re, we find that for both, their oe plays the
role of de-centering and their re behaves like de-correlation. In this sense, we bring two seemingly
irrelevant frameworks into a unified perspective with disentangled de-centering and de-correlation.

**Beyond SimSiam and InfoNCE. In SSL, there is a trend of performing explicit manipulation of**
de-centering and de-correlation, for which W-MSE (Ermolov et al., 2021), Barlow-twins (Zbontar
et al., 2021), DINO (Caron et al., 2021) are three representative works. They often achieve performance comparable to those with InfoNCE or SimSiam. Towards a unified understanding of recent
progress in SSL, our work is most similar to a concurrent work (Bardes et al., 2021). Their work
is mainly inspired by Barlow-twins (Zbontar et al., 2021) but decomposes its loss into three explicit
components. By contrast, our work is motivated to answer the question of how SimSiam prevents


-----

50

40

30

20

10


1.5

1.0

0.5

0.0


0.2 0.4 0.6 0.8 1.0

Temperature

(a)


|Col1|Col2|
|---|---|
|||
||= 0.1|
||= 0.4|
||= 0.8|
||= 1.0|


50 100 150 200

= 0.1
= 0.4
= 0.8
= 1.0

Epoch

(b)


50 100 150 200

Epoch

(c)


Figure 7: Influence of temperature. (a) Entropy of λi with regard to temperature; (b) Top-1 accuracy
trend with various temperature; (c) Covariance trend with various temperature.

collapse without negative samples. Their work claims that variance component (equivalent to decentering) is an indispensable component for preventing collapse, while we find that de-correlation
itself alleviates collapse. Overall, our work helps understand various frameworks in SSL from an
unified perspective, which also inspires an investigation of inter-anchor hardness-awareness Zhang
et al. (2022) for further bridging the gap between CL and non-CL frameworks in SSL.


TOWARDS SIMPLIFYING THE PREDICTOR IN SIMSIAM


Based on our understanding of how SimSiam prevents collapse, we demonstrate that simple components (instead of a non-linear MLP in SimSiam) in the predictor are sufficient for preventing
collapse. For example, to achieve dimensional de-correlation, a single FC layer might be sufficient
because a single FC layer can realize the interaction among various dimensions. On the other hand,
to achieve de-centering, a single bias layer might be sufficient because a bias vector can represent the
center vector. Attaching an l2-normalization layer at the end of the encoder, i.e. before the predictor,
is found to be critical for achieving the above goal.

**Pridictor with FC layers. To learn the dimensional**
correlation, an FC layer is sufficient theoretically but
can be difficult to train in practice. Inspired by the
property that Multiple FC layers make the training more One FC Tanh(FC) 64.82
stable even though they can be mathematically equiva- One bias Bias 49.82
lent to a single FC layer (Bell-Kligler et al., 2019), we

|Method|Predictor Top-1 (%)|
|---|---|
|SimSiam Two FC One FC One bias|Non-linear MLP 66.9 FC+FC+Bias 66.7 Tanh(FC) 64.82 Bias 49.82|

adopt two consecutive FC layers which are equivalent Table 5: Linear evaluation on CIFAR100.
to removing the BN and ReLU in the original predictor.
The training can be made more stable if a Tanh layer is applied on the adopted single FC after every
iteration. Table 5 shows that they achieve performance comparable to that with a non-linear MLP.

**Predictor with a bias layer. A predictor with a single**
bias layer can be utilized for preventing collapse (see Bias (1) single bias (2) bias in MLP
Table 5) and the trained bias vector is found to have

|Bias|(1) single bias (2) bias in MLP|
|---|---|
|Similarity|0.99 0.89|

a cosine similarity of 0.99 with the center vector (see

Table 6: Similarity between center vector

Table 6). A bias in the MLP predictor also has a high
cosine similarity of 0.89, suggesting that it is not a co- and (1) single bias layer (bp), (2) the last

_bias layer of MLP in the predictor._

incidence. A theoretical derivation for justifying such a
high similarity as well as how this single bias layer prevents collapse are discussed in Appendix A.8.

6 CONCLUSION


We point out a hidden flaw in prior works for explaining the success of SimSiam and propose to
decompose the representation vector and analyze the decomposed components of extra gradient. We
find that its center vector gradient helps prevent collapse via the de-centering effect and its residual
gradient achieves de-correlation which also alleviates collapse. Our further analysis reveals that
InfoNCE achieve the two effects in a similar manner, which bridges the gap between SimSiam and
InfoNCE and contributes to a unified understanding of recent progress in SSL. Towards simplifying
the predictor we have also found that a single bias layer is sufficient for preventing collapse.


-----

ACKNOWLEDGEMENT

This work was partly supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) under grant No.2019-001396 (Development of framework for analyzing, detecting, mitigating of bias in AI model and
training data), No.2021-0-01381 (Development of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and No.2021-0-02068 (Artificial
Intelligence Innovation Hub). During the rebuttal, multiple anonymous reviewers provide valuable
advice to significantly improve the quality of this work. Thank you all.

REFERENCES

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing
mutual information across views. arXiv preprint arXiv:1906.00910, 2019.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization
for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.

Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using
an internal-gan. NeurIPS, 2019.

Jane Bromley, James W Bentz, L´eon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard
S¨ackinger, and Roopak Shah. Signature verification using a “siamese” time delay neural network.
_International Journal of Pattern Recognition and Artificial Intelligence, 1993._

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. _arXiv preprint_
_arXiv:2006.09882, 2020._

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. _arXiv preprint_
_arXiv:2104.14294, 2021._

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In ICML, 2020a.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision
transformers. ICCV, 2021.

Victor G. Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. Solo-learn: A library
of self-supervised methods for visual representation learning, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
_the North American Chapter of the Association for Computational Linguistics: Human Language_
_Technologies, Volume 1 (Long and Short Papers), 2019._

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand
Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Crosscovariance image transformers. arXiv preprint arXiv:2106.09681, 2021.

Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for selfsupervised representation learning. In ICML. PMLR, 2021.

Abe Fetterman and Josh Albrecht. Understanding self-supervised and contrastive learning with
”bootstrap your own latent” (byol), 2020.
[https://untitled-ai.github.io/](https://untitled-ai.github.io/understanding-self-supervised-contrastive-learning.html)
[understanding-self-supervised-](https://untitled-ai.github.io/understanding-self-supervised-contrastive-learning.html)
[contrastive-learning.html.](https://untitled-ai.github.io/understanding-self-supervised-contrastive-learning.html)


-----

Jean-Bastien Grill, Florian Strub, Florent Altch´e, Corentin Tallec, Pierre Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar,
et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural
_Information Processing Systems, 2020._

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In CVPR, 2020.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron
Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. _arXiv preprint_
_arXiv:2004.11362, 2020._

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020.

Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and
Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv
_preprint arXiv:2106.09785, 2021._

Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. Dc-bert: Decoupling question and document for efficient contextual encoding. In Proceedings of the 43rd
_International ACM SIGIR Conference on Research and Development in Information Retrieval,_
2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners. OpenAI blog, 2019.

Pierre H Richemond, Jean-Bastien Grill, Florent Altch´e, Corentin Tallec, Florian Strub, Andrew
Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without
batch statistics. arXiv preprint arXiv:2010.10241, 2020.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), 2015._

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS,
2016.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. {VL}-{bert}:
Pre-training of generic visual-linguistic representations. In ICLR, 2020.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint
_arXiv:1906.05849, 2019._

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics
without contrastive pairs. arXiv preprint arXiv:2102.06810, 2021.

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance discrimination. In CVPR, 2018.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised
learning via redundancy reduction. ICML, 2021.

Chaoning Zhang, Kang Zhang, Trung X. Pham, Changdong Yoo, and In-So Kweon. Towards understanding and simplifying moco: Dual temperature helps contrastive learning without many
negative samples. In CVPR, 2022.


-----

A APPENDIX

A.1 EXPERIMENTAL SETTINGS

**Self-supervised encoder training: Below are the settings for self-supervised encoder training. For**
simplicity, we mainly use the default settings in a popular open library termed solo-learn (da Costa
et al., 2021).

_Data augmentation and normalization:_ We use a series of transformations including Ran_domResizedCrop with scale [0.2, 1.0], bicubic interpolation. ColorJitter (brightness (0.4), contrast_
(0.4), saturation (0.4), hue (0.1)) is randomly applied with the probability of 0.8. Random gray
scale RandomGrayscale is applied with p = 0.2 Horizontal flip is applied with p = 0.5 The images
are normalized with the mean (0.4914, 0.4822, 0.4465) and Std (0.247, 0.243, 0.261).

_Network architecture and initialization: The backbone architecture is ResNet-18. The projection_
head contains three fully-connected (FC) layers followed by Batch Norm (BN) and ReLU, for which
ReLU in the final FC layer is removed, i.e. FC1 +BN +ReLU +FC2 +BN +ReLU +FC3 +BN .
All projection FC layers have 2048 neurons for input, output as well as the hidden dimensions. The
predictor head includes two FC layers as follows: FC1 + BN + ReLU + FC2. Input and output
of the predictor both have the dimension of 2048, while the hidden dimension is 512. All layers of
the network are by default initialized in Pytorch.

_Optimizer: SGD optimizer is used for the encoder training._ The batch size M is 256 and
the learning rate is linearly scaled by the formula lr × M/256 with the base learning rate lr set
to 0.5. The schedule for learning rate adopts the cosine decay as SimSiam. Momentum 0.9 and
weight decay 1.0 × 10[−][5] are used for SGD. We use one GPU for each pre-training experiment.
Following the practice of SimSiam, the learning rate of the predictor is fixed during the training.
We use warmup training for the first 10 epochs. If not specified, by default we train the model for
1000 epochs.

**Online linear evaluation:** For the online linear revaluation, we also follow the practice in
the solo-learn library (da Costa et al., 2021). The frozen features (2048 dimensions) from the
training set are extracted (from the self-supervised pre-trained model) to feed into a linear classifier
(1 FC layer with the input 2048 and output of 100). The test is performed on the validation set. The
learning rate for the linear classifier is 0.1. Overall, we report Top-1 accuracy with the online linear
evaluation in this work.

A.2 TWO SUB-PROBLEMS IN AO OF SIMSIAM

In the sub-problem η[t] arg minη (θ[t], η), η[t] indicating latent representation of images at step t
_←_ _L_

is actually obtained through ηx[t] _[←]_ [E][T] _Fθt_ (T (x)), where they in practice ignore ET [·] and sample

only one augmentationpredictor to EOA. _T_ _[′], i.e. ηx[t]_ _[←F]h_ _[θ][t]_ [(][T][ ′][(][x][))]i[. Conceptually, Chen & He equate the role of]

A.3 EXPERIMENTAL DETAILS FOR EXPLICIT EOA IN TABLE 1


In the Moving average experiment, we follow the setting in SimSiam (Chen & He, 2021) without
predictor. In the Same batch experiment, multiple augmentations, 10 augmentations for instance,
are applied on the same image. With multi augmentations, we get the corresponding encoded representation, i.e. zi, i [1, 10]. We minimize the cosine distance between the first representation z1
and the average of the remaining vectors, ∈ _i.e. ¯z =_ [1]9 10i=2 _[z][i][. The gradient stop is put on the aver-]_

aged vector. We also experimented with letting the gradient backward through more augmentations,
however, they consistently led to collapse. P


-----

A.4 EXPERIMENTAL SETUP AND RESULT TREND FOR TABLE 2.

**Mirror SimSiam. Here we provide the pseudocode for Mirror SimSiam. In the Mirror SimSiam ex-**
periment which relates to Fig. 1 (c). Without taking symmetric loss into account, the pseudocode is
shown in Algorithm 1. Taking symmetric loss into account, the pseudocode is shown in Algorithm 2.

**Algorithm 1 Pytorch-like Pseudocode: Mirror SimSiam**

# f: encoder (backbone + projector)
# h: predictor

for x in loader: # load a minibatch x with n samples

x_a, x_b = aug(x), aug(x) # augmentation
z_a, z_b = f(x_a), f(x_b) # projections

p_b = h(z_b.detach()) # detach z_b but still allowing gradient p_b

L = D_cosine(z_a, p_b) # loss

L.backward() # back-propagate
update(f, h) # SGD update

def D_cosine(z, p): # negative cosine similarity

z = normalize(z, dim=1) # l2-normalize
p = normalize(p, dim=1) # l2-normalize
return -(z*p).sum(dim=1).mean()

**Algorithm 2 Pytorch-like Pseudocode: Mirror SimSiam**

# f: encoder (backbone + projector)
# h: predictor

for x in loader: # load a minibatch x with n samples

x_a, x_b = aug(x), aug(x) # augmentation
z_a, z_b = f(x_a), f(x_b) # projections

p_b = h(z_b.detach()) # detach z_b but still allowing gradient p_b
p_a = h(z_a.detach()) # detach z_a but still allowing gradient p_a

L = D_cosine(z_a, p_b)/2 + D_cosine(z_b, p_a)/2 # loss

L.backward() # back-propagate
update(f, h) # SGD update

def D_cosine(z, p): # negative cosine similarity

z = normalize(z, dim=1) # l2-normalize
p = normalize(p, dim=1) # l2-normalize
return -(z*p).sum(dim=1).mean()

**Symmetric Predictor. To implement the SimSiam with Symmetric Predictor as in Fig. 2 (b), we**
can just perceive the predictor as part of the new encoder, for which the pseudocode is provided in
Algorithm 3. Alternatively, we can additionally train the predictor similarly as that in SimSiam, for
which the training involves two losses, one for training the predictor and another for training the
new encoder (the corresponding pseudocode is provided in Algorithm 4). Moreover, for the second
implementation, we also experiment with another variant that fixes the predictor while optimizing
the new encoder and then train the predictor alternatingly. All of them lead to collapse with a similar
trend as long as the symmetric predictor is used for training the encoder. For avoiding redundancy,
in Fig. 8 we only report the result of the second implementation.

**Result trend. The result trend of SimSiam, Naive Siamese, Mirror SimSiam, Symmetric Predictor**
are shown in Fig. 8. We observe that all architectures lead to collapse except for SimSiam. Mirroe
SimSiam was stopped in the middle because a NaN value was returned from the loss.

A.5 EXPERIMENTAL DETAILS FOR INVERSE PREDICTOR.

In the inverse predictor experiment which relates to Fig. 2 (c), we introduce a new predictor which
has the same structure as that of the original predictor. The training loss consists of 3 parts: predictor
training loss, inverse predictor training and new encoder (old encoder+predictor) training. The new


-----

**Algorithm 3 Pytorch-like Pseudocode: Symmetric Predictor**

# f: encoder (backbone + projector)
# h: predictor


for x in loader: # load a minibatch x with n samples

x_a, x_b = aug(x), aug(x) # augmentation
z_a, z_b = f(x_a), f(x_b) # projections
p_a, p_b = h(z_a), h(z_b) # predictions

L = D(p_a, p_b)/2 + D(p_b, p_a)/2 # loss


L.backward() # back-propagate
update(f, h) # SGD update

def D(p, z): # negative cosine similarity

z = z.detach() # stop gradient
p = normalize(p, dim=1) # l2-normalize
z = normalize(z, dim=1) # l2-normalize
return -(p*z).sum(dim=1).mean()


**Algorithm 4 Pytorch-like Pseudocode: Symmetric Predictor (with additional training on predictor)**

# f: encoder (backbone + projector)
# h: predictor


for x in loader: # load a minibatch x with n samples

x_a, x_b = aug(x), aug(x) # augmentation
z_a, z_b = f(x_a), f(x_b) # projections
p_a, p_b = h(z_a), h(z_b) # predictions

d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach()) # detached predictor output


# predictor training loss
L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2

# encoder training loss
L_enc = D(p_a, d_p_b)/2 + D(p_b, d_p_a)/2


L = L_pred + L_enc

L.backward() # back-propagate
update(f, h) # SGD update


def D(p, z): # negative cosine similarity with detach on z

z = z.detach() # stop gradient
p = normalize(p, dim=1) # l2-normalize
z = normalize(z, dim=1) # l2-normalize
return -(p*z).sum(dim=1).mean()

60

50

20

10


200 400 600 800 1000

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
||SimSiam|||
|||||
||Symmetric Predicto|r||
||Mirror SimSiam Naive Siamese|||
|||||
|||||
|||||


Epoch


Figure 8: Result trend of Naive Siamese, Mirror SimSiam, Symmetric Predictor.

encoder F consists of the old encoder f + predictor h. The practice of gradient stop needs to be
considered in the implementation. We provide the pseudocode in Algorithm 5.


-----

**Algorithm 5 Pytorch-like Pseudocode: Trainable Inverse Predictor**

# f: encoder (backbone + projector)
# h: predictor
# h_inv: inverse predictor

for x in loader: # load a minibatch x with n samples

x_a, x_b = aug(x), aug(x) # augmentation
z_a, z_b = f(x_a), f(x_b) # projections
p_a, p_b = h(z_a), h(z_b) # predictions

d_p_a, d_p_b = h(z_a.detach()), h(z_b.detach()) # detached predictor output
# predictor training loss
L_pred = D(d_p_a, z_b)/2 + D(d_p_b, z_a)/2 # to train h

inv_p_a, inv_p_b = h_inv(p_a.detach()), h_inv(p_b.detach()) # to train h_inv
# inverse predictor training loss
L_inv_pred = D(inv_p_a, z_a)/2 + D(inv_p_b, z_b)/2

# encoder training loss
L_enc = D(p_a, h_inv(p_b))/2 + D(p_b, h_inv(p_a))

L = L_pred + L_inv_pred + L_enc

L.backward() # back-propagate
update(f, h, h_inv) # SGD update

def D(p, z): # negative cosine similarity with detach on z

z = z.detach() # stop gradient
p = normalize(p, dim=1) # l2-normalize
z = normalize(z, dim=1) # l2-normalize
return -(p*z).sum(dim=1).mean()

A.6 REGULARIZATION LOSS

Following Zbontar et al. (2021), we compute covariance regularization loss of encoder output along
the mini-batch. The pseudocode for de-correlation loss calculation is put in Algorithm 6.

**Algorithm 6 Pytorch-like Pseudocode: De-correlation loss**

# Z_a: representation vector
# N: batch size
# D: the number of dimension for representation vector

Z_a = Z_a - Z_a.mean(dim=0)

cov = Z_a.T @ Z_a / (N-1)
diag = torch.eye(D)

loss = cov[˜diag.bool()].pow_(2).sum() / D

A.7 GRADIENT DERIVATION AND TEMPERATURE ANALYSIS FOR INFONCE

With · indicating the cosine similarity between vectors, the InfoNCE loss can be expressed as

exp(Za **_Zb/τ_** )
_InfoNCE =_ log _·_
_L_ _−_ exp(Za **_Zb/τ_** ) + _i=1_ [exp(][Z][a][ ·][ Z][i][/τ] [)]

_·_ (5)

exp(Za **_Zb/τ_** )
= − log _N_ _·_ [P][N],
_i=0_ [exp(][Z][a][ ·][ Z][i][/τ] [)]
P

where N indicates the number of negative samples and Z0 = Zb for simplifying the notation. By
treating Za **_Zi as the logit in a normal CE loss, we have the corresponding probability for each_**
_·_ exp(Za **_Zi/τ_** )
negative sample as λi = _Ni=0_ [exp(]·[Z][a][·][Z][i][/τ] [)] [, where][ i][ = 0][,][ 1][,][ 2][, ..., N][ and we have][ P]i[N]=0 _[λ][i][ = 1][.]_
P


-----

The negative gradient of the InfoNCE on the representation Za is shown as


= [1]

_−_ _[∂][L][InfoNCE]∂Za_ _τ_ [(1][ −] _[λ][0][)][Z][b][ −]_ _τ[1]_


_λiZi_
_i=1_

X


= [1]

_τ_ [(][Z][b][ −]

= [1]

_τ_ [(][Z][b][ −]


_λiZi)_
_i=0_

X

_N_

_λi(oz + ri))_
_i=0_

X


(6)


= [1]

_τ_ [(][Z][b][ + (][−][o][z][ −]


_λiri)_
_i=0_

X


_∝_ **_Zb + (−oz −_**


_λiri)_
_i=0_

X


where _τ[1]_ [can be adjusted through learning rate and is omitted for simple discussion. With][ Z][b][ as the]

basic gradient, Ge = −oz − [P]i[N]=0 _[λ][i][r][i][, for which][ o][e][ =][ −][o][z][ and][ r][e][ =][ −]_ [P]i[N]=0 _[λ][i][r][i][.]_

When the temperature is set to a large value, λi = _Niexp(=0_ [exp(]Za·[Z]Z[a]i[·]/τ[Z][i])[/τ] [)] [, approaches] _N1+1_ [, indicated]

by a high entropy value (see Fig. 7). InfoNCE will degenerate to a simple contrastive loss,P _i.e._
_Lsimple = −Za · Zb +_ _N1+1_ _Ni=0_ **_[Z][a][ ·][ Z][i][, which repulses every negative sample with an equal]_**

force. In contrast, a relative smaller temperature will give more relative weight, i.e. larger λ, to
P
negative samples that are more similar to the anchor (Za).

The influence of the temperature on the covariance and accuracy is shown in Fig. 7 (b) and (c).
We observe that a higher temperature tends to decrease the effect of de-correlation, indicated by
a higher covariance value, which also leads to a performance drop. This verifies our hypothesis
regarding on how re in InfoNCE achieves de-correlation because a large temperature causes more
balanced weights λi, which is found to alleviate the effect of de-correlation. For the setup, we note
that the encoder is trained for 200 epochs with the default setting in Solo-learn for the SimCLR
framework.

A.8 THEORETICAL DERIVATION FOR A SINGLE BIAS LAYER

With the cosine similarity loss defined as Eq 7 Eq 8:

_a_ _b_
_cossim(a, b) =_ _·_ (7)
_√a[2]_ _b[2][,]_

_·_

for which the derived gradient on the vector a is shown as


_∂_ _b1_ (8)

_∂a_ _[cossim][(][a, b][) =]_ _|a| · |b| [−]_ _[cossim][(][a, b][)][ ·][ a]|a|[1][2][ .]_


The above equation is used as a prior for our following derivations. As indicated in the main
manuscript, the encoder output za is l2-normalized before feeding into the predictor, thus pa =
**_Za + bp, bp denotes the bias layer in the predictor. The cosine similarity loss (ignoring the symme-_**
try for simplicity) is shown as

_Lcosine = −Pa · Zb_

**_zb_** (9)

=
_−_ **_[p]p[a]a_** **_zb_**

_||_ _|| [·]_ _∥_ _∥_


-----

The gradient on pa is derived as

**_zb_** **_pa_**
=

_−_ _[∂][L]∂[cosine]pa_ **_zb_** **_pa_** **_pa_**

_∥_ _∥· ∥_ _∥_ _[−]_ _[cossim][(][Z][a][,][ Z][b][)][ ·]_ _||_ _||[2]_

1 **_zb_**
=

**_pa_** **_zb_**
_∥_ _∥_  _∥_ _∥_ _[−]_ _[cossim][(][Z][a][,][ Z][b][)][ ·][ P][a]_

1
= **_Zb_** _cossim(Za, Zb)_ **_[Z][a][ +][ b][p]_**

**_pa_** _−_ _·_ **_pa_**
_∥_ _∥_  _∥_ _∥_ 

1
= (oz + rb) (oz + ra + bp)

**_pa_** _−_ _[cossim]p[(]a[Z][a][,][ Z][b][)]_ _·_
_∥_ _∥_  _∥_ _∥_


(10)


=

**_pa_**
_∥_ _∥_ [((][o][z][ +][ r][b][)][ −] _[m][ ·][ (][o][z][ +][ r][a][ +][ b][p][))]_

1
=

**_pa_**
_∥_ _∥_ [((1][ −] _[m][)][o][z][ −]_ _[m][b][p][ +][ r][b][ −]_ _[m][ ·][ r][a][)][,]_

where m = _[cossim]∥p[(]a[Z]∥[a][,][Z][b][)]_ .

Given that pa = Za + bp, the negative gradient on bp is the same as that on pa as


=

_−_ _[∂][L]∂[cosine]bp_ _−_ _[∂][L]∂[cosine]pa_ (11)

1
=

**_pa_**
_∥_ _∥_ [((1][ −] _[m][)][o][z][ −]_ _[m][b][p][ +][ r][b][ −]_ _[m][ ·][ r][a][)][ .]_


We assume that the training is stable and the bias layer converges to a certain value when

_∂bp_ = 0. Thus, the converged bp satisfies the following constraint:

_−_ _[∂cossim][(][Z][a][,][Z][b][)]_

1

_∥pa∥_ [((1][ −] _[m][)][o][z][ −]_ _[m][b][p][ +][ r][b][ −]_ _[m][r][a][)) = 0]_ (12)

**_bp = [1][ −]_** _[m]_ **_oz + [1]_**

_m_ _m_ **_[r][b][ −]_** **_[r][a][.]_**


With a batch of samples, the average of _m[1]_ **_[r][b][ and][ r][a][ is expected to be close to 0 by the definition of]_**

residual vector. Thus, the bias layer vector is expected to converge to:


**_bp = [1][ −]_** _[m]_


**_oz._** (13)


**Rational behind the high similarity between bp and oz. The above theoretical derivation shows**
that the parameters in the bias layer are excepted to converge to a vector [1][−]m[m] **_[o][z][. This theoretical]_**

derivation justifies why the empirically observed cosine similarity between bp and oz is as high as
0.99. Ideally, it should be 1, however, such a small deviation is expected with the training dynamics
taken into account.

**Rational behind how a single bias layer prevents collapse. Given that pa = Za +** **_bp, the negative_**
gradient on Za is shown as

=

_−_ _[∂][L]∂[cosine]Za_ _−_ _[∂][L]∂[cosine]pa_


1

**_Zb_** _cossim(Za, Zb)_ **_[Z][a][ +][ b][p]_**

**_pa_** _−_ _·_ **_pa_**
_∥_ _∥_  _∥_ _∥_ 

1

**_Za_** **_bp._**

**_pa_** **_pa_** _−_ _[cossim]p[(]a[Z][a][,][ Z][b][)]_
_∥_ _∥_ **_[Z][b][ −]_** _[cossim]∥_ [(][Z]∥[2][a][,][ Z][b][)] _∥_ _∥[2]_


(14)


Here, we highlight that since the loss −Za · Za = −1 is a constant having zero gradients on the encoder, **_pa_** **_Za can be seen as a dummy term. Considering Eq 13 and m =_** _[cossim]p[(]a[Z][a][,][Z][b][)]_,
_−_ _[cossim]∥_ [(][Z]∥[2][a][,][Z][b][)] _∥_ _∥_


-----

we have b = ( _cossim∥p(aZ∥a,Zb)_

_[−]_ [1)][o][z][. The above equation is equivalent to]

1
= **_bp_**

_−_ _[∂][L]∂[cosine]Za_ **_pa_** **_pa_**

_∥_ _∥_ **_[Z][b][ −]_** _[cossim]∥_ [(][Z]∥[2][a][,][ Z][b][)]

1 **_pa_**
= ( _∥_ _∥_

**_pa_** **_pa_** _cossim(Za, Zb)_
_∥_ _∥_ **_[Z][b][ −]_** _[cossim]∥_ [(][Z]∥[2][a][,][ Z][b][)] _[−]_ [1)][o][z]

1 1
= )oz

**_pa_** **_pa_** **_pa_**
_∥_ _∥_ **_[Z][b][ −]_** _∥_ _∥_ [(1][ −] _[cossim]∥_ [(][Z]∥[a][,][ Z][b][)]

**_Zb_** (1 )oz.
_∝_ _−_ _−_ _[cossim]p[(]a[Z][a][,][ Z][b][)]_

_∥_ _∥_


(15)


With Zb as the basic gradient, the extra gradient component Ge = (1 **_pa_** )oz. Given
_−_ _−_ _[cossim]∥_ [(][Z]∥[a][,][Z][b][)]

that pa = Za + bp and ∥Za∥ = 1, thus ∥pa∥ _< 1 only when Za is negatively correlated with bp. In_
practice, however, Za and bp are often positively correlated to some extent due to their shared center
vector component. In other words, **_pa_** _> 1. Moreover, cossim(Za, Zb) is smaller than 1, thus_
_∥_ _∥_
(1 **_pa_** ) < 0, suggesting Ge consists of negative oz with the effect of de-centerization.
_−_ _−_ _[cossim]∥_ [(][Z]∥[a][,][Z][b][)]

This above derivation justifies the rationale why a single bias layer can help alleviate collapse.

B DISCUSSION: DOES BN HELP AVOID COLLAPSE?


1.0

0.8

0.6

0.4

0.2

0.0


1.0

0.8

0.6

0.4

0.2

0.0


0.00100

0.00075

0.00050

0.00025

0.00000


40

20


50 100 150 200

MSE
SimSiam

Epoch


50 100 150 200

Epoch


50 100 150 200

Epoch


50 100 150 200

Epoch


Figure 9: BN with MSE helps prevent collapse without predictor or stop gradient. Its performance,
however, is inferior to the cosine loss-based SimSiam (with predictor and stop gradient).

To our knowledge, our work is the first to revisit and refute the explanatory claims in (Chen & He,
2021). Several works, however, have attempted to demystify the success of BYOL (Grill et al.,
2020), a close variant of SimSiam. The success has been ascribed to BN in (Fetterman & Albrecht,
2020), however, (Richemond et al., 2020) refutes their claim. Since the role of intermediate BNs is
ascribed to stabilize training (Richemond et al., 2020; Chen & He, 2021), we only discuss the final
BN in the SimSiam encoder. Note that with our Conjecture1, the final BN that removes the mean
of representation vector is supposed to have de-centering effect. BY default SimSiam has such a
BN at the end of its encoder, however, it still collapses with the predictor and stop gradient. Why
would such a BN not prevent collapse in this case? Interestingly, we observe that such BN can help
alleviate collapse with a simple MSE loss (see Fig. 9), however, its performance is is inferior to the
cosine loss-based SimSiam (with predictor and stop gradient) due to the lack of the de-correlation
effect in SimSiam. Note that the cosine loss is in essence equivalent to a MSE loss on the l2normalized vectors. This phenomenon can be interpreted as that the l2-normalization causes another
mean after the BN removes it. Thus, with such l2-normalization in the MSE loss, i.e. adopting the
default cosine loss, it is important to remove the oe from the optimization target. The results with
the loss ofabove interpretation. −Za · sg(Zb + oe) in Table 3 show that this indeed prevents collapse and verifies the


-----

