# UNDERSTANDING DIMENSIONAL COLLAPSE IN CON## TRASTIVE SELF-SUPERVISED LEARNING

**Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian**
Facebook AI Research
_{ljng, pascal, yann, yuandong}@fb.com_

ABSTRACT

Self-supervised visual representation learning aims to learn useful representations
without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the
same image. Various methods have been proposed to solve the collapsing problem
where all embedding vectors collapse to a trivial constant solution. Among these
methods, contrastive learning prevents collapse via negative sample pairs. It has
been shown that non-contrastive methods suffer from a lesser collapse problem of
a different nature: dimensional collapse, whereby the embedding vectors end up
spanning a lower-dimensional subspace instead of the entire available embedding
space. Here, we show that dimensional collapse also happens in contrastive learning. In this paper, we shed light on the dynamics at play in contrastive learning
that leads to dimensional collapse. Inspired by our theory, we propose a novel
contrastive learning method, called DirectCLR, which directly optimizes the representation space without relying on a trainable projector. Experiments show that
_DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet._

1 INTRODUCTION

Self-supervised learning aims to learn useful representations of the input data without relying on human annotations. Recent advances in self-supervised visual representation learning based on joint
embedding methods (Misra & Maaten, 2020b; He et al., 2020; Chen et al., 2020a; Chen & He, 2020;
Grill et al., 2020; Zbontar et al., 2021; Bardes et al., 2021; Chen et al., 2020b; Dwibedi et al., 2021;
Li et al., 2021; Misra & Maaten, 2020a; HaoChen et al., 2021; Assran et al., 2021; Caron et al., 2021)
show that self-supervised representations have competitive performances compared with supervised
ones. These methods generally aim to learn representations invariant to data augmentations by maximizing the agreement between embedding vectors from different distortions of the same images.

As there are trivial solutions where the model maps all input to the same constant vector, known
as the collapsing problem, various methods have been proposed to solve this problem that rely on
different mechanisms. Contrastive methods like Chen et al. (2020a) and He et al. (2016) define ‘positive’ and ‘negative’ sample pairs which are treated differently in the loss function. Non-contrastive
methbods like Grill et al. (2020) and Chen & He (2020) use stop-gradient, and an extra predictor to
prevent collapse without negative pairs; Caron et al. (2018; 2020) use an additional clustering step;
and Zbontar et al. (2021) minimize the redundant information between two branches.

These self-supervised learning methods are successful in preventing complete collapse whereby all
representation vectors shrink into a single point. However, it has been observed empirically in noncontrastive learning methods (Hua et al., 2021; Tian et al., 2021) that while embedding vectors do not
completely collapse; they collapse along certain dimensions. This is known as dimensional collapse
(Hua et al., 2021), whereby the embedding vectors only span a lower-dimensional subspace.

In contrastive methods that explicitly use positive and negative pairs in the loss function, it seems
intuitive to speculate that the repulsive effect of negative examples should prevent this kind of dimensional collapse and make full use of all dimensions. However, contrary to intuition, contrastive
learning methods still suffer from dimensional collapse (See Fig. 7). In this work, we theoretically
study the dynamics behind this phenomenon. We show there are two different mechanisms that


-----

cause collapsing: (1) along the feature direction where the variance caused by the data augmentation is larger than the variance caused by the data distribution, the weight collapses. Moreover, (2)
even if the covariance of data augmentation has a smaller magnitude than the data variance along
all dimensions, the weight will still collapse due to the interplay of weight matrices at different layers known as implicit regularization. This kind of collapsing happens only in networks where the
network has more than one layer.

Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which
directly optimizes the encoder (i.e., representation space) without relying on a trainable projector.
_DirectCLR outperforms SimCLR with a linear trainable projector on ImageNet._

We summarize our contributions as follows:

-  We empirically show that contrastive self-supervised learning suffers from dimensional
collapse whereby all the embedding vectors fall into a lower-dimensional subspace instead
of the entire available embedding space.

-  We showed that there are two mechanisms causing the dimensional collapse in contrastive
learning: (1) strong augmentation along feature dimensions (2) implicit regularization driving models toward low-rank solutions.

-  We propose DirectCLR, a novel contrastive learning method that directly optimizes the representation space without relying on a trainable projector. DirectCLR outperforms SimCLR
with a linear trainable projector.

2 RELATED WORKS

**Self-supervised Learning Methods Joint embedding methods are a promising approach in self-**
supervised learning, whose principle is to match the embedding vectors of augmented views of a
training instance. Contrastive methods (Chen et al., 2020a; He et al., 2016) directly compare training samples by effectively viewing each sample as its own class, typically based on the InfoNCE
contrastive loss (van den Oord et al., 2018) which encourages representations from positive pairs of
examples to be close in the embedding space while representations from negative pairs are pushed
away from each other. In practice, contrastive methods are known to require a large number of
negative samples. Non-contrastive methods do not directly rely on explicit negative samples. These
include clustering-based methods (Caron et al., 2018; 2020), redundancy reduction methods (Zbontar et al., 2021; Bardes et al., 2021) and methods using special architecture design (Grill et al., 2020;
Chen & He, 2020).

**Theoretical Understanding of Self-supervised Learning Although self-supervised learning mod-**
els have shown success in learning useful representations and have outperformed their supervised
counterpart in several downstream transfer learning benchmarks (Chen et al., 2020a), the underlying dynamics of these methods remains somewhat mysterious and poorly understood. Several
theoretical works have attempted to understand it. Arora et al. (2019b); Lee et al. (2020); Tosh
et al. (2021) theoretically proved that the learned representations via contrastive learning are useful for downstream tasks. Tian et al. (2021) explained why non-contrastive learning methods like
BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2020) work: the dynamics of the alignment
of eigenspaces between the predictor and its input correlation matrix play a key role in preventing
complete collapse.

**Implicit Regularization It has been theoretically explained that gradient descent will drive adjacent**
matrices aligned in a linear neural network setting (Ji & Telgarsky, 2019). Under the aligned matrix
assumption, Gunasekar et al. (2018) prove that gradient descent can derive minimal nuclear norm
solution. Arora et al. (2019a) extend this concept to the deep linear network case by theoretically
and empirically demonstrating that a deep linear network can derive low-rank solutions. In general,
over-parametrized neural networks tend to find flatter local minima (Saxe et al., 2019; Neyshabur
et al., 2019; Soudry et al., 2018; Barrett & Dherin, 2021).

3 DIMENSIONAL COLLAPSE

Self-supervised learning methods learn useful representation by minimizing the distances between
embedding vectors from augmented images (Figure 1a). On its own, this would result in a collapsed


-----

Embeddings

|x|Col2|Col3|z|
|---|---|---|---|



Augmentation InfoNCE


|x'|Col2|Col3|z'|
|---|---|---|---|


Input

~x


Encoder


InfoNCE

loss


(a) embedding space (b) complete collapse (c) dimensional collapse

Figure 1: Illustration of the collapsing problem. For complete collapse, the embedding vectors collapse to
same point. For dimensional collapse, the embedding vectors only span a lower dimensional space.


solution where the produced representation becomes constant (Figure 1b). Contrastive methods
prevent complete collapse via the negative term that pushes embedding vectors of different input
images away from each other. In this section, we show that while they prevent complete collapse,
contrastive methods still experience a dimensional collapse in which the embedding vectors occupy
a lower-dimensional subspace than their dimension (Figure 1c).


We train a SimCLR model (Chen et al. (2020a))
with a two-layer MLP projector. We followed
the standard recipe and trained the model on
ImageNet for 100 epoch. We evaluate the dimensionality by collecting the embedding vectors on the validation set. Each embedding vector has a size of d = 128. We compute the co_variance matrix C ∈_ R[d][×][d] of the embedding
layer (here ¯z := _i=1_ **[z][i][/N][ and][ N][ is the total]**
number of samples):

_N_

[P][N]

_C = [1]_ (zi ¯z)(zi ¯z)[T] (1)

_N_ _i=1_ _−_ _−_

X

Figure 2 shows singular value decomposition
on this matrix (C = USV _[T]_, S = diag(σ[k])). in
sorted order and logarithmic scale ({log(σ[k])}).
We observe that a number of singular values
collapse to zero, thus representing collapsed dimensions.


10

15


20


25

|Col1|Col2|
|---|---|
|0 20 40 60 80 Singular Value Rank In|100 120 dex|



Figure 2: Singular value spectrum of the embedding
space. The embedding vectors are computed from a
pretrained SimCLR model on the validation set of ImageNet. Each embedding vector has a dimension of 128.
The spectrum contains the singular values of the covariance matrix of these embedding vectors in sorted order and logarithmic scale. A number of singular values
drop to zero, indicating collapsed dimensions.


4 DIMENSIONAL COLLAPSE CAUSED BY STRONG AUGMENTATION

4.1 LINEAR MODEL


In this section, we explain one scenario for contrastive learning to have collapsed embedding dimensions, where the augmentation surpasses the input information. We focus on a simple linear network
setting. We denote the input vector as x and the augmentation is an additive noise. The network is a
single linear layer with weight matrix is W . Hence, the embedding vector is z = W **x. We focus on**
a typical contrastive loss, InfoNCE (van den Oord et al., 2018):

_N_

exp( **zi** **z[′]i[|][2][/][2)]**

_L =_ log _−|_ _−_ (2)
_−_ _i=1_ _j≠_ _i_ [exp(][−|][z][i][ −] **[z][j][|][2][/][2) + exp(][−|][z][i][ −]** **[z]i[′]** _[|][2][/][2)]_

X

where zi and z[′]i [are a pair of embedding vectors from the two branches,]P **[ z][j][ indicates the negative]**
samples within the minibatch. When all zi and z[′]i [are normalized to be unit vector, the negative]
distancestochastic gradient descent without momentum or weight decay. −|zi − **z[′]i[|][2][/][2][ can be replaced by inner products][ z][T]i** **[z]i[′]** [. The model is trained with a basic]


-----

4.2 GRADIENT FLOW DYNAMICS

We study the dynamics via gradient flow, i.e., gradient descent with an infinitesimally small learning
rate.

**Lemma 1. The weight matrix in a linear contrastive self-supervised learning model evolves by:**

_W˙_ = −G (3)

_where G =_ _i[(][g]zi_ **_[x]i[T]_** [+][ g]zi[′] **_[x]i[′][T]_** [)][, and][ g]zi _[is the gradient on the embedding vector][ z][i][ (similarly][ g]z[′]i_ _[).]_

This can be easily proven based on the chain rule. See proof in Appendix B.1. For InfoNCE loss

[P]

defined in Eqn 2, the gradient of the embedding vector for each branch can be written as


**gzi =**


Xj≠ _i_ _αij(zj −_ **z[′]i[) +]**


Xj≠ _i_ _αji(zj −_ **zi),** **gz′i** [=]


_αij(z[′]i_ (4)

Xj≠ _i_ _[−]_ **[z][i][)]**


where {αij} are the softmax of similarity of between zi and {zj}, defined by αij = exp(−|zi −
**zj|[2]/2)/Zi, αii = exp(−|zi** _−z[′]i[|][2][/][2)][/Z][i][, and][ Z][i][ =][ P]j≠_ _i_ [exp(][−|][z][i] _[−][z][j][|][2][/][2)+exp(][−|][z][i]_ _[−][z]i[′]_ _[|][2][/][2)][.]_

Hence, _j_ _[α][ij][ = 1][. Since][ z][i][ =][ W]_ **[x][i][, we have]**

[P] _G = −WX_ (5)

where


_X :=_ _αij(x[′]i_ _αji(xi_ **xj)** **x[T]i**
_−_ Xi  _j≠_ _i_ _[−]_ **[x][j][) +]** Xj≠ _i_ _−_  _[−]_

[X] 

**Lemma 2. X is a difference of two PSD matrices:**


_T_
(1 _αii)(x[′]i_ _i_ (6)
_−_ _[−]_ **[x][i][)][x][′]**


_X = Σ[ˆ]_ 0 Σ1 (7)
_−_ [ˆ]

_Here_ Σ[ˆ] 0 = _i,j_ _[α][ij][(][x][i][ −]_ **_[x][j][)(][x][i][ −]_** **_[x][j][)][T][ is a weighted data distribution covariance matrix and]_**

Σˆ 1 = _i[(1][ −][P][α][ii][)(][x]i[′]_ _[−]_ **_[x][i][)(][x]i[′]_** _[−]_ **_[x][i][)][T][ is a weighted augmentation distribution covariance matrix.]_**

See proof in Appendix B.2. Therefore, the amplitude of augmentation determines whether X is a

[P]

positive definite matrix. Similar to Theorem 3-4 in Tian et al. (2020), Lemma 2 also models the
time derivative of weight W as a product of W and a symmetric and/or PSD matrices. However,
Lemma 2 is much more general: it applies to InfoNCE with multiple negative contrastive terms,
remains true when αij varies with sample pair (i, j), and holds with finite batch size N . In contrast,
Theorem 4 in Tian et al. (2020) only works for one negative term in InfoNCE, holds only in the
population sense (i.e., N → +∞), and the formulation has residual terms, if αij are not constants.

Next, we look into the dynamics of weight matrix W given property of X.

**Theorem 1. With fixed matrix X (defined in Eqn 6) and strong augmentation such that X has**
_negative eigenvalues, the weight matrix W has vanishing singular values._

See proof in Appendix B.3.

**Corollary 1 (Dimensional Collapse Caused by Strong Augmentation). With strong augmentation,**
_the embedding space covariance matrix becomes low-rank._

The embedding space is identified by the singular value spectrum of the covariance matrix on the

has vanishing singular values,embedding (Eqn. 1), C = _i C[(][z] is also low-rank, indicating collapsed dimensions.[i][ −]_ [¯]z)(zi − ¯z)[T] _/N =_ _i_ _[W]_ [(][x][i][ −] **x[¯])(xi −** ¯x)[T] _W_ _[T]_ _/N_ . Since W

Numerical simulation verifies our theory. We choice input data as isotropic Gaussian with covari-[P] [P]
ance matrix _i,j[(][x][i][ −]_ **[x][j][)(][x][i][ −]** **[x][j][)][T][ /N][ =][ I][. We set the augmentation as additive Gaussian with]**

covariance matrix equal tohas the size of 8x8. We plot the weight matrix singular value spectrum in Figure 3 with various[P] _i[(][x]i[′]_ _[−]_ **[x][i][)(][x]i[′]** _[−]_ **[x][i][)][T][ /N][ =][ block diagonal][(][0][, k][ ∗]** _[I][)][, where the block]_

augmentation amplitude k. This proves that under linear network setting, strong augmentation leads

[P]

to dimensional collapse in embedding space.


-----

Our theory in this section is limited to linear network
settings. For more complex nonlinear networks,
the collapsing condition will still depend on “strong
augmentation” but interpreted differently. A strong
augmentation will be determined by more complicated properties of the augmentation (higher-order
statistics of augmentation, manifold property of augmentation vs. data distribution) conditioned on the
capacity of the networks.

5 DIMENSIONAL COLLAPSE
CAUSED BY IMPLICIT REGULARIZATION

5.1 TWO-LAYER LINEAR MODEL



1.6

1.4

1.2

k=0.1

1.0 k=0.3

k=0.5

0.8 k=0.6

k=0.7

0.6 k=0.8

k=0.9

Singular values0.4 k=1

k=1.2
k=1.5

0.2 k=2

0 2 4 6 8 10 12 14

Singular Value Rank Index


CAUSED BY IMPLICIT REGULARIZATION Figure 3: Weight matrix singular value spectrum

with different augmentation amplitude k. The setting is a single layer linear toy model with each

5.1 TWO-LAYER LINEAR MODEL

weight matrix of the size of 16x16, where the
block has the size of 8x8. Strong augmentation

With strong augmentation, a linear model under In
results in vanishing singular values in weight ma
foNCE loss will have dimensional collapse. How- trices.
ever, such scenarios rely on the condition that the
network has a limited capacity which may not hold
for real cases. On the other hand, when there is no
strong augmentation (Σ[ˆ] 1 Σ0) and thus X matrix remains PSD, a single linear model won’t have
dimensional collapsing. However, interestingly, for deep networks, dimensional collapsing still ≺ [ˆ]
happens in practice. In the following, we will show that it stems from a different nature: implicit
regularization, where over-parametrized linear networks tend to find low-rank solutions.


To understand this counter-intuitive phenomena, we
start with the simplest over-parametrized setting
by choosing the network as a two-layer linear MLP
without bias. The weight matrices of these two layers are denoted bySimilar to the setting in Sec 4, the input vector is W1 ∈ R[d][×][d] and W2 ∈ R[d][×][d].
denoted as x and the augmentation is an additive
noise. The embedding vector from each branch is
**z = W2W1x, hence z ∈** R[n]. We do not normalize
**z. See Figure 4. We use InfoNCE loss defined in**
Eqn 2. The model is trained with a basic stochastic
gradient descent without momentum or weight decay.

5.2 GRADIENT FLOW DYNAMICS


|x|Col2|W1|Col4|W2|Col6|z|
|---|---|---|---|---|---|---|


InfoNCE

loss

|x'|Col2|W1|Col4|W2|Col6|z'|
|---|---|---|---|---|---|---|


Input

~x


Figure 4: Two-layer Linear Model


Similar to Lemma 1, we derive the gradient flow on the two weight matrices W1 and W2.
**Lemma 3. The weight matrices of the two layer linear contrastive self-supervised learning model**
_evolves by (G =_ _i[(][g]zi_ **_[x]i[T]_** [+][ g]z[′]i **_[x]i[′][T]_** [)][ is defined in Lemma 1):]

_W˙_ 1 = _W2[T]_ _[G,]_ _W˙_ 2 = _GW1[T]_ (8)

[P] _−_ _−_

This can be easily proven based on the chain rule. See proof in Appendix B.4. For the two layer
case, similar to Eqn 5, we have the specific form of G:

_G =_ _W2W1X_ (9)
_−_

where X is defined in Eqn 6. According to Lemma 2, we know that with small augmentation,
_X = Σ[ˆ]_ 0 Σ1 0 is a positive-definite matrix.
_−_ [ˆ] _≻_

5.3 WEIGHT ALIGNMENT

Since we have two matrices W1 and W2, the first question is how they interact with each other.
We apply singular value decomposition on both matrices W1 and W2, i.e., W1 = U1S1V1[T] [,][ W][2] [=]
_U2S2V2[T]_ [and][ S][1] [=][ diag][([][σ]1[k][])][,][ S][2] [=][ diag][([][σ]2[k][])][. The alignment is now governed by the interaction]


-----

between the adjacent orthonormal matrices V2 := [v2[k][]][ and][ U][1] [= [][u][k]1[]][. This can be characterized]
by the alignment matrix A = V2[T] _[U][1][, whose][ (][k, k][′][)][-entry represents the alignment between the][ k][-th]_
right singular vector v2[k] [of][ W][2] [and the][ k][′][-th left singular vector][ u]1[k][′] [of][ W][1][. The following shows]
that indeed W1 and W2 aligns.
**Theorem 2 (Weight matrices align). If for all t, W2(t)W1(t) ̸= 0, X(t) is positive-definite and**
_W1(+_ ), W2(+ ) have distinctive singular values, then the alignment matrix A = V2[T] _[U][1]_
_∞_ _∞_ _[→]_ _[I][.]_

See proof in Appendix B.5. Here, we also empirically demonstrate that under InfoNCE loss, the
absolute value of the alignment matrix A converges to an identity matrix. See Figure 5.

The alignment effect has been studied in other scenarios (Ji & Telgarsky, 2019; Radhakrishnan et al.,
2020). In the real case, when some of our assumptions are not satisfied, e.g., there are degenerate
singular values in weight matrices, we will not observe a perfect alignment. This can be easily understood by the fact that the singular decomposition is no longer unique given degenerate singular
values. In our toy experiment, we specifically initialize the weight matrices to have non-degenerate
singular values. In real scenario, when weight matrices are randomly initialized, we will only observe the alignment matrix to converge to a block-diagonal matrix, with each block representing a
group of degenerate singular values.

Given the fact that singular vectors corresponding to
the same singular value align, we can now study the
dynamics of the singular values of each weight matrix W1 and W2.
**Theorem 3. If W2 and W1 are aligned (i.e., V2 =**
_U1[T]_ _[), then the singular values of the weight matrices]_
_W1 and W2 under InfoNCE loss evolve by:_


_T_ _k_
_σ˙_ 1[k] [=][ σ]1[k][(][σ]2[k][)][2][(][v]1[k] _Xv1[)]_ (10)

_T_ _k_ Figure 5: Visualization of the alignment matrix
_σ˙_ 2[k] [=][ σ]2[k][(][σ]1[k][)][2][(][v]1[k] _Xv1[)]_ (11) _A = V2[T]_ _[U]1_ [after training. The setting is a 2-layer]

linear toy model with each weight matrix of the

See proof in Appendix B.6. According to Eqn. 10, size of 16x16. The alignment matrix converges to
(σ1[k][)][2] = (σ2[k][)][2][ +][ C][.] We solve the singular an identity matrix.
˙
value dynamics analytically: _σ1[k]_ = σ1[k][((][σ]1[k][)][2][ +]
_C)(v[k]1_ _T Xvk1[)][. This shows that a pair of singular values (singular values with same ranking from]_
the other matrix) have gradients proportional to themselves. Notice that X is a positive definite ma_T_ _k_
trix, the term v[k]1 _Xv1_ [is always non-negative. This explains why we observe that the smallest group]
of singular values grow significantly slower. See demonstrative experiment results in Figure 6a and
6b.


W1 Spectrum W2 Spectrum Embedding Space Spectrum

0 0 0

2 1 5

10

4 2

15

6 3

20

log singular values 8 log singular values 4 log singular values

25

0 500 1000 1500iterations2000 2500 3000 3500 4000 0 500 1000 1500iterations2000 2500 3000 3500 4000 0 500 1000 1500iterations2000 2500 3000 3500 4000

(a) W1


1500iterations2000

(b) W2


500 1000 1500iterations2000 2500 3000 3500

(c) Embedding Space


Figure 6: Evolution of the singular values of the weight matrices and the embedding space covariance matrix.
The setting is a 2-layer linear toy model with each weight matrix of the size of 16x16. The lowest few singular
values of each weight matrix remain significantly smaller.

**Corollary 2 (Dimensional Collapse Caused by Implicit Regularization). With small augmentation**
_and over-parametrized linear networks, the embedding space covariance matrix becomes low-rank._

The embedding space is identified by the singular value spectrum of the covariance matrix on the
embedding vectors, C = (z − ¯z)(z − ¯z)[T] _/N =_ _W2W1(x −_ ¯x)(x − ¯x)[T] _W1[T]_ _[W][ T]2_ _[/N]_ [. As]

[P] [P]


-----

_W2W1 evolves to be low-rank, C is low-rank, indicating collapsed dimensions. See Figure 6c for_
experimental verification.

Our theory can also be extended to multilayer networks and nonlinear setting. Please see Appendix C

6 DIRECTCLR

6.1 MOTIVATION

We now leverage our theoretical finding to design novel algorithms. Here we are targeting the
projector component in contrastive learning.

Empirically, adding a projector substantially improves the quality of the learned representation and
downstream performance (Chen et al., 2020a). Checking the spectrum of the representation layer

|x|Col2|Col3|r|Col5|Col6|z|
|---|---|---|---|---|---|---|


|x'|Col2|Col3|r'|Col5|Col6|z'|
|---|---|---|---|---|---|---|


|w/ projector w/o projector|w/ projector w/o projector|Col3|
|---|---|---|
|0 500 1000 1500 20 Singular Value Rank Index||00|


also reveals a difference with/without a projector. To see this, we train two SimCLR models with and
without a projector. The representation space spectrum are shown in Figure 7b. The dimensional
collapse in representation space happens when the model is trained without a projector. Thus, the
projector prevents the collapse in the representation space.

0

Representations Embeddings w/ projectorw/o projector

5

x r z

Input

10

~x Augmentation InfoNCE

loss 15

x' r' z' Log of singular values 20

25

Encoder Projector 0 500 1000 1500 2000

Singular Value Rank Index

(a) representation and embedding (b) Representation space spectrum

Figure 7: (a) Definition of representation and the embedding space; (b) Singular value spectrums of the
representation space of pretrained contrastive learning models (pretrained with or without a projector). The
representation vectors are the output from the ResNet50 encoder and directly used for downstream tasks. Each
representation vector has a dimension of 2048. Without a projector, SimCLR suffers from dimensional collapse


in the representation space.

The projector in contrastive learning is essential to prevent dimensional collapse in the representation space. We claim the following propositions regarding a linear projector in contrastive learning
models.
**Proposition 1. A linear projector weight matrix only needs to be diagonal.**
**Proposition 2. A linear projector weight matrix only needs to be low-rank.**

Based on our theory on implicit regularization dynamics, we expect to see adjacent layers W1(=
_U1S1V1[T]_ [)][ and][ W][2][(=][ U][2][S][2][V][ T]2 [)][ to be aligned such that the overall dynamics is only governed by]
their singular values S1 and S2. And the orthogonal matrices V2[T] [and][ U][1] [are redundant as they will]
evolve to V2[T] _[U][1]_ [=][ I][, given][ S][1] [and][ S][2][.]

Now, let’s consider the linear projector SimCLR model and only focus on the channel dimension.
_W1 is the last layer in the encoder, and W2 is the projector weight matrix. Our propositions claim_
that for this projector matrix W2, the orthogonal component V2 can be omitted. Because the previous
layer W1 is fully trainable, its orthogonal component (U1) will always evolve to satisfy V2[T] _[U][1]_ [=][ I][.]
Therefore, the final behavior of the projector is only determined by the singular values (S2 ) of the
projector weight matrix. This motivates Proposition 1: the orthogonal component of the weight
matrix doesn’t matter. So we can set the projector matrix as a diagonal matrix.

Also, according to our theory, the weight matrix will always converge to the low-rank. The singular
value diagonal matrix naturally becomes low-rank, so why not just set it low-rank directly? This is
the motivation of Proposition 2.


-----

These propositions are verified via ablation studies in Sec 6.3. Given these two propositions, we
propose DirectCLR, which is effectively using a low-rank diagonal projector.

6.2 MAIN IDEA


We propose to remove the projector in contrastive

Representations

learning by directly sending a sub-vector of the
representation vector to the loss function. We x r
call our method DirectCLR. In contrast to all re- Input

|x|Col2|Col3|z|
|---|---|---|---|
||||r|

cent state-of-the-art self-supervised learning meth- ~x Augmentation InfoNCE
ods, our method directly optimizes the representa- loss
tion space. See Figure 8, DirectCLR picks a subvec- z'
tor of the representation z = r[0 : d0], where d0 x' r'

Input

~

x

is a hyperparameter. Then, it applies a standard In
|Col1|Col2|Col3|z'|
|---|---|---|---|
|x'|||z'|
||||r'|

foNCE loss on this normalized subvectorL = _i_ [log] exp(ˆj [exp(ˆ]zi·zˆzi[′]i·ˆ[)]zj ) [.] ˆz = z/|z|, Encoder

Figure 8: DirectCLR: no trainable projector, sim
P

We train DirectCLR with a standard recipe of Sim- ply apply InfoNCE loss on the a fixed sub-vector

[P]

CLR for 100 epochs on ImageNet. The backbone of the representations
encoder is a ResNet50. More implementation details
can be found in the Appendix D. DirectCLR demonstrates better performance compared to SimCLR
with a trainable linear projector on ImageNet. The linear probe accuracies for each model are listed
in Table 1.

Loss function Projector Accuracy


SimCLR 2-layer nonlinear projector 66.5
SimCLR 1-layer linear projector 61.1
SimCLR no projector 51.5
_DirectCLR_ no projector 62.7

Table 1: Linear probe accuracy on ImageNet. Each model is trained on ImageNet for 100 epochs with standard
training recipe. The backbone encoder is a ResNet50. DirectCLR outperforms SimCLR with 1-layer linear
projector.


We visualize the learnt representation space spectrum in Figure 9. DirectCLR prevents dimensional
collapse in the representation space similar to the functionality of a trainable projector in SimCLR.

0

SimCLR: 2-layer nonlinear projectorSimCLR: 1-layer linear projector residual connection


z InfoNCE


10

15

|Col1|Col2|Col3|residual connection|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||h||full-rank|low-r||ank|z r||
||||||ow-r||||


hidden layer nonlinear representations
(full-rank) conv block

Figure 10: Why is the whole representation vector r
meaningful in DirectCLR while only part of it receives
gradient? It takes advantage of the residual connection
in the backbone. Thus, the gradient passing through the
representation vector is low-rank where only the first d0
channel dimensions are non-zero. When the gradient
enters the ResNet backbone and passes through the last
nonlinear conv block, it becomes full rank. Therefore,
this hidden layer h receives gradients on all channels.
During forward pass, h is directly fed to the representation vectors via the residual connection. Therefore, the
entire representation vector r is meaningful.


20

25


SimCLR: 2-layer nonlinear projector
SimCLR: 1-layer linear projector
SimCLR: no projector
DirectCLR: no projector


|SimCLR: 2-layer nonlinear projector SimCLR: 1-layer linear projector SimCLR: no projector DirectCLR: no projector|SimCLR: 2-layer nonlinear projector SimCLR: 1-layer linear projector SimCLR: no projector DirectCLR: no projector|Col3|
|---|---|---|
|0 500 1000 1500 20 Singular Value Rank Index||00|


Figure 9: Representation space spectrum of Di_rectCLR compared to SimCLR (a) with a 2-layer_
nonlinear projector (b) with a 1-layer linear projector (c) without projector. The spectrums are
computed based on the output from the backbone,
using ImgaeNet validation set. Similar to SimCLR with projectors, DirectCLR is able to prevent
dimensional collapse in the representation space.


-----

One may suspect that the contrastive loss in DirectCLR does not apply a gradient on the rest part of
the representation vector r[d0 :], then why these dimensions would contain useful information?

Here, we show that the entire representation vector r contains useful information. See Figure 10.
First, the gradient backpropagating through the representation vector is low-rank, where only the
first d0 channel dimensions are non-zero. When the gradient enters the ResNet backbone and passes
through the last nonlinear conv block, it becomes full rank. Therefore, this hidden layer h receives
gradients on all channels. Note that h and r have a same channel dimension of 2048. Next, we
consider the forward pass. This hidden layer h is directly fed to the representation vectors via the
residual connection. As a result, the rest part of the representation vector r[d0 :] is not trivial. In
addition, we run an ablation study in Sec F to test the linear probe accuracy based only on the
“directly” optimized vector. This verifies that the whole representation vector is meaningful.

6.3 ABLATION STUDY

Projector diagonal low-rank Top-1 Accuracy

no projector 51.5
orthogonal projector 52.2
trainable projector 61.1
trainable diagonal projector ✓ 60.2
fixed low-rank projector ✓ 62.3
fixed low-rank diagonal projector ✓ ✓ 62.7

Table 2: Ablation study: top-1 accuracies on ImageNet by SimCLR model with different projector settings.

To further verify our hypothesis, we have perform ablation studies.

Proposition 1 matches the fact that: (a) an orthogonal constrained projector performs the same as the
non-projector setting; (b) fixed low-rank projector performs the same as a fixed diagonal projector;
(c) trainable linear projector performs the same as a trainable diagonal projector.

Proposition 2 matches the observation that a low-rank projector has the highest accuracy.

Please see more detailed ablation study discuss and additional ablation experiments in Appendix F.

7 CONCLUSIONS

In this work, we showed that contrastive self-supervised learning suffers from dimensional collapse,
where the embedding vectors only span a lower-dimensional subspace. We provided the theoretical
understanding of this phenomenon and showed that there are two mechanisms causing dimensional
collapse: strong augmentation and implicit regularization. Inspired by our theory, we proposed a
novel contrastive self-supervised learning method DirectCLR that directly optimizes the representation space without relying on a trainable projector. DirectCLR outperforms SimCLR with a linear
projector on ImageNet.

ACKNOWLEDGEMENT

We thank Yubei Chen, Jiachen Zhu, Adrien Bardes, Nicolas Ballas, Randall Balestriero, Quentin
Garrido for useful discussions.

REPRODUCIBILITY STATEMENT

We provide detailed proof for all the lemmas and theorems in the Appendices. Code (in PyTorch) is
[available at https://github.com/facebookresearch/directclr](https://github.com/facebookresearch/directclr)


-----

REFERENCES

Sanjeev Arora, Nadav Cohen, W. Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In NeurIPS, 2019a.

Sanjeev Arora, H. Khandeparkar, M. Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical
analysis of contrastive unsupervised representation learning. In ICML, 2019b.

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by non-parametrically
predicting view assignments with support samples. ArXiv, abs/2104.13963, 2021.

Adrien Bardes, J. Ponce, and Y. LeCun. Vicreg: Variance-invariance-covariance regularization for
self-supervised learning. ArXiv, abs/2105.04906, 2021.

D. Barrett and B. Dherin. Implicit gradient regularization. ArXiv, abs/2009.11162, 2021.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and M. Douze. Deep clustering for unsupervised
learning of visual features. In ECCV, 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv’e J’egou, J. Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. ArXiv, abs/2104.14294,
2021.

Mario Lezcano Casado and David Mart´ınez-Rubio. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. ArXiv, abs/1901.08428,
2019.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework
for contrastive learning of visual representations. 2020a.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020.

Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. ArXiv, abs/2003.04297, 2020b.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With
a little help from my friends: Nearest-neighbor contrastive learning of visual representations.
_ArXiv, abs/2104.14548, 2021._

Jean-Bastien Grill, Florian Strub, Florent Altch´e, Corentin Tallec, Pierre H. Richemond, Elena
Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko. Bootstrap your own
latent: A new approach to self-supervised learning. In NeurIPS, 2020.

Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan
Srebro. Implicit regularization in matrix factorization. 2018 Information Theory and Applications
_Workshop (ITA), pp. 1–10, 2018._

Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. ArXiv, abs/2106.04156, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for
unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision
_and Pattern Recognition (CVPR), pp. 9726–9735, 2020._

Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. On feature
decorrelation in self-supervised learning. ArXiv, abs/2105.00470, 2021.


-----

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. ArXiv,
abs/1810.02032, 2019.

L. Jing, J. Zbontar, and Y. LeCun. Implicit rank-minimizing autoencoder. ArXiv, abs/2010.00679,
2020.

J. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. ArXiv, abs/2008.01064, 2020.

Junnan Li, Pan Zhou, Caiming Xiong, R. Socher, and S. Hoi. Prototypical contrastive learning of
unsupervised representations. ArXiv, abs/2005.04966, 2021.

Ishan Misra and L. V. D. Maaten. Self-supervised learning of pretext-invariant representations. 2020
_IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6706–6716,_
2020a.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, 2020b.

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Y. LeCun, and Nathan Srebro. Towards
understanding the role of over-parametrization in generalization of neural networks. _ArXiv,_
abs/1805.12076, 2019.

Adityanarayanan Radhakrishnan, Eshaan Nichani, D. Bernstein, and Caroline Uhler. On alignment
in deep linear neural networks. arXiv: Learning, 2020.

Andrew M. Saxe, James L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116:11537 –
11546, 2019.

Daniel Soudry, E. Hoffer, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient
descent on separable data. ArXiv, abs/1710.10345, 2018.

Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning
with dual deep networks. arXiv preprint arXiv:2010.00578, 2020.

Yuandong Tian, Xinlei Chen, and S. Ganguli. Understanding self-supervised learning dynamics
without contrastive pairs. ArXiv, abs/2102.06810, 2021.

Christopher Tosh, A. Krishnamurthy, and Daniel J. Hsu. Contrastive learning, multi-view redundancy, and linear models. ArXiv, abs/2008.10150, 2021.

A¨aron van den Oord, Y. Li, and Oriol Vinyals. Representation learning with contrastive predictive
coding. ArXiv, abs/1807.03748, 2018.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised
learning via redundancy reduction. arXiv preprint arxiv:2103.03230, 2021.

A USEFUL LEMMAS

We adapt two useful lemmas from Arora et al. (2019a).

**Lemma 4. Given a matrix W and the dynamics that W evolves by** _W[˙]_ _, the singular values of this_
_matrix evolve by:_

˙
_σ[k]_ = u[kT][ ˙]W **_v[k]_** (12)

_where u[k]_ _and v[k]_ _are singular value σ[k]’s corresponding left and right singular vectors. i.e. the k-th_
_column of matrices U and V respectively._


-----

_Proof. Given a matrix W and its singular value decomposition W = USV_ _[T]_ . We have the dynamics
of the matrix
_W˙_ = USV[˙] _[T]_ + U _SV[˙]_ _[T]_ + USV[˙] _[T]_

Multiplying U _[T]_ from the left and multiplying V from the right, considering U and V are orthogonal
matrices, we have
_U_ _[T][ ˙]WV = U_ _[T][ ˙]US + S[˙] + SV[˙]_ _[T]_ _V_
Since S = diag(σ[k]) is a diagonal matrix, we have

˙
_σ[k]_ = u[kT][ ˙]W **v[k]** **u[kT][ ˙]u[k]σ[k]** _σ[k][ ˙]v[kT]_ **v[k]**
_−_ _−_

Again, considering u[k] and v[k] have unit-norm, we have u[kT][ ˙]u[k] = 0 and **v[˙][kT]** **v[k]** = 0. Therefore, we
derive
˙
_σ[k]_ = u[kT][ ˙]W **v[k]**

**Lemma 5. Given a matrix W and the dynamics that W evolves by** _W[˙]_ _, the singular vectors of this_
_matrix evolve by:_

_U˙_ = U (H ⊙ (U _[T][ ˙]WV S + SV_ _[T][ ˙]W_ _[T]_ _U_ )) (13)

_V˙ = V (H ⊙_ (V _[T][ ˙]W_ _[T]_ _US + SU_ _[T][ ˙]WV ))_ (14)

_where ⊙_ _represents Hadamard element-wise multiplication. H is a skew-symmetric matrix_

1/(σ[k][2] _σ[k][′]_ [2]) _if k_ = k[′]

_H_ _[k,k][′]_ = _−_ _̸_ (15)

(0 _if k = k[′]_

_Proof. Same as proof for Lemma 1, we start from the following equation_

_U_ _[T][ ˙]WV = U_ _[T][ ˙]US + S[˙] + SV[˙]_ _[T]_ _V_

Considering the fact that U _[T][ ˙]U and_ _V[˙]_ _[T]_ _V are skew-symmetric matrices, whose diagonal terms are_
all zero, we Hadamard-multiply _I[¯] to both sides of the equation. Here,_ _I[¯] has all diagonal values equal_
zeros and all off-diagonal values equal to one, we have

_I¯ ⊙_ _U_ _[T][ ˙]WV = U_ _[T][ ˙]US + SV[˙]_ _[T]_ _V_ (16)

Taking transpose, we have
_I¯ ⊙_ _V_ _[T][ ˙]WU = −SU_ _[T][ ˙]U −_ _V[˙]_ _[T]_ _V S_ (17)
Right-multiplying S to Eqn 16 and left-multiplying S to Eqn 17, then adding them up, we have


_U_ _[T][ ˙]US[2]_ _−_ _S[2]U_ _[T][ ˙]U = I[¯] ⊙_ (U _[T][ ˙]WV S + SV_ _[T][ ˙]WU_ )

Therefore, we have
_U˙_ = U (H ⊙ (U _[T][ ˙]WV S + SV_ _[T][ ˙]W_ _[T]_ _U_ ))
where

1/(σ[k][2] _σ[k][′]_ [2]) if k = k[′]

_H_ _[k,k][′]_ = _−_ _̸_

(0 if k = k[′]

Similar proof applies to Eqn 14.

**Lemma 6 (Alignment matrix dynamics). The alignment matrix A, defined by A = V2[T]** _[U][1][, evolves]_
_by:_
_A˙ =_ _A(H1_ (A[T] _F + F_ _[T]_ _A)) + (H2_ (AF _[T]_ + FA[T] ))A (18)
_−_ _⊙_ _⊙_
_where ⊙_ _represents Hadamard (element-wise) multiplication. Hl is a skew-symmetric matrix, whose_
(k, k[′])-entry is given by

2 _k[′]_ 2

_Hl[k,k][′]_ = (10/(σl[k] _−_ _σl_ ) _ifif k k ̸ == k k[′][′]_ (19)

_and F is defined by_
_F = S2U2[T]_ _[GV][1][S][1]_ (20)


-----

_Proof. According to Lemma. 5, we have_

_U˙1 = U1(H1 ⊙_ (U1[T] _W˙_ 1V1S1 + S1V1[T] _W˙_ 1[T] _[U][1][))]_
_V˙2 = V2(H2_ (V2[T] _W˙_ 2[T] _[U][2][S][2]_ [+][ S][2][U][ T]2 _W˙_ 2V2))
_⊙_

Plugging the above two equations and Eqn 8, the dynamics of the alignment matrix A = V2[T] _[U][1]_ [can]
be written as

_A˙_ = _V2[T]_ _U[˙]_ 1 + V[˙]2[T] _[U][1]_
= _V2[T]_ _[U][1][(][H][1]_ 1 _W[˙]_ 1V1S1 + S1V1[T] _W˙_ 1[T] _[U][1][)) + (][H][2]_ 2 _W˙_ 2[T] _[U][2][S][2]_ [+][ S][2][U][ T]2 _W[˙]_ 2V2))[T] _V2[T]_ _[U][1]_

_[⊙]_ [(][U][ T] _[⊙]_ [(][V][ T]
= _−A(H1 ⊙_ (U1[T] _[W][ T]2_ _[GV][1][S][1]_ [+][ S][1][V][ T]1 _[G][T][ W][2][U][1][)) + (][H][2]_ _[⊙]_ [(][S][2][U][ T]2 _[GW][ T]1_ _[V][2]_ [+][ V][ T]2 _[W][1][G][T][ U][2][S][2][))][A]_

= _−A(H1 ⊙_ (U1[T] _[V][2][S][2][U][ T]2_ _[GV][1][S][1]_ [+][ S][1][V][ T]1 _[G][T][ U][2][S][2][V][ T]2_ _[U][1][))]_

+(H2 ⊙ (S2U2[T] _[GV][1][S][1][U][ T]1_ _[V][2]_ [+][ V][ T]2 _[U][1][S][1][V][ T]1_ _[G][T][ U][2][S][2][))][A]_

= _−A(H1 ⊙_ (A[T] _S2U2[T]_ _[GV][1][S][1]_ [+][ S][1][V][ T]1 _[G][T][ U][2][S][2][A][)]_

+(H2 ⊙ (S2U2[T] _[GV][1][S][1][A][T][ +][ AS][1][V][ T]1_ _[G][T][ U][2][S][2][))][A]_

= _A(H1_ (A[T] _F + F_ _[T]_ _A)) + (H2_ (AF _[T]_ + FA[T] ))A
_−_ _⊙_ _⊙_

where
_F = S2U2[T]_ _[GV][1][S][1]_


**Lemma 7 (Singular value dynamics). The singular values of the weight matrices W1 and W2 evolve**
_by:_


_T_ _T_
(v[k]2[′] **_uk1[)][σ]2[k][′]_** [(][u]2[k][′] _Gvk1[)]_ (21)
_k[′]_

X

(u[k]1[′] _T vk2[)][σ]1[k][′]_ [(][u]2[k]T Gv1k[′] [)] (22)
_k[′]_

X


˙
_σ1[k]_ [=][ −]

˙
_σ2[k]_ [=][ −]


_Proof. According to Lemma 4,_
_σ˙_ 1[r] [=][ u]1[r] _T ˙W1v[r]1_
Plugging in Eqn 8, we have

_σ˙_ 1[k] = _−u[k]1_ _T W T2_ _[G][v]1[k]_

= _−u[k]1_ _T V2S2U T2_ _[G][v]1[k]_

_T_ _T_

= (v2[k][′] **uk1[)][σ]2[k][′]** [(][u]2[k][′] _Gvk1[)]_
_−_

_k[′]_

X


Similar proof applies to Eqn 22.

B DELAYED PROOFS

B.1 PROOF OF LEMMA 1

The gradient on matrix W is


_∂zi_

_∂W_ [+][ ∂L]∂z[′]i


_∂z[′]i_

_∂W_ [)]


_dL_

_dW_ [=]


( _[∂L]_

_∂zi_


We denote the gradient on zi and z[′]i [as][ g]zi [and][ g]z[′]i [, respectively. Since][ ∂]∂W[z][i] [=][ x][i][ and][ ∂]∂W[z]i[′] [=][ x]i[′] [, we]

get
_W˙_ = ( _[dL]_ (gzi **x[T]i** [+][ g]z[′]i **[x]i[′]** _T )_
_−_ _dW_ [)][T][ =][ −]

X


-----

B.2 PROOF OF LEMMA 2

_Proof. X is defined in Eqn 6._

_X_ = ( _αij(x[′]i_
Xi Xj≠ _i_ _[−]_ **[x][j][) +]**


_αji(xi_ **xj))x[T]i**

Xj≠ _i_ _−_ _[−]_


(1 _αii)(x[′]i_ _i_
_−_ _[−]_ **[x][i][)][x][′]**


Xj≠ _i_ _αijx[′]i[x][T]i_ _[−]_ Xi Xj≠ _i_ _αijxjx[T]i_ [+] Xi Xj≠ _i_ _αji(xi −_ **xj)(xi −** **xj)[T]**

_αji(xi_ **xj)x[T]j** (1 _αii)(x[′]i_ _i_ (1 _αii)(x[′]i_

Xi Xj≠ _i_ _−_ _[−]_ Xi _−_ _[−]_ **[x][i][)(][x][′]** _[−]_ **[x][i][)][T][ −]** Xi _−_ _[−]_ **[x][i][)][x][i]**


Given the fact that _j=i_ _[α][ij][ = 1][ −]_ _[α][ii][, we have][ P]i_ _j=i_ _[α][ij][x]i[′]_ **[x][T]i** [=][ P]i[(1][ −] _[α][ii][)][x]i[′]_ **[x][T]i** [. Also,]

_̸_ _̸_

since _i_ _j=i_ [iterates all pairs of][ i, j][, we can replace the index between][ i][ and][ j][, we have]

_̸_ P

_i_ _j≠_ _i_ P[α][ij][x][j][x]i[T] [=][P][ P]i _j≠_ _i_ _[α][ji][x][i][x]j[T]_ [.]

[P]

ThereforeP P P

_X =_ _αji(xi_ **xj)(xi** **xj)[T]** (1 _αii)(x[′]i_ _i_
Xi Xj≠ _i_ _−_ _−_ _−_ Xi _−_ _[−]_ **[x][i][)(][x][′]** _[−]_ **[x][i][)][T]**

B.3 PROOF OF THEOREM 1

_Proof. According to Lemma 1, we have_


_d_

(23)
_dt_ _[W][ =][ WX]_

For a fixed X, we solve this equation analyically,

_W_ (t) = W (0) exp(Xt)

Apply eigen-decomposition on X, X = U ΛU _[T]_ . Then we have exp(Xt) = U exp(Λt)U _[T]_ . Therefore,
_W_ (t) = W (0)U exp(Λt)U _[T]_

Because X has negative eigenvalues, i.e., Λ has negative terms, we have for t →∞, exp(Λt) is rank
deficient. Therefore, we know that W (∞) is also rank deficient, the weight matrix W has vanishing
singular values.

B.4 PROOF OF LEMMA 3

_Proof. The gradient on matrix W2 is_


_∂zi_ + _[∂L]_ _∂z[′]i_ ) (24)

_∂W2_ _∂z[′]i_ _∂W2_


_dL_

_dW2_


( _[∂L]_

_∂zi_


We denote the gradient on zi and z[′]i [as][ g]zi [and][ g]z[′]i [, respectively. Since] _∂W∂zi2_ [=][ W][1][x][i][ and] _∂W∂z[′]i2_ [=]

_W1x[′]i[, we get]_

_W˙_ 2 = ( _[dL]_ )[T] = (gzi **x[T]i** [+][ g]z[′]i **[x]i[′]** _T )W T1_ (25)
_−_ _dW2_ _−_

_i_

X

Similar proof applies to W1.


-----

B.5 PROOF OF THEOREM 2

Here, we prove that under the assumption that singular values are non-degenerate, the alignment
matrix A = V2[T] _[U][1]_ [converges to identity matrix.]

_Proof. According to Lemma 3, we have_
_d_

1 [) =][ −][W][1][G][T][ W][2] 2 _[GW][ T]1_
_dt_ [(][W][1][W][ T] _[−]_ _[W][ T]_

_d_

2 _[W][2][) =][ −][W][ T]2_ _[GW][ T]1_
_dt_ [(][W][ T] _[−]_ _[W][1][G][T][ W][2]_

therefore,
_d_

1 2 _[W][2][) = 0]_
_dt_ [(][W][1][W][ T] _[−]_ _[W][ T]_
or
_W1W1[T]_ 2 _[W][2]_ [=][ C]

_[−]_ _[W][ T]_

Next, we show that the Frobenius norm of each weight matrix grow to infinitely.


_d_

_F_ [=][ d] 1 [) =][ −][tr][(][W][ T]2 _[GW][ T]1_ [)][ −] _[tr][(][W][1][G]1[T]_ _[W][2][)]_
_dt_ _[||][W][1][||][2]_ _dt_ _[tr][(][W][1][W][ T]_

According to Eqn 9, G = _W2W1X, we have_
_−_

_−tr(W2[T]_ _[GW][ T]1_ [)] = _tr(W2[T]_ _[W][2][W][1][XW][ T]1_ [)]

= _tr(W2W1XW1[T]_ _[W][ T]2_ [)]

Because X is a positive definite matrix and for all t, W2(t)W1(t) = 0, we know B :=
_̸_
_W2W1XW1[T]_ _[W][ T]2_ [is positive semi-definite and][ B][ ̸][= 0][. Therefore,][ tr][(][B][) =][ P]k _[λ][k][(][B][)][ >][ 0][ since]_

not all eigenvalues of B are zero.

Therefore, we know _W1_ _F_ _F_
_||_ _||[2]_ _[→]_ [+][∞] [(similarly][ ||][W][2][||][2] _[→]_ [+][∞][). In the limit][ t][−] _[>][ +][∞][, we have]_

_W1W1[T]_ [=][ W][ T]2 _[W][2]_

Plug in the singular value decomposition of W1 and W2, we have U1S1[2][U][ T]1 [=][ V][2][S]2[2][V][ T]2 [. Assuming]
_W1 and W2 have non-degenerate singular values, due to the uniqueness of eigen-decomposition, we_
have
_U1 = V2_
therefore,
_V2[T]_ _[U][1]_ [=][ I]

**Remark. Note that when the non-degenerate singular value assumption does not hold, the corre-**
sponding singular vectors are not unique and we will not observe the corresponding dimensions
becoming aligned.

B.6 PROOF OF THEOREM 3

_Proof. According to Theorem 2, for σ1[k]_ [and][ σ]2[k] [with same index, the corresponding singular vector]

_T_
pairsto **v[k]2** [and][ u]1[k] [will get aligned, i.e.,][ v]2[k][′] **uk1** _[→]_ _[δ][i,j][. Therefore, Eqn 21 and Eqn 22 can be simplified]_
˙ _T_ _k_
_σ1[k]_ 2 [(][u][k]2 _Gv1[)]_
˙ _[→−][σ][k]_ _T_ _k_
_σ2[k]_ 1 [(][u][k]2 _Gv1[)]_

_[→−][σ][k]_

Insert Eqn 9 and considering the alignment, we derive


_σσ˙˙_ 21[k][k] _[→][→]_ _[σ][σ]21[k][k][(][(][σ][σ]12[k][k][)][)][2][2][(][(][v][v][k]1[k]1_ _TT X Xvvk1k1[)][)]_


-----

C EFFECT OF MORE LAYERS AND NONLINEARITY

In our toy model, we focused on a two-layer linear MLP setting. Here, we empirically show that our
theory extends to multilayer and nonlinear cases, as shown in Figure 11a.

Stronger over-parametrization leads to a stronger collapsing effect, which has been shown theoretically (Arora et al., 2019a; Barrett & Dherin, 2021) and empirically (Jing et al., 2020). This can be
explained by the fact that more adjacent matrices getting aligned, and the collapsing in the product
matrix gets amplified. Note that for a single-layer case, L = 1, there is no dimensional collapse in
the embedding space, which is consistent with our analysis.


0 0

5 5

10 10

15 15

20 L=1 20 L=1

Log of singular values L=2 Log of singular values L=2

L=3 L=3
L=4 L=4

25 25

0 2 4 6 8 10 12 14 0 2 4 6 8 10 12 14

Singular Value Rank Index Singular Value Rank Index

(a) multiple layers


Singular Value Rank Index

(b) nonlinear


Figure 11: Embedding space singular value spectrum with different layers on (a) linear and (b) nonlinear
networks. All models use weight matrices with a size of 16x16. Adding more layers in the network leads to
more collapsed dimensions. Adding nonlinearity leads to a similar collapsing effect.

We empirically show that the collapsing effect also applies to the nonlinear scenario. We insert
ReLU between linear layers and observe a similar singular value collapse compared to the linear
case. See Figure 11b.

D IMPLEMENTATION DETAIL


D.1 AUGMENTATIONS

Each input image is transformed twice to produce the two distorted views for contrastive loss. The
image augmentation pipeline includes random cropping, resizing to 224x224, random horizontal
flipping, color jittering, grayscale, Gaussian blurring, and solarization.


D.2 NETWORK

Throughout the ImageNet experiments in this paper, we use a ResNet-50 (He et al., 2016) as an
encoder. This network has an output of dimension 2048, which is called a representation vector.


D.3 OPTIMIZATION

We use a LARS optimizer and train all models for 100 epochs. The batch size is 4096, which fits
into 32 GPUs during training. The learning rate is 4.8 as in SimCLR (Chen et al., 2020a), which
goes through a 10 epoch of warming up and then a cosine decay schedule.


E HYPERPARAMETER TUNING ON d0

Here, we list the ImageNet accuracy with various d0 value in Figure 12. It’s easy to see that when
_d0_ 0, there’s too little gradient information coming from the loss, the performance drops. When
_d0 →_ 2048, the model converges to standard SimCLR without a projector, which we know suffers
from dimensional collapse in representation space. →


-----

Top-1 ImageNet Accuracy

62

60

58

Accuracy (%) 56

54

52

0 250 500 750 1000 1250 1500 1750 2000

d_0


Figure 12: Hyperparameter tuning on d0 based on ImageNet linear probe Top-1 accuracy.

F ABLATION STUDY DETAIL

**Fixed low-rank projector vs Fixed low-rank diagonal projector: DirectCLR is equivalent to**
SimCLR with a fixed low-rank diagoanl projector. It performs the same as a SimCLR with fixed
low-rank projector, which achieves 62.3% linear probe accuracy. Specifically, the singular values of
this low-rank matrix are set to have d0 numbers of 1 and 0 for the rest, then left- and right- multiply
a fixed orthogonal matrix. Therefore, their only difference is that this fixed projector has an extra
fixed orthogonal matrix in between.

**Trainable projector vs trainable diagonal projector: We trained a SimCLR model with a trainable**
projector that is constrained be diagonal. The model achieves 60.2% linear probe accuracy on
ImageNet, which is close to a SimCLR with a 1-layer linear projector.

**Orthogonal projector vs no projector: We train a single layer projector SimCLR model with**
orthogonal constraint using ExpM parametrization (Casado & Mart´ınez-Rubio, 2019). Therefore,
the projector weight matrix has all singular values fixed to be 1. This model reaches 52.2% accuracy
on ImageNet which is close to a SimCLR without projector.

These ablation studies verify the propostion 1 that the SimCLR projector only needs to be diagonal. Also, according to Table 2, we find that low-rank projector setting consistently improves the
performance, which verifies proposition 2.

**Linear probe on subvector instead of the entire vector: For DirectCLR, we perform a linear**
probe only on the sub-vector z and get 47.9% accuracy on ImageNet. This shows that the rest of r
still contains useful information even though it does not see gradient directly coming from the loss
function.

**Random dropout instead of fixed subvector: Since DirectCLR drops out a number of dimensions**
for the loss function, it would be natural to ask whether random dropping out can reach the same
performance. We train a SimCLR model without a projector and randomly feed d0 number of
features to InfoNCE loss every iteration. This model reaches only 43.0% accuracy on ImageNet.
This demonstrates the importance of applying a fixed subvector, which allows the alignment effect
to happen.


-----

