# LEARNING TOWARDS THE LARGEST MARGINS

**Xiong Zhou[1][∗], Xianming Liu[1][,][2][†], Deming Zhai[1], Junjun Jiang[1][,][2], Xin Gao[3][,][2][,][4], Xiangyang Ji[5]**

1Harbin Institute of Technology 2Peng Cheng Laboratory 3King Abdullah University of Science and Technology
4Gaoling School of Artificial Intelligence, Renmin University of China 5Tsinghua University

ABSTRACT

One of the main challenges for feature representation in deep learning-based classification is the design of appropriate loss functions that exhibit strong discriminative power. The classical softmax loss does not explicitly encourage discriminative
learning of features. A popular direction of research is to incorporate margins in
well-established losses in order to enforce extra intra-class compactness and interclass separability, which, however, were developed through heuristic means, as
opposed to rigorous mathematical principles. In this work, we attempt to address
this limitation by formulating the principled optimization objective as learning
_towards the largest margins. Specifically, we firstly define the class margin as_
the measure of inter-class separability, and the sample margin as the measure of
intra-class compactness. Accordingly, to encourage discriminative representation
of features, the loss function should promote the largest possible margins for both
classes and samples. Furthermore, we derive a generalized margin softmax loss
to draw general conclusions for the existing margin-based losses. Not only does
this principled framework offer new perspectives to understand and interpret existing margin-based losses, but it also provides new insights that can guide the
design of new tools, including sample margin regularization and largest margin
_softmax loss for the class-balanced case, and zero-centroid regularization for the_
class-imbalanced case. Experimental results demonstrate the effectiveness of our
strategy on a variety of tasks, including visual classification, imbalanced classification, person re-identification, and face verification.

1 INTRODUCTION

Recent years have witnessed the great success of deep neural networks (DNNs) in a variety of tasks,
especially for visual classification (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al.,
2016; Howard et al., 2017; Zoph et al., 2018; Touvron et al., 2019; Brock et al., 2021; Dosovitskiy
et al., 2021). The improvement in accuracy is attributed not only to the use of DNNs, but also to the
elaborated losses encouraging well-separated features (Elsayed et al., 2018; Musgrave et al., 2020).

In general, the loss is expected to promote the learned features to have maximized intra-class compactness and inter-class separability simultaneously, so as to boost the feature discriminativeness.
Softmax loss, which is the combination of a linear layer, a softmax function, and cross-entropy loss,
is the most commonly-used ingredient in deep learning-based classification. However, the softmax
loss only learns separable features that are not discriminative enough (Liu et al., 2017). To remedy the limitation of softmax loss, many variants have been proposed. Liu et al. (2016) proposed
a generalized large-margin softmax loss, which incorporates a preset constant m multiplying with
the angle between samples and the classifier weight of the ground truth class, leading to potentially
larger angular separability between learned features. SphereFace (Liu et al., 2017) further improved
the performance of L-Softmax by normalizing the prototypes in the last inner-product layer. Subsequently, Wang et al. (2017) exhibited the usefulness of feature normalization when using feature
vector dot products in the softmax function. Coincidentally, in the field of contrastive learning, Chen
et al. (2020) also showed that normalization of outputs leads to superior representations. Due to its
effectiveness, normalization on either features or prototypes or both becomes a standard procedure
in margin-based losses, such as SphereFace (Liu et al., 2017), CosFace/AM-Softmax (Wang et al.,
2018b;a) and ArcFace (Deng et al., 2019). However, there is no theoretical guarantee provided yet.

_∗This work was done as intern at Peng Cheng Laboratory._
_†Correspondence to: Xianming Liu <csxm@hit.edu.cn>_


-----

Despite their effectiveness and popularity, the existing margin-based losses were developed through
heuristic means, as opposed to rigorous mathematical principles, modeling and analysis. Although
they offer geometric interpretations, which are helpful to understand the underlying intuition, the
theoretical explanation and analysis that can guide the design and optimization is still vague. Some
critical issues are unclear, e.g., why is the normalization of features and prototypes necessary? How
can the loss be further improved or adapted to new tasks? Therefore, it naturally raises a fundamental
question: how to develop a principled mathematical framework for better understanding and design
of margin-based loss functions? The goal of this work is to address these questions by formulating
the objective as learning towards the largest margins and offering rigorously theoretical analysis as
well as extensive empirical results to support this point.

To obtain an optimizable objective, firstly, we should define measures of intra-class compactness and
inter-class separability. To this end, we propose to employ the class margin as the measure of interclass separability, which is defined as the minimal pairwise angle distance between prototypes that
reflects the angular margin of the two closest prototypes. Moreover, we define the sample margin
following the classic approach in (Koltchinskii et al., 2002, Sec. 5), which denotes the similarity
difference of a sample to the prototype of the class it belongs to and to the nearest prototype of other
classes and thus measures the intra-class compactness. We provide a rigorous theoretical guarantee
that maximizing the minimal sample margin over the entire dataset leads to maximizing the class
margin regardless of feature dimension, class number, and class balancedness. It denotes that the
sample margin also has the power of measuring inter-class separability.

According to the defined measures, we can obtain categorical discriminativeness of features by
the loss function promoting the largest margins for both classes and samples, which also meets to
tighten the margin-based generalization bound in (Kakade et al., 2008; Cao et al., 2019). The main
contributions of this work are highlighted as follows:

-  For a better understanding of margin-based losses, we provide a rigorous analysis about the
necessity of normalization on prototypes and features. Moreover, we propose a generalized
margin softmax loss (GM-Softmax), which can be derived to cover most of existing marginbased losses. We prove that, for the class-balance case, learning with the GM-Softmax loss
leads to maximizing both class margin and sample margin under mild conditions.

-  We show that learning with existing margin-based loss functions, such as SphereFace,
NormFace, CosFace, AM-Softmax and ArcFace, would share the same optimal solution.
In other words, all of them attempt to learn towards the largest margins, even though they
are tailored to obtain different desired margins with explicit decision boundaries. However,
these losses do not always maximize margins under different hyper-parameter settings. Instead, we propose an explicit sample margin regularization term and a novel largest margin
_softmax loss (LM-Softmax) derived from the minimal sample margin, which significantly_
improve the class margin and the sample margin.

-  We consider the class-imbalanced case, in which the margins are severely affected. We
provide a sufficient condition, which reveals that, if the centroid of prototypes is equal
to zero, learning with GM-Softmax will provide the largest margins. Accordingly, we
propose a simple but effective zero-centroid regularization term, which can be combined
with commonly-used losses to mitigate class imbalance.

-  Extensive experimental results are offered to demonstrate that the strategy of learning
towards the largest margins significantly can improve the performance in accuracy and
class/sample margins for various tasks, including visual classification, imbalanced classification, person re-identification, and face verification.

2 MEASURES OF INTRA-CLASS COMPACTNESS AND INTER-CLASS
SEPARABILITY

With a labeled dataset D = {(xi, yi)}i[N]=1 [(where][ x][i][ denotes a training example with label][ y][i][, and]
_yi_ [1, k] = 1, 2, ..., k ), the softmax loss for a k-classification problem is formulated as
_∈_ _{_ _}_

_N_ exp(wy[T]i **_[z][i][)]_** _N_ exp( **_wyi_** 2 **_zi_** 2 cos(θiyi ))

_L = N[1]_ _−_ log _k_ = _−_ log _k_ _∥_ _∥_ _∥_ _∥_ _,_ (2.1)

_i=1_ _j=1_ [exp(][w]j[T][z][i][)] _i=1_ _j=1_ [exp(][∥][w][j][∥][2][∥][z][i][∥][2][ cos(][θ][ij][))]

X X

P P


-----

where zi = φΘ(xi) ∈ R[d] (usually k ≤ _d + 1) is the learned feature representation vector; φΘ_
denotes the feature extraction sub-network; W = (w1, ..., wk) ∈ R[d][×][k] denotes the linear classifier
which is implemented with a linear layer at the end of the network (some works omit the bias and use
an inner-product layer); θij denotes the angle between zi and wj; and ∥· ∥2 denotes the Euclidean
norm, where w1, ..., wk can be regarded as the class centers or prototypes (Mettes et al., 2019). For
simplicity, we use prototypes to denote the weight vectors in the last inner-product layer.

The softmax loss intuitively encourages the learned feature representation zi to be similar to the
corresponding prototype wyi, while pushing zi away from the other prototypes. Recently, some
works (Liu et al., 2016; 2017; Deng et al., 2019) aim to achieve better performance by modifying
the softmax loss with explicit decision boundaries to enforce extra intra-class compactness and interclass separability. However, they do not provide the theoretical explanation and analysis about
the newly designed losses. In this paper, we claim that a loss function to obtain better inter-class
separability and intra-class compactness should learn towards the largest class and sample margin,
and offer rigorously theoretical analysis as support. All proofs can be found in the Appendix A.

In the following, we define class margin and sample margin as the measures of inter-class separability and intra-class compactness, respectively, which serve as the base for our further derivation.

2.1 CLASS MARGIN

With prototypes w1, ..., wk ∈ R[d], the class margin is defined as the minimal pairwise angle distance:

**_wi[T][w][j]_**
_mc(_ **_wi_** _i=1[) = min]_ max _,_ (2.2)
_{_ _}[k]_ _i≠_ _j_ [∠][(][w][i][,][ w][j][) = arccos]  _i≠_ _j_ _∥wi∥2∥wj∥2_ 


where ∠(wi, wj) denotes the angle between the vec- 1.3 MNIST CIFAR-10 CIFAR-100
tors wi and wj. Note that we omit the magnitudes of
the prototypes in the definition, since the magnitudes
tend to be very close according to the symmetry prop- 1.225
erty. To verify this, we compute the ratio between the
maximum and minimum magnitudes, which tends to 1.15
be close to 1 on different datasets, as shown in Fig. 1.

To obtain better inter-class separability, we seek the 1.075
largest class margin, which can be formulated as

1

max _mc(_ **_wi_** _i=1[) =]_ max min 1 22 43 64 85 106 127 148 169 190
_{wi}i[k]=1_ _{_ _}[k]_ _{wi}i[k]=1_ _i≠_ _j_ [∠][(][w][i][,][ w][j][)][.] Epochs

Since magnitudes do not affect the solution of the max
Figure 1: The curves of ratio between max
min problem, we perform ℓ2 normalization for each wi

imum and minimum magnitudes of prototypes

to effectively restrict the prototypes on the unit sphere

on MNIST and CIFAR-10/-100 using the CE

S[d][−][1] with center at the origin. Under this constraint, loss. The ratio is roughly close to 1 (< 1.3).
the maximization of the class margin is equivalent to
the configuration of k points on S[d][−][1] to maximize their minimum pairwise distance:

arg max arg max (2.3)
_{wi}i[k]=1[⊂][S][d][−][1][ min]i≠_ _j_ [∠][(][w][i][,][ w][j][) =] _{w}i[k]=1[⊂][S][d][−][1][ min]i≠_ _j_ _[∥][w][i][ −]_ **_[w][j][∥][2][,]_**

The right-hand side is well known as the k-points best-packing problem on spheres (often called
the Tammes problem), whose solution leads to the optimal separation of points (Borodachov et al.,
2019). The best-packing problem turns out to be the limiting case of the minimal Riesz energy:


1

= arg max (2.4)
_∥wi −_ **_wj∥2[t]_** _{w}i[k]=1[⊂][S][d][−][1][ min]i≠_ _j_ _[∥][w][i][ −]_ **_[w][j][∥][2][.]_**


arg min
_t_
**_w_** _i=1[⊂][S][d][−][1][ lim]→∞_
_{_ _}[k]_


_i≠_ _j_


Interestingly, Liu et al. (2018) utilized the minimum hyperspherical energy as a generic regularization for neural networks to reduce undesired representation redundancy. Whenk ≤ _d + 1, and t > 0, the solution of the best-packing problem leads to the minimal Riesz w1, ..., wk t ∈-energy:S[d][−][1],_
_RieszLemma 2.1. t-energy and For any k-points best-packing configurations are uniquely given by the vertices of regular w1, ..., wk ∈_ S[d][−][1], d ≥ 2, and 2 ≤ _k ≤_ _d + 1, the solution of minimal_
(k − 1)-simplices inscribed in S[d][−][1]. Furthermore, wi[T][w][j][ =] _k−−11_ _[,][ ∀][i][ ̸][=][ j][.]_


-----

This lemma shows that the maximum of mc({wi}i[k]=1[)][ is][ arccos(][ −]k−[1]1 [)][ when][ k][ ≤] _[d][ + 1][, which]_

is analytical and can be constructed artificially. However, when k > d + 1, the optimal k-point
configurations on the sphere S[d][−][1] have no generic analytical solution, and are only known explicitly
for a handful of cases, even for d = 3.

2.2 SAMPLE MARGIN

According to the definition in (Koltchinskii et al., 2002), for the network f (x; Θ, W ) =
_W_ [T]φΘ(x) : R[m] _→_ R[k] that outputs k logits, the sample margin for (x, y) is defined as

_γ(x, y) = f_ (x)y max _y_ **_[z][ −]_** [max] _j_ **_[z][,]_** (2.5)
_−_ _j≠_ _y_ **_[f]_** [(][x][)][j][ =][ w][T] _j≠_ _y_ **_[w][T]_**

where z = φΘ(x) denotes the corresponding feature. Let nj be the number of samples in class j
and Sj = {i : yi = j} denote the sample indices corresponding to class j. We can further define the
sample margin for samples in class j as

_γj = min_ (2.6)
_i_ _Sj_ _[γ][(][x][i][, y][i][)][.]_
_∈_

Accordingly, the minimal sample margin over the entire dataset is γmin = min{γ1, ..., γk}. Intuitively, learning features and prototypes to maximize the minimum of all sample margins means
making the feature embeddings close to their corresponding classes and far away from the others:
**Theorem 2.2. For w1, ..., wk, z1, ..., zN ∈** S[d][−][1] _(where nj > 0 for each j ∈_ [1, k]), the optimal
_solution {wi[∗][}]i[k]=1[,][ {][z]i[∗][}]i[N]=1_ [= arg max]{wi}i[k]=1[,][{][z][i][}]i[N]=1w[∗][γ][min][ is obtained if and only if][ {][w]i[∗][}]i[k]=1 _[max-]_

_yi_ _[−][w]yi[∗]_
_imizes the class margin mc({wi}i[k]=1[)][, and][ z]i[∗]_ [=] _∥wyi[∗]_ _[−][w][∗]yi_ _[∥][2]_ _[, where][ w]y[∗]i_ _[denotes the centroid of]_

_the vectors {wj : j maximizes wy[T]i_ **_[w][j][, j][ ̸][=][ y][i][}][.]_**

As shown in the proof A, Theorem 2.2 guarantees that maximizing γmin will provide the solution
of the Tammes problem with respect to any feature dimension d, class number k, and both classbalanced and class-imbalance cases. When 2 ≤ _k ≤_ _d + 1, we can derive the following proposition:_
**Proposition 2.3. For any w1, ..., wk, z1, ..., zN** S[d][−][1], d 2, and 2 _k_ _d + 1, the maximum_
_of γmin is_ _k_ _k_ 1 _[, which is obtained if and only if][ ∀] ∈[i][ ̸][=][ j][,][ w]i[T] ≥[w][j][ =][ −]_ _k_ 1 ≤1 _[, and] ≤[ z][i][ =][ w][y][i]_ _[.]_

_−_ _−_

Theorem 2.2 and Proposition 2.3 show that the best separation of prototypes is obtained when maximizing the minimal sample margin γmin.

On the other hand, let Lγ,j[f ] = Prx∼Pj [maxj′≠ _j f_ (x)j′ > f (x)j − _γ] denote the hard margin_
loss on samples from class j, and _Lγ,j denote its empirical variant. When the training dataset is_
separable (which indicates that there exists f such that γmin > 0), Cao et al. (2019) provided a finegrained generalization error bound under the setting with balanced test distribution by considering

[b]
the margin of each class, i.e., for γj > 0 and all f ∈F, with a high probability, we have


Pr(x,y)[f (x)y < max
_l≠_ _y_ _[f]_ [(][x][)][l][]][ ≤] _k[1]_


_Lγj_ _,j[f_ ] + γ[4]j Rj(F) + εj(γj) _._ (2.7)



b b


_j=1_


In the right-hand side, the empirical Rademacher complexity _γ4j_ Rj( ) has a big impact. From the

_F_
perspective of our work, a straightforward way to tighten the generalization bound is to enlarge the
minimal sample margin γmin, which further leads to the larger margin γj for each class j.

[b]

3 LEARNING TOWARDS THE LARGEST MARGINS

3.1 CLASS-BALANCED CASE

According to the above derivations, to encourage discriminative representation of features, the loss
function should promote the largest possible margins for both classes and samples. In (Mettes
et al., 2019), the pre-defined prototypes positioned through data-independent optimization are used
to obtain a large class margin. As shown in Figure 2, although they keep the particularly large margin


-----

(a) Test Accuracy (b) Class Margin (c) Sample Margin

Figure 2: Test accuracies, class margins and sample margins on CIFAR-10 and CIFAR-100 with
and without fixed prototypes, where fixed prototypes are pre-trained for very large class margins.

from the beginning, the sample margin is smaller than that optimized without fixed prototypes,
leading to insignificant improvements in accuracy.

In recent years, in the design of variants of softmax loss, one popular approach (Bojanowski &
Joulin, 2017; Wang et al., 2017; Mettes et al., 2019; Wang & Isola, 2020) is to perform normalization on prototypes or/and features, leading to superior performance than unnormalized counterparts
(Parkhi et al., 2015; Schroff et al., 2015; Liu et al., 2017). However, there is no theoretical guarantee
provided yet. In the following, we provide a rigorous analysis about the necessity of normalization.
Firstly, we prove that minimizing the original softmax loss without normalization for both features
and prototypes may result in a very small class margin:

**Theorem 3.1. ∀ε ∈** (0, π/2], if the range of w1, ..., wk or z1, ..., zN is R[d] _(2 ≤_ _k ≤_ _d + 1), then_
_there exists prototypes that achieve the infimum of the softmax loss and have the class margin ε._

This theorem reveals that, unless both features and prototypes are normalized, the original softmax
loss may produce an arbitrary small class margin ε. As a corroboration of this conclusion, L-Softmax
(Liu et al., 2016) and A-Softmax (Liu et al., 2017) that do not perform any normalization or only do
on prototypes, cannot guarantee to maximize the class margin. To remedy this issue, some works
(Wang et al., 2017; 2018a;b; Deng et al., 2019) proposed to normalize both features and prototypes.

A unified framework (Deng et al., 2019) that covers A-Softmax (Liu et al., 2017) with feature normalization, NormFace (Wang et al., 2017), CosFace/AM-Softmax (Wang et al., 2018b;a), ArcFace
(Deng et al., 2019) as special cases can be formulated with hyper-parameters m1, m2 and m3:

exp(s(cos(m1θiyi + m2) _m3))_
_L[′]i_ [=][ −] [log] _−_ (3.1)

exp(s(cos(m1θiyi + m2) _m3)) +_ _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_
_−_ _̸_

where θij = ∠(wj, zi). The hyper-parameters setting usually guarantees that cos(m1θiyi + m2)

[P] _−_

_βm =3 ≤ −mcos3 < m 02 cos, then we have θiyi −_ _m3, and m2 is usually set to satisfy cos m2 ≥_ 2[1] [. Let][ α][ = cos][ m][2][ and]

exp(s(α cos θiyi + β))
_L[′]i_ _[≥−]_ [log] exp(s(α cos θiyi + β)) + _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_ (3.2)

_̸_

which indicates that the existing well-designed normalized softmax loss functions are all considered

[P]

as the upper bound of the right-hand side, and the equality holds if and only if θiyi = 0.

**Generalized Margin Softmax Loss. Based on the right-hand side of (3.2), we can derive a more**
general formulation, called Generalized Margin Softmax (GM-Softmax) loss:

exp(s(αi1 cos θiyi + βi1))
_Li =_ log (3.3)
_−_ exp(s(αi2 cos θiyi + βi2)) + _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_

_̸_

where αi1, αi2, βi1 and βi2 are hyper-parameters to handle the margins in training, which are set

[P]

_s >specifically for each sample instead of the same in ( 0, βi1, βi2 ∈_ R. For class-balanced case, each sample is treated equally, thus setting3.2). We also require that αi1 ≥ [1]2 [,][ α] α[i][2]i1[ ≤] = α[α][i]1[1],[,]


-----

_αi2 = α2, βi1 = β1 and βi2 = β2, ∀i. For class-imbalanced case, the setting relies on the data_
distribution, e.g., the LDAM loss (Cao et al., 2019) achieves the trade-off of margins with αi1 =
_αi2 = 1 and βi1 = βi2 = −Cn[−]yi[1][/][4]. It is worth noting that we merely use the GM-Softmax loss as_
a theoretical formulation and will derive a more efficient form for the practical implementation.

Wang et al. (2017) provided a lower bound for normalized softmax loss, which relies on the assumption that all samples are well-separated, i.e., each sample’s feature is exactly the same as its
corresponding prototype. However, this assumption could be invalid during training, e.g., for binary
classification, the best feature of the first class z obtained by minimizing − log exp(swexp(1[T][z][)+exp(]sw1[T][z][)][s][w]2[T][z][)]

is _∥ww11−−ww22∥2_ [rather than][ w][1][. In the following, we provide a more general theorem, which does not]

mizing the GM-Softmax loss will maximize both class margin and sample margin.rely on such a strong assumption. Moreover, we prove that the solutions {wj[∗][}]j[k]=1[,][ {][z]i[∗][}]i[N]=1 [mini-]
_dTheorem 3.2. + 1, learning with GM-Softmax (where For class-balanced datasets, αi w1 =1, ..., α1 w, αki,2 z =1, ..., α2 z, βNi ∈1 =S β[d][−]1 and[1], d ≥ βi22 =, and β2 2) leads to ≤_ _k ≤_
_maximizing both class margin and sample margin._

As can be seen, for anyproduces the same optimal solution or leads to α1 ≥ [1]2 [,][ α][2][ ≤] _[α][1][,][ s >][ 0] neural collapse[, and][ β][1][, β][ ∈]_ [R] ([, minimizing the GM-Softmax loss]Papyan et al., 2020)), even though

they are intuitively designed to obtain different decision boundaries. Moreover, we have
_kProposition 3.3. ≤_ _d + 1, learning with the loss functions A-Softmax ( For class-balanced datasets, w1, ...,Liu et al. wk, z1, ...,, 2017 zN ∈) with feature normalization,S[d][−][1], d ≥_ 2, and 2 ≤
_NormFace (Wang et al., 2017), CosFace (Wang et al., 2018b) or AM-Softmax (Wang et al., 2018a),_
_and ArcFace (Deng et al., 2019) share the same optimal solution._

Although these losses theoretically share the same optimal solution, in practice they usually meet
sub-optimal solutions under different hyper-parameter settings when optimizing a neural network,
which is demonstrated in Table 1. Moreover, these losses are complicated and possibly redundantly
designed, leading to difficulties in practical implementation. Instead, we suggest a concise and easily
implemented regularization term and a loss function in the following.

**Sample Margin Regularization. In order to encourage learning towards the largest margins, we**
may explicitly leverage the sample margin (2.5) as the loss, which is defined as:

_Rsm(x, y) =_ (wy[T][z][ −] [max] _j_ **_[z][)][.]_** (3.4)
_−_ _j=y_ **_[w][T]_**
_̸_

Noticeably, the empirical risk _N1_ _Ni=1_ _[R][sm][(][x][i][, y][i][)][ is a lower-bounded surrogate of][ −][γ][min][,][ i.e.][,]_

_−networks. Whenγmin ≥_ _N1_ PNi=1 k ≤[R][sm]d+1[(][x][i], learning with[, y][i][)][, while directly minimizing]P _Rsm will promote the learning towards the largest margins:[ −][γ][min][ is too difficult to optimize neural]_

_dTheorem 3.4. + 1, learning with For class-balanced datasets, Rsm leads to the maximization of both class margin and sample margin. w1, ..., wk, z1, ..., zN ∈_ S[d][−][1], d ≥ 2, and 2 ≤ _k ≤_

Although learning with Rsm theoretically achieves the largest margins, in practical implementation,
the optimization by the gradient-based methods shows unstable and non-convergent results for large
scale datasets. Alternatively, we turn to combine Rsm as a regularization or complementary term
with commonly-used losses, which is referred to as sample margin regularization. The empirical
results demonstrate its superiority in learning towards the large margins, as depicted in Table 1.

**Largest Margin Softmax Loss (LM-Softmax). Theorem 2.2 provides a theoretical guarantee that**
maximizing γmin will obtain the maximum of class margin regardless of feature dimension, class
number, and class balancedness. It offers a straightforward approach to meet our purpose, i.e.,
learning towards the largest margins. However, directly maximizing γmin is difficult to optimize a
neural network with only one sample margin. As a consequence, we introduce a surrogate loss for
balanced datasets, which is called the Largest Margin Softmax (LM-Softmax) loss:

exp(swy[T][z][)]

_L(x, y; s) =_ exp(s(wj **_wy)[T]z)_** (3.5)
_−_ [1]s [log] _j≠_ _y_ [exp(][s][w]j[T][z][) = 1]s [log] Xj≠ _y_ _−_

which is derived by the limiting case of theP logsumexp operator, i.e.. we have _γmin_ =
_−_
lims→∞ [1]s [log(][P]i[N]=1 _j≠_ _yi_ [exp(][s][(][w]j[T][z][i][ −] **_[w]y[T]i_** **_[z][i][)))][. Moreover, since][ log][ is strictly concave, we]_**

P


-----

can derive the following inequality


_N_ _N_

1

exp(s(wj[T][z][i] _yi_ **_[z][i][)))][ ≥]_** [1] _L(xi, yi; s) + [1]_ (3.6)

_s_ [log(]Xi=1 _jX≠_ _yi_ _[−]_ **_[w][T]_** _N_ Xi=1 _s_ [log][ N.]

Minimizing the right-hand side of (3.6) usually leads to that _j=yi_ [exp(][s][(][w]j[T][z][ −] **_[w]y[T]i_** **_[z][))][ is a]_**

_̸_
constant, while the equality of (3.6) holds if and only if _j=yi_ [exp(][s][(][w]j[T][z][ −] **_[w]y[T]i_** **_[z][))][ is a constant.]_**

_̸_
Thus, we can achieve the maximum of γmin by minimizing L(x[P], y; s) defined in (3.5).

It can be found that, LM-Softmax can be regarded as a special case of the GM-Softmax loss when[P] _α2_
or β2 approaches −∞, which can be more efficiently implemented than the GM-Softmax loss. With
respect to the original softmax loss, LM-Softmax removes the term exp(swy[T][z][)][ in the denominator.]

3.2 CLASS-IMBALANCED CASE


Class imbalance is ubiquitous and inherent in real-world classification problems (Buda et al., 2018;
Liu et al., 2019). However, the performance of deep learning-based classification would drop significantly when the training dataset suffers from heavy class imbalance effect. According to (2.7),
enlarging the sample margin can tighten the upper bound in case of class imbalance. To learn towards
the largest margins on class-imbalanced datasets, we provide the following sufficient condition:
**Theorem 3.5. For class-balanced or -imbalanced datasets, w1, ..., wk, z1, ..., zN ∈** S[d][−][1], d ≥ 2,
_and 2 ≤_ _k ≤_ _d + 1, if_ _i=1_ **_[w][i][ = 0][, learning with GM-Softmax in (][3.3][) leads to maximizing both]_**
_class margin and sample margin._

[P][K]

This theorem reveals that, if the centroid of prototypes is equal to zero, learning with GM-Softmax
will provide the largest margins.

**Zero-centroid Regularization. As a consequence, we propose a straight regularization term as**
follows, which can be combined with commonly-used losses to remedy the class imbalance effect:


_Rw{wj}j[k]=1_ [=][ λ]


**_wj_** 2[.] (3.7)
_j=1_

X

[2]


The zero-centroid regularization is only applied to prototypes at the last inner-product layer.

4 EXPERIMENTS

In this section, we provide extensive experimental results to show superiority of our method on a
variety of tasks, including visual classification, imbalanced classification, person ReID, and face
verification. More experimental analysis and implementation details can be found in the appendix.

4.1 VISUAL CLASSIFICATION

To verify the effectiveness of the proposed sample margin regularization in improving inter-class
separability and intra-class compactness, we conduct experiments of classification on balanced
datasets MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009).
We evaluate performance with three metrics: 1) top-1 validation accuracy acc; 2) the class margin
_mcls defined in (2.2); 3) the average of sample margins msamp. We use a 4-layer CNN, ResNet-18,_
and ResNet-34 on MNIST, CIFAR-10, and CIFAR-100, respectively. Moreover, some commonlyused neural units are considered, such as ReLU, BatchNorm, and cosine learning rate annealing.
We use CE, CosFace, ArcFace, NormFace as the compared baseline methods. Note that CosFace,
ArcFace, NormFace have one identical hyper-parameter s, which is used for comprehensive study.

**Results. As shown in Table 1, all baseline losses fail in learning with large margins for all s, in which**
the class margin decreases as s increases. There is no significant performance difference among
them. In contrast, by coupling with the proposed sample margin regularization Rsm, the losses turn
to have larger margins. The results demonstrate that the proposed sample margin regularization is
really beneficial to learn towards the possible largest margins. Moreover, the enlargement on class
margin and sample margin means better inter-class separability and intra-class compactness, which
further brings the improvement of classification accuracy in most cases.


-----

Table 1: Test accuracies (acc), class margins (mcls) and sample margins (msamp) on MNIST, CIFAR-10 and
CIFAR-100 using loss functions with/without Rsm in (3.4). The results with positive gains are highlighted.

|Dataset|MNIST|CIFAR-10|CIFAR-100|
|---|---|---|---|


|Metric|acc mcls msamp|acc mcls msamp|acc mcls msamp|
|---|---|---|---|


|CE CE + 0.5Rsm|99.11 87.39◦ 0.5014 99.13 95.41◦ 1.026|94.12 81.73◦ 0.6203 94.45 96.31◦ 0.9744|74.56 65.38◦ 0.1612 74.96 90.00◦ 0.4955|
|---|---|---|---|


|CosFace (s = 10) CosFace (s = 20) CosFace (s = 64)|98.98 95.93◦ 0.9839 99.06 93.24◦ 0.8376 99.25 89.50◦ 0.7581|94.39 96.00◦ 0.9168 94.13 91.22◦ 0.7955 93.53 64.14◦ 0.6969|74.44 83.31◦ 0.4578 73.26 79.17◦ 0.3078 73.87 72.56◦ 0.2233|
|---|---|---|---|


|CosFace (s = 10) + 0.5Rsm CosFace (s = 20) + 0.5Rsm CosFace (s = 64) + 0.5Rsm|99.16 95.56◦ 1.033 99.24 95.41◦ 1.030 99.27 95.35◦ 1.019|94.42 96.26◦ 0.9675 94.27 96.18◦ 0.9490 94.20 95.48◦ 0.9075|73.76 90.21◦ 0.5089 74.41 89.02◦ 0.4780 74.53 85.31◦ 0.3817|
|---|---|---|---|


|ArcFace (s = 10) ArcFace (s = 20) ArcFace (s = 64)|99.05 94.64◦ 0.8225 99.11 90.84◦ 0.6091 99.21 82.63◦ 0.4038|94.50 91.23◦ 0.8501 94.11 53.98◦ 0.5707 − − −|73.96 76.91◦ 0.4313 74.74 60.91◦ 0.3010 − − −|
|---|---|---|---|


|ArcFace (s = 10) + 0.5Rsm ArcFace (s = 20) + 0.5Rsm ArcFace (s = 64) + 0.5Rsm|99.14 95.42◦ 1.034 99.19 91.38◦ 1.030 99.14 95.29◦ 1.019|94.21 96.27◦ 0.9651 94.32 96.15◦ 0.9571 − − −|74.47 90.13◦ 0.5143 74.64 88.73◦ 0.4804 − −|
|---|---|---|---|


|NormFace (s = 10) NormFace (s = 20) NormFace (s = 64)|99.06 94.34◦ 0.7750 99.09 89.27◦ 0.5263 99.00 82.08◦ 0.2621|94.16 94.40◦ 0.8004 94.09 74.32◦ 0.6001 94.01 36.50◦ 0.2633|74.23 79.10◦ 0.4250 73.87 77.47◦ 0.2498 73.42 52.37◦ 0.0993|
|---|---|---|---|


|NormFace (s = 10) + 0.5Rsm NormFace (s = 20) + 0.5Rsm NormFace (s = 64) + 0.5Rsm|99.16 95.38◦ 1.034 99.19 95.37◦ 1.031 99.34 95.29◦ 1.021|94.23 96.28◦ 0.9650 94.38 96.17◦ 0.9519 94.42 93.87◦ 0.9508|74.54 90.10◦ 0.5160 74.75 88.86◦ 0.4773 74.33 76.02◦ 0.3665|
|---|---|---|---|



4.2 IMBALANCED CLASSIFICATION

To verify the effectiveness of the proposed zero-centroid regularization in handling class-imbalanced
effect, we conduct experiments on imbalanced classification with two imbalance types: long-tailed
imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). The compared baseline losses
include CE, Focal Loss, NormFace, CosFace, ArcFace, and the Label-Distribution-Aware Margin
Loss (LDAM) with hyper-parameter s = 5. We follow the controllable data imbalance strategy
in (Maas et al., 2011; Cao et al., 2019) to create the imbalanced CIFAR-10/-100 by reducing the
number of training examples per class and keeping the validation set unchanged. The imbalance
ratio ρ = maxi ni/ mini ni is used to denote the ratio between sample sizes of the most frequent
and least frequent classes. We add zero-centroid regularization to the margin-based baseline losses
and the proposed LM-Softmax to verify its validity. We report the top-1 validation accuracy acc and
class margin mcls of compared methods.

Table 2: Test accuracies (acc) and class margins (mcls) on imbalanced CIFAR-10. The results with
positive gains are highlighted (where * denotes coupling with zero-centroid regularization term).

|Dataset|Imbalanced CIFAR-10|Imbalanced CIFAR-100|
|---|---|---|


|Imbalance Type|long-tailed|step|long-tailed|step|
|---|---|---|---|---|


|Imbalance Ratio|100|10|100|10|100|10|100|10|
|---|---|---|---|---|---|---|---|---|


|Metric|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|
|---|---|---|---|---|---|---|---|---|


|CE Focal|70.88 77.41◦ 66.30 74.14◦|88.17 79.63◦ 87.33 74.48◦|62.21 76.50◦ 60.55 63.31◦|85.06 82.24◦ 84.49 75.16◦|40.38 64.73◦ 38.04 54.67◦|60.42 66.24◦ 60.09 59.29◦|42.36 60.32◦ 41.90 55.98◦|56.88 62.82◦ 57.84 55.72◦|
|---|---|---|---|---|---|---|---|---|


|CosFace CosFace*|69.28 58.77◦ 69.52 91.90◦|87.02 81.61◦ 87.55 95.46◦|53.64 19.78◦ 62.49 95.86◦|84.86 75.96◦ 85.59 96.12◦|34.91 4.731◦ 40.98 80.93◦|60.60 70.81◦ 60.77 84.97◦|40.36 0.764◦ 41.17 41.59◦|47.56 8.559◦ 57.97 83.93◦|
|---|---|---|---|---|---|---|---|---|


|ArcFace ArcFace*|72.20 65.86◦ 72.23 92.30◦|89.00 85.23◦ 89.22 96.23◦|62.48 54.29◦ 64.38 93.51◦|86.32 80.51◦ 86.65 96.23◦|42.77 13.22◦ 44.68 56.60◦|63.21 67.73◦ 63.80 73.45◦|41.47 0.497◦ 44.26 32.10◦|58.89 0.369◦ 60.79 79.85◦|
|---|---|---|---|---|---|---|---|---|


|NormFace NormFace*|72.37 62.72◦ 72.07 94.95◦|89.19 82.60◦ 89.30 94.50◦|63.69 51.00◦ 64.07 93.06◦|86.37 77.82◦ 86.49 96.28◦|43.71 16.11◦ 44.25 64.85◦|63.50 71.26◦ 63.81 79.85◦|41.93 1.363◦ 44.51 36.30◦|59.85 21.32◦ 60.22 80.83◦|
|---|---|---|---|---|---|---|---|---|


|LDAM LDAM*|72.86 73.30◦ 72.86 91.75◦|88.92 88.19◦ 89.51 96.26◦|63.27 61.42◦ 64.99 96.04◦|87.04 85.21◦ 86.74 96.26◦|43.28 7.733◦ 45.23 70.96◦|63.62 73.19◦ 64.18 85.03◦|41.65 0.852◦ 44.48 43.26◦|58.32 6.085◦ 60.83 75.22◦|
|---|---|---|---|---|---|---|---|---|


|LM-Softmax LM-Softmax*|65.32 4.420◦ 73.21 92.57◦|88.69 68.91◦ 89.12 95.73◦|50.47 0.452◦ 65.91 93.84◦|86.08 52.20◦ 87.07 96.05◦|41.52 4.500◦ 45.28 69.53◦|63.26 68.31◦ 63.77 81.99◦|41.53 0.467◦ 46.23 43.15◦|55.44 1.372◦ 60.73 74.78◦|
|---|---|---|---|---|---|---|---|---|



**Results. As can be seen from Table 2, the baseline margin-based losses have small class margins,**
although their classification performances are better than CE and Focal, which largely attribute to
the normalization on feature and prototype. We can further improve their classification accuracy by


-----

enlarging their class margins through the proposed zero-centroid regularization, as demonstrated by
results in Table 2. Moreover, it can be found that the class margin of our LM-Softmax loss is fairly
low in the severely imbalanced cases, since it is tailored for balanced case. We can also achieve
significantly enlarged class margins and improved accuracy by the zero-centroid regularization.

4.3 PERSON RE-IDENTIFICATION

We conduct experiments on the task of per- Table 3: The results on Market-1501 and DukeMTMC
son re-identification. Specifically, we use for person re-identification task. The best three results are

**highlighted.**

the off-the-shelf baseline (Luo et al., 2019)

|Dataset|Market-1501|DukeMTMC|
|---|---|---|


|Method|mAP Rank1 Rank@5|mAP Rank@1 Rank@5|
|---|---|---|


|CE|82.8 92.7 97.5|73.0 83.5 93.0|
|---|---|---|


|ArcFace (s = 10) ArcFace (s = 20) ArcFace (s = 64)|67.5 84.1 92.1 79.1 90.8 96.5 80.4 92.6 97.4|37.7 58.7 72.7 61.4 78.3 88.6 67.6 83.4 91.4|
|---|---|---|


periments are conducted on Market-1501

|CosFace (s = 10) CosFace (s = 20) CosFace (s = 64)|68.0 84.9 92.7 80.5 92.0 97.1 78.7 92.0 97.1|39.3 60.6 73.1 64.2 81.3 89.7 68.2 83.1 92.5|
|---|---|---|


|NormFace (s = 10) NormFace (s = 20) NormFace (s = 64)|81.2 91.6 96.3 83.2 93.5 97.9 77.5 90.0 96.9|63.7 79.3 88.5 71.6 83.8 93.3 60.1 75.2 88.1|
|---|---|---|


|LM-Softmax (s = 10) LM-Softmax (s = 20) LM-Softmax (s = 64)|83.3 92.8 97.1 84.7 93.8 97.6 84.6 93.9 98.1|72.2 85.8 92.4 74.1 86.4 93.5 74.2 86.6 93.5|
|---|---|---|


(Rank@5) matching rate. Moreover, LM-Softmax exhibits significant robustness for different parameters, while ArcFace, CosFace, and NormFace show worse performance than ours and are more
sensitive to parameter settings.

4.4 FACE VERIFICATION
Table 4: Face verification results on IJBC-C,

We also verify our method on face verification that Age-DB30, CFP-FP and LFW. The results with
highly depends on the discriminability of feature em- positive gains are highlighted.
beddings. Following the settings in (An et al., 2020), Method IJB-C Age-DB30 CFP-FP LFW
we train the compared models on a large-scale dataset ArcFace 99.4919 98.067 97.371 99.800

|Method|IJB-C Age-DB30 CFP-FP LFW|
|---|---|

MS1MV3 (85K IDs/ 5.8M images) (Guo et al., 2016) CosFace 99.4942 98.033 97.300 99.800
and test on LFW (Huang et al., 2008), CFP-FP (Sen
|ArcFace CosFace LM-Softmax|99.4919 98.067 97.371 99.800 99.4942 98.033 97.300 99.800 99.4721 97.917 97.057 99.817|
|---|---|

gupta et al., 2016), AgeDB-30 (Moschoglou et al., ArcFace† **99.5011** **98.117** **97.400 99.817**
2017) and IJBC (Maze et al., 2018). We use ResNet34 ArcFaceCosFace‡† **99.513399.5112** **98.08398.150** **97.471 99.81797.371 99.817**
as the backbone, and train it with batch size 512 for CosFace‡ **99.5538** 97.900 **97.500 99.800**
all compared methods. The comparison study includes LM-Softmax‡ 99.5086 **98.167** **97.429 99.833**
CosFace, ArcFace, NormFace, and our LM-Softmax. _† and ‡ denotes training with Rsm and Rw, respectively._

|ArcFace† ArcFace‡ CosFace† CosFace‡ LM-Softmax‡|99.5011 98.117 97.400 99.817 99.5133 98.083 97.471 99.817 99.5112 98.150 97.371 99.817 99.5538 97.900 97.500 99.800 99.5086 98.167 97.429 99.833|
|---|---|

As shown in Table 4, Rsm (sample margin regularization) and Rw (zero-centroid regularization) can
improve the performance of these baselines in most cases. Moreover, it is worth noting that the
results of LM-Softmax are slightly worse than ArcFace and CosFace, which is due to that in these
large-scale datasets there exists class imbalanced effect more or less. We can alleviate this issue by
adding Rw, which can improve the performance further.


5 CONCLUSION

In this paper, we attempted to develop a principled mathematical framework for better understanding
and design of margin-based loss functions, in contrast to the existing ones that are designed heuristically. Specifically, based on the class and sample margins, which are employed as measures of
intra-class compactness and inter-class separability, we formulate the objective as learning towards
the largest margins, and offer rigorously theoretical analysis as support. Following this principle,
for class-balance case, we propose an explicit sample margin regularization term and a novel largest
margin softmax loss; for class-imbalance case, we propose a simple but effective zero-centroid regularization term. Extensive experimental results demonstrate that the proposed strategy significantly
improves the performance in accuracy and margins on various tasks.


-----

**Acknowledgements.** This work was supported by National Key Research and Development
Project under Grant 2019YFE0109600, National Natural Science Foundation of China under Grants
61922027, 6207115 and 61932022.

REFERENCES

Xiang An, Xuhan Zhu, Yang Xiao, Lan Wu, Ming Zhang, Yuan Gao, Bin Qin, Debing Zhang, and
Fu Ying. Partial fc: Training 10 million identities on a single machine. In Arxiv 2010.05222,
2020.

Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International
_Conference on Machine Learning, pp. 517–526. PMLR, 2017._

Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets.
Springer, 2019.

Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale
image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021.

Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance
problem in convolutional neural networks. Neural Networks, 106:249–259, 2018.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced
datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing
_Systems, 2019._

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning,
pp. 1597–1607. PMLR, 2020.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on
effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision
_and Pattern Recognition, pp. 9268–9277, 2019._

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin
loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision
_and Pattern Recognition, pp. 4690–4699, 2019._

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at
scale. In International Conference on Learning Representations, 2021.

Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large
margin deep networks for classification. Advances in neural information processing systems, 31,
2018.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In
_Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp._
315–323. JMLR Workshop and Conference Proceedings, 2011.

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset
and benchmark for large-scale face recognition. In European conference on computer vision, pp.
87–102. Springer, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand,
Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for
mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.


-----

Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled faces in the wild:
A database forstudying face recognition in unconstrained environments. In Workshop on faces
_in’Real-Life’Images: detection, alignment, and recognition, 2008._

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In International conference on machine learning, pp. 448–456.
PMLR, 2015.

Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: risk
bounds, margin bounds, and regularization. In Proceedings of the 21st International Conference
_on Neural Information Processing Systems, pp. 793–800, 2008._

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth
_International Conference on Learning Representations (ICLR), 2020._

Vladimir Koltchinskii, Dmitry Panchenko, et al. Empirical margin distributions and bounding the
generalization error of combined classifiers. Annals of statistics, 30(1):1–50, 2002.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer
_Science Department, University of Toronto, Tech. Rep, 1, 01 2009._

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to
document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference on International
_Conference on Machine Learning-Volume 48, pp. 507–516, 2016._

Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep
hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer
_vision and pattern recognition, pp. 212–220, 2017._

Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning
towards minimum hyperspherical energy. Advances in Neural Information Processing Systems,
31:6222–6233, 2018.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Largescale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on
_Computer Vision and Pattern Recognition, pp. 2537–2546, 2019._

I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR 2017
_(5th International Conference on Learning Representations), 2016._

Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and a strong baseline
for deep person re-identification. In Proceedings of the IEEE/CVF Conference on Computer
_Vision and Pattern Recognition Workshops, pp. 0–0, 2019._

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts.
Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the
_association for computational linguistics: Human language technologies, pp. 142–150, 2011._

Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K
Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. Iarpa janus benchmark-c: Face
dataset and protocol. In 2018 International Conference on Biometrics (ICB), pp. 158–165. IEEE,
2018.

Pascal Mettes, Elise van der Pol, and Cees G M Snoek. Hyperspherical prototype networks. In
_Advances in Neural Information Processing Systems, 2019._

Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia,
and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Pro_ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp._
51–59, 2017.


-----

Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European
_Conference on Computer Vision, pp. 681–699. Springer, 2020._

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal
phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652–24663, 2020.

Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Ma_chine Vision Association, 2015._

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. arXiv preprint arXiv:1912.01703, 2019.

Ergys Ristani, Francesco Solera, Roger S. Zou, R. Cucchiara, and Carlo Tomasi. Performance
measures and a data set for multi-target, multi-camera tracking. In ECCV Workshops, 2016.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern
_recognition, pp. 815–823, 2015._

Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and
David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference
_on Applications of Computer Vision (WACV), pp. 1–9. IEEE, 2016._

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image
recognition. arXiv preprint arXiv:1409.1556, 2014.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In
_Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9, 2015._

Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution
discrepancy. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates,
Inc., 2019.

Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multime_dia, pp. 1041–1049, 2017._

Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926–930, 2018a.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei
Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE
_conference on computer vision and pattern recognition, pp. 5265–5274, 2018b._

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp.
9929–9939. PMLR, 2020.

L. L. Whyte. Unique arrangements of points on a sphere. The American Mathematical Monthly, 59
[(9):606–611, 1952. ISSN 00029890, 19300972. URL http://www.jstor.org/stable/](http://www.jstor.org/stable/2306764)
[2306764.](http://www.jstor.org/stable/2306764)

Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person
re-identification: A benchmark. In Proceedings of the IEEE international conference on computer
_vision, pp. 1116–1124, 2015._

Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 16489–16498, 2021.


-----

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures
for scalable image recognition. In Proceedings of the IEEE conference on computer vision and
_pattern recognition, pp. 8697–8710, 2018._


-----

## Appendix for “Learning Towards the Largest Margin”

A PROOFS

**LemmaRiesz t-energy and 2.1. For any k-points best-packing configurations are uniquely given by the vertices of regular w1, ..., wk ∈** S[d][−][1], d ≥ 2, and 2 ≤ _k ≤_ _d + 1, the solution of minimal_
(k − 1)-simplices inscribed in S[d][−][1]. Furthermore, wi[T][w][j][ =] _k−−11_ _[,][ ∀][i][ ̸][=][ j][.]_

_Proof. See in Borodachov et al. (2019, Theorem 3.3.1)._

**Theorem 2.2. For w1, ..., wk, z1, ..., zN ∈** S[d][−][1] _(where nj > 0 for each j ∈_ [1, k]), the optimal
_solution {wi[∗][}]i[k]=1[,][ {][z]i[∗][}]i[N]=1_ [= arg max]{wi}i[k]=1[,][{][z][i][}][N]i=1w[∗][γ][min][ is obtained if and only if][ {][w]i[∗][}]i[k]=1 _[max-]_

_imizes the class margin mc({wi}i[k]=1[)][, and][ z]i[∗]_ [=] _∥wyi[∗]yi[−][−][w][w][∗]yi[∗]yi[∥][2]_ _[, where][ w]y[∗]i_ _[denotes the centroid of]_

_the vectors {wj : j maximizes wy[T]i_ **_[w][j][, j][ ̸][=][ y][i][}][.]_**


_Proof. According to the definition of γmin, we have_

arg maxw maxz _γmin = arg maxw_ maxz mini **_wy[T]i_** **_[z][i]_** _[−]_ [max]j≠ _yi_ **_[w]j[T][z][i]_**

= arg maxw mini maxzi **_[w]y[T]i_** **_[z][i]_** _[−]_ [max]j≠ _yi_ **_[w]j[T][z][i],_**

= arg maxw mini maxzi **_[w]y[T]i_** **_[z][i]_** _[−]_ **_[w]k[T][z][i]_**

= arg maxw mini _∥wyi −_ **_wk∥2_**

where k = arg maxj≠ _yi wj[T][z][i][, and][ z][i][ =]_ _∥wwyiyi−−wwkk∥2_ [. Notice that][ w]k[T][z][i][ =][ −] 2[1] _[∥][w][y][i]_ _[−]_ **_[w][k][∥][2][, then]_**

_k = arg minj=yi_ **_wy[T]i_**
_̸_
_∥_ _[−]_ **_[w][j][∥][2][. Therefore, we have]_**

arg max max _γmin = max_ min min
**_w_** **_z_** **_w_** [min]i _k≠_ _yi_ _[∥][w][y][i][ −]_ **_[w][k][∥][2][ = arg max]w_** _i≠_ _j_ _[∥][w][i][ −]_ **_[w][j][∥][2][,]_**

i.e., maximizing γmin will provide the solution of the Tammes Problem, which also maximizes the
class margin.


On the other hand, zi[∗] [maximizes][ w]y[∗]i[T][z][i] _i_ **_[w]j[∗][T][z][i][,][ i.e.][,]_**

_[−]_ [max][j][̸][=][y]

**_zi[∗]_** [= arg max]zi∈S[d][−][1][ w]y[∗]i[T][z][i] _[−]_ [max]j≠ _yi_ **_[w]j[∗][T][z][i]_**

= arg maxzi S[d][−][1][ w]y[∗]i[T][z][i] _[−]_ **_[w]y[∗]i[T][z][i]_** _,_
_∈_

= **_wy[∗]i_** _[−]_ **_[w]y[∗]i_**

**_wy[∗]i_** _yi_ _[∥][2]_
_∥_ _[−]_ **_[w][∗]_**

where w[∗]yi [denotes the centroid of the vectors][ {][w][j] [:][ j][ maximizes][ w]y[T]i **_[w][j][, j][ ̸][=][ y][i][}][.]_**

**Proposition 2.3. For any w1, ..., wk, z1, ..., zN** S[d][−][1], d 2, and 2 _k_ _d + 1, the maximum_
_of γmin is_ _k_ _k_ 1 _[, which is obtained if and only if][ ∀] ∈[i][ ̸][=][ j][,][ w]i[T] ≥[w][j][ =][ −]_ _k_ 1 ≤1 _[, and] ≤[ z][i][ =][ w][y][i]_ _[.]_

_−_ _−_

the class margin andProof. Based on Theorem zi = 2.2∥wwyiyi, the maximum of−−wwyiyi∥2 [,][ i.e.][,][ w]i[T][w] γmin[j][ =] is obtained if and only if[ −] _k−1_ 1 [according to Lemma] {w[ 2.3]i}i[k]=1[. At this time,][maximizes]

we have zi = _∥wwyiyi−−wwyiyi∥2_ [=] _∥wwyiyi−−((−−wwyiyi))∥2_ [=][ w][y][i] [.]

**Theorem 3.1. ∀ε ∈** (0, π/2], if the range of w1, ..., wk or z1, ..., zN is R[d] _(2 ≤_ _k ≤_ _d + 1), then_
_there exists prototypes that achieve the infimum of the softmax loss and have the class margin ε._


-----

_Proof. With the softmax loss, the goal is to optimize the following problem_


_N_ exp(wy[T]i **_[z][i][)]_**

min _L =_ log _K_ _._
_{wj_ _}j[k]=1[,][{][z][i][}][N]i=1_ _−_ _N[1]_ _i=1_ _j=1_ [exp(][w]j[T][z][i][)]

X

P

_∀ε ∈_ (0, _[π]2_ []][, we can easily obtain][ k][ ≤] _[d][ + 1][ vectors][ w]1[′]_ _[, ...,][ w]k[′]_ [on the unit sphere][ S][d][−][1][, such that]

the angle between any two of them is ε ∈ (0, _[π]2_ [)][.]

(1) If the domain of w1, ..., wk is R[d], then let wj = swj[′] [, and][ z][i][ =][ w][y]i [. In this way, we have]
**_wy[T]i_** **_[z][i]_** _[>][ w]j[T][z][i][,][ ∀][j][ ̸][=][ y][i][. The infimum of softmax loss can be obtained by directly increasing][ s][.]_

(2) If the domain of z1, ..., zk is R[d], then let wj = wj[′] [, and][ z][i][ =][ s][w][y]i [. In this way, we have]
**_wy[T]i_** **_[z][i]_** _[>][ w]j[T][z][i][,][ ∀][j][ ̸][=][ y][i][. The infimum of softmax loss can be obtained by directly increasing][ s][.]_

In conclusion, without both normalization for both features and prototypes, the original softmax loss
may produce an arbitrary small class margin ε.

**Theorem 3.2. For class-balanced datasets (i.e., each class has the same number of samples),**
_αwi11, ..., = α w1k, α, zi12, ..., = α z2N, ∈ βi1 =S[d][−] β[1]1, d and ≥ β2i2, and = β 22) leads to maximizing both the class margin and the ≤_ _k ≤_ _d + 1, learning with GM-Softmax (where_
_sample margin. More specifically, the optimal solution_


1 _N_ exp(s(α1wy[T]i **_[z][i]_** [+][ β][1][))]
**_wj[∗]_** _j=1[,][ {][z]i[∗]_ _i=1_ [= arg min] log
_{_ _[}][k]_ _[}][N]_ **_wj_** _,zi∈S[d][−][1]_ _N_ _i=1_ _−_ exp(s(α2wy[T]i **_[z][i]_** [+][ β][2][)) +][ P]j≠ _yi_ [exp(][s][w]j[T][z][i][)]

X

_have the largest class margin m[∗]c_ [= arccos][ −]k [1]1 _[and the largest sample margin][ γ]min[∗]_ [=] _k_ _k_ 1 _[. The]_

_lower bound of the risk iswhich is obtained if and only if log[exp( ∀i ̸=s( jα,1 w +i[T] β[w]1[j]− −[ =]αk2−− −11_ _[, and]β2)) + ([ z][i][ =]k[ w] −[y]1) exp([i]_ _[.]_ _−s(_ _k−1_ 1 [+][ α][1][ +]− _[ β][1][))]][,]_

_Proof. Since the function exp is strictly convex, using the Jensen’s inequality, we have_


_N_ exp(s(α1wy[T]i **_[z][i]_** [+][ β][1][))]

_i=1_ _−_ log exp(s(α2wy[T]i **_[z][i]_** [+][ β][2][)) +][ P]j≠ _i_ [exp(][s][w]j[T][z][i][)]

X

_N_ exp(s(α1wy[T]i **_[z][i]_** [+][ β][1][))]

log _s_

Xi=1 _−_ exp(s(α2wy[T]i **_[z][i]_** [+][ β][2][)) + (][k][ −] [1) exp(] _k−1_

P


_L = [1]_

_N_

_≥_ _N[1]_


_j=i_ **_[w]j[T][z][i][)]_**
_̸_


Let w = _k[1]_ _ki=1_ **_[w][i][,][ α][ =][ α][2][ −]_** _[α][1][,][ β][ =][ β][2][ −]_ _[β][1][,][ σ][ =]_ _k_ _k_ 1 [, and][ δ][ =] _k_ 1 1 [+][ α][1][, then we have]

_−_ _−_

P _N_

_L_ log exp(s(αwy[T]i **_[z][i]_** [+][ β][) + (][k][ −] [1) exp(][s][(][σ][w][ −] _[δ][w][y]i_ [)][T][z][i]
_≥_ _N[1]_ _i=1_ _[−]_ _[sβ][1][)]_

XN  

_≥_ _N[1]_ _i=1_ log[exp(sα + sβ) + (k − 1) exp(−s∥σw − _δwyi_ _∥2 −_ _sβ1)]_ _,_

X

_k_

= [1] log[exp(sα + sβ) + (k 1) exp( _s_ _σw_ _δwi_ 2 _sβ1)]_

_k_ _i=1_ _−_ _−_ _∥_ _−_ _∥_ _−_

X

where we use the facts that αwy[T]i **_[z][i]_** _i_ [)][T][z][i] [when]

_[≥]_ _[α][ when][ α][ ≤]_ [0][,][ (][σ][w][ −] _[δ][w][y]_ _[≥−∥][σ][w][ −]_ _[δ][w][i][∥][2]_
**_zi ∈_** S[d][−][1]. Due the convexity of the function log[1 + exp(ax + b)] (a > 0), we use the Jensen’s


-----

inequality and obtain that


exp(s(α + β)) + (k 1) exp(
_−_ _−_ _k[s]_


_L ≥_ log

_≥_ log


_σw_ _δwi_ 2 _sβ1)_
_i=1_ _∥_ _−_ _∥_ _−_ #

X

_k_

_k_ _σw_ _δwi_ 2
vu _i=1_ _∥_ _−_ _∥[2]_ _[−]_ _[sβ][1][)]_
u X
t 

_k(kδ[2]_ _−_ 2kσδ∥w∥2[2] [+][ kσ][2][∥][w][∥]2[2][)][ −] _[sβ][1][)]_

q


log exp(s(α + β)) + (k 1) exp(
_≥_  _−_ _−_ _k[s]_

= log exp(s(α + β)) + (k 1) exp(
_−_ _−_ _k[s]_



log[exp(s(α + β)) + (k 1) exp( _s(δ + β1))]_
_≥_ _−_ _−_

1
= log exp(s(α2 _α1 + β2_ _β1)) + (k_ 1) exp( _s(_
_−_ _−_ _−_ _−_ _k_ 1 [+][ α][1][ +][ β][1][))]
 _−_ 

where in the second inequality we used the Cauchy–Schwarz inequality, and the third inequality is
based on that σ ≤ 2δ ⇔ _α1 ≥_ 2kk−−22 [, which holds since][ α][1][ ≥] [1]2 [.]

**_wAccording to the above derivation, the equality holds if and only ify[T]i+1[z][i]_** [=][ ...][ =][ w]k[T][z][i][,][ w]y[T]i **_[z][i]_** [= 1][,][ z][i] [=][ −] _∥σσww−−δδwwyiyi∥2_ [,][ ∥][σ][w][ −] _[δ] ∀[w]i,[1] w[∥][2][ =]1[T][z][ ...][i]_ [=][ =][ ...][ ∥][ =][σ][w][ w][ −]y[T]i−[δ][w]1[z][k][i][∥][=][2][,]

and w = 0. The condition can be simplified as ∀i ̸= j, wi[T][w][j][ =] _k−−11_ [, and][ z][i][ =][ w][y][i] [when][ 2][ ≤] _[d]_

and 2 ≤ _k ≤_ _d + 1._

**Propositionk ≤** _d + 1, learning with the loss functions A-Softmax ( 3.3. For class-balanced datasets, w1, ...,Liu et al. wk, z1, ...,, 2017 zN ∈) with feature normalization,S[d][−][1], d ≥_ 2, and 2 ≤
_NormFace (Wang et al., 2017), CosFace (Wang et al., 2018b) or AM-Softmax (Wang et al., 2018a),_
_and ArcFace (Deng et al., 2019) share the same optimal solution._

_Proof. A unified framework for A-Softmax with feature normalization, NormFace, LMLC/AM-_
Softmax and ArcFace can be implemented with hyper-parameters m1, m2 and m3, i.e.,

exp(s(cos(m1θiyi + m2) _m3))_
_L[′]i_ [=][ −] [log] _−_

exp(s(cos(m1θiyi + m2) _m3)) +_ _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_
_−_ _̸_

where θij = ∠(wj, zi). The setting of these hyper-parameters always guarantees that cos(m1θiyi +

[P]

_m2)_ _m3_ cos m2 cos θiyi _m3, and m2 is usually set to satisfy cos m2_ 2 [. Let][ α][ = cos][ m][2]

and β − = − ≤m3 < 0, then we have − _≥_ [1]

exp(s(α cos θiyi + β))
_L[′]i_ _[≥−]_ [log] exp(s(α cos θiyi + β)) + _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_ (A.1)

_̸_

where the equality holds if and only if θiyi = 0.

[P]

According to Theorem 3.2, we know that the empirical risk of the loss function in the right-hand
side of (3.2) has a lower bound, then we obtain


_N_

1 exp(s(α + β))

_N_ _i=1_ _L[′]i_ _[≥−]_ [log] exp(s(α + β)) + _j≠_ _yi_ [exp(][−] _k−s_ 1 [)] (A.2)

X

The equality holds if and only if _i_ = j, wi[T][w][j][ =] _k−11_ [, and][P] **_[ z][i][ =][ w][y][i]_** [. Since][ z][i][ =][ w][y][i] [means]
_∀_ _̸_ _−_

_θiyi = 0, indicating that the equality in (3.2) holds. Then the optimal solution is the same for_
A-Softmax with feature normalization, NormFace, CosFace, and ArcFace.

_dTheorem + 1, learning with 3.4. For class-balanced datasets, Rsm = −wy[T][z][ + max][j] w[̸][=][y]1[w], ...,j[T][z] w[ leads to the maximization of the class margin]k, z1, ..., zN ∈_ S[d][−][1], d ≥ 2, and 2 ≤ _k ≤_
_and the sample margin._


-----

_Proof. Let L(z, y) = −wy[T][z][ + max][j][̸][=][y]_ **_[w]j[T][z][,][ w][ =][ 1]k_** _ki=1_ **_[w][i][, then we have]_**

_N_ _N_

1 P

_L(zi, yi) = [1]_ ( **_wy[T]i_** **_[z][i]_** [+ max] _j_ **_[z][i][)]_**

_N_ _N_ _−_ _j=yi_ **_[w][T]_**

_i=1_ _i=1_ _̸_

X X

_N_

1
( **_wy[T]i_** **_[z][i]_** [+] **_wj[T][z][i][)]_**

_≥_ _N[1]_ _−_ _k_ 1

Xi=1 _−_ _jX≠_ _yi_


(wyi **_w)[T]zi_**
_i=1_ _−_ _−_

X

_N_

**_wyi_** **_w_** 2
_i=1_ _−∥_ _−_ _∥_

X


_N_ (k − 1)

_k_

_N_ (k − 1)


(A.3)


**_wi_** **_w_** 2
_i=1_ _−∥_ _−_ _∥_

X


_k −_ 1


_k(_ **_wi_** **_w_** 2[)]
vu _i=1_ _∥_ _−_ _∥[2]_
u X
t

_k(k −_ _k∥w∥2[2][)]_

q


_k −_ 1 _[−]_

1

_k −_ 1 _[−]_

_k_


_≥−_ _k_ 1

_−_

where the equality holds if and only if ∀i ̸= j, wi[T][w][j][ =] _k−−11_ [, and][ z][i][ =][ w][y][i] [.]


**Theorem 3.5. For class-balanced or -imbalanced cases, w1, ..., wk, z1, ..., zN ∈** S[d][−][1], d ≥ 2,
_and 2 ≤_ _k ≤_ _d + 1, if_ _i=1_ **_[w][i][ = 0][, then learning with the GM-Softmax loss in (][3.3][) leads to]_**
_maximizing both the class margin and the sample margin. More specifically, the optimal solution_
_{marginwj[∗][}]j[K]=1 γmin[∗][,][ {][z]i[=][∗][}]i[N]k=1k_ 1[has the largest class margin][. The lower bound of the risk is][P][K] _[ m][ 1][(]N[W][ ∗][) = arccos]Ni=1_ [log[exp(][s]K−[(]−[α]11[i][1][ +][and the largest sample][ β][i][1][ −] _[α][i][2][ −]_ _[β][i][2][)) +]_

(k − 1) exp(−s( _−k−1_ 1 [+][ α][i][1][ +][ β][i][1][))]][, which is obtained if and only if]P _[ ∀][i][ ̸][=][ j][,][ w]i[T][w][j][ =]_ _K−−11_ _[, and]_

**_zi = wyi_** _, i.e., the optimal solution maximizes that class margin and sample margin._


exp(s(αi1wyi[T] **_[z][i][+][β][i][1][))]_**
_Proof. For the GM-Softmax loss Li = −_ log exp(s(αi2wyi[T] **_[z][i][+][β][i][2][))+][P]j≠_** _yi_ [exp(][s][w]j[T][z][i][)] [, let][ α][i][ =][ α][i][2] _[−]_

_αi1 ≤_ 0, βi = βi2 − _βi1. If_ _i=1_ **_[w][i][ = 0][, then we have]_**

exp(s(αi1wy[T]i **_[z][i]_** [+][ β][i][1][))]
_Li =_ log [P][K]
_−_ exp(s(αi2wy[T]i **_[z][i]_** [+][ β][i][2][)) +][ P]j=yi [exp(][s][w]j[T][z][i][)]

_̸_

exp(s(αi1wy[T]i **_[z][i]_** [+][ β][i][1][))]
log 1
_≥−_ exp(s(αi2wy[T]i **_[z][i]_** [+][ β][i][2][)) + (][k][ −] [1) exp(] _k_ 1 _j=yi_ _[s][w]j[T][z][i][)]_

_−_ _̸_

exp(s(αi1wy[T]i **_[z][i]_** [+][ β][i][1][))] P _._ (A.4)
= − log exp(s(αi2wy[T]i **_[z][i]_** [+][ β][i][2][)) + (][k][ −] [1) exp(][−] _k_ _s_ 1 **_[w]y[T]i_** **_[z][i][)]_**

_−_

_s_
= log exp(sαiwy[T]i **_[z][i]_** [+][ sβ][i][) + (][k][ −] [1) exp(][−] _yi_ **_[z][i]_** _yi_ **_[z][i]_** [+][ β][i][1][))]

_k_ 1 **_[w][T]_** _[−]_ _[s][(][α][i][1][w][T]_

 _−_ 

_≥_ log exp(s(αi1 + βi1 − _αi2 −_ _βi2)) + (k −_ 1) exp(−s(1/(k − 1) + αi1 + βi1))

where in the first inequality we used the Jensen’s inequality, and the last inequality comes from the 
facts that αiwy[T]i **_[z][i]_** _[≥]_ _[α][i]_ [and][ −] _k−1_ 1 **_[w]y[T]i_** **_[z][i]_** _[−]_ _[α][i][1][w]y[T]i_ **_[z][i]_** _[≥−]_ _k−1_ 1 _[−]_ _[α][i][1][.]_

Therefore, we have the lower bound of the risk _N1_ _Ni=1_ _[L][i][ ≥]_ _N1_ _Ni=1_ [log[exp(][s][(][α][i][1][ +][ β][i][1][ −]

_αi2_ _βi2)) + (k_ 1) exp( _s(_ _k_ 1 1 [+][ α][i][1][ +][ β][i][1][))]][, where the equality holds if and only if][ ∀] _[i][,]_
_−_ _−_ _−_ _−_ P P


-----

**_w1[T][z][i]_** [=][ ...][ =][ w]y[T]i 1[z][i] [=][ w]y[T]i+1[z][i] [=][ ...][ =][ w]k[T][z][i][, and][ w]y[T]i **_[z][i]_** [= 1][. The condition can be simplified]
as ∀i ̸= j, wi[T][w][j][ =]− _k−−11_ [, and][ z][i][ =][ w][y][i] [when][ 2][ ≤] _[d][ and][ 2][ ≤]_ _[k][ ≤]_ _[d][ + 1][.]_

B MORE ANALYSIS

In this section, we provide more analysis about the unified framework of margin-based losses in
(3.2), Sample Margin Regularization, Largest-Margin Softmax (LM-Softmax) loss.

B.1 A UNIFIED FRAMEWORK

A unified framework that covers A-Softmax (Liu et al., 2017) with feature normalization, NormFace
(Wang et al., 2017), CosFace/AM-Softmax (Wang et al., 2018b;a) and ArcFace (Deng et al., 2019)
as special cases can be formulated with hyper-parameters m1, m2 and m3:

exp(s(cos(m1θiyi + m2) _m3))_
_L[′]i_ [=][ −] [log] _−_ (B.1)

exp(s(cos(m1θiyi + m2) _m3)) +_ _j=yi_ [exp(][s][ cos][ θ][ij][)] _[,]_
_−_ _̸_

where θij = ∠(wj, zi). In the following, we provide the details of the derivation from (3.1) to (3.2)

[P]

For the parameter m1, it satisfies that cos(m1θ) cos(θ) in SphereFace Liu et al. (2017). There_≤_
fore, based on the definition of the multiplicative-angular operator, we have cos(m1θiyi + m2) ≤
cos(θiyi + m2). To better understand the theoretical optimal solution, we make the constraint that

CosFace, and ArcFace, should satisfyθiyi ∈ [0, _[π]2_ []][, which is reasonable because the unique minimizer of these losses, like SphereFace,] θiy[∗] _i_ [= 0][, rather than belongs to][ (][ π]2 _[, π][]][.]_

As forOtherwise, the minimum of ArcFace will be obtained at m2, ArcFace did not analyze its range. Instead, we can easily derive that θiyi = π, since cos(θiyi + m 0 ≤2) ≤mcos(2 ≤π +π2 [.]
_m2) when m2 >_ _[π]2_ [, which is ridiculous. Therefore, for][ θ][iy][i] _[, m][2][ ∈]_ [[0][,][ π]2 []][, we have][ cos(][θ][iy][i] [+][m][2][) =]

cos θiyi cos m2 − sin θiyi sin m2 ≤ cos m2 cos θiyi, which is the main derivation from (3.1) to (3.2).

B.2 ON THE SAMPLE MARGIN REGULARIZATION AND BEYOND

The sample margin regularization term in (3.4) actually encourages the feature representation z to
be similar to the corresponding prototype wy, and push z away from the most similar one of the
other prototypes. This concept is similar to contrastive learning, where the most similar one of the
other prototypes can be regarded as the hardest negative representation. And we also have


_Rsm(x, y) ≤−wy[T][z][ +]_


**_wj[T][z][,]_** (B.2)

Xj≠ _y_


_k −_ 1


where the right side can be regarded as pushing z away from the centroid _k_ 1 1 _j=y_ **_[w][j][ or pushing]_**

_−_ _̸_

**_z away from other negative representations. Intuitively, we can also use the right side of Eq. B.2 as_**

P

a sample margin regularization.

B.3 MORE CLARIFICATIONS

As shown in the main paper, GM-Softmax loss, LM-Softmax loss, Sample Margin regularization,
and Zero-centroid regularization serve different purposes. More specifically,

-  The GM-Softmax loss is only derived as a theoretical formulation, which is not used for
practical implementation.

-  The LM-Softmax loss is tailored to obtain large margins with only one hyper-parameter.
It can be used to replace popular margin-based losses, such as CosFace, and ArcFace,
to obtain better discriminativeness of feature representations. Compared with NormFace
Wang et al. (2017), LM-Softmax achieves much better performance on the task of person
ReID, as shown in Table 3. This demonstrates that removing the term exp(swy[T][z][)][ in the]
denominator is helpful, which enforces LM-Softmax to have a stronger fitting ability.


-----

(a) NormFace (68.50[◦]) (b) CosFace (71.86[◦]) (c) ArcFace (67.74[◦]) (d) LM-Softmax (74.46[◦])

Figure 3: Visualization of the learned prototypes (red arrows) and features (green points) using
NormFace, CosFace, ArcFace and LM-Softmax on S[2] for eight classes. The optimal solution of
Tammes problem for N = 8 have the class margin 74.86[◦] (Whyte, 1952), where the class margin of
learning with the losses NormFace, CosFace, ArcFace and LM-Softmax are 68.50[◦], 71.86[◦], 67.74[◦]
and 74.46[◦], respectively. We note that this phenomenon coincides with the recent popular concept—
_neural collapse (Papyan et al., 2020)._

-  The sample margin regularization Rsm serves as a general regularization term to significantly improve the ability of learning towards the largest margins by combining it with
the commonly-used losses. Sample margin is not new, but to the best of our knowledge,
we are the first one to use it in deep learning to obtain feature representations with interclass separability and intra-class compactness. Although theoretically learning with Rsm
can achieve the largest margins, we verify by experiments that directly maximizing sample
margin cannot optimize neural networks well on complex datasets, such as CIFAR-100,
as shown in Table 5. It can be found that learning with Rsm suffers from the underfitting
problem on CIFAR-100, whose performance is much worse than CE. Alternatively, we
turn to use Rsm as a regularization term, which can significantly improve the performance
of commonly-used CE loss. These results demonstrate that using sample margin as the
regularization term is more beneficial than using it as the loss. This is our new contribution
to the classical sample margin.

-  The zero-centroid regularization Rw is specially tailored for class-imbalanced cases, which
is only applied to prototypes at the last inner-product layer. Therefore, it can be easily
embedded into the DNN-based methods to handle class imbalance.

C EXPERIMENTS

In this section, we provide the experimental details, including datasets, network architectures, parameter settings, analysis, and more results. All codes are implemented by PyTorch (Paszke et al.,
2019).

regularizationWe first recall the sample margin regularization Rw = ∥ _k[1]_ _ki=1_ **_[w][i][∥]2[2][, which are used to enlarge margins for baseline methods. As] Rsm = −wy[T][z][+max][j][̸][=][y]_** **_[w]j[T][z][ and the zero-centroid]_**

for the trade-off parameter settings µ and λ in the following experiments. We use µ and λ[′] = 100λ
denote the trade-off parameters forP _Rsm and Rw, i.e., L + µRsm and L + 100λRw, respectively. In_
the following, we set µ = 0.5, 1.0 for Rsm, and λ = 1, 2, 5, 10, 20 for Rw.

C.1 TOY EXPERIMENT

We conduct a toy experiment to show the inter-class separability and intra-class compactness using
different losses, where we randomly generate prototypes W ∈ R[k][×][d] (we set d = 3 and k = 8), and
initialize features Z ∈ R[N] _[×][d]_ (we set N = 10k). Our goal is to optimize both W and Z to learn
the largest class margin and sample margin with different losses. According to the Tammes problem
for N = 8, the optimal solution of W and Z satisfies that mc(W ) = 74.86[◦] (Whyte, 1952). The
number of training epochs is set 500,000. We use cosine learning rate annealing with Tmax=10,000,
and SGD optimizer with momentum 0.9 and weight decay 1e − 4.


-----

(a) CosFace (s = 64) (b) CosFace (s = 64) + 0.5Rsm (c) CosFace (s = 64) + Rsm

(d) CosFace (s = 64) (e) CosFace (s = 64) + 0.5Rsm (f) CosFace (s = 64) + Rsm

Figure 4: Histogram of similarities and sample margins for CosFace with/without sample margin
regularization Rsm on CIFAR-10. (a-c) denote the cosine similarities between samples and their
corresponding prototypes, and (d-f) denote the sample margins.

**Results. We use green points and red arrows to denote the learned feature vectors and prototype**
vectors, respectively. As shown in Fig. 3, the learned prototypes are separated well with NormFace, CosFace, ArcFace, and LM-Softmax. Specifically, the class margin of learning with the losses
NormFace, CosFace, ArcFace, and LM-Softmax are 68.50[◦], 71.86[◦], 67.74[◦] and 74.46[◦], respectively. As we can seen, ArcFace has a smaller class margin (67.74[◦]) than the others, and the intraclass compactness for NormFace and CosFace is worse than LM-Softmax. The features in the blue
box of Fig. 3(a) and Fig. 3(b) are not compact enough, but ArcFace and LM-Softmax do. Moreover,
our proposed LM-Softmax shows better performance in class margin and sample margins, where
the learned prototypes have the class margin close to the theoretical optima, and the features are
perfectly optimized to be their corresponding prototypes.

C.2 VISUAL CLASSIFICATION

We introduce three metrics to evaluate whether a loss function owns good inter-class separability and intra-class compactness. The first one is the top-1 test accuracy acc to measure the
generalization of the trained models. The second one is the class margin mcls defined in Eq.
(2). And the last one we define as the average of sample margins with cosine similarities, i.e.,

1 _N_ **_wyi[T]_** _[φ][Θ][(][x][i][)]_ **_wyi[T]_** _[φ][Θ][(][x][i][)]_
_msamp =_ _N_ _i=1_ _∥wyi_ _∥∥φΘ(xi)∥_ _[−]_ [max][j][̸][=][y][i] _∥wj_ _∥∥φΘ(xi)∥_ [. Then we experiments with a 4-layer]

CNN, ResNet-18 and ResNet-34 (He et al., 2016) on MNIST (LeCun et al., 1998), CIFAR-10 and

P

CIFAR-100 (Krizhevsky & Hinton, 2009), respectively. Moreover, some commonly-used neural
layers are considered, such as ReLU (Glorot et al., 2011), BatchNorm (Ioffe & Szegedy, 2015), and
cosine learning rate annealing (Loshchilov & Hutter, 2016).

**Datasets. We empirically investigate the performance of learning towards the largest margins on**
benchmark datasets including MNIST (LeCun et al., 1998), CIFAR-10 and CIFAR-100 (Krizhevsky
& Hinton, 2009).

**Training details. We use a simple CNN which consists of Conv(1, 32, 3) →** _BatchNorm (Ioffe &_
_Szegedy, 2015) →_ _ReLU (Glorot et al., 2011) →_ _MaxPool(2,2) →_ _Conv(32, 64, 3) →_ _BatchNorm →_
_ReLU →_ _MaxPool(2,2) →_ _Linear() for MNIST, a ResNet-18 (He et al., 2016) for CIFAR-10, and_
a ResNet-34 (He et al., 2016) for CIFAR-100. The number of training epochs is set 100, 200 and


-----

Table 5: Test accuracies, class margins and sample margins on MNIST, CIFAR-10 and CIFAR100 using loss functions with/without sample margin regularization Rsm, where we simply set the
regularization parameter to 0.5. The results with positive gains are highlighted.

|Dataset|MNIST|CIFAR-10|CIFAR-100|
|---|---|---|---|

|Metric|acc mcls msamp|acc mcls msamp|acc mcls msamp|
|---|---|---|---|

|CosFace (s = 5) CosFace (s = 10) CosFace (s = 20) CosFace (s = 40) CosFace (s = 64)|99.11 95.85◦ 1.020 98.98 95.93◦ 0.9839 99.06 93.24◦ 0.8376 99.18 90.69◦ 0.7650 99.25 89.50◦ 0.7581|94.02 96.33◦ 0.9619 94.39 96.00◦ 0.9168 94.13 91.22◦ 0.7955 93.84 76.09◦ 0.7617 93.53 64.14◦ 0.6969|75.37 84.20◦ 0.5037 74.44 83.31◦ 0.4578 73.26 79.17◦ 0.3078 73.54 77.48◦ 0.2380 73.87 72.56◦ 0.2233|
|---|---|---|---|

|CosFace (s = 5) + 0.5Rsm CosFace (s = 10) + 0.5Rsm CosFace (s = 20) + 0.5Rsm CosFace (s = 40) + 0.5Rsm CosFace (s = 64) + 0.5Rsm|99.07 95.60◦ 1.036 99.16 95.56◦ 1.033 99.24 95.41◦ 1.030 99.32 95.41◦ 1.026 99.27 95.35◦ 1.019|94.20 96.32◦ 0.9740 94.42 96.26◦ 0.9675 94.27 96.18◦ 0.9490 94.42 95.93◦ 0.9238 94.20 95.48◦ 0.9075|75.52 90.41◦ 0.5230 73.76 90.21◦ 0.5089 74.41 89.02◦ 0.4780 74.58 86.91◦ 0.4251 74.53 85.31◦ 0.3817|
|---|---|---|---|

|ArcFace (s = 5) ArcFace (s = 10) ArcFace (s = 20) ArcFace (s = 40) ArcFace (s = 64)|99.05 95.46◦ 0.9956 99.05 94.64◦ 0.8225 99.11 90.84◦ 0.6091 99.13 86.13◦ 0.4606 99.21 82.63◦ 0.4038|93.90 96.33◦ 0.9473 94.50 91.23◦ 0.8501 94.11 53.98◦ 0.5707 93.88 35.68◦ 0.3195 − − −|75.08 78.28◦ 0.4884 73.96 76.91◦ 0.4313 74.74 60.91◦ 0.3010 − − − − − −|
|---|---|---|---|

|ArcFace (s = 5) + 0.5Rsm ArcFace (s = 10) + 0.5Rsm ArcFace (s = 20) + 0.5Rsm ArcFace (s = 40) + 0.5Rsm ArcFace (s = 64) + 0.5Rsm|99.00 95.59◦ 1.034 99.14 95.42◦ 1.034 99.19 91.38◦ 1.030 99.24 95.34◦ 1.026 99.14 95.29◦ 1.019|94.17 96.32◦ 0.9731 94.21 96.27◦ 0.9651 94.32 96.15◦ 0.9571 94.07 95.69◦ 0.9434 − − −|74.72 90.37◦ 0.5081 74.47 90.13◦ 0.5143 74.64 88.73◦ 0.4804 − − − − −|
|---|---|---|---|

|NormFace (s = 5) NormFace (s = 10) NormFace (s = 20) NormFace (s = 40) NormFace (s = 64)|99.03 95.68◦ 0.9836 99.06 94.34◦ 0.7750 99.09 89.27◦ 0.5263 99.06 85.44◦ 0.3473 99.00 82.08◦ 0.2621|94.34 96.34◦ 0.9452 94.16 94.40◦ 0.8004 94.09 74.32◦ 0.6001 94.11 47.52◦ 0.3825 94.01 36.50◦ 0.2633|75.56 85.37◦ 0.5076 74.23 79.10◦ 0.4250 73.87 77.47◦ 0.2498 73.73 66.67◦ 0.1439 73.42 52.37◦ 0.0993|
|---|---|---|---|

|NormFace (s = 5) + 0.5Rsm NormFace (s = 10) + 0.5Rsm NormFace (s = 20) + 0.5Rsm NormFace (s = 40) + 0.5Rsm NormFace (s = 64) + 0.5Rsm|99.15 95.55◦ 1.035 99.16 95.38◦ 1.034 99.19 95.37◦ 1.031 99.14 95.36◦ 1.026 99.34 95.29◦ 1.021|94.11 96.32◦ 0.9739 94.23 96.28◦ 0.9650 94.38 96.17◦ 0.9519 94.18 95.59◦ 0.9495 94.42 93.87◦ 0.9508|74.82 90.38◦ 0.5124 74.54 90.10◦ 0.5160 74.75 88.86◦ 0.4773 74.48 84.78◦ 0.4181 74.33 76.02◦ 0.3665|
|---|---|---|---|

|NormFace (s = 5) + Rsm NormFace (s = 10) + Rsm NormFace (s = 20) + Rsm NormFace (s = 40) + Rsm NormFace (s = 64) + Rsm|99.14 95.48◦ 1.029 99.12 95.37◦ 1.028 99.11 95.35◦ 1.028 99.11 95.36◦ 1.026 99.14 95.34◦ 1.025|94.42 96.34◦ 0.9798 94.31 96.32◦ 0.9758 94.16 96.25◦ 0.9656 93.98 95.87◦ 0.9583 94.04 94.35◦ 0.9570|74.89 90.45◦ 0.5134 73.16 90.31◦ 0.5183 74.23 89.72◦ 0.5004 74.22 88.73◦ 0.4731 74.24 81.57◦ 0.4386|
|---|---|---|---|


CE 99.11 87.39[◦] 0.5014 94.12 81.73[◦] 0.6203 74.56 65.38[◦] 0.1612
_Rsm_ 99.07 95.38[◦] 1.036 94.13 96.28[◦] 0.9791 62.08 58.58[◦] 0.3793
CE + 0.5Rsm **99.13** **95.41[◦]** **1.026** **94.45** **96.31[◦]** **0.9744** **74.96** **90.00[◦]** **0.4955**


CosFace (s = 5) + Rsm **99.15** 95.59[◦] **1.032** **94.38** **96.35[◦]** **0.9817** 75.18 **90.44[◦]** **0.5228**
CosFace (s = 10) + Rsm **99.09** 95.48[◦] **1.029** **94.49** **96.32[◦]** **0.9770** 73.93 **90.36[◦]** **0.5237**
CosFace (s = 20) + Rsm **99.08** **95.37[◦]** **1.028** **94.36** **96.24[◦]** **0.9640** **73.79** **89.63[◦]** **0.4958**
CosFace (s = 40) + Rsm 99.12 **95.38[◦]** **1.027** **94.31** **96.18[◦]** **0.9510** **74.43** **88.83[◦]** **0.4736**
CosFace (s = 64) + Rsm 99.18 **95.38[◦]** **1.025** **94.60** **96.02[◦]** **0.9443** **74.05** **87.83[◦]** **0.4390**


ArcFace (s = 5) + Rsm **99.17** **95.53[◦]** **1.030** **94.40** **96.35[◦]** **0.9825** 74.85 **90.41[◦]** **0.5156**
ArcFace (s = 10) + Rsm **99.09** **95.37[◦]** **1.029** 94.14 **96.32[◦]** **0.9713** 73.76 **90.30[◦]** **0.5259**
ArcFace (s = 20) + Rsm 99.11 **95.36[◦]** **1.028** **94.45** **96.25[◦]** **0.9676** 74.61 **89.65[◦]** **0.5033**
ArcFace (s = 40) + Rsm 99.02 **95.34[◦]** **1.026** **94.39** **96.04[◦]** **0.9621** **_−_** **_−_** **_−_**
ArcFace (s = 64) + Rsm 99.13 **95.30[◦]** **1.024** **_−_** **_−_** **_−_** **_−_** **_−_** **_−_**


-----

(a) LM-Softmax (s = 10) (b) LM-Softmax (s = 20) (c) LM-Softmax (s = 40) (d) LM-Softmax (s = 64)

(e) LM-Softmax (s = 10) (f) LM-Softmax (s = 20) (g) LM-Softmax (s = 40) (h) LM-Softmax (s = 64)

Figure 5: Histogram of similarities and sample margins for LM-Softmax on CIFAR-10. (a-d) denote
the cosine similarities between samples and their corresponding prototypes, and (e-h) denote the
sample margins.

250 for MNIST, CIFAR-10, and CIFAR-100, respectively. For all training, we use SGD optimizer
with momentum 0.9 and cosine learning rate annealing (Loshchilov & Hutter, 2016) when Tmax is
equal to the corresponding epochs. Weight Decay is set to 1 × 10[−][4] for MNIST, CIFAR-10, and
CIFAR-100. The initial learning rate is set to 0.01 for MNIST and 0.1 for CIFAR-10 and CIFAR100.Moreover, batch size is set to 256. Typical data augmentations including random width/height
shift and horizontal flip are applied.

**Baselines and hyper-parameter settings.** We consider the baseline methods, including the
commonly-used loss function CE, and margin-based loss functions NormFace, CosFace, and ArcFace with normalization for both feature vectors and class centers, and our proposed LM-Softmax
loss. We have tuned their hyper-parameters for the best performance, and the specific settings are:
for CosFace, we set m = 0.1; for ArcFace, we set m = 0.1. To learn towards the largest margins,
we boost them with the sample margin regularization, and the trade-off parameter is set to 0.5 and
1. Moreover, we tune their identical hyper-parameter s, and show them for a comprehensive study.

**Results. The test accuracy, class margin and the average of all sample margins are reported in Table**
1. As we can see, the baseline methods fail in learning large margins for all s, and there is no significant difference in the performance of these losses. More specifically, the class margin decreased
as s increases, while the losses with the sample margin regularization Rsm usually remain the large
class margins, and the class margins are close to the optimal results (arccos(−1/9) = 96.37[◦] for
MNIST and CIFAR-10, and arccos(−1/99) = 90.57[◦] for CIFAR-100). To better describe the the
inter-class separability and intra-class compactness, we provide the histograms of sample margins
and similarities between the learned features and their corresponding prototype that they belong to.
In Fig. 9, the similarities in Fig. 9(a) are mainly concentrated in 0.8 for CosFace with s = 64,
while the similarities in Fig. 9(b) and 9(c) are very close to 1. This indicates that the sample margin
regularization significantly improves the inter-class compactness (the learned features in the same
class are very similar to their corresponding prototype.) Moreover, the histograms of our proposed
LM-Softmax on CIFAR-10 and CIFAR-100 are reported in Fig. 5 and 6, respectively. The similarities and sample margins keep very large with different s. More visualizations are provided in the
following figures.

**Clarification. As shown in table 5, the proposed method results in both more larger class margin**
and more larger sample margin than the compared methods, however, the accuracy of the proposed
method is slightly better than accuracies of the compared methods. acc actually evaluate the proportion of samples whose sample margin is larger than 0, i.e., acc = _N[1]_ _Ni=1_ [I][(][γ][(][x][i][, y][i][)][ >][ 0)][.][ acc][ is a]

good evaluation criterion for classification but is not good enough to measure the quality of feature
representation. This is also one of the motivations of the previous works to improve the originalP


-----

softmax loss. In this paper, we measure the inter-class separability and intra-class compactness by
class margin and sample margin, which can be used as two criteria to evaluate the quality of feature
representations. Thus, acc, class margin, and sample margin can be regarded as different criteria.

Although the relationship of acc and margins is not so straightforward, enlarging the margins can
improve acc to some extent. As shown in Table 1, we can see that enlarging the margins of other
losses by adding the sample margin regularization Rsm can improve the accuracy in most cases.
Moreover, as shown in Table 2, the results on imbalanced learning are noteworthy, where the zerocentroid regularization for learning towards the largest margins on imbalanced classification shows
obvious improvements in both class margins and accuracy in most cases, and even can improve the
performance of LDAM that is tailored for imbalanced learning

(a) LM-Softmax (s = 10) (b) LM-Softmax (s = 20) (c) LM-Softmax (s = 40) (d) LM-Softmax (s = 64)

(e) LM-Softmax (s = 10) (f) LM-Softmax (s = 20) (g) LM-Softmax (s = 40) (h) LM-Softmax (s = 64)

Figure 6: Histogram of similarities and sample margins for LM-Softmax on CIFAR-100. (a-d)
denote the cosine similarities between samples and their corresponding prototypes, and (e-h) denote
the sample margins.

C.3 IMBALANCED CLASSIFICATION

**Imbalanced CIFAR-10 and CIFAR-100. The original version of CIFAR-10 and CIFAR-100 con-**
tains 50,000 training images and 10,000 test images of size 32 × 32 with 10 and 100 classes, respectively. To create their imbalanced version, we follow the setting in (Buda et al., 2018; Cui et al.,
2019; Cao et al., 2019), where we reduce the number of training examples per class, and keep the
test set unchanged. To ensure that our methods apply to a variety of settings, we consider two types
of imbalance: long-tailed imbalance (Cui et al., 2019) and step imbalance (Buda et al., 2018). We
use the imbalance ratio ρ to denote the ratio between sample sizes of the most frequent and least
frequent class, i.e., ρ = maxi _ni_ _/ mini_ _ni_ . Long-tailed imbalance utilizes an exponential decay
_{_ _}_ _{_ _}_
in sample sizes across different classes. For step imbalance setting, all minority classes have the
same sample size, as do all frequent classes. This gives a clear distinction between minority classes
and frequent classes, and the fraction for minority classes is defined as µ. We follow (Cao et al.,
2019) and set µ = 0.5 by default.

We report the top-1 test accuracy acc and class margin mcls of various baseline methods, including CE, Focal Loss, NormFace, CosFace, ArcFace, and the Label-Distribution-Aware Margin Loss
(LDAM) with hyper-parameter s = 5. Moreover, the proposed LM-Softmax loss actually is greatly
affected by data imbalance since it will pay much attention to enlarge the margin between frequent
classes and minority classes than other losses rather than any two classes. And we experiment with
the LM-Softmax to verify the validity of the enlarging margin method. Moreover, we add the zerocentroid regularization to the losses whose feature and prototypes are normalized for better margins.

**Training details. We use ResNet-18 for imbalanced CIFAR-10, and ResNet-34 for imbalanced**
CIFAR-100. Following in (Cao et al., 2019), we use SGD optimizer with momentum 0.9 and weight
decay 2 _×_ 10[−][4]. The number of training epochs is set 200, and batch size is 128. The initial learning


-----

80

70

60

50

40

30

20

10


70

60

50

40

30

20

10


40

30

20

10


80

60

40

20

|Col1|Col2|
|---|---|
|||
||CE CosFace|
||CosFace* Focal LDAM|
||LDAM* LNorm LNorm*|
||Norm Norm*|

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
||||CE CosFace CosFace* Focal LDAM|
||||LDAM* LNorm LNorm* Norm Norm*|

|Col1|Col2|
|---|---|
|||
||CE CosFace CosFace* Focal LDAM|
||LDAM* LNorm LNorm* Norm Norm*|

|Col1|Col2|
|---|---|
|*||
|||
|||
|||
|||


CE
CosFace
CosFace*
Focal
LDAM
LDAM*
LNorm
LNorm*
Norm
Norm*

50 100 150 200

Epoch

(a) Test Accuracy


CE
CosFace
CosFace*
Focal
LDAM
LDAM*
LNorm
LNorm*
Norm
Norm*

50 100 150 200

Epoch

(b) Class Margin


CE
CosFace
CosFace*
Focal
LDAM
LDAM*
LNorm
LNorm*
Norm
Norm*

50 100 150 200

Epoch

(c) Test Accuracy


CE
CosFace
CosFace*
Focal
LDAM
LDAM*
LNorm
LNorm*
Norm
Norm*

50 100 150 200

Epoch

(d) Class Margin


Figure 7: Test accuracies and class margins using different loss functions with and without the zerocentroid regularization on imbalanced CIFAR-10 and CIFAR-100. (a) and (b) are test accuracies
and class margins on imbalanced CIFAR-10, respectively. (c) and (d) are test accuracies and class
margins on imbalanced CIFAR-10, respectively.

rate is set to 0.1. Moreover, we use the cosine learning rate annealing strategy (Loshchilov & Hutter,
2016) when Tmax is equal to the corresponding epochs.

**Baselines and their hyper-parameter settings. We consider the baseline methods, including CE,**
Focal loss, CosFace, NormFace, ArcFace, LM-Softmax, and the label-distribution-aware margin
(LDAM) loss. We set γ = 1 for Folcal, m = 0.35 for CosFace, m = 0.1 for ArcFace with stable
results, and the identical hyper-parameter s is set to 5.

**Results. The experimental results of imbalanced CIFAR-10 and CIFAR-100 are reported in Ta-**
ble 6. As we can see, the class margin of the LM-Softmax loss is fairly low in the severely
imbalanced cases, while the other losses with feature and weight normalization have better performance than CE and Focal. However, their class margins are still small. With the role of the
zero-centroid regularization, the class margin has a very obvious improvement in all cases, where
the class margins are close to the optimal one (arccos(−1/9) = 96.37[◦] for imbalanced CIFAR-10,
and arccos(−1/99) = 90.57[◦] for imbalanced CIFAR-100). This conclusion holds for any choice
of λ. As for the accuracy, there are also good improvements in most cases, especially for imbalanced CIFAR-100. Moreover, compared with the performance of NormFace, it is worth noticing
that the improvements of LDAM may heavily rely on the features and prototype normalization even
if LDAM is designed for label-distribution-aware margin trade-off. As illustrated in Fig. 20-28,
the zero-centroid regularization improves the intra-class compactness, where the cosine similarities
between features and their corresponding prototypes they belong to are more concentrated around 1.
The experimental results on the task of imbalanced classification. In the class imbalanced scenario,
the stronger fitting ability of LM-Softmax however would make the learner care more about the
majority classes but neglect the minority classes. This is the reason why LM-Softmax is less stable,
which can be alleviated by applying the proposed zero-centroid regularization.

**More Comparisons. To better show the effectiveness of zero-centroid regularization, we also con-**
struct more comparison to other related works of imbalanced learning, including two-stage methods
cRT (Kang et al., 2020) and MiSLAS (Zhong et al., 2021). cRT works in a two-stage manner: firstly
learn feature representation from the original imbalanced data, and then retrain the classifier using class-balanced sampling with the first-stage representation frozen. Our proposed zero-centroid
regularization Rw can not only render zero-centroid classifier but also produce feature representations with larger margins when directly learning with imbalanced datasets. Thus, our proposed
zero-centroid regularization can benefit these two-stage methods. To verify this point, we conduct
experiments on ImageNet-LT with backbone ResNet-50 where the experimental settings follow a
recent two-stage decoupling method MiSLAS. As shown in the following table, the performance
comparison of CE and CE + Rw demonstrate that zero-centroid regularization can significantly improve the representation learning ability of the first stage. Moreover, our zero-centroid regularization
_Rw can be easily integrated into well-developed two-stage decoupling methods, such as cRT, MiS-_
LAS. As demonstrated by the following results, adding Rw into 1st stage (representation learning
only) or both stages (representation learning and classifier learning) all can improve the performance
of the original methods.


-----

Table 6: Test accuracies (acc) and class margins (mcls) on imbalanced CIFAR-10. The results with
positive gains are highlighted (where λ denotes the regularization coefficient of the zero-centroid
regularization term).

|Dataset|Imbalanced CIFAR-10|Imbalanced CIFAR-100|
|---|---|---|


|Imbalance Type|long-tailed|step|long-tailed|step|
|---|---|---|---|---|


|Imbalance Ratio|100|10|100|10|100|10|100|10|
|---|---|---|---|---|---|---|---|---|


|Metric|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|acc mcls|
|---|---|---|---|---|---|---|---|---|


|CE Focal|70.88 87.41◦ 66.30 74.14◦|88.17 79.63◦ 87.33 74.48◦|64.21 76.50◦ 60.55 63.30◦|85.06 82.24◦ 84.49 75.16◦|40.38 64.73◦ 38.04 54.67◦|60.42 66.24◦ 60.09 59.30◦|42.36 60.32◦ 41.90 55.98◦|56.88 62.83◦ 57.84 55.72◦|
|---|---|---|---|---|---|---|---|---|


|CosFace (λ = 0) CosFace (λ = 1) CosFace (λ = 2) CosFace (λ = 5) CosFace (λ = 10) CosFace (λ = 20)|69.28 58.77◦ 68.86 96.17◦ 69.40 95.61◦ 69.18 93.73◦ 68.83 92.49◦ 69.52 91.90◦|87.02 81.61◦ 87.24 96.16◦ 87.16 96.26◦ 87.34 96.24◦ 86.94 96.23◦ 87.55 95.46◦|53.64 19.78◦ 62.24 95.93◦ 62.49 95.86◦ 62.13 95.84◦ 61.99 95.35◦ 62.38 94.36◦|84.86 75.96◦ 84.98 96.26◦ 84.69 95.88◦ 85.07 96.24◦ 85.59 96.12◦ 85.15 95.88◦|34.91 4.73◦ 40.53 65.42◦ 40.53 65.13◦ 40.58 55.27◦ 40.98 80.93◦ 39.92 80.30◦|60.60 70.82◦ 60.37 84.84◦ 60.77 84.97◦ 60.34 84.47◦ 59.15 85.07◦ 59.66 83.46◦|40.36 0.76◦ 40.90 42.85◦ 40.84 42.96◦ 41.12 43.79◦ 40.97 34.65◦ 41.17 41.59◦|47.56 8.56◦ 56.50 74.41◦ 56.73 71.08◦ 57.22 75.42◦ 56.97 84.09◦ 57.97 83.93◦|
|---|---|---|---|---|---|---|---|---|


|ArcFace (λ = 0) ArcFace (λ = 1) ArcFace (λ = 2) ArcFace (λ = 5) ArcFace (λ = 10) ArcFace (λ = 20)|72.20 65.86◦ 71.69 95.08◦ 71.91 93.78◦ 72.23 92.30◦ 71.99 91.92◦ 71.75 91.42◦|89.00 85.23◦ 88.86 96.26◦ 88.78 96.24◦ 89.22 96.23◦ 88.99 94.68◦ 88.99 92.85◦|62.48 54.29◦ 63.10 95.83◦ 63.05 94.84◦ 64.38 95.01◦ 63.59 94.97◦ 63.56 93.29◦|86.32 80.51◦ 86.49 96.23◦ 86.18 96.23◦ 86.56 96.24◦ 86.65 96.23◦ 86.15 95.83◦|42.77 13.22◦ 43.97 52.75◦ 44.19 55.95◦ 44.68 56.60◦ 43.89 75.58◦ 43.55 75.28◦|63.21 67.73◦ 63.67 71.52◦ 63.54 72.68◦ 63.80 73.45◦ 63.55 82.11◦ 62.10 81.00◦|41.47 0.50◦ 44.45 0.71◦ 44.41 0.81◦ 43.79 0.61◦ 44.11 31.54◦ 44.26 32.10◦|58.89 0.37◦ 61.11 62.38◦ 60.71 0.62◦ 60.30 63.68◦ 60.44 69.63◦ 60.79 79.85◦|
|---|---|---|---|---|---|---|---|---|


|NormFace (λ = 0) NormFace (λ = 1) NormFace (λ = 2) NormFace (λ = 5) NormFace (λ = 10) NormFace (λ = 20)|72.37 62.72◦ 72.07 94.95◦ 71.92 94.29◦ 70.79 92.37◦ 72.04 91.95◦ 71.36 91.14◦|89.19 82.60◦ 89.18 96.27◦ 88.93 96.28◦ 88.84 96.17◦ 89.30 94.50◦ 89.08 93.40◦|63.69 51.00◦ 62.40 96.15◦ 63.21 96.14◦ 62.83 95.38◦ 63.45 94.75◦ 64.07 93.06◦|86.37 77.82◦ 86.46 96.29◦ 86.26 96.30◦ 86.49 96.28◦ 86.06 96.29◦ 86.50 95.94◦|43.71 16.11◦ 44.18 59.42◦ 44.20 60.39◦ 44.25 64.85◦ 43.71 74.87◦ 43.67 75.71◦|63.50 71.26◦ 63.81 79.85◦ 63.90 77.69◦ 63.60 77.74◦ 63.17 82.71◦ 62.66 82.18◦|41.93 1.36◦ 43.77 41.25◦ 44.51 36.30◦ 44.14 36.62◦ 43.61 36.47◦ 43.70 28.94◦|59.85 21.32◦ 61.04 64.55◦ 60.49 71.70◦ 60.30 73.08◦ 60.22 80.83◦ 60.16 81.66◦|
|---|---|---|---|---|---|---|---|---|


|LDAM (λ = 0) LDAM (λ = 1) LDAM (λ = 2) LDAM (λ = 5) LDAM (λ = 10) LDAM (λ = 20)|72.86 73.30◦ 72.50 96.25◦ 72.41 95.85◦ 71.99 93.83◦ 72.21 92.49◦ 72.86 91.75◦|88.92 88.19◦ 88.97 96.24◦ 89.01 96.24◦ 89.51 96.25◦ 88.92 96.18◦ 89.20 95.59◦|63.27 61.42◦ 64.31 96.10◦ 64.99 96.04◦ 64.79 96.12◦ 64.48 96.16◦ 64.66 94.55◦|87.04 85.21◦ 86.74 96.26◦ 86.55 96.28◦ 86.62 96.16◦ 86.69 96.29◦ 86.60 96.05◦|43.28 7.73◦ 44.18 71.00◦ 44.90 67.95◦ 45.23 70.96◦ 43.53 81.42◦ 43.85 79.65◦|63.62 73.19◦ 63.95 84.49◦ 64.12 85.81◦ 64.18 85.03◦ 63.05 85.62◦ 62.64 84.87◦|41.65 0.85◦ 44.14 39.14◦ 44.40 36.96◦ 43.80 40.03◦ 44.48 43.26◦ 44.17 37.66◦|58.32 6.08◦ 60.52 71.43◦ 60.83 75.22◦ 60.83 72.27◦ 60.39 83.06◦ 60.28 84.31◦|
|---|---|---|---|---|---|---|---|---|


|LM-Softmax (λ = 0) LM-Softmax (λ = 1) LM-Softmax (λ = 2) LM-Softmax (λ = 5) LM-Softmax (λ = 10) LM-Softmax (λ = 20)|65.32 4.42◦ 72.25 96.06◦ 72.57 95.83◦ 72.53 93.65◦ 73.21 92.57◦ 73.20 91.95◦|88.69 68.91◦ 88.47 96.26◦ 88.69 96.31◦ 88.60 96.26◦ 88.49 96.25◦ 89.12 95.73◦|50.47 0.45◦ 64.18 91.44◦ 65.58 93.23◦ 65.18 95.20◦ 65.91 93.84◦ 65.39 93.23◦|86.08 52.20◦ 86.66 96.14◦ 86.70 96.11◦ 87.07 96.05◦ 86.96 96.09◦ 86.95 96.03◦|41.52 4.50◦ 45.22 68.02◦ 44.90 67.90◦ 45.28 69.53◦ 44.13 78.90◦ 44.22 80.53◦|63.26 68.31◦ 63.77 81.99◦ 63.39 82.93◦ 63.60 83.32◦ 62.89 85.39◦ 63.40 83.80◦|41.53 0.47◦ 45.40 39.87◦ 45.17 38.29◦ 46.23 43.15◦ 45.06 46.69◦ 45.97 64.84◦|55.44 1.37◦ 60.57 73.19◦ 60.73 74.78◦ 60.22 74.37◦ 60.48 7.94◦ 60.23 76.77◦|
|---|---|---|---|---|---|---|---|---|



Table 7: Top-1 validation accuracy on ImageNet-LT, where * denotes that the results are borrowed
from MiSLAS, X+Rw denotes adding Rw to the corresponding stages, and the trade-off parameter
_λ is set 100. The results with positive gains are highlighted._

|Method|Many Medium Few All|
|---|---|

|CE CE+R w|66.76 36.87 7.06 43.61 68.42 39.42 10.69 45.90|
|---|---|

|cRT* cRT+mixup* cRT+mixup cRT+mixup+R (adding R for 1st stage) w w cRT+mixup+R (adding R for 1st and 2nd stage) w w|62.5 47.4 29.5 50.3 63.9 49.1 30.2 51.7 65.72 48.78 25.89 51.61 64.03 49.89 32.81 52.59 64.12 49.99 32.73 52.65|
|---|---|

|MiSLAS* MiSLAS MiSLAS+ R (adding R for 1st stage) w w MiSLAS+R (adding R for 1st and 2nd stage) w w|61.7 51.3 35.8 52.7 63.30 50.06 33.52 52.50 63.11 50.56 34.24 52.76 63.20 50.69 34.21 52.85|
|---|---|


-----

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 8: Histogram of similarities and sample margins for LM-Softmax using the zero-centroid
regularization with different λ on long-tailed imbalanced CIFAR-10 with ρ = 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.

C.4 PERSON RE-IDENTIFICATION

We conduct experiments on the task of person re-identification. Specifically, we use the off-the-shelf
baseline (Luo et al., 2019) as the main code to verify the efficiency of our proposed LM-Softmax.

**Training Details. We followed the default parameter settings and training strategy. More specif-**
ically, we train the ResNet50 with pre-trained parameters for 60 epochs. Two benchmark datasets
Market-1501 (Zheng et al., 2015) and DukeMTMC (Ristani et al., 2016) are evaluated. Moreover, all models are trained with Triplet Loss + the compared losses, including CE, ArcFace, CosFace, NormFace, and the proposed LM-Softmax. Experiments were conducted on Market-1501 and
DukeMTMC. As shown in Table 3, our proposed LM-Softmax obtains obvious improvements in
mAP, Rank@1 and Rank@5, which also exhibits significant robustness for different parameters.
In contrast, ArcFace, CosFace, and NormFace show worse performance than ours and are more
sensitive to parameter settings.

Table 8: The results on Market-1501 and DukeMTMC for person re-identification task. The best
four results are highlighted.

|Dataset|Market-1501|DukeMTMC|
|---|---|---|

|Method|mAP Rank@1 Rank@5 Rank@10|mAP Rank@1 Rank@5 Rank@10|
|---|---|---|

|Softmax|82.8 92.7 97.5 98.7|73.0 83.5 93.0 95.2|
|---|---|---|

|ArcFace (s = 10) ArcFace (s = 20) ArcFace (s = 32) ArcFace (s = 64)|67.5 84.1 92.1 94.9 79.1 90.8 96.5 98.1 80.5 92.1 97.1 98.4 80.4 92.6 97.4 98.4|37.7 58.7 72.7 77.8 61.4 78.3 88.6 91.6 66.7 82.9 91.2 93.4 67.6 83.4 91.4 94.1|
|---|---|---|

|CosFace (s = 10) CosFace (s = 20) CosFace (s = 32) CosFace (s = 64)|68.0 84.9 92.7 95.2 80.5 92.0 97.1 98.2 81.7 93.4 97.6 98.3 78.7 92.0 97.1 98.3|39.3 60.6 73.1 78.7 64.2 81.3 89.7 92.8 69.4 83.5 92.3 94.4 68.2 83.1 92.5 94.4|
|---|---|---|

|NormFace (s = 10) NormFace (s = 20) NormFace (s = 32) NormFace (s = 64)|81.2 91.6 96.3 98.0 83.2 93.5 97.9 98.8 77.5 90.0 96.9 98.3 77.5 90.0 96.9 98.3|63.7 79.3 88.5 91.0 71.6 83.8 93.3 95.1 66.2 80.2 90.5 93.8 60.1 75.2 88.1 91.7|
|---|---|---|

|LM-Softmax (s = 10) LM-Softmax (s = 20) LM-Softmax (s = 32) LM-Softmax (s = 64)|83.3 92.8 97.1 98.2 84.7 93.8 97.6 98.6 84.3 93.4 97.7 98.4 84.6 93.9 98.1 98.8|72.2 85.8 92.4 94.8 74.1 86.4 93.5 94.9 73.3 86.0 93.2 95.1 74.2 86.6 93.5 95.2|
|---|---|---|


-----

(a) µ = 0 (b) µ = 0.5 (c) µ = 1 (d) µ = 0 (e) µ = 0.5 (f) µ = 1

(g) µ = 0 (h) µ = 0.5 (i) µ = 1 (j) µ = 0 (k) µ = 0.5 (l) µ = 1

Figure 9: Histogram of similarities and sample margins for CosFace (s = 64) with/without sample
margin regularization Rsm on CIFAR-10 and CIFAR-100. (a-c) and (g-i) denote the cosine similarities on CIFAR-10 and CIFAR-100, respectively. (d-f) and (j-l) denote the sample margins on
CIFAR-10 and CIFAR-100, respectively.

C.5 FACE VERIFICATION

**Datasets. We also verify our method on the task of face verification whose performance highly**
depends on the discriminability of feature embeddings. We follow the training settings in (An et al.,
2020)[1]. The model is trained on MS1MV3 with 5.8M images and 85K ids (Guo et al., 2016) and
testing on LFW (Sengupta et al., 2016), CFP-FP [3], AgeDB-30 (Moschoglou et al., 2017) and IJBC
(Maze et al., 2018). The detailed results on IJBC-C are shown in Table 9


**Training Details We use ResNet34 as the fea-**
ture embedding model and train it on two GPUs
NVIDIA Tesla v100 with batch size 512 for all
compared methods. The compared method includes ArcFace, CosFace, NormFace, and our proposed LM-Softmax.

**Baselines and hyper-parameter settings.** We
use the baseline methods including CosFace, ArcFace, NormFace, and our proposed LM-Softmax.
For CosFace and ArcFace, we use the hyperparameters followed their original paper, i.e., s =
64 and m = 0.35 for CosFace; s = 64 and
_m = 0.5 for ArcFace; For NormFace and LM-_
Softmax, we set s = 64 and s = 32, respectively.

[1https://github.com/deepinsight/insightface/](https://github.com/deepinsight/insightface/tree/master/recognition/arcface_torch)


Table 9: Different evaluation metrics of fave verification on IJB-C. The results with positive gains are
**highlighted.**

|Method|1e-5 1e-4 AUC|
|---|---|

|ArcFace ArcFace+R sm ArcFace+R w|93.21 95.51 99.4919 93.26 95.41 99.5011 93.27 95.53 99.5133|
|---|---|

|CosFace CosFace+R sm CosFace+R w|93.27 95.63 99.4942 93.28 95.68 99.5112 93.29 95.69 99.5538|
|---|---|

|LM-Softmax LM-Softmax+R w|91.85 94.80 99.4721 93.17 95.47 99.5086|
|---|---|


-----

(a) µ = 0 (b) µ = 0.5 (c) µ = 1 (d) µ = 0 (e) µ = 0.5 (f) µ = 1

(g) µ = 0 (h) µ = 0.5 (i) µ = 1 (j) µ = 0 (k) µ = 0.5 (l) µ = 1

Figure 10: Histogram of similarities and sample margins for ArcFace (s = 20) with/without sample
margin regularization Rsm on CIFAR-10 and CIFAR-100. (a-c) and (g-i) denote the cosine similarities on CIFAR-10 and CIFAR-100, respectively. (d-f) and (j-l) denote the sample margins on
CIFAR-10 and CIFAR-100, respectively.

(a) µ = 0 (b) µ = 0.5 (c) µ = 1 (d) µ = 0 (e) µ = 0.5 (f) µ = 1

(g) µ = 0 (h) µ = 0.5 (i) µ = 1 (j) µ = 0 (k) µ = 0.5 (l) µ = 1

Figure 11: Histogram of similarities and sample margins for NormFace (s = 64) with/without
sample margin regularization Rsm on CIFAR-10 and CIFAR-100. (a-c) and (g-i) denote the cosine
similarities on CIFAR-10 and CIFAR-100, respectively. (d-f) and (j-l) denote the sample margins
on CIFAR-10 and CIFAR-100, respectively.

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 12: Histogram of similarities and sample margins for LM-Softmax using the zero-centroid
regularization with different λ on step imbalanced CIFAR-10 with ρ = 10. (a-f) denote the cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.


-----

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 13: Histogram of similarities and sample margins for CosFace using the zero-centroid regularization with different λ on long-tailed imbalanced CIFAR-10 with ρ = 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 14: Histogram of similarities and sample margins for ArcFace using the zero-centroid regularization with different λ on long-tailed imbalanced CIFAR-10 with ρ = 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 15: Histogram of similarities and sample margins for NormFace using the zero-centroid
regularization with different λ on long-tailed imbalanced CIFAR-10 with ρ = 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.


-----

(a) λ = 0 (b) λ = 1 (c) λ = 2 (d) λ = 5 (e) λ = 10 (f) λ = 20

(g) λ = 0 (h) λ = 1 (i) λ = 2 (j) λ = 5 (k) λ = 10 (l) λ = 20

Figure 16: Histogram of similarities and sample margins for LDAM using the zero-centroid regularization with different λ on long-tailed imbalanced CIFAR-10 with ρ = 10. (a-f) denote the
cosine similarities between samples and their corresponding prototypes, and (g-l) denote the sample
margins.


-----

