# UNDERSTANDING SQUARE LOSS IN TRAINING OVER## PARAMETRIZED NEURAL NETWORK CLASSIFIERS

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Deep learning has achieved many breakthroughs in modern classification tasks.
Numerous architectures have been proposed for different data structures but when
it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In
particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding
of square loss in classification by systematically investigating how it performs for
overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration
error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established
for both misclassification rate and calibration error. When classes are separable,
the misclassification rate improves to be exponentially fast. Further, the resulting
margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and
translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss
in both synthetic low-dimensional data and real image data. Comparing to crossentropy, square loss has comparable generalization error but noticeable advantages
in robustness and model calibration.

1 INTRODUCTION

The pursuit of better classifiers has fueled the progress of machine learning and deep learning research. The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provides
test fields for all kinds of new classification models, especially those based on deep neural networks
(DNN). With the introduction of CNN, ResNets, and transformers, DNN classifiers are constantly
improving and catching up to the human-level performance. In contrast to the active innovations
in model architecture, the training objective remains largely stagnant, with cross-entropy loss being
the default choice. Despite its popularity, cross-entropy has been shown to be problematic in some
applications. Among others, Yu et al. (2020) argued that features learned from cross-entropy lack
interpretability and proposed a new loss aiming for maximum coding rate reduction. Pang et al.
(2019) linked the use of cross-entropy to adversarial vulnerability and proposed a new classification
loss based on latent space matching. Guo et al. (2017) discovered that the confidence of most DNN
classifiers trained with cross-entropy is not well-calibrated.

Recently, several alternative losses have seen revived interests for deep classifiers. In particular,
many existing works have presented empirical evidence promoting the use of square loss over crossentropy. Hui & Belkin (2020) conducted large-scale experiments comparing the two and found that
square loss tends to perform better in natural language processing related tasks while cross-entropy
usually yields slightly better accuracy in image classification. Similar comparisons are also made
in Demirkaya et al. (2020). Kornblith et al. (2020) compared a variety of loss functions and output
layer regularization strategies on the accuracy and out-of-distribution robustness, and found that
square loss has greater class separation and better out-of-distribution robustness.

In comparison to the empirical investigation, theoretical understanding of square loss in training
deep learning classifiers is still lacking. Through our lens, square loss has its uniqueness among


-----

classic classification losses, and we argue that it has great potentials for modern classification tasks.
Below we list our motivations and reasons why.

**Explicit feature modeling** Deep learning’s success can be largely attributed to its superior ability

as feature extractors. For classification, the ideal features should be separated between classes and
concentrated within classes. However, when optimizing cross-entropy loss, it’s not clear what the
learned features should look like (Yu et al., 2020). In comparison, square loss uses the label codings
(one-hot, simplex etc.) as features, which can be modeled explicitly to control class separations.

**Model Calibration** An ideal classifier should not only give the correct class prediction, but also

with the correct confidence. Calibration error measures the closeness of the predicted confidence
to the underlying conditional probability ⌘. Using square loss in classification can be essentially
viewed as regression where it treats discrete labels as continuous code vectors. It can be shown that
the optimal classifier under square loss is 2⌘ _−_ 1, linear with the ground truth. This distinguishing
property allows it to easily recover ⌘. In comparison, the optimal classifiers under the hinge loss and
cross-entropy are sign(2⌘ _−_ 1) and log( 1−⌘ _⌘_ [)][, respectively. Therefore, hinge loss doesn’t provide]

reliable information on the prediction confidence, and cross-entropy can be problematic when ⌘ is
close to 0 or 1 (Zhang, 2004). Hence, in terms of model calibration, square loss is a natural choice.

**Connections to popular approaches** Mixup (Zhang et al., 2017) is a popular data augmentation

technique where augmented data are constructed via convex combinations of inputs and their labels.
Like in square loss, mixup treats labels as continuous and is shown to improve the generalization
of DNN classifiers. In knowledge distillation (Hinton et al., 2015), where a student classifier is
trying to learn from a trained teacher, Menon et al. (2021) proved that the “optimal” teacher with
the ground truth conditional probabilities provides the lowest variance in student learning. Since
classifiers trained using square loss is a natural consistent estimator of ⌘, one can argue that it is a
better teacher. In supervised contrastive learning (Khosla et al., 2020), the optimal features are the
same as those from square loss with simplex label coding (Graf et al., 2021) (details in Section 4).

Despite its lack of popularity in practice, square loss has many advantages that can be easily overlooked. In this work, we systematically investigate from a statistical estimation perspective, the
properties of deep learning classifiers trained using square loss. The neural networks in our analysis are required to be sufficiently overparametrized in the neural tangent kernel (NTK) regime.
Even though this restricts the implication of our results, it is a necessary first step towards a deeper
understanding. In summary, our main contributions are:

-  Generalization error bound: We consider two cases, according to whether classes are separable

or not. In the general non-separable case, we adopt the classical binary classification setting
with smooth conditional probability. Fast rate of convergence is established for overparametrized
neural network classifiers with Tsybakov’s noise condition. If two classes are separable with
positive margins, we show that overparametrized neural network classifiers can provably reach
zero misclassification error with probability exponentially tending to one. To the best of our
knowledge, this is the first such result for separable but not linear separable classes. Furthermore,
we bridge these two cases and offer a unified view by considering auxiliary random noise injection.

-  Robustness (margin property): When two classes are separable, the decision boundary is not

unique and large-margin classifiers are preferred. In the separable case, we further show that
the decision boundary of overparametrized neural network classifiers trained by square loss cannot be too close to the data support and the resulting margin is lower bounded away from zero,
providing theoretical guarantees for robustness.

-  Calibration error: We show that classifiers trained using square loss are inherently well-calibrated,

i.e., the trained classifier provides consistent estimation of the ground-truth conditional probability
in L norm. Such property doesn’t hold for cross-entropy.
_1_

-  Empirical evaluation: We corroborate our theoretical findings with empirical experiments in both

synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has
comparable generalization error but noticeable advantages in robustness and model calibration.

This work contributes towards the theoretical understanding of deep classifiers, from an estimation
point of view, which has been a classic topic in statistics literature. Among others, Mammen &
Tsybakov (1999) established the optimal convergence rate for 0-1 loss excess risk when the decision
boundary is smooth. Zhang (2004); Bartlett et al. (2006) extended the analysis to various surrogate


-----

losses. Audibert & Tsybakov (2007); Kohler & Krzyzak (2007) studied the convergence rates for
plug-in classifiers from local averaging estimators. Steinwart et al. (2007) investigated the convergence rate for support vector machine using Gaussian kernels. We build on and extend classic results
to neural networks in the NTK regime. Comparing to existing works on deep learning classification,
e.g., Kim et al. (2018) derived fast convergence rates of ReLU DNN classifiers that minimize the
empirical hinge loss, our results incorporate the training algorithm and apply to trained classifiers.

We require the neural network to be overparametrized, which has been extensively studied recently,
under the umbrella term NTK. Most such results are in the regression setting with a handful of exceptions. Ji & Telgarsky (2019) showed that only polylogarithmic width is sufficient for gradient
descent to overfit the training data using logistic loss. Hu et al. (2020) proved generalization error
bound for regularized NTK in classification. Cao & Gu (2019; 2020) provided optimization and
generalization guarantees for overparametrized network trained with cross-entropy. In comparison,
our results are sharper in the sense that we take the ground truth data assumptions into consideration.
This allows a faster convergence rate, especially when the classes are separable, where the exponential convergence rate is attainable. The NTK framework greatly reduces the technical difficulty for
our theoretical analysis. However, our results are mainly due to properties of the square loss itself
and we expect them to hold for a wide range of classifiers.

There are other works investigating the use of square loss for training (deep) classifiers. Han et al.
(2021) uncovered that the “neural collapse” phenomenon also occurs under square loss where the
last-layer features eventually collapse to their simplex-style class-means. Muthukumar et al. (2020)
compared classification and regression tasks in the overparameterized linear model with Gaussian
features, illustrating different roles and properties of loss functions used at the training and testing phases. Poggio & Liao (2019) made interesting observations on effects of popular regularization techniques such as batch normalization and weight decay on the gradient flow dynamics under
square loss. These findings support our theoretical results’ implication, which further strengthens
our beliefs that the essence comes from the square loss and our analysis can go beyond NTK regime.

The rest of this paper is arranged as follows. Section 2 presents some preliminaries. Main theoretical
results are in Section 3. The simplex label coding is discussed in Section 4 followed by numerical
studies in Section 5 and conclusions in Section 6. Technical proofs and details of the numerical
studies can be found in the Appendix.

2 PRELIMINARIES

**Notation** For a function f : ⌦ _! R, let kf_ _k1 = supx2⌦_ _|f_ (x)| and kf _kp = (_ ⌦ _[|][f]_ [(][x][)][|][p][d][x][)][1][/p][.]

For a vector x, **_x_** _p denotes its p-norm, for 1_ _p_ . Lp and lp are used to distinguish function
_k_ _k_ __ _1_ R
norms and vector norms. For two positive sequences {an}n2N and {bn}n2N, we write an . bn if
there exists a constantan . bn and bn . an. Let C > [ 0N such that] = {1, . . ., N an } forCbn N for all sufficiently large 2 N, I be the indicator function, and n. We write a In ⇣d be thebn if
_d ⇥_ _d identity matrix. N_ (µ, ⌃) represents Gaussian distribution with mean µ and covariance ⌃.

**Classification problem settings** Let P be an underlying probability measure on ⌦ _⇥_ **_Y, where_**

⌦ _⇢_ R[d] is compact and Y = {1, −1}. Let (X, Y ) be a random variable with respect to P . Suppose
we have observations (xi, yi) _i=1_

task is to predict the unobserved label { _}[n]_ _[⇢] y[(⌦] given a new input[⇥]_ _[Y][ )][n][ i.i.d. sampled according to] x 2 ⌦. Let ⌘_ defined on[ P] [. The classification] ⌦ denote the
conditional probability, i.e., ⌘(x) = P(y = 1|x). Let PX be the marginal distribution of P on X.
The key quantity of interest is the misclassification error, i.e., 0-1 loss. In the population level, the
0-1 loss can be written as

_L(f_ ) = E(X,Y )⇠P I{sign(f (X)) 6= Y } =EX⇠PX [(1 − _⌘(X))I{f_ (X) ≥ 0} + ⌘(X)I{f (X) < 0}],

(2.1)

where the expectation is taken with respect to the probability measure P . Clearly, an optimal classifier with the minimal 0-1 loss is 2⌘ _−_ 1.

According to whether labels are deterministic, there are two scenarios of interest. If ⌘ only takes
values from {0, 1}, i.e., labels are deterministic, we call this case the separable case[1]. Let ⌦1 =

1In the separable case we consider, the classes are not limited to linearly separable but can be arbitrarily

complicated.


-----

_{(0x,| 1)⌘(}x is non-zero, i.e., the labels contain randomness, we call this case the) = 1}, ⌦2 = {x|⌘(x) = 0} and ⌦= ⌦1 [ ⌦2. If the probability measure of non-separable case {x|⌘(x. In) 2_
the separable case, we further assume that there exists a positive margin, i.e., dist(⌦1, ⌦2) 2γ > 0,
_≥_
wherequantify the difficulty of classification, we adopt the well-established Tsybakov’s noise condition γ is a constant, and dist(⌦1, ⌦2) = inf **_x2⌦1,x02⌦2 kx −_** **_x[0]k2. In the non-separable case, to_**
(Audibert & Tsybakov, 2007), which measures how large the “difficult region” is where ⌘(x) ⇡ 1/2.
**Definition 2.1 (Tsybakov’s noise condition). Let ** _2 [0, 1]. We say P has Tsybakov noise expo-_
nent  if there exists a constant C, T > 0 such that for all 0 < t < T, PX ( 2⌘(X) 1 _< t)_ _C_ _t[]._
_|_ _−_ _|_ __ _·_

A large value of  implies the difficult region to be small. It is expected that a larger  leads to a faster
convergence rate of a neural network classifier. This intuition is verified for the overparametrized
neural network classifier trained by square loss and `2 regularization. See Section 3 for more details.

**Neural network setup** We mainly focus on the one-hidden-layer ReLU neural network family F

with m nodes in the hidden layer, denoted by

_m_

1
_fW,a(x) =_ _pm_ _arσ(Wr[>][x][)][,]_

_r=1_

X

where x 2 ⌦, W = (W1, · · ·, Wm) 2 R[d][⇥][m] is the weight matrix in the hidden layer, a =
(a1, · · ·, am)[>] _2 R[m]_ is the weight vector in the output layer, σ(z) = max{0, z} is the rectified
linear unit (ReLU). The initial values of the weights are independently generated from

**_Wr(0)_** _N_ (0, ⇠[2]Im), ar unif 1, 1 _,_ _r_ [m].
_⇠_ _⇠_ _{−_ _}_ _8_ _2_

Based on the observations {(xi, yi)}i[n]=1[, the goal of training a neural network is to find a solution to]


_l(fW,a(xi), yi) + µ_ (W, a), (2.2)
_R_


min


_i=1_


where l is the loss function, R is the regularization, and µ ≥ 0 is the regularization parameter.
Note in Equation 2.2 that we only consider training the weights W . This is because a · σ(z) =
sign(a) _σ(_ _a_ _z), which allows us to reparametrize the network to have all ai’s to be either 1 or_
_·_ _|_ _|_
_−1. In this work, we consider square loss associated with `2 regularization, i.e., l(fW,a(xi), yi) =_
(fW,a(xi) − _yi)[2]_ and R(W, a) = kW k2[2][.]

A popular way to train the neural network is via gradient based methods. It has been shown that
the training process of DNNs can be characterized by the neural tangent kernel (NTK) (Jacot et al.,
2018). As is usually assumed in the NTK literature (Arora et al., 2019; Hu et al., 2020; Bietti &
Mairal, 2019; Hu et al., 2021), we consider data on the unit sphere S[d][−][1], i.e., kxik2 = 1, 8i 2 [n],
and the neural network is highly overparametrized (m ≫ _n) and trained by gradient descent (GD)._
For details about NTK and GD in one-hidden-layer ReLU neural networks, we refer to Appendix
A. In the rest of this work, we use fW (k),a to denote the GD-trained neural network classifier under
square loss associated with `2 regularization, where k is the iteration number satisfying Assumption
D.1 and W (k) is the weight matrix after k-th iteration.

3 THEORETICAL RESULTS

In this section, we present our main theoretical results. Throughout the analysis, we assume that the
overparametrized neural network fW,a and the training process via GD satisfy Assumption D.1 (see
Appendix D), which essentially requires the neural network to be sufficiently overparametrized (with
a finite width), and imposes conditions on the learning rate and iteration number. Our theoretical
results consist of three parts: generalization error, robustness, and calibration error.

3.1 GENERALIZATION ERROR BOUND

In classification, the generalization error is typically referred to as the misclassification error, which
can be quantified by L(f ) defined in Equation 2.1. In the non-separable case, the excess risk,
defined by L(f ) _−_ _L[⇤], is used to evaluate the quality of a classifier f_, where L[⇤] = L(2⌘ _−_ 1), which
minimizes the 0-1 loss. The following theorem states that the overparametrized neural network with
GD and `2 regularization can achieve a small excess risk in the non-separable case.


-----

**Theorem 3.1 (Excess risk in the non-separable case). Suppose Assumptions D.1, D.2, and D.4**
hold. Assume the conditional probability ⌘(x) satisfies Tsybakov’s noise condition with component
_. Let µ_ _n_ 2dd−−11 . Then
_⇣_ _d(+1)_

_L(fW (k),a) = L[⇤]_ + OP(n[−] (2d−1)(+2) ). (3.1)

From Theorem 3.1, we can see that as  becomes larger, the convergence rate becomes faster, which
is intuitively true. Generalization error bounds in this setting is scarce. To the best of the authors’ knowledge, Hu et al. (2020) is the closest work (the labels are randomly flipped), where the
bound is in the order of OP(1/[p]n). Our bound is faster, especially with larger . It is known

_d(+1)_
that the optimal convergence rate under Assumptions D.2 and D.4 is OP(n[−] _d+4d−2 ) (Audibert_

& Tsybakov, 2007). The differences between Equation 3.1 and the optimal convergence rate is
that there is an extra (d 1) in the denominator of the convergence rate in Equation 3.1 (since

_d(+1)_ _−d(+1)_
_n[−]_ (2d−1)(+2) = n[−] (d−1)+d+4d−2 ). If the conditional probability ⌘ has a bounded Lipschitz con
stant, then Kohler & Krzyzak (2007) showed that the convergence rate based on the plug-in kernel
_+1_
estimate is OP(n[−] _+3+d ), which is slower than the rate in Equation 3.1 if d is large._

Now we turn to the separable case. Since ⌘ only takes value from {0, 1} in the separable case, ⌘ is
bounded away from 1/2. Therefore, one can trivially take  _! 1 in Equation 3.1 and obtain the_
convergence rate OP(n[−][d/][(2][d][−][1)]). However, this rate can be significantly improved in the separable
case, as stated in the following theorem.
**Theorem 3.2 (Generalization error in the separable case). Suppose Assumptions D.1, D.3, and D.5**
hold. Let µ = o(1). There exist positive constants C1, C2 such that the misclassification rate is 0%
with probability at least 1 − _δ −_ _C1 exp(−C2n), and δ can be arbitrarily small[2]_ by enlarging the
neural network’s width.

Note that in Theorem 3.2, the regularization parameter can take any rate that converges to zero. In
particular, µ can be zero, and the corresponding classifier overfits the training data. Theorem 3.2
states that the convergence rate in the separable case is exponential, if a sufficiently wide neural
network is applied. This is because the observed labels are not corrupted by noise, i.e., P(y = 1|x)
is either one or zero. Therefore, it is easier to classify separable data, which is intuitively true.

3.2 ROBUSTNESS AND CALIBRATION ERROR

If two classes are separable with positive margin, the decision boundary is not unique. Practitioners
often prefer the decision boundary with large margins, which are robust against possible perturbation
on input points (Elsayed et al., 2018; Ding et al., 2018). The following theorem states that the square
loss trained margin can be lower bounded by a positive constant. Recall that in the separable case,
⌦= ⌦1 [ ⌦2, where ⌦1 = {x|⌘(x) = 1} and ⌦2 = {x|⌘(x) = 0}.
**Theorem 3.3 (Robustness in the separable case). Suppose the assumptions of Theorem 3.2 are**
satisfied. Let µ = o(1). Then there exist positive constants C, C1, C2 such that for all n,

min

**_x2DT,x[0]2⌦1[⌦2_**

_[k][x][ −]_ **_[x][0][k][2][ ≥]_** _[C,]_

and the misclassification rate is 0% with probability at least 1 − _δ −_ _C1 exp(−C2n), where DT is_
the decision boundary, and δ is as in Theorem 3.2.

**Remark 1. Note that** **_x_** **_x[0]_** _pd_ **_x_** **_x[0]_** 2, thus Theorem 3.3 also indicates l robustness.
_k_ _−_ _k1 ≥_ _k_ _−_ _k_ _1_

In the non-separable case, ⌘(x) varies within (0,1) and practitioners may not only want a classifier with a small excess risk, but also want to recover the underlying conditional probability ⌘.
Therefore, square loss is naturally preferred since it treats the classification problem as a regression
problem. The following theorem states that, one can recover the conditional probability ⌘ by using
an overparametrized neural network with `2 regularization and GD training.
**Theorem 3.4 (Calibration error). Suppose the conditions in Theorem 3.1, Assumption D.3 and D.4**
are fulfilled. Let µ ⇣ _n_ 2dd−−11 . Then 1

_k(fW (k),a + 1)/2 −_ _⌘kL1 = OP(n[−]_ 4d−2 ). (3.2)

2The term δ only depends on the width of the neural network. A smaller δ requires a wider neural network.

If δ = 0, then the number of nodes in the hidden layer is infinity.


-----

Theorem 3.4 states that the underlying conditional probability in the non-separable case can be
recovered by (fW (k),a + 1)/2. The form (fW (k),a + 1)/2 is to account for the {−1, 1} label
coding. Under 0,1 coding, the estimator would be fW (k),a itself. The L consistency doesn’t
_{_ _}_ _1_ _⌘_
hold for cross-entropy trained neural networks, due to the form of the optimal solution log( 1 _⌘_ [)][.]

_−_

With limited capacity, the network’s confidence prediction is bounded away from 0 and 1 (Zhang,
2004). In practice, we want to control the complexity of the neural network thus it is usually the
case that _fW (k),a_ _< C for some constant C. Hence, it cannot accurately estimate ⌘(x) when_
_k_ _k1_
_⌘(x) >_ 1+e[C]e[C][ or][ ⌘][(][x][)][ <] 1+1e[C][, which makes the calibration error under the cross-entropy loss]

always bounded away from zero. However, square loss does not have such a problem.

Notice that the calibration error bound in Theorem 3.4 does not depend on the Tsybakov’s noise
condition, and is slower than the excess risk. This is because, a small calibration error is much
stronger than a small excess risk, since the former requires the conditional probability estimation
to be uniformly accurate, not just matching the sign of ⌘(x) − 1/2. To be more specific, a good
estimated _⌘_ can always result in a low risk plug-in classifier _f_ (x) = 2⌘(x) − 1, but not vice versa.

**Remark 2 (Technical challenge). Despite the similar forms of regression and classification using**
square loss, most of the regression analysis techniques cannot be directly applied to the classifi- b [b] b
cation problem, even if the supports of two classes are non-separable. Moreover, it is clear that
classification problems in the separable case are completely different with regression problems.

**Remark 3 (Extension on NTK). Although our analysis only concerns overparametrized one-**
hidden-layer ReLU neural networks, it can potentially apply to other types of neural networks in
the NTK regime. Recently, it has been shown that overparametrized multi-layer networks correspond to the Laplace kernel (Geifman et al., 2020; Chen & Xu, 2020). As long as the trained neural
networks can approximate the classifier induced by the NTK, our results can be naturally extended.

3.3 TRANSITION FROM SEPARABLE TO NON-SEPARABLE

The general non-separable case and the special separable case can be connected via Gaussian noise
injection. In practice, data augmentation is an effective way to improve robustness and the simplest
way is Gaussian noise injection (He et al., 2019). In this section, we only consider it as an auxiliary
tool for theoretical analysis purpose and not for actual robust training. Injecting Gaussian noise
amounts to convoluting a Gaussian distribution N (0, υ[2]Id) to the marginal distribution PX, which
enlarges both ⌦1 and ⌦2 to R[d] and a unique decision boundary Dυ can be induced. Correspondingly,
the “noisy” conditional probability, denoted as _⌘υ, is also smoothed to be continuous on R[d]. As_

_υ_ 0, _⌘υ_ _⌘_ 0 on ⌦1 and ⌦2 and the limiting _⌘0 is a piecewise constant function with_

discontinuity at the induced decision boundary. ! _k_ _−_ _k1 !_

e

**Lemma 3.5e** (Tsybakov’s noise condition under Gaussian noises) e **. Let the margin be 2γ > 0, the**
noise be N (0, υ[2]Id). Then there exist some constants T, C > 0 such that for any 0 < t < T,

_PX_ ( 2⌘υ(X) 1 _< t)_ exp _t._
_|_ _−_ _|_ __ _[Cυ]γ_ [2] _−_ 2[γ]υ[2][2] _·_

✓ ◆

e

**Theorem 3.6 (Exponential convergence rate). Suppose the classes are separable with margin 2γ >**
0. No matter how complicated ⌦1 [ ⌦2 are, the excess risk of the over parameterized neural network
classifier satisfying Assumptions D.1 and D.4 has the rate OP(e[−][nγ/][7]).

The proof of Theorem 3.6 involves taking the auxiliary noise to zero, e.g., v = vn 1/[p]n. The
exponential convergence rate is a direct outcome of Lemma 3.5 and Theorem 3.1. Note that our ⇣
exponential convergence rate is much faster than existing ones under the similar separable setting
(Ji & Telgarsky, 2019; Cao & Gu, 2019; 2020), which are all polynomial with n, e.g., OP(1/[p]n).

**Remark 4. Theorems 3.4 and 3.6 share the same gist that the over parameterized neural network**
classifiers can have exponential convergence rate when data are separable with positive margin,
while the result of Theorem 3.6 is weaker than that of Theorem 3.4, but with milder conditions.
Nevertheless, Theorem 3.6 bridges the non-separable case and separable case.


-----

4 MULTICLASS CLASSIFICATION

In binary classification, the labels are usually encoded as −1 and 1. When there are K > 2 classes,
the default label coding is one-hot. However, it is empirically observed that this vanilla square loss
struggles when the number of classes are large, for which scaling tricks have been proposed (Hui
& Belkin, 2020; Demirkaya et al., 2020). Another popular coding scheme is the simplex coding
(Mroueh et al., 2012), which takes maximally separated K points on the sphere as label features.
When K = 2, this reduces to the typical −1, 1 coding. Many advantages of the simplex coding
have been discussed, including its relationship with cross-entropy loss and supervised contrastive
learning (Papyan et al., 2020; Han et al., 2021; Graf et al., 2021; Fang et al., 2021).

In this work, we adopt the simplex coding. More discussion and empirical comparison about the
coding choices can be found in Appendix G.2. Given the label coding, one can easily generalize the
theoretical development in Section 3 by employing the following objective function


(fj,W,a(xi) − _yi,j)[2]_ + µkW k2[2][,]


min


_j=1_


_i=1_


where fW,a : ⌦ _7! R[K], and yi = (yi,1, ..., yi,K)[>]_ is the label of i-th observation.

The following proposition states a relationship between the simplex coding scheme and the conditional probability.
**Proposition 4.1 (Conditional probability). Let f** _[⇤]_ : ⌦ _! R[K]_ minimize the mean square error
EX (f _[⇤](X) −_ **_vy)[2], where vy is the simplex coding vector of label y. Then we have_**

_⌘k(x) := P (y = k|x) =_ (K − 1)f _[⇤](x)[>]vk + 1_ _/K._ (4.1)

' (

Unlike the softmax function when using cross entropy, the estimated conditional probability using
square loss is not guaranteed to be within 0 and 1. This will cause issues for adversarial attacks,
which will be discussed in detail in Appendix G.2.

5 NUMERICAL STUDIES

Although our theoretical results are for overparametrized neural network in the NTK regime, we
expect our conclusions to generalize to practical network architectures. The focus of this section is
not on improving the state-of-the-art performance for deep classifiers, but to illustrate the difference
between cross-entropy and square loss. We provide experiment results on both synthetic and real
data, to support our theoretical findings and illustrate the practical benefits of square loss in training
overparametrized DNN classifiers. Compared with cross-entropy, the square loss has comparable
generalization performance, but with stronger robustness and smaller calibration error.

5.1 SYNTHETIC DATA

We consider the square loss based and cross-entropy based overparametrized neural networks
(ONN) with `2 regularization, denoted as SL-ONN + `2 and CE-ONN + `2, respectively. The
chosen ONNs are two-hidden-layer ReLU neural networks with 500 neurons for each layer, and the
parameter µ is selected via a validation set. More implementation details are in Appendix G.1.

**Separable case** We consider two separated classes with spiral curve like supports. We also present

the performance of the cross-entropy based ONN without `2 regularization (CE-ONN). Figure 1
shows one instance of the test misclassification rate and decision boundaries attained by SL-ONN +
_`2 (Left), CE-ONN + `2 (Center), and CE-ONN (Right). From this example and other examples in_
Appendix G.1, it can be seen that SL-ONN + `2 has a smaller test misclassification rate and a much
smoother decision boundary. In particular, in the red region, where the training data are sparse,
SL-ONN + `2 fits the correct data distribution best.

**Non-separable case** We consider the conditional probability ⌘(x) = sin(p2⇡||x||2), x 2

[−1, 1][2], and the calibration performance of SL-ONN + `2 and CE-ONN + `2, where the classifiers
are denoted by _f[b]l2 and_ _f[b]ce, respectively. The results are presented in Figure G.8 in the Appendix._


-----

CE-ONN + `2 (Center); CE-ONN (Right) for the separable case.

Figure 1: Test misclassification rates and decision boundaries predicted by: SL-ONN + `2 (Left);


The error bar plot of the test calibration error shows that _fl2 has the smaller mean and standard_

deviation than _fce. It suggests that square loss generally outperforms cross entropy in calibration._

The histogram and kernel density estimation of the test calibration errors for one case show that the[b]
pointwise calibration errors on the test points of[b] _fl2 are more concentrated around zero than those_

of _fce. Moreover, despite a comparable misclassification rate with_ _fce,_ _fl2 has a smaller calibration_

error. Figure G.8 demonstrates that SL-ONN + `2 recovers[b] _⌘_ much better than CE-ONN + `2.

[b] [b] [b]

5.2 REAL DATA

To make a fair comparison, we adopt popular architectures, ResNet (He et al., 2016) and Wide
ResNet (Zagoruyko & Komodakis, 2016) and evaluate them on the CIFAR image classification
datasets, with only the training loss function changed, from cross-entropy (CE) to square loss with
simplex coding (SL). Further, we don’t employ any large scale hyper-parameter tuning and all the
parameters are kept as default except for the learning rate (lr) and batch size (bs), where we are
choosing from the better of (lr=0.01, bs=32) and (lr=0.1, bs=128). Each experiment setting is replicated 5 times and we report the average performance followed by its standard deviation in the parenthesis. (lr=0.01, bs=32) works better for the most cases except for square loss trained WRN-16-10
on CIFAR-100. More experiment details and additional results can be found in Appendix G.2.

**Generalization** In both CIFAR-10 and CIFAR-100, the performance of cross-entropy and square

loss with simplex coding are quite comparable, as observed in Hui & Belkin (2020). Cross-entropy
tends to perform slightly better for ResNet, especially on CIFAR-100 with an advantage of less than
1%. There is a more significant gap with Wide ResNet where square loss outperforms cross-entropy
by more than 1% on both CIFAR-10 and CIFAR-100. The details can be found in Table 1.

Table 1: Test accuracy on CIFAR datasets. Average accuracy larger than 0 but less than 0.1 is
denoted as 0[⇤] without standard deviation.

|Dataset|Network|Loss|Clean acc %|PGD-100 (l -strength) 1|Col6|Col7|AutoAttack (l -strength) 1|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||2/255|4/255|8/255|2/255|4/255|8/255|
|CIFAR-10|ResNet-18|CE|95.15 (0.11)|8.81 (1.61)|0.65 (0.24)|0|2.74 (0.09)|0|0|
|||SL|95.04 (0.07)|30.53 (0.92)|6.64 (0.67)|0.86 (0.24)|4.10 (0.50)|0⇤|0|
||WRN-16-10|CE|93.94 (0.16)|1.04 (0.10)|0|0|0.33 (0.06)|0|0|
|||SL|95.02 (0.11)|37.47 (0.61)|23.16 (1.28)|7.88 (0.72)|5.37 (0.50)|0⇤|0|
|CIFAR-100|ResNet-50|CE|79.82 (0.14)|2.31 (0.07)|0⇤|0|0.99 (0.10)|0⇤|0|
|||SL|78.91 (0.14)|13.76 (1.30)|4.63 (1.20)|1.21 (0.80)|3.67 (0.60)|0.16 (0.05)|0|
||WRN-16-10|CE|77.89 (0.21)|0.83 (0.07)|0⇤|0|0.42 (0.07)|0|0|
|||SL|79.65 (0.15)|6.48 (0.40)|0.42 (0.04)|0⇤|2.73 (0.20)|0⇤|0|



**Adversarial robustness** Normally trained deep classifiers are found to be adversarially vulnerable

and adversarial attacks provide a powerful tool to evaluate classification robustness. For our experiment, we consider the black-box Gaussian noise attack, the classic white-box PGD attack (Madry
et al., 2017) and the state-of-the-art AutoAttack (Croce & Hein, 2020), with attack strength level
2/255, 4/255, 8/255 in l norm. AutoAttack contains both white-box and black-box attacks and
_1_
offers a more comprehensive evaluation of adversarial robustness. The Gaussian noises results are


-----

Table 2: Performance on CIFAR-10 dataset for ResNet-18 under standard PGD adversarial training.

|CIFAR10|Loss|Acc (%)|PGD steps|Strength(l ) 1|Autoattack|
|---|---|---|---|---|---|
|ResNet-18|CE|86.87|3|8/255|37.08|
|||84.50|7|8/255|41.88|
|ResNet-18|SL|87.31|3|8/255|40.46|
|||84.52|7|8/255|44.76|



presented in Table G.3 in the Appendix. At different noise levels, square loss consistently outperforms cross-entropy, especially for WRN-16-10, with around 2-4% accuracy improvement. More
details can be found in Appendix G.2. The PGD and AutoAttack results are reported in Table 1.
Even though classifiers trained with square loss is far away from adversarially robust, it consistently
gives significantly higher adversarial accuracy. The same margin can be carried over to standard
adversarial training as well. Table 2 lists results from standard PGD adversarial training with CE
and SL. By substituting cross-entropy loss to square loss, the robust accuracy increased around 3%
while maintaining higher clean accuracy.

One thing to notice is that when constructing white-box attacks, square loss will not work well since
it doesn’t directly reflect the classification accuracy. More specifically, for a correctly classified
image (x, y), maximizing the square loss may result in linear scaling of the classifier f (x), which
doesn’t change the predicted class (see Appendix G.2 for more discussion). To this end, we consider
a special attack for classifiers trained by square loss by maximizing the cosine similarity between
_f_ (x) and vy. We call this angle attack and also utilize it for the PGD adversarial training paired
with square loss in Table 2. In our experiments, this special attack rarely outperforms the standard
PGD with cross-entropy and the reported PGD accuracy are from the latter settings. This property
of square loss may be an advantage in defending adversarial attacks.

**Model calibration** The predicted class probabilities for square loss can be obtained from Equa
tion 4.1. Expected calibration error (ECE) measures the absolute difference between predicted confidence and the actual accuracy. Deep classifiers are usually found to be over-confident (Vaicenavicius
et al., 2019). Using ResNet as an example, we report the typical reliability diagram in Figure 2. On
CIFAR-10 with ResNet-18, the average ECE for cross-entropy is 0.028 (0.002) while that for square
loss is 0.0097 (0.001). On CIFAR-100 with ResNet-50, the average ECE for cross-entropy is 0.094
(0.005) while that for square loss is 0.068 (0.005). Square loss results are much more calibrated with
significantly smaller ECE.

Figure 2: Reliability diagrams of ResNet-18 on CIFAR-10 and ResNet-50 on CIFAR-100. Square
loss trained models behave more well-calibrated while cross-entropy trained ones tend to be visibly
more over-confident.

6 CONCLUSIONS

Classification problems are ubiquitous in deep learning. As a fundamental problem, any progress
in classification can potentially benefit numerous relevant tasks. Despite its lack of popularity in
practice, square loss has many advantages that can be easily overlooked. Through both theoretical
analysis and empirical studies, we identify several ideal properties of using square loss in training
neural network classifiers, including provable fast convergence rates, strong robustness, and small
calibration error. We encourage readers to try square loss in your own application scenarios.


-----

**Ethnics Statement** We acknowledge the ICLR Code of Ethics. This submission is mostly theo
retical and the authors could not think of any potential violations of them in this submission.

**Reproducibility Statement** For our theoretical results, explanations of assumptions can be found

in Appendix D and a complete proof of the claims can be found in the Appendix E and Appendix F.
Our experiment details can be found in Appendix G, with both data and training descriptions.

REFERENCES

N. Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc., 68(3):337–404, 1950.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of

optimization and generalization for overparameterized two-layer neural networks. arXiv preprint
_arXiv:1901.08584, 2019._

Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The

_Annals of statistics, 35(2):608–633, 2007._

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds.

_Journal of the American Statistical Association, 101(473):138–156, 2006._

Colin Bennett and Robert C Sharpley. Interpolation of Operators. Academic press, 1988.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. arXiv preprint

_arXiv:1905.12173, 2019._

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and

deep neural networks. Advances in Neural Information Processing Systems, 32:10836–10846,
2019.

Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over
parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelli_gence, volume 34, pp. 3349–3356, 2020._

Lin Chen and Sheng Xu. Deep neural tangent kernel and Laplace kernel have the same RKHS.

_arXiv preprint arXiv:2009.10683, 2020._

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble

of diverse parameter-free attacks. In International Conference on Machine Learning, pp. 2206–
2216. PMLR, 2020.

Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass

classification. In 2020 54th Annual Conference on Information Sciences and Systems (CISS), pp.
1–5. IEEE, 2020.

Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct

input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637,
2018.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes

over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018.

David Eric Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators,

volume 120. Cambridge University Press, 2008.

Gamaleldin F Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large

margin deep networks for classification. arXiv preprint arXiv:1803.05598, 2018.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer
peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy
_of Sciences, 118(43), 2021._

Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and infer
ence. Econometrica, 89(1):181–213, 2021.


-----

Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Ronen Basri. On

the similarity between the Laplace and neural tangent kernels. arXiv preprint arXiv:2007.01580,
2020.

Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised con
strastive learning. In International Conference on Machine Learning, pp. 3821–3830. PMLR,
2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural

networks. In International Conference on Machine Learning, pp. 1321–1330. PMLR, 2017.

XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and

dynamics on the central path. arXiv preprint arXiv:2106.02073, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog
nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
770–778, 2016.

Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness

to improve deep neural network robustness against adversarial attack. In Proceedings of the

_IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 588–597, 2019._

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv

_preprint arXiv:1503.02531, 2015._

Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A nonparamet
ric perspective on overparametrized neural network. In International Conference on Artificial
_Intelligence and Statistics, pp. 829–837. PMLR, 2021._

Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on

noisily labeled data with generalization guarantee. In International Conference on Learning Rep_resentations, 2020._

Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross
entropy in classification tasks. arXiv preprint arXiv:2006.07322, 2020.

Arthur Jacot, Franck Gabriel, and Cl´ement Hongler. Neural tangent kernel: Convergence and gen
eralization in neural networks. In Advances in Neural Information Processing Systems, pp. 8571–
8580, 2018.

Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbi
trarily small test error with shallow ReLU networks. arXiv preprint arXiv:1909.12292, 2019.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron

Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. _arXiv preprint_

_arXiv:2004.11362, 2020._

Yongdai Kim, Ilsang Ohn, and Dongha Kim. Fast convergence rates of deep neural networks for

classification. arXiv preprint arXiv:1812.03599, 2018.

Michael Kohler and Adam Krzyzak. On the rate of convergence of local averaging plug-in classifica
tion rules under a margin condition. IEEE Transactions on Information Theory, 53(5):1735–1742,
2007.

Simon Kornblith, Honglak Lee, Ting Chen, and Mohammad Norouzi. Demystifying loss functions

for classification. 2020.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient

descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157–
8166, 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.

Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083,
2017.


-----

Enno Mammen and Alexandre B Tsybakov. Smooth discrimination analysis. The Annals of Statis
_tics, 27(6):1808–1829, 1999._

Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statisti
cal perspective on distillation. In International Conference on Machine Learning, pp. 7632–7642.
PMLR, 2021.

Youssef Mroueh, Tomaso Poggio, Lorenzo Rosasco, and Jean-Jacques Slotine. Multiclass learning

with simplex coding. arXiv preprint arXiv:1209.1360, 2012.

Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and

Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function
matter? arXiv preprint arXiv:2005.08054, 2020.

Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under

neural tangent kernel regime. arXiv preprint arXiv:2006.12297, 2020.

Atsushi Nitanda, Geoffrey Chinot, and Taiji Suzuki. Gradient descent can learn less

over-parameterized two-layer neural networks on classification problems. _arXiv preprint_

_arXiv:1905.09870, 2019._

Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax

cross-entropy loss for adversarial robustness. arXiv preprint arXiv:1905.10626, 2019.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal

phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652–24663, 2020.

Tomaso Poggio and Qianli Liao. Generalization in deep network classifiers trained with the square

loss. Technical report, CBMM Memo No, 2019.

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation

function. The Annals of Statistics, 48(4):1875–1897, 2020.

Ingo Steinwart, Clint Scovel, et al. Fast rates for support vector machines using Gaussian kernels.

_The Annals of Statistics, 35(2):575–607, 2007._

Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas

Sch¨on. Evaluating model calibration in classification. In The 22nd International Conference on

_Artificial Intelligence and Statistics, pp. 3459–3467. PMLR, 2019._

Holger Wendland. Scattered Data Approximation. Cambridge University Press, 2004.


Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and

discriminative representations via the principle of maximal coding rate reduction. Advances in
_Neural Information Processing Systems, 33, 2020._

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. _arXiv preprint_

_arXiv:1605.07146, 2016._

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical

risk minimization. arXiv preprint arXiv:1710.09412, 2017.

Tong Zhang. Statistical behavior and consistency of classification methods based on convex risk

minimization. The Annals of Statistics, pp. 56–85, 2004.


-----

