# OPTIMAL TRANSPORT FOR LONG-TAILED RECOGNITION WITH LEARNABLE COST MATRIX

**Hanyu Peng, Mingming Sun, Ping, Li**

Cognitive Computing Lab
Baidu Research
No.10 Xibeiwang East Road, Beijing 100193, China
10900 NE 8th St. Bellevue, Washington 98004, USA
{penghanyu,sunmingming01,liping11}@baidu.com

ABSTRACT

It is attracting attention to the long-tailed recognition problem, a burning issue that
has become very popular recently. Distinctive from conventional recognition is that
it posits that the allocation of the training set is supremely distorted. Predictably, it
will pose challenges to the generalisation behaviour of the model. Approaches to
these challenges revolve into two groups: firstly, training-aware methods, with the
aim of enhancing the generalisability of the model by exploiting its potential in the
training period; and secondly, post-hoc correction, liberally coupled with trainingaware methods, which is intended to refine the predictions to the extent possible in
the post-processing stage, offering the advantages of simplicity and effectiveness.
This paper introduces an alternative direction to do the post-hoc correction, which
goes beyond the statistical methods. Mathematically, we approach this issue from
the perspective of optimal transport (OT), yet, choosing the exact cost matrix
when applying OT is challenging and requires expert knowledge of various tasks.
To overcome this limitation, we propose to employ linear mapping to learn the
cost matrix without necessary configurations adaptively. Testing our methods
in practice, along with high efficiency and excellent performance, our method
surpasses all previous methods and has the best performance to date.

1 INTRODUCTION

Classification problems in the real world are generally challenged by the long-tailed label distribution,
i.e., having a small number of samples for a majority of labels, and a dominant number of samples for
a minority of labels (Van Horn & Perona, 2017; Buda et al., 2018; Liu et al., 2019). It is also known
as imbalanced recognition, which has been widely studied in the past decades (Cardie & Nowe, 1997;
Chawla et al., 2002; Qiao & Liu, 2009; Cui et al., 2019). These distribution biases pose a significant
challenge to predictive modeling; conceivably, models often suffer from poor generalisation and
undesirable estimation bias (Cao et al., 2019; Kang et al., 2020; Zhou et al., 2020).

Recently, a renewed interest in the problem of long-tail recognition has emerged following the context
of neural networks, as numerous publications in the literature endeavour to resolve the problem albeit
in different ways including decouple (Kang et al., 2020), meta-learning (Ren et al., 2020; Wang
et al., 2020; Li et al., 2021), post-hoc correction (Tang et al., 2020; Hong et al., 2021), etc (Liu
et al., 2019; Cao et al., 2019; Tang et al., 2020). One of the representative methods of post-hoc
correction, Logit Adjustment Menon et al. (2021), provides a statistical correction to the prediction,
receiving widespread attention for its simplicity and validity. But the downside is that it is conducted
on individual samples, the rectified marginal distribution may not satisfy the desired distribution.

Figuring out exact flaws of Logit Adjustment, our explicit modeling of the problem mathematically
turns into an equational constraint, meanwhile to minimise the difference between refined distribution
and the original one, this minimisation is motivated upon the inner-product similarity. A little further,
the resulting problem can be linked to OT. Drawing on this linkage, we develop it further by proposing


-----

a linear mapping to automatically learn cost matrix, thereby circumventing the requirement for expert
knowledge to configure this matrix. In summary, our contributions are as follows:

-  We propose an alternative direction based on convex optimisation to do post-hoc correction,
which goes beyond previous direction from the statistical view.

-  Imposing marginal distributions to align ideal ones, we derive an optimisation problem tied
to OT that is solved using Sinkhorn. More further, for better learning of the cost matrix, we
present a linear mapping enabling elegant learning with one-layer network.

-  The experimental evidence shows the high efficiency and best performance on three benchmarks. It verifies that addressing the post-hoc problem via OT is helpful and effective.

2 PRELIMINARIES

In this section, we begin with notational definition, followed by an introduction to the long-tailed
recognition problem. Finally, we briefly review the OT and Logit Adjustment Menon et al. (2021).

**NotationsN** _K:_ In what follows, for two matrices X, Y _∈_ R[N] _[×][K], we denote ⟨X, Y ⟩_ =
_n=1_ _k=1_ **_[X][nk][Y][nk][ as the Frobenius dot-product.][ δ][(][·][)][ stands for the Dirac function,][ p][(][·][)][ repre-]_**
sents the probability distribution. U (r, c) = **_P_** R[N]+ _[×][K]_ **_P 1K = r, P_** [⊺]1N = c, where 1N and
P P _{_ _∈_ _|_ _}_
**1K are N** -dimension and K-dimension vector whose elements are all 1. r and c refer to the vectors
of size N and K, U (r, c) include all matrices with row and column sums r and c respectively.

2.1 PROBLEM FORMULATION

Having a collection of training samples {(x[s]n[, y]n[s] [)][}][N]n=1[s] [, validation samples][ {][(][x]n[v] _[, y]n[v]_ [)][}][N]n=1[v] [and test]
samples {(x[t]n[, y][n][)][t][}][N]n=1[t] [for classification with][ K][ labels and input][ x][ ∈] [R][d][, long-tailed recognition]
assumes that the class-prior distribution for training data p(y[s]) is different from that for validation
data p(y[v]) and test data p(y[t]). Specifically, long-tailed recognition means the distribution p(y[s]) is
_highly skewed, that is, some classes have the dominant number of samples, while tailed labels own_
_a very small number of samples. We can use imbalance ratio to measure the skewness in training_

_max_
data set, which can be defined as R = _[N]N[ s]min[s]_ [, where][ N][ s]max [and][ N][ s]min [denote the largest and smallest]

number of samples in the training data set, respectively. In this paper, we assume that the marginal
_distribution of the test set is known, we consider it as an implicit prior knowledge to be applied._
Stepping back, even if we do not know the marginal distribution of the test dataset in advance. There
are still ways to estimate the marginal distribution of the test dataset relatively precisely, such as
methods in Hendrycks et al. (2018); Azizzadenesheli et al. (2019).

Obviously, most models trained on imbalanced training data set would suffer from extremely limited
generalisation ability. Hence the ultimate goal is to learn a model that minimises the empirical risk:


_Ns_

_L (Φ(x[s]n[)][, y]n[s]_ [)][,] (1)
_n=1_

X


(Φ (x[s]n[)][, y]n[s] [) = 1]
_J_ _Ns_


where Φ(x[s]n[)][ ∈] [R][K][ denotes logits with associated sample,][ Φ(][·][) :][ R][d][ →] [R][K][ represents the mapping]
via neural networks, L stands for the loss function, typically cross entropy for classification problem.

2.2 REMINDERS ON OPTIMAL TRANSPORT

OT is used to calculate the cost of transporting one probability measure to another. We next present a
brief introduction to OT to help us better view the long-tailed problem from an OT perspective.

For two random variables X and Y, we denote its corresponding probability measures as r and
_c. Besides, C(X, Y ) : X × Y →_ R+ stands for cost function which measures the expense of
transporting X to Y . Based on these, we can define OT distance between X and Y as


_X×Y_ _C(x, y)π(x, y)dxdy,_ (2)


_d(r, c) =_ min
**_π∈Π(r,c)_**


-----

where Π (r, c) = _[π][(][x][,][ y][)][d][y][ =][ r,]_ _X_ _[π][(][x][,][ y][)][d][x][ =][ c]_ is the joint probability measure with r

_Y_

and c. When we extend the above to the discrete situation, we consider following discrete distributions:nR R o


_pi(yj)δ(yj)_ (3)
_j=1_

X


**_r =_**


_pi(xi)δ(xi)_ **_c =_**
_i=1_

X


where pi(xi) and pi(yj) represent the probability mass to the sample xi and yj respectively. In this
context, OT distance can be expressed as:

_dM_ (r, c) = min (4)
**_P_** _U_ (r,c)[⟨][P][,][ M] _[⟩][.]_
_∈_

where M stands for the cost matrix constructed by Mij = C(xi, yj). The goal of OT is to find a
transportation matrix P that minimizes the distance dM (r, c)

As we can see, OT is a distance measure between two probability distributions under some cost
matrix (Villani, 2008). However, when we use network simplex or interior point methods to solve the
above optimisation problem, it often comes at the cost of heavy computational demands. To tackle
this issue, OT with entropy constraint is proposed to allow the optimisation at small computational
cost in sufficient smoothness (Burges et al., 2013). By adding a Lagrangian multiplier to the entropy
constraint, the new formulation can be defined as follows:

_d[λ]M_ [(][r][,][ c][) =][ ⟨][P][ λ][,][ M] _[⟩]_ where **_P_** _[λ]_ = arg minP ∈U (r,c)⟨P, M _⟩−_ _λh(P ),_ (5)

where λ ∈ [0, +∞], h(P ) = − [P]n[N]=1 _Kk=1_ **_[P][nk][ log][ P][nk][,][ d]M[λ]_** [(][r][,][ c][)][ is also known as dual-Sinkhorn]
divergence, besides, it can be calculated with matrix scaling algorithms for cheaper computational
demand. The following lemma guarantees the convergence and uniqueness of the solution.P

**Lemma 1 For λ > 0, the solution P** _[λ]_ _is unique and has the form P_ _[λ]_ = diag(u)Kdiag(v), where
**_u and v are two non-negative vectors uniquely defined up to a multiplicative factor and K = e[−][M]_** _[/λ]_
_is the element-wise exponential of −M_ _/λ._

The above lemma states the uniqueness of P _[λ]_ (Sinkhorn, 1974), and P _[λ]_ can be efficiently computed
via Sinkhorn’s fixed point iteration u, v ← **_r./Kv, c./K[⊺]u._**

2.3 A QUICK RECAP OF LOGIT ADJUSTMENT

We give a brief introduction to Logit Adjustment (Menon et al., 2021; Hong et al., 2021). For the
model Φ(·), it is trained by the standard cross-entropy loss function on imbalanced training data set,
and evaluated on test data. In this algorithm, the test logit is adjusted as follows:

Φ(x[t]n[) = Φ(][x]n[t] [)][ −] [log][ p][(][y][s][)] (6)

This simple procedure is derived from the Bayes optimal rule. It is apparent that Logit Adjustment
involves a post hoc correction on an individual sample, which does not necessarily guarantee that the
marginal distribution of the whole dataset matches the desired distribution.

3 METHODOLOGY

The first part of this section explores post-hoc correction from an OT perspective, proceeds to the
automatic learning of the cost matrix via linear mapping. Lastly, we demonstrate how it can be
achieved simply with one-layer neural network.

3.1 POST-HOC CORRECTION FORMALISED FROM AN OT PERSPECTIVE

Since Logit Adjustment applies adjustment at the individual sample level. It doesn’t assure that the
marginal distribution of the overall data set fulfils our desired distribution. In this respect, we clearly
put the constraint into an equation:

**_Y_** [⊺]1N = µ, (7)


-----

where Y ∈ R[N] _[×][K]_ indicates the refined prediction value in matrix form, µ represents the expected
distribution on the test set. Alternatively, it is desirable to preserve another characteristic of Y,
namely, remaining almost as similar to the original prediction as possible. We consider inner-product
based similarity to measure this, which is a straightforward yet useful similarity measure.

maximize _C( Z[ˆ]), Y_ _,_ (8)
**_Y_** _⟨_ _⟩_

where **_Z[ˆ] represents the original prediction in matrix form, C(·) denotes to some transformation to_** **_Z[ˆ],_**
it can be some simple function, like Logarithmic function log(z), exponential function z[α]. Here we
select − log(·) as the cost function. This choice was driven by the requirement that the cost matrix
must be positive definite, whereas the transformation of the original prediction by − log(·) satisfies
this condition. In addition, as log likelihood represents the local probability density of the associated
samples, it can also be used to substitute **_Z[ˆ] for the similarity approximation. In brief, the resulting_**
numerical form can be put in formal terms as follows:

minimize log( Z[ˆ]), Y (9)
**_Y_** _⟨−_ _⟩_

subject to **_Y_** [⊺]1N = µ, **_Y 1K = 1N_** _._ (10)

Extra constraint on Y is imposed simply cos the tuned estimation has to fulfil the basic probabilistic
requirement that its sum is one. Comparing Eq. (9-10) with Eq. (4), we can see that if we substitute
**_P with Y, and substitute r and c with 1N and µ respectively, we find that the above optimisation_**
problem is actually a special case of OT.

In preliminaries, the entropy regularised OT (EOT) is introduced. By adding entropy regularisation to
OT, the given equation can be solved efficiently by Sinkhorn algorithm. Specifically, the equation is

minimize log( Z[ˆ]), Y + λY log(Y ) (11)
**_Y_** _⟨−_ _⟩_

subject to **_Y_** [⊺]1N = µ, **_Y 1K = 1N_** _._

The associated algorithmic flow for solving Eq. (11) is outlined in detail in Algorithm 1.

**Algorithm 1: Solve OT-related algorithm efficiently in the post-hoc correction via Sinkhorn**
Algorithm.


**Input: Cost matrix M = −** log( Z[ˆ]), trade-off parameter λ, max number of iterations NT,
iteration number t, error threshold ϵ, current error σ, row and column sums r = 1N and
**_c = µ, |·| denotes the vector norm._**

**Result: Refined predictions Y**

**1 Initialise K = e[−][M]** _[/λ],uold = 1N_, v = 1K, t = 0;

**2 while t ≤** _NT and δ ≤_ _ϵ do_

**3** **_u = r./Kv;_**

**4** **_v = c./K[⊺]u;_**

**5** _σ =_ **_uold_** **_u_** ;

**6** **_uold | = u; −_** _|_

**7** _t = t + 1;_

**8 end**

**9 Output Y = diag(u)Kdiag(v)**

Assigning λ with 1, it is observed that we equate our objective function to the KL divergence, thus
illustrating the extensive nature of our approach. DARP (Kim et al., 2020a) has previously applied it
to long-tailed semi-supervised classification.

**Remark** We would like to illustrate the non-applicable scenarios of our method. Firstly, our method
requires a large number of samples for evaluation. This is because if the batch size is small, we can
not guarantee that the desired marginal distribution can be satisfied within the batch. In some online
scenarios, the sample-wise correction method is more suitable. In addition, our method assumes that
the marginal distribution is already known. We assume that it is consistent with a uniform distribution.


-----

3.2 COST FUNCTION LEARNING VIA LINEAR MAPPING

Simple functions are likely to be sub-optimal for the real data sets; this suggests the design of a better
cost function to better fit the long-tailed recognition problem. However, manually designed cost
_functions require expert knowledge in different domains. Thus, we propose to use a linear mapping to_
_automatically learn the cost function, which relieves the need to configuration._

More specifically, for predictions **_Z[˜] generated by Softmax operation via leveraging linear transforma-_**
tion matrix W, W ∈ R[K][×][K] is learned so that the following objective function is minimised.

minimize **_Y, log Z[˜]_** + λ **_Y, log Y_** _,_ (12)
**_Y ∈R_** _−⟨_ _⟩_ _⟨_ _⟩_

**_Y 1K = 1N_** **_Y_** [⊺]1N = µ, **_Z˜nk =_** exp(K **_W_** [⊺]Φ (xn))k _._
_k_ [exp(][W][ ⊺][Φ (][x][n][))]k

The resulting formula can be illustrated using a simple one-layer network of weight parameter W,

P

together with an error function in Eq. (12). We initialise W with an identity matrix and use small
learning rate to learn W . One could also absorb the term − log p(y[s]) as the fixed bias parameter into
the network. Motivated by this description, we can use a general gradient descent algorithm, such as
SGD, to optimise the error function. Taking into account the implementation in practice, the direct
calculation of Eq. (12) can be done using the Sinkhorn iterations (Burges et al., 2013; Frogner et al.,
2015; Peyré et al., 2019) in mini-batch training efficiently. Besides, We term the proposed method as
OTLM (Optimal Transport via Linear Mapping), Figure 1 illustrates the overview of OTLM.

|Col1|Col2|-|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||


**A single layer feed-**
**forward network**

**Optimal Transport** **Refined prediction**

**Cost matrix**

−()*(%&) !, #$ 
**Input logits** &%

.(/)


Figure 1: Our proposed framework OTLM. The logit Φ (x) is fed into a single-layer feed-forward
network to infer **_Z[˜]. The cost matrix M is set to −_** log Z[˜]. By solving the optimal transport problem
via Sinkhorn algorithm, we can obtain the refined prediction Y .


**Compute the gradients w.r.t W** With the optimisation in Algorithm 1, it was conducted on the
overall data set with a comparatively large number of data. Quite a different scenario now, as minibatch training is more favoured when it comes to neural networks. For this reason, the optimisation
workflow in Algorithm 1, as pointed by Peyré et al. (2019); Viehmann (2019), will have a problem of
batch stablisation. To this end, we perform logarithmic transformation to step 3-4 in Algorithm 1.

1
log u = log r log (Mv) = log r logsumexp (13)
_−_ _−_ _λ_ **_[M][ −]_** [log][ v]
 

1
log v = log c log (M [⊺]u) = log c logsumexp _._ (14)
_−_ _−_ _λ_ **_[M][ −]_** [log][ u]
 

with log-sum-exp operation logsumexp (x) = log([P]i [exp][ x][i][)][. Assuming convergence is achieved in]
the Sinkhorn’s loops, it is unnecessary for us manually to compute the derivative of the loss function
in Eq. (12) w.r.t W . As long as we make sure that the entire optimisation process is differentiable,
modern deep learning libraries can automatically implement end-to-end derivations, as it allows us to
differentiate between the results of Sinkhorn’s loops as a mere composition of elementary operations.

Performance degradation caused by imbalanced distributions can be addressed with meta-learning
based methods (Ren et al., 2020; Wang et al., 2020; Li et al., 2021). Recent works have illustrated neural networks can learn more meaningful representations from the validation data set {(x[v]n[, y]n[v] [)][}][N]n=1[v] [.]
We also take a validation data set to optimise the parameter W, but with the still significant difference
_in that we require no labelling information, thus avoiding a large expanse of labeling samples._


-----

4 EXPERIMENTS

In this section, we first conduct experiments comparing our approach versus extant post-hoc correction
methods on three data sets, including CIFAR-100-LT (Cao et al., 2019), ImageNet-LT (Liu et al.,
2019), and iNaturalist (Horn et al., 2018) with varying backbones. Finally, we empirically make a
comparison of our algorithm with alternative cutting-edge long-tailed recognition methods. Observe
that, for one thing, a plausible coupling of our algorithm to any training-aware long-tailed recognition
_method exists. As an illustration of the potency and strong generalisation of our approach, we_
conducted post-hoc correction of the pseudo-predictions of both methods, which are fairly typical
for the training stage of all data sets: The first is based on the cross entropy (CE), and the other is
the recently proposed RIDE (Wang et al., 2021), which is based on multi experts. We conducted
_experiments in the appendix for semi-supervised learning, to some extent to imitate the online_
_situation. All our experiments are implemented in the PaddlePaddle deep learning platform._

4.1 DATA SETS AND BASELINES

The baseline methods and data sets are briefly described here, with implementation details placed in
appendix. We assume that the marginal distribution is uniform due to the characteristics of data sets.

**Baselines: We compare our methods with (i) post-hoc correction methods including Logit Adjust-**
ment (Menon et al., 2021), DARP (Kim et al., 2020a), (ii) state-of-the-art methods including Focal
Loss (Lin et al., 2017), LDAM (Cao et al., 2019), BBN (Zhou et al., 2020), Balanced Softmax (Ren
et al., 2020), Causal Norm (Tang et al., 2020), LADE (Hong et al., 2021), M2m (Kim et al., 2020b),
Decouple (Kang et al., 2020), LFME (Xiang et al., 2020), RIDE (Wang et al., 2021)

**Long-tailed data set: We take experiments on three data sets including CIFAR-100-LT, ImageNet-**
LT, and iNaturalist. We build the imbalanced version of CIFAR-100 by downsampling samples per
class following the profile in Liu et al. (2019); Kang et al. (2020) with imbalanced ratios 10, 50, and
100. For all the benchmarks, we evaluate the performance on the test data set using a model trained
on the training data set, and report the results using top-1 accuracy.

4.2 MAIN RESULTS

Table 1 illustrates the results on CIFAR-100-LT data set. For CE-based methods, it shows that
Logit Adjustment is indeed a simple yet effective method for post-hoc correction. Remarkably, it
outperforms the baseline methods by 2.6%, 4.1%, and 4.1% under the imbalance ratio of 10, 50, and
100 respectively. With a quick look at the results based on the OT approach, the superior results stress
the advantages of our approach over the DARP and Logit adjustments. OT algorithm can further
improve the performance by 0.62%, 0.43%, 0.70%, for OTLM, it outperforms Logit Adjustment by
0.55%, 0.73%, 0.99% under the three imbalance ratios on CIFAR-100-LT data set. For RIDE-based
training-aware methods, there is also a 0.6% and 1.23% improvement in the accuracy of our method.

Table 1: Comparison on the top-1 accuracy with post-hoc correction methods on CIFAR-100-LT data
set using ResNet-32 backbone, ImageNet-LT and iNaturalist data set using ResNeXt-50-32x4d and
ResNet-50 backbones respectively. Better results are marked in bold, the small red font indicates the
increase in accuracy when our method is compared to Logit Adjustment and RIDE. ’-’ means the
results are not reported. CE indicates that a cross-entropy loss function is used in the training stage.

Data set CIFAR-100 ImageNet-LT iNaturalist

Imabalced Ratio 10 50 100 -  - 

Baseline (CE) 59.00 45.50 41.00 43.10 65.00
Logit Adjustment+CE 61.63 49.62 45.11 51.62 69.44

DARP+CE 61.78 49.52 45.13 51.20 71.18
OT+CE 62.25+0.62 50.05+0.43 45.81+0.70 51.87+0.25 71.20+1.76

OTLM+CE 62.18+0.55 **50.35+0.73** **46.10+0.99** **52.35+0.73** **72.30+2.86**


RIDE -  -  50.20 57.50 72.90

OT+RIDE -  -  50.82+0.62 58.45+0.95 75.85+2.95

OTLM+RIDE -  -  **51.43+1.23** **58.71+1.21** **76.12+3.22**


-----

Table 1 also provides the results on ImageNet-LT and iNaturalist data sets. The fact again attracts
our attention that for CE-based training-aware methods, OT, which is based on convex optimisation,
consistently outperforms Logit Adjustment by a large margin (1.76%) on iNaturalist data set. The
accuracy boosting indicates the great potential of methods based on optimisation for post-hoc correction. Besides, not surprisingly, OTLM can further enhance the prediction accuracy, it outperforms
Logit Adjustment by 0.73% and 2.86% on ImageNet-LT and iNaturalist respectively. As for RIDE,
the gain in accuracy is also significant, about 3% on iNaturalist.

4.3 COMPARISON WITH THE STATE-OF-THE-ART METHODS

Armed with a demonstration of the validity of OT and OTLM, we switched to a comparison of our
performance with existing methods that achieved the most advanced results on three benchmarks. As
shown in Table 2, in particular, on ImageNet-LT and iNaturalist data sets, the experimental results are
remarkable and impressive, our method outperforms RIDE by 1.0% and 3.0%, respectively. Since
_iNaturalist is a very difficult and fine-grained data set consisting of 8,142 categories. These huge_
_performance gains come at a fraction of the cost, as we show in the following subsection._

Table 2: Comparison on the top-1 accuracy with state-of-the-arts on CIFAR-100-LT data set using
ResNet-32 backbone, on ImageNet-LT and iNaturalist data set using ResNeXt-50-32x4d and ResNet50 backbones respectively. − denotes the results are not reported, results underlined are the ones
being compared, best results are marked in bold, the small red font denotes performance gain.

Data Set CIFAR-100 ImageNet-LT iNaturalist

Imabalced Ratio 10 50 100 _−_ _−_

Cross Entropy (CE) 55.7 45.5 38.3 44.4 61.7
Focal Loss 55.8 44.3 38.4 43.7 _−_
CB-Focal _−_ _−_ _−_ _−_ 61.1
OLTR _−_ _−_ _−_ 46.3 63.9
NCM _−_ _−_ _−_ 47.3 63.1
norm _−_ _−_ _−_ 49.4 69.3
cRT _−_ _−_ _−_ 49.6 67.6
LWS _−_ _−_ _−_ 49.9 69.5
LDAM _−_ _−_ _−_ _−_ 64.6
LDAM+DRW 58.7 46.6 42.0 _−_ 68.0
BBN 59.1 47.0 42.6 _−_ 69.3
Causal Norm 59.6 50.3 44.1 51.8 _−_
M2m 58.2 _−_ 43.5 _−_ _−_
LFME _−_ _−_ 43.8 _−_ _−_
Balanced Softmax 61.6 49.9 45.1 _−_ _−_
LADE 61.7 50.5 45.4 51.9 70.0
RIDE (6 experts) _−_ _−_ 50.2 57.5 72.9

Logit Adjustment 61.6 49.6 45.1 51.6 69.4
DARP 61.8 49.5 45.1 51.2 71.2
OT+CE **62.3+0.7** 50.1 45.8 51.9 71.2
OTLM+CE 62.2 **50.4+0.8** 46.1 52.4 72.3
OT+RIDE (6 experts) _−_ _−_ **50.8+0.6** **58.5+1.0** **75.9+3.0**

OTLM+RIDE (6 experts) _−_ _−_ **51.4+1.2** **58.7+1.2** **76.1+3.2**


4.4 COMPUTATION COST OF OT AND OTLM

As we have highlighted, the additional computational cost of OT and OTLM is particularly small,
compared to that of training-aware methods. In Table 3, exact time of the evaluations on the ImageNetLT and iNaturalist data sets is provided. Please note, the time here are measured from the start of
each method to the final best performance. For the comparison between OT and OTLM, because
OTLM runs on the GPU, one optimisation iteration of OTLM is in fact much faster than OT. Except
for OTLM, which was run on an NVidia card (V100), the results come from a 28-core machine
(2.20 Ghz Xeon). Firstly, we can observe that all post-hoc correction methods here are not quite
_time-consuming. For comparison, ImageNet-LT and iNaturalist can be trained on 4 NVidia cards_


-----

Table 3: Time for different methods to execute on the ImageNet-LT and iNaturalist data sets. The
times here are measured from the start of each method to the final best performance. Their running
time are counted in seconds. Coupled with the prior results, an observation can be made that OT can
be the best coordinate to trade off performance and efficiency in post-hoc correction methods.

Data Set ImageNet-LT iNaturalist

Logit Adjustment 0.3 1.2
OT 32.7 82.5
OTLM 134.4 256.5

(V100) for extremely long periods of time. Using ResNeXt-50-32x4d on the iNaturalist training data
set, for example, with a batch size of 256, the amount of training time (in seconds) to perform 1
iteration is approximately 850. Also, if one differs the performance of each method, we gather that
_OT and OTLM provide the best balance of performance and running costs to match._

4.5 CONVERGENCE BEHAVIOR OF OTLM

To verify that our adopted OTLM algorithm is really efficient. With ResNeXt-50-32x4d as the
backbone, we plotted curves depicting the decreasing absolute error σ with the number of iterations t
on the ImageNet-LT validation data set. Image on the left in Figure 2 gives us a visualisation of the
optimisation process. It is found that the absolute error σ converges at around 60 iterations. In fact,
we discovered that 10 or even fewer iterations were enough to yield satisfactory performance.

Figure 2: The left image shows the convergence of OTLM: The absolute error σ decreases with
iterations on the ImageNet-LT validation data set (20k samples), and we used ResNet-50 as the
backbone to obtain the original predictions Z. The right image demonstrates the block in W, we
trained on CIFAR-100 to produce the original predictions with OTLM to minimise the difference
between the ideal and actual distributions. For ease of display, the values of the diagonal elements
are relatively large and we will replace these values with zeros.

4.6 DETAILED ANALYSIS ON DISTRIBUTION MATCHING

Upon distribution fitting, preferably the estimation bias is lessened, which implies that the KL
divergence distance between the refined prediction and the target distribution is decreased. Table 4
displays the KL distances on the ImageNet-LT and iNaturalist data sets, and we can find that OT
possesses a smaller KL distance than Logit Adjustment, which suggests that OT decreases the
estimation bias more than Logit Adjustment by a much larger margin.

4.7 VISUALISATION ON WEIGHT PARAMETER IN OTLM

As a natural consequence, one wonders about the exact structure of this weight matrix W, and no
better way to present its results than through visualisation. Towards this end, we applied OTLM


-----

Table 4: KL distance between refined probability distribution and target distribution computed on
ImageNet-LT and iNaturalist data sets. Smaller distance indicates two distributions are more similar.

Data Set ImageNet-LT iNaturalist

Logit Adjustment 1.78e-05 5.81e-06
DARP 8.49e-08 1.07e-08
OT 8.62e-10 4.72e-10

to refine the original predictions on CIFAR-100 data set with an imbalanced ratio R = 100 and
ResNet-32 backbone, and trained a one-layer model until it converged to extract the value of W,
where the shape of is 100 × 100. We visualised a sub-block of size 20 × 20, the illustration is
presented on the right image in Figure 2. The visualised matrix is very dense, this non-sparsity
suggests that the OTLM manages to correct the predictions of the model in the training stage, whilst
minimising the distance between the desired and the actual distribution.

5 RELATED WORK

**Long-tailed recognition: The great majority of current methods applied to long-tailed recognition**
can be divided into either training-aware and post-hoc correction methods. The training-aware
methods aim to improve the generalisation ability of models trained on imbalanced training data set.
Among them, re-sampling and re-weighting are two major methods. Re-sampling controls the class
numbers by the means of over-sampling (Chawla et al., 2002) and under-sampling (Barandela et al.,
2004; Drummond et al., 2003). The re-weighting approaches achieve balanced learning by giving
smaller weights to majority classes and larger weights to minority classes (Tang et al., 2008; Zadrozny
et al., 2003; Lin et al., 2017; Lee et al., 2017; Cui et al., 2019; Khan et al., 2019). Other approaches
have also been proposed to resolve the imbalanced problem. For example, Cao et al. introduces a
theoretically-principled loss function motivated by minimizing the generalisation error bound (Cao
et al., 2019), Kang et al. proposes to decouple the learning phases into representation learning and
classifier tuning (Kang et al., 2020). Apart from training-aware methods, post-hoc correction also
offers us an avenue for long-tailed recognition problem, it can be applied to the normalisation of
classifier weights (Kang et al., 2020; Zhang et al., 2019; Kim & Kim, 2020) or logits based on label
frequency (Provost, 2000; Collell et al., 2016). Among them, Logit Adjustment (Menon et al., 2021)
is a simple yet powerful method to use. Compared with Logit Adjustment, our approach performs the
post-hoc correction from the optimisation view.

**Optimal transport: OT distance is a important family of distances for probability measures (Villani,**
2008; Santambrogio, 2015). It has been extensively applied in different fields in machine learning.
Courty et al. (Courty et al., 2016) proposes a regularised unsupervised optimal transportation model
to perform the domain adaption. Other applications include 3D shape matching (Su et al., 2015),
generative model (Arjovsky et al., 2017; Salimans et al., 2018; Bunne et al., 2019; Deshpande
et al., 2019; Bhagoji et al., 2019; An et al., 2021), graph matching (Xu et al., 2019a;b), model
designs Kandasamy et al. (2018); Chizat & Bach (2018). To the best of our knowledge, we are the
first to perform the post-hoc correction from the OT perspective.

6 CONCLUSION

In this paper, we present two techniques for post-hoc corrections from an optimisation perspective.
Our approach reviews the basic mathematical formulation of the distribution alignment problem
and relates the resulting formulation to OT, which we find, moreover, offers greater possibilities
and flexibility to solve the problem. In order to avoid manual configuration of the cost matrix, we
propose to learn the cost matrix automatically by linear mapping, which leads to further performance
improvements. The experimental results confirm the superiority of our proposed approach. We
believe our work can open up more possibilities and imagination for post-hoc corrections. There
are potential challenges that remain to be addressed. These include, for example, how to apply OT
to correction of predictions under unknown marginal distribution. Meanwhile, an extension of our
approach to other applications where label shift is involved is considered possible.


-----

REFERENCES

Dongsheng An, Jianwen Xie, and Ping Li. Learning deep latent variable models by short-run MCMC
inference with optimal transport correction. In Proceedings of the IEEE Conference on Computer
_Vision and Pattern Recognition (CVPR), pp. 15415–15424, virtual, 2021._

Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In
_Proceedings of the International Conference on Machine Learning (ICML), pp. 214–223, Sydney,_
Australia, 2017.

Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning
for domain adaptation under label shifts. In Proceedings of International Conference on Learning
_Representations (ICLR), New Orleans, LA, 2019._

Ricardo Barandela, Rosa Maria Valdovinos, José Salvador Sánchez, and Francesc J. Ferri. The
imbalanced training sample problem: Under or over sampling? In Proceedings of the Joint IAPR
_International Workshops on Structural, Syntactic, and Statistical Pattern Recognition (SSPR and_
_SPR), pp. 806, Lisbon, Portugal, 2004._

Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness
from optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), pp.
7496–7508, Vancouver, BC, Canada, 2019.

Paula Branco, Luís Torgo, and Rita P. Ribeiro. A survey of predictive modeling on imbalanced
domains. volume 49, pp. 1–50. ACM New York, NY, USA, 2016.

Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance
problem in convolutional neural networks. Neural Networks, 106:249–259, 2018.

Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative
models across incomparable spaces. In Proceedings of International Conference on Machine
_Learning (ICML), pp. 851–861, Long Beach, CA, 2019._

Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger. Sinkhorn
distances: Lightspeed computation of optimal transport. In Advances in neural information
_processing systems (NeurIPS), pp. 2292–2300, Lake Tahoe, Nevada, United States, 2013._

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced
datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing
_Systems (NeurIPS), pp. 1565–1576, Vancouver, Canada, 2019._

Claire Cardie and Nicholas Nowe. Improving minority class prediction using case-specific feature
weights. In Proceedings of the International Conference on Machine Learning (ICML, pp. 57–65,
Nashville, Tennessee, USA, 1997. Morgan Kaufmann.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic
minority over-sampling technique. Journal of artificial intelligence research (JMLR), 16:321–357,
2002.

Lénaïc Chizat and Francis R. Bach. On the global convergence of gradient descent for overparameterized models using optimal transport. In Annual Conference on Neural Information
_Processing Systems, (NeurIPS), pp. 3040–3050, Montréal, Canada, 2018._

Guillem Collell, Drazen Prelec, and Kaustubh R. Patil. Reviving threshold-moving: a simple plug-in
bagging ensemble for binary and multiclass imbalanced data. arXiv preprint arXiv:1606.08698,
2016.

Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for
domain adaptation. IEEE transactions on pattern analysis and machine intelligence (PAMI), 39(9):
1853–1865, 2016.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based
on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and
_Pattern Recognition (CVPR), pp. 9268–9277, Long Beach, CA, USA, 2019._


-----

Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen
Zhao, David A. Forsyth, and Alexander G. Schwing. Max-sliced wasserstein distance and its use
for gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), pp. 10648–10656, Long Beach, CA, USA, 2019._

Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why undersampling beats over-sampling. In Workshop on learning from imbalanced datasets II, volume 11,
pp. 1–8. Citeseer, 2003.

Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, and Tomaso A. Poggio.
Learning with a wasserstein loss. In Advances in neural information processing systems (NeurIPS),
pp. 2053–2061, Montreal, Quebec, Canada, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), pp. 770–778, Las Vegas, NV, USA, 2016._

Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train
deep networks on labels corrupted by severe noise. In Advances in neural information processing
_systems (NeurIPS), pp. 10477–10486, Montréal, Canada, 2018._

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang.
Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE
_Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6626–6636, virtual, 2021._

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig
Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection
dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), pp. 8769–8778, Salt Lake City, UT, USA, 2018._

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for
imbalanced classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern
_Recognition (CVPR), pp. 5375–5384. IEEE Computer Society, 2016._

Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric P. Xing.
Neural architecture search with bayesian optimisation and optimal transport. In Annual Conference
_on Neural Information Processing Systems, (NeurIPS), pp. 2020–2029, Montréal, Canada, 2018._

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis
Kalantidis. Decoupling representation and classifier for long-tailed recognition. 2020.

Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right
balance with uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern
_Recognition (CVPR), pp. 103–112, Long Beach, CA, USA, 2019._

Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. 8:
81674–81685, 2020.

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in
_Neural Information Processing Systems (NeurIPS), pp. 1565–1576, Vancouver, Canada, 2020a._

Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via majorto-minor translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern
_Recognition (CVPR), pp. 13896–13905, Seattle, WA, USA, 2020b._

Wonji Lee, Chi-Hyuck Jun, and Jong-Seok Lee. Instance categorization by support vector machines
to adjust weights in adaboost for imbalanced data classification. Information Sciences, 381:92–103,
2017.

Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug:
Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE
_Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5212–5221, virtual, 2021._


-----

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), pp. 2999–3007, Venice,Italy, 2017._

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale
long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer
_Vision and Pattern Recognition (CVPR), pp. 2537–2546, Long Beach, CA, USA, 2019._

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and
Sanjiv Kumar. Long-tail learning via logit adjustment. In Proceedings of International Conference
_on Learning Representations (ICLR), Virtual Event, Austria, 2021. OpenReview.net._

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data
science. Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.

Foster Provost. Machine learning from imbalanced data sets 101. In Proceedings of the AAAI’2000
_workshop on imbalanced data sets, volume 68, pp. 1–3. AAAI Press, 2000._

Xingye Qiao and Yufeng Liu. Adaptive weighted learning for unbalanced multicategory classification.
_Biometrics, 65(1):159–168, 2009._

Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced
meta-softmax for long-tailed visual recognition. In Advances in neural information processing
_systems (NeurIPS), virtual, 2020._

Tim Salimans, Han Zhang, Alec Radford, and Dimitris N. Metaxas. Improving gans using optimal transport. In Proceedings of International Conference on Learning Representations, ICLR,
Vancouver, BC, Canada, 2018. OpenReview.net.

Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94,
2015.

Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. ii.
_Proceedings of the American Mathematical Society, 45(2):195–198, 1974._

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus
Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with
consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS),
virtual, 2020.

Zhengyu Su, Yalin Wang, Rui Shi, Wei Zeng, Jian Sun, Feng Luo, and Xianfeng Gu. Optimal mass
transport for shape matching and comparison. IEEE transactions on pattern analysis and machine
_intelligence (PAMI), 37(11):2246–2259, 2015._

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good
and removing the bad momentum causal effect. In Advances in Neural Information Processing
_Systems (NeurIPS), virtual, 2020._

Yuchun Tang, Yan-Qing Zhang, Nitesh V. Chawla, and Sven Krasser. Svms modeling for highly
imbalanced classification. IEEE Trans. Syst. Man Cybern. Part B, 39(1):281–288, 2008.

Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild.
_arXiv preprint arXiv:1709.01450, 2017._

Thomas Viehmann. Implementation of batched sinkhorn iterations for entropy-regularized wasserstein
loss. arXiv preprint arXiv:1907.01729, 2019.

Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media,
2008.

Renzhen Wang, Kaiqin Hu, Yanwen Zhu, Jun Shu, Qian Zhao, and Deyu Meng. Meta feature
modulator for long-tailed recognition. arXiv preprint arXiv:2008.03428, 2020.


-----

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition
by routing diverse distribution-aware experts. In Proceedings of International Conference on
_Learning Representations (ICLR), Virtual, 2021._

Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced
knowledge distillation for long-tailed classification. In Proceedings of European Conference on
_Computer Vision (ECCV), pp. 247–263, Glasgow,UK, 2020. Springer._

Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual
transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer
_Vision and Pattern Recognition (CVPR), pp. 6626–6636, Honolulu, HI, USA, 2021._

Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstein learning for graph
partitioning and matching. In Advances in Neural Information Processing Systems (NeurIPS), pp.
3046–3056, Vancouver, BC, Canada, 2019a.

Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin. Gromov-wasserstein learning for
graph matching and node embedding. In Proceedings of International Conference on Machine
_Learning (ICML), pp. 6932–6941, Long Beach, California, USA, 2019b. PMLR._

Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate
example weighting. In Third IEEE international conference on data mining (ICDM), pp. 435–442,
Melbourne, Florida, USA, 2003. IEEE.

Junjie Zhang, Lingqiao Liu, Peng Wang, and Chunhua Shen. To balance or not to balance: A simpleyet-effective approach for learning with long-tailed distributions. arXiv preprint arXiv:1912.04486,
2019.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with
cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE Conference on
_Computer Vision and Pattern Recognition (CVPR), pp. 9719–9728, Seattle, WA, USA, 2020._


-----

A OPTIMAL TRANSPORT ON LONG-TAILED SEMI-SUPERVISED
CLASSIFICATION

Together, with the extension of our approach, we adapt it to semi-supervised long-tailed classification tasks. We bring our approach together with the advanced semi-supervised learning methods
FixMatch Sohn et al. (2020). Adhering to the configuration in DARP Kim et al. (2020a), we also
create synthetically long-tailed variants of CIFAR-100 with imbalanced ratio R = 20. The sample
size of the unlabeled data is twice as large as the labelled data, with the same proportion of imbalance between them. To assess the performance, we report two popular metrics: balanced accuracy
(bACC) Huang et al. (2016) and geometric mean scores (GM) Branco et al. (2016). Experimentally,
with the exception of an OT to align the marginal distribution of the unlabelled data, all settings are
consistent with DARP, and we set λ to 0.005. The results are reported in Table 5. It can be seen
that our approach also achieves much better performance on semi-supervised learning tasks. It can
_somehow mimic the online scenarios. And the advantages of OT coping with such cases can be_
_demonstrated._

Table 5: Comparison of classification performance (bACC/GM) on CIFAR-100 with imbalanced
ratio R = 20, We report standard deviation and mean values for each evaluation metric. Better results
are marked in bold.

Method CIFAR-100

bACC GM

DARP 54.9±0.05 46.4±0.41
OT **56.6±0.08** **48.4±0.32**


B PARAMETER STUDY OF λ

We conduct an additional experiment on iNaturalist data set to study the effect of λ on the final
performance. The value of λ is selected from {0.01, 05, 0.1, 1.0.2.0, 5.0}, all other experimental
settings are the same, including various hyperparameters, network structure, etc. As illustrated in
Figure 3, we can find that OT achieves the best performance when λ = 1.

Figure 3: Visualisation of the accuracy change curve with λ on iNaturalist data set.

C IMPLEMENTATION DETAILS

The specific implementation details for each data set under the different methods are described below.

C.1 IMPLEMENTATION DETAILS OF CROSS ENTROPY

**CIFAR-100-LT: During the training stage, following the recipe of (Tang et al., 2020; Hong et al.,**
2021), we apply SGD with batch size 256 and weight decay 0.0005 to train a ResNet-32 (He et al.,
2016) model for 200 epochs, we employ the linear warm-up learning rate schedule for the first five


-----

epochs. We also set the base learning rate to 0.2 and reduce it at epoch 120 and 160 by a factor of
100. During the post-hoc stage, we use Adam with batch size 10k and weight decay 0.001 to train the
one-layer feed-forward network for 20 iterations. We set λ to 0.1, T to 200, ∆ to 0.001. The learning
rate is constant and set to 0.001.

**ImageNet-LT: During the training stage, following the implementation of (Hong et al., 2021), we**
train the ResNeXt-50-32x4d (Xie et al., 2021) for 90 epochs and perform a cosine learning rate
scheme with an initial learning rate of 0.05. SGD is also employed to optimise the neural network
with batch size 256, weight decay 0.0005, and momentum 0.9. During the post-hoc stage, we use
SGD with batch size 10k, weight decay 0.001 to train the one-layer feed-forward network for 100
iterations. The constant learning rate is 0.2. We set λ to 1.2, T to 7, ∆ to 0.001.

**iNaturalist: During the training stage, ResNet-50 (He et al., 2016) is chosen as the backbone network.**
We use SGD with momentum 0.9, batch size 256 to train the network. We utilise the cosine learning
rate schedule gradually decaying from 0.1 to 0. During the post-hoc stage, we use SGD with batch
size 23,826, weight decay 0.001, λ to 1.0, ∆ to 0.001 and T to 10. The constant learning rate is set to
0.05. As iNaturalist has no test data set, we directly report the performance on validation data set.

C.2 IMPLEMENTATION DETAILS OF RIDE

The training details we use in RIDE share the same as the original paper, there are 6 experts in our
RIDE on all three data sets.

**CIFAR-100-LT** We experimented with R = 100 and ResNet-32 as the base network. SGD is
utilised as our solver with momentum 0.9, batch size 128, and epoch 200. The learning rate is
initialised as 0.1 and decayed by 0.01 at epoch 120 and 160 respectively.

**ImageNet-LT** We take experiments with ResNeXt-50 as backbone with 100 epochs. We set the
initial learning rate to 0.1 and reduce it by 10 at epoch 60 and 80. SGD with batch size 256 and
momentum 0.9 is also adopted as the optimisation solver here.

**iNaturalist** We conduct experiments with ResNet-50 as backbone. The training details are the
same as ImageNet-LT except from batch size 512.


-----

