Under review as a conference paper at ICLR 2022
ENHANCING SEMI-SUPERVISED LEARNING VIA SELF-
INTERESTED COALITIONAL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Semi-supervised learning holds great promise for many real-world applications,
due to its ability to leverage both unlabeled and expensive labeled data. However,
most semi-supervised learning algorithms still heavily rely on the limited labeled
data to infer and utilize the hidden information from unlabeled data. We note
that any semi-supervised learning task under the self-training paradigm also hides
an auxiliary task of discriminating label observability. Jointly solving these two
tasks allows full utilization of information from both labeled and unlabeled data,
thus alleviating the problem of over-reliance on labeled data. This naturally leads
to a new learning framework, which we call Self-interested Coalitional Learning
(SCL). The key idea of SCL is to construct a semi-cooperative “game”, which
forges cooperation between a main self-interested semi-supervised learning task
and a companion task that infers label observability to facilitate main task train-
ing. We show with theoretical deduction its connection to loss reweighting on
noisy labels. Through comprehensive evaluation on both classiﬁcation and re-
gression tasks, we show that SCL can consistently enhance the performance of
semi-supervised learning algorithms.
1
INTRODUCTION
Despite the huge success of supervised learning with deep neural networks, they require large
amounts of labeled data for model training to achieve high performance (Meng et al., 2019; Prabhu
& Varma, 2014). Collecting labeled data can be very difﬁcult and costly in practice (Meng et al.,
2020). The challenge can be further exacerbated as in many cases, human or machine labeled data
may contain errors or noises (Natarajan et al., 2017; Schnabel et al., 2016). Semi-supervised learn-
ing (SSL) (Chapelle et al., 2006) that leverages both labeled and unlabeled data has become an
increasingly promising yet still challenging area, and is widely applicable to real-world problems.
To utilize the hidden information in the unlabeled data, a general and popular semi-supervised learn-
ing paradigm is through self-training (Yarowsky, 1995; Lee et al., 2013). Self-training uses a previ-
ously learned model to predict labels for the unlabeled data (pseudo-labeling) which are then used
for subsequent model training (Yarowsky, 1995; Lee et al., 2013; Laine & Aila, 2017; Tarvainen &
Valpola, 2017b; Iscen et al., 2019b). Despite some empirical successes, self-training methods still
suffer from two core challenges, which are over-reliance on labeled data and error accumulation.
Most existing semi-supervised learning methods (Yarowsky, 1995; Lee et al., 2013; Laine & Aila,
2017; Sohn et al., 2020; Xie et al., 2020; Kipf & Welling, 2017) assume the labeled and unlabeled
data follow the same or similar data distribution, thus the data-label mapping mechanisms learned
from the labeled data are somewhat transferable to unlabeled data. Essentially, the algorithm is still
reinforcing the information in the labeled data rather than mining additional information from the
unlabeled data. Second, the pseudo-labels of unlabeled data can be incorrectly predicted. Using such
biased information for training in subsequent epochs could increase conﬁdence in erroneous predic-
tions, and eventually leading to a vicious circle of error accumulation (Cai et al., 2013; Arazo et al.,
2020). The situation can be even worse when the labeled data contain errors or noises, as the model
learned from labeled data will make more mistakes, resulting in more severe error accumulation.
Alleviating the issue of over-reliance on labeled data requires extracting additional sources of infor-
mation from the unlabeled data. One recent approach is to introduce input consistency regularization
(Sajjadi et al., 2016). Consistency regularization enforces the stability of the predictions with respect
1
Under review as a conference paper at ICLR 2022
to the transformations of the unlabeled data (e.g. data augmentation on images such as rotation, ﬂip
and shift, etc.). Although such data perturbation or augmentation are well-deﬁned for images, they
are not directly transferable to general settings, hence the success of consistency regularization meth-
ods are mostly restricted to image classiﬁcation tasks (Sajjadi et al., 2016; Liu et al., 2019; Xie et al.,
2020; Sohn et al., 2020). We provide a new insight by noting that, under the self-training paradigm,
every semi-supervised learning task hides another auxiliary task of discriminating whether the data
label is real or a pseudo-label predicted by a machine oracle. Jointly solving the main task together
with the auxiliary task allows sufﬁcient utilization of the hidden information in unlabeled data.
To break the vicious circle of error accumulation, one needs to not always trust the machine-labeled
data. A natural idea is to consider the reliability or conﬁdence of data labels and treat them differ-
ently during training. A connected approach in supervised learning literature is to use the notion of
“soft label” with conﬁdence probability or weights, which has been veriﬁed in a number of noisy
label learning problems (Natarajan et al., 2017; Schnabel et al., 2016; Shen et al., 2020; Algan &
Ulusoy, 2021). This treatment can also help a semi-supervised learning algorithm gain the ability to
robustly learn from noisy labels.
Combining previous insights, we develop a novel semi-supervised learning framework, called Self-
interested Coalitional Learning (SCL). SCL contains three ingredients. First, it simultaneously
solves a companion task by learning a discriminator as a critic to predict label observability (whether
it is a true label or a pseudo-label), which is used to mine additional information in the unlabeled
data while facilitating main task learning. Second, the output of the discriminator is in fact a measure
of label reliability, thus can be interpreted as the conﬁdence probabilities of labels (Qin et al., 2021).
This converts the original problem into a soft label learning problem. Finally and most importantly,
SCL introduces a cooperative-yet-competitive learning scheme to boost the performance of both
the main task and the companion task. Speciﬁcally, SCL forges cooperation between both tasks by
providing extra information to each other (the main task provides prediction loss, companion task
provides label conﬁdence measures), leading to improved performance of both tasks. Moreover,
we consider the main task model to be self-interested which tries to challenge the discriminator by
providing as little information as possible. This design forms a semi-cooperative “game” with a par-
tially adversarial main task model, which can be shown theoretically equivalent to a loss reweighting
mechanism on noisy soft labels. Under this design, the main task and the companion task in SCL
are neither fully cooperative as in multi-objective optimization (Deb, 2014) nor fully adversarial to
each other as in adversarial learning (Goodfellow et al., 2014; Matyasko & Chau, 2018), but balance
the beneﬁt of cooperation and competition. In summary, the contribution of this work include:
• We develop a novel and very general self-interested coalitional learning framework for SSL that
fully utilizes the hidden information in the unlabeled data while addressing the error accumulation
issue in the self-training paradigm of SSL.
• SCL can be easily incorporated into a wide range of semi-supervised algorithms (e.g. image clas-
siﬁcation, label propagation and data imputation) with limited changes on the original method.
We show with comprehensive empirical experiments that SCL consistently achieves superior per-
formance and robustness compared with the original and self-training version of different semi-
supervised learning algorithms.
• SCL naturally provides the data conﬁdence measures from the discriminator as a byproduct of
the learning process, which offers additional interpretability of the semi-supervised learning task.
This can be particularly useful for many practical applications, while also providing extra beneﬁts
for scenarios with noisy data labels.
2
PRELIMINARIES
2.1
PROBLEM DEFINITIONS AND FORMULATIONS
We formulate the semi-supervised learning problem as learning a model f(·) with input data x ∈X
to predict the label y ∈Y . In SSL, only a partial set of data labels YL with ground truth are given in
the training set, the rest are unlabeled YU (Y = YL
S YU). For convenience of later discussion, we
denote L as the set of indices of data samples with labels and U as the set of indices for unlabeled
data. In many real-world SSL problems, the size of labeled data |L| is often limited. In extreme
cases, YL may potentially contain errors or noises. We further deﬁne a mask vector M to denote the
2
Under review as a conference paper at ICLR 2022
observability of labels over each data sample. We set Mi ∈M equals to 1 if i ∈L and Mj ∈M
equals to 0 if j ∈U. In our proposed SCL framework, we consider following common settings
of semi-supervised classiﬁcation (inductive classiﬁcation, label propagation) and regression (data
imputation) tasks:
Inductive classiﬁcation: a typical SSL task setting is to construct a classiﬁer to predict labels
y ∈Y for any object in the input space x ∈X. Common examples include semi-supervised image
classiﬁcation tasks (Sajjadi et al., 2016; Sohn et al., 2020; Xie et al., 2020), where only a subset of
data samples are given the known labels YL and the rest data labels YU are unknown.
Label propagation: another class of SSL tasks under transductive setup is to use all input samples
X and observed labels YL to train a classiﬁer to predict on unseen labels YU. Examples include
classifying nodes in a graph given only small subset of node labels YL (Kipf & Welling, 2017).
Data imputation: one of the special case of SSL problem is that the missing state of input and
output data are strongly correlated (Richardson et al., 2020; Qin et al., 2021), such as the data
imputation tasks. In data imputation, the inputs X are partially ﬁlled and the labels Y are equivalent
to a reconstructed version of X obtained through regression.
2.2
AUXILIARY TASK IN SEMI-SUPERVISED LEARNING PROBLEMS
We begin our discussion by ﬁrst noting that, if we feed the unknown labels YU with the model f
predicted labels ˜
YU = ˆ
Y (k)
U
( ˆ
Y (k)
U
= f(X) represents the labels from the k-th round of pseudo-
labeling) or other machine-generated labels, there actually hides an auxiliary task of discriminating
whether the data label is real or a pseudo-label. Denote all the data labels under pseudo-labeling as
˜
Y = YL
S ˜
YU. We can train a discriminator d(·) to tell the conﬁdence measure p ∈P of whether
x →˜
y (x ∈X, ˜
y ∈˜
Y ) is a valid mapping. This is always learnable since the ground truth of P is
exactly the observability mask M. With a slight abuse of notation, we formulate the main SSL task
A and the auxiliary companion task B as follows:
A : f(X) = ˆ
Y ,
LA = lossA(YL, ˆ
YL)
B : d(X, ˜
Y ) = P,
LB = lossB(P, M)
(1)
where lossA is the original loss function of the main task and lossB can be any binary classiﬁcation
loss between P and M, such as the binary cross entropy (BCE) loss. Jointly solving the above
two tasks allows exploiting the underlying relationship between input data X and data label Y from
another angle, which can potentially provide more information to facilitate main task training.
There are some other works that embody the similar idea of jointly learning two tasks. For ex-
ample, the multi-task style multi-objective optimization methods (Ruder, 2017; Deb, 2014) exploit
the shared information and underlying commonalities between two tasks and solve the problem by
minimizing an augmented loss. The generative adversarial learning (Goodfellow et al., 2014) makes
a generator and a discriminator learn against each other, thus improves the performance of both
tasks. However, na¨
ıvely applying these two approaches has some drawbacks. For example, multi-
objective optimization cannot handle potential contradictions of the two tasks in certain settings,
where jointly minimizing the augmented loss may impede both tasks from achieving the best per-
formance (Qin et al., 2021). Whereas under the adversarial learning framework, lossA is implicitly
optimized through the discriminator, which could result in potential loss of information. Moreover,
solving the minimax optimization problem under a GAN-style model is much harder compared with
directly minimize both loss functions explicitly in (1) in a supervised learning fashion.
3
SELF-INTERESTED COALITIONAL LEARNING
In this section, we introduce a new learning strategy to jointly solving the two tasks in Problem (1)
for semi-supervised learning, which is called self-interested coalitional learning (SCL). As its name
suggests, SCL forges cooperation between a self-interested main task and an unselﬁsh companion
task for boosted performance. We show with theoretical derivation that under the pseudo-labeling
paradigm, this special learning scheme will lead to a soft label learning formulation for the original
SSL task. We also provide an intuitive interpretation of the SCL scheme.
3
Under review as a conference paper at ICLR 2022
𝑓
𝑿
𝒀
𝒀𝐿
𝒀𝑈
𝑙𝑜𝑠𝑠𝐴
𝑙𝑜𝑠𝑠𝑊
𝑙𝑜𝑠𝑠𝐵
𝑑
𝑷
𝑴
Pseudo-labeling periodically 
Reweighting
loss
Information from the main task
Figure 1: Illustration of self-interested coalitional learning
3.1
ALTERNATIVE FORMULATION UNDER SCL
We consider an alternative formulation of Problem (1) by modifying the loss of the main task LA
and the inputs of the companion task to establish cooperation and information sharing:
A : f(X) = ˆ
Y ,
LA = lossA( ˜
Y , ˆ
Y ) + α · lossW ( ˜
Y , ˆ
Y , P)
B : d(X, ˆ
Y , g( ˜
Y , ˆ
Y )) = P,
LB = lossB(P, M).
(2)
where g(˜
y, ˆ
y) is some form of information provided by the main task. We can simply model the
information g(·) as the element-wise loss term of the main task, i.e. g(˜
y, ˆ
y) = lossA(˜
y, ˆ
y), such as
the cross entropy loss for classiﬁcation tasks and mean square error (MSE) loss for regression tasks.
Moreover, lossW ( ˜
Y , ˆ
Y , P) is a corrective additional loss term impacted by the label conﬁdence
measure information P provided by the discriminator d, and α is its weight parameter. Due to the
existence of the corrective information from lossW , we now train the main task with all data labels
˜
Y rather than only with observed labels YL, thus allows sufﬁcient utilization of information from
the unlabeled data. We illustrate the strategy of SCL in Figure 1.
The design of SCL has several interesting properties. First, since additional information are provided
to both tasks, they can be both learned better. Second, a less well-learned model f will produce large
element-wise error lossA(˜
y, ˆ
y) for a predicted label ˆ
y, which in turn provides more information in
g(·) for the discriminator d to distinguish whether a label is real or a pseudo-label. As f is learned
better during the cooperative learning process, when g no longer provides useful information for
discriminating the label observability, then we may have reason to believe f has achieved satisfactory
inference performance. Hence minimizing the information in g(·) coincides with improving the
learning of f. This suggests that if we alter the learning direction of f using lossW to encourage
it providing as little information in g(·) as possible, then we might obtain a better learned f. This
in fact forms a cooperative-yet-competitive “game” between both tasks, with the main task been
self-interested and partially competitive. Finally, it also allows the discriminator to detect errors or
noises in observed data labels. As the erroneous (x, y) data pairs may possess different x →y
mapping patterns against normal data, thus are more likely to be identiﬁed as pseudo-labels. Such
negative impact can be further corrected using lossW with label conﬁdence measure P.
3.2
DERIVATION OF THE ADDITIONAL LOSS TERM lossW
Until this point, the only mystery about SCL is the exact form of lossW that embodies the self-
interested behavior for the main task model f. We resort to calculus of variations to provide theo-
retical derivation of lossW .
Under the alternative formulation (2), both the discriminator d and its loss LB are impacted by the
information g(·) provided by f, thus they are now functionals of f. We wish to inspect how the
variation of model f impacts LB. Denote θf, θd as the model parameters of model f and d. Here,
we ﬁx the the discriminator d and its parameters θd to exclude its own inﬂuence, then d(θd, x, f)
can be replaced by d(x, f). Similarly, with d ﬁxed, we can deﬁne the functional J(f) for LB and
further eliminate the effect of changes in x on d by integrating over x, mathematically:
J(f) = LB =
Z ∂LB
∂d dx =
Z
F(x, f, f ′)dx
(3)
4
Under review as a conference paper at ICLR 2022
where F(x, f, f ′) = ∂LB/∂d is assumed to be a continuously differentiable function with respect
to x, f and f ′. Although the discriminator d aims to minimize its loss LB, as mentioned previously,
we want the main task model f to challenge the discriminator by providing as little information in
g as possible. This essentially requires changing the behavior of f such that the functional J(f)
attains its maxima. The extrema (maxima or minima) of functional J(f) can be obtained by ﬁnding
a function f where the functional derivative of J(f) is equal to zero. By Taylor expansion,
J(f) = J(fθ0
f ) + J′(fθ0
f )δf + o(δf)
(4)
where δf is the variation of f. Since o(δf) is a higher order inﬁnitesimal term at θf →θ0
f, and
assuming that o(δf) is small and ignorable, we can obtain the ﬁnite difference approximation of the
functional J(f) by plugging it into Eq.(3),
∆J(f) = J(f) −J(fθ0
f ) = J′(fθ0
f )δf = d
R
F(x, f, f ′)dx
dθf
δf
(5)
If the uniform continuity of F and its derivative is satisﬁed, we can simplify the derivative of J(f)
with respect to θf as follows, while swapping the order of dx with δf:
∆J(f) =
R
∂F(x, f, f ′)dx
∂θf
δf =
Z ∂F(x, f, f ′)
∂θf
δfdx
(6)
As f can be any function in an arbitrary function space F. Ensuring the functional derivative of
J(f) equal to zero (∆J(f) = 0) requires ∂F(x, f, f ′)/∂θf = 0, which is
∂F(x, f, f ′)
∂θf
= ∂F
∂g · ∂g
∂f · ∂f
∂θf
= ∂F
∂g · ∇θf lossA = 0
(7)
The last step derivation in above equation is due to our design of letting g(·) be the element-wise
loss function of f (g(˜
y, ˆ
y) = lossA(˜
y, ˆ
y)).
As an example, suppose we use BCE loss for LB, and write pi as the output value of
d(X, ˆ
Y , g( ˜
Y , ˆ
Y )) with index i, the Eq.(7) can be written as
∂F
∂θf
= ∂F
∂g · ∇θf lossA = −
X
i∈L
1
pi
∇θf lossA(˜
yi, ˆ
yi) +
X
j∈U
1
1 −pj
∇θf lossA(˜
yj, ˆ
yj)
(8)
Note that the result of above equation can be equivalently perceived as the gradient of a new term
Lf( ˜
Y , ˆ
Y , P) (∂F/∂θf = −∂Lf/∂θf) with following form:
−Lf( ˜
Y , ˆ
Y , P) =
X
i∈L
1
pi
· lossA(˜
yi, ˆ
yi) −
X
j∈U
1
1 −pj
· lossA(˜
yj, ˆ
yj)
(9)
Consequently, minimizing −Lf with respect to θf will drive the functional derivative of J(f) to
zero (−∂Lf/∂θf = ∂F/∂θf = 0
=
⇒
∆J(f) = 0), which also lead to the maxima of J(f)
(update with gradient direction −∂F/∂θf). We now uncover the exact form of lossW , which can be
set as lossW ( ˜
Y , ˆ
Y , P) = −Lf( ˜
Y , ˆ
Y , P) to realize the self-interested behavior of f. The resulting
lossW can be considered as an additional reweighting loss that complements the original loss term
lossA based on the estimated conﬁdence measures P from the discriminator d.
3.3
INTERPRETATION OF SCL
Adding lossW to the original loss term lossA, we can obtain the complete loss for the main task as:
LA =
X
i∈L

1 + α
pi

· lossA(yi, ˆ
yi) +
X
j∈U

1 −
α
1 −pj

· lossA(˜
yj, ˆ
yj)
(10)
This essentially transforms the original main task into a cost-sensitive learning problem (Zadrozny
et al., 2003) by imposing following soft-labeling weights on the errors of each data label as:
Soft-labeling weights =

1 + α/pi,
i ∈L
1 −α/(1 −pj),
j ∈U
(11)
5
Under review as a conference paper at ICLR 2022
We notice that the above soft-labeling weights induce very different behaviors on the loss
lossA(˜
y, ˆ
y) of labeled and pseudo-labeled samples. As pi represents the judgment of the discrimina-
tor d on whether yi is labeled or not. For labeled samples, the loss are boosted by additional weight
of α/pi, which means if the discriminator d judges an labeled samples yi ∈YL to be unlabeled or
unreliable (conﬁdence pi →0), the loss of this sample will get signiﬁcantly boosted. This forces
the main task model f to pay more attention to infer the problematic labeled samples. As unlabeled
samples yj ∈YU are periodically ﬁlled with pseudo-labels generated by f (initially ﬁlled with ran-
dom values), SCL does not fully trust these labels and may even choose to enlarge the gap with
the pseudo-label when p > 1 −α (α ≤1). Speciﬁcally, the more likely an label yj is considered
reliable by the discriminator (pj →1), the stronger it forces the learning process to ignore further
improvement on inferring yj.
Note that uniform continuity needs to be satisﬁed in Eq.(6), however, the derivative of BCE loss
will result in 1/pi and 1/(1 −pi) which can potentially take inﬁnite value, violating the continuity
assumption. We thus clip their values as min(1/pi, H) and min(1/(1 −pi), H) in our practical
algorithm, where H is a predetermined threshold (H = 10 in our implementation). Algorithm 1
presents the practical algorithmic framework of SCL.
Algorithm 1 Self-interested coalitional learning framework
Require: Training samples {X, Y } = {XL
S XU, YL
S YU}, observability mask M, weight parameter α
1: Randomly initialized θf, θd, YU, and let ˜
YU ←YU
2: while training do
3:
Inference the all data labels ˆ
Y using the main task model f
4:
Calculating the element-wise loss of the main task lossA( ˜
Y , ˆ
Y ) as g( ˜
Y , ˆ
Y )
5:
Predict the label conﬁdence measures P using the discriminator d and g( ˜
Y , ˆ
Y )
6:
Update θt+1
d
by minimizing LB in Eq.(2)
// Update d with information from main task
7:
Update θt+1
f
by minimizing LA in Eq.(10)
// Update f with information from companion task
8:
update the pseudo-label ˜
YU ←ˆ
YU
9: end while
4
EXPERIMENTS
The proposed SCL framework can be easily incorporated in a variety of SSL tasks. We choose three
representative SSL tasks and apply SCL on their ofﬁcial implementations to evaluate the perfor-
mance of SSL, including: image classiﬁcation (He et al., 2016) for inductive classiﬁcation, label
propagation (Kipf & Welling, 2017) for transductive classiﬁcation and data imputation (Gondara &
Wang, 2018) for regression.
Image classiﬁcation: We use RestNet18 for image classiﬁcation on the fashion-mnist dataset (Xiao
et al., 2017), with 6,000/10,000 training/testing set partition. Note that the training set only contains
10% of the original fashion-mnist dataset which aims to substantially increase the task difﬁculty.
Label propagation: We trained a two-layer GCN (Kipf & Welling, 2017) integrated with SCL
and evaluate the prediction accuracy on Cora dataset(Sen et al., 2008). The test set consists of 1000
labeled examples extracted from a knowledge graph with 2,708 nodes and 5,429 edges. The labeling
rate represents the number of labeled nodes used for training divided by the total number of nodes
in the training set.
Data imputation: We use a deep denoising autoencoder (DAE) (Gondara & Wang, 2018) for data
imputation task under regression setting. We use the UCI online news popularity dataset (Asuncion
& Newman, 2007) for evaluation. The dataset is split the training/testing set with the ratio of 0.8/0.2.
For all tasks, we remove certain proportion (referred as missing rate) of data labels in the training
set and treat them as unlabeled data in the semi-supervised setting. At initial step of SCL, as the
model is not yet trained to provide pseudo-labels, we ﬁll the initial pseudo-labels for unlabeled
data with random labels. For the regression task (data imputation), we compute the mean µ of the
labeled data and ﬁll the missing entries with µ adding random noise; for the classiﬁcation tasks,
we ﬁll the missing label YU by randomly selecting a data label. We present the average results
over 10 random seeds for all the experiments. The implementation details and extra comparative
experiments between SCL and several advanced SSL algorithms are presented in the Appendix.
6
Under review as a conference paper at ICLR 2022
Missing
rate (%)
Image classiﬁcation
Label propagation
Data imputation
Accuracy (%)
Accuracy (%)
MSE
ResNet18
ResNet18+SCL
GCN
GCN+SCL
DAE
DAE+SCL
00
83.85 ± 0.12
86.23 ± 0.13
-
-
-
-
10
83.57 ± 0.12
84.69 ± 0.08
82.18 ± 1.59
84.33 ± 0.56
0.8055
0.7899
20
83.06 ± 0.16
85.15 ± 0.13
82.18 ± 1.69
84.00 ± 1.81
0.9029
0.8716
30
82.89 ± 0.21
85.38 ± 0.16
81.80 ± 1.75
83.33 ± 0.68
0.9335
0.8867
40
82.43 ± 0.13
84.98 ± 0.21
82.28 ± 2.69
83.00 ± 0.95
0.9761
0.9229
50
81.94 ± 0.16
84.61 ± 0.21
81.93 ± 2.56
81.33 ± 2.28
0.9726
0.9412
60
81.23 ± 0.16
83.87 ± 0.20
80.03 ± 6.37
82.33 ± 2.10
0.9787
0.9727
70
79.70 ± 0.13
82.90 ± 0.21
77.84 ± 7.52
83.67 ± 3.73
1.1593
1.1582
80
78.84 ± 0.35
81.31 ± 0.42
74.33 ± 7.00
80.33 ± 1.00
1.1429
1.1425
90
76.02 ± 0.41
77.96 ± 0.47
69.66 ± 7.66
74.26 ± 0.53
-
Table 1: Comparison of the original models and the SCL-integrated version for different tasks (with
mean ± std). The std results in the imputation task are omitted because they are less than 1e-04.
4.1
PERFORMANCE EVALUATION OF SCL
We compared our method with the original model in three representative SSL tasks to demonstrate
the effectiveness and robustness of SCL under diverse problem setting. Results are summarized
in Table 1. We report the classiﬁcation accuracies and imputation error measured by MSE. SCL
consistently achieves superior performance over all tasks, which shows the universality of SCL
under diverse SSL problem setting. We also use 10 different levels of missing rates to verify the
robustness of SCL. By randomly dropping some labels, we increase the missing rate of the training
set from 0% (fully supervised) to 90%. As label propagation and data imputation tasks are not well-
deﬁned under fully supervised setting, only the image classiﬁcation task under 0% missing rates is
shown in the table. It is observed in Table 1 that most of the original models experience substantial
performance deterioration with the increase of missing rate. By contrast, SCL remains high level of
robustness with limited performance drop in most tasks.
Supprisingly, for the inductive image classiﬁcation task, we observe that in the case of 0% missing
rate (fully supervised), SCL can achieve even higher accuracy compared with the fully supervised
base model (accuracy: 86.2 ± 0.13% (SCL) vs 83.85 ± 0.13% (base model)). Moreover, SCL
achieves similar level of accuracy as the fully supervised base model even with 60% missing rate.
These results suggests that SCL can also be used in supervised learning problems to boost model
performance. SCL performs cost-sensitive learning for different data samples to maximally utilize
the hidden information from both labeled and unlabeled data, which lead to more stable and accurate
inference under different missing rates. For the transductive classiﬁcation task like label propaga-
tion, all the input data X and the labeled data YL are used for inference on the unobserved labels
YU. The amount of information from the labeled data plays a vital role for overall prediction accu-
racy. Hence as the missing rate becomes large, we observe a substantial performance drop for the
original GCN model. SCL, on the other hand, maintains high accuracy in spite of high missing rate
by continuously soft pseudo-labeling the data to ensure the reliable pseudo-labels are properly used
to facilitate the main task training, thus exhibits high robustness under missing information.
4.2
COMPARISON OF SCL AND SELF-TRAINING VIA PSEUDO-LABELING
To further validate the effectiveness of SCL under semi-supervised settings, we compare the per-
formance of SCL with the widely used pseudo-labeling approach in self-training (Yarowsky, 1995;
Lee et al., 2013; Laine & Aila, 2017; Iscen et al., 2019b). We compare their performances in three
different aspects, i.e., convergence speed, accuracy under different missing rates and the degree of
error accumulation under pseudo-labeling. Figure 2 shows us the convergence of image classiﬁca-
tion, label propagation, and data imputation for missing rates of 40%, 20%, and 30%, respectively.
In Figure 2, we can see that the learning speed of the main task model with the corrective reweight-
ing loss term lossW in SCL is signiﬁcantly faster than the original model. Meanwhile, since the
hard pseudo-labels in self-training are used as true label for model learning rather than carefully
reweighted as soft labels in SCL, certain degree of oscillation is observed during the learning pro-
cess, which impacts the learning speed as well as the ﬁnal performance of the model. This demon-
strates the beneﬁt of using soft-labeling weights to correct unreliable label information in SCL.
7
Under review as a conference paper at ICLR 2022
0
25
50
75
100 125 150 175 200
Epoch
74
76
78
80
82
84
Accuracy (%)
ResNet18 + SCL
ResNet18 + self-training
ResNet18
(a) Image classiﬁcation
0
30
60
90
120
150
Epoch
0
20
40
60
80
100
Accuracy (%)
GCN + SCL
GCN + self-training
GCN
(b) Label propagation
0
30
60
90
120
150
180
Epoch
0.86
0.88
0.90
0.92
0.94
0.96
0.98
1.00
1.01
1.03
MSE (Normalized)
DAE
DAE + SCL
(c) Data imputation
Figure 2: The convergence of test accuracy under different methods and tasks
20
30
40
50
60
70
80
Missing rate (%)
75
80
85
90
Accuracy (%)
ResNet18
Self-training
SCL
(a) Image classiﬁcation
20
30
40
50
60
70
80
Missing rate (%)
65
70
75
80
85
90
Accuracy (%)
GCN
Self-training
SCL
(b) Label propagation
20
30
40
50
60
70
80
Missing rate (%)
0.71
0.79
0.87
0.95
1.03
1.11
1.19
1.27
MSE (Normalized)
DAE
SCL
(c) Data imputation
Figure 3: Performance under different missing rate
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Rounds of pseudo-labeling
80
85
90
95
100
Accuracy of pseudo-labels (%)
GCN + SCL
GCN + self-training
(a) Accuracy of pseudo-labels under 20% missing rate
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Rounds of pseudo-labeling
0
20
40
60
80
100
Accuracy of pseudo-labels (%)
GCN + SCL
GCN + self-training
(b) Accuracy of pseudo-labels under 70% missing rate
Figure 4: Error accumulation of different methods in the label propagation task
Figure 3 shows the accuracy of the original, self-trained and SCL versions of models for classiﬁca-
tion tasks with different missing rates. We observe that SCL maintains high accuracy under different
missing rates and consistently outperforms the original and self-training versions. In the label prop-
agation task, although self-training increases the training samples by periodically pseudo-labeling
high conﬁdence unlabeled data, the accuracy of the model could decreases due to the introduction of
wrong labeling information, which results in subsequent error accumulation and performance drop.
To further investigate the impact of error accumulation in pseudo-labels during model training, we
record the accuracy of pseudo-labels after each round of pseudo-labeling in self-training. The test
accuracy of these samples in the SCL version of the model are also recorded for comparison. To
ensure the stability of self-training, we add pseudo-labels every 10 training epochs. Different from
self-training that only label high conﬁdence unlabeled samples, SCL introduces pseudo-labels for
all unlabeled data YU but with soft-labeling weights. Figure 4 shows the error accumulation trends
for self-training and SCL in the label propagation task. When the missing rate is low (Figure 4(a)),
self-training maintains a high accuracy at the beginning, but model performance will ﬂuctuate and
decrease with the involvement of more potentially problematic pseudo-labels. On the other hand,
when the missing rate is high and available information is scarce (Figure 4(b)), gradually involv-
ing pseudo-labeled data is beneﬁcial for self-training, resulting in slowly increasing test accuracy,
although the model still struggles to achieve high accuracy. Despite the occurrence of error accu-
mulation in self-training, we barely observe such phenomenon in SCL. The test accuracy of the
pseudo-labels remains high and stable across different rounds of pseudo-labeling, which clearly
suggests the ability of SCL in avoiding error accumulation in pseudo-labels.
8
Under review as a conference paper at ICLR 2022
0.0
0.2
0.4
0.6
0.8
1.0
Output value of the discriminator
0
10
20
30
40
Epoch:  30
Label
Fake label
0.0
0.2
0.4
0.6
0.8
1.0
Output value of the discriminator
0
10
20
30
40
Epoch:  70
Label
Fake label
0.0
0.2
0.4
0.6
0.8
1.0
Output value of the discriminator
0
10
20
30
40
Epoch: 140
Label
Fake label
0.0
0.2
0.4
0.6
0.8
1.0
Output value of the discriminator
0
10
20
30
40
Epoch: 235
Label
Fake label
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Probability density
Figure 5: The probability distribution given by the discriminator during the training process
4.3
INTERPRETABILITY OF DISCRIMINATOR OUTPUTS
To evaluate the judgement of the discriminator d, we analyze its output value distribution on the
labeled and intentionally introduced fake/noisy labeled data in different training stages of image
classiﬁcation task. The four plots in Figure 5 clearly show the transition of an improved discrimina-
tor, which gradually gains better capability to discriminate true labels (p →1) and fake/noisy labels
(p →0). The interpretability of the discriminator is a byproduct of SCL, which can be particularly
useful for many real-world SSL tasks or scenarios involving error or noise in labeled data.
5
RELATED WORK
Self-training via pseudo-labeling. Pseudo-labeling (Triguero et al., 2015; Yarowsky, 1995; Lee
et al., 2013) is one of the most widely used self-training method in semi-supervised learning, which
uses a supervised model that is iteratively trained on both labeled and pseudo-labeled data in previ-
ous iterations of the algorithm. The basic assumption in most models (Yarowsky, 1995; Lee et al.,
2013; Laine & Aila, 2017; Tarvainen & Valpola, 2017b; Iscen et al., 2019b) is that an unlabeled
instance can be labeled for training if the underlying model has high conﬁdence in its prediction.
When the model makes inaccurate predictions, it is likely to cause error accumulation that nega-
tively impacts training (Cai et al., 2013; Arazo et al., 2020). To alleviate this problem, many studies
(Natarajan et al., 2017; Schnabel et al., 2016; Shen et al., 2020; Rizve et al., 2021; ?) guide the
model learning based on various label conﬁdence measures and avoid the model from being overly
conﬁdent on incorrect pseudo-labels. The above works only focus on leveraging both labeled and
unlabeled data, but not fully applicable to scenarios with noisy labeled data. SCL considers the
contribution of both labeled and unlabeled data through soft labeling, which better addresses afore-
mentioned drawbacks.
Learning from noisy labels. The lack of high-quality labels is common in many real-world scenar-
ios (Natarajan et al., 2017; Schnabel et al., 2016). Erroneous or noisy labels can severely degrade
the generalization performance of deep neural networks. Learning from noisy labels (Song et al.,
2020) is becoming an increasinly important task in modern deep learning applications. Early studies
(Cesa-Bianchi et al., 2011) require unbiased gradient estimates of the loss to provide learning guar-
antees. Some recent studies (Natarajan et al., 2017; Schnabel et al., 2016; Shen et al., 2020; Algan
& Ulusoy, 2021) provide unbiased estimators for evaluating and training under noise data. How-
ever, these methods still require that the observed data have a relatively well-behaved (noise-free)
underlying distribution, thus the noisy data are weighted according to the same distribution. Satis-
fying such requirements is difﬁcult in practice and the observed distribution may be biased across
the data. In contrast, SCL bypasses the dependence on the original data distribution and learn label
conﬁdence measures adaptive using an additional discriminator to facilitate model training.
6
CONCLUSION
We introduce a novel self-interested coalitional learning framework, which can be easily incorpo-
rated into a wide range of semi-supervised learning algorithms for enhanced performance. SCL im-
proves semi-supervised model learning by constructing a semi-cooperative “game” that establishes
cooperation between the main self-interested semi-supervised learning task and a companion task
of inferring the observability of labels. Meanwhile, it also offers additional conﬁdence measures for
the original and inferred data labels, which can be particularly useful for the practical applications
with noisy labels. Through comprehensive evaluations, we show that SCL can consistently improve
the performance and robustness of semi-supervised classiﬁcation and regression algorithms.
9
Under review as a conference paper at ICLR 2022
REFERENCES
G¨
orkem Algan and Ilkay Ulusoy. Meta soft label generation for noisy labels. In 2020 25th Interna-
tional Conference on Pattern Recognition (ICPR), pp. 7142–7148. IEEE, 2021.
Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudo-labeling
and conﬁrmation bias in deep semi-supervised learning. In 2020 International Joint Conference
on Neural Networks (IJCNN), pp. 1–8. IEEE, 2020.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin
Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural In-
formation Processing Systems 32: Annual Conference on Neural Information Processing Systems
2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5050–5060, 2019.
Xiao Cai, Feiping Nie, Weidong Cai, and Heng Huang. Heterogeneous image features integra-
tion via multi-modal semi-supervised learning model. In Proceedings of the IEEE International
Conference on Computer Vision, pp. 1737–1744, 2013.
Nicol`
o Cesa-Bianchi, Shai Shalev-Shwartz, and Ohad Shamir. Online learning of noisy data. IEEE
Trans. Inf. Theory, 57(12):7907–7931, 2011.
Olivier Chapelle, Bernhard Sch¨
olkopf, and Alexander Zien (eds.). Semi-Supervised Learning. The
MIT Press, 2006.
Kalyanmoy Deb. Multi-objective optimization. In Search methodologies, pp. 403–449. Springer,
2014.
Xavier Glorot and Yoshua Bengio. Understanding the difﬁculty of training deep feedforward neural
networks. In Proceedings of the thirteenth international conference on artiﬁcial intelligence and
statistics, pp. 249–256. JMLR Workshop and Conference Proceedings, 2010.
Lovedeep Gondara and Ke Wang. MIDA: multiple imputation using denoising autoencoders. In
Advances in Knowledge Discovery and Data Mining - 22nd Paciﬁc-Asia Conference, PAKDD
2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III, pp. 260–272, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information
processing systems, 27:2672–2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectiﬁers: Surpassing
human-level performance on imagenet classiﬁcation. In 2015 IEEE International Conference on
Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026–1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-
supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2019, Long Beach, CA, USA, June 16-20, 2019, pp. 5070–5079, 2019a.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-
supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 5070–5079, 2019b.
Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy S. J. Ren, and Rynson W. H. Lau. Dual student:
Breaking the limits of the teacher in semi-supervised learning. In 2019 IEEE/CVF International
Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2,
2019, pp. 6727–6735, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter-
national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9,
2015, Conference Track Proceedings, 2015.
10
Under review as a conference paper at ICLR 2022
Thomas N. Kipf and Max Welling. Semi-supervised classiﬁcation with graph convolutional net-
works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France,
April 24-26, 2017, Conference Track Proceedings, 2017.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th Interna-
tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017,
Conference Track Proceedings, 2017.
Dong-Hyun Lee et al. Pseudo-label: The simple and efﬁcient semi-supervised learning method for
deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3,
pp. 896, 2013.
Lu Liu, Yiting Li, and Robby T Tan.
Decoupled certainty-driven consistency loss for semi-
supervised learning. arXiv preprint arXiv:1901.05657, 2019.
Alexander Matyasko and Lap-Pui Chau. Improved network robustness with adversary critic. In
Advances in Neural Information Processing Systems 31: Annual Conference on Neural Infor-
mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr´
eal, Canada, pp.
10601–10610, 2018.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. Weakly-supervised hierarchical text classi-
ﬁcation. In Proceedings of the AAAI conference on artiﬁcial intelligence, volume 33, pp. 6826–
6833, 2019.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text
classiﬁcation using label names only: A language model self-training approach. In Proceedings
of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,
Online, November 16-20, 2020, pp. 9006–9017, 2020.
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Cost-sensitive
learning with noisy labels. J. Mach. Learn. Res., 18:155:1–155:33, 2017.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow. Realis-
tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information
Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018,
NeurIPS 2018, December 3-8, 2018, Montr´
eal, Canada, pp. 3239–3250, 2018.
Yashoteja Prabhu and Manik Varma. Fastxml: a fast, accurate and stable tree-classiﬁer for extreme
multi-label learning. In The 20th ACM SIGKDD International Conference on Knowledge Dis-
covery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, 2014, pp. 263–272,
2014.
Huiling Qin, Xianyuan Zhan, Yuanxun Li, Xiaodu Yang, and Yu Zheng.
Network-wide trafﬁc
states imputation using self-interested coalitional learning. In KDD ’21: The 27th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18,
2021, pp. 1370–1378, 2021.
Trevor W. Richardson, Wencheng Wu, Lei Lin, Beilei Xu, and Edgar A. Bernal. Mcﬂow: Monte
carlo ﬂow models for data imputation. In 2020 IEEE/CVF Conference on Computer Vision and
Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 14193–14202, 2020.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah.
In defense
of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised
learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event,
Austria, May 3-7, 2021, 2021.
Sebastian Ruder.
An overview of multi-task learning in deep neural networks.
arXiv preprint
arXiv:1706.05098, 2017.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor-
mations and perturbations for deep semi-supervised learning. Advances in neural information
processing systems, 29:1163–1171, 2016.
11
Under review as a conference paper at ICLR 2022
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims.
Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of the 33nd
International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-
24, 2016, pp. 1670–1679, 2016.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad.
Collective classiﬁcation in network data. AI magazine, 29(3):93–93, 2008.
Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kunag. Stable learning via sample reweighting. In
Proceedings of the AAAI Conference on Artiﬁcial Intelligence, volume 34, pp. 5692–5699, 2020.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive
semi-supervised deep learning using min-max features. In Computer Vision - ECCV 2018 - 15th
European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V, pp. 311–
327, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do-
gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learn-
ing with consistency and conﬁdence. In Advances in Neural Information Processing Systems 33:
Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December
6-12, 2020, virtual, 2020.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with
deep neural networks: A survey. CoRR, abs/2007.08199, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con-
sistency targets improve semi-supervised deep learning results. In 5th International Conference
on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track
Proceedings, 2017a.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis-
tency targets improve semi-supervised deep learning results. In Proceedings of the 31st Interna-
tional Conference on Neural Information Processing Systems, pp. 1195–1204, 2017b.
Isaac Triguero, Salvador Garc´
ıa, and Francisco Herrera. Self-labeled techniques for semi-supervised
learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):
245–284, 2015.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con-
sistency training for semi-supervised learning. In Proceedings of the Twenty-Eighth International
Joint Conference on Artiﬁcial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp.
3635–3641, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark-
ing machine learning algorithms. CoRR, abs/1708.07747, 2017.
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. Self-training with noisy student
improves imagenet classiﬁcation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern
Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 10684–10695, 2020.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd
annual meeting of the association for computational linguistics, pp. 189–196, 1995.
Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate
example weighting. In Third IEEE international conference on data mining, pp. 435–442. IEEE,
2003.
12
Under review as a conference paper at ICLR 2022
A
EXPERIMENT SETTINGS
• Image classiﬁcation.The base learner (f) is a ResNet18 (He et al., 2016) where the input
channels of the ﬁrst convolutional layer are replaced with 1. We use the Kaiming Normal-
ization (He et al., 2015) to initialize the parameters of f. All input samples of training set
and test set are transformed with a mean and a standard deviation for normalization. The
discriminator consists of two feature extractors for X and hatY , respectively, and a fusion
layer whose inputs are the extracted X and hatY features and the main task information g.
The feature extractor of X is also a ResNet18 with the number of input channels of the ﬁrst
convolutional layer as 1, and it outputs features with dimension 1. The feature extractor for
hatY is a network with three-layer FC, where the ﬁrst two layers are each followed by a
ReLU layer. We fuse the extracted features of X and hatY by the Hadamard product. And
concatenate the loss g to the fused embeddings of X, hatY and fuse them through a one-
layer FC. We feed the output into a sigmoid layer to output the probabilities. We use the
Adam optimizer with a learning rate of 0.001 to train the base learner f and discriminator
d simultaneously for a total of 300 epochs. We set α = 0.6 in all experiments for this task.
• Label propagation. We train a two-layer GCN as described in (Kipf & Welling, 2017)
and evaluate prediction accuracy on a test set of 1,000 labeled examples. We initialize
weights using the initialization described in (Glorot & Bengio, 2010) and accordingly (row-
)normalize input feature vectors. For discriminator d, we use a ﬁve-layer and two-layer FC
to encode the input X and ˆ
Y , and we fuse the hidden embedding of X and ˆ
Y by hadamard
product. We connect g to the fusion embedding of X, ˆ
Y and fuse it through one-layer FC
and then fed to a sigmoid layer to give the ﬁnal probability. All FC layers is followed by
a ReLU operations except the output layer. We train all models for a maximum of 150
epochs (training iterations) using Adam (Kingma & Ba, 2015) with a learning rate of 0.01.
We set α = 1 in all experiments for this task.
• Data imputation. The denoise autoencoder we used in the data imputation task as de-
scribed in (Gondara & Wang, 2018) consists of a dropout layer and an encoder composed
of four-layer FC and a decoder composed of four-layer FC, where the dropout layer has
p = 0.1. For discriminator d, we use three two-layer FC to encode the input X, ˆ
Y and g,
and we fuse the hidden embeddings by two one-layer FC and then fed to a sigmoid layer to
give the ﬁnal probability. All FC layers in the discriminator are followed by ReLU opera-
tions, while all FC layers in the DAE are followed by tanh operations, with the exception of
the output layer. We train all models for a maximum of 100 epochs using Adam (Kingma
& Ba, 2015) with a learning rate of 10−5 and early stopping with a window size of 10, i.e.
we stop training if the validation loss does not decrease for 10 consecutive epochs. We set
α = 1 in all experiments for this task.
B
ADDITIONAL EXPERIMENT RESULTS
B.1
ADDITIONAL ABLATION STUDY
In this section we present the ablation study of the weight parameter α under the image classiﬁ-
cation task to analyze the impact of α in SCL. We selected three different levels of missing rates
in our experiments - low (30%), medium (50%) and high (80%). Figure 6 shows the classiﬁcation
accuracy obtained by SCL using different α values under different missing rates. We observe that a
larger α will make the main task favors more on minimizing the reweighting loss rather than simply
minimizing the difference between model predicted labels and the given labels ˜
Y . In our exeriment
results in Figure 6, it actually suggests that with the α taking value from the recommended range
[0.1, 0.9], SCL is not very sensitive to α. The maximum difference of accuracy between the worst
and best choice of α is only about 2% under different missing rates. The insensitivity to α is another
advantage of SCL, as we do not need to perform complex parameter tuning in order to learn a robust
and high performance model, which is very implementation friendly.
13
Under review as a conference paper at ICLR 2022
0.1
0.3
0.5
0.7
0.9
Value of 
78
79
80
81
82
83
84
85
86
Accuracy (%)
Missing rate
80%
50%
30%
(a) Image classiﬁcation
0.1
0.3
0.5
0.7
0.9
Value of 
67
71
75
79
83
87
Accuracy (%)
Missing rate
80%
50%
30%
(b) Label propagation
Figure 6: Test accuracy under different weight parameter α
B.2
COMPARISON WITH ADVANCED SSL METHODS
We conduct supplementary experiments on the semi-supervised image classiﬁcation task on CIFAR-
10 dataset with 4000 labeled data samples. We compare SCL with several advanced semi-supervised
learning methods, including the state-of-the-art method: uncertainty-aware pseudo-label selection
(UPS) (Rizve et al., 2021). We use the CNN-13 architecture that is commonly used in benchmarking
SSL algorithms (Oliver et al., 2018) and set α = 0.9 for SCL in this task. For a fair comparison,
we compare against the methods which report results using the same CNN-13 backbone architec-
ture: DeepLP (Iscen et al., 2019a), TSSDL (Shi et al., 2018), MT (Tarvainen & Valpola, 2017a),
MT + DeepLP, ICT (Verma et al., 2019), DualStudent (Ke et al., 2019), and MixMatch (Berthelot
et al., 2019). We keep the experiment setting the same as in the UPS paper (Rizve et al., 2021) for
comparison. The technical details of above baselines algorithms please refer to their papers.
We present in Table 2 the mean and standard deviation of each method across three different random
splits on training and testing data. SCL achieves comparable results to the state-of-the-art method
UPS (92.2% accuracy as compared to 93.6% in UPS), despite that SCL is much more light-weighted
and simpler as compared to UPS. Under the same hardware setting (8-core 4.7GHz CPU and Nvidia
RTX3070 GPU), it takes 19 hours to train UPS to reach 92% accuracy and at least 60 hours to fully
converge; while SCL takes only 7 hours to reach ﬁnal convergence, a 60% reduction in training
time to reach the same level of accuracy. UPS employs the uncertainty measure to select high-
quality pseudo-labels and also uses negative sample learning to improve the model performance.
The uncertainty-aware pseudo-label selection scheme used in UPS is very costly, which lead to high
training cost. By contrast, in SCL, all unlabeled data are pseudo-labelled and used for training, but
with a soft-labeling weights to unevenly update the loss of different samples. It does not involve any
pseudo-label selection process as used in most existing pseudo-labeling methods. Moreover, SCL is
a very general SSL framework which broadly applicable to semi-supervised inductive classiﬁcation,
transductive classiﬁcation and regression tasks, whereas most of the previous baseline algorithms
are only applicable to semi-supervised inductive classiﬁcation tasks. This shows the great potential
and computational efﬁciency of using SCL solving real-world large-scale applications.
Table 2: Error rate (%) on the CIFAR-10 test set. Methods with * are pseudo-labeling based.
Methods
CIFAR-10 (4000 labels)
DeepLP*
12.69 ± 0.29
TSSDL*
10.90 ± 0.23
MT
11.41 ± 0.25
MT+DeepLp
10.61 ± 0.28
ICT
7.29 ± 0.02
DualStudent
8.89 ± 0.09
MixMatch
6.48
UPS*
6.39 ± 0.03
SCL
7.85 ± 0.03
14
