# RETHINKING CLASS-PRIOR ESTIMATION FOR POSITIVE-UNLABELED LEARNING

**Yu Yao[1]** **Tongliang Liu[1][†]** **Bo Han[2]** **Mingming Gong[3]**

**Gang Niu[4]** **Masashi Sugiyama[4][,][5]** **Dacheng Tao[6][,][1]**

1The University of Sydney 2Hong Kong Baptist University 3The University of Melbourne
4RIKEN AIP 5The University of Tokyo 6JD Explore Academy, China

ABSTRACT

Given only positive (P) and unlabeled (U) data, PU learning can train a binary
classifier without any negative data. It has two building blocks: PU class-prior
_estimation (CPE) and PU classification; the latter has been well studied while the_
former has received less attention. Hitherto, the distributional-assumption-free
CPE methods rely on a critical assumption that the support of the positive data
_distribution cannot be contained in the support of the negative data distribution._
If this is violated, those CPE methods will systematically overestimate the class
prior; it is even worse that we cannot verify the assumption based on the data. In
this paper, we rethink CPE for PU learning—can we remove the assumption to
make CPE always valid? We show an affirmative answer by proposing Regrouping
CPE (ReCPE) that builds an auxiliary probability distribution such that the support
of the positive data distribution is never contained in the support of the negative
data distribution. ReCPE can work with any CPE method by treating it as the base
method. Theoretically, ReCPE does not affect its base if the assumption already
holds for the original probability distribution; otherwise, it reduces the positive
_bias of its base. Empirically, ReCPE improves all state-of-the-art CPE methods on_
various datasets, implying that the assumption has indeed been violated here.

1 INTRODUCTION


_Positive-unlabeled (PU) learning can date back to 1990s (Denis, 1998; De Comite et al.´_, 1999;

Letouzey et al., 2000), and there has been a surge of interest in this learning scenario in recent years
because of the difficulty to annotate large-scale datasets (Ren et al., 2014; du Plessis et al., 2014;
2015; Christoffel et al., 2016; Jain et al., 2016; Ramaswamy et al., 2016; Sakai et al., 2018; Kato
et al., 2018; Bekker & Davis, 2018; Gong et al., 2019; Bai et al., 2021; Xia et al., 2021; Yao et al.,
2021). It is also fallen into different applications, such as knowledge-base completion (Galarraga´
et al., 2015; Neelakantan et al., 2015), text classification (Lee & Liu, 2003; Li & Liu, 2003), and
medical diagnosis (Claesen et al., 2015; Zuluaga et al., 2011).

PU learning can be divided into two different settings based on different data generation processes.
The first setting is called censoring PU learning (Elkan & Noto, 2008), which follows a one-sample
configuration. Specifically, a sample S is randomly drawn from the unlabeled data distribution Pu,
and a positive sample Sp is then distilled from it, i.e., randomly selecting some positive instances
contained in the unlabeled data to be the positive sample. The second setting is called case-control
PU learning (Kiryo et al., 2017). In this setting, a positive sample Sp = {xi}i[k]=1 [is randomly]
drawn from the positive class-conditional distribution Pp = P (X|Y = 1), and an unlabeled sample
_Su = {xi}i[n]=k+1_ [is randomly drawn from the unlabeled data distribution][ P][u][. Because case-control]
PU learning is more general than censoring PU learning (Niu et al., 2016), therefore, we will focus
on the setting of case-control PU learning.

Under the setting of case-control PU learning, a lot of classification methods have been proposed
(Ren et al., 2014; du Plessis et al., 2014; 2015; Christoffel et al., 2016; Sakai et al., 2018; Kato
et al., 2018; Bekker & Davis, 2018; Kwon et al., 2019; Tanielian & Vasile, 2019; Gong et al., 2019).

_†Correspondence to Tongliang Liu (tongliang.liu@sydney.edu.au)._


-----

0.30 Pp

0.25 Pu

0.20

P0.15

0.10

0.05

0.00 3 2 1 0 1 2 3

X

(a)


0.30 0.5Pp

0.25 0.5Pu = 0.5Pn Pn + 0.5Pp

0.20

P0.15

0.10

0.05

0.00 3 2 1 0 1 2 3

X

(b)


0.30 0.7Pp

0.25 0.3Pu = 0.3M M + 0.7Pp

0.20

P0.15

0.10

0.05

0.00 3 2 1 0 1 2 3

X

(c)


0.30 0.51Pp[′]

0.25 0.49Pn[′]

Pu = 0.49Pn[′] [+ 0.51][P]p[′]

0.20

P0.15

0.10

0.05

0.00 3 2 1 0 1 2 3

X

(d)


Figure 1: (a) The unlabeled data distribution Pu and the positive class-conditional distribution Pp
are given. (b) Assume that the latent negative class-conditional distribution Pn is fixed, i.e., 0.5Pn
is shown by the green curve, and that the class-prior π is 0.5, i.e., Pu = 0.5Pn + 0.5Pp. (c) The
existing distributional-assumption-free CPE methods will output 0.7 instead of 0.5 because they
always output the maximum proportion κ[∗] of Pu in Pp. (d) Applying the proposed ReCPE method, a
auxiliary distribution Pp′ will be created and the existing CPE methods will output π[′] = 0.49 instead
of 0.7 with input Pp′ and Pu instead of Pp and Pu.

However, the class-prior estimation (CPE) (Elkan & Noto, 2008; Jain et al., 2016; Ramaswamy et al.,
2016; Christoffel et al., 2016; Kato et al., 2018) has received less attention. Formally, CPE is defined
as a problem of estimating π = P (y = 1) (0, 1) given a sample from the marginal distribution Pu
_∈_
and a sample from positive class-conditional distribution Pp. The marginal distribution Pu is mixed
with both positive and negative class-conditional distributions, i.e., Pu = πPp + (1 − _π)Pn. CPE acts_
as a crucial building block for state-of-the-art PU classification methods, and it is essential to build
statistically-consistent PU classifiers (du Plessis et al., 2014; Scott, 2015; Jain et al., 2016; Kiryo et al.,
2017). The formulation of these classification methods involves the class-prior π, but π is usually
unknown in practice. If π is poorly estimated, the classification accuracy of the state-of-the-art PU
classification methods (du Plessis et al., 2014; 2015; Kiryo et al., 2017) could be degraded.

The mixture proportion estimation (MPE) is closely related to CPE (Blanchard et al., 2010; Scott,
2015). In the setting of MPE, there is a mixture distribution

_F = (1 −_ _κ[∗])G + κ[∗]H,_ (1)

where H and G are called component distributions. Given the samples randomly drawn from
_F and H, respectively, MPE aims to estimate the maximum proportion κ[∗]_ _∈_ (0, 1) of H in F .
Thereby, if the maximum proportion κ[∗] is identical to the class-prior π, the MPE methods can
be employed to obtain π by letting Pu and Pp be the mixture distribution F and the component
distribution H, respectively; otherwise, the MPE methods cannot be employed. To the best of our
knowledge, most of state-of-the-art CPE methods (Blanchard et al., 2010; Liu & Tao, 2015; Scott,
2015; Ramaswamy et al., 2016; Jain et al., 2016) are based on MPE, which do not rely on assumptions
that the data are drawn from a given parametric family of probability distributions (i.e., they are
distributional-assumption-free methods).

To let these distributional-assumption-free methods can be used to identify class-prior π, κ[∗] must
be identical to the class-prior π. The irreducibility assumption (Blanchard et al., 2010) has been
proposed to make them identical, which is employed by all these CPE methods implicitly or explicitly,
to the best of our knowledge. It assumes that the support of the positive class-conditional distribution
_Pp is not contained in the support of the negative class-conditional distribution Pn. However, it is_
strong and hard to be verified in PU learning, since Pn is a latent distribution, such that we do not
have any prior knowledge about it. Additionally, since the applications of PU learning are diverse
(Hsieh et al., 2019; Bekker & Davis, 2020), it is hard to guarantee that the support of Pp is not in the
support of Pn.

If the irreducibility assumption cannot be satisfied, the existing distributional-assumption-free CPE
methods will suffer from an overestimation of π. For example, in Figure 1(a), we show both the
unlabeled data distribution Pu and the component distribution Pp. In Figure 1(b), we assume the
latent negative class-conditional distribution Pn is fixed as shown in the green color, and the positive
class-prior π = 0.5. In Figure 1(c), we show the existing distributional-assumption-free CPE methods
will output the biased class-prior 0.7. It is different from the ground truth 0.5, since the support of Pp
is contained in the support of Pn. When the irreducibility assumption is not held, how to improve the
estimations of distributional-assumption-free PU learning methods is challenging but useful.


-----

Because the irreducibility assumption is impossible to check without making any assumption on
_Pn. Thereby, in this paper, we rethink those CPE methods and propose a novel method called_
_Regrouping CPE (ReCPE) which improves the estimations of the current PU learning methods_
without irreducibility assumption. The main idea of our method is that, instead of estimating
the maximum proportion of Pp in Pu, we build a new CPE problem by creating a new auxiliary
distribution Pp′ always guaranteeing the irreducibility assumption. Then we use the existing CPE
method to obtain the maximum proportion of Pp′ in Pu, which is denoted by π[′]. We show that, with
both theoretical analyses and experimental validations, when the irreducibility assumption holds,
our ReCPE method does not affect the prediction of the existing estimators; when the irreducibility
assumption does not hold, our method will help the current estimators have less estimation bias,
which could improve the performances of PU classification tasks. For example, in Figure 1(d), we
create a new class-conditional (auxiliary) distribution Pp′ . By solving it, π[′] = 0.51. The estimation
bias of the existing estimators will reduce to π[′] _−_ _π = 0.01 instead of κ[∗]_ _−_ _π = 0.2._

The rest of the paper is organized as follows. In Section 2, we review the irreducibility assumption
and its variants. We discuss the difficulty of checking the assumptions. In Section 3, we provide
the estimation biases of the existing consistent distributional-assumption-free CPE methods. Then
we propose our method ReCPE, followed by theoretically analysis of its estimation bias and the
implementation details. All the proofs are listed in Appendix A. The experimental validations are
given in Section 4. Section 5 concludes the paper.

2 IRREDUCIBILITY OF CPE

In this section, we briefly review the assumptions used for existing distributional-assumption-free CPE
estimators. Then we provide the estimation bias introduced by consistent distributional-assumptionfree CPE methods when the assumptions do not hold.

**The irreducibility assumption. Let Pp and Pu be probability measures (distributions) on a measur-**
able space (X _, S), where X is the sample space, and S is the σ-algebra. Let κ[∗]_ be the maximum
proportion of Pp in Pu. To let κ[∗] be identical to π, the irreducibility assumption was proposed by
Blanchard et al. (2010).

**Definition 1 (Irreducibility). Pn and Pp are said to satisfy the irreducibility assumption if Pn is not**
_a mixture containing Pp. That is, there does not exist a decomposition Pn = (1 −_ _β)Q + βPp, where_
_Q is a probability distribution on the measurable space (X_ _, S), and 0 < β ≤_ 1.

Equivalently, the assumption assumes the support of Pp is hardly contained in the support of Pn. It
means that with the selection of different sets S, the probability Pn(S) can be arbitrarily close to 0,
and Pp(S) > 0. Suppose we can access the distributions Pu, Pp and the set C containing all possible
latent distributions, then the class-prior π can be found as follows:

_Pu(S)_
_π = κ[∗]_ ≜ sup _α_ _Pu = (1_ _α)K + αPp, K_ = inf (2)
_{_ _|_ _−_ _∈C}_ _S∈S,Pp(S)>0_ _Pp(S)_ _[.]_

To the best of our knowledge, all existing distributional-assumption-free CPE methods (Blanchard
et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015; Ramaswamy et al., 2016; Ivanov, 2019)
are variants of estimating the maximum proportion κ[∗] of Pp in Pu. Many of them are statistically
consistent estimators (Blanchard et al., 2010; Scott et al., 2013; Liu & Tao, 2015; Scott, 2015).

**The variants of the irreducibility. Based on the irreducibility assumption, estimators can be**
designed with theoretical guarantees that they will converge to the class-prior π (Blanchard et al.,
2010). However, the convergence rate can be arbitrarily slow (Scott, 2015). The reason is that the
irreducibility assumption implies the following fact (Blanchard et al., 2010; Scott et al., 2013)

_Pn(S)_
inf (3)
_S∈S,Pp(S)>0_ _Pp(S) [= 0][,]_

i.e., the maximum proportion of Pp in Pn approaches to 0. To obtain the class-prior π, it requires
finding a sequence of the sets S converging to the infimum, which empirically can be hard to find.
Therefore, the convergence rate of the designed estimators based on Eq. (3) will be arbitrarily slow. To
ensure a fixed rate of convergence, the anchor set assumption, a stronger variant of the irreducibility


-----

assumption, has been proposed (Scott, 2015; Liu & Tao, 2015; Xia et al., 2019; 2020; Yao et al.,
2020). It assumes that

_Pn(S)_
min (4)
_S∈S,Pp(S)>0_ _Pp(S) [= 0][,]_

i.e., there exists a set can achieve the minimum 0, which is called an anchor set. Another stronger
variant is the separability assumption (Ramaswamy et al., 2016) which extends the anchor set
assumption to a function space. It is proposed to bound the convergence rate of the method based on
kernel-mean-matching (KMM) technique (Gretton et al., 2012).

3 REGROUPING FOR CPE (RECPE)

In this section, we propose a general method named regrouping for CPE (ReCPE). We discuss how
to theoretically and empirically mitigate the overestimation problem of the class-prior π.

3.1 MOTIVATION

In general, it is impossible to verify the irreducibility assumption for CPE. To check the assumption,
we need to make Pn itself to be observable and verify that whether the distribution Pn is a mixture
containing the distribution Pp, which obviously contradicts the setting of PU learning. However,
in practice, the irreducibility assumption may not hold for many real-world problems, because the
negative class is diverse (Hsieh et al., 2019; Bekker & Davis, 2020) in PU learning. If the assumption
does not hold, Pn is said to be reducible to Pp, and distributional-assumption-free CPE methods will
introduce an estimation bias.
**Proposition 1. Let β[∗]** = inf _S∈S,Pp(S)>0_ _PPnp((SS))_ _[be the maximum proportion of][ P][p][ in][ P][n][, given]_
_Pu = (1 −_ _π)Pn + πPp, for 0 < π ≤_ 1, we have

_Pn(S)_
_κ[∗]_ = π + (1 _π)_ inf (5)
_−_ _S∈S,Pp(S)>0_ _Pp(S) [=][ π][ + (1][ −]_ _[π][)][β][∗][.]_

According to Proposition 1, if the irreducibility assumption does not hold, then there exists β > 0.
In this case, maximum proportion κ[∗] can still be obtained, but it is different from π but equal to
_π + (1 −_ _π)β[∗]. In this case, if we directly employ existing distributional-assumption-free CPE_
methods, they could introduce an arbitrary estimation bias (1 _π)β[∗]_ which depends on Pn.
_−_

To reduce the estimation bias, we propose ReCPE. The process of regrouping is to change the original
class-conditional distributions Pn and Pp into new class-conditional distributions Pn[′] and Pp[′] by
transporting the probability mass of the set A from the negative class to the positive class. After
regrouping, new class-conditional distributions are guaranteed to satisfy the irreducibility assumption,
and therefore, the new positive class-prior π[′] can be identified by current CPE methods. To get the
intuition, we provide a concrete example as follows.

Suppose that Pp is the uniform on [ [1]2 _[,][ 1]][,][ P][n][ is uniform on][ [0][,][ 1]][, and][ π][ =][ 1]2_ [. Then we have][ P][u][ such]

that it is uniform on [0, [1]2 [)][ and][ [][ 1]2 _[,][ 1]][, respectively. Specifically, the probabilities are]_



[0, [1] = [1]

2 [)] 4 _[, P][u]_




[ [1] = [3]

2 _[,][ 1]]_ 4 _[.]_



_Pu_


In this case, by Eq. 5, the maximum proportion of Pp in Pu is κ[∗] = [3]4 [. Let][ ρ >][ 0][ be a small constant,]

and let A = (1 − _ρ, 1]. In this case, the mass of A in Pu from Pn is πPn(A) =_ _[ρ]2_ [. After transporting]

the mass _[ρ]2_ [from][ P][n][ to][ P][p][, we have a new positive class-prior][ π][′][ and a new class-conditional][ P][p][′]

which is uniform on [ [1]2 _[,][ 1][ −]_ _[ρ][)][ and][ [1][ −]_ _[ρ,][ 1]][, respectively. Specifically,]_


_π[′]_ = [1 +][ ρ]


3ρ

_, Pp′_ [ [1] = [1][ −] [2][ρ]

2 _[,][ 1][ −]_ _[ρ][)]_ 1 + ρ [, P][p][′][ ([1][ −] _[ρ,][ 1]) =]_ 1 + ρ _[.]_

 


The left equation above shows that the new class-prior π[′] is dependent on ρ or the size of A. By
controlling set A or ρ to be small, π[′] can be as close to π as possible. This is the intuition of how
regrouping works.


-----

**Algorithm 1 ReCPE**

**Input: An unlabeled sample Su i.i.d. drawn from Pu, a positive sample Sp i.i.d. drawn from Pp,**
and the percentage p of the sample needed to copy from Su to Sp.

1: Train a binary classifier h with the unlabeled sample Su and positive sample Sp by treating Su
as a negative sample;

2: Assign each example x ∈ _Su with the negative class-posterior probability P_ (Y = −1|X = x)
predicted by the trained classifier h;

3: Obtain Sp′ by copying p _Su_ examples with the smallest negative class-posterior probability
_× |_ _|_
_P_ (Y = −1|X = x) from Su to Sp;

4: Estimate the class-prior π[′] by employing an algorithm based on Eq. (5) with inputs Su and Sp′ .

**Output: The estimated new class-prior ˆπ[′].**

3.2 PRACTICAL IMPLEMENTATION

In practice, we have to implement the aforementioned idea of regrouping based on positive sample
_Sp and unlabeled sample Su. Since the negative sample is unavailable, we cannot “cut and paste” any_
example from negative class to positive sample Sp; instead, we can “copy and past” some unlabeled
examples to Sp. When doing so, we should select a small set of samples _A[ˆ][∗]_ which look the most
similar to the positive class and dissimilar to the negative class, which could encourage the difference
between the original Pn, Pp, and ρ and π, Pp[′], and π[′] to be small. This is why _A[ˆ][∗]_ = (1 _ρ, 1] was_
_−_
selected in the above intuitive example, i.e., _A[ˆ][∗]_ belongs geometrically and visually to the positive
class with the highest confidence among all subsets of [0, 1] of size ρ.

A hyper-parameter p ∈ (0, 1) is introduced to control the size of set _A[ˆ][∗], theoretically, we prefer the_
set _A[ˆ][∗]_ to have a small size. Empirically, p cannot be so small: the existing estimators are insensitive
to tiny modifications (they are designed to be robust in such a way, in order to be good estimators).
For example, the difference between the estimated class-priors by employing samples Su and Sp and
the one by employing samples Su and Sp′ can be hardly observed if Sp and Sp′ only differ from
in one or two points. Specifically, p = 10% is selected for the experiments on all datasets, which
leads to a significant improvement of the estimation accuracy. The details on the selection of the
hyper-parameter value will be explained in Section 4.1. The algorithm is summarized in Algorithm 1.

There are two fundamental concerns for copying _A[ˆ][∗]_ to Sp. 1). When we have irreducibility, might
regrouping make ˆπ[′] be a worse approximation? 2). When we lack irreducibility, must regrouping
make ˆπ[′] be a better approximation? While these concerns will be formally clarified later, we give
here intuitive implications of regrouping.

1). If we have irreducibility, the Pn( A[ˆ][∗]) should be rather small (if not zero), and _A[ˆ][∗]_ should be drawn
from the positive component Pp of the mixture Pu. In this case, regrouping will generally have small
influence to Pp. Hence, it will not make ˆπ[′] worse.

2). If we lack irreducibility, _A[ˆ][∗]_ may be drawn from either Pp or Pn. By regrouping, _A[ˆ][∗]_ becomes
present in Sp[′], which encourages the probability of the set _A[ˆ][∗]_ in Pp[′] to be large. This will modify Pp
as we expected towards irreducibility. As a consequence, regrouping will make ˆπ[′] better.

3.3 THEORETICAL JUSTIFICATION

In the regrouping approach described above, the auxiliary class-conditional distribution Pp′ and
_Pn′ are created by regrouping a small set A from Pp and Pn. Here, we analyze the properties of_
regrouping and theoretically justify it.

**A formal definition of regrouping In order to analyze the properties, we need to formally define**
how to split, transport, and regroup a set A (or the mass of A).

**Definition 2. Let M be a probability measure on a measurable space (X** _, S). Given a set A ∈_ S,
_we define a measure M_ _[A]_ _on the σ-algebra S as follows:_

_∀S ∈_ S, M _[A](S) = M_ (S ∩ _A)._ (6)


-----

It is easy to see that given two measures M _[A]_ and M _[A][c]_ obtained according to Definition 2, where
_A[c]_ = X \ A, then M _[A]_ and M _[A][c]_ have the following property.
**Lemma 1. Let M be a probability measure over a measurable space (X** _, S). For any set A ∈_ S,
_we have M_ _[A]_ + M _[A][c]_ = M _._

Now, we introduce the theory of regrouping. Fixing a set A ∈ S, we split Pn as Pn[A][c] and Pn[A][,]
transport Pn[A] [to][ P][p] [to regroup them together, i.e.,]

_Pu = (1_ _π)Pn + πPp = (1_ _π)(Pn[A]_ [+][ P]n[ A][c] ) + πPp = (1 _π)Pn[A][c]_ + ((1 _π)Pn[A]_ [+][ πP][p]).
_−_ _−_ _−_ _−_
split into two regroup as one

Finally, we can rewrite the unlabeled data distribution Pu as a mixture of two new class-conditional

| {z } | {z }

distributions Pn′ and Pp′ defined in Theorem 1 by normalization.
**Theorem 1. Let Pu = (1** _π)Pn + πPp. Let A_ support(Pu). By regrouping Pn[A] _[to][ P][p][,][ P][u]_ _[can]_
_−_ _⊂_
_be written as a mixture, i.e., Pu = (1_ _π[′])Pn′ + π[′]Pp′_ _, where_
_−_
_π[′]_ = π + (1 _π)Pn(A),_ (7)
_−_

_Pn[A][c]_ n [+][ πP][p]
_Pn′ =_ (8)

_Pn(A[c])_ _[, P][p][′][ = (1](1[ −]_ _π[π])[)]P[P]n[ A](A) + π [,]_

_−_

_and Pn′ and Pp′ satisfy the anchor set assumption._

When class-conditional distributions Pn and Pp do not satisfy the irreducibility assumption, π cannot
be obtained by using CPE methods based on MPE, which will lead to an estimation bias discussed
before. However, Theorem 1 shows that the new proportion π[′] is always identifiable as Pn[′] and
_Pp[′] always satisfy the anchor set assumption. Thus, after regrouping, π[′]_ is identifiable and can be
estimated by the existing CPE methods.

**Bias reduction According to Theorem 1, to make π[′]** closer to π, we expect to find the set A looks
most dissimilar to the negative class, i.e., Pn(A) is small.

**Theorem 2. Let Pp′ and Pn′ be obtained by regrouping a set A[∗]** := arg minA S _PPnp((AA))_ 1 from Pp
_∈_

_and Pn. 1). If Pn and Pp satisfy the irreducibility assumption, then π[′]_ = π; 2). if Pn and Pp
_dissatisfy the irreducibility assumption, then π < π[′]_ _< π + (1 −_ _π)β[∗]_ = κ[∗].

Theorem 2 shows how to properly select a set used for regrouping to make π[′] a good approximation of
_π. Specifically, once A[∗]_ is selected for regrouping, if Pn and Pp satisfy the irreducibility assumption,
the new estimation π[′] will be identical to π; if Pn and Pp dissatisfy the irreducibility assumption, π[′]
obtained by employing the distributions Pu and Pp′ will contain a smaller estimation bias compared
to κ[∗] obtained by employing the distributions Pu and Pp.

**Convergence analysis For completeness, we illustrate the convergence property of ReCPE, which**
is presented by employing the estimator proposed by Blanchard et al. (2010). Let Su, Sp and Sp′
be the samples i.i.d. drawn from Pu, Pp and Pp′, respectively. Let A be the set used for regrouping.
Let h : X → R, h ∈H, be a function that predicts 1 for all elements in the set A and 0 otherwise,
where denotes a hypothesis space. Let _S_ denote the cardinality of a set S. Let 1 _h(x)=1_ be an
_H_ _|_ _|_ _{_ _}_
indicator function which returns 1 if h(x) predicts 1 and 0 otherwise. Then Pu(A) can be expressed
as _x_ _[p][p][(][x][)][1][{][h][(][x][)=1][}][d][x][, where][ p][p][ is the density function of the distribution][ P][u][. Let][ ˆ]Pu(A) be_

the empirical version ofR _∈X_ _Pu(A), i.e.,_ _P[ˆ]u(A) =_ _|S1u|_ _x∈X_ [1][{][h][(][x][)=1][}][. Similarly, let][ ˆ]Pp′ (A) be the

empirical version of Pp[′] (A). Let the error ϵδ, (Su) denote the difference between Pu(A) and _P[ˆ]u(A)_
_H_ P
obtained by exploiting the empirical Rademacher complexity (Mohri et al., 2018). Similarly, let
_ϵδ,H(Sp′_ ) denote the difference between Pp′ (A) and _P[ˆ]p′_ (A). We have the following theorem.

**Theorem 3.probability 1 Let −** 2 Pδ, the estimated class-prioru = (1 − _π)Pn + πPp. By selecting a set ˆπ[′]_ _based on solving A and regrouping inf_ _S∈S, ˆPp′_ (S P)>n[A]0 _[to]PPˆˆp[ P]u′((S[p]S)[. Then, with])_ _[satisfies]_

_|πˆ[′]_ _−_ _π| ≤_ _Pˆp′_ (Aϵδ,) +H( ϵSδ,pH′ )(Sp′ ) + _Pˆp′_ (Aϵ) +δ,H ϵ(Sδ,uH)(Sp′ ) + (1 − _π)Pn(A),_ (9)

1We have defined that the fraction tends to infinite if its numerator is larger than 0 and its denominator is
0. Additionally, the infimum may not always exist, if it does not exist, we could use a sequence of sets that
converges to the infimum value, but the convergence rate can be arbitrarily slow (Scott, 2015).


-----

log2|Sδ[4]| _[, and][ ˆ]RS(H) is the empirical Rademacher complexity of H._


_where ϵδ,_ (S) ≜ 2 R[ˆ] _S(_ ) + 3
_H_ _H_


To make ϵδ,H(S) converge to 0 with the increasing of the sample size of S, a universal approximation
assumption has been proposed by Scott (2015) to ensure that the hypothesis space is large enough to
represent a wide variety of interesting functions. Under the assumption, Scott (2015) proved that,
with increasing of the size of samples Su and Sp′, the error between Pu(A) and _P[ˆ]u(A) will converge_

to 0 at a rate logS |uSu|, similarly to Pp′ (A) and _P[ˆ]p′_ (A). Since the empirical Rademacher
_O_ _|_ _|_

complexity R[ˆ] _X_ (q) of a hypothesis space can be upper-bounded by its VC-dimension (Mohri
_H_ _H_
et al., 2018), the both errors based on the empirical Rademacher complexity will also converge to
zero with increasing of the sample size. Consequently, the estimation ˆπ[′] = _PPˆˆpu′((AA))_ [will converge to]

_π[′]_ = _P[P]p[u]′[(]([A]A[)])_ [=][ π][ + (1][ −] _[π][)][P][n][(][A][)][ at a rate][ O]_ log(min(min(|S|uS|u,||S,|pS′p|′)|)) .

r 

**Computationally efficient identification of A[∗]** The following theorem presents how to identify A[∗]
with Pu and Pp. Let us define another auxiliary distribution q(X, C), where C ∈{0, 1} is the positivevs-unlabeled label i.e., a class label distinguishing between the positive component and the whole
mixture. Specifically, priors are q(C = 1) := 1 _ππ_ [and][ q][(][C][ = 0) :=] 1 1π [; conditional densities]

_−_ _−_

are q(X|C = 1) := Pp and q(X|C = 0) := Pu; class-posterior probabilities are q(C = 0|X) and
_q(C = 1|X). We have the following theorem._

**Theorem 4. Let pu and pp be density functions of Pu and Pp, respectively. Let q = P** (C =
0)the setpu + A P, and(C = 1) 0 otherwise. Then the setpp. Let 1A : X →{0 A, 1[∗]} be the identity function which outputs= arg minA S EExx∼qq((XX))[[11AA((XX==xx))qq((CC=0=1| 1XX== ifxx x)])] _∈X[.]_ _is in_
_∈_ _∼_ _|_

For the above optimization, its objective function has two expectations over q, which can have
the “exact” empirical solution obtained by replacing expectations with empirical averages: A[∗] =
arg minA⊂S Pxx∈∈AA _[q][q][(][(][C][C][=1][=0][|][|][X][X][=][=][x][x][)][)]_ [.]

P

**Approximation of Pp′ with a surrogate As we do not have examples drawn from Pn, it is hard to**

create Pp′, let alone sample from it. We approximate Pp′ by using Pp˜[′][ =] _PPuu[A](A[+])+1[P][p]_ [. The following]

proposition shows that when Pu(A) is small, Pp˜[′][ is almost identical to][ P]p[′] [.]

**Proposition 2. Let Pp˜[′][ =]** _PPuu[A](A[+])+1[P][p]_ _[and][ P][u][(][A][)][ < ϵ][.][ ∀][ϵ >][ 0][ and][ S][ ∈]_ [S][,][ |][P][p][′] [(][S][)][ −] _[P]p[˜][′]_ [(][S][)][| ≤O][(][ϵ][)][.]

Since the gap has the same order as Pu(A) uniformly over S, it is guaranteed that whenever Pu(A)
is small, the gap is also small. Practically, we can control the parameter ϵ in the above proposition to
be small. Specifically, using a small value of the hyper-parameter p in Algorithm 1 will lead to the
set A in Theorem 1 to be small, as well as Pu(A). As a consequence, the practical implementation of
regrouping is a good approximation of the theory of regrouping as we expected. By now, we have
analyzed all of the properties of regrouping and theoretically justified all of the points in its design.

4 EXPERIMENTS

We run experiments on 2 synthetic datasets and 9 real word datasets[2]. The objectives of employing
synthetic datasets are to validate whether the proposed regrouping CPE method reduces the estimation
error of the consistent distributional-assumption-free CPE method on the dataset satisfying the
irreducibility assumption and does not influence the prediction of the CPE method on the dataset
dissatisfying the irreducibility assumption. The hyper-parameter p is also selected from the synthetic
datasets. The real-world datasets are used to illustrate the effectiveness of our methods. Although
we have introduced a hyper-parameter p and used approximations in the implementation, empirical
results on all synthetic and real-world datasets consistently show the superiority of ReCPE.

To have a rigorous performance evaluation, for each dataset, 6 × 3 × 10 experiments are conducted
via random sampling. Specifically, we select {0.25, 0.5, 0.75} fraction of positive examples to be the

[2The real word datasets are downloaded from the UCL machine learning database. Multi-class datasets are](https://archive.ics.uci.edu/ml/index.php)
used as binary datasets by either grouping or ignoring classes.


-----

sample of the positive distribution Pp. We let the rest of the examples be the sample of the unlabeled
distribution Pu. In such a way, 3 pairs of empirical positive and unlabeled distributions are generated.
Then, we create other 3 pairs of distributions by flipping the labels of all instances in the original
datasets. For each pair of distributions, we randomly draw positive and unlabeled samples with
sizes of 800, 1600, and 3200, respectively, which are used as input data. Note that, the positive and
unlabeled samples have the same size as did in Ramaswamy et al. (2016). For each sample size, 10
repeated experiments are carried out with random sampling. For all experiments, we employ a neural
network [3] with 2 hidden layers. Each hidden layer contains 50 hidden units. The batch normalization
(Ioffe & Szegedy, 2015) is also employed. The stochastic gradient descent optimizer is used with the
batch size 50. The network is trained for 350 epochs with a learning rate 0.01 and momentum 0. The
weight decay is set to 1e − 5. The model with the best validation accuracy is used to estimate the
positive class-posterior probability P (Y = 1|X = x). We sample the validation set with 20% of the
training data size.

4.1 EXPERIMENTS ON SYNTHETIC DATASETS


We create two datasets with one satisfying the irreducibility assumption while the other not. The dataset satisfying
the irreducibility assumption is created by sampling from 2
different 10-dimensional Gaussian distributions as the component distributions. One of the distributions has zero means
and a unit covariance matrix. Another one has unit means
and unit covariance matrix. The dataset dissatisfying the irreducibility assumption is also created by drawing examples
from 2 different 10-dimensional Gaussian distributions. One
of the distributions has zero means and unit covariance matrix. Another one has unit means and covariance matrix. Then
we remove all the data points with P (Y = 1|X) ≥ 0.98 or
_P_ (Y = 1|X) ≤ 0.02. For simplicity, in Figure 2, we name
two datasets irreducible data and reducible data, respectively.

To validate the correctness of our method and to select a suitable value of the hyper-parameter p, we carry out two experiments. The consistent CPE method KM2 is used as the
baseline, which is compared to our method ReKM2, i.e., regrouping version of the KM2. Firstly, we compare the magnitude differences between ˆπ and ˆπ[′] (i.e., ˆπ − _πˆ[′]) with the_
different fractions of points to be copied from the mixture
sample to the component sample, which is illustrated in Figure 2(a). Then we compare differences of the absolute error
(i.e., |πˆ − _π| −|πˆ[′]_ _−_ _π|) between the baseline and our method_
with the increasing of the copy fractions. Note that each point
in Figure 2 is obtained by averaging over 6 × 3 × 10 experiments.


difference on irreduicible data difference on reducible data


0.025

0.020

0.015

ReKM2

0.010

KM2 0.005

0.000

0.05 0.10 0.15 0.20

Copy Fraction p

(a)


0.025

0.020

0.015

ReKM2 AAE

0.010

0.005

KM2 AAE

0.000

0.05 0.10 0.15 0.20

Copy Fraction p

(b)


with the increasing of the copy fractions. Note that each point

Figure 2: Experiments of the hyper
in Figure 2 is obtained by averaging over 6 3 10 experi_×_ _×_ parameter selection on synthetic
ments.

datasets. With increasing of the copy

Figure 2(a) validates the correctness of our Theorem 2 and fraction p, (a) average estimation
Eq. (7). Theorem 2 states that, by properly selecting the set A, differences between KM2 and Reon the dataset dissatisfying the irreducibility assumption (re- grouping KM2 (ReKM2) and (b) avducible data), π[′] should be smaller than the maximum propor- erage differences of the absolute ertion κ[∗]; on the dataset satisfying the irreducibility assumption ror between KM2 and Regrouping
(irreducible data), π[′] should be close to π. Figure 2(a) perfectly KM2 (ReKM2).
matches this statement. It shows that, on the reducible data,
the values of ˆπ[′] are continuously smaller than ˆπ with the copy fraction ≤ 22.5%; on the irreducible
data, ˆπ[′] and ˆπ have the similar values until the copy fraction ≥ 17.5%. According to Eq. (7), the
positive bias of our estimator should become larger with the increase of Pn(A). This fact is reflected
by the differences of ˆπ − _πˆ[′]_ become smaller on both datasets when the copy fraction > 15%.

Figure 2(b) illustrates the average differences of absolute error between the baseline and the proposed
method. On the reducible data, our method continuously outperforms the baseline with the copy


3We employ the neural network because it has a high approximation capability (Cs´aji et al., 2001).


-----

|Col1|AM ReAM|DPL ReDPL|EN ReEN|KM1 ReKM1|KM2 ReKM2|ROC ReROC|RPG ReRPG|
|---|---|---|---|---|---|---|---|
|adult (800)|0.127 0.13|0.122 0.108∗|0.316 0.295|0.255 0.132|0.164 0.153|0.176 0.153|0.135 0.134|
|adult (1600)|0.122 0.124|0.089∗ 0.089∗|0.31 0.29|0.131 0.091|0.12 0.13|0.121 0.095|0.123 0.137|
|adult (3200)|0.105 0.086|0.054 0.057|0.297 0.279|0.054 0.04∗|0.082 0.089|0.089 0.067|0.114 0.128|
|avila (800)|0.168 0.152|0.129 0.147|0.447 0.422|0.105 0.075∗|0.104 0.081|0.263 0.228|0.119 0.111|
|avila (1600)|0.165 0.132|0.104 0.084|0.439 0.418|0.086 0.076∗|0.108 0.092|0.191 0.16|0.123 0.121|
|avila (3200)|0.156 0.133|0.05∗ 0.061|0.436 0.42|0.092 0.078|0.112 0.092|0.131 0.095|0.121 0.122|
|bank (800)|0.135 0.158|0.116∗ 0.132|0.282 0.264|0.356 0.216|0.266 0.238|0.163 0.15|0.163 0.185|
|bank (1600)|0.117 0.167|0.087∗ 0.105|0.262 0.244|0.178 0.128|0.203 0.198|0.129 0.118|0.157 0.167|
|bank (3200)|0.104 0.127|0.073∗ 0.091|0.248 0.237|0.124 0.09|0.15 0.16|0.093 0.106|0.159 0.18|
|card (800)|0.131 0.127∗|0.174 0.161|0.465 0.444|0.293 0.176|0.203 0.158|0.247 0.233|0.177 0.155|
|card (1600)|0.173 0.14|0.14 0.14|0.459 0.437|0.19 0.135|0.159 0.129|0.194 0.163|0.126 0.115∗|
|card (3200)|0.164 0.134|0.127 0.12|0.455 0.435|0.161 0.113|0.142 0.122|0.159 0.152|0.11 0.108∗|
|covtype (800)|0.16 0.123|0.155 0.151|0.367 0.343|0.157 0.142|0.122 0.13|0.291 0.258|0.116 0.105∗|
|covtype (1600)|0.12 0.1∗|0.132 0.109|0.364 0.339|0.116 0.113|0.121 0.123|0.199 0.161|0.109 0.108|
|covtype (3200)|0.128 0.09|0.093 0.083∗|0.354 0.334|0.097 0.109|0.124 0.128|0.157 0.113|0.109 0.107|
|egg (800)|0.153 0.106∗|0.218 0.225|0.505 0.505|0.173 0.264|0.119 0.131|0.476 0.396|0.171 0.124|
|egg (1600)|0.137 0.12|0.121 0.142|0.486 0.489|0.234 0.214|0.116 0.108∗|0.315 0.238|0.151 0.114|
|egg (3200)|0.126 0.113|0.057∗ 0.073|0.485 0.489|0.26 0.193|0.134 0.113|0.163 0.139|0.142 0.102|
|magic04 (800)|0.099 0.077|0.072 0.071|0.312 0.296|0.111 0.1|0.071 0.064|0.141 0.124|0.055 0.054∗|
|magic04 (1600)|0.071 0.056|0.044 0.043∗|0.292 0.274|0.084 0.072|0.079 0.065|0.1 0.073|0.058 0.052|
|magic04 (3200)|0.069 0.054|0.035∗ 0.036|0.274 0.258|0.07 0.047|0.085 0.063|0.065 0.047|0.054 0.052|
|robot (800)|0.053 0.062|0.049 0.047∗|0.19 0.187|0.232 0.215|0.111 0.114|0.119 0.144|0.077 0.084|
|robot (1600)|0.053 0.038∗|0.087 0.054|0.139 0.132|0.15 0.141|0.098 0.099|0.08 0.075|0.076 0.079|
|robot (3200)|0.052 0.039∗|0.156 0.119|0.091 0.085|0.079 0.077|0.084 0.084|0.063 0.043|0.06 0.066|
|shuttle (800)|0.083 0.031|0.016∗ 0.02|0.041 0.035|0.058 0.083|0.035 0.065|0.042 0.047|0.035 0.051|
|shuttle (1600)|0.09 0.045|0.011∗ 0.018|0.04 0.034|0.048 0.079|0.024 0.05|0.029 0.043|0.026 0.039|
|shuttle (3200)|0.076 0.028|0.012∗ 0.021|0.043 0.038|0.046 0.07|0.018 0.03|0.038 0.045|0.028 0.042|
|average|0.116 0.1|0.094 0.092∗|0.311 0.297|0.146 0.121|0.117 0.111|0.157 0.136|0.106 0.105|


Table 1: Absolute estimation errors on real-world datasets. The first column provides the names of the
datasets and sample size. We bold the smaller average estimation errors by comparing each baseline
method with its regrouped version. The smallest average estimation error among all methods for
each row is highlighted with ∗. The last row is obtained by averaging the results of all experiments.
Variances and the results of Wilcoxon signed-rank test are reported in Appendix B. The proposed
Regrouping methods are significantly better than most of the baselines.

fraction ≤ 22.5%. However, the differences of average absolute error start to decrease with the copy
fraction > 15%. On the irreducible data, the differences of average absolute error are close to zero
until the copy fraction > 15%.

By observing Figure 2, we found the prediction of the KM2 estimator will not change much if the
copy fraction p is too small. For example, the difference between the estimated mixture proportion
by employing samples Su and XP and the ones by employing samples Su and XP[′] [can be hardly]
observed if XP and XP[′] [only differ from in one or two points. For simplicity and consistency, we]
select hyper-parameter p to be 10% for all the following experiments.

4.2 EXPERIMENTS ON REAL-WORLD DATASETS

We illustrate the absolute estimation errors of different estimators on the real-world datasets. Totally,
7 baseline methods are used in the experiments, which are AlphaMax (AM) (Jain et al., 2016),
DEDPUL (DPL) (Ivanov, 2019), Elkan-Noto (EN) (Elkan & Noto, 2008), KM1, KM2 (Ramaswamy
et al., 2016), ROC (Scott, 2015), and Rankpruning (RPG) (Northcutt et al., 2017). By using our
method, the regrouped version of them are implemented, which are called ReAM, ReDPL, ReEN,
ReKM1, ReKM2, ReROC, and ReRPG. In Table 1, we compare the absolute estimation errors of
each baseline with those of its regrouped version on different datasets with different sample lengths.
Each number in Table 1 is the average over 6 × 10 experiments.

Table 1 reflects the effectiveness of our regrouping CPE method. Overall, by using our method, the
estimation accuracy is increased for most of the popular CPE methods among most of the datasets
with different sample lengths. By observing the last row, the regrouped version of the estimators has
much smaller average estimation errors except DPL, KM2, and RPG. On the real-world datasets,
Regrouping AlphaMax (ReAM) has the smallest average estimation error among all methods.

5 CONCLUSION

In this paper, we investigate how to reduce the estimation bias of the distributional-assumption-free
CPE method without irreducibility assumption for PU learning. We have proposed regrouping CPE
which can be employed on top of most existing CPE methods. We have also theoretically analyzed
the estimation bias of ReCPE. Empirically, it improves all popular CPE methods on various datasets.
One future work will focus on how to generate a sample from Pp′ instead of using an approximation.


-----

REPRODUCIBILITY STATEMENT

For theoretical results, we have clearly explained any assumptions. A complete proof of the claims can
be founded in the appendix. We have also included an anonymous source code in our supplementary
material. For any datasets used in the experiments, a complete description of the data processing
steps is provided In Section 4.

ACKNOWLEDGMENTS

TL was partially supported by Australian Research Council Projects DP180103424, DE-190101473,
IC-190100031, and DP-220102121. BH was supported by the RGC Early Career Scheme No.
22200720 and NSFC Young Scientists Fund No. 62006202. MG is supported by ARC DE210101624.
GN and MS were supported by JST AIP Acceleration Research Grant Number JPMJCR20U3, Japan.
MS was also supported by the Institute for AI and Beyond, UTokyo.

REFERENCES

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang
Liu. Understanding and improving early stopping for learning with noisy labels. Advances in
_Neural Information Processing Systems, 34, 2021._

Jessa Bekker and Jesse Davis. Estimating the class prior in positive and unlabeled data through
decision tree induction. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: a survey. Mach. Learn.,
109(4):719–760, 2020.

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of
_Machine Learning Research, 11(Nov):2973–3009, 2010._

Marthinus Christoffel, Gang Niu, and Masashi Sugiyama. Class-prior estimation for learning from
positive and unlabeled data. In Asian Conference on Machine Learning, pp. 221–236, 2016.

Marc Claesen, Frank De Smet, Pieter Gillard, Chantal Mathieu, and Bart De Moor. Building classifiers
to predict the start of glucose-lowering pharmacotherapy using belgian health expenditure data.
_arXiv preprint arXiv:1504.07389, 2015._

Balazs Csan´ ad Cs´ aji et al. Approximation with artificial neural networks.´ _Faculty of Sciences, Etvs_
_Lornd University, Hungary, 24(48):7, 2001._

Francesco De Comite, Fran´ c¸ois Denis, Remi Gilleron, and Fabien Letouzey. Positive and unlabeled´
examples help learning. In International Conference on Algorithmic Learning Theory, pp. 219–230.
Springer, 1999.

Franc¸ois Denis. Pac learning from positive statistical queries. In International Conference on
_Algorithmic Learning Theory, pp. 112–126. Springer, 1998._

Marthinus du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from
positive and unlabeled data. In International conference on machine learning, pp. 1386–1394,
2015.

Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and
unlabeled data. In Advances in neural information processing systems, pp. 703–711, 2014.

Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In
_Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data_
_mining, pp. 213–220. ACM, 2008._

Luis Galarraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in´
ontological knowledge bases with amie. The VLDB Journal, 24(6):707–730, 2015.


-----

Chen Gong, Hong Shi, Tongliang Liu, Chuang Zhang, Jian Yang, and Dacheng Tao. Loss decomposition and centroid estimation for positive and unlabeled learning. IEEE transactions on pattern
_analysis and machine intelligence, 2019._

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander Smola. A¨
kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723–773, 2012.

Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and
biased negative data. In International Conference on Machine Learning, pp. 2820–2829, 2019.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In International conference on machine learning, pp. 448–456.
PMLR, 2015.

Dmitry Ivanov. Dedpul: Method for mixture proportion estimation and positive-unlabeled classification based on density estimation. arXiv preprint arXiv:1902.06965, 2019.

Shantanu Jain, Martha White, Michael W Trosset, and Predrag Radivojac. Nonparametric semisupervised learning of class proportions. arXiv preprint arXiv:1601.01944, 2016.

Masahiro Kato, Liyuan Xu, Gang Niu, and Masashi Sugiyama. Alternate estimation of a classifier
and the class-prior from positive and unlabeled data. arXiv preprint arXiv:1809.05710, 2018.

Ryuichi Kiryo, Gang Niu, Marthinus C du Plessis, and Masashi Sugiyama. Positive-unlabeled
learning with non-negative risk estimator. In Advances in neural information processing systems,
pp. 1675–1685, 2017.

Yongchan Kwon, Wonyoung Kim, Masashi Sugiyama, and Myunghee Cho Paik. Principled analytic classifier for positive-unlabeled learning via weighted integral probability metric. Machine
_Learning, pp. 1–20, 2019._

Wee Sun Lee and Bing Liu. Learning with positive and unlabeled examples using weighted logistic
regression. In ICML, volume 3, pp. 448–455, 2003.

Fabien Letouzey, Franc¸ois Denis, and Remi Gilleron. Learning from positive and unlabeled examples.´
In International Conference on Algorithmic Learning Theory, pp. 71–85. Springer, 2000.

Xiaoli Li and Bing Liu. Learning to classify texts using positive and unlabeled data. In IJCAI,
volume 3, pp. 587–592, 2003.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE
_Transactions on pattern analysis and machine intelligence, 38(3):447–461, 2015._

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT
press, 2018.

Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. Compositional vector space models
for knowledge base completion. In ACL, 2015.

Gang Niu, Marthinus Christoffel du Plessis, Tomoya Sakai, Yao Ma, and Masashi Sugiyama. Theoretical comparisons of positive-unlabeled learning against positive-negative learning. In Advances
_in neural information processing systems, pp. 1199–1207, 2016._

Curtis G Northcutt, Tailin Wu, and Isaac L Chuang. Learning with confident examples: Rank pruning
for robust classification with noisy labels. stat, 1050:9, 2017.

Harish Ramaswamy, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel
embeddings of distributions. In International Conference on Machine Learning, pp. 2052–2060,
2016.

Yafeng Ren, Donghong Ji, and Hongbin Zhang. Positive unlabeled learning for deceptive reviews
detection. In Proceedings of the 2014 conference on empirical methods in natural language
_processing (EMNLP), pp. 488–498, 2014._


-----

Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Semi-supervised auc optimization based on
positive-unlabeled learning. Machine Learning, 107(4):767–794, 2018.

Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning
from noisy labels. In Artificial Intelligence and Statistics, pp. 838–846, 2015.

Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise:
Consistency and maximal denoising. In Conference On Learning Theory, pp. 489–511, 2013.

Ugo Tanielian and Flavian Vasile. Relaxed softmax for pu learning. In Proceedings of the 13th ACM
_Conference on Recommender Systems, pp. 119–127, 2019._

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama.
Are anchor points really indispensable in label-noise learning? Advances in Neural Information
_Processing Systems, 32, 2019._

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu,
Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent
label noise. Advances in Neural Information Processing Systems, 33:7597–7610, 2020.

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama.
Sample selection with uncertainty of losses for learning with noisy labels. _arXiv preprint_
_arXiv:2106.00445, 2021._

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama.
Dual t: Reducing estimation error for transition matrix in label-noise learning. Advances in neural
_information processing systems, 33:7260–7271, 2020._

Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent
label-noise learning under a structural causal model. Advances in Neural Information Processing
_Systems, 34, 2021._

Maria A Zuluaga, Don Hush, Edgar JF Delgado Leyton, Marcela Hernandez Hoyos, and Maciej´
Orkisz. Learning from only positive and unlabeled data to detect lesions in vascular ct images. In
_International Conference on Medical Image Computing and Computer-Assisted Intervention, pp._
9–16. Springer, 2011.


-----

