# MAKO: SEMI-SUPERVISED CONTINUAL LEARNING
## WITH MINIMAL LABELED DATA VIA DATA PROGRAM- MING

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Lifelong machine learning (LML) is a well-known paradigm mimicking the human learning process by utilizing experiences from previous tasks. Nevertheless,
an issue that has been rarely addressed is the lack of labels at the individual task
level. The state-of-the-art of LML largely addresses supervised learning, with a
few semi-supervised continual learning exceptions which require training additional models, which in turn impose constraints on the LML methods themselves.
Therefore, we propose Mako, a wrapper tool that mounts on top of supervised
LML frameworks, leveraging data programming. Mako imposes no additional
knowledge base overhead and enables continual semi-supervised learning with a
limited amount of labeled data. This tool achieves similar performance, in terms
of per-task accuracy and resistance to catastrophic forgetting, as compared to fully
labeled data. We ran extensive experiments on LML task sequences created from
standard image classification data sets including MNIST, CIFAR-10 and CIFAR100, and the results show that after utilizing Mako to leverage unlabeled data,
LML tools are able to achieve 97% performance of supervised learning on fully labeled data in terms of accuracy and catastrophic forgetting prevention. Moreover,
when compared to baseline semi-supervised LML tools such as CNNL, ORDisCo
and DistillMatch, Mako significantly outperforms them, increasing accuracy by
0.25 on certain benchmarks.

1 INTRODUCTION

Since 1995, researchers have been actively seeking machine learning (ML) frameworks that can
better mimic the human learning process by retaining memories from past experiences (Thrun &
Mitchell, 1995; Liu, 2017). Hence, in contrast to traditional isolated ML where knowledge is never
accumulated, the concept of lifelong machine learning (LML) is defined as training ML models for
continual learning over task sequences, with a knowledge base to store information that could be
helpful in the future. Under this definition, LML can be seen as continual transfer learning (Pan &
Yang, 2015) with memory or sequential multi-task learning (Caruana, 1997).

Nevertheless, at the level of individual tasks, the current state-of-the-art of LML still remains largely
supervised. A challenge that has rarely been mentioned is that labels can still be expensive and rarely
available for each task, such that supervised training can hardly provide acceptable performance. For
LML, performance is not only measured at task level, but also for the entire lifetime, with attributes
such as resistance to catastrophic forgetting and space-efficiency of knowledge base.

We take a more generic approach to semi-supervised lifelong learning and develop a technique that
can wrap around any existing continual learning algorithm. Specifically, we propose to use data
_programming (Ratner et al., 2016b) to label new data under the supervision of a set of weak labeling_
functions trained from a limited number of labeled data points. Data programming has proven
successful for isolated learning, and so this is the first work integrating it with lifelong learning.

Our approach, called Mako, sits atop an existing lifelong learner to turn it into a semi-supervised
learner. For each task, Mako takes as input a small labeled dataset plus an unlabeled dataset with
no restriction on size, generates labels for the unlabeled data and then updates the lifelong learner
supervised by both sets. This approach builds upon the widely used Snuba algorithm (Varma & R´e,


-----

2016), and we show that its theoretical guarantees still hold in the lifelong setting. Mako’s procedure
leverages automatic hyperparameter tuning, data programming and confidence calibration, in order
to adapt automatic labeling to constantly varying LML tasks. We show that Mako produces adequately high performance, in terms of both task accuracy and prevention of catastrophic forgetting,
that approaches that of training on fully labeled data without costing extra knowledge base space
overhead.

We extensively evaluated Mako on various partially labeled LML task sequences generated from
commonly used datasets including MNIST, CIFAR-10 and CIFAR-100 (LeCun & Cortes, 2010;
Krizhevsky, 2009), while mounting on a diversity of supervised LML frameworks such as Deconvolutional Factorized CNN (DF-CNN), Dynamic Expandable Network (DEN) and Tensor Factorization (TF) (Lee et al., 2019; Yoon et al., 2018; Bulat et al., 2020). We compared the results to
supervised learning on fully labeled data, as well as the same partially labeled data with existing
semi-supervised LML tools: CNNL, ORDisCo and DistillMatch (Baucum et al., 2017; Wang et al.,
2021; Smith et al., 2021). Empirically, we show that the performance of LML methods improves
as giving more training data even with Mako labels, achieving at least 97% performance relative to
LML methods trained on ground-truth labels and being able to beat existing semi-supervised LML
tools by approximately 0.25 higher accuracy.

**Contributions Summary:**

1. Adapting automatic label generation by semi-supervised learning/data programming to LML in
the scenario where labeled training data is expensive to obtain compared to unlabeled for all tasks.
2. Implementing a LML wrapper, namely Mako, for the scenario of expensive labels, such that,
when given partially labeled data of limited size and compared to the current supervised state-ofthe-art with fully labeled data, each LML task sequence (1) requires significantly fewer labeled
data to achieve high accuracy on individual tasks, (2) maintains similar resistance to catastrophic
forgetting and (3) does not cost extra knowledge base storage.
3. Extensive experiments on lifelong binary and multiclass image classification on 3 commonly
used datasets, including comparison to supervised and semi-supervised LML baselines, accomplishing very close performance to supervised LML on fully labeled data and significantly outperforming
baseline semi-supervised LML frameworks.

2 RELATED WORK

Lifelong machine learning (LML) is a concept of continual learning among a sequence of tasks.
Specifically, given multiple machine learning tasks arriving continuously, the LML framework accumulates the knowledge of previous tasks and utilizes it for future ones (Chen & Liu, 2016; Ruvolo
& Eaton, 2013b). The method of retaining and adopting the knowledge base varies depending on
the characteristics of tasks and learning models. For instance, a latent matrix is used to keep prototypical weights for linear models (Kumar & Daume, 2012; Ruvolo & Eaton, 2013a), and for deep
neural networks, either weights of layers (Yoon et al., 2018; Lu et al., 2017; Bulat et al., 2020; Lee
et al., 2019) or extracted features of layers (Rusu et al., 2016; Misra et al., 2016; Gao et al., 2019;
Schwarz et al., 2018) are considered as the knowledge base to transfer. Besides research on models
to transfer knowledge across tasks, the LML community has also been working on task order sorting
for higher overall performance (Sun et al., 2018; Ruvolo & Eaton, 2013b). Nevertheless, all these
works assume that each individual task training is supervised with ground-truth labels for the entire
training set. In many real-life ML tasks, obtaining labels has been and continues to be expensive
(Settles, 2009). Only a few works have addressed this issue on LML in very specific domains, such
as continual sentimental analysis with words partially labeled (Wang et al., 2016) and reinforcement
learning in a sequence of state spaces without explicit rewards (Finn et al., 2017). Different from
previous work, Mako enables high performance in continual learning on generic ML tasks given
partially labeled data, with the cardinality of labels restricted at a small number.

To tackle expensive labels in ML, researchers have purposed various methods such as active learning
(Settles, 2009) or semi-supervised learning (Olivier et al., 2006), which is an ML framework that
takes both labeled and unlabeled data as inputs. The unlabeled data can directly assist the training,
or can serve as a target data set yet to be labeled. In recent years, with the rising enthusiasm and
need in deep learning, various methods have been used to assist semi-supervised learning with DNN
models (Rasmus et al., 2015; Laine & Aila, 2016; Tarvainen & Valpola, 2017). One branch of


-----

semi-supervised learning, namely data programming or weak supervision (Ratner et al., 2016b),
focuses on auto-generation of labels on an unlabeled dataset with the help of a small labeled dataset.
Targeting this problem, researchers leverage weak labeling functions that each can label with an
accuracy slightly higher than a coin flip. Then, ensembling algorithms are able to bag these weak
labelers and produce highly accurate final labels. In other words, the weak labels supervise the
training of the final ensembled generative model. There exists different implementations of such
generative model training such as Snorkel (Ratner et al., 2016a).

Semi-supervised learning is an underexplored problem in the LML setting; all the works mentioned
above have addressed problems in isolated ML only. Early work in semi-supervised incremental
learning, in which all tasks are trained simultaneously as batches of unlabeled data are incrementally observed by a classifier, has identified challenges pertaining to selection of the label generating
model architecture and confidence calibration of the generated labels (Baucum et al., 2017), processes which Mako undertakes autonomously through automatic search for weak labelers before ensembling, and confidence calibration after labels are produced. Pioneering work in semi-supervised
LML requires training of additional models for out-of-distribution detection (Smith et al., 2021) or
generative replay (Wang et al., 2021). Conversely, by focusing on label generation through data programming, our approach achieves similar behavior through a pre-processing workflow that does not
impose additional constraints on the LML approach or require additional knowledge base storage.

For data programming tools, it has been a long standing issue to obtain weak labeling functions
as input. One approach is manually designing the weak labelers by domain experts (Ratner et al.,
2016b;a), while other researchers seek automatic labeler generation. For instance, Snuba (Varma
& R´e, 2016) outputs weak supervisors with an accuracy guarantee on the final ensembled data
programming model. Thus far, Snuba has been evaluated on various isolated ML tasks, with weak
labelers in form of k-nearest-neighbors, decision stumps and logistic regressors, and has been shown
to ace in natural language processing and image classification. It also succeeds in more complex
applications such as classifying bone tumors in medical domain. In our work, we build a weak
labeling function generation module in Mako based on Snuba, aiming to support a larger variety of
labeler models for LML, while maintaining the theoretical accuracy guarantee of generated labels.

3 PROBLEM FORMULATION

**LML: Given a sequence of ML tasks: T1, T2, . . ., Tm, where m is an unbounded integer, each**
task Ti has an underlying data space Xi and label space Yi. Ti has ni,train labeled training data
(Xi,train, Yi,train) _i, such that Xi,train_ _i[n][i,train]_ and Yi,train _i_, following some
task-specific i.i.d. distribution ∼D _Di. The task also has ∈X_ _ni,test testing data ∈Y (Xi,test[n][i,train], Yi,test) drawn from_
the same distribution.

Traditionally, an LML model encounters the tasks in order. At Ti, it trains on the full set of
(Xi,train, Yi,train) with supervised learning and obtain model fi : _j=1_ _j=1_

_[X][i][ 7→]_ [S][i] _[Y][i][. Im-]_
mediately after this training, it predicts the j-th task with _Y[ˆ]ij = fi(Xj,test) for each j ≤_ _i. There_
are two objectives: [S][i]

1. High test accuracy. Formally, denote testing accuracy of task Tj right after the training of task Ti
on (Xi,train, Yi,train) as

_aij =_ _nk=1j,test_ **1(ˆyk = yk)** (1)

_nj,test_

P

with ˆyk and yk the k-th label in _Y[ˆ]ij and Yi,test, respectively. One LML objective in the current_
state-of-the-art is to maximize the peak per-task accuracy up to task i defined as


_a˜i = [1]_


_ajj_ (2)
_j=1_

X


Peak per task accuracy is defined on all tasks i = 1, . . ., m, and shows the LML performance on
the current training task on the basis of the accumulated knowledge.

2. Low catastrophic forgetting. A critical aspect of LML is to prevent catastrophic forgetting of
previous tasks. The resistance to catastrophic forgetting can be quantified by the average testing
accuracy of all tasks encountered so far. For each task Ti, denote a second objective, final accuracy


-----

**up to task i as**


_i_

_a¯i = [1]_ _aij_ (3)

_i_

_j=1_

X


The second objective is to maximize ¯ai for all i = 1, . . ., m.The gap between peak per-task accuracy
and final accuracy expresses how much knowledge of previous tasks LML method can preserve
through the streams of tasks. Therefore, we define a third metric to be maximized, catastrophic
**forgetting ratio up to task i, as**


_i_

_ci = [1]_

_i_

_j=1_

X


_aij_ (4)
_ajj_


This metric is less than 1 if the LML model loses its performance on the earlier tasks, and it is greater
than 1 if there is positive knowledge transfer from the later tasks to the earlier ones.

**Semi-supervised LML with data programming:** The key challenge we want to address is the
cost of labeling data in all LML tasks. That is, the number of available labeled training data is
small. Therefore, we would like to split (Xi,train, Yi,train) in the traditional LML problem into
disjoint (Xi,L, Yi,L, Xi,U ), where (Xi,L, Yi,L) is a set of labeled training data with restriction on
size and Xi,U is a set of unlabeled data with no such restriction. In other words, |Xi,L| is small, e.g.
_Xi,L_ 150 for a 10-way image classification problem.
_|_ _| ≤_

Apparently, the LML model is now trained with semi-supervised learning. For task Ti, we would like
This model automatically labelsto leverage data programming on X (Xi,Ui,L with, Yi,L Y, Xi,U[π]i,U )[=] to first train a generative model[ π][i][(][X][i,U] [)][. Then, the LML framework will] πi : Xi 7→Yi.
be trained on (Xi,L, Yi,L, Xi,U _, Yi,U[π]_ [)][. Denote the LML model immediately after this training as]
_fi[π]_ [:][ S]j[i] =1 _[X][i][ 7→]_ [S]j[i] =1 _[Y][i][, and][ ˆ]Yij[π]_ [=][ f][ π]i [(][X][j,test][)][ for each task][ j][ ≤] _[i][.]_

In this scenario, we redefine testing accuracy in Equation 1 as

_nk=1j,test_ **1(ˆyk[π]** [=][ y][k][)]
_a[π]ij_ [=] (5)

_nj,test_

P

where the only difference is that ˆyk[π] [is the][ k][-th label in][ ˆ]Yij[π] [instead of][ ˆ]Yij. From Equation 5, we can
then redefine metrics in Equations 2, 3 and 4 as


_a[π]ij_ (8)

_a[π]jj_


_a˜[π]i_ [= 1]

_i_


_i_ _i_

_a[π]jj_ (6) _a¯[π]i_ [= 1] _a[π]ij_ (7) _c[π]i_ [= 1]

_i_ _i_

_j=1_ _j=1_

X X


_j=1_


Our research problem is to design a semi-supervised LML framework leveraging data programming
that minimizes the performance between using partially labeled data, (Xi,L, Yi,L, Xi,U _, Yi,U[π]_ [)][, and]
the upper-bound performance using fully labeled data, (Xi,L, Yi,L, Xi,U _, Yi,U_ ), in terms of these
three metrics while imposing no additional knowledge base storage overhead in the LML framework.

4 MAKO: LIFELONG MACHINE LEARNING FRAMEWORK

We implemented Mako, a semi-supervised LML framework that can be mounted on top of any existing LML tool, with (Xi,L, Yi,L, Xi,U ) as input for each task Ti. The automatic label generation
of Yi,U[π] [is done by data programming, consisting of generating weak labeling functions and ensem-]
bling a generative model. Nevertheless, different from data programming on isolated ML, due to the
variation of tasks throughout LML, we need to actively adapt both the weak and the strong models
to maximize the performance at each task. This is accomplished by automatic search on architecture and training hyperparameter search before weak labeler generation and confidence calibration
after the ensembled model πi is trained. The following subsections explains the Mako workflow
step-by-step, as illustrated in Figure 1.


-----

|Data Programming|Col2|
|---|---|
|Bootstrapping|Ensemble|
|LF Generation||


Confidence Calibration LML Tools

Strong Temperature Scaling DF-CNN DEN TF EWC
Labels Calibrated

Logits Labels

Thresholding Testing Accuracy

Satisfying the Theoretical Guarantee

Hyperparameter Search Data Programming

Architecture Training Bootstrapping

Parameters Labeler

Learning Model

Rates LF Generation

Batch Size Epochs

Repeat Until Sufficient Labelers


Figure 1: Mako Framework

4.1 AUTOMATIC HYPERPARAMETER SEARCH

Before producing weak labeling functions, two questions must be answered: (1) How do we choose
the model architecture for our weak labeling functions? (2) How do we train these models after
architecture is decided? Formally, these questions lead to the decision of architectural and training
hyperparameters of labelers, which Mako accomplishes automatically.

The only prior knowledge required is a search space, manually designed, of the architecture and
training hyperparameters. For instance, given that the task is image classification, the search space
of labeler architecture will include CNNs with the same input and output dimensions, but different
configurations of hidden layers. Similarly, the training hyperparameters will be searched on different
learning rates, batch sizes and number of epochs.

The automatic tuning of weak labelers therefore traverses the search space until a satisfactory configuration is found. The stopping criterion is defined as follows: after training ten labelers with a
given configuration of architecture and training hyperparameters on different bootstrapped subsets
of (Xi,L, Yi,L), 9 out of 10 achieves an accuracy on the entire (Xi,L, Yi,L) above some threshold
_a. This criterion shows the generalization capability of sampled labelers and alternative criteria can_
be used, such as 8 out of 10 or average accuracy ≥ _a. We accept such a configuration because we_
train the labelers on bootstrapped subsets of data during data programming, which will be further
explained in 4.2. If no configuration in the search space is satisfactory, we decrease a by 0.05 and
repeat the search. Hence, we can start from a sufficiently high a such as 0.9. For efficiency, Mako
selects the configuration with the smallest number of training epochs among all the acceptable ones.

There exists a efficiency-automation trade-off in this hyperparameter search. The less prior knowledge we have for the task, the larger search space we need to input to Mako and thus a larger time
overhead. In contrast, more manual exploration on the task will lead to a smaller search space and
therefore efficiency.

4.2 GUARANTEED DATA PROGRAMMING WITH BOOTSTRAPPING ON TRAINING SET

After deciding the architecture and training hyperparameters of weak labeling functions, Mako kicks
off weak labeler training and ensembling of generative model πi for each task Ti. For the training
process, we are inspired by Snuba (Varma & R´e, 2016) due to the fact that it has a theoretical
accuracy guarantee in the final generative model. Nonetheless, in order to increase variance among
the labelers, Snuba trains them on different selected subsets of features. This procedure does little
favor in LML, where we might encounter complex tasks in the sequence that requires the entire
feature space. For instance, in image classification, excluding pixels in training may lead to a failure
in capturing important feature-target relationships, especially when we want to take advantage of
local connectivity of CNNs. Therefore, Mako uses bootstrapping on (Xi,L, Yi,L) and trains each
labeler with a sampled subset. The bootstrapping size can be either manually given or searched as a
training hyperparameter.

Besides bootstrapping on data, the rest of weak labeler generation process remains unchanged from
Snuba in order to maintain its theoretical guarantee. Snuba has such a guarantee because it checks
an exit condition on whether the committed labelers have labeled sufficient number of data points in
_Xi,L with acceptable confidence. If so, these labelers guarantee that the final generative model will_
have a learned accuracy on Xi,L close to the labeling accuracy on Xi,U . Formally, this guarantee
can be stated as Proposition 1.

**Proposition 1 (Theoretical Guarantee of Snuba): Given h committed weak labelers, denote the**
_empirical accuracies of all committed weak labelers on Xi,L as a vector ai,L,w. The generative_


-----

_model training outputs learned accuracies of these labelers on Xi,L as ai,L,g before ensembling._
_Moreover, the unknown true accuracies of the labelers on Xi,U after generative model training are_
_denoted as ai,U,g, with ai,L,w, ai,L,g, ai,U,g_ R[h]. We have a measured error _ai,L,w_ _ai,L,g_
_ϵ. Then, if each labeler labels a minimum of ∈_ _||_ _−_ _||∞_ _≤_

1 2h2
_d_ (9)
_≥_ 2(γ _ϵ)[2][ log]_ _δ_

_−_  

_data points in Xi,L with above some given confidence threshold ν for all iterations, we can guaran-_
_tee that ||ai,U,g −_ _ai,L,g||∞_ _< γ for all iterations with probability 1 −_ _δ._

The proof of Proposition 1 from the original Snuba paper is included in Appendix A. We would like
to show that our bootstrapping technique maintains this guarantee, as in Proposition 2.

**Proposition 2 (Theoretical Guarantee of Mako Weak Labeler Generation): Despite the modi-**
_fications from Snuba, the Mako weak labeler generation process satisfies the theoretical guarantees_
_described in Proposition 1._

**Proof of Proposition 2: Although our weak labelers are generated with (1) different model archi-**
tectures and (2) sampling by bootstrapping on data instead of features, it still satisfies the definition
of weak heuristics required by Snuba. That is, a function f : _i_ _i_ _abstain_, where abstain
is a special label assigned to data points with a confidence lower than the threshold X _7→Y_ _∪{_ _} ν._

Consequently, by Proposition 2, if Mako checks the same theoretical exit condition at each iteration,
the final generative model satisfies the same theoretical guarantee as in Snuba. The hyperparameter
search described in 4.1 can be understood as looking for labelers with high ai,L,w. Based on this
probablistically achieve high labeling accuracyguarantee, we will have ||ai,U,g − _ai,L,w||∞_ _≤ ai,U,gϵ +. γ with probability 1 −_ _δ, and hence we can_

The weak labeling functions are then used to label both Xi,L and Xi,U and the labels are input for
the training of generative model πi. We label Xi,L as well for confidence calibration, which will be
explained in the following subsection.

4.3 CONFIDENCE CALIBRATION AND THRESHOLDING

Considering the variability and complexity of LML tasks, we believe the generative model labels
can be further improved to approach the ground truth. Therefore, Mako takes an extra step to adjust
the trained generative model πi by confidence calibration on the produced logits on Xi,U .

Confidence calibration (Guo et al., 2017) is the process to align the confidence of labeling a data
point to the accuracy, which is an estimation of the probability that the data point is labeled correctly.
The procedure of confidence calibration is adjusting the logits in order to minimize the expected
calibration error (ECE), formalized as
_pmin[π]k,i,U_ Ep[π]k,i,U [[][|][ Pr[][y]k,i,U[π] [=][ y][k,i,U] _[|][ p]k,i,U[π]_ [=][ p][]][ −] _[p][|][]]_ (10)

where yk,i,U is the ground truth label of the k-th data point in Xi,U, yk,i,U[π] [is its predicted label by]
the data programming generative model and p[π]k,i,U [is the associated confidence. However, since the]
ground truth labels of Xi,U is unknown, this approach is infeasible. This is the reason for training the
generative model under the supervision of weak labels on both Xi,U and Xi,L. The model is hence
expected to have the similar ECE on labeled and unlabeled data. Mako then performs temperature
scaling (Platt, 1999) to minimize ECE on Xi,L, and then apply the same temperature on Xi,U to
adjust the logits.

After calibration, there could still exist data points labeled with low confidence. We then threshold
_Xi,U with a confidence of 1/nclasses + β for some small positive β. For example, in a 10-way_
classification problem, pick β = 0.01. We only keep the subset of Xi,U with calibrated confidence
_≥_ 0.1 + 0.01 = 0.11. Denote this kept portion of unlabeled data and labels as (Xi,U,c, Yi,U,c[π] [)][.]

4.4 LML TASK TRAINING AND EVALUATION

Finally, Xi,L, Yi,L, Xi,U,c and Yi,U,c[π] [will be input into the mounted LML tool for the training of]
task Ti. The evaluation of the current model will be performed on all the hold-out testing data set
(Xtest,1, Ytest,1), . . ., (Xtest,i, Ytest,i).


-----

Mako does not alter the internal procedures of the LML tool. Instead, it generates labels for unlabeled data and then feeds it to the LML tool as a black box. This design enables high modularity
and the LML tool can be easily replaced. In the experiment section, we will demonstrate Mako’s
performance when mounting on various of LML paradigms such as DF-CNN, DEN and TF. This
design also imposes no additional knowledge space storage in LML.

5 EXPERIMENTS

5.1 DATASETS AND LML TASK SEQUENCES

We evaluate Mako on LML task sequences generated from 3 commonly-used image classification
datasets: MNIST (LeCun & Cortes, 2010), CIFAR-10 and CIFAR-100 (Krizhevsky, 2009).

We mount Mako on existing supervised LML tools: DF-CNN, and TF (Lee et al., 2019; Bulat
et al., 2020) to enable semi-supervised learning, and compare the performance to supervised LML
on labeled data only as well as all data fully labeled. The task sequences are:

1. Binary MNIST: for each task, pick classes 0 vs 1, 0 vs 2, ..., 8 vs 9, with 45 tasks in total.
2. Binary CIFAR-10: same construction as binary MNIST.

Additionally, we compare Mako-mounted supervised LML tools to existing semi-supervised LML:
CNNL, ORDisCo, and DistillMatch (Baucum et al., 2017; Wang et al., 2021; Smith et al., 2021).
We generate the same task sequences as in their papers:

3. 10-way CIFAR-10: the entire CIFAR-10 as one task.
4. 5-way CIFAR-100: for each task, pick each superclass in CIFAR-100, with 20 tasks in total.
5. 5-way MNIST: for each task, pick classes 0-4 and 5-9 in MNIST, with 2 tasks in total.

Details of the tasks such as data splits, hyperparameters searched and labeling accuracy are explained
in Appendix B.

5.2 COMPARISON TO SUPERVISED LML

For each task, data is split into a labeled training set, an unlabeled training set and a hold-out test
set. The details of data split and weak labeler models searched are included in Appendix B.

Table 1 and 2 summarize the performance of LML methods according to the amount of training data:
**L standing for the labeled training set and U with number denoting the number of instances in the**
unlabeled training set used during training of each task. These tables show peak per-task accuracy,
final accuracy and catastrophic forgetting ratio up to the last task (Eqn. 6, 7 and 8, respectively).
Additionally, we trained the LML methods on the same data with true labels instead of the Makogenerated labels to quantify the quality of the generated labels. LML methods achieve better peak

|LML|Data|Performance|Col4|Col5|Relative to True Labels|Col7|
|---|---|---|---|---|---|---|
|||Per-Task Acc.|Final Acc.|Forget. Ratio|Per-Task Acc.|Final Acc.|
|TF|L|95.6 0.4 ±|96.8 0.6 ±|1.02 0.01 ±|-|-|
||L+U30|95.9 0.4 ±|97.0 0.4 ±|1.01 0.01 ±|0.99|1.00|
||L+U60|96.1 0.3 ±|96.8 0.5 ±|1.01 0.01 ±|0.99|0.99|
||L+U120|96.2 0.2 ±|96.9 0.2 ±|1.01 0.00 ±|0.99|0.99|
||L+U240|96.5 0.2 ±|96.4 0.2 ±|1.00 0.00 ±|0.98|0.98|
|DF-CNN|L|93.8 0.4 ±|95.4 0.3 ±|1.02 0.01 ±|-|-|
||L+U30|94.6 0.3 ±|96.2 0.4 ±|1.02 0.01 ±|0.98|1.00|
||L+U60|94.9 0.2 ±|96.1 0.7 ±|1.01 0.01 ±|0.98|0.99|
||L+U120|95.2 0.3 ±|95.8 0.9 ±|1.01 0.01 ±|0.98|0.99|
||L+U240|95.9 0.2 ±|94.5 1.3 ±|0.99 0.01 ±|0.98|0.97|



Table 1: Supervised LML experiments on binary MNIST tasks, showing mean ± standard deviation.
LML models are trained on either Labeled data only or Labeled data and instances of Unlabeled data
with Mako-generated labels, and evaluated three metrics up to task 45 (Eqn. 6, 7 and 8) as well as
accuracy metrics relative to LML models trained on the same data with true labels instead.


-----

|LML|Data|Performance|Col4|Col5|Relative to True Labels|Col7|
|---|---|---|---|---|---|---|
|||Per-Task Acc.|Final Acc.|Forget. Ratio|Per-Task Acc.|Final Acc.|
|TF|L|81.5 0.2 ±|76.8 0.7 ±|0.95 0.01 ±|-|-|
||L+U200|83.4 0.4 ±|76.8 1.6 ±|0.93 0.02 ±|0.99|1.00|
||L+U400|84.3 0.3 ±|76.7 1.4 ±|0.92 0.02 ±|-|-|
|DF-CNN|L|80.7 0.3 ±|80.4 0.7 ±|1.01 0.01 ±|-|-|
||L+U200|82.7 0.3 ±|80.0 0.5 ±|0.98 0.01 ±|0.99|0.99|
||L+U400|84.0 0.3 ±|80.0 0.4 ±|0.96 0.01 ±|-|-|


Table 2: Supervised LML experiments on binary CIFAR-10 tasks, mean ± standard deviation.

per-task accuracy as more Mako-labeled data is provided for training. This shows that Mako is able
to generate useful labels for training while the current task, and especially data distribution, keeps
changing in the lifelong learning setting. Training with Mako labels has no significant improvement
in final accuracy, but, compared to LML models trained on the true data, both peak per-task accuracy
and final accuracy are at least 97% of the counterparts. Learning curves for each experiment are
shown in Appendix C.

5.3 COMPARISON TO SEMI-SUPERVISED LML

As discussed in Section 2, there is little prior work in semi-supervised LML settings. We identify
three approaches for comparison: CNNL, ORDisCo, and DistillMatch (Baucum et al., 2017; Wang
et al., 2021; Smith et al., 2021). Each baseline approach contains multiple modules and networks
which each require their own tuning; to enable fair comparisons, we replicate the experimental conditions (data set, amount of labeled vs. unlabeled data, task definitions, and network architecture)
and compare our approach to the best results reported in each baseline’s original publication. Specifically, we replicate the instance-incremental learning experiments of CNNL on MNIST and CIFAR10, and the class-incremental learning experiments of ORDisCo and DistillMatch on CIFAR-10
and CIFAR-100, respectively. We briefly describe each experiment below; for more details, see
Appendix B for data splits and the original publications for full experimental descriptions.

**Instance-Incremental Experiments.** We compare semi-supervised learning using Mako in an
instance-incremental setting (i.e., where all tasks are present at every epoch, but subsequent epochs
contain different batches of unlabeled data), using an identical two-convolutional-layer CNN as described in (Baucum et al., 2017). The MNIST experiment uses a labeled data set of 150 images,
with 1000 unlabeled images introduced in each of 30 instance-incremental batches. The CIFAR10 experiment uses a labeled data set of 2000 samples, with subsequent batches of 1000 unlabeled
images. Each Mako experiment was run over 10 random seeds. For additional context, we include
the performance of the same network trained in the instance-incremental setting where all data is
labeled with the ground truth, representing an upper bound on semi-supervised performance. As
shown in Table 3, using Mako labels in place of CNNL’s semi-supervised labeler results in comparable classification accuracy with better sample efficiency (MNIST), or strictly higher classification
accuracy (CIFAR-10). Learning curves for each experiment are shown in Appendix D.

|Col1|MNIST|Col3|CIFAR-10|Col5|
|---|---|---|---|---|
|Approach|Final Acc.|Batches to Saturation|Final Acc.|Batches to Saturation|
|CNNL Mako Labeled Fully Labeled|90.0 90.0 0.4 ± 99.0 0.1 ±|26 3.3 1.6 ± 17.0 4.0 ±|45.7 54.2 0.5 ± 57.6 0.5 ±|25 26.5 1.1 ± 27.1 1.7 ±|


MNIST CIFAR-10

Approach Final Acc. Batches to Saturation Final Acc. Batches to Saturation

CNNL **90.0** 26 45.7 25

Mako Labeled **90.0 ± 0.4** 3.3 ± 1.6 **54.2 ± 0.5** 26.5 ± 1.1

_Fully Labeled_ _99.0 ± 0.1_ _17.0 ± 4.0_ _57.6 ± 0.5_ _27.1 ± 1.7_


Table 3: Instance-incremental semi-supervised LML experiments, showing mean ± standard deviation (where available). Final accuracy (see Eqn. 7) is measured on the data set’s standard held out
test set, batches to saturation is measured as the first epoch where a 3-batch sliding window average
meets or exceeds the final accuracy

**Class-Incremental Experiments. We compare state-of-the-art LML methods (DF-CNN, DEN,**
and TF (Lee et al., 2019; Yoon et al., 2018; Bulat et al., 2020)) using Mako labels to perform semisupervised learning in a class-incremental setting (i.e., the model is sequentially presented with tasks
containing new sets of classes). For all experiments below, each Mako-enabled approach was run


-----

over 10 random seeds, and we also include the performance of the same using all ground truth labels.
We first compare Mako to ORDisCo over five binary CIFAR-10 tasks using 400 labeled instances,
with the remaining data unlabeled. We note that we could not directly replicate the ORDisCo classification network architecture due to a lack of details in the original publication, but instead show
that using Mako labels achieve a higher total classification accuracy with DF-CNN and TF using
a significantly smaller network (4 convolutional layers for the Mako approaches, compared to ORDisCo’s 9 convolutional layers), shown in Table 4. Additionally, we compare Mako to DistillMatch
over the 5-way CIFAR-100 using 20% labeled data, replicating DistillMatch’s ParentClass task. As
shown in Table 4, all of the LML methods enabled by Mako labeling meet or exceed DistillMatch’s
performance over the learning task. Learning curves for each experiment are shown in Appendix D.

|Col1|CIFAR-10|Col3|CIFAR-100|Col5|
|---|---|---|---|---|
||Approach|Final Acc.|Approach|Final Acc.|
|Baseline|ORDisCo|74.8|DistillMatch|24.4 0.4 ±|
|Semi- Supervised|Mako-labeled DF-CNN Mako-labeled DEN Mako-labeled TF|86.8 1.3 ± 61.4 2.3 ± 82.1 2.7 ±|Mako-labeled DF-CNN Mako-labeled DEN Mako-labeled TF|50.0 0.8 ± 24.1 0.6 ± 48.9 2.1 ±|
|Supervised|Fully-labeled DF-CNN Fully-labeled DEN Fully-labeled TF|86.8 1.4 ± 61.4 3.7 ± 82.5 4.6 ±|Fully-labeled DF-CNN Fully-labeled DEN Fully-labeled TF|52.4 1.4 ± 23.7 0.4 ± 51.0 1.8 ±|



Table 4: Class-incremental semi-supervised LML experiments, showing mean ± standard deviation
(where available). Final accuracy (see Eqn. 7) is measured on the data set’s standard held out test
set, evaluated on all tasks after the final task completes training. For a breakdown of individual task
accuracy and catastrophic forgetting ratios, see Appendix D.

6 CONCLUSION AND FUTURE WORK

In this paper, we identified the challenge that collecting task-level labeled data for LML is expensive.
We address this challenge with data programming, where labels are automatically generated, and
implemented the Mako framework that can be mounted on top of existing LML tools. Mako takes
in a limited number of labeled data and an unlimited number of unlabeled data, aiming to minimize
the performance gap to using the same data but fully labeled. We demonstrated Mako can achieve
sufficiently high accuracy per task as well as resistance to catastrophic forgetting, while costing no
additional knowledge base storage, over a set of common image classification LML task sequences.

Future work on this topic can consider how to better resolve the issue of expensive labels at individual LML tasks. This includes how to extend Mako to a larger variety of LML task sequences. For
instance, what alternative methods can be used for labeler hyperparameter tuning other than searching, what prior knowledge of tasks could assist automatic labeling, and how should the efficiencycapability trade-off of different models be balanced. Outside of the Mako framework, alternative
solutions can modify existing LML tools directly, compromising a certain degree of modularity
while possibly improving the overall LML performance.

Another interesting future direction is to consider the scenario where the labeled data and unlabeled
data input per task are drawn from different distributions. This can be enabled by optimal transport
theory, specifically the minimization of Sinkhorn distance between two distributions (Cuturi, 2013).
Ideally, this approach could allow unlabeled data from other datasets to assist LML, resolving the
issue of not only expensive labels but also expensive data collection.


-----

7 REPRODUCIBILITY STATEMENT

We encourage researchers to replicate our experiments. As discussed in Section 4, we have provided the step-by-step explanation of Mako workflow. We have included more experiment details
in Appendix B for convenience. Additionally, we pushed a sample code to an anonymous GitHub
[repository (https://github.com/mako-anon/mako). All readers are welcomed to pull our](https://github.com/mako-anon/mako)
code and execute Mako on the example data themselves.

REFERENCES

M. Baucum, D. Belotto, S. Jeannet, E. Savage, P. Mupparaju, and C. W. Morato. Semi-supervised
deep continuous learning. In Proceedings of the International Conference on Deep Learning
_Technologies, pp. 11–18, 2017._

A. Bulat, J. Kossaifi, G. Tzimiropoulos, and M. Pantic. Incremental multi-domain learning with network latent tensor factorization. In Proceedings of the AAAI Conference on Artificial Intelligence.
AAAI Press, 2020.

R. Caruana. Multitask Learning. PhD thesis, Carnegie Mellon University, 1997.

Z. Chen and B. Liu. Lifelong Machine Learning. Synthesis Lectures on Artificial Intelligence and
Machine Learning. Morgan & Claypool Publishers, 2016.

M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural
_Information Processing Systems (NIPS), 2:2292–2300, 2013._

C. Finn, T. Yu, J. Fu, P. Abbeel, and S. Levine. Generalizing skills with semi-supervised reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR),
2017.

Y. Gao, Q. She, J. Ma, M. Zhao, W. Liu, and A. L. Yuille. NDDR-CNN: Layer-wise feature fusing
in multi-task cnn by neural discriminative dimensionality reduction. In Proceedings of the IEEE
_Conference on Computer Vision and Pattern Recognition, pp. 3205–3214, 2019._

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. Pro_ceedings of the International Conference on Machine Learning (ICML), 70:1321–1330, 2017._

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. IEEE Conference
_on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016._

A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University
of Toronto, 2009.

A. Kumar and H. Daume. Learning task grouping and overlap in multi-task learning. In Proceedings
_of the 29th International Conference on Machine Learning, pp. 1383–1390. Omnipress, July_
2012. ISBN 978-1-4503-1285-1.

S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. Proccedings of the Inter_national Conference on Learning Representations (ICLR), 2016._

[Y. LeCun and C. Cortes. Mnist handwritten digit database. 2010. URL http://yann.lecun.](http://yann.lecun.com/exdb/mnist/)
[com/exdb/mnist/.](http://yann.lecun.com/exdb/mnist/)

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.

S. Lee, J Stokes, and E. Eaton. Learning shared knowledge for deep lifelong learning using deconvolutional networks. In Proceedings of the International Joint Conference on Artificial Intelligence,
pp. 2837–2844, 2019.

B. Liu. Lifelong machine learning: a paradigm for continuous learning. Frontier of Computer
_Science, 11(3):359–361, 2017._

10


-----

Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. Feris. Fully-adaptive feature sharing in multitask networks with applications in person attribute classification. In Proceedings of the IEEE
_Conference on Computer Vision and Pattern Recognition, July 2017._

I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994–
4003, 2016.

C. Olivier, S. Bernhard, and Z. Alexander. Semi-supervised learning. IEEE Transactions on Neural
_Networks, 20, 2006._

S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transaction on Knowledge and Data
_Engineering, 22(10):1345–1359, 2015._

J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61–74, 1999.

A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko. Semi-supervised learning with
ladder networks. Advances in Neural Information Processing Systems (NIPS), 2015.

A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. R´e. Snrokel: rapid training data creation
with weak supervision. Proceedings of the VLDB Endowment, 11(3), 2016a.

A. Ratner, S. De Sa, C.and Wu, D. Selsam, and C. R´e. Data programming: Creating large training
sets, quickly. Advances in Neural Information Processing Systems (NIPS), pp. 3574–3582, 2016b.

A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu,
and R. Hadsell. Progressive neural networks. CoRR, abs/1606.04671, 2016.

P. Ruvolo and E. Eaton. ELLA: An efficient lifelong learning algorithm. Proceedings of the Inter_national Conference on Machine Learning (PMLR), 28(1), 2013a._

P. Ruvolo and E. Eaton. Active task selection for lifelong machine learning. Proceedings of the
_AAAI Conference on Artificial Intelligence, 27(1), 2013b._

J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the
_International Conference on Machine Learning, pp. 4528–4537, 2018._

B. Settles. Active learning literature survey. Technical report, Department of Computer Science,
University of Wisconsin-Madison, 2009.

J. Smith, J. Balloch, Y. Hsu, and Z. Kira. Memory-efficient semi-supervised continual learning: The
world is its own replay buffer. In Proceedings of the International Joint Conference on Neural
_Networks, 2021._

G. Sun, Y. Cong, and X. Xu. Active lifelong learning with ”watchdog”. In Thirty-Second AAAI
_Conference on Artificial Intelligence, 2018._

A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency
targets improve semi-supervised deep learning results. Advances in Neural Information Process_ing Systems (NIPS), 2017._

S. Thrun and T. M. Mitchell. Lifelong robot learning. In The Biology and Technology of Intelligent
_Autonomous Agents, pp. 165–196. Springer, Berlin, 1995._

P. Varma and C. R´e. Snuba: automating weak supervision to label training data. Proceedings of the
_VLDB Endowment, 12, 2016._

L. Wang, K. Yang, C. Li, L. Hong, Z. Li, and J. Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEE/CVF
_Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5383–5392, June 2021._

S. Wang, Z. Chen, and B. Liu. Mining aspect-specific opinion using a holistic lifelong topic model.
_Proceedings of the International Conference on World Wide Web (WWW), pp. 167–176, 2016._

J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learning with dynamically expandable networks.
_Proccedings of the International Conference on Learning Representations (ICLR), 2018._

11


-----

APPENDIX A SNUBA THEORETICAL GUARANTEE AND PROOF
Proposition 1 in Section 4.2 and its proof are given by the original Snuba paper (Varma & R´e, 2016).
We adapt their notations to our paper to support the propositions in Section 4.2 on an arbitrary LML
task Ti. For convenience, we remove the subscript i in all notations.

**Proof of Proposition 1:** First, the probability of the l error between the labeling accuracies of
_∞_
the generative model on XU and XL is greater than or equal to γ can be bounded by the following
triangular inequality, with the empirical accuracy of one weak labeler in the middle.
Pr[||aU,g − _aL,g||∞_ _≥_ _γ] ≤_ Pr[||aU,g − _aL,w||∞_ + ||aL,w − _aL,g||∞_ _≥_ _γ]_ (11)
_≤_ Pr[||aU,g − _aL,w||∞_ + ϵ ≥ _γ]_

Then, since we have h weak labelers, the following union bound holds for an individual labeler j in
the committed set and its scalar accuracy.


Pr[ _aU,g,j_ _aL,w,j_ + ϵ _γ]_ (12)
_j=1_ _|_ _−_ _|_ _≥_

X


Pr[||aU,g − _aL,w||∞_ + ϵ ≥ _γ] ≤_


In XL, suppose there are dj data points not labeled as abstain by labeler j, the definition of the
empirical accuracy of weak labeler j is


_aL,w,j = [1]_

_dj_


**1(yk = ˆyj,k)** (13)


where yk is the ground truth label of data point k in the non-abstain subset and _yj,kˆ_ is the label
given by heuristic j. By combining Equations 12 and 13, and using Hoeffding’s inequality on the
independent data points, we can get
Pr[||aU,g − _aL,w||∞ϵ ≥_ _γ] = Pr[||aU,g −_ _aL,w||∞+ ≥_ _γ −_ _ϵ]_

_h_

Pr[ _aU,g,j_ _aL,w,j_ _γ_ _ϵ]_

_≤_ _j=1_ _|_ _−_ _| ≥_ _−_

X


_h_

Pr _aU,g,j_
_j=1_ _|_ _−_ _d[1]j_

X h


(14)


**1(yk = ˆyj,k|) ≥** _γ −_ _ϵ_


2 exp( 2(γ _ϵ)[2]dj)_

_≤_ _−_ _−_

_j=1_

X

2h exp( 2(γ _ϵ)[2]_ min(d1, . . ., dh))
_≤_ _−_ _−_

Equation 14 bounds the probability of Snuba’s failure to keepAssume without loss of generality that we commit one more weak labeler per iteration, so for the all ||aU,g − _aL,g||∞_ _< γ in one iteration._
_h iterations, we apply union bound again:_
Pr[||aU,g − _aL,w||∞+ ≥_ _γ −_ _ϵ for any iteration]_


_j=1_ Pr[||aU,g − _aL,w||∞+ ≥_ _γ −_ _ϵ for one iteration]_

X


(15)


2h[2] exp( 2(γ _ϵ)[2]_ min(d1, . . ., dh))
_≤_ _−_ _−_

Therefore, if we denote this probability of failure in any iteration as δ, we have the bound
_δ_ 2h[2] exp( 2(γ _ϵ)[2]_ min(d1, . . ., dh)) (16)
_≤_ _−_ _−_
and thus obtain Equation 9 by letting d = min(d1, . . ., dh).

12


-----

APPENDIX B WEAK LABELER MODEL HYPERPARAMETERS IN
EXPERIMENTS

The LML task configurations are listed in Table 5 for the convenience of replicating our procedure.
Because even after hyperparameter search (Section 4.1), there are still differences in the architectures
and training hyperparameters throughout a single LML sequence, we have provided shrunken search
spaces in weak labeler architecture and training hyperparameters.




|LML Task Sequence|# Tasks|Data Split of X, X, X i,L i,U i,test|Weak Labeler Architecture|Training Hyperparams|
|---|---|---|---|---|
|Binary MNIST|45|120/11880/2000|Arch A|lr [1e-4, 1e-2] ∈ n [5, 10] batches ∈ n [30, 100] epochs ∈ bootstrap = 30|
|Binary CIFAR-10|45|400/9600/2000|Arch B|lr [0.8e-3, 1e-3] ∈ n = 30 batches n [30, 60] epochs ∈ bootstrap = 350|
|10-way CIFAR-10|1|2000/30000/10000|Arch B but final out = 10|lr = 1e-2 n = 30 batches n = 50 epochs bootstrap = 750|
|5-way CIFAR-100|20|500/2000/500|Arch B but final out = 5|lr [0.8e-3, 1e-2] ∈ n = 20 batches n [60, 140] epochs ∈ bootstrap = 200|
|5-way MNIST|2|150/29750/5000|Arch A but final out = 5|lr [0.8e-3, 1.5e-3] ∈ n = 10 batches n [180, 220] epochs ∈ bootstrap = 50|

|conv_1: out_c=6, k=5, p=2, s=1, sigmoid|dropout_1: avgpool_1: rate=0.2 k=2, p=0, s=2|conv_2: out_c=16, k=5, p=0, s=1, sigmoid|d rr ao tp eo =u 0t_ .32: k=a 2v,g lp ap = to t0 eo, nl _ s2 =: 2, d soe iu gn t ms =e 4 o_ 5 id1,: d rr ao tp eo =u 0t_ .53: d soe iu gn t ms =e 2 o_ 1 id2,: d se o on u fts t m=e 2_ a,3 x: f|
|---|---|---|---|


For example, to re-create our binary MNIST LML tasks, one needs to first arrange the 45 tasks as

LML Task Data Split of Weak Labeler Training
# Tasks
Sequence _Xi,L, Xi,U_, Xi,test Architecture Hyperparams

_lr ∈_ [1e-4, 1e-2]

Binary MNIST 45 120/11880/2000 Arch A _nnepochsbatches ∈[30[5,, 100] 10]_

_bootstrap ∈_ = 30

_lr ∈_ [0.8e-3, 1e-3]
_nbatches = 30_

Binary CIFAR-10 45 400/9600/2000 Arch B

_nepochs_ [30, 60]
_bootstrap ∈_ = 350

_lr = 1e-2_

Arch B _nbatches = 30_
10-way CIFAR-10 1 2000/30000/10000
but final out = 10 _nepochs = 50_

_bootstrap = 750_

_lr ∈_ [0.8e-3, 1e-2]

Arch B _nbatches = 20_
5-way CIFAR-100 20 500/2000/500
but final out = 5 _nepochs_ [60, 140]

_bootstrap ∈_ = 200

_lr ∈_ [0.8e-3, 1.5e-3]

Arch A _nbatches = 10_
5-way MNIST 2 150/29750/5000
but final out = 5 _nepochs_ [180, 220]

_bootstrap ∈_ = 50

Table 5: LML Task Configurations in Section 5. Please refer to Figure 2 for weak labeler architectures.

out_c=6, k=5,
p=2, s=1,
conv_1:
sigmoid dropout_1:
rate=0.2 k=2, p=0, s=2avgpool_1: out_c=16, k=5,
p=0, s=1,
conv_2:
sigmoid dropout_2:
rate=0.3 k=2, p=0, s=2,
avgpool_2:
flatten dense_1:
sigmoidout=45, dropout_3:
rate=0.5 dense_2:
sigmoidout=21, dense_3:
softmaxout=2,

(a) Architecture A: used for MNIST and Fashion-MNIST

conv_1:

out_c=16, k=3,
batchnorm,p=1, s=1, out_c=16block_1: out_c=32block_2: out_c=64block_3: k=2, p=0, s=0,
avgpool_1:
flatten dense_1:
softmaxout=2,

relu

(b) Architecture B: used for CIFAR-based experiments

out_c=block_out_c,k=3, p=1, s=3,
batchnorm,conv_1:
relu out_c=block_out_c,k=3, p=1, s=1,
batchnormconv_2: out_c=block_out_c,k=3, p=1, s=2,
batchnorm,conv_4:
relu out_c=block_out_c,k=3, p=1, s=1,
batchnormconv_5:

sum, sum,

relu relu

conv_3: conv_6:

out_c=block_out_c, out_c=block_out_c,

k=1, p=0, s=1, k=1, p=0, s=1,

batchnorm batchnorm

(c) Block structure of Architecture B, where block out c is the out c input into the block

Figure 2: Searched Architectures of Weak Labeling Functions

0 vs 1, 0 vs 2, ..., 8 vs 9. Then, split the labeled training data, unlabeled training data and testing

13


-----

data into 120/11880/2000. All the data splits have balanced classes. Finally, perform our procedure
described in Section 4, starting from the hyperparameter search in the shrunken search space.

Figure 2 illustrates the searched architectures. The search spaces were inspired by various previous
works on CNN designs (Lecun et al., 1998; He et al., 2016). Nonetheless, the final architectures are
much smaller since we need fast training of multiple weak labelers. The notations of the figures are:
_out c: output channels/number of filters, k: size (width and height) of a filter, p: padding and s:_
stride.

Figure 3 shows the labeling accuracies of the Mako-labeled data input into LML.

Figure 3: Labeling accuracies of the task sequences after confidence calibration and thresholding,
with threshold = 1/num classes + 0.01 for all tasks, i.e. labeling accuracies of (Xi,U,c, Yi,U,c[π] [)]
described in 4.3

14


-----

APPENDIX C ADDITIONAL SUPERVISED LML EXPERIMENT ANALYSIS

We visualize peak per-task accuracy and final accuracy of supervised LML experiments (Figure 4)
and learning curve of final accuracies in these experiments (Figure 5), summarized in Table 1 and
2. In these plots, we used the same notation L and U of the main text to specify the training data
setting, except TrueU denoting the case trained on the same instances of U but using the true labels
instead.


0.98

0.96

0.94

0.92

0.9

0.9


0.98

0.96

0.94

0.92

0.9

0.9

|Col1|Col2|Col3|
|---|---|---|
|||Peak Per-task Accuracy Final Accuracy|

|Col1|Col2|Col3|
|---|---|---|
|||Peak Per-task Accuracy Final Accuracy|


Peak Per-task Accuracy
Final Accuracy


(a) TF on Binary MNIST

|Col1|Peak Per-t Final Accu|Col3|Col4|
|---|---|---|---|
|||Peak Per-t Final Accu|ask Accuracy racy|


Peak Per-task Accuracy
Final Accuracy


Label L+U200 L+U400 L+TrueU200


(b) DF-CNN on Binary MNIST


0.85

0.8


0.85

0.8


0.75

0.7


0.75

0.7

|Col1|Peak Per-t Final Accu|Col3|Col4|
|---|---|---|---|
|||Peak Per-t Final Accu|ask Accuracy racy|


Peak Per-task Accuracy
Final Accuracy


Label L+U200 L+U400 L+TrueU200

(d) DF-CNN on Binary CIFAR-10


(c) TF on Binary CIFAR-10


Figure 4: Supervised LML experiments, showing mean and 95% confidence interval. Peak per-task
accuracy increases as more mako-labeled data is provided as training, showing the benefit of the
generated labels.

(a) TF on Binary MNIST (b) DF-CNN on Binary MNIST


(c) TF on Binary CIFAR-10 (d) DF-CNN on Binary CIFAR-10

Figure 5: Learning Curve Comparisons


15


-----

APPENDIX D ADDITIONAL SEMI-SUPERVISED LML EXPERIMENT
ANALYSIS

We include learning curves for each of the instance-incremental semi-supervised LML experiments
in Figure 6 and each of the class-incremental semi-supervised LML experiments in Figure 7. All
training curves show final accuracy up to task i as defined in Equation 3, where task i is the current
task being trained. For easier tasks with high labeling accuracy, such as CIFAR-10, the Makoenabled semi-supervised LML methods perform similarly to the equivalent supervised fully-labeled
task sequence. Otherwise, the Mako-enabled semi-supervised methods approach fully supervised
performance.

(a) 5-way MNIST Experiment (b) 10-way CIFAR-10 Experiment

Figure 6: Instance-incremental semi-supervised vs fully supervised learning curve comparisons

(a) Binary CIFAR-10 Experiment

(b) 5-way CIFAR-100 Experiment

Figure 7: Class-incremental semi-supervised vs fully supervised learning curve comparisons.

We additionally analyze the class-incremental semi-supervised LML baseline experiments for catastrophic forgetting, using the catastrophic forgetting ratio defined in Equation 4. Catastrophic forgetting measures are not available for the baseline methods ORDisCo and DistillMatch as they were
not reported in the original publications. The results are shown in Table 6.

For a further breakdown of results, we include task-specific performance measures for all of the
semi-supervised LML experiments in Tables 7, 8, 9, and 10.

16


-----

|Col1|CIFAR-10|Col3|CIFAR-100|Col5|
|---|---|---|---|---|
||Approach|Forget. Ratio|Approach|Forget. Ratio|
|Semi- Supervised|Mako-lab. DF-CNN Mako-lab. DEN Mako-lab. TF|0.98 0.04 ± 0.72 0.16 ± 0.92 0.09 ±|Mako-lab. DF-CNN Mako-lab. DEN Mako-lab. TF|0.85 0.18 ± 0.48 0.16 ± 0.83 0.16 ±|
|Supervised|Fully-lab. DF-CNN Fully-lab. DEN Fully-lab. TF|0.98 0.04 ± 0.72 0.17 ± 0.93 0.10 ±|Fully-lab. DF-CNN Fully-lab. DEN Fully-lab. TF|0.83 0.19 ± 0.48 0.16 ± 0.82 0.16 ±|


Table 6: Catastrophic forgetting ratios (see Equation 4) for class-incremental semi-supervised LML
experiments, showing mean ± standard deviation

|Col1|Col2|Mako-labeled (Semi-supervised)|Col4|Fully-labeled (Supervised)|Col6|
|---|---|---|---|---|---|
|Approach|Task|Final Acc.|Forget. Ratio|Final Acc.|Forget. Ratio|
|DF-CNN|0 1 2 3 4|88.1 1.4 ± 75.9 4.8 ± 84.4 2.0 ± 93.2 0.5 ± 92.5 1.1 ±|0.953 0.015 ± 0.942 0.063 ± 0.987 0.020 ± 0.999 0.007 ± 1.000 0.000 ±|86.7 5.3 ± 76.6 1.7 ± 84.7 1.1 ± 93.5 0.7 ± 92.6 0.6 ±|0.944 0.061 ± 0.944 0.024 ± 0.989 0.014 ± 0.999 0.003 ± 1.000 0.000 ±|
|TF|0 1 2 3 4|80.5 7.2 ± 67.8 6.7 ± 78.1 8.8 ± 91.3 3.5 ± 92.9 0.5 ±|0.874 0.078 ± 0.845 0.085 ± 0.922 0.105 ± 0.972 0.035 ± 1.000 0.000 ±|79.0 11.2 ± 71.0 5.0 ± 77.4 9.4 ± 92.4 1.4 ± 92.8 1.0 ±|0.853 0.119 ± 0.884 0.064 ± 0.905 0.111 ± 0.987 0.009 ± 1.000 0.000 ±|
|DEN|0 1 2 3 4|50.4 0.7 ± 50.2 3.3 ± 58.1 9.7 ± 60.7 7.5 ± 87.4 3.9 ±|0.559 0.013 ± 0.664 0.049 ± 0.716 0.111 ± 0.678 0.086 ± 1.000 0.000 ±|51.7 4.3 ± 50.8 9.4 ± 54.5 8.7 ± 59.9 5.4 ± 90.0 0.6 ±|0.574 0.045 ± 0.678 0.118 ± 0.674 0.106 ± 0.662 0.063 ± 1.000 0.000 ±|


Mako-labeled (Semi-supervised) Fully-labeled (Supervised)

Approach Task Final Acc. Forget. Ratio Final Acc. Forget. Ratio

0 88.1 ± 1.4 0.953 ± 0.015 86.7 ± 5.3 0.944 ± 0.061

1 75.9 ± 4.8 0.942 ± 0.063 76.6 ± 1.7 0.944 ± 0.024

DF-CNN 2 84.4 ± 2.0 0.987 ± 0.020 84.7 ± 1.1 0.989 ± 0.014

3 93.2 ± 0.5 0.999 ± 0.007 93.5 ± 0.7 0.999 ± 0.003

4 92.5 ± 1.1 1.000 ± 0.000 92.6 ± 0.6 1.000 ± 0.000

0 80.5 ± 7.2 0.874 ± 0.078 79.0 ± 11.2 0.853 ± 0.119

1 67.8 ± 6.7 0.845 ± 0.085 71.0 ± 5.0 0.884 ± 0.064

TF 2 78.1 ± 8.8 0.922 ± 0.105 77.4 ± 9.4 0.905 ± 0.111

3 91.3 ± 3.5 0.972 ± 0.035 92.4 ± 1.4 0.987 ± 0.009

4 92.9 ± 0.5 1.000 ± 0.000 92.8 ± 1.0 1.000 ± 0.000

0 50.4 ± 0.7 0.559 ± 0.013 51.7 ± 4.3 0.574 ± 0.045

1 50.2 ± 3.3 0.664 ± 0.049 50.8 ± 9.4 0.678 ± 0.118

DEN 2 58.1 ± 9.7 0.716 ± 0.111 54.5 ± 8.7 0.674 ± 0.106

3 60.7 ± 7.5 0.678 ± 0.086 59.9 ± 5.4 0.662 ± 0.063

4 87.4 ± 3.9 1.000 ± 0.000 90.0 ± 0.6 1.000 ± 0.000


Table 7: Class-incremental Binary CIFAR-10 experiment results broken down by task, showing
mean ± standard deviation

|Col1|Col2|Mako-labeled (Semi-supervised)|Col4|Fully-labeled (Supervised)|Col6|
|---|---|---|---|---|---|
|Approach|Task|Final Acc.|Forget. Ratio|Final Acc.|Forget. Ratio|
|DF-CNN|0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19|33.9 5.4 ± 35.1 6.5 ± 32.8 4.0 ± 41.6 6.6 ± 37.5 7.5 ± 45.8 5.4 ± 49.3 5.1 ± 46.7 3.5 ± 57.2 3.1 ± 64.2 2.7 ± 70.1 3.1 ± 53.0 3.5 ± 61.7 1.7 ± 54.1 1.8 ± 33.0 1.7 ± 50.0 1.3 ± 48.1 2.2 ± 53.1 1.2 ± 62.7 2.1 ± 70.5 1.8 ±|0.663 0.104 ± 0.547 0.103 ± 0.556 0.067 ± 0.640 0.101 ± 0.530 0.104 ± 0.776 0.080 ± 0.831 0.083 ± 0.768 0.049 ± 0.901 0.048 ± 0.930 0.038 ± 0.944 0.047 ± 0.917 0.055 ± 0.967 0.033 ± 0.970 0.034 ± 0.987 0.032 ± 1.006 0.029 ± 1.002 0.024 ± 1.000 0.013 ± 1.001 0.015 ± 1.000 0.000 ±|31.8 6.0 ± 33.8 6.3 ± 32.8 5.7 ± 38.8 5.9 ± 42.0 7.2 ± 50.7 7.6 ± 51.5 4.5 ± 53.6 3.7 ± 58.2 5.1 ± 65.9 2.3 ± 71.1 3.0 ± 57.7 2.5 ± 64.2 1.7 ± 55.4 1.8 ± 36.7 1.9 ± 52.7 2.5 ± 50.0 2.3 ± 56.1 1.6 ± 67.7 1.8 ± 77.3 1.9 ±|0.583 0.110 ± 0.491 0.093 ± 0.509 0.081 ± 0.562 0.084 ± 0.564 0.092 ± 0.788 0.117 ± 0.804 0.065 ± 0.789 0.066 ± 0.873 0.069 ± 0.923 0.035 ± 0.930 0.040 ± 0.952 0.046 ± 0.963 0.033 ± 0.922 0.029 ± 0.977 0.039 ± 0.971 0.035 ± 0.967 0.021 ± 0.994 0.018 ± 0.996 0.009 ± 1.000 0.000 ±|



Table 8: Class-incremental 5-way CIFAR-100 experiment, DF-CNN, results broken down by task,
showing mean ± standard deviation

17


-----

|Col1|Col2|Mako-labeled (Semi-supervised)|Col4|Fully-labeled (Supervised)|Col6|
|---|---|---|---|---|---|
|Approach|Task|Final Acc.|Forget. Ratio|Final Acc.|Forget. Ratio|
|TF|0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19|36.9 6.7 ± 43.4 4.8 ± 36.2 7.0 ± 56.4 5.2 ± 40.6 13.2 ± 51.0 7.4 ± 46.3 7.1 ± 48.8 9.3 ± 49.6 8.7 ± 60.4 7.1 ± 57.5 11.4 ± 48.3 8.3 ± 53.6 5.2 ± 48.4 5.5 ± 31.0 2.0 ± 41.5 6.0 ± 45.1 3.0 ± 50.0 4.3 ± 62.8 2.7 ± 69.6 2.2 ±|0.729 0.135 ± 0.692 0.082 ± 0.639 0.122 ± 0.870 0.085 ± 0.585 0.193 ± 0.839 0.124 ± 0.767 0.126 ± 0.804 0.152 ± 0.773 0.111 ± 0.885 0.106 ± 0.777 0.154 ± 0.842 0.140 ± 0.881 0.072 ± 0.888 0.097 ± 0.924 0.070 ± 0.836 0.134 ± 0.956 0.073 ± 0.939 0.083 ± 0.974 0.032 ± 1.000 0.000 ±|38.8 9.1 ± 49.9 9.9 ± 37.5 7.0 ± 54.0 5.5 ± 42.4 10.7 ± 50.9 13.6 ± 52.6 5.0 ± 50.8 7.4 ± 47.0 8.5 ± 58.5 6.0 ± 56.9 14.0 ± 46.9 6.0 ± 57.6 3.8 ± 52.9 8.2 ± 31.6 2.9 ± 48.1 2.8 ± 45.2 5.2 ± 54.5 2.4 ± 67.8 2.2 ± 76.4 2.7 ±|0.736 0.174 ± 0.750 0.138 ± 0.617 0.116 ± 0.791 0.072 ± 0.600 0.151 ± 0.760 0.206 ± 0.832 0.085 ± 0.804 0.081 ± 0.732 0.122 ± 0.828 0.095 ± 0.732 0.180 ± 0.798 0.115 ± 0.905 0.064 ± 0.847 0.126 ± 0.945 0.090 ± 0.928 0.046 ± 0.936 0.093 ± 0.969 0.033 ± 0.962 0.023 ± 1.000 0.000 ±|


Mako-labeled (Semi-supervised) Fully-labeled (Supervised)

Approach Task Final Acc. Forget. Ratio Final Acc. Forget. Ratio

0 36.9 ± 6.7 0.729 ± 0.135 38.8 ± 9.1 0.736 ± 0.174

1 43.4 ± 4.8 0.692 ± 0.082 49.9 ± 9.9 0.750 ± 0.138

2 36.2 ± 7.0 0.639 ± 0.122 37.5 ± 7.0 0.617 ± 0.116

3 56.4 ± 5.2 0.870 ± 0.085 54.0 ± 5.5 0.791 ± 0.072

4 40.6 ± 13.2 0.585 ± 0.193 42.4 ± 10.7 0.600 ± 0.151

5 51.0 ± 7.4 0.839 ± 0.124 50.9 ± 13.6 0.760 ± 0.206

6 46.3 ± 7.1 0.767 ± 0.126 52.6 ± 5.0 0.832 ± 0.085

7 48.8 ± 9.3 0.804 ± 0.152 50.8 ± 7.4 0.804 ± 0.081

8 49.6 ± 8.7 0.773 ± 0.111 47.0 ± 8.5 0.732 ± 0.122

9 60.4 ± 7.1 0.885 ± 0.106 58.5 ± 6.0 0.828 ± 0.095

TF 10 57.5 ± 11.4 0.777 ± 0.154 56.9 ± 14.0 0.732 ± 0.180

11 48.3 ± 8.3 0.842 ± 0.140 46.9 ± 6.0 0.798 ± 0.115

12 53.6 ± 5.2 0.881 ± 0.072 57.6 ± 3.8 0.905 ± 0.064

13 48.4 ± 5.5 0.888 ± 0.097 52.9 ± 8.2 0.847 ± 0.126

14 31.0 ± 2.0 0.924 ± 0.070 31.6 ± 2.9 0.945 ± 0.090

15 41.5 ± 6.0 0.836 ± 0.134 48.1 ± 2.8 0.928 ± 0.046

16 45.1 ± 3.0 0.956 ± 0.073 45.2 ± 5.2 0.936 ± 0.093

17 50.0 ± 4.3 0.939 ± 0.083 54.5 ± 2.4 0.969 ± 0.033

18 62.8 ± 2.7 0.974 ± 0.032 67.8 ± 2.2 0.962 ± 0.023

19 69.6 ± 2.2 1.000 ± 0.000 76.4 ± 2.7 1.000 ± 0.000

Table 9: Class-incremental 5-way CIFAR-100 experiment, TF, results broken down by task, showing
mean ± standard deviation

|Col1|Col2|Mako-labeled (Semi-supervised)|Col4|Fully-labeled (Supervised)|Col6|
|---|---|---|---|---|---|
|Approach|Task|Final Acc.|Forget. Ratio|Final Acc.|Forget. Ratio|
|DEN|0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19|21.1 2.9 ± 18.6 2.4 ± 20.3 2.2 ± 20.1 1.6 ± 24.5 4.7 ± 21.9 1.4 ± 19.3 1.0 ± 22.4 2.5 ± 22.0 1.3 ± 24.0 2.6 ± 22.7 1.8 ± 22.9 1.9 ± 21.9 2.1 ± 24.4 3.0 ± 20.9 1.5 ± 22.8 3.0 ± 21.2 0.8 ± 22.2 1.6 ± 23.1 2.1 ± 65.2 3.5 ±|0.433 0.059 ± 0.373 0.061 ± 0.424 0.056 ± 0.381 0.080 ± 0.420 0.106 ± 0.444 0.079 ± 0.405 0.110 ± 0.490 0.171 ± 0.451 0.058 ± 0.421 0.077 ± 0.350 0.048 ± 0.502 0.058 ± 0.408 0.073 ± 0.506 0.056 ± 0.689 0.082 ± 0.497 0.068 ± 0.516 0.059 ± 0.429 0.033 ± 0.416 0.065 ± 1.000 0.000 ±|19.6 1.1 ± 22.7 2.3 ± 21.0 2.4 ± 20.5 3.0 ± 20.1 1.5 ± 21.8 2.5 ± 19.6 2.7 ± 21.8 2.3 ± 21.8 1.3 ± 22.0 2.7 ± 23.1 3.1 ± 22.8 2.5 ± 22.3 1.4 ± 24.4 3.3 ± 20.2 1.1 ± 20.9 1.6 ± 21.1 2.1 ± 21.6 1.7 ± 23.0 2.0 ± 64.1 5.2 ±|0.388 0.033 ± 0.455 0.092 ± 0.477 0.097 ± 0.361 0.058 ± 0.373 0.069 ± 0.423 0.047 ± 0.350 0.062 ± 0.452 0.095 ± 0.472 0.058 ± 0.402 0.062 ± 0.359 0.046 ± 0.477 0.050 ± 0.432 0.041 ± 0.494 0.058 ± 0.737 0.083 ± 0.479 0.051 ± 0.527 0.065 ± 0.434 0.038 ± 0.446 0.064 ± 1.000 0.000 ±|



Table 10: Class-incremental 5-way CIFAR-100 experiment, DEN, results broken down by task,
showing mean ± standard deviation

18


-----

