# MAML IS A NOISY CONTRASTIVE LEARNER
## IN CLASSIFICATION

**Chia-Hsiang Kao[†]** **Wei-Chen Chiu[†]** **Pin-Yu Chen[‡]**

_†National Yang Ming Chiao Tung University, Taiwan_ _‡IBM Research_
chkao.md04@nycu.edu.tw walon@cs.nctu.edu.tw pin-yu.chen@ibm.com

ABSTRACT

Model-agnostic meta-learning (MAML) is one of the most popular and widely
adopted meta-learning algorithms, achieving remarkable success in various learning problems. Yet, with the unique design of nested inner-loop and outer-loop
updates, which govern the task-specific and meta-model-centric learning, respectively, the underlying learning objective of MAML remains implicit, impeding a
more straightforward understanding of it. In this paper, we provide a new perspective of the working mechanism of MAML. We discover that MAML is analogous
to a meta-learner using a supervised contrastive objective in classification. The
query features are pulled towards the support features of the same class and against
those of different classes. Such contrastiveness is experimentally verified via an
analysis based on the cosine similarity. Moreover, we reveal that vanilla MAML
has an undesirable interference term originating from the random initialization
and the cross-task interaction. We thus propose a simple but effective technique,
the zeroing trick, to alleviate the interference. Extensive experiments are conducted on both mini-ImageNet and Omniglot datasets to validate the consistent
improvement brought by our proposed method. [1]

1 INTRODUCTION

Humans can learn from very few samples. They can readily establish their cognition and understanding of novel tasks, environments, or domains even with very limited experience in the corresponding
circumstances. Meta-learning, a subfield of machine learning, aims at equipping machines with such
capacity to accommodate new scenarios effectively (Vilalta & Drissi, 2002; Grant et al., 2018). Machines learn to extract task-agnostic information so that their performance on unseen tasks can be
improved (Hospedales et al., 2020).

One highly influential meta-learning algorithm is Model Agnostic Meta-Learning (MAML) (Finn
et al., 2017), which has inspired numerous follow-up extensions (Nichol et al., 2018; Rajeswaran
et al., 2019; Liu et al., 2019; Finn et al., 2019; Jamal & Qi, 2019; Javed & White, 2019). MAML
estimates a set of model parameters such that an adaptation of the model to a new task only requires
some updates to those parameters. We take the few-shot classification task as an example to review
the algorithmic procedure of MAML. A few-shot classification problem refers to classifying samples from some classes (i.e. query data) after seeing a few examples per class (i.e. support data).
In a meta-learning scenario, we consider a distribution of tasks, where each task is a few-shot classification problem and different tasks have different target classes. MAML aims to meta-train the
base-model based on training tasks (i.e., the meta-training dataset) and evaluate the performance of
the base-model on the testing tasks sampled from a held-out unseen dataset (i.e. the meta-testing
dataset). In meta-training, MAML follows a bi-level optimization scheme composed of the inner
loop and the outer loop, as shown in Appendix A (please refer to Section 2 for detailed definition). In the inner loop (also known as fast adaptation), the base-model θ is updated to θ[′] using
the support set. In the outer loop, a loss is evaluated on θ[′] using the query set, and its gradient is
computed with respect to θ to update the base-model. Since the outer loop requires computing the
gradient of gradient (as the update in the inner loop is included in the entire computation graph), it
is called second-order MAML (SOMAML). To prevent computing the Hessian matrix, Finn et al.

[1Code available at https://github.com/IandRover/MAML_noisy_contrasive_learner](https://github.com/IandRover/MAML_noisy_contrasive_learner)


-----

(2017) propose first-order MAML (FOMAML) that uses the gradient computed with respect to the
inner-loop-updated parameters θ[′] to update the base-model.

The widely accepted intuition behind MAML is that the models are encouraged to learn generalpurpose representations which are broadly applicable not only to the seen tasks but also to novel
tasks (Finn et al., 2017; Raghu et al., 2020; Goldblum et al., 2020). Raghu et al. (2020) confirm
this perspective by showing that during fast adaptation, the majority of changes being made are in
the final linear layers. In contrast, the convolution layers (as the feature encoder) remain almost
static. This implies that the models trained with MAML learn a good feature representation and
that they only have to change the linear mapping from features to outputs during the fast adaptation.
Similar ideas of freezing feature extractors during the inner loop have also been explored (Lee et al.,
2019; Bertinetto et al., 2019; Liu et al., 2020), and have been held as an assumption in theoretical
works (Du et al., 2021; Tripuraneni et al., 2020; Chua et al., 2021).

While this intuition sounds satisfactory, we step further and ask the following fundamental questions:
(1) In what sense does MAML guide any model to learn general-purpose representations? (2) How
do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve
so? (3) What is the role of support and query data, and how do they interact with each other? In
this paper, we answer these questions and give new insights on the working mechanism of MAML,
which turns out to be closely connected to supervised contrastive learning (SCL)[2].

Here, we provide a sketch of our analysis in Figure 1. We consider a setting of (a) a 5-way 1-shot
paradigm of few-shot learning, (b) the mean square error (MSE) between the one-hot encoding of
groundtruth label and the outputs as the objective function, and (c) MAML with a single inner-loop
update. At the beginning of the inner loop, we set the linear layer w[0] to zero. Then, the inner loop
update of w[0] is equivalent to adding the support features to w[0]. In the outer loop, the output of a
query sample q1 is actually the inner product between the query feature ϕ(q1) and all support features
(the learning rate is omitted for now). As the groundtruth is an one-hot vector, the encoder is trained
to either minimize the inner product between the query features and the support features (when they
are from different classes, as shown in the green box), or to pull the inner product between the query
features and the support features to 1 (when they have the same label, as shown in the red box).
Therefore, the inner loop and the outer loop together manifest a SCL objective. Particularly, as the
vanilla implementation of MAML uses non-zero (random) initialization for the linear layer, we will
show such initialization leads to a noisy SCL objective which would impede the training.

In this paper, we firstly review a formal definition of SCL, present a more general case of MAML
with cross entropy loss in classification, and show the underlying learning protocol of vanilla MAML
as an interfered SCL in Section 2. We then experimentally verify the supervised contrastiveness of
MAML and propose to mitigate the interference with our simple but effective technique of the zeroinitialization and zeroing trick (cf. Section 3). In summary, our main contributions are three-fold:

-  We show MAML is implicitly an SCL algorithm in classification and the noise comes from the
randomly initialized linear layer and the cross-task interaction.

-  We verify the inherent contrastiveness of MAML based on the cosine similarity analysis.

-  Our experiments show that applying the zeroing trick induces a notable improvement in testing
accuracy during training and that that during meta-testing, a pronounced increase in the accuracy
occurs when the zeroing trick is applied.

2 WHY MAML IS IMPLICITLY A NOISY SUPERVISED CONTRASTIVE
ALGORITHM?

2.1 PRELIMINARY: SUPERVISED CONTRASTIVE LEARNING

In this work, we aim to bridge MAML and supervised contrastive learning (SCL) and attribute the
success of MAML to SCL’s capacity in learning good representations. Thus, we would like to
introduce SCL briefly.

2We use the term supervised contrastiveness to refer to the setting of using ground truth label information
to differentiate positive samples and negative samples (Khosla et al., 2020). This setting is different from
(unsupervised/self-supervised) contrastive learning.


-----

|Col1|−|Col3|
|---|---|---|

|Col1|−|Col3|
|---|---|---|


**Setting:** **Model:**
5-way 1-shot using MAML with one Image 𝜙 𝑤 Output
inner-loop update under MSE loss. Encoder Linear Layer **0       1**

**0. Set linear layer 𝒘[𝟎]** **to zero.**

**1. Forward single support data**
(𝒔𝟏, 𝒕𝟏).

Image𝑠1 𝜙 Encoder𝜙 𝑀𝑆𝐸𝑙𝑜𝑠𝑠

𝜙𝑠1 [⊤] output 𝑡1 Inner loop

gradient of 𝒘[𝟎]

**2. For single data, 𝒘[𝟎]** **is** **3. Overall, the 𝒌[𝒕𝒉]** **columns of 𝒘[𝟎]** **is added with**
**updated by adding the** **support features of class 𝒌** **and becomes 𝒘[𝟏]** **.**
**support feature:**

𝑤[′] = 𝑤[0] −𝜂(−𝜙𝑠1 𝑡1⊤)

gradient 𝑤[1] = 𝑤[0] −𝜂σ𝑘 (−𝜙𝑠𝑘 𝑡𝑘⊤)

**4. In the outer loop, the loss is**
**computed using query data (𝒒𝟏, 𝒖𝟏)**

Image 𝜙 Encoder

𝑞1 𝜙

𝑀𝑆𝐸

𝑙𝑜𝑠𝑠

𝜙𝑞1 [⊤] output 𝑢1

**5. As 𝒘[𝟏]** **contains the support features, the outer loop essentially adopts a supervised contrastive loss.**

⊤ Outer

sample loss Negative − 2 ∼ ∙ −0 2 Minimize the inner product of query features and support features of loop

different classes (𝑢1 ≠𝑡1).

𝜙𝑞1 [⊤] ∙𝜙𝑠1 Guide the encoder 𝜙 to learn

⊤ more disentangled features

sample loss Positive − 2 ∼ ∙ −0 2 Force the inner product of query features and support features of the

same class (𝑢1 = 𝑡3) to be one.

𝜙𝑞1 [⊤] ∙𝜙𝑠3


Figure 1: A step-by-step illustration showing the SCL objective underlying MAML. Assume the
linear layer w[0] to be zero, we find that, during the inner loop, the i[th] column of w[0] is added with
the support features of class i. In other words, the support features are memorized by the linear layer
during the inner loop. In the outer loop, the output of a query sample is the inner product of ϕ(q1)
and w[1], which is essentially the inner product of the query features and all the support features. The
outer loop loss aims to minimize the MSE between the inner products and the one-hot label. Thus,
MAML displays the characteristic of supervised contrastiveness. Besides, the support data acts as
positive samples when the labels of support and query data match, and vice versa.

Supervised contrastive learning, proposed by Khosla et al. (2020), is a generalization of several
metric learning algorithms, such as triplet loss and N-pair loss (Schroff et al., 2015; Sohn, 2016),
and has shown the best performance in classification compared to SimCLR and CrossEntropy. In
Khosla et al. (2020), SCL is described as “contrasts the set of all samples from the same class as
positives against the negatives from the remainder of the batch” and “embeddings from the same
class are pulled closer together than embeddings from different classes.” For a sample s, the label
information is leveraged to indicate positive samples (i.e., samples having the same label as sample s) and negative samples (i.e., samples having different labels to sample s). The loss of SCL
is designed to increase the similarity (or decrease the metric distance) of embeddings of positive
samples and to reduce the similarity (or increase the metric distance) of embeddings of negative
samples (Khosla et al., 2020). In essence, SCL combines supervised learning and contrastive learning and differs from supervised learning in that the loss contains a measurement of the similarity (or
distance) between the embedding of a sample and embeddings of its positive/negative sample pairs.

Now we give a formal definition of SCL. For a set of N samples drawn from a n-class dataset. Let
_i ∈_ _I = {1, ..., N_ _} be the index of an arbitrary sample. Let A(i) = I \ {i}, P_ (i) be the set of
indices of all positive samples of sample i, and N (i) = A(i) \ P (i) be the set of indices of all
negative samples of sample i. Let zi indicates the embedding of sample i.


-----

**Definition 1 Let Msim be a measurement of similarity (e.g., inner product, cosine similarity).**
_Training algorithms that adopt loss of the following form belong to SCL:_

_LSCL =_ _c[−]p,i[M][sim][(][z][i][, z][p][) +]_ _c[+]n,i[M][sim][(][z][i][, z][n][) +][ c]_

(1)

Xi _p∈XP (i)_ Xi _n∈XN_ (i)

_where c[−]p,i_ _[<][ 0][ and][ c]n,i[+]_ _[>][ 0][ for all][ n][,][ p][ and][ i][; and][ c][ is a constant independent of samples.]_

_We further define that a training algorithm that follows Eq.(1), but with either (a) c[+]n,i_ _[<][ 0][ for some]_
_n, i or (b) c is a constant dependent of samples, belongs to noisy SCL._

2.2 PROBLEM SETUP

We provide the detailed derivation to show that MAML is implicitly a noisy SCL, where we adopt
the few-shot classification as the example application. In this section, we focus on the meta-training
period. Consider drawing a batch of tasks _T1, . . ., TNbatch_ from a meta-training task distribution
_{_ _}_
_D. Each task Tn contains a support set Sn and a query set Qn, where Sn =_ (sm, tm) _m=1_,
_{_ _}[N][way][×][N][shot]_
_Qn =_ (qm, um) _m=1_, sm, qm **R[N][in]** are data samples, and tm, um 1, ..., Nway
are labels. We denote { _}[N][way] N[×]way[N][query] the number of classes in each task, and ∈_ _{Nshot, Nquery ∈{} respectively}_
the number of support and query samples per class. The architecture of our base-model comprises
of a convolutional encoder ϕ : R[N][in] _→_ **R[N][f]** (parameterized by φ), a fully connected linear head
**w ∈** **R[N][f][ ×][N][way]**, and a Softmax output layer, where Nf is the dimension of the feature space. We
denote the k[th] column of w as wk. Note that the base-model parameters θ consist of φ and w.

As shown in Appendix A, both FOMAML and SOMAML adopt a training strategy comprising the
inner loop and the outer loop. At the beginning of a meta-training iteration, we sample Nbatch tasks.
For each task Tn, we perform inner loop updates using the inner loop loss (c.f. Eq. (2)) evaluated
on the support data, and then evaluate the outer loop loss (c.f. Eq. (3)) on the updated base-model
using the query data. In the i[th] step of the inner loop, the parameters {φ[i][−][1], w[i][−][1]} are updated to
_{φ[i], w[i]} using the multi-class cross entropy loss evaluated on the support dataset Sn as_

_Nway_ exp(ϕ[i](s)[⊤]wji)

_L{φi,wi},Sn =_ (s,tE)∼Sn _j=1_ **1j=t[−** log _Nk=1way_ exp(ϕ[i](s)[⊤]wk[i]) ] (2)

X

After Nstep inner loop updates, we compute the outer loop loss using the query dataP _Qn:_

_L{φNstep,wNstep },Qn =_ (q,uE)∼Qn[− log _Nk=1exp(way_ exp(ϕ[N][step]ϕ[N]([step]q)[⊤](wq)u[⊤]Nwstepk[N])[step] ) ] (3)

Then, we sum up the outer loop losses of all tasks, and perform gradient descent to update the

P

base-model’s initial parameters {φ[0], w[0]}.

To show the supervised contrastiveness entailed in MAML, we adopt an assumption that the Encoder
_ϕ is Frozen during the Inner Loop (the EFIL assumption) and we discuss the validity of the as-_
sumption in Section 2.6. Without loss of generality, we consider training models with MAML with
_Nbatch = 1 and Nstep = 1, and we discuss the generalized version in Section 2.6. For simplicity,_

the k[th] element of model output _Nwayj=1exp(ϕexp((s)[⊤]ϕ(ws)k[⊤]0)wj_ [0]) [(respectively] _Nwayj=1exp(ϕexp((q)[⊤]ϕ(wq)k[⊤]1)wj_ [1]) [) of sample]

_s (respectively q) is denoted as skP (respectively qk)._ P

2.3 INNER LOOP AND OUTER LOOP UPDATE OF LINEAR LAYER AND ENCODER

In this section, we primarily focus on the update of parameters in the case of FOMAML. The full
derivation and discussion of SOMAML are provided in Appendix B.

**Inner loop update of the linear layer. In the inner loop, the linear layer w[0]** is updated to w[1] with a
learning rate η as shown in Eq. (4) in both FOMAML and SOMAML. In contrast to the example in
Figure 1, the columns of the linear layer are added with the weighted sum of the features extracted
from support samples (i.e., support features). Compared to wk[0], wk[1] is pushed towards the support
features of the same class (i.e., class k) with strength of 1 sk, while being pulled away from the
_−_
support features of different classes with strength of sk.

**wk1 = wk0** _η [∂L][{][φ,][w][0][}][,S]_ = wk0 + η **E** (4)
_−_ _∂wk[0]_ (s,t) _S[(][1][k=t][ −]_ [s][k][)][ϕ][(][s][)]

_∼_


-----

**Outer loop update of the linear layer. In the outer loop, w[0]** is updated using the query data with
a learning rate ρ. For FOMAML, the final linear layer is updated as follows.

0 0 0
**wk[′]** = wk _ρ_ _[∂L][{][φ,][w][1][}][,Q]_ = wk + ρ **E** (5)
_−_ _∂wk[1]_ (q,u) _Q[(][1][k=u][ −]_ [q][k][)][ϕ][(][q][)]

_∼_

Note that the computation of qk requires the inner-loop updated w[1]. Generally speaking, Eq. (5)
resembles Eq. (4). It is obvious that, in the outer loop, the query features are added weightedly to
the linear layer, and the strength of change relates to the output value. In other words, after the outer
loop update, the linear layer memorizes the query features of current tasks. This can cause a cross_task interference because in the next inner loop there would be additional inner products between_
the support features of the next tasks and the query features of the current tasks.

**Outer loop update of the encoder. Using the chain rule, the gradient of the outer loop loss with re-**
spect to φ (i.e., the parameters of the encoder) is given by _∂L{φ,∂φw1_ _},Q_ = E(q,u)∼Q _∂L{∂ϕφ,(wq1)},Q_ _∂ϕ∂φ(q)_ [+]

**E(s,t)∼S** _∂L{∂ϕφ,w(s1)},Q_ _∂ϕ∂φ(s)_ [, where the second term can be neglected when FOMAML is considered.]

Below, we take a deeper look at the backpropagated error of one query data (q, u) ∼ _Q. The full_
derivation is provided in Appendix B.2.


_Nway_ _Nway_

(qj **1j=u)wj0 + η** **E** qjsj) + su + qt **1t=u]ϕ(s)** (6)
_j=1_ _−_ (s,t)∼S[[][−][(] _j=1_ _−_

X X


_∂L{φ,w1},q_

_∂ϕ(q)_


2.4 MAML IS A NOISY CONTRASTIVE LEARNER

**Reformulating the outer loop loss for the encoder as a noisy SCL loss. We can observe from**
Eq. (6) that the actual loss for the encoder (evaluated on a single query data (q, u) ∼ _Q) is as the_
following.


_Nway_

(qj **1j=u)wj0⊤**

Xj=1 _−stop gradient_


_Nway_

qjsj + su + qt **1t=u]ϕ(s)[⊤]**
_j=1_ _−_

X

stop gradient


_L_ _φ,w1_ _,q =_
_{_ _}_


_ϕ(q) + η_ **E**
(s,t) _S_ [[][−]
_∼_


_ϕ(q)_
(7)


For SOMAML, the range of “stop gradient” in the second term is different:


_Nway_

(qj **1j=u)wj0⊤**

Xj=1 _−stop gradient_


_Nway_

qjsj + su + qt **1t=u]**
_j=1_ _−_

X

stop gradient


_ϕ(s)[⊤]ϕ(q)_
(8)


_ϕ(q) + η_ **E**
(s,t) _S_ [[][−]
_∼_


_L_ _φ,w1_ _,q =_
_{_ _}_


With these two reformulations, we observe the essential difference between FOMAML and SOMAML is the range of stop gradient. We provide detailed discussion and instinctive illustration in
Appendix B.5 on how this explains the phenomenon that SOMAML often leads to faster convergence. To better deliberate the effect of each term in the reformulated outer loop loss, we define the
first term in Eq. (7) or Eq. (8) as interference term, the second term as noisy contrastive term, and
the coefficients _j=1_ qjsj + su + qt **1t=u as contrastive coefficients.**
_−_ [P][N][way] _−_

**Understanding the interference term. In the case of j = u, the outer loop loss forces the model to**
minimize (qj **1)wj[0][⊤]ϕ(q). This can be problematic because (a) at the beginning of training, w[0]**
_−_

is assigned with random values and (b) w[0] is added with query features of previous tasks as shown
in Eq. (5). Consequently, ϕ(q) is pushed to a direction composed of previous query features or to a
random direction, introducing an unnecessary cross-task interference or an initialization interference
that slows down the training of the encoder. Noting that the cross-task interference also occurs at
the testing period, since, at testing stage, w[0] is already added with query features of training tasks,
which can be an interference to testing tasks.

**Understanding the noisy contrastive term. When the query and support data have the same label**
(i.e., u = t), e.g., class 1, the contrastive coefficients becomes _j=2_ qjsj q1s1 + s1 + q1 **1,**
_−_ [P][N][way] _−_ _−_

which is _j=2_ qjsj (1 q1)(1 s1) < 0. This indicates the encoder would be updated
_−_ [P][N][way] _−_ _−_ _−_
to maximize the inner product between ϕ(q) and the support features of the same class. However,
when the query and support data are in different classes, the sign of the contrastive coefficient can


-----

sometimes be negative. The outer loop loss thus cannot well contrast the query features against the
support features of different classes, making this loss term not an ordinary SCL loss.

To better illustrate the influence of the interference term and the noisy contrastive term, we provide
an ablation experiment in Appendix B.7. Theorem 1 below formally connects MAML to SCL.

**Theorem 1 With the EFIL assumption, FOMAML is a noisy SCL algorithm. With assumptions of**
_(a) EFIL and (b) a single inner-loop update, SOMAML is a noisy SCL algorithm._

Proof: For FOMAML, both Eq. (7) (one inner loop update step) and Eq. (26) (multiple inner loop
update steps) follows Definition 1. For SOMAML, Eq. (8) follows Definition 1.

**Introduction of the zeroing trick makes Eq. (7) and Eq. (8) SCL losses. To tackle the interference**
term and make the contrastive coefficients more accurate, we introduce the zeroing trick: setting the
**w[0]** to be zero after each outer loop update, as shown in Appendix A. With the zeroing trick, the
original outer loop loss (of FOMAML) becomes

_L_ _φ,w1_ _,q = η_ **E** _ϕ(q)_
_{_ _}_ (s,t) _S_ [(][q][t][ −] **[1][t=u][)][ϕ][(][s][)][⊤]** (9)
_∼_ stop gradient

For SOMAML, the original outer loop loss becomes

_L_ _φ,w1_ _,q = η_ **E** _ϕ(s)[⊤]ϕ(q)_
_{_ _}_ (s,t) _S_ [(][q][t][ −] **[1][t=u][)]** (10)
_∼_ stop gradient


The zeroing trick brings two nontrivial effects: (a) eliminating the interference term in both Eq. (7)
and Eq. (8); (b) making the contrastive coefficients follow SCL. For (b), since all the predictive
values of support data become the same, i.e., sk = _Nway1_ [, the contrastive coefficient becomes q][t]

_[−]_
**1t=u, which is negative when the support and query data have the same label, and positive otherwise.**
With the zeroing trick, the contrastive coefficient follows the SCL loss, as summarized below.

**Corollary 1 With mild assumptions of (a) EFIL, (b) a single inner-loop update and (c) training with**
_the zeroing trick (i.e., the linear layer is zeroed at the end of each outer loop), both FOMAML and_
_SOMAML are SCL algorithms._

Proof: Both Eq. (9) and Eq. (10) follow Definition 1.

The introduction of the zeroing trick makes the relationship between MAML and SCL more straightforward. Generally speaking, by connecting MAML and SCL, we can better understand other
MAML-based meta-learning studies.

2.5 RESPONSES TO QUESTIONS IN SECTION 1

**In what sense does MAML guide any model to learn general-purpose representations? Under**
the EFIL assumption, MAML is a noisy SCL algorithm in a classification paradigm. The effectiveness of MAML in enabling models to learn general-purpose representations can be attributed to the
SCL characteristics of MAML.

**How do the inner and outer loops in the training mechanism of MAML collaboratively prompt**
**to achieve so? MAML adopts the inner and outer loops to perform noisy SCL sequentially. In the**
inner loop, the features of support data are memorized by w via inner-loop update. In the outer loop,
the softmax output of the query data thus contains the inner products between the support features
and the query feature.

**What is the role of support and query data, and how do they interact with each other? We show**
that the original loss in MAML can be reformulated as a loss term containing the inner products of
the embedding of the support and query data. In FOMAML, the support features act as the reference,
while the query features are updated to move towards the support features of the same class and
against those of the different classes.

2.6 GENERALIZATION OF OUR ANALYSIS

In Appendix C, we provide the analysis where Nbatch 1 and Nstep 1. For the EFIL assumption,
it can hardly be dropped because the behavior of the updated encoder is intractable. Besides, Raghu ≥ _≥_


-----

et al. (2020) show that the representations of intermediate layers do not change notably during the
inner loop of MAML, and thus it is understood that the main function of the inner loop is to change
the final linear layer. Furthermore, the EFIL assumption is empirically reasonable, since previous
works (Raghu et al., 2020; Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020) yield comparable
performance while leaving the encoder untouched during the inner loop.

With our analysis, one may notice that MAML is approximately a metric-based few-shot learning algorithm. From a high-level perspective, under the EFIL assumption, second-order MAML is similar
to metric-based few-shot learning algorithms, such as MatchingNet (Vinyals et al., 2016), Prototypical network (Snell et al., 2017), and Relation network (Sung et al., 2018). The main difference
lies in the metric and the way prototypes are constructed. Our work follows the setting adopted by
MAML, such as using negative LogSoftmax as objective function, but we can effortlessly generalize
our analysis to a MSE loss as had been shown in Figure 1. As a result, our work points out a new
research direction in improving MAML by changing the objective functions in the inner and the
outer loops, e.g., using MSE for the inner loop but negative LogSoftmax for the outer loop. Besides,
in MAML, we often obtain the logits by multiplying the features by the linear weight w. Our work
implies future direction as to alternatively substitute this inner product operation with other metrics
or other similarity measurements such as cosine similarity or negative Euclidean distance.

3 EXPERIMENTAL RESULTS

In this section, we provide empirical evidence of the supervised contrastiveness of MAML and show
that zero-initialization of w[0], reduction in the initial norm of w[0], or the application of zeroing trick
can speed up the learning profile. This is applicable to both SOMAML and FOMAML.

3.1 SETUP

We conduct our experiments on the mini-ImageNet dataset (Vinyals et al., 2016; Ravi & Larochelle,
2017) and the Omniglot dataset (Lake et al., 2015). For the results on the Omniglot dataset, please
refer to Appendix E. For the mini-ImageNet, it contains 84 × 84 RGB images of 100 classes from
the ImageNet dataset with 600 samples per class. We split the dataset into 64, 16 and 20 classes
for training, validation, and testing as proposed in (Ravi & Larochelle, 2017). We do not perform
hyperparameter search and thus are not using the validation data. For all our experiments of applying
MAML into few-shot classification problem, where we adopt two experimental settings: 5-way 1shot and 5-way 5-shot, with the batch size Nbatch being 4 and 2, respectively (Finn et al., 2017). The
few-shot classification accuracy is calculated by averaging the results over 400 tasks in the test phase.
For model architecture, optimizer and other experimental details, please refer to Appendix D.1.

3.2 COSINE SIMILARITY ANALYSIS VERIFIES THE IMPLICIT CONTRASTIVENESS IN MAML

In Section 2, we show that the encoder is updated so that the query features are pushed towards the
support features of the same class and pulled away from those of different classes. Here we verify
this supervised contrastiveness experimentally. Consider a relatively overfitting scenario where there
are five classes of images and for each class there are 20 support images and 20 query images. We
fix the support and query set (i.e. the data is not resampled every iteration) to verify the concept that
the support features work as positive and negative samples. Channel shuffling is used to avoid the
undesirable channel memorization effect (Jamal & Qi, 2019; Rajendran et al., 2020). We train the
model using FOMAML and examine how well the encoder can separate the data of different classes
in the feature space by measuring the averaged cosine similarities between the features of each class.
The results are averaged over 10 random seeds.

As shown in the top row of Figure 2, the model trained with MAML learns to separate the features
of different classes. Moreover, the contrast between the diagonal and the off-diagonal entries of the
heatmap increases as we remove the initialization interference (by zero-initializing w[0], shown in
the middle row) and remove the cross-task interference (by applying the zeroing trick, shown in the
bottom row). The result agrees with our analysis that MAML implicitly contains the interference
term which can impede the encoder from learning a good feature representation. For experiments
on semantically similar classes of images, the result is shown in Section D.3.


-----

Figure 2: The supervised contrastiveness entailed in MAML manifests when zero initialization
**or the zeroing trick is applied. The value in the heatmap is calculated by averaging the pair-**
wise cosine similarities between query features or between query features and support features. We
consider the setting of having randomly initialized linear layer (top), zero-initialized linear layer
(middle), and the zeroing trick (bottom), and experiment with various numbers of outer loop updates. The readers are encouraged to compare the results between different rows.

3.3 ZEROING LINEAR LAYER AT TESTING TIME INCREASES TESTING ACCURACY


Before starting our analysis on benchmark datasets, we note that the cross-task interference can also
occur during meta-testing. In the meta-testing stage, the base-model is updated in the inner loop
using support data S and then the performance is evaluated using query data Q, where S and Q
are drawn from a held-out, unseen meta-testing dataset. Recall that at the end of the outer loop (in
meta-training stage), the query features are added weightedly to the linear layer w[0]. In other words,
at the beginning of meta-testing, w[0] is already added with the query features of previous training
tasks, which can drastically influence the performance on the unseen tasks.

To validate this idea, we apply the zeroing trick at meta-testing time (which we refer to zeroing w[0]
at the beginning of the meta-testing time) and show such trick increases the testing accuracy of the
model trained with FOMAML. As illustrated in Figure 3, compared to directly entering meta-testing
(i.e. the subplot at the left), additionally zeroing the linear layer at the beginning of each meta-testing
time (i.e. the subplot at the right) increases the testing accuracy of the model whose linear layer is
randomly initialized or zero-initialized (denoted by the red and orange curves, respectively). And
the difference in testing performance sustains across the whole training session.

In the following experiments, we evaluate the testing performance only with zeroing the linear layer
at the beginning of the meta-testing stage. By zeroing the linear layer, the potential interference
brought by the prior (of the linear layer) is ignored. Then, we can fully focus on the capacity of the
encoder in learning a good feature representation.


Original FOMAML


w/ zeroing trick applied in testing stage


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000

Iteration (outer loop) Iteration (outer loop)

Figure 3: Using the zeroing trick at meta-testing stage improves performance. The left/right

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||||
||||||
||||||
||||||
||||||
||||||
|0 2000|4000 6000 8000 100001200014000 Iteration (outer loop)||||


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
||||||||
||||||||
||||||||
||set||||||
|||ting f|or line|ar lay|er w0||
||ra ze|ndom ro-init|ly initia ialized|lized|||
||ze|roed|every o|uter lo|op||
||(z|eroing|trick a|t traini|ng stag|e)|
|0 20|00 4000 6000 8000 100001200014000 Iteration (outer loop)||||||

subplot shows the performance of models without/with w[0] zeroed at the beginning of meta-testing
time. The curves in red: w[0] is randomly initialized. The curves in yellow: w[0] is zeroed at initialization. The curves in green: the models trained with training trick applied in the training stage.


-----

3.4 SINGLE INNER LOOP UPDATE SUFFICES WHEN USING THE ZEROING TRICK

In Eq. (4) and Eq. (21), we show that the features of the support data are added to the linear layer in
the inner loop. Larger number of inner loop update steps can better offset the effect of interference
brought by a non-zeroed linear layer. In other words, when the models are trained with the zeroing
trick, a larger number of inner loop updates can not bring any benefit. We validate this intuition in
Figure 4 under a 5-way 1-shot setting. In the original FOMAML, the models trained with a single
inner loop update step (denoted as red curve) converge slower than those trained with update step
of 7 (denoted as purple curve). On the contrary, when the models are trained with the zeroing trick,
models with various inner loop update steps converge at the same speed.



w/ zeroing trick applied in training stage


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000

Iteration (outer loop) Iteration (outer loop)

Figure 4: With the zeroing trick, a larger number of inner loop update steps is not necessary. In

|Col1|Original FOMAML|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|0 2000 4|000 6000 8000 100001200014000 Iteration (outer loop)|||


|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
|||numbe|r of in|ner lo 1|op steps|
|||||2 3||
|||||5 7||
|||||||
|0 2|000 4000 6000 8000 10000120001400 Iteration (outer loop)|||||

original MAML, a larger number of inner loop update steps is preferred as it generally yields better
testing accuracy even with zeroing trick applied in the meta-testing stage (refer to the left figure).
However, models trained using the zeroing trick do not show this trend (refer to the right figure).


3.5 EFFECT OF INITIALIZATION AND THE ZEROING TRICK

In Eq. (7), we observe an interference derived from the historical task features or random initialization. We validate our formula by examining the effects of (1) reducing the norm of w[0] at initialization and (2) applying the zeroing trick. From Figure 5, the performance is higher when the initial
norm of w[0] is lower. Compared to random initialization, reducing the norm via down-scaling w[0] by
0.7 yields visible differences. Besides, the testing accuracy of MAML with zeroing trick (the purple
curve) outperforms that of original MAML.



5-way 5-shot


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0.625

0.600

0.575

0.550

0.525

0.500

0.475

0.450


Figure 5: Effect of initialization and the zeroing trick on testing performance. Both reducing

|Col1|5-way 1-shot|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|0 2000 4||||


|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
||s|etting fo randoml|r linear y initializ|layer w0 ed||
|||randoml randoml|y initializ y initializ|ed (×0.7) ed (×0.5)||
|||zero-initi zeroed e|alized very out|er loop||
|||(zeroing|trick at t|raining st|age)|
|0 2|500 5000 7500 10000 12500 15000 Iteration (outer loop)|||||

the norm of w[0] and zeroing w[0] each outer loop (i.e., the zeroing trick) increase the testing accuracy.
The curves in red: models with w[0] randomly initialized. The curves in orange/green: reducing the
value of w[0] at initialization by a factor of 0.7/ 0.5. The curve in blue: w[0] is zero-initialized. The
curve in blue: models trained with the zeroing trick.

4 CONCLUSION


This paper presents an extensive study to demystify how the seminal MAML algorithm guides the
encoder to learn a general-purpose feature representation and how support and query data interact. Our analysis shows that MAML is implicitly a supervised contrastive learner using the support
features as positive and negative samples to direct the update of the encoder. Moreover, we unveil an interference term hidden in MAML originated from the random initialization or cross-task
interaction, which can impede the representation learning. Driven by our analysis, removing the
interference term by a simple zeroing trick renders the model unbiased to seen or unseen tasks. Furthermore, we show constant improvements in the training and testing profiles with this zeroing trick,
with experiments conducted on the mini-ImageNet and Omniglot datasets.


-----

REFERENCES

Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In International
_Conference on Learning Representations (ICLR), 2019._

Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR),
2019.

Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. arXiv
_preprint arXiv:2105.02221, 2021._

Tristan Deleu. Model-agnostic meta-learning. [https://github.com/tristandeleu/](https://github.com/tristandeleu/pytorch-maml)
[pytorch-maml, 2020.](https://github.com/tristandeleu/pytorch-maml)

Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning
the representation, provably. In International Conference on Learning Representations (ICLR),
2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation
of deep networks. In International Conference on Learning Representations (ICLR), 2017.

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In
_International Conference on Machine Learning (ICML), 2019._

Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein.
Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Interna_tional Conference on Machine Learning (ICML), 2020._

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradientbased meta-learning as hierarchical bayes. In International Conference on Learning Representa_tions (ICLR), 2018._

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural
networks: A survey. arXiv preprint arXiv:2004.05439, 2020.

Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In
_IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019._

Khurram Javed and Martha White. Meta-learning representations for continual learning. In Ad_vances in Neural Information Processing Systems (NeurIPS), 2019._

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron
Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural
_Information Processing Systems (NeurIPS), 2020._

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning
through probabilistic program induction. Science, 2015.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with
differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recog_nition (CVPR), 2019._

Chen Liu, Chengming Xu, Yikai Wang, Li Zhang, and Yanwei Fu. An embarrassingly simple
baseline to one-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), 2020._

Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased metareinforcement learning. In International Conference on Machine Learning (ICML), 2019.

Liangqu Long. Maml-pytorch implementation. [https://github.com/dragen1860/](https://github.com/dragen1860/MAML-Pytorch)
[MAML-Pytorch, 2018.](https://github.com/dragen1860/MAML-Pytorch)

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv
_preprint arXiv:1803.02999, 2018._


-----

Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse?
towards understanding the effectiveness of maml. In International Conference on Learning Rep_resentations (ICLR), 2020._

Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta-augmentation. In
_Advances in Neural Information Processing Systems (NeurIPS), 2020._

Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit
gradients. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International
_Conference on Learning Representations (ICLR), 2017._

Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal
meta-policy search. In International Conference on Learning Representations (ICLR), 2019.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), 2015._

Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning.
_Advances in Neural Information Processing Systems (NeurIPS), 2017._

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in
_Neural Information Processing Systems (NeurIPS), 2016._

Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, and Yunhao
Tang. Es-maml: Simple hessian-free meta learning. In International Conference on Learning
_Representations (ICLR), 2020._

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.
Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer
_Vision and Pattern Recognition (CVPR), 2018._

Nilesh Tripuraneni, Chi Jin, and Michael I Jordan. Provable meta-learning of linear representations.
_arXiv preprint arXiv:2002.11684, 2020._

Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial
_Intelligence Review, 2002._

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems
_(NeurIPS), 2016._

Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning
without memorization. In International Conference on Learning Representations (ICLR), 2020.


-----

APPENDIX

A ORIGINAL MAML AND MAML WITH THE ZEROING TRICK

**Algorithm 1 Second-order MAML**

**Require: Task distribution D**
**Require: η, ρ: inner loop and outer loop learning rates**
**Require: Randomly initialized base-model parameters θ**

1: while not done do
2: Sample tasks _T1, . . . TNbatch_ from D
_{_ _}_

3: **for n = 1, 2, . . ., Nbatch do**

4: _Sn, Qn_ sample from Tn
_{_ _} ←_

5: _θn = θ_

6: **for i = 1, 2, . . ., Nstep do**

7: _θn_ _θn_ _η_ _θn_ _Lθn,Sn_

8: **end for ←** _−_ _∇_

9: **end for**

10: Update θ _θ_ _ρ_ _n=1_ _θLθn,Qn_
_←_ _−_ _∇_

11: end while

[P][N][batch]

**Algorithm 2 First-order MAML**

**Require: Task distribution D**
**Require: η, ρ: inner loop and outer loop learning rates**
**Require: Randomly initialized base-model parameters θ**

1: while not done do
2: Sample tasks _T1, . . . TNbatch_ from D
_{_ _}_

3: **for n = 1, 2, . . ., Nbatch do**

4: _Sn, Qn_ sample from Tn
_{_ _} ←_

5: _θn = θ_

6: **for i = 1, 2, . . ., Nstep do**

7: _θn_ _θn_ _η_ _θn_ _Lθn,Sn_

8: **end for ←** _−_ _∇_

9: **end for**

10: Update θ _θ_ _ρ_ _n=1_ _θn_ _Lθn,Qn_
_←_ _−_ _∇_

11: end while

[P][N][batch]

**Algorithm 3 Second-order MAML with the zeroing trick**

**Require: Task distribution D**
**Require: η, ρ: inner loop and outer loop learning rates**
**Require: Randomly initialized base-model parameters θ**

1: Set w ← 0 (the zeroing trick)
2: while not done do
3: Sample tasks _T1, . . . TNbatch_ from D
_{_ _}_

4: **for n = 1, 2, . . ., Nbatch do**

5: _Sn, Qn_ sample from Tn
_{_ _} ←_

6: _θn = θ_

7: **for i = 1, 2, . . ., Nstep do**

8: _θn_ _θn_ _η_ _θn_ _Lθn,Sn_

9: **end for ←** _−_ _∇_

10: **end for**

11: Update θ _θ_ _ρ_ _n=1_ _θLθn,Qn_
_←_ _−_ _∇_

12: Set w ← 0 (the zeroing trick)

13: end while

[P][N][batch]


-----

B SUPPLEMENTARY DERIVATION

In this section, we provide the full generalization and further discussion that supplement the main
paper. We consider the case of Nbatch = 1 and Nstep = 1 under the EFIL assumption. We
provide the outer loop update of the linear layer under SOMAML in Section B.1. Next, we offer
the full derivation of the outer loop update of the encoder in Section B.2. Then, we reformulate the
outer loop loss for the encoder in both FOMAML and SOMAML in Section B.3 and Section B.4.
Afterward, we discuss the main difference in FOMAML and SOMAML in detail in Section B.5.
Finally, we show the performance of the models trained using the reformulated loss in Section B.6.

B.1 THE DERIVATION OF OUTER LOOP UPDATE FOR THE LINEAR LAYER USING SOMAML

Here, we provide the complete derivation of the outer loop update for the linear layer. Using SOMAML with support set S and query set Q, the update of the linear layer follows


_Nway_

_m=1_

X

_Nway_


**wk[′][0]** [=][ w]k0 _ρ_ _[∂L][{][φ,][w][1][}][,Q]_ = wk0 _ρ_
_−_ _∂wk[0]_ _−_


_∂wm1_

_∂wk[0][ ·][ ∂L]∂[{][φ,]w[w]m[1][1][}][,Q]_

_∂wm1_

_∂wk[0][ ·][ ∂L]∂[{][φ,]w[w]m[1][1][}][,Q]_


1
= wk0 − _ρ_ _[∂]∂[w]wkk[0][ ·][ ∂L]∂[{][φ,]w[w]k[1][1][}][,Q]_ _−_ _ρ_


**k** **k** **k** **m**

_m≠_ _k_

= wk0 + ρ[I _η_ **E** _k[)][ϕ][(][s][)][ϕ][(][s][)][T][ ]]_ **E**
_−_ (s,t) _S[(][s][k][ −]_ [s][2] (q,u) _Q[(][1][k=u][ −]_ [q][k][)][ϕ][(][q][)]
_∼_ _∼_

+ ρη [ **E** **E**

(s,t) _S[(][s][m][s][k][)][ϕ][(][s][)][ϕ][(][s][)][T][ ][](q,u)_ _Q[(][1][m=u][ −]_ [q][m][)][ϕ][(][q][)]]
_m=k_ _∼_ _∼_

X̸

= wk0 + ρ[I _η_ **E** **E**
_−_ (s,t) _S_ [s][k][ϕ][(][s][)][ϕ][(][s][)][T][ ]] (q,u) _Q[(][1][k=u][ −]_ [q][k][)][ϕ][(][q][)]
_∼_ _∼_


(11)


_Nway_

[ **E** **E**
(s,t) _S[(][s][m][s][k][)][ϕ][(][s][)][ϕ][(][s][)][T][ ][](q,u)_ _Q[(][1][m=u][ −]_ [q][m][)][ϕ][(][q][)]]
_m=1_ _∼_ _∼_

X


+ ρη


We can further simplify Eq. (11) to Eq. (12) with the help of the zeroing trick.

**wk[′][0]** [=][ ρ][[][I][ −] _[η]_ **E** **E** (12)
(s,t) _S_ [s][k][ϕ][(][s][)][ϕ][(][s][)][T][ ]] (q,u) _Q[(][1][k=u][ −]_ [q][k][)][ϕ][(][q][)]
_∼_ _∼_

This is because the zeroing trick essentially turns the logits of all support samples to zero, and consequently the predicted probability (softmax) output sm becomes _Nway1_ [for all channel][ m][. Therefore,]

the third term in Eq. (11) turns out to be zero (c.f. Eq. (13)). The equality of Eq. (13) holds since
the summation of the (softmax) outputs is one.


_Nway_

[ **E** **E**
_m=1_ (s,t)∼S _[ϕ][(][s][)][ϕ][(][s][)][T][ ][](q,u)∼Q[(][1][m=u][ −]_ [q][m][)][ϕ][(][q][)]]

X


_ρη_

_Nway[2]_


(13)


_ρη_ _Nway_

_Nway[2]_ [(s,tE)∼S _[ϕ][(][s][)][ϕ][(][s][)][T][ ]]_ (q,uE)∼Q _[ϕ][(][q][)]_ _m=1_ (1m=u − qm) = 0

X


B.2 THE FULL DERIVATION OF THE OUTER LOOP UPDATE OF THE ENCODER.

As the encoder ϕ is parameterized by φ, the outer loop gradient with respect to φ is given by
_∂L{φ,∂φw1_ _},Q_ = E(q,u)∼Q _∂L{∂ϕφ,(wq1)},Q_ _∂ϕ∂φ(q)_ + E(s,t)∼S _∂L{∂ϕφ,w(s1)},Q_ _∂ϕ∂φ(s)_ [. We take a deeper look at the]

backpropagated error _∂L{∂ϕφ,(wq1)},Q_ of the feature of one query data (q, u) _Q, based on the following_

_∼_


-----

form:


_Nway_

(qjwj1) =
_j=1_

X


_Nway_

(1j=u qj)wj
_j=1_ _−_

X


_−_ _[∂L]∂ϕ[{][φ,]([w]q[1])[}][,Q]_ = wu1 −


_Nway_

(1j=u qj)wj0 + η
_j=1_ _−_

X


_Nway_

[1j=u qj][ **E**
_j=1_ _−_ (s,t)∼S[(][1][j=t][ −] [s][j][)][ϕ][(][s][)]]

X


(14)


_Nway_ _Nway_

= (1j=u qj)wj0 + η **E** qjsj) su qt + 1t=u]ϕ(s)

_j=1_ _−_ (s,t)∼S[[(] _j=1_ _−_ _−_

X X

B.3 REFORMULATION OF THE OUTER LOOP LOSS FOR THE ENCODER AS NOISY SCL LOSS.

We can derive the actual loss (evaluated on a single query data (q, u) ∼ _Q) that the encoder uses_
under FOMAML scheme as follows:


_Nway_ _Nway_

(qj **1j=u)wj0⊤** _ϕ(q)_ _η_ **E** qjsj) su qt + 1t=u]ϕ(s)[⊤]
_j=1_ _−stop gradient_ _−_ (s,t)∼S [[(] _j=1_ _−_ _−_

X X

stop gradient


_L_ _φ,w1_ _,q =_
_{_ _}_


_ϕ(q)_
(15)


For SOMAML, we need to additionally plug Eq. (4) into Eq. (3).


_Nway_ _Nway_

(qj **1j=u)wj0⊤** _ϕ(q)_ _η_ **E** qjsj) su qt + 1t=u]
_j=1_ _−stop gradient_ _−_ (s,t)∼S [[(] _j=1_ _−_ _−_

X X

stop gradient


_ϕ(s)[⊤]ϕ(q)_
(16)


_L_ _φ,w1_ _,q =_
_{_ _}_


B.4 INTRODUCTION OF THE ZEROING TRICK MAKES EQ. (7) AND EQ. (8) SCL LOSSES.

Apply the zeroing trick to Eq. (7) and Eq. (8), we can derive the actual loss Eq. (17) and Eq. (18)
that the encoder follows.

_L_ _φ,w1_ _,q = η_ **E** _ϕ(q)_
_{_ _}_ (s,t) _S_ [(][q][t][ −] **[1][t=u][)][ϕ][(][s][)][⊤]** (17)
_∼_ stop gradient


_L_ _φ,w1_ _,q = η_ **E** _ϕ(s)[⊤]ϕ(q)_
_{_ _}_ (s,t) _S_ [(][q][t][ −] **[1][t=u][)]** (18)
_∼_ stop gradient

With these two equations, we can observe the essential difference in FOMAML and SOMAML
is the range of stopping gradients. We would further discuss the implication of different ranges of
gradient stopping in Appendix B.5.

B.5 DISCUSSION ABOUT THE DIFFERENCE BETWEEN FOMAML AND SOMAML

Central to the mystery of MAML is the difference between FOMAML and SOMAML. Plenty of
work is dedicated to approximating or estimating the second-order derivatives in the MAML algorithm in a more computational-efficient or accurate manner (Song et al., 2020; Rothfuss et al., 2019;
Liu et al., 2019). With the EFIL assumption and our analysis through connecting SCL to these algorithms, we found that we can better understand the distinction between FOMAML and SOMAML
from a novel perspective. To better understand the difference, we can compare Eq. (7) with Eq. (8)
or compare Eq. (9) with Eq. (10). To avoid being distracted by the interference terms, we provide
the analysis of the latter.

The main difference between Eq. (9) and Eq. (10) is the range of gradient stopping and we will show
that this difference results in a significant distinction in the feature space. To begin with, by chain
rule, we have _[∂L]∂φ_ [=][ E][(][q,u][)][∼][Q] _∂ϕ∂L(q)_ _∂ϕ∂φ(q)_ + E(s,t)∼S _∂ϕ∂L(s)_ _∂ϕ∂φ(s)_ [. As we specifically want to know]

how the encoded features are updated given different losses, we can look at the terms _∂ϕ∂L(q)_ [and]

_∂L_

_∂ϕ(s)_ [by differentiating Eq. (9) and Eq. (10) with respect to the features of query data][ q][ and support]
data s, respectively.


-----

FOMAML:

SOMAML:


_∂L_

**E**
_∂ϕ(q) [=][ η]_ (s,t) _S[(][q][t][ −]_ **[1][t=u][)][ϕ][(][s][)]**
_∼_ (19)

_∂L_

_∂ϕ(s) [= 0]_


_∂L_

**E**
_∂ϕ(q) [=][ η]_ (s,t) _S[(][q][t][ −]_ **[1][t=u][)][ϕ][(][s][)]**
_∼_

(20)

_∂L_

**E**
_∂ϕ(s) [=][ η]_ (s,t) _S[(][q][t][ −]_ **[1][t=u][)][ϕ][(][q][)]**
_∼_

Obviously, as the second equation in Eq (19) is zero, we know that in FOMAML, the update of
the encoder does consider the change of the support features. The encoder is updated to move the
query features closer to support features of the same class and further to support features of different
classes in FOMAML. On the contrary, we can tell from the above equations that in SOMAML, the
encoder is updated to make support features and query features closer if both come from the same
class and make support features and query features further if they come from different classes.

We illustrate the difference in Figure 6. For simplicity, we do not consider the scale of the coefficients but their signs. The subplot on the left indicates that this FOMAML loss guides the encoder
to be updated so that the feature of the query data moves 1) towards the support feature of the same
class, and 2) against the support features of the different classes. On the other hand, the SOMAML
loss guides the encoder to be updated so that 1) when the support data and query data belong to the
same class, their features move closer, and otherwise, their features move further. This generally
explains why models trained using SOMAML generally converge faster than those trained using
FOMAML.

Figure 6: Illustration of the distinction of FOMAML and SOMAML. Conceptually speaking,
the objective function of FOMAML aims to change the features of the query data; in contrast,
that of SOMAML seeks to change the query’s features and support data simultaneously. In this
figure, the support data and query data features are plotted as solid and hollow circles, respectively.
The different colors represent different classes. The solid and hollow arrows indicate the gradient
calculated from positive and negative samples, respectively. Note that this is in the feature space,
not the weight space.

B.6 EXPLICITLY COMPUTING THE REFORMULATING LOSS USING EQ. (7) AND EQ. (8)

Under the EFIL assumption, we show that MAML can be reformulated as a loss taking noisy SCL
form. Below, we consider a setting of 5-way 1-shot mini-ImageNet few-shot classification task,
under the condition of no inner loop update of the encoder. (This is the assumption that our derivation
heavily depends on. It means that we now only update the encoder in the outer loop.) We empirically
show that explicitly computing the reformulated losses of Eq. (7), Eq. (17) and Eq. (18) yield almost
the same curves as MAML (with the EFIL assumption). Please note that the reformulated losses are
used to update the encoders, for the linear layer w[0], we explicitly update it using Eq. (5). Note that


-----

although the performance models training using FOMAML, FOMAML with the zeroing trick, and
SOMAML converge to similar testing accuracy, the overall testing performance during the training
process is distinct. The results are averaged over three random seeds.

0.48

0.46

0.44


Figure 7: Updating the encoder using the reformulated outer loop loss. We experimentally

|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
||FOMAML, rando FOMAML, rando|m-initia m-initia|lized lized (co|ntrastiv|e)|
||FOMAML, zero-i FOMAML, zero-i|nitialize nitialize|d d (contr|astive)||
||SOMAML, zero-i|nitialize|d|||
||SOMAML, zero-i|nitialize|d (contr|astive)||
|0 2000||||||

validate that the testing accuracy of models trained using MAML (with no inner loop update of
encoder) consists with that of models using their corresponding supervised contrastive losses, i.e.,
Eq. (7), Eq. (17) and Eq. (18).


B.7 THE EFFECT OF INTERFERENCE TERM AND NOISY CONTRASTIVE TERM

Reformulating the loss of MAML into a noisy SCL form enables us to further investigate the effects
brought by the interference term and the noisy contrastive term, which we presume both effects to
be negative.

To investigate the effect of the interference term, we simply consider the loss adopted by firstorder MAML as in Eq. (7) but with the interference term dropped (denoted as “n1 ”). As for
the noisy contrastive term, the noise comes from the fact that “when the query and support data ×
are in different classes, the sign of the contrastive coefficient can sometimes be negative”, as being
discussed in Section 2.4. To mitigate this noise, we consider the loss in Eq. (7) with the term
([P][N]j=1[way] qjsj) + su dropped from the contrastive coefficient, and denote it as “n2 ”. On the
_−other hand, we also implement a loss with “n1_, n2 ”, which is actually Eq. (9). We adopt the ×
same experimental setting as Section B.6. _×_ _×_

In Figure 8, we show the testing profiles of the original reformulated loss (i.e., the curve in red,
labeled as “n1 ✓, n2 ✓”), dropping the interference term (i.e., the curve in orange, labeled as “n1
_×, n2 ✓”), dropping the noisy part of the contrastive term (i.e., the curve in green, labeled as “n1_
✓dropping the interference term or dropping dropping the noisy part of contrastive coefficients yield, n2 ×”) or dropping both (i.e., the curve in blue, labeled as “n1 ×, n2 ×”). We can see that either
profound benefit.


0.48

0.46

0.44

0.42

0.40

0.38

0.36


|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
||n|||||
|||, n|(origin|al MAM|L)|
||1 n1|2, n2||||
||n1 n|, n2, n|(zeroin|g trick)||
||1|2||||
|0 2000||||||


Figure 8: The effect of the interference term and the noisy contrastive term. We perform an
ablation study of the reformulated loss in Eq. (7) by dropping the interference term (denoted as
“n1”) or dropping the noisy part in the noisy contrastive term (marked as “n2”).

To better understand how noisy is the noisy contrastive term, i.e., how many times the sign of
the contrastive coefficient is negative when the query and support data are in different classes, we
explicitly record the ratio of the contrastive term being positive or negative. We adopt the same
experimental setting as Section B.6.


-----

The result is shown in Figure 9. When the zeroing trick is applied, the ratio of contrastive term being
negative (shown as the red curve on the right subplot) is 0.2, which is _Nway1_ [where][ N][way][ = 5][ in]

our setting. On the other hand, when the zeroing trick is not applied, the ratio of contrastive term
being negative (shown as the orange color on the right subplot) is larger than 0.2. This additional
experiment necessitates the application of the zeroing trick.

|Col1|Col2|Col3|Col4|Col5|Col6|with with|zeroing out zer|trick oing tri|ck|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||


1.0

with zeroing trick

0.8 without zeroing trick

0.6

0.4

being positive being negative

Ratio of contrastive term 0.2 Ratio of contrastive term

0.0

0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000

Iteration (outer loop) Iteration (outer loop)


Figure 9: The ratio of the contrastive term being positive or negative during training. With the
zeroing trick, the MAML becomes a SCL algorithm, so its ratio of negative contrastive term is 0.2.
The original MAML, however, is a noisy SCL algorithm, and thus its ratio of negative contrastive
term is larger than 0.2


-----

C A GENERALIZATION OF OUR ANALYSIS

In this section, we derive a more general case of the encoder update in the outer loop. We consider
drawing Nbatch tasks from the task distribution D and having Nstep update steps in the inner loop
while keeping the EFIL assumption.

To derive a more general case, we use wk[i,n] to denote the k[th] column of w[i,n], where w[i,n] is
updated from w[0] using support data Sn for i inner-loop steps. For simplicity, the k[th] channel
softmax predictive output _Nwayj=1exp(ϕexp((s)[⊤]ϕ(ws)k[⊤]i,nw)j_ _[i,n])_ [of sample][ s][ (using][ w][i][−][1][,n][) is denoted as s]k[i,n][.]
P

**Inner loop update for the linear layer We yield the inner loop update for the final linear layer in**
Eq. (21) and Eq. (22).

**wki,n = wki−1,n −** _η [∂L][{]∂[φ,]w[w]k[i][i][−][−][1][1][,n][,n][}][,S][n]_ = wki−1,n + η (s,tE) _Sn(1k=t −_ s[i]k[−][1][,n])ϕ(s) (21)

_∼_

_Nstep_

_Nstep,n_ 0
**wk** = wk _η_ **E** (1k=t s[i]k[−][1][,n])ϕ(s) (22)
_−_ _i=1_ (s,t)∼Sn _−_

X


**Outer loop update for the linear layer We derive the outer loop update for the linear layer in**
SOMAML, with denoting I = 1, 2, ..., Nway :
_{_ _}_


_Nbatch_

_n=1_

X

_Nbatch_

_n=1_

X


_∂L{φ,wk_ _Nstep,n},Qn_

_∂wk[0]_

_Nstep−1_

[(
_p0=k,p1∈I,...,pX_ _Nway ∈I_ _iY=0_


**wk[′]** 0 = wk0 _ρ_
_−_

= wk0 _ρ_
_−_


(23)


_∂w∂pwi+p1i_ _ii,n+1,n_ ) _∂L∂w{φ,pNstepwNstep,nNstep},Q,nn_ []]


When it comes to FOMAML, we have


_Nbatch_

_n=1_

X


_Nbatch_

**E** (1k=u qNk _step,n)ϕ(q)_ (24)
_n=1_ (q,u)∼Qn _−_

X


_∂L{φ,wk_ _Nstep,n},Qn_ = wk[0] [+][ ρ]

_∂wk[N][step][,n]_


**wk[′]** 0 = wk0 _ρ_
_−_


**Outer loop update for the encoder We derive the outer loop update of the encoder under FOMAML**
as below. We consider the back-propagated error of the feature of one query data (q, u) _Qn. Note_
_∼_
that the third equality below holds by leveraging Eq. (21).


_∂L{φ,w∂ϕNstep,n(q)_ _},Qn_ = wuNstep,n − _Nway(qNi_ _step,nwiNstep,n)_

_i=1_

X


_Nway_

(1i=u qNi _step,n)wiNstep,n_
_i=1_ _−_

X


_Nway_

(1i=u qNi _step,n)[wi[0]_ [+][ η]
_i=1_ _−_

X

_Nway_

(1i=u qNi _step,n)wi[0]_
_i=1_ _−_

X


_Nstep_

**E** (1i=t s[p]i _[−][1][,n])ϕ(s)]_
_p=1_ (s,t)∼Sn _−_

X


_Nway_

+ η (1i=u qNi _step,n_

_i=1_ _−_

X

_Nway_

(1i=u qNi _step,n)wi[0]_
_i=1_ _−_

X


_Nstep_

**E** (1i=u s[p]i _[−][1][,n])ϕ(s)_
_p=1_ (s,t)∼Sn _−_

X


_Nstep_ _Nway_

+ η **E** [( qNj _step,ns[p]j_ _[−][1][,n])_ s[p]u[−][1][,n] qNt _step,n_ + 1t=u]ϕ(s)
(s,t)∼Sn _p=1_ _j=1_ _−_ _−_

X X

(25)


-----

**Reformulating the Outer Loop Loss for the Encoder as Noisy SCL Loss. From Eq. (25), we**
can derive the generalized loss (of one query sample (q, u) _Qn) that the encoder uses under_
_∼_
FOMAML scheme.

_Nway_

_L_ _φ,wNstep,n_ _,q =_ (1i=u qNi _step,n)wi[0]⊤_ _ϕ(q)_
_{_ _}_ _i=1_ _−stop gradient_

X


(26)
_ϕ(q)_


_Nstep_ _Nway_

[( qNj _step,ns[p]j_ _[−][1][,n])_ s[p]u[−][1][,n] qNt _step,n_ + 1t=u]ϕ(s)[⊤]

_−_ _−_

_p=1_ _j=1_

X X

stop gradient


+ η **E**
(s,t)∼Sn


D EXPERIMENTS ON MINI-IMAGENET DATASET

D.1 EXPERIMENTAL DETAILS IN MINI-IMAGENET DATASET

The model architecture contains four basic blocks and one fully connected linear layer, where each
block comprises a convolution layer with a kernel size of 3 × 3 and filter size of 64, batch normalization, ReLU nonlineartity and 2 × 2 max-poling. The models are trained with the softmax cross
entropy loss function using the Adam optimizer with an outer loop learning rate of 0.001 (Antoniou
et al., 2019). The inner loop step size η is set to 0.01. The models are trained for 30000 iterations (Raghu et al., 2020). The results are averaged over four random seeds, and we use the shaded
region to indicate the standard deviation. Each experiment is run on either a single NVIDIA 1080-Ti
or V100 GPU. The detailed implementation is based on Long (2018) (MIT License).

D.2 THE EXPERIMENTAL RESULT OF SOMAML

The results with SOMAML are shown in Figure 10. Note that as it is possible that longer training
can eventually overcome the noise factor and reach similar performance as the zeroing trick, the
benefit of the zeroing trick is best seen at the observed faster convergence results when compared to
vanilla MAML.


5-way 1-shot setting 5-way 5-shot setting

SOMAML FOMAML


FOMAML


SOMAML


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|||||||||
|||||||||
|||||||||
|||||||||
|||||||||
|0||||||||


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||s|etting rando|for lin mly initi|ear lay alized|er w0||
|||||zero-i zeroe (zeroi|nitialize d every ng trick|d outer lo at traini|op ng stag|e)|
||2000 4000 6000 8000 100001200014000 Iteration (outer loop)||||||||


0.48 0.625 0.625

0.46 0.600 0.600

0.44 0.575 0.575

0.42 0.550 0.550

0.40 0.525 0.525 setting for linear layer

0.38 0.500 0.500 randomly initializedzero-initialized

0.36 0.475 0.475 zeroed every outer loop (zeroing trick at training stage)

0.450 0.450

0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000 0 2000 4000 6000 8000 100001200014000

Iteration (outer loop) Iteration (outer loop) Iteration (outer loop) Iteration (outer loop)

Figure 10: Both FOMAML and SOMAML benefit from the zeroing trick. We examine if re
ducing or removing the interference can increase the testing performance in models trained with
FOMAML and SOMAML. The results suggest that SOMAML also suffers from the interference
term. Note that the second subplots from the right shows lower testing performance of models
trained with the zeroing trick as compared to the zero-initialized model. This may result from the
overfitting problem. The curves in red: models trained with original MAML. The curve in orange:
**w[0]** is zero-initialized. The curve in green: models trained with the zeroing trick.

D.3 COSINE SIMILARITY ANALYSIS ON SEMANTICALLY SIMILAR CLASSES VERIFIES THE
IMPLICIT CONTRASTIVENESS IN MAML

In Figure 2, we randomly sample five classes of images under each random seed. Given the rich
diversity of the classes in mini-ImageNet, we can consider that the five selected classes as semantically dissimilar or independent for each random seed. Here, we also provide the experimental
outcomes using a dataset composed of five semantically similar classes selected from the miniImageNet dataset: French bulldog, Saluki, Walker hound, African hunting dog, and Golden retriever.
Likewise to the original setting, we train the model using FOMAML and average the results over
ten random seeds. As shown in Figure 11, the result is consistent with Figure 2. In conclusion, we


-----

show that the supervised contrastiveness is manifested with the application of the zeroing trick even
if a semantically similar dataset is considered.

Figure 11: The supervised contrastiveness is verified even using dataset composing of semanti**cally similar classes of images. Considering a dataset composing of different species of dogs, we**
again observe the tendency that the supervised contrastiveness is manifested when we zero-initialize
the linear weight and apply the zeroing trick.


D.4 EXPERIMENTAL RESULTS ON LARGER NUMBER OF SHOTS

To empirically verify if our theoretical derivation generalizes to the setting where the number of
shots is large, we conduct experiment of a 5-way 25-shot classification task using FOMAML with
four random seeds where we adopt mini-ImageNet as the example dataset. As shown in Figure 12,
we observe that models trained with the zeroing trick again yield the best performance, consistent
with our theoretical work that MAML with the zeroing trick is SCL without noises and interference.


5-way 25-shot


0.70

0.65

0.60

0.55

0.50

0.45

|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
||||MAML|setting||
||||MAML MAML|+ zero initia + zeroing tr|lization ick|


0 2000 4000 6000 8000

setting

MAML
MAML + zero initialization
MAML + zeroing trick

Iteration (outer loop)

Figure 12: The zeroing trick works when it comes to larger number of shots. To examine if our
work generalized to the scenario when the number of shots increases, we perform a 5-way 25-shot
classification task. The result agrees with our results of 5-way 1-shot and 5-way 5-shot.


D.5 THE ZEROING TRICK MITIGATES THE CHANNEL MEMORIZATION PROBLEM

The channel memorization problem (Jamal & Qi, 2019; Rajendran et al., 2020) is a known issue occurring in a non-mutually-exclusive task setting, e.g., the task-specific class-to-label is not randomly
assigned, and thus the label can be inferred from the query data alone (Yin et al., 2020). Consider
a 5-way K-shot experiment where the total number of training classes is 5 × L. Now we construct
tasks by assigning the label t to a class sampled from class tL to (t + 1)L. It is conceivable that
the model will learn to directly map the query data to the label without using the information of
the support data and thus fails to generalize to unseen tasks. This phenomenon can be explained
from the perspective that the t[th] column of the final linear layer already accumulates the query features from tL[th] to (t + 1)L[th] classes. Zeroing the final linear layer implicitly forces the model to
use the imprinted information from the support features for inferring the label and thus mitigates


-----

this problem. We use the mini-ImageNet dataset and consider the case of L = 12. As shown in
Figure 13, the zeroing trick prevents the model from the channel memorization problem whereas
zero-initialization of the linear layer only works out at the beginning. Besides, the performance of
models trained with the zeroing trick under this non-mutually-exclusive task setting equals the ones
under the conventional few-shot setting as shown in Figure 5. As the zeroing trick clears out the final
linear layer and equalizes the value of logits, our result essentially accords with Jamal & Qi (2019)
that proposes a regularizer to maximize the entropy of prediction of the meta-initialized model.


5-way 1-shot


5-way 5-shot


0.48

0.46

0.44

0.42

0.40

0.38

0.36


0.60

0.55

0.50

0.45


Figure 13: The performance of the models trained on non-mutually exclusive tasks. The mod
|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
|0|||||||||


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|||||||||
|||||||||
||||||setting fo randomly|r linear lay initialized|er w0|
||||||zero-initi zeroed ev (zeroing t|alized ery outer l rick at train|oop ing stage)|
|||||||||
||5000 10000 15000 20000 25000 300 Iteration (outer loop)|||||||

els are trained under a non-mutually exclusive tasks setting where there is a one-to-one assignment
between class and channel. Under this circumstance, the zeroing trick tackles the channel memorization problem and yields a performance similar to conventional few-shot settings.

E EXPERIMENTS ON OMNIGLOT DATASET


Omniglot is a hand-written character dataset containing 1623 character classes, each with 20 drawn
samples from different people (Lake et al., 2015). The dataset set is splitted into training (1028
classes), validation (172 classes) and testing (423 classes) sets (Vinyals et al., 2016). Since we follow
Finn et al. (2017) for setting hyperparamters, we do not use the the validation data. The character
images are resized to 28 × 28. For all our experiments, we adopt two experimental settings: 5way 1-shot and 5-way 5-shot where the batch size Nbatch is 32 and Nquery is 15 for both cases
(Finn et al., 2017). The inner loop learning rate η is 0.4. The models are trained for 3000 iterations
using FOMAML or SOMAML. The few-shot classification accuracy is calculated by averaging the
results over 1000 tasks in the test stage. The model architecture follows the architecture used to train
on mini-ImageNet, but we substitute the convolution with max-pooling with strided convolution
operation as in Finn et al. (2017). The loss function, optimizer, and outer loop learning rate are the
same as those used in the experiments on mini-ImageNet. Each experiment is run on either a single
NVIDIA 1080-Ti. The results are averaged over four random seeds, and the standard deviation is
illustrated with the shaded region. The models are trained using FOMAML unless stated otherwise.
The detailed implementation is based on Deleu (2020) (MIT License).

We revisit the application of the zeroing trick at the testing stage on Omniglot in Figure 14 and
observe the increasing testing accuracy, in which such results are compatible with the ones on miniImageNet (cf. Figure 3 in the main manuscript). In the following experiments, we evaluate the
testing performance only after applying the zeroing trick.

In Figure 15, the distinction between the performance of models trained with the zeroing trick
and zero-initialized models is prominent, sharing remarkable similarity with the results in miniImageNet (cf. Figure 5 in the main manuscript) in both 5-way 1-shot and 5-way 5-shot settings. We
also show the testing performance of models trained using SOMAML in Figure 16 under a 5-way
5-shot setting, where there is little distinction in performance (in comparison to the results on miniImageNet, cf. Figure 10 in the main manuscript) between the models trained with the zeroing trick
and the ones trained with random initialization.

For channel memorization task, we construct non-mutually-exclusive training tasks by assigning the
label t (where 1 ≤ _t ≤_ 5 in a few-shot 5-way setting) to a class sampled from class tL to (t + 1)L
where L is 205 on Omniglot. The class-to-channel assignment is not applied to the testing tasks.
The result is shown in Figure 17. For a detailed discussion, please refer to Section D.5.


-----

Original FOMAML w/ zeroing trick applied in testing stage

5-way 5-shot 5-way 1-shot 5-way 5-shot


5-way 1-shot


1.000

0.975

0.950

0.925

0.900

0.875

0.850

0.825

0.800




|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||||
||||||
||||||
||||||
||||||
||||||
|0|20 40 60 80 Iteration (outer loop)||||

|Col1|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|||||||
|||||||
|||||||
|||||||
|||||||
|||||||
|||||||
|0|20 40 60 80 1 Iteration (outer loop)|||||

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|20 40 60 80 10 Iteration (outer loop)||||

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
||||||||
||||||||
||||||||
||||||||
|||||setting fo randomly|r linear lay initialized|er w0|
|||||zero-initi|alized||
|||||z(zeerrooeidn ge t|vreicryk oaut tterar iln|oionpg stage)|
|0|20 40 60 80 10 Iteration (outer loop)||||||


Figure 14: Zeroing the final linear layer before testing time improves the testing accuracy on
**Omniglot. The two subplots on the left: original testing setting. The two subplots at the right: the**
final linear layer is zeroed before testing time. The curves in red: the models whose linear layer is
randomly initialized. The curves in yellow: the models whose linear layer is zeroed at initialization.
The curves in green: the models whose linear layer is zeroed after each outer loop at training stage.

5-way 1-shot 5-way 5-shot

0 20 40 60 80 100 0 20 40 60 80 100

Iteration (outer loop) Iteration (outer loop)

Figure 15: Effect of initialization and the zeroing trick in testing accuracy on Omniglot. The

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|20 40 60 80 10 Iteration (outer loop)||||


|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
||setting f|or linear la|yer w0|
||random random|ly initialized ly initialized|(× 0.5)|
||zero-init zeroed|ialized every outer l|oop|
||(zeroing|trick at trai|ning stage)|
|20 40 60 80 10 Iteration (outer loop)||||

test performance of the models with reducing the initial norm of the weights of final linear layer
is similar to that with the final linear layer being zero-initialized. The distinction in performance
between models trained using the zeroing trick and zero-initialized model is more prominent in 5

Figure 16: The effect of the zeroing trick on models trained using FOMAML and SOMAML

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|||||
|||||
|20 40 60 80 100 Iteration (outer loop)||||


|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
||setting fo randoml|r linear lay y initialized|er w0|
||zero-init zeroed e|ialized very outer l|oop|
||(zeroing|trick at train|ing stage)|
|20 40 60 80 10 Iteration (outer loop)||||

**on Omniglot. The results suggest that both zero-initialization and zeroing trick mitigate the inter-**

5-way 1-shot 5-way 5-shot

0 20 40 60 80 100 0 20 40 60 80 100

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
|||||
|||||
|20 40 60 80 10 Iteration (outer loop)||||

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
||setting f|or linear lay|er w0|
||randoml zero-init|y initialized ialized||
||zeroed e (zeroing|very outer l trick at trai|oop ning stage)|
|20 40 60 80 10 Iteration (outer loop)||||


setting for linear layer w[0]

randomly initialized
zero-initialized
zeroed every outer loop (zeroing trick at training stage)

0 20 40 60 80 100 0 20 40 60 80

Iteration (outer loop) Iteration (outer loop)

**Zeroing the final linear layer before testing time improves the testing accuracy on**
The two subplots on the left: original testing setting. The two subplots at the right: the

5-way 5-shot

setting for linear layer w[0]

randomly initialized
randomly initialized (× 0.5)

zero-initialized
zeroed every outer loop
(zeroing trick at training stage)

0 20 40 60 80 100

Iteration (outer loop)

The

SOMAML

setting for linear layer w[0]

randomly initialized
zero-initialized
zeroed every outer loop
(zeroing trick at training stage)

0 20 40 60 80 100

Iteration (outer loop)

**The effect of the zeroing trick on models trained using FOMAML and SOMAML**
. The results suggest that both zero-initialization and zeroing trick mitigate the inter
5-way 5-shot

setting for linear layer w[0]

randomly initialized
zero-initialized
zeroed every outer loop
(zeroing trick at training stage)


Figure 17: The performance of the models trained on non-mutually exclusive tasks on Om**niglot. The results are compatible to those on mini-ImageNet (cf. Figure 13 in the main manuscript),**


0 20 40 60 80 100 0 20 40 60 80 100

Iteration (outer loop) Iteration (outer loop)

5-way 1-shot

1.000

0.975

0.950

0.925

0.900

0.875

Testing accuracy

0.850

0.825

0.800

0 20 40 60 80 100

Iteration (outer loop)

way 5-shot setting.

FOMAML

1.000

0.975

0.950

0.925

0.900

0.875

0.850

0.825

0.800

0 20 40 60 80 100

Iteration (outer loop)

**on Omniglot**
ference empirically.

5-way 1-shot

0.95

0.90

0.85

0.80

Testing accuracy 0.75

0.70

0.65

0 20 40 60 80 100

Iteration (outer loop)

suggesting that the zeroing trick alleviates the channel memorization problem.


-----

