# CONFESS: A FRAMEWORK FOR SINGLE SOURCE CROSS-DOMAIN FEW-SHOT LEARNING

**Debasmit Das, Sungrack Yun & Fatih Porikli**
Qualcomm AI Research.[∗]
```
 {debadas,sungrack,fporikli}@qti.qualcomm.com

```
ABSTRACT

Most current few-shot learning methods train a model from abundantly labeled
base category data and then transfer and adapt the model to sparsely labeled novel
category data. These methods mostly generalize well on novel categories from
the same domain as the base categories but perform poorly for distant domain
categories. In this paper, we propose a framework for few-shot learning coined as
**ConFeSS (Contrastive Learning and Feature Selection System) that tackles large**
domain shift between base and novel categories. The first step of our framework
trains a feature extracting backbone with the contrastive loss on the base category
data. Since the contrastive loss does not use supervision, the features can generalize
better to distant target domains. For the second step, we train a masking module to
select relevant features that are more suited to target domain classification. Finally,
a classifier is fine-tuned along with the backbone such that the backbone produces
features similar to the relevant ones. To evaluate our framework, we tested it on
a recently introduced cross-domain few-shot learning benchmark. Experimental
results demonstrate that our framework outperforms all meta-learning approaches
and produces competitive results against recent cross-domain methods. Additional
analyses are also performed to better understand our framework.

1 INTRODUCTION

Recently, there has been an expansion in the quality and quantity of datasets (Zhang et al., 2018; Sun
et al., 2017), computing resources (Jeon et al., 2019), and deep neural architectures (Dhillon & Verma,
2020). When trained with vast amounts of data, these deep neural network models deliver improved
performance on applications like image recognition, action localization, speaker verification, text
analysis, and gene sequence prediction (Nguyen et al., 2018; Yun et al., 2019; Yao et al., 2019;
Zhou et al., 2018). However, data collection and annotation at a large scale incurs substantial labor
and cost, which are particularly difficult for specialized domains such as medical imaging and satellite
imagery, where domain expertise is needed. Moreover, most neural networks fail to generalize to
unseen categories when trained with a few labeled samples. To address these limitations, research on
few-shot learning has gained significant attention.

Few-shot learning methods (Wang et al., 2020) aim to uncover the data structure and model the
concept of new categories with only a few labeled samples. A popular strategy to tackle few-shot
learning is meta-learning which consists of two stages: meta-train and meta-test. In the meta-train
stage, a backbone network is trained to classify the base category correctly by leveraging the labeled
source data while mimicking a few-shot regime where only a limited number of samples are available
per class in each learning episode. In the meta-test stage, with the trained backbone, the novel
categories with only a few target class samples can be added (or enrolled) and tested. Here, the
backbone network can be adapted to the target samples.

Nonetheless, most few-shot learning approaches exhibit insufficient generalization capacity when
there is a big gap between the source and target data. To investigate this problem, there have been
considerable efforts (Triantafillou et al., 2020; Chen et al., 2019; Tseng et al., 2020) in establishing
cross-domain few-shot learning (CDFSL) benchmarks. Still, these datasets limit their focus to natural

_∗Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc._


-----

images and fail to capture more pragmatic domain shifts where target data may come from more
diverse domains such as satellite and medical imagery. Very recently, Guo et al. (2020) introduced
a challenging benchmark to evaluate generalization capability on distant target domains. Its target
domain datasets consist of images from natural, medical and satellite domains with wide variations
of context, color, and perspective. Thus, it represents practical applications where the generic
model needs to be adapted to a particular use case. On this benchmark, the popular meta-learning
approaches (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017; Sung et al., 2018; Lee et al.,
2019) have been found to produce poor recognition performance.

This paper proposes a novel contrastive learning and feature selection system (ConFeSS) for singlesource cross-domain few-shot learning. Our framework consists of three steps: pre-training a
backbone network on a single-source dataset, learning a feature masking module on the target
dataset, and fine-tuning the backbone network. In the first step, a backbone network is trained in an
unsupervised fashion, where a self-supervised learning approach is considered with the contrastive
loss (Chen et al., 2020). This is in contrast to meta-learning approaches, which use supervision during
the pre-training stage. Although the label of the source dataset is given at this step, we consider
the unsupervised learning to alleviate the supervision collapse problem (Doersch et al., 2020) and
also to generalize better to the distant target domains. In the second step, a feature masking module
is learned with target domain data to generate masks for separating task-relevant features from
irrelevant features. This step is required because there is a large discrepancy between the source and
target datasets, and hence all the features useful for the source task might not be helpful or even be
detrimental to the target task. Furthermore, we expect the generalization performance to be improved
with fewer features in the few-shot regime due to the Vapnik-Chervonenkis (VC) dimension reduction.
In the final step, both the pre-trained backbone network and the classifier are fine-tuned to adapt to
the target categories by a proper regularization with the relevant features.

Our main contributions can be summarized as follows: (i) Learning a feature masking module with
appropriate constraints to select relevant features for few-shot target samples; (ii) Fine-tuning the
backbone by regularizing it with the selected relevant features; (iii) Our extensive experimental
evaluation and analyses show that our method produces competitive recognition performance on the
new CDFSL benchmark (Guo et al., 2020).

2 RELATED WORK

**Meta-learning for Few-shot Learning These methods use episodic pre-training to simulate test**
conditions followed by fast adaptation to novel category samples. One of the earliest meta-learning
methods was MatchNet (Vinyals et al., 2016) which learns a mapping function to project labeled and
unlabeled samples to their corresponding labels. ProtoNet (Snell et al., 2017) extended this work
by learning a representation and assigning a class depending on the distance of query samples to
class prototypes while RelationNet (Sung et al., 2018) learns an additional deep metric function.
MetaOpt (Lee et al., 2019) takes a different approach where an SVM-like classifier is learned on top
of the features for better generalization. Finally, MAML (Finn et al., 2017) is an optimization-based
method that learns to adapt to few-shot novel categories in a few iterations. All these meta-learning
methods have performed poorly on the CDFSL benchmark (Guo et al., 2020). There are many
other meta-learning works but they have not been evaluated on the CDFSL benchmark. One can
refer (Hospedales et al., 2021) for a comprehensive survey on this topic.

**Domain Adaptation In this problem, we have source and target domains with the same categories,**
and the goal is to reduce domain discrepancy between them. Hence, domain adaptation methods
cannot be directly used for CDFSL, where the labels between source and target are disjoint. The
universal domain adaptation (UDA) setting (You et al., 2019) might be more similar to the CDFSL
setting because it has different source and target categories. However, in UDA, there is some overlap
between the source and the unknown target categories with lots of unlabeled target data available
while CDFSL considers completely novel target categories each containing only few labeled data.

**Cross-domain Few-shot Learning There have been very few works on cross-domain few-shot**
learning. A recent work (Tseng et al., 2020) uses a noisy transformation layer on top of features to
simulate cross-domain distributions and produce better generalization. In (Chen et al., 2019), the
authors compare different meta-learning frameworks and propose a competitive fine-tuning-based
baseline against these methods for the cross-domain setting. However, the datasets used for evaluating


-----

these methods contain only natural images. As a result, there is no significant domain shift between
the source and target datasets even though the source and target labels are disjoint. Guo et al. (2020)
introduce a novel CDFSL benchmark and show that most meta-learning methods along with the
feature-wise transformation (Tseng et al., 2020) approach perform poorly compared to simple finetuning methods. In our paper, the fine-tuning step is augmented with a feature selection mechanism
to select relevant features. More recent methods that evaluate on the CDFSL benchmark include
CHEF (Adler et al., 2020), ATA (Wang & Deng, 2021) and STARTUP (Phoo & Hariharan, 2021).
CHEF addresses large domain shift by fusion of Hebbian learners applied on different layers. This is
done to increase the importance of low and mid-level features for distant domain recognition. ATA is
a plug-and-play method that improves robustness of models through adversarial task augmentation.
STARTUP assumes access to large unlabelled data from the target domain and proposed combining
knowledge distillation and contrastive learning to learn the target model. In our framework, we only
assume access to few labeled data from the target domain.

**Self-supervision for Few-shot Learning Self-supervised learning has been used in the form of**
different pretext tasks (He et al., 2020; Noroozi & Favaro, 2016; Gidaris et al., 2018) to pre-train
representations that can be used for down-stream tasks as well. These representations have been able
to generalize well in the few-shot regime. Recent works (Gidaris et al., 2019; Su et al., 2020; Chen
et al., 2021) show that adding self-supervised loss functions for representation learning improves fewshot recognition performance. In our paper, we solely use self-supervision in the form of contrastive
loss (Chen et al., 2020) during pre-training because it mitigates supervision collapse as observed
in (Doersch et al., 2020). Furthermore, contrastive losses have been theoretically proven (Saunshi
et al., 2019) to produce better representations for few-shot learning but have not been evaluated for
their generalization ability on few-shot novel categories with distant domains.

**Feature Selection for Few-shot Learning Feature selection is useful for deriving relevant features**
for a particular task or for preventing overfitting on few-shot samples. Zhao et al. (2018) separate
the features into orthogonal components where the sparse signal component facilitates the feature
selection. It is similar to our approach where we use a mask to select relevant and irrelevant features,
yet we impose different constraints on these decomposed features. Liu et al. (2017) use a greedy
feature selection mechanism followed by multiple dropouts to reduce gradient variance of few-shot
samples. However, this method is not applicable for transferring to novel tasks with large domain
differences. A more recent work (Dvornik et al., 2020) select features from a universal representation
learned from multiple source domains by optimizing the selection coefficients for different domains.
This is quite different from our method, which can work even with a single source domain by
selecting relevant features instead of relevant source domains. Berriel et al. (2019) use budget-aware
mechanism of optimizing a switch vector to select domain-relevant feature channels from a pretrained architecture. Additionally, masking has been used to adapt single network weights to multiple
new tasks (Mancini et al., 2018; Mallya et al., 2018).

|𝒙 𝒙𝟏|Source Feature Extractor|
|---|---|


3 PROPOSED FRAMEWORK

𝒎

𝒇𝟐 𝒎 𝒙 **Feature Source** 𝒇 **GeneratorMask**

𝒙 **Feature Source** 𝒇 **Feature Source** 𝒇 **GeneratorMask** **Extractor**

**Extractor**

**Extractor** 𝒙

𝒙𝟏 𝒇𝟏 ⊙

𝑳𝒄𝒐𝒏(𝒇, 𝒇𝟏, 𝒇𝟐) 𝑳𝒏𝒆𝒈(𝒇[−]) 𝒇[−] = (𝟏−𝒎)⊙𝒇 ⊙ **Target** 𝒇[+] = 𝒎⊙𝒇

𝑳𝒅𝒊𝒗(𝒇[+], 𝒇𝑳[−]𝒑𝒐𝒔) (𝒇[+]) 𝒇[+] = 𝒎⊙𝒇 **ExtractorFeature** 𝒇[𝒕] 𝑳𝒓𝒆𝒈(𝒇[𝒕], 𝒇[+])

𝑳𝒕𝒂𝒔𝒌(𝒇[𝒕])

**(a)**
**(b)** **(c)**

Figure 1: Our framework consists of three steps: (a) Pre-training the backbone using a self-supervised contrastive
loss; (b) Learning the masking module on the target data to select relevant features, and (c) Fine-tuning the
backbone using a regularized loss with positively relevant features.


-----

3.1 PROBLEM DESCRIPTON AND NOTATION

For the CDFSL problem, we have a source domain and a target domain. Each domain has an associated
joint distribution P over the input space X and the label space Y. The marginal distribution of the
input space is denoted as P . Instances (x, y) can be sampled from P, where x is the input and y is
_X_
the corresponding label. Accordingly, the source domain can be represented as ( _s,_ _s) and the target_
_X_ _Y_
domain as (Xt, Yt) with joint distributions Ps and Pt, respectively. Due to the domain difference, the
source marginal distribution PXs will be significantly different from the target marginal distribution
_P_ _t. Moreover, the target domain classes are novel; hence there is no overlap between_ _s and_ _t._
_X_ _Y_ _Y_
The goal is to first learn a model from abundant data sampled from the source distribution Ps. Then
the model is adapted to few data sampled from the target distribution Pt. Finally, the adapted model
is evaluated on held-out test data sampled from the target distribution. In our framework, we learn the
model from the source distribution, using a self-supervised contrastive loss function. The adaptation
step on the target data involves learning a mask generator followed by regularized fine-tuning. Our
framework is depicted in Fig. 1, and the details are described in the following subsections.

3.2 UNSUPERVISED TRAINING OF BACKBONE

The backbone (the feature extraction module) is trained in an unsupervised manner inspired from
recent works on contrastive learning (Chen et al., 2020) and unsupervised pre-training (Doersch
et al., 2020). Contrastive learning has been found to be effective for transfer learning (?). This
is because contrastively learned features focus more on mid and low-level features, which are
easily adapted. Furthermore, such features produce better reconstruction by learning a holistic
representation of images rather than focusing only on discriminative regions. Thus, contrastive
learning is an effective pre-training strategy for transferring representations to distant target domains.
In our pre-training stage, we augment samples from the existing samples in the training batch using
various transformations and use these augmented samples and original samples to determine a
contrastive loss. Specifically, let there be Nb training samples in a batch, where the samples are
represented as {xi}i[N]=1[b] [. For each sample][ x][i][, we obtain][ N][t][ random transformations where the][ t][th]
transformed instance is represented as xit and t ∈{1, 2..., Nt}. Following the idea of (Doersch et al.,
2020), we enforce the the transformed instances xit to be close to xi and far from xk, k ̸= i using
the following cross-entropy loss,

1 _Nb_ _Nt_ exp( _d(φs(xit), φs(xi)))_
_Lcon=_ _−_ _NbNt_ log _Nb_ _−_ _._ (1)

_i=1_ _t=1_ _k=1[exp(][−][d][(][φ][s][(][x][it][)][, φ][s][(][x][k][)))]_

XX

Here, φs( ) represents the feature extraction module, andP _d(_ ) is a distance metric. Snell et al. (2017)

_·_ _·_
showed that Euclidean distances model Bregman divergence of mixture densities, which consistently
performs better for the few-shot setting, and so we choose the same metric. The appendix discusses
the theoretical support of constrastive learning for few-shot learning.

3.3 LEARNING THE FEATURE MASKING MODULE

The feature masking module is used to generate masks that can select task-relevant and task-irrelevant
features. For simplicity, we call task-relevant and irrelevant features positive and negative features,
respectively. It is important to note that we cannot afford a large masking sub-network because it
might overfit to few-shot target domain samples. So, we just mask on features fed to the classifier
with appropriate regularization during fine-tuning. Let the feature extraction module learned from the
source domain be denoted as φs(·). Given a batch of target domain samples {(xi, yi)}i[N]=1[, we can]
obtain the feature fi = φs(xi) ∈ R[d] for each sample. We feed the feature into the mask generating
module M (·) to obtain the mask mi = M (fi). This mask is then used to produce positive (fi[+][) and]
negative (fi[−][) features, such that]

**fi[+]** [=][ m][i][ ⊙] **[f][i][,]** **fi[−]** [= (][1][ −] **[m][i][)][ ⊙]** **[f][i]** (2)

whereis a mask vector consisting of ⊙ is the Hadamard product, and d elements where the 1 is a vector of ones of the appropriate dimension. j[th] element is represented as mij. To generate mi ∈ R[d]
binary masks mij, we follow the probabilistic procedure introduced in (Maddison et al., 2017; Jang
et al., 2017). Let zij be the unbounded output logit from the mask module corresponding to the


-----

_i[th]_ sample and the j[th] dimension. We generate logistic noise l such that l = log(u) − log(1 − _u)_
and u uniform(0, 1). The noise is then added to the logits to produce mask mij, such that
_∼_
_mij = σ(_ _[z][ij]τ[+][l]_ ) where σ( ) is the sigmoid operation, and τ is the temperature scale. The noise is

_·_
added to the logits to explore different binary masks suitable for the target task. To back-propagate
discrete masks during training, we follow the straight-through estimator (Bengio et al., 2013) where
we use sigmoid during the backward pass and hard-threshold operation during the forward pass. The
hard-threshold operation involves setting mij to 1 if mij > 0.5 or 0 otherwise. During inference
mode, the hard-threshold operation of the mask is carried out but with the logistic noise l = 0

To train the feature masking module M (·), we want to make sure that the positive features fi[+] [are]
discriminative while the negative features fi[−] [are not. To produce discriminative positive features][ f][ +]i [,]
we use the cross-entropy criterion such that

_Lpos(fi[+][) =][ L][XEnt][(][C]_ [+][(][f][ +]i [)][, y][i][)][,] (3)

where LXEnt( ) is the cross-entropy criterion, and C [+]( ) is a linear classifier used for the positive

_·_ _·_
features fi[+][. To produce negative features][ f][ −]i [, we use the maximum entropy criterion such that]

_Lneg(fi[−][) =][ −][L][Ent][(][C]_ _[−][(][f][ −]i_ [))][,] (4)

where LEnt(·) is the entropy of the softmax outputs of C _[−](fi[−][)][, and][ C]_ _[−][(][·][)][ is a linear classifier]_
used for the negative features fi[−][. The maximum entropy criterion makes sure that output class]
probabilities are uncertain causing the negative features to be less class discriminative.

The design of the mask only makes positive and negative features apart. However, they can still be
statistically similar in arrangement of clusters and higher-order statistics. Hence, a divergence measure
to maximize the statistical distance between the positive and the negative features is required. If we
let sd(·) be the statistical distance between two sets of features: the positive set F[+] = {(fi[+][)]i[N]=1[}]
and the negative set F[−] = {(fi[−][)]i[N]=1[}][, then we would minimize the divergence loss,]

_Ldiv(F[+], F[−]) = e[−][s][d][(][F][+][.][F][−][)]._ (5)

The exponent term is used to provide more stable and smaller gradients when close to optimality. The
loss terms in Eq. 3, 4 and 5 are weighted and combined to obtain

_Lmask = λposLpos + λnegLneg + λdivLdiv._ (6)

Here, Lpos and Lneg are averaged over the batch samples, while Ldiv is an aggregated loss function
over all the batch samples. These loss terms are combined to obtain the final loss Lmask, which is
back-propagated across M (·), C [+](·) and C _[−](·) to update the respective parameters._

3.4 FINE-TUNING

The fine-tuning stage is the final step of adaptation to the target domain. In this step, we train both the
feature extractor and the classifier on the target domain data. Since the target domain contains only
a few labeled data, we regularize the feature extractor to produce positive features using the mask
generator that has been trained in the previous step. Let φt( ) be the target domain feature extractor

_·_
that is initialized from the parameters of the source domain feature extractor φs( ). Given a batch of

_·_
target domain samples {(xi, yi)}i[N]=1[, for each sample we can obtain the feature][ f][ t]i [=][ φ][t][(][x][i][)][ ∈] [R][d][.]
This feature fi[t] [is fed into a linear classifier][ C][(][·][)][ such that we obtain the cross-entropy loss,]

_Ltask(fi[t][) =][ L][XEnt][(][C][(][f][ t]i_ [)][, y][i][)][.] (7)

To regularize the network, we want to make sure that the target domain feature fi[t] [is close to the target]
relevant (positive) feature fi[+] [=][ M] [(][φ][s][(][x][i][))][ ⊙] _[φ][s][(][x][i][)][. This is realized by minimizing the loss,]_

_Lreg =_ **fi[t]** _i_ 2[,] (8)
_||_ _[−]_ **[f][ +][||][2]**

where ||·||2 is the 2-norm. The regularization ensures that the network does not catastrophically forget
the positively relevant features and does not allow the negatively relevant features to be transferred.
Additionally, distance-based regularization has been shown to promote tighter generalization (Gouk
et al., 2021) as discussed in the appendix. The loss terms in Eq. 7 and 8 are combined as

_Lft = Ltask + λregLreg._ (9)


-----

_Lft is then averaged over the training samples in a batch to compute the final loss, which is back-_
propagated across φt( ) and C( ) to update the respective parameters. We can choose not to fine-tune

_·_ _·_
_φt(_ ) but ablation studies in Table 2 show that fine-tuning backbone is effective for CDFSL. This

_·_
completes the fine-tuning stage. All the stages of our proposed framework, including the pre-training
and the fine-tuning steps, are summarized in Algorithm 1. For a test sample xte, we use C(φt(xte))
followed by the softmax operation to obtain the class probabilities and the most probable class.

**Algorithm 1: ConFeSS framework**
**Given: Source dataset Ds and Target dataset Dt**
**Hyper-parameters: λpos, λneg, λdiv, λreg**
**Step 1: Pre-train backbone φs(·) on Ds**
**For each sampled batch of source data**
**For each sampled augmentation**
Take gradient descent step of Eq. 1 with respect to φs( )

_·_
**Step 2: Obtain mask generator M** (·) from Dt
**For each sampled batch of target data**
Take gradient descent step of Eq. 6 with respect to M (·), C [+](·) and C _[−](·)_
**Step 3: Fine-tune backbone φt(·) on Dt**
Initialize φt( ) from optimized φs( )

_·_ _·_
**For each sampled batch of target data**
Take gradient descent step of Eq. 9 with respect to φt( ) and C( )

_·_ _·_
**Step 4: Predict test sample class using optimized φt(·) and C(·)**

4 EXPERIMENTAL RESULTS

4.1 DATASET DESCRIPTION

To evaluate our proposed framework, we test it on the CDFSL benchmark introduced by Guo et al.
(2020). This benchmark uses mini-ImageNet (Vinyals et al., 2016), which is a subset of the ImageNet (Deng et al., 2009) dataset as the source domain that contains abundantly labeled natural
categories. The model learned on the mini-Imagenet dataset is then tested on target datasets containing
only a few labeled training data. These target datasets have large domain differences from the source
domain, and in order of increasing dissimilarity, they consist of the following: a) CropDiseases (Mohanty et al., 2016), containing images of different plant disease types, b) EuroSAT (Helber et al.,
2019), consisting of different classes of satellite imagery, c) ISIC2018 (Tschandl et al., 2018; Codella
et al., 2018), which contains different dermoscopic images of skin lesions, and d) ChestX (Wang et
_al., 2017), a collection of chest X-Ray images of different lung disease types._

Table 1: Results of our approach (ConFeSS) as compared with previous methods on the ChestX, ISIC, EuroSAT
and CropDisease datasets. The best results are shown in boldface. NWKS means N-way K-shot test setting.

|Col1|ChestX|Col3|Col4|ISIC|Col6|Col7|EuroSAT|Col9|Col10|CropDisease|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Method|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|
|MatchNet MatchNet+FWT MAML ProtoNet ProtoNet+FWT RelationNet RelationNet+FWT MetaOpt STARTUP CHEF FT-All ATA ConFeSS|22.40 21.26 23.48 24.05 23.77 22.96 22.74 22.53 26.94 24.72 25.97 24.43 27.09|23.61 23.23 27.53 28.21 26.87 26.63 26.75 25.53 33.19 29.71 31.32 – 33.57|22.12 23.01 – 29.32 30.12 28.45 27.56 29.35 36.91 31.25 35.49 – 39.02|36.74 30.40 40.13 39.57 38.87 39.41 35.54 36.28 47.22 41.26 48.11 45.83 48.85|45.72 32.01 52.36 49.50 43.78 41.77 43.31 49.42 58.63 54.30 59.31 – 60.10|54.58 33.17 – 51.99 49.84 49.32 46.38 54.80 64.16 60.86 66.48 – 65.34|64.45 56.04 71.70 73.29 67.34 61.31 61.16 64.44 82.29 74.15 79.08 83.75 84.65|77.10 63.38 81.95 82.27 75.74 74.43 69.40 79.19 89.26 83.31 87.64 – 90.40|54.44 62.75 – 80.48 78.64 74.91 73.84 83.62 91.99 86.55 90.89 – 92.66|66.39 62.74 78.05 79.72 72.72 68.99 64.91 68.41 93.02 86.87 89.25 90.59 88.88|76.38 74.90 89.75 88.15 85.82 80.45 78.43 82.89 97.51 94.78 95.51 – 95.34|58.53 75.68 – 90.81 87.17 85.08 81.14 91.76 98.45 96.77 97.68 – 97.56|



4.2 IMPLEMENTATION DETAILS

For a fair comparison, we use the ResNet-10 backbone introduced by Guo et al. (2020), which
produces a 512 dimension feature space. We use Adam as the optimizer with a learning rate of 0.001.
The statistical distance sd( ) used in Eq. 5 is maximum mean discrepancy (MMD) (Gretton et al.,

_·_
2012). MMD between two distributions P and Q over feature space is defined as EX _P [φ(X)]_
_X_ _||_ _∼_ _−_


-----

EY _Q[φ(Y )]_ where φ :, and is a reproducing kernel Hilbert space. The MMD can
_∼_ _||H_ _X →H_ _H_
be easily computed using the kernel trick while we use the Gaussian kernel in our experiments.
Unless explicitly mentioned, we use the pre-training batch size Nb = 50 and the augmentation
size Nt = 3, where the augmentations were chosen as in (Chen et al., 2020) . For larger values
of Nt, we found a dip in performance probably because the extra augmentations do not represent
transformations in the target domain. The results of different values of Nt are reported and analyzed
later in the paper. The masking module M (·) consists of a small two-layer feed-forward network
with a hidden layer dimension of 256. We use a small subnetwork for masking to prevent overfitting
issues. We set temperature τ = 1. Also, we set λpos = 10[−][3], λneg = 10[−][2], λdiv = 10[−][2], and
_λreg = 10[−][2]. The numbers of training epochs for Step 1, Step 2, and Step 3 in Algorithm 1 are_
set as 400, 15, and 50, respectively. The hyper-parameters are kept fixed because it is not possible
to create a small-to-medium validation set from the few-shot target dataset. The epoch number of
400 for pre-training is kept the same as fine-tuning methods described in (Guo et al., 2020). For
all experiments, the average accuracy over 600 episodes of N-way K-shot setting is reported. Each
episode contains randomly sampled K-shot samples per class for adaptation and 15 query samples
per class for evaluation, where N is the number of sampled classes.

4.3 COMPARISONS

We compare our proposed approach against several meta-learning based few-shot learning methods introduced in the CDFSL benchmark (Guo et al., 2020): MatchNet (Vinyals et al., 2016), MAML (Finn
et al., 2017), ProtoNet (Snell et al., 2017), RelationNet (Sung et al., 2018), and MetaOpt (Lee et al.,
2019). Furthermore, Feature-wise Transformation (FWT) (Tseng et al., 2020) was added to the
backbones to simulate the cross-domain setting of the benchmark. We also include the FT-All (Guo
et al., 2020) baseline for the comparison that fine-tunes the full network with only the cross-entropy
loss. Additionally, we compare with recent methods, STARTUP, CHEF and ATA, which have been
evaluated on the CDFSL benchmark. It is to be noted that SENet (Hu et al., 2017) scales channels
similar to the way we select features. However, it does not decompose features into relevant and
irrelevant ones. Besides, SENet cannot tackle domain-shift because it is only used as a module in a
feature extraction block. Hence, using SENet as the backbone for comparison would not be useful
(or even fair) because all the compared baselines use the ResNet-10 backbone. The results of the
comparison for 5-way 5-shot, 5-way 20-shot, and 5-way 50-shot test settings are shown in Table 1.

From Table 1, we see that our proposed framework ConFeSS outperforms all meta-learning methods
by a large margin. Specifically, for the 5-shot setting, our method produces improvements of 12.64 %,
21.72 %, 15.50 % and 11.49 % over the best meta-learning method on the ChestX, ISIC, EuroSAT,
and CropDisease datasets, respectively. Veritably, the improvement margin increases further as the
number of shots increases, especially for medical datasets such as ChestX and ISIC. This is because
medical datasets have similar classes and require more annotations to perform reasonably well. Also,
the FWT module fails to generalize to target datasets and sometimes negatively affects these methods.
As expected, the performance of our framework also follows the rank of domain similarity with
miniImagenet: least performance for ChestX and best performance for CropDisease.

The meta-learning methods use supervision for pre-training and cannot mimic distant domain datasets,
which causes them to overfit source data with poor generalization to distant target domains. In comparison, our contrastively learned backbone only learns the inherent structure of data transformations
and can generalize more effectively to other domains. Secondly, the masking module selects only
relevant features for fine-tuning on the target domain, thus preventing overfitting. Our method also
performs better than CHEF on all settings. Compared to FT-All, ATA and STARTUP, our method
achieves much higher scores in most settings except for the CropDisease dataset, which is the easiest
benchmark for the cross-domain task as it is the most similar to miniImageNet. STARTUP uses large
amount of unlabelled target domain data while our proposed approach does not use any. Still, our
method outperforms STARTUP in more difficult 9 out of 12 settings.

4.4 ADDITIONAL ANALYSES

**Ablation study: Table 2 shows ablation study results. The masking module uses losses functions**
_Lneg and Ldiv. Also, the fine-tuning step uses loss Lreg. w/o Feature Mask implies that the pre-_
trained network is fine-tuned using only cross-entropy loss, without using a masking module. We
can also choose not to fine-tune the whole backbone, which is denoted as w/o FT BB in Table 2 for


-----

Table 2: Ablation results. ↑ shows increase in performance with ablation.


|Col1|5-way 5-shot|Col3|Col4|Col5|5-way 20-shot|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|Setting|CropDisease|EuroSAT|ISIC|ChestX|CropDisease|EuroSAT|ISIC|ChestX|
|Full Framework w/o Cont. Learn. w/o FT BB w/o Feature Mask w/o Ldiv w/o Lneg w/o Lreg Direct Positive|88.88 87.26 85.18 87.57 87.95 89.03 ↑ 87.83 87.15|84.65 83.15 83.14 83.87 84.23 84.41 83.98 83.94|48.85 47.66 42.25 47.10 48.62 48.04 48.34 47.25|27.09 26.06 25.76 26.09 26.92 26.60 26.73 26.58|95.34 95.47 ↑ 93.52 95.49 ↑ 94.74 94.72 94.43 93.65|90.40 88.78 89.70 89.93 89.31 90.37 90.31 89.40|60.10 59.96 52.61 61.08 ↑ 59.20 60.14 ↑ 59.83 59.66|33.57 32.12 31.12 33.20 32.77 32.81 32.69 31.92|


the 5-way 5-shot and 5-way 20-shot settings. As shown, in most cases, removing these components
result in a drop in performance, suggesting that all these loss functions and components are essential.
An important step in the framework is fine-tuning the whole backbone, where the most significant
drop is observed when it is absent. This indicates that for large domain differences between source
and target domain, fine-tuning the backbone is essential. In the 5-shot setting, there is always a
drop in performance when removing the feature masking module. However, for the 20-shot setting,
performance improves slightly on the ISIC and CropDisease datasets. This demonstrates that the
feature masking module is more critical for fewer shot settings. Among Ldiv, Lreg, and Lneg,
there is no clear winner since their order of importance depends on the dataset and the shot. In the
table, w/o Cont. Learn. implies when the contrastive pre-training step is replaced by traditional
supervised pre-training using cross-entropy loss. The results show that using supervised pre-training
produces lower recognition accuracy than contrastive pre-training. However, the standard supervised
pre-training along with masking and fine-tuning still performs better than most of the other compared
methods in Table 1. We also consider the Direct Positive setting for the ablation study, where instead
of using Lreg, we directly use the positive features obtained using the mask generator to fine-tune the
feature extractor and classifier. The results obtained using this technique are competitive compared to
previous work but still worse than the Full framework and w/o Lreg ablation.

**The number of features selected: We also analyzed the number of features selected for different**
datasets in Fig. 2 (a). According to VC theory, the number of features selected for 5-shot setting is
less than that of the 20-shot setting to prevent overfitting. The number of features selected for the
50-shot setting is less because the 50-shot setting has lots of training samples and does not require a
masking module. Also, CropDisease has the highest number of features selected while ChestX has
the least number of features selected because CropDisease dataset contains natural images and is
more similar to miniImageNet.


350

340

330

320

310

300

290

280

270

260

250

|1shot 5shot 20shot 50shot|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||


1shot
5shot
20shot
50shot


ChestX ISIC EuroSAT CropDisease

(a)


20 20

Feat neg

15 Feat pos 15

10 10

5 5 Feat neg

Feat pos

0 0

-5 -5

-10 -10

-15 -15

-15 -10 -5 0 5 10 -15 -10 -5 0 5 10


(b)


(c)


Figure 2: (a) Average number of positive features selected for different datasets and shots. T-SNE plot of positive
and negative features of CropDisease dataset for (b) 5-way 5-shot, (c) 5-way 20-shot setting
**Comparison between positive and negative features: We also investigate the difference between**
the positive and negative features. In practice, the positive features are more relevant to the classification task, and therefore expected to be more discriminative than the negative features. We quantify the
discrimination ability of features using the metric DS = Tr(Sb)/Tr(Sw), where Sb is the betweenclass scatter matrix, Sw is the within-class scatter matrix and Tr(·) is the trace operation. The scatter
matrices are defined as Sb = _i=1_ _[n][i][(][µ][i][ −]_ _[µ][)(][µ][i][ −]_ _[µ][)][T][ and][ S][w][ =][ P]j[M]=1[(][x][j][ −]_ _[µ][y]j_ [)(][x][j][ −] _[µ][y]j_ [)][T][,]
where µi is the sample mean of the i[th] class, µ is the mean of all the samples, and ni is the number
of samples in the i[th] class with a total of[P][N] _N classes. (xj, yj) is the j[th]_ sample-label pair out of M
total samples. Higher values of DS indicate better discrimination. We compared DS across different
shots and datasets in Table 3 for the 5-way setting. As expected, DS scores for the positive features
are higher than that of the negative features for both the 5-shot and 20-shot settings. The DS scores
get higher for the positive features of the 20-shot setting since more training samples produce better


-----

Table 3: Discrimination scores (DS) of positive (+) and negative features (-) for different shots (S) and datasets

|Col1|ChestX|Col3|Col4|Col5|ISIC|Col7|Col8|Col9|EuroSAT|Col11|Col12|Col13|CropDisease|Col15|Col16|Col17|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Setting|5S+|5S-|20S+|20S-|5S+|5S-|20S+|20S-|5S+|5S-|20S+|20S-|5S+|5S-|20S+|20S-|
|DS|0.018|0.016|0.03|0.02|0.13|0.10|0.17|0.09|0.67|0.42|0.62|0.28|0.63|0.41|0.77|0.31|



clusters. The DS difference between positive and negative features for the ChestX dataset is low
mostly because the dataset is very hard to cluster. Visualization using t-SNE (Maaten & Hinton, 2008)
for 5-shot and 20-shot settings are shown in Fig. 2 (b) and (c) respectively. Results show that positive
features produce better clusters compared to negative ones. As expected, the positive features for the
20-shot setting are more discriminative compared to the 5-shot setting. Also, the internal statistics of
positive and negative features are different because of Ldiv, which maximizes the statistical distance
between the two sets of features.


90 90 90 ISIC 90

80 ISIC 80 ISIC 80 EuroSAT 80 ISIC

70 EuroSATCropDisease 70 EuroSATCropDisease 70 CropDiseaseChestX 70 EuroSATCropDisease

60 ChestX 60 ChestX 60 60 ChestX

50 50 50 50

40 40 40 40

30 30 30 30

20-3 -2 -1 0 1 2 3 20-3 -2 -1 0 1 2 3 20-3 -2 -1 0 1 2 3 20-3 -2 -1 0 1 2 3


(a)


(b)


(c)


(d)


Figure 3: Accuracy with 5W5S setting as (a) log10 λneg (b) log10 λdiv, (c) log10 λreg and (d) log10 λpos vary.

**Hyper-parameter sensitivity: We study our framework’s performance as the hyper-parameters**
_λneg, λpos, λdiv, and λreg are varied. The results for the 5-shot setting are shown in Fig. 3, which_
shows that the recognition performance is stable with respect to λneg, λpos, and λdiv, while the
performance drops for larger values of λreg. This is because λneg, λpos, and λdiv affect the learning
of a much smaller feature masking network. On the other hand, the value of λreg affects the learning
of a much larger network - the target feature extractor, which eventually plays a direct role in the
inference stage. The plot in Fig. 3 (c) shows that we should choose λreg < 1 for better performance.

**The impact of the number of augmentations: We report how the number of augmentations Nt used**
in contrastive pre-training affects cross-domain few-shot recognition performance. Results for 5-way
5-shot, 5-way 20-shot, and 5-way 50 shot settings are shown in Table 4. Results show a sharp drop in

Table 4: Recognition performance on the N-way K-shot setting as Nt is varied during pre-training.

|Col1|5-way 5-shot|Col3|Col4|Col5|5-way 20-shot|Col7|Col8|Col9|5-way 50-shot|Col11|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Dataset/Nt|3|10|20|30|3|10|20|30|3|10|20|30|
|CropDisease EuroSAT ISIC ChestX|88.88 84.65 48.85 27.09|70.11 62.28 37.59 23.30|72.28 60.38 38.08 23.29|70.51 59.93 36.64 23.16|95.34 90.40 60.10 33.57|86.21 72.45 48.60 25.68|86.85 70.40 48.80 25.24|87.08 69.50 47.99 25.37|97.56 92.66 65.34 39.02|92.48 76.87 53.86 27.72|92.10 75.99 54.78 27.34|92.42 75.29 53.78 27.25|



recognition performance when the number of augmentations is increased beyond 3. This is because
additional augmentations in the source dataset do not represent the possible augmentations in the
target datasets. With higher Nt, there is a propensity to have augmentations that are not valid for target
classes. The target datasets consist of specialized domains like medical and satellite imagery, which
also exuberate inconsistent categories when the target datasets are transformed using arbitrary source
augmentation policies. As a result, contrastive representations learned using those augmentations
might not generalize well. For example, in ChestX, random cropping or Gaussian blur might affect
discriminative regions in images. This phenomenon has also been recently studied in (Xiao et al.,
2021), where color augmented representations do not transfer well for color discrimination tasks.

5 CONCLUSION
We presented a framework called ConFeSS (Contrastive Learning and Feature Selection System) to
learn a generalizable representation followed by a feature selection mechanism while fine-tuning
on the target domain. We introduce novel loss constraints on selecting relevant and irrelevant
features for the target domain. Extensive experiments conducted on the cross-domain few-shot
learning benchmark show our approach’s advantages over the meta-learning and other CDFSL
methods. Additional analyses also provide insights into the feature selection mechanism and justify
the importance of each component of our framework.


-----

REFERENCES

Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David Kreil, Michael Kopp,
Günter Klambauer, and Sepp Hochreiter. Cross-domain few-shot learning by representation fusion.
_arXiv preprint arXiv:2010.06498, 2020._

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through
stochastic neurons for conditional computation. arXiv:1308.3432, 2013.

Rodrigo Berriel, Stephane Lathuillere, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe,
and Elisa Ricci. Budget-aware adapters for multi-domain learning. In International Conference on
_Computer Vision, pp. 382–391, 2019._

Zhong Cao, Jiang Lu, Jian Liang, and Changshui Zhang. A theory of self-supervised framework for
[few-shot learning, 2021. URL https://openreview.net/forum?id=-aThAo4b1zn.](https://openreview.net/forum?id=-aThAo4b1zn)

Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. Self-supervised learning
for few-shot image classification. In International Conference on Acoustics, Speech, and Signal
_Processing, pp. 1745–1749, 2021._

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International Conference on Machine Learning,
2020.

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look
at few-shot classification. In International Conference on Learning Representations, 2019.

Noel CF Codella, David Gutman, Celebi, et al. Skin lesion analysis toward melanoma detection. In
_International Symposium on Biomedical Imaging, pp. 168–172, 2018._

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pp.
248–255, 2009.

Anamika Dhillon and Gyanendra K Verma. Convolutional neural network: a review of models,
methodologies and applications to object detection. Progress in Artificial Intelligence, 9(2):85–112,
2020.

Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot
transfer. In Conference on Neural Information Processing Systems, 2020.

Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a multi-domain
representation for few-shot classification. In European Conference on Computer Vision, 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of
deep networks. In International Conference on Machine Learning, pp. 1126–1135, 2017.

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by
predicting image rotations. In International Conference on Learning Representations, 2018.

Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Pérez, et al. Boosting few-shot visual learning
with self-supervision. In International Conference on Computer Vision, pp. 8059–8068, 2019.

Henry Gouk, Timothy Hospedales, and massimiliano pontil. Distance-based regularisation of deep
networks for fine-tuning. In International Conference on Learning Representations, 2021.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A
kernel two-sample test. Journal of Machine Learning Research, 13(1):723–773, 2012.

Yunhui Guo, Noel C Codella, et al. A broader study of cross-domain few-shot learning. In European
_Conference on Computer Vision, 2020._

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In Conference on Computer Vision and Pattern
_Recognition, pp. 9729–9738, 2020._


-----

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and
deep learning benchmark for land use and land cover classification. Journal of Selected Topics in
_Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019._

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural
networks: A survey. Transactions on Pattern Analysis and Machine Intelligence, 2021.

Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks.
_Transactions on Pattern Analysis and Machine Intelligence, 2017._

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In
_International Conference on Learning Representations, 2017._

Myeongjae Jeon et al. Analysis of large-scale multi-tenant GPU clusters for DNN training workloads.
In USENIX Annual Technical Conference, pp. 947–960, 2019.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses
for scene geometry and semantics. In Conference on Computer Vision and Pattern Recognition, pp.
7482–7491, 2018.

Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few-shot
image classification. In Conference on Neural Information Processing Systems, 2019.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, et al. Meta-learning with differentiable
convex optimization. In Conference on Computer Vision and Pattern Recognition, pp. 10657–
10665, 2019.

Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample
size data. In International Joint Conference on Artificial Intelligence, pp. 2287–2293, 2017.

Laurens Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning
_Research, 9(Nov):2579–2605, 2008._

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous
relaxation of discrete random variables. In International Conference on Learning Representations,
2017.

Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple
tasks by learning to mask weights. In European Conference on Computer Vision, pp. 67–82, 2018.

Massimiliano Mancini, Elisa Ricci, Barbara Caputo, and Samuel Rota Bulo. Adding new tasks to
a single network with weight transformations using binary masks. In European Conference on
_Computer Vision Workshops, pp. 180–189, 2018._

Carlos Medina, Arnout Devos, and Matthias Grossglauser. Self-supervised prototypical transfer
learning for few-shot classification. arXiv preprint arXiv:2006.11325, 2020.

Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based
plant disease detection. Frontiers in Plant Science, 7:1419, 2016.

Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action localization
by sparse temporal pooling network. In Conference on Computer Vision and Pattern Recognition,
pp. 6752–6761, 2018.

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw
puzzles. In European Conference on Computer Vision, pp. 69–84, 2016.

Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task
differences. In International Conference on Learning Representations, 2021.

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, et al. A theoretical analysis of contrastive
unsupervised representation learning. In International Conference on Machine Learning, pp.
5628–5637, 2019.


-----

John Shawe-Taylor and N Cristianini. An introduction to support vector machines and other kernel_based learning methods. 2000._

Jake Snell, Kevin Swersky, et al. Prototypical networks for few-shot learning. In Conference on
_Neural Information Processing Systems, 2017._

Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot
learning? In European Conference on Computer Vision, pp. 645–666, 2020.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable
effectiveness of data in deep learning era. In International Conference on Computer Vision, pp.
843–852, 2017.

Flood Sung, Yongxin Yang, Li Zhang, et al. Learning to compare: Relation network for few-shot
learning. In Conference on Computer Vision and Pattern Recognition, pp. 1199–1208, 2018.

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, et al. Meta-dataset: A dataset of datasets for
learning to learn from few examples. In International Conference on Learning Representations,
2020.

Philipp Tschandl et al. The ham10000 dataset, a large collection of multi-source dermatoscopic
images of common pigmented skin lesions. Scientific Data, 2018.

Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot
classification via learned feature-wise transformation. In International Conference on Learning
_Representations, 2020._

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one
shot learning. In Conference on Neural Information Processing Systems, pp. 3630–3638, 2016.

Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task
augmentation. In International Joint Conference on Artificial Intelligence, pp. 1075–1081, 2021.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples:
A survey on few-shot learning. Computing Surveys, 53(3):1–34, 2020.

Xiaosong Wang et al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weaklysupervised classification and localization of common thorax diseases. In Conference on Computer
_Vision and Pattern Recognition, pp. 2097–2106, 2017._

Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in
contrastive learning. In International Conference on Learning Representations, 2021.

Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In
_Conference on Artificial Intelligence, volume 33, pp. 7370–7377, 2019._

Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal
domain adaptation. In Conference on Computer Vision and Pattern Recognition, pp. 2720–2729,
2019.

Sungrack Yun, Janghoon Cho, et al. An end-to-end text-independent speaker verification framework
with a keyword adversarial network. In Conference of the International Speech Communication
_Association, 2019._

Qingchen Zhang, Laurence T Yang, Zhikui Chen, and Peng Li. A survey on deep learning for big
data. Information Fusion, 42:146–157, 2018.

Bo Zhao, Xinwei Sun, et al. Msplit lbi: Realizing feature selection and dense estimation simultaneously in few-shot and zero-shot learning. In International Conference on Machine Learning, pp.
5912–5921, 2018.

Jian Zhou et al. Deep learning sequence-based ab initio prediction of variant effects on expression
and disease risk. Nature genetics, 50(8):1171–1179, 2018.


-----

A ADDITIONAL EXPERIMENTAL DETAILS

All our experiments were conducted in a cluster of nodes where the nodes contain NVIDIA Tesla V100
GPUs with a mix of 16GB and 32GB memory. The details of the benchmark used for comparison
[have been introduced in the following repository: https://github.com/IBM/cdfsl-benchmark. We](https://github.com/IBM/cdfsl-benchmark)
only consider the single source domain setting where the source domain is miniImageNet [1], and the
target domains are ChestX [2], ISIC [3], EuroSAT [4] and CropDisease [5]. Additional experimental details
include: (a) Image size: 224 × 224 (b) Batch size during adaptation: 5 (c) MMD kernel numbers: 5
(d) MMD kernel multiplier: 2.0. The masking module architecture is as follows: [Linear(512,256) ReLU - Linear(256,512) - Gumbel Sigmoid]

B IMPLEMENTATION OF MMD

We use a multi-kernel approach to implement MMD. Specifically, for the positive and negative
feature matrix:Here, [XX]ij = F[+] _∈k=0R[N][exp][×][d]_ [(]and[−] _||F F[+]i:bm[−][−][F][k]∈[+]j:[||]2[2]R)[N], [[×]Y Y[d], MMD ]ij =_ =k mean=0 [exp](XX[(][−] _|| +F[−]i Y Y:bm[−][F][k]j[−] −:[||]2[2]_ )XY, [XY − ]Y Xij =).

_nk_ 1 **F[+]i:** _[−][F][−]j:[||]2[2]_ **F[−]i:** _j:[||]2[2]_
_k=0−_ [exp][(][−] _||_ _bm[k]_ ) and [Y X]ij = _k=0_ [exp][(][−] _||_ _bm[−][F][k][+]_ ). F[+]i: [is the][ i][th][ row of][ F][+][.]

[P][n][k][−][1] [P][n][k][−][1]

Pm is the kernel multiplier and nk is the number of kernels. b is the bandwidth and is computed as

sum(D) [P][n][k][−][1]
_b =_ (4N [2] 2N )(m[floor][(0][.][5][nk] [)]) [where][ [][D][]][ij][ =][ ||][F][i][:][ −] **[F][j][:][||]2[2]** [and][ F][ ∈] [R][2][N] _[×][d][ is the concatenation of][ F][+]_

_−_

and F[−].


C DATA AUGMENTATION FOR CONTRASTIVE PRE-TRAINING

The augmentation policies are the same as those used for pre-training the SimCLR framework (Chen
et al., 2020). They include the following transformations:

-  Random Crop and Resize: The random cropping has scale in the range [0.08, 1.0] and
aspect ratio in the range [3/4, 4/3]. The random cropping is always followed by a horizontal
flip with each flip type having a probability of 0.5. This is followed by resizing of image to
the desired size.

-  Color Distortion: Color jitter is applied with a probability of 0.8. The jitter strength values
of brightness, contrast, saturation and hue are set as 0.8, 0.8, 0.8 and 0.2, respectively. This
is followed by color drop operation (convert to grayscale) with a probability of 0.2.

-  Gaussian Blur: This is applied with a probability of 0.5, and radius of blur is selected
randomly from the range [0.1, 2.0].

D EFFECT OF DIFFERENT WAYS

We also test our framework on higher number of ways i.e. higher number of classes per episodic
evaluation. The results of the experiment are shown in Table 5. As expected, higher number of ways
leads to drop in performance because of more difficulty in discrimination. Surprisingly, the drop
in performance is less for easier datasets like EuroSAT and CropDisease. This might be probably
because the additional classes for higher ways leads to less confusion compared to those of ChestX
and ISIC.

[1100 categories. Downloaded from: https://drive.google.com/file/d/1uxpnJ3Pmmwl-6779qiVJ5JpWwOGl48xt/view](https://drive.google.com/file/d/1uxpnJ3Pmmwl-6779qiVJ5JpWwOGl48xt/view)
[27 categories. License and download information available at: https://www.kaggle.com/nih-chest-xrays/data](https://www.kaggle.com/nih-chest-xrays/data)
[37 categories. License and download information available at: https://challenge.isic-archive.com/data#2018](https://challenge.isic-archive.com/data##2018)
[410 categories. Downloaded from: http://madm.dfki.de/files/sentinel/EuroSAT.zip](http://madm.dfki.de/files/sentinel/EuroSAT.zip)
[538 categories. License and download information available at: https://www.kaggle.com/saroz014/plant-disease](https://www.kaggle.com/saroz014/plant-disease)


-----

Table 5: Recognition results of our approach with different ways and shots (S) on the ChestX, ISIC, EuroSAT
and CropDisease datasets. The entries with – imply that testing is not possible because of lesser number of total
categories present.

|Col1|ChestX|Col3|Col4|ISIC|Col6|Col7|EuroSAT|Col9|Col10|CropDisease|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Setting|5S|20S|50S|5S|20S|50S|5S|20S|50S|5S|20S|50S|
|5-way 7-way 10-way 19-way|27.09 20.72 – –|33.57 27.15 – –|39.02 32.33 – –|48.85 40.38 – –|60.10 52.57 – –|65.34 59.29 – –|84.65 80.23 76.03 –|90.40 87.66 84.59 –|92.66 90.90 88.15 –|88.88 85.37 82.10 75.31|95.34 94.00 92.63 88.45|97.56 96.51 95.48 93.39|



E COMPARISON IN 1-SHOT SETTING

Of all the compared methods, only STARTUP (Phoo & Hariharan, 2021) and ATA (Wang & Deng,
2021) have been evaluated on the 1-shot setting. Hence, we evaluate our model on the 1-shot setting
and report the results in Table 6. Results show that our method is still competitive with respect to
existing methods.

Table 6: Comparison on the 1-shot setting.

|Col1|ChestX|ISIC|EuroSAT|CropDisease|
|---|---|---|---|---|
|Method|1S|1S|1S|1S|
|STARTUP ATA ConFeSS|23.09 22.14 23.67|32.66 34.70 33.46|63.88 65.94 65.51|75.93 77.82 76.49|



F COMPARISON WITH UNSUPERVISED META-TRAINING

We also compare our method with UMTRA (Khodadadeh et al., 2019) - an unsupervised metatraining framework derived from the popular meta-learning framework MAML (Finn et al., 2017).
UMTRA uses a similar algorithm as MAML but extends it to the case of unlabeled training data.
Class membership for unlabeled data is determined such that a sample and its augmentation belong
to the same class. We compare against two versions: UMTRA-ProtoNet and UMTRA-ProtoTune as
reported in (Medina et al., 2020) on the CDFSL benchmark (Guo et al., 2020). UMTRA-ProtoTune
extends UMTRA-ProtoNet by fine-tuning on target domain data. The results in Table 7 show that our
method outperforms the two variants of UMTRA on all shots and all datasets.

Table 7: Comparison against unsupervised meta-training with different shots (S) and 5-way setting on the
**ChestX, ISIC, EuroSAT and CropDisease datasets.**

|Col1|ChestX|Col3|Col4|ISIC|Col6|Col7|EuroSAT|Col9|Col10|CropDisease|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Method|5S|20S|50S|5S|20S|50S|5S|20S|50S|5S|20S|50S|
|UMTRA-ProtoNet UMTRA-ProtoTune ConFeSS|24.94 25.00 27.09|28.04 30.41 33.57|29.88 35.63 39.02|39.21 38.47 48.85|44.62 51.60 60.10|46.48 60.12 65.34|74.91 68.11 84.65|80.42 81.56 90.40|82.24 85.05 92.66|79.81 82.67 88.88|86.84 92.04 95.34|88.44 95.46 97.56|



G EFFECT OF DIFFERENT EPOCH NUMBERS

The numbers of epochs for training the masking module and fine-tuning are fixed at 15 and 50
respectively because we do not have validation split from few-shot target domain dataset to set an
optimal value. The epoch numbers (15 and 50) are kept substantially low compared to pre-training
epoch number (400) so that the network is less prone to over-fitting on few-shot data. To show
the effect of different epoch numbers, we perform the following experiments on the 5-way 5-shot
setting. Firstly, the number of epochs for training the masking module is varied and then the number
of epochs for fine-tuning is varied. In Table 8, we show the results for varying epochs for training
masking module while keeping epochs for fine-tuning fixed at 50 as well as the results for varying
epochs for fine-tuning while keeping epochs for training masking module fixed at 15. From the
results, it seems that the recognition performance is not very sensitive to the number of epochs used
for training masking module or for fine-tuning. However, increasing fine-tuning epochs beyond 50
tends to decrease performance slightly for ISIC, EuroSAT and CropDisease datasets.


-----

Table 8: Effect of varying epochs for training masking module and for fine-tuning. In the first column, number
without parantheses is the epoch number for training masking module while that with parantheses is the epoch
number for fine-tuning. The results are shown in corresponding labelled super-columns.

|Col1|Varying epoch number for training masking module|Col3|Col4|Col5|(Varying epoch number for fine-tuning)|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|Setting|ChestX|ISIC|EuroSAT|CropDisease|ChestX|ISIC|EuroSAT|CropDisease|
|5 (25) 10 (50) 15 (75) 20 (100) 25 (125)|26.50 27.26 27.09 26.87 27.27|48.82 48.24 48.85 48.48 48.34|83.64 83.90 84.65 84.18 83.69|87.64 88.78 88.88 88.01 88.04|26.82 27.09 27.09 27.33 27.07|48.54 48.85 48.23 47.83 47.48|84.23 84.65 83.10 82.86 82.31|88.27 88.88 88.56 88.23 88.05|



H ALTERNATIVE MODEL DESIGN CHOICES

In this section, we consider the following alternative model designs and report recognition performance on the 5-way 5-shot, 5-way 20-shot and 5-way 50-shot setting. The evaluation setup is similar
to that described in Section 4.2:

-  L1 norm: In this design of the ConFeSS framework, we just replace the L2 norm **fi[t]** _i_ 2
in Eq. 8 with the L1 norm **fi[t]** _i_ 1[.] _||_ _[−][f][ +][||][2]_
_||_ _[−]_ **[f][ +][||][2]**

-  Source Mask: In this setup, we train the two layer mask module (defined in appendix A)
on the source dataset rather than the target dataset. The source dataset used is miniImageNet.
Specifically, the mask module is trained on top of the contrastively learned pre-trained
feature extractor with the miniImageNet dataset. In the final step, the target feature extractor
is fine-tuned on the target dataset.



-  Neg. Reg.: In this setup, we use negative features for the regularization. Specifically, we
use ||fi[t] _[−]_ **[f][ −]i** _[||]2[2]_ [in Eq. 8 instead of][ ||][f][ t]i _[−]_ **[f][ +]i** _[||]2[2][.]_

-  Dir. Neg.: We also consider the Direct Negative setting for the ablation study, where instead
of using Lreg, we directly use the negative features obtained using the mask generator to
fine-tune the feature extractor and classifier.

-  K Layer Mask Mod.: Here, we study the effect of having masking module of different
sizes. Here, K stands for the number of layers used for the masking module. Specifically, we
study the effect for K = 3, 4, 5. The masking module architecture for K = 3, 4, 5 are [Linear(512,256) - BatchNorm1D(256) - ReLU - Linear(256,128) - BatchNorm1D(128) - ReLU

-  Linear(128, 512) - Gumbel Sigmoid], [Linear(512,256) - BatchNorm1D(256) - ReLU - Linear(256,128) - BatchNorm1D(128) - ReLU - Linear(128, 256) - BatchNorm1D(256) - ReLU

-  Linear(256, 512) - Gumbel Sigmoid], and [Linear(512,256) - BatchNorm1D(256) - ReLU

-  Linear(256,128) - BatchNorm1D(128) - ReLU - Linear(128, 64) - BatchNorm1D(64) ReLU - Linear(64, 128) - BatchNorm1D(128) - ReLU - Linear(128, 512) - Gumbel Sigmoid],
respectively.

-  Joint Training: We consider the setup where the the masking module and the target
feature extractor are trained together in one stage instead of the proposed two stages, using
combined losses in Eq. 6 and Eq. 9.

The results of comparing these alternative model designs with our proposed framework ConFeSS are
shown in Table 9. Results show that among all these alternative model designs, especially L1 norm,
Source Mask, Neg. Reg., and Dir. Neg. perform poorly compared to our original ConFeSS framework.
In most of the cases, having a larger masking module produces similar or slightly better performance
compared to ConFeSS because of better representation capacity of output masks. Joint training of
masking module and target feature extractor produces poorer recognition performance for 5 shot
setting compared to ConFeSS. However, for higher shot setting, the joint training procedure produces
similar or better performance compared to ConFeSS. This is because joint training encompasses
optimization of larger number of parameters, which might cause the network to overfit on lower shots
while exploit additional amount of training data for higher shots.

I RESULTS WITH CONFIDENCE INTERVAL

In this section, we re-report comparison studies: Table 10 and Table 11 show the performance with
95 % confidence interval for Table 1 and Table 2, respectively.


-----

Table 9: Recognition performance on alternative model designs along with 95 % confidence interval shown in
parentheses for different shots and datasets.

|Col1|ChestX|Col3|Col4|ISIC|Col6|Col7|EuroSAT|Col9|Col10|CropDisease|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Method|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|
|L1 norm Source Mask Neg. Reg. Dir. Neg. 3 Layer Mask Mod. 4 Layer Mask Mod. 5 Layer Mask Mod. Joint Training ConFeSS|25.78 (0.44) 25.82 (0.34) 24.05 (0.11) 24.23 (0.41) 26.79 (0.36) 27.19 (0.38) 26.96 (0.42) 25.93 (0.41) 27.09 (0.24)|31.79 (0.47) 30.59 (0.47) 28.21 (0.25) 27.48 (0.36) 34.12 (0.28) 34.11 (0.22) 34.63 (0.21) 34.33 (0.32) 33.57 (0.31)|38.41 (0.52) 36.55 (0.21) 29.32 (0.23) 30.32 (0.22) 40.04 (0.12) 40.34 (0.20) 39.79 (0.20) 39.57 (0.30) 39.02 (0.12)|45.45 (0.35) 44.04 (0.32) 39.57 (0.28) 39.17 (0.44) 48.84 (0.27) 48.79 (0.18) 48.70 (0.37) 47.24 (0.41) 48.85 (0.29)|56.40 (0.35) 53.69 (0.24) 49.50 (0.51) 48.24 (0.38) 60.26 (0.23) 60.79 (0.29) 60.24 (0.15) 59.62 (0.23) 60.10 (0.33)|62.32 (0.30) 61.17 (0.43) 51.99 (0.50) 50.72 (0.51) 65.88 (0.46) 65.95 (0.11) 66.39 (0.39) 66.12 (0.34) 65.34 (0.45)|81.29 (0.42) 79.57 (0.60) 73.29 (0.37) 72.69 (0.36) 84.13 (0.34) 84.16 (0.32) 83.74 (0.23) 83.01 (0.30) 84.65 (0.38)|89.81 (0.39) 85.33 (0.48) 82.27 (0.62) 81.34 (0.55) 90.59 (0.23) 90.63 (0.26) 90.65 (0.14) 90.23 (0.12) 90.40 (0.24)|90.34 (0.17) 90.78 (0.39) 85.48 (0.35) 83.98 (0.17) 91.52 (0.27) 91.70 (0.28) 91.20 (0.14) 91.50 (0.38) 92.66 (0.36)|83.80 (0.28) 85.12 (0.38) 79.72 (0.46) 80.32 (0.41) 88.36 (0.37) 88.38 (0.15) 88.10 (0.50) 88.27 (0.43) 88.88 (0.51)|92.65 (0.67) 90.26 (0.25) 88.15 (0.33) 86.15 (0.12) 95.72 (0.41) 95.56 (0.44) 95.93 (0.56) 95.71 (0.24) 95.34 (0.48)|96.19 (0.23) 94.75 (0.24) 90.81 (0.22) 91.24 (0.53) 97.67 (0.10) 97.61 (0.33) 97.64 (0.12) 97.60 (0.18) 97.56 (0.43)|



Table 10: Table 1 results along with 95 % confidence interval shown in parentheses.

|Col1|ChestX|Col3|Col4|ISIC|Col6|Col7|EuroSAT|Col9|Col10|CropDisease|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|Method|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|5W5S|5W20S|5W50S|
|MatchNet MatchNet+FWT MAML ProtoNet ProtoNet+FWT RelationNet RelationNet+FWT MetaOpt STARTUP CHEF FT-All ATA ConFeSS|22.40 (0.7) 21.26 (0.31) 23.48 (0.96) 24.05 (1.01) 23.77 (0.42) 22.96 (0.88) 22.74 (0.40) 22.53 (0.91) 26.94 (0.94) 24.72 (0.14) 25.97 (0.41) 24.43 (0.2) 27.09 (0.24)|23.61 (0.86) 23.23 (0.37) 27.53 (0.43) 28.21 (1.15) 26.87 (0.43) 26.63 (0.92) 26.75 (0.41) 25.53 (1.02) 33.19 (0.46) 29.71 (0.27) 31.32 (0.45) – (–) 33.57 (0.31)|22.12 (0.88) 23.01 (0.34) – 29.32 (1.12) 30.12 (0.46) 28.45 (1.20) 27.56 (0.40) 29.35 (0.99) 36.91 (0.50) 31.25 (0.20) 35.49 (0.45) – (–) 39.02 (0.12)|36.74 (0.53) 30.40 (0.48) 40.13 (0.58) 39.57 (0.57) 38.87 (0.52) 39.41 (0.58) 35.54 (0.55) 36.28 (0.50) 47.22 (0.61) 41.26 (0.34) 48.11 (0.64) 45.83 (0.3) 48.85 (0.29)|45.72 (0.53) 32.01 (0.48) 52.36 (0.57) 49.50 (0.55) 43.78 (0.47) 41.77 (0.49) 43.31 (0.51) 49.42 (0.60) 58.63 (0.58) 54.30 (0.34) 59.31 (0.48) – (–) 60.10 (0.33)|54.58 (0.65) 33.17 (0.43) – 51.99 (0.52) 49.84 (0.51) 49.32 (0.51) 46.38 (0.53) 54.80 (0.54) 64.16 (0.58) 60.86 (0.18) 66.48 (0.56) – (–) 65.34 (0.45)|64.45 (0.63) 56.04 (0.65) 71.70 (0.72) 73.29 (0.71) 67.34 (0.76) 61.31 (0.72) 61.16 (0.70) 64.44 (0.73) 82.29 (0.60) 74.15 (0.27) 79.08 (0.61) 83.75 (0.4) 84.65 (0.38)|77.10 (0.57) 63.38 (0.69) 81.95 (0.55) 82.27 (0.57) 75.74 (0.70) 74.43 (0.66) 69.40 (0.64) 79.19 (0.62) 89.26 (0.43) 83.31 (0.14) 87.64 (0.47) – (–) 90.40 (0.24)|54.44 (0.67) 62.75 (0.76) – 80.48 (0.57) 78.64 (0.57) 74.91 (0.58) 73.84 (0.60) 83.62 (0.58) 91.99 (0.36) 86.55 (0.15) 90.89 (0.36) – (–) 92.66 (0.36)|66.39 (0.78) 62.74 (0.90) 78.05 (0.68) 79.72 (0.67) 72.72 (0.70) 68.99 (0.75) 64.91 (0.79) 68.41 (0.73) 93.02 (0.45) 86.87 (0.27) 89.25 (0.51) 90.59 (0.3) 88.88 (0.51)|76.38 (0.67) 74.90 (0.71) 89.75 (0.42) 88.15 (0.51) 85.82 (0.51) 80.45 (0.64) 78.43 (0.59) 82.89 (0.54) 97.51 (0.21) 94.78 (0.12) 95.51 (0.31) – (–) 95.34 (0.48)|58.53 (0.73) 75.68 (0.78) – 90.81 (0.43) 87.17 (0.50) 85.08 (0.53) 81.14 (0.56) 91.76 (0.38) 98.45 (0.17) 96.77 (0.88) 97.68 (0.21) – (–) 97.56 (0.43)|



J FEATURE MASKING AND VC THEORY

The generalization ability of a machine learning model is related to the Vapnik-Chervonenkis (VC)
theory. The VC dimension (Shawe-Taylor & Cristianini, 2000) measures the capacity or complexity
of a machine learning model. For a model family, the VC dimension is the maximum number
of training points that can be shattered by that family. The VC dimension of a set of separating
hyperplanes is d + 1 where d is the feature space dimensionality. Vapnik proved that with probability
1 _η, the test loss (_ _te) is upper bounded as_
_−_ _L_


_γ + log(2N_ ) − log( _[η]4_ [)]


(10)


_Lte ≤Ltr +_


where Ltr is the training loss, N is the number of training samples, and γ is the VC dimension. For
better generalization, the goal is to reduce the upper bound, which can be decreased by having more
training samples N . However, when N is small in the few-shot setting, the upper bound increases,
triggering generalization performance to drop. If we reduce γ, we can decrease the upper bound. For
a linear classifier, γ is upper bounded by the number of features. Hence, if we reduce the number of
features, we also reduce the upper bound of γ and subsequently the generalization upper bound. This
is realized with the masking module M (·), which selects a fraction of features before forwarding them


-----

Table 11: Table 2 results along with 95 % confidence interval shown in parentheses.

|Col1|5-way 5-shot|Col3|Col4|Col5|5-way 20-shot|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|Setting|CropDisease|EuroSAT|ISIC|ChestX|CropDisease|EuroSAT|ISIC|ChestX|
|Full Framework w/o Cont. Learn. w/o FT BB w/o Feature Mask w/o Ldiv w/o Lneg w/o Lreg Direct Positive|88.88 (0.51) 87.26 (0.32) 85.18 (0.42) 87.57 (0.44) 87.95 (0.24) 89.03 (0.12) ↑ 87.83 (0.30) 87.15 (0.16)|84.65 (0.38) 83.15 (0.21) 83.14 (0.25) 83.87 (0.26) 84.23 (0.21) 84.41 (0.36) 83.98 (0.26) 83.94 (0.15)|48.85 (0.29) 47.66 (0.18) 42.25 (0.73) 47.10 (0.13) 48.62 (0.22) 48.04 (0.18) 48.34 (0.32) 47.25 (0.24)|27.09 (0.24) 26.06 (0.42) 25.76 (0.52) 26.09 (0.15) 26.92 (0.50) 26.60 (0.30) 26.73 (0.24) 26.58 (0.14)|95.34 (0.48) 95.47 (0.53) ↑ 93.52 (0.56) 95.49 (0.26) ↑ 94.74 (0.34) 94.72 (0.17) 94.43 (0.22) 93.65 (0.21)|90.40 (0.24) 88.78 (0.17) 89.70 (0.25) 89.93 (0.20) 89.31 (0.25) 90.37 (0.38) 90.31 (0.22) 89.40 (0.11)|60.10 (0.33) 59.96 (0.28) 52.61 (0.34) 61.08 (0.24) ↑ 59.20 (0.25) 60.14 (0.18) ↑ 59.83 (0.08) 59.66 (0.16)|33.57 (0.31) 32.12 (0.28) 31.12 (0.24) 33.20 (0.18) 32.77 (0.09) 32.81 (0.34) 32.69 (0.12) 31.92 (0.20)|



to the linear classifier. Thus, feature selection has theoretical support for improving generalization
performance in the few-shot setting. Also, empirical results in Fig. 2 (a) show different datasets and
shots selecting different number of features and hence realizing different upper bounds of γ.

K THEORY OF CONTRASTIVE LEARNING AND FEW-SHOT LEARNING

In (Cao et al., 2021), the authors proved the following bound:

_Lsup ≤_ _γ0LU[−]_ [+][ γ][1][s][(][f][k][)][.] (11)

Here, Lsup is the supervised evaluation metric for learned representations. LU[−] [is the unsupervised]
contrastive evaluation metric for true negative samples. s(fk) is the intra-class deviation using the key
encoder fk. γ0 and γ1 are coefficients depending on class distributions. Lsup can be the training loss
of any supervised few-shot meta-learning method which can generalize to novel categories. Since
_LU[−]_ [upper bounds][ L][sup][, decreasing][ L]U[−] [amounts to decreasing][ L][sup][. Also,][ L]U[−] [can be decreased]
arbitrarily because it is evaluated only on true negative samples. Hence, contrastive losses can be
useful for learning representations that are effective for few-shot learning.

L THEORY OF DISTANCE-BASED REGULARIZATION FOR FINE-TUNING

In (Gouk et al., 2021), the authors proved that with probability 1 _η, the test loss (_ _te) is upper_
_−_ _L_
bounded as

_L_ _Dj[F]_ _L_ log(2/η)

_Lte ≤Ltr + κ_ _j=1_ 2Bj[F] _ji=1_ _√ni_ _j=1_ 2Bj[F] _√nj + 3r_ 2m _._ (12)

X Y

Here, Ltr is the training loss. κ is a coefficient depending on the properties of dataset.Q _m is the_
number of training samples. Bj[F] [is the upper bound of the Frobenius norm of weight parameter of]
layer j of both pre-trained and fine-tuned model. Dj[F] [is the upper bound of the Frobenius norm of]
the difference between weight parameter of layer j of pre-trained and fine-tuned model. nj is the
number of columns in weight parameter of layer j. According to the bound, the generalization gap
between Ltr and Lte decreases if Dj[F] [’s of all the layers can be decreased. However, minimizing the]
weights between pre-trained and fine-tuned model for all layers might be cumbersome. Hence, we
choose to minimize the Frobenius norm of difference in features for our regularization term Lreg.

M LIMITATIONS OF OUR FRAMEWORK

Although our framework produces competitive performance on the CDFSL benchmark, it has the
following limitations: (a) In online setting, when target domain samples arrive in a streaming


-----

fashion, our method might not be applicable. This is mainly because of the presence of the mask
generator. Even though the mask generator is a small network, it still requires a small batch of
samples for learning the parameters. In the online setting, samples arrive one at a time, and the small
masking network might overfit. A workaround to prevent overfitting can be selectively updating
only certain parameters during online learning. (b) Another limitation of our framework is the
use of large number of hyperparameters in the adaptation step for weighing the loss functions i.e.
_λpos, λneg, λdiv and λreg. For practical few-shot adaptation, it is difficult to set aside sufficient_
number of validation samples to tune the optimal hyperparameter configuration. Hence, we just
fixed the hyperparameter values in our experiments. Another possible workaround involves learning
hyperparameters themselves within the framework of multi-task uncertainty (Kendall et al., 2018).


-----

