# AUTO-TRANSFER: LEARNING TO ROUTE TRANSFER## ABLE REPRESENTATIONS

**Keerthiram Murugesan[1][∗]** **Vijay Sadashivaiah[2][∗]** **Ronny Luss[1]**

**Karthikeyan Shanmugam[1]** **Pin-Yu Chen[1]** **Amit Dhurandhar[1]**

1IBM Research, Yorktown Heights 2Rensselaer Polytechnic Institute, New york

keerthiram.murugesan@ibm.com sadasv2@rpi.edu
rluss@us.ibm.com karthikeyan.shanmugam2@ibm.com
pin-yu.chen@ibm.com adhuran@us.ibm.com

ABSTRACT

Knowledge transfer between heterogeneous source and target networks and tasks
has received a lot of attention in recent times as large amounts of quality labelled
data can be difficult to obtain in many applications. Existing approaches typically
constrain the target deep neural network (DNN) feature representations to be close
to the source DNNs feature representations, which can be limiting. We, in this
paper, propose a novel adversarial multi-armed bandit approach which automatically learns to route source representations to appropriate target representations
following which they are combined in meaningful ways to produce accurate target
models. We see upwards of 5% accuracy improvements compared with the stateof-the-art knowledge transfer methods on four benchmark (target) image datasets
CUB200, Stanford Dogs, MIT67 and Stanford40 where the source dataset is ImageNet. We qualitatively analyze the goodness of our transfer scheme by showing
individual examples of the important features our target network focuses on in
different layers compared with the (closest) competitors. We also observe that our
improvement over other methods is higher for smaller target datasets making it an
effective tool for small data applications that may benefit from transfer learning.[1]

1 INTRODUCTION

Deep learning models have become increasingly good at learning from large amounts of labeled data.
However, it is often difficult and expensive to collect sufficient a amount of labeled data for training a
deep neural network (DNN). In such scenarios, transfer learning (Pan & Yang, 2009) has emerged as
one of the promising learning paradigms that have demonstrated impressive gains in several domains
such as vision, natural language, speech, etc., and tasks such as image classification (Sun et al., 2017;
Mahajan et al., 2018), object detection (Girshick, 2015; Ren et al., 2015), segmentation (Long et al.,
2015; He et al., 2017), question answering (Min et al., 2017; Chung et al., 2017), and machine
translation (Zoph et al., 2016; Wang et al., 2018). Transfer learning utilizes the knowledge from
information-rich source tasks to learn a specific (often information-poor) target task.

There are several ways to transfer knowledge from source task to target task (Pan & Yang, 2009),
but the most widely used approach is fine-tuning (Sharif Razavian et al., 2014) where the target
DNN being trained is initialized with the weights/representations of a source (often large) DNN
(e.g. ResNet (He et al., 2016)) that has been pre-trained on a large dataset (e.g. ImageNet (Deng
et al., 2009)). In spite of its popularity, fine-tuning may not be ideal when the source and target
tasks/networks are heterogeneous i.e. differing feature spaces or distributions (Ryu et al., 2020;
Tsai et al., 2020). Additionally, the pretrained source network can get overwritten/forgotten which
prevents its usage for multiple target tasks simultaneously. Among the myriad of other transfer techniques, the most popular approach involves matching the features of the output (or gradient of the
output) of the target model to that of the source model (Jang et al., 2019; Li et al., 2018; Zagoruyko
& Komodakis, 2016). In addition to the output features, a few methods attempt to match the features
of intermediate states between the source and target models. Here, in this paper, we focus on the
latter by guiding the target model with the intermediate source knowledge representations.

_∗Equal contribution, ordered alphabetically._
[1Code available at https://github.com/IBM/auto-transfer](https://github.com/IBM/auto-transfer)


-----

Figure 1: Illustration of our proposed approach. During training, an input image is first forward passed through
the source network (such as ResNet34 trained on ImageNet) and the internal feature representations are saved.
An adversarial multi-armed bandit (AMAB), for each layer of the target network (such as ResNet18), selects
the useful source features (if any) to receive knowledge. Feature representations are then combined and fed into
the next layer. In this example the following (target, source) pairs are selected: (1,2), (2,1), (3,3), (4, None).
Parameters for AMAB and combination modules are optimized over training data. At test time, given an input
image, representations mapping best feature representation between source-target layers, based on our method,
are combined for the target network to make a decision.

While common approaches allow knowledge transfer between heterogeneous tasks/networks, it is
also important to recognize that constraining the target DNN representations to be close to certain
source DNN representations may be sub-optimal. For example, a source model, trained to classify
cats vs dogs may be accessed at different levels to provide internal representations of tiger or wolf
images to guide the target task in classifying tigers vs wolves. Since the source model is trained
with a large number of parameters and labeled examples of cats and dogs, it will have learned
several patterns that distinguish cat images from dog images. It is postulated that concepts or representations such as the shape of the tail, eyes, mouth, whiskers, fur, etc. are useful to differentiate
them (Neyshabur et al., 2020), and it is further possible to reuse these learned patterns to generalize
to new (related) tasks by accessing representations at the appropriate level. This example raises
three important questions related to knowledge transfer between the source-target models: 1) What
knowledge to transfer? 2) Where to transfer? 3) How to transfer the source knowledge?

While the what and where have been considered in prior literature (Rosenbaum et al., 2018; Jang
et al., 2019), our work takes a novel and principled approach to the questions of what, where and
_how to transfer knowledge in the transfer learning paradigm. Specifically, and perhaps most impor-_
tantly, we address the question of how to transfer knowledge, going beyond the standard matching
techniques, and take the perspective that it might be best to let the target network decide what source
knowledge is useful rather than overwriting one’s knowledge to match the source representations.
Figure 1 illustrates our approach to knowledge transfer where the question of what and where is
addressed by an adversarial multi-armed bandit (routing function) and the how is addressed by an
aggregation operation detailed later. In building towards these goals, we make the following contributions:

_• We propose a transfer learning method that takes a novel and principled approach to automatically_
decide which source layers (if any) to receive knowledge from. To achieve this, we propose an
adversarial multi-armed bandit (AMAB) to learn the parameters of our routing function.

_• We propose to meaningfully combine feature representations received from the source network_
with the target network-generated feature representations. Among various aggregation operations
that are considered, AMAB also plays a role in selecting the best one. This is in contrast with
existing methods that force the target representation to be similar to source representation.

_• Benefits of the proposed method are demonstrated on multiple datasets. Significant improvements_
are observed over seven existing benchmark transfer learning methods, particularly when the
target dataset is small. For example, in our experiment on ImageNet-based transfer learning on
the target Stanford 40 Actions dataset, our auto-transfer learning method achieved more than 15%
improvement in accuracy over the best competitor.


-----

2 RELATED WORK

Transfer learning from a pretrained source model is a well-known approach to handle target tasks
with a limited label setup. A key aspect of our work is that we seek to transfer knowledge between
heterogeneous DNNs and tasks. Recent work focused on feature and network weight matching to
address this problem where the target network is constrained to be near the source network weights
and/or feature maps. Network matching based on L[2]−SP regularization penalizes the ℓ2 distance of
the pretrained source network weights and weights of the target networks to restrict the search space
of the target model and thereby hinder the generalization (Xuhong et al., 2018). Recent work (Li
et al., 2018) has shown that it is better to regularize feature maps of the outer layers than the network
weights and reweighting the important feature via attention. Furthermore, attention-based feature
distillation and selection (AFDS) matches the features of the output of the convolutional layers
between the source-target models and prunes the unimportant features for computational efficiency.
Similar matching can also be applied to match the Jacobians (change in output with respect to input
rather than matching the output) between source and target networks (Srinivas & Fleuret, 2018).
Previous works (Dhurandhar et al., 2018; 2020) also suggested that rather than matching the output
of a complex model, it could also be used to weight training examples of a smaller model.

Learning without forgetting (LwF) (Li & Hoiem, 2017) leverages the concept of distillation (Hinton
et al., 2015) and takes it further by introducing the concept of stacking additional layers to the
source network, retraining the new layers on the target task, and thus adapting to different source
and target tasks. SpotTune (Guo et al., 2019) introduced an adaptive fine-tuning mechanism, where
a policy network decides which parts of a network to freeze vs fine-tune. FitNet (Romero et al.,
2014) introduced an alternative to fine-tuning, where the internal feature representations of teacher
networks were used as a guide to training the student network by using ℓ2 matching loss between the
two feature maps. Attention Transfer (AT) (Zagoruyko & Komodakis, 2016) used a similar approach
to FitNet, except the matching loss was based on attention maps. The most relevant comparison to
our work is that of Learning to Transfer (L2T-ww) (Jang et al., 2019), which matches source and
target feature maps but uses a meta-learning based approach to learn weights for useful pairs of
source-target layers for feature transfer. Unlike L2T-ww, our method uses a very different principled
approach to combine the feature maps in a meaningful way (instead of feature matching) and let the
target network decide what source knowledge is useful rather than overwriting one’s knowledge to
match the source representations. Finally, Ji et al. (2021) uses knowledge distillation based approach
to transfer knowledge between source and target networks.

3 AUTO-TRANSFER METHOD

In this section, we describe our main algorithm for Auto-Transfer learning and explain in detail the
adversarial bandit approach that dynamically chooses the best way to combine source and target
representations in an online manner when the training of the target proceeds.

_What is the best way to train a target network such that it leverages pre-trained source representa-_
_tions speeding up training on the target task in terms of sample and time efficiency? We propose a_
routing framework to answer this: At every target layer, we propose to route one of the source representations from different layers and combine it with a trainable operation (e.g. a weighted addition)
such that the composite function can be trained together (see Figure 10 for an example of combined
representations). We propose to use a bandit algorithm to make the routing/combination choices in
an online manner, i.e. which source layer’s representation to route to a given target layer and how
to combine, while the training of the target network proceeds. The bandit algorithm intervenes once
every epoch of training to make choices using rewards from evaluation of the combined network on
a hold out set, while the latest choice made by the bandit is used by the training algorithm to update
the target network parameters on the target task. We empirically show the benefit of this approach
with other baselines on standard benchmarks. We now describe this framework of source-target
representation transfer along with the online algorithm.

3.1 ROUTING REPRESENTATIONS

For a given image x, let {fS[1][(][x][)][, f]S[ 2][(][x][)][,][ · · ·][, f]S[ N] [(][x][)][}][ and][ {][f][ 1]T [(][x][)][, f]T[ 2] [(][x][)][,][ · · ·][, f][ M]T [(][x][)][}][ be the inter-]
mediate feature representations for image x from the source and the target networks, respectively.


-----

Let us assume the networks have trainable parametersdt are the total number of trainable parameters of the networks. Clearly, the representations are a WS ∈ R[d][s] and WT ∈ R[d][t] where ds and
function of the trainable parameters of the respective networks. We assume that the source network
is pre-trained. These representations could be the output of the convolutional or residual blocks of
the source and target networks.

_Our Key Technique: For the i-th target representation fT[i]_ [, our proposed method a) maps][ i][ to one]
of the N intermediate source representations, fS[j] [, or][ NULL][ (zero valued) representation; b) uses]
_Tj, a trainable transformation of the representation fS[j]_ [, to get][ ˜]fS[j] [, i.e.][ ˜]fS[j] [(][x][) =][ T][j][(][f][ j]S[(][x][))][; and c)]
combines transformed source _f[˜]S[j]_ [and the target representations][ f][ i]T [using another trainable operation]
chosen from a set of operations M. Let Wi,jL [be the set of trainable parameters associated with]

the operator chosen. We describe the various possible operations below. The target network uses the
combined representation in place of the originalL _i-th target representation:_

_f˜T[i]_ [(][x][) =][ T][j][(][f][ j]S[(][x][))] _fT[i]_ [(][x][)] (1)
M

In the above equation, the trainable parameters of the operator depend on the i and j (that dependence is hidden for convenience in notation). The set of choices are discrete, that is, P =
_{[N_ ] ∪ NULL} × M where [N ] denotes set of N source representations. Each choice has a set of
trainable parameters Tj, Wi,jL [in addition to the trainable parameters][ W][T][ of the target network.]

3.2 LEARNING THE CHOICE THROUGH ADVERSARIAL BANDITS

To pick the source-target mapping and the operator choice, we propose an adversarial bandit-based
online routing function (Auer et al., 2002) that picks one of the choices (with its own trainable
parameters) containing information on what, where and how to transfer to the target representation
_i. Briefly, adversarial bandits choose actions at from a discrete choice of actions at time t, and the_
environment presents an adversarial reward rt(at) for that choice. The bandit algorithm minimizes
the regret with respect to the best action a[∗] in hindsight. In our non-stationary problem setting,
the knowledge transfer from the source model changes the best action (and the reward function) at
every round as the target network adapts to this additional knowledge. This is the key reason to use
adversarial bandits for making choices as it is agnostic to an action dependent adversary.

_Bandit Update: We provide our main update Algorithm 1 for a given target representation i from_
layer (ℓ). At each round t, the update algorithm maintains a probability vector πt over a set of all
possible actions from routing choice space P. The algorithm chooses a routing choice at = (jt →
_ℓ,_ ) randomly drawn according to the probability vector πt (in Line 7). Here jt is the selected
source representation to be transfered to the target layer l and combined with target representation i
using the operator .

[L][t]

_Reward function: The reward rt for the selected routing choice is then computed by evaluating_
gain in the loss due to the chosen source-target combination as follows: the prediction gain is the[L][t]
difference between the target network’s losses on a hold out set Dv with and without the routing
choice at i.e., L(fT[M] [(][x][))][ −L][( ˜]fT[M] [(][x][))][ for a given image][ x][ from the hold out data. This is shown]
in the Algorithm 3 EVALUATE. The reward function is used in Lines 4 and 5 to update the
probability vector πp,t almost identical to the update in the classical EXP3.P algorithm of (Auer
et al., 2002). Note that if the current version of the trainable parameters is not available, then a
random initialization is used. In our experiments, this reward value is mapped to the [−1, 1] range
to feed as a reward to the bandit update algorithm.

_Environment Update: Given the choice j →_ _i and the operator_, the target network is trained
for one epoch over all samples in the training data DT for the target task. Algorithm 2 TRAINTARGET updates the target network weights WT and other trainable parameters[L] (Wi,jL[, T][j][)][ of the]

routing choice at for each epoch on the entire target training dataset. Our main goal is to train the
best target network that can effectively combine the best source representation chosen. Here, L is
the loss function which operates on the final representation layer of the target network. αt = 1/t
and β is the exploration parameter. We set β = 0.4 and γ = 10[−][3].


-----

**Algorithm 1 AMAB - Update Algorithm for Target Layer ℓ**

1: Inputs: Learning rate αt, Exploration parameter β, Number of Epochs E. Routing choice set
_P_

2: forInitialize: t ∈ [1 : w E0],p do, ˜r0,p ← 0.
3: **for p** **do**
_∈P_ _αt_

4:5: _wt,p ←_ log (1 − _αt) exp_ _wt−1,p + γr˜t−1,p_ + _K−1_ _j≠_ _p_ [exp] _wt−1,j + γr˜t−1,j_
  P  


_e[w][t,p]_
_πt,p ←_ (1 − _β)_ _Kj=1_ _[e][w][t,j][ +][ β]K_ (2)

6: **end for** P

7: Choose action at _πt. Let at = (jt_ _ℓ,_ ).

_t_

_∼_ _→_

8: Obtain current version of trainable parameters: _T, Tjt_ _,_ _i,j_ . Use the standard random
_W_ _W_ L

initialization if not initialized. _t[L][t]_  

9: _rt,at_ EVALUATE(at, _T, Tjt_ _,_ _i,j_ )
_←_ _t_ _W_ _W_ L _t_

10: _T, Tjt_ _,_ _i,j_ TRAIN -TARGET(at, _T, Tjt_ _,_ _i,j_ )

_W_ _W_ L _←_ _W_ _W_ L

_rt,p_

11: r˜t,p _πt,p_ if p = at,  
_←_ (0 otherwise

12: end for

**Algorithm 2 TRAIN-TARGET - Train Target Network**

1: Inputs: Target training dataset DT, Target loss L(·). Routing choice: (j → _i,_ ). Seed weight
parameters: WT [0], Tj[0], Wi,jL[[0]][.]

2: Randomly shuffle DT . [L]
3:4: forx k ← ∈D[1 :T [ |k]D. _T |] do_

5:6: end forWT [k], Tj[k], Wi,jL[[][k][]] _←_ WT [−k −ηk∇1][], TWjT[,Tk −j _,W1]i,jL, WLi,jL( ˜f[[]T[M][k][ −][(][x][1]][))]_

7: Output: Last iterate of _T, Tj,_ _i,j_
W _W_ L


**Algorithm 3 EVALUATE - Evaluate Target Network**

1: Inputs: Routing Choice: (j → _i,_ ). Weight parameters: WT, Tj, Wi,jL[. Target Loss][ L][()][.]

Target task hold out set Dv.

2: Output: _D1v_ (fT[M] [(][x][))][ −L][( ˜][L]fT[M] [(][x][))][.]

_|_ _|_ _x∈Dv_ _L_
P

3.3 ROUTING CHOICES


The routing choice (j → _i,_ _i,j[)][ can be seen as deciding][ where, what and how][ to transfer/combine]_

the source representations with the target network.

_Where to transfer? The routing function[L]_ _j →_ _i decides which one of the N intermediate source_
features is useful for a given target feature fT[i] [. In addition to these combinations, we allow the]
routing function to ignore the transfer using the NULL option. This allows the target network to
potentially discard the source knowledge if it’s unrelated to the target task.

_What to transfer? Once a pair of source-task (j →_ _i) combination is selected, the routing func-_
tion decides what relevant information from the source feature fS[j] [should be transferred to the target]


-----

network using the transformation Tj. We use a Convolution-BatchNorm block to transfer useful features to the target network _f[˜]S[j]_ [= BN(Conv(][f][ j]S[))][. Here,][ T][j][ = BN(Conv(][·][))][. The convolution layer]
can select for relevant channels from the source representation and the batch normalization (Ioffe &
Szegedy, 2015) addresses the covariant-shift between the source and the target representations, we
believe that this combination is sufficient to ”match” the two representations. This step also ensures
that the source feature has a similar shape to that of the target feature.

_How to transfer (i.e. combine the representations)? Given a pair of source and target feature repre-_
sentations (j → _i), the routing function chooses one of the following operations (i.e._ ) to combine
them. We describe the class of operations M, i.e. the various ways (1) is implemented.

[L]

1. Identity (Iden) operation allows the target network just to use the target representation fT[i] [after]
looking at the processed source representation _f[˜]S[j]_ [from the previous Conv-BN step.]

2. Simple Addition (sAdd) adds the source and target features: _f[˜]T[i]_ [= ˜]fS[j] [+][ f][ i]T [.]

3. Weighted Addition (wAdd) modifies sAdd with weights for the source and target features. These
weights constitute Wi,jL[. i.e. the trainable parameters of this operation choice:][ ˜]fT[i] [=][ w][S,i,j][ ∗]

_f˜S[j]_ [+][ w][T,i,j][ ∗] _[f][ i]T_ [.]

4. Linear Combination (LinComb) uses the linear block (without bias term) along with the average
pooling to weight the features: fT[i] [= Lin][S,i,j][( ˜]fS[j] [)][ ∗] _f[˜]S[j]_ [+ Lin][T,i,j][(][f][ i]T [)][ ∗] _[f][ i]T_ [where][ Lin][·][,i,j][ is a]
linear transformation with its own trainable parameters.

5. Feature Matching (FM) follows the earlier work and forces the target feature to be similar to the
source feature. This operation adds a regularization term wi,j _fS[j]_ _T_
when we train. _∥_ [˜] _[−]_ _[f][ i]_ _[∥]_ [to the target objective][ L]

6. Factorized Reduce (FactRed) use two convolution modules to reduce the number of
channels c in the source and target features to c/2 and concat them together: _fT[i]_ =
concat(Conv[c/]S,i,j[2] [( ˜]fS[j] [)][,][ Conv][c/]T,i,j[2] [(][f][ i]T [))][.]

An action a from the search space is given by [(j → _i),_ _i,j[]][. The total number of choice combi-]_
nations is O((N + 1)M ). Typically N and M are very small numbers, for instance, when Resnet
is used as a source and target networks, we have N = 4[L], M = 5. For large action search spaces,
action pruning (Even-Dar et al., 2006) and greedy approaches (Bayati et al., 2020) can be used to
efficiently learn the best combinations as demonstrated in our experiment section.

4 EXPERIMENTS

In this section, we present experimental results to validate our Auto-Transfer methods. We first show
the improvements in model accuracy that can be achieved over various baselines on six different
datasets (section A.3) and two network/task setups. We then demonstrate superiority in limited
sample size and limited training time usecases. Finally, we use visual explanations to offer insight
as to why performance is improved using our transfer method. Experimental results on a toy example
can be found in the supplement section A.1.

4.1 EXPERIMENTAL SETUP

Our transfer learning method is compared against existing baselines on two network/task setups. In
the first setup, we transfer between similar architectures of different complexities; we use a 34-layer
ResNet (He et al., 2016) as the source network pre-trained on ImageNet and an 18-layer ResNet
as the target network. In the second setup, we transfer between two very different architectures;
we use an 32-layer ResNet as the source network pretrained on TinyImageNet and a 9-layer VGG
(Simonyan & Zisserman, 2014) as the target network. For ImageNet based transfer, we apply our
method to four target tasks: Caltech-UCSD Bird 200 (Wah et al., 2011), MIT Indoor Scene Recognition (Quattoni & Torralba, 2009), Stanford 40 Actions (Yao et al., 2011) and Stanford Dogs (Khosla
et al., 2011). For TinyImageNet based transfer, we apply our method on two target tasks: CIFAR100
(Krizhevsky et al., 2009), STL-10 (Coates et al., 2011).

We investigate different configurations of transfer between source and target networks. In the full
configuration, an adverserial multi-armed bandit (AMAB) based on Exponential-weight algorithm


-----

for Exploration and Exploitation (EXP3) selects (source, target) layer pairs as well as one of one
of five aggregation operations to apply to each pair (operations are independently selected for each
pair). In the route configuration, the AMAB selects layer pairs but the aggregation operation is fixed
to be weighted addition. In the fixed configuration, transfer is done between manually selected pairs
of source and target layers. Transfer can go between any layers, but the key is that the pairs are
manually selected. In each case, during training, the source network is passive and only shares the
intermediate feature representation of input images hooked after each residual block. After pairs are
decided, the target network does aggregation of each pair of source-target representation in feedforward fashion. The weight parameters of aggregation are trained to act as a proxy to how much
source representation is useful for the target network/task. For aggregating features of different
spatial sizes, we simply use a bilinear interpolation.

4.2 EXPERIMENTS ON TRANSFER BETWEEN SIMILAR AND DIFFERENT ARCHITECTURES

In the first setup, we evaluate all three Auto-Transfer configurations, full, fixed, and route, on various
visual classification tasks, where transfer is from a Resenet-34 model to a Resnet-18 model. Our
findings are compared with an independently trained Resnet-18 model (Scratch), another Resnet-18
model tuned for ImageNet and finetuned to respective tasks (Finetune), and the following existing baselines: Learning without forgetting (LwF) (Li & Hoiem, 2017), Attention Transfer (AT)
(Zagoruyko & Komodakis, 2016), Feature Matching (FM) (Romero et al., 2014), Learning What
and Where to Transfer (L2T-ww) (Jang et al., 2019) and Show, Attend and Distill (SAaD) (Ji et al.,
2021). Results are shown in Table 6. Each experiment is repeated 3 times.

First, note that the Auto-Transfer Fixed configuration already improves performance on (almost) all
tasks as compared to existing benchmarks. The fixed approach lets the target model decide how
much source information is relevant when aggregating the representations. This result supports our
approach to feature combination and demonstrates that it is more effective than feature matching.
This even applies to the benchmark methods that go beyond and learn where to transfer to. Next,
note that the Auto-Transfer Route configuration further improves the performance over the one-toone configuration across all tasks. For example, on the Stanford40 dataset, Auto-Transfer Route
improves accuracy over the second best baseline by more than 15%. Instead of manually choosing
source and target layer pairs, we automatically learn the best pairs through our AMAB setup (Table
5 shows example set of layers chosen by AMAB). This result suggests that learning the best pairs
through our AMAB setup to pick source-target pairs is a useful strategy over manual selection
as done in the one-to-one configuration. To further justify the use of AMAB in our training, we
conducted an ablation experiment (section A.6) where we retrain Auto-Transfer (fixed) with bandit
chosen layer pairs, and found that the results were sub-optimal.

Next, note that Auto-Transfer Full, which allows all aggregation operations, does well but does
not outperform Auto-Transfer Route. Indeed, the Auto-Transfer Full results showed that selected
operations were all leaning to weighted addition, but other operations were still used as well. We
conjecture that weighted addition is best for aggregation, but the additional operations allowed in
Auto-Transfer Full introduce noise and make it harder to learn the best transfer procedure. Additionally, we conducted experiments by fixing aggregation to each of 5 operations and running
Auto-Transfer Route and found that weighted addition gave best performance Table 8.

In order to demonstrate that our transfer method does not rely on the source and target networks
being similar architectures, we proceed to transfer knowledge from a Resnet-32 model to a VGG-9
model. Indeed, Table 6 in the appendix demonstrates that Auto-Transfer significantly improves over
other baselines for CIFAR100 and STL-10 datasets. Finally, we conducted experiments on matched
configurations, where both Auto-Transfer (Route) and FineTune used same sized source and target
models and found that Auto-Transfer outperforms FineTune (Figure 7 and Table 3).

4.3 EXPERIMENTS ON LIMITED AMOUNTS OF TRAINING SAMPLES

Transfer learning emerged as an effective method due to performance improvements on tasks with
limited labelled training data. To evaluate our Auto-Transfer method in such data constrained scenario, we train our Auto-Transfer Route method on all datasets by limiting the number of training
samples. We vary the samples per class from 10% to 100% at 10% intervals. At 100%, Stanford40
has ∼100 images per class. We compare the performance of our model against Scratch and L2T-ww


-----

Table 1: Transfer between Resnet models: Classification accuracy (%) of transfer learning from ImageNet
(224 × 224) to Caltech-UCSD Bird 200 (CUB200), Stanford Dogs datasets, MIT Indoor Scene Recognition
(MIT67) and Stanford 40 Actions (Stanford40). ResNet34 and ResNet18 are used as source and target networks
respectively. Best results are bolded and each experiment is repeated 3 times. *DNR: did not report

Source task ImageNet

Target task CUB200 Stanford Dogs MIT67 Stanford40

Scratch 39.11±0.52 57.87±0.64 48.30±1.01 37.42±0.55
Finetune 41.38±2.96 54.76±3.56 48.50±1.42 37.15±3.26
LwF 45.52±0.66 66.33±0.45 53.73±2.14 39.73±1.63
AT 57.74±1.17 69.70±0.08 59.18±1.57 59.29±0.91
LwF+AT 58.90±1.32 72.67±0.26 61.42±1.68 60.20±1.34
FM 48.93±0.40 67.26±0.88 54.88±1.24 44.50±0.96
L2T-ww 65.05±1.19 78.08±0.96 64.85±2.75 63.08±0.88
SAaD 68.29±DNR 76.06±DNR 66.47±DNR 67.92±DNR

Auto-Transfer

-  full 67.86±0.70 84.07±0.42 74.79±0.60 77.40±0.74

-  fixed 64.86±0.06 86.10±0.08 69.44±0.41 77.27±0.32

-  route **74.76±0.39** **86.16±0.24** **75.86±1.01** **80.10±0.58**

for Stanford40 and report results in Figure 2 (top). Auto-Transfer Route significantly improves the
performance over existing baselines. For example, at 60% training set (∼60 images per class), our
method achieves 77.90% whereas Scratch and L2T-ww achieve 29% and 46%, respectively. To put
this in perspective, Auto-Transfer Route requires only 10% images per class to achieve better accuracy than achieved by L2T-ww with 100% of the images. We see similar performance with other
three datasets: CUB200, MIT67, Stanford Dogs (Figure 9).

4.4 IMPROVEMENTS IN TRAINING & INFERENCE TIMES


In order to assess training metrics and stability of learning, we visualize the test accuracy over training steps in
Figure 2 (bottom) for the Stanford40 dataset. The results
show that our method learns significantly quicker relative
to the second closest baseline. For example, at epoch 25,
our method achieves 74.55% accuracy whereas L2T-ww
and Scratch achieve 25.55% and 21.69%, respectively. In
terms of training time, Auto-Transfer Route took ∼300
minutes to train 200 epochs on the Stanford40 dataset,
whereas L2T-ww and Scratch models took 610 and 170
minutes, respectively. Taken together, our method significantly improves performance over the second baseline with less than half the runtime. We report additional
experiments with training curves plotted against training
time in appendix (Figure 7) and inference times plotted
against test accuracy (Figure 8). In Table 4 we show
that for inference time matched models, Auto-Transfer
(Route) outperforms FineTune by significant margin. Figure 2: Above we see test accuracies as

a function of (target) training sample size
(top) and number of epochs (bottom) on the

4.5 VISUAL EXPLANATIONS Stanford40 dataset for the Scratch model,

L2T-ww (our closest competitor) and our

In order to qualitatively analyze what bandit Auto- method Auto-Transfer. Qualitatively similar
Transfer Route is learning, Grad-CAM (Selvaraju et al., behavior is also seen on the other datasets.
2017) based visual explanations are presented in Figure 3 Experiments repeated 3 times.
(additional explanations are in Figures 11, 12, and 13 in the appendix). Grad-CAM highlights pixels
that played an important role in correctly labelling the input image. For each target task, we present
a (random) example image that is correctly labelled by bandit Auto-Transfer but incorrectly classi

-----

Figure 3: Layer-wise Grad-CAM images highlighting important pixels that correspond to predicted output
class. We show examples from MIT67 and CUB200 (ImageNet based transfer) where the independently trained
scratch model predicted the input image incorrectly, but our bandit based auto-transfer method predicted the
right class for that image. Correctly predicted class is indicated in green text and incorrectly classified class is
indicated in red text. Class probability for these predictions is also provided.

fied by Scratch, along with layer-wise Grad-CAM images that illustrate what each layer of the target
model focuses on. For each image, we report the incorrect label, correct label and class probability
for correct (pc) and incorrect (pi) labels.

Overall, we observe that our method pays attention to relevant visual features in making correct decisions. For example, in the first image from MIT67 dataset, the Scratch model incorrectly labelled
it as a gameroom while the correct class is bedroom (pi = 0.67, pc = 0.007). The Grad-CAM
explanations show that layers 1-3 of the Scratch model pay attention to the green floor which is
atypical to a bedroom and common in gamerooms (e.g. pool tables are typically green). The last
layer focuses on the surface below the window that looks like a monitor/tv that is typically found in
gamerooms. On the other hand, our model correctly identifies the class as bedroom (pc = 0.57) by
paying attention to the bed and surrounding area at each layer.

To visualize an example from a harder task, consider the indigo bunting image from the CUBS
dataset. The Scratch model classifies the image as a bluejay (pi = 0.85, pc = 0.09), but our model
correctly predicts it as a bunting (pc = 0.99). Indigo buntings and blue jays are strikingly similar, but
blue jays have white faces and buntings have blue faces. We clearly see this attribute picked up by
the bandit Auto-Transfer model in layers 2 and 3. We hypothesize that the source model, trained on
millions of images, provides useful fine-grained information useful for classifying similar classes.

5 CONCLUSION

In this paper, we have put forth a novel perspective where we leverage and adapt an adversarial
multi-armed bandit approach to transfer knowledge across heterogeneous tasks and architectures.
Rather than constraining target representations to be close to the source, we dynamically route source
representations to appropriate target representations also combining them in novel and meaningful
ways. Our best combination strategy of weighted addition leads to significant improvement over
state-of-the-art approaches on four benchmark datasets. We also observe that we produce accurate
target models faster in terms of (training) sample size and number of epochs. Further visualization
based qualitative analysis reveals that our method produces robust target models that focus on salient
features of the input more so than its competitors, justifying our superior performance.


-----

ACKNOWLEDGMENT

We would like to thank Clemens Rosenbaum, Matthew Riemer, and Tim Klinger for their comments on an earlier version of this work. This work was supported by the Rensselaer-IBM
[AI Research Collaboration (http://airc.rpi.edu), part of the IBM AI Horizons Network](http://airc.rpi.edu)
[(http://ibm.biz/AIHorizons).](http://ibm.biz/AIHorizons)

REFERENCES

Peter Auer, Nicol`o Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32:48–77, 2002.

Mohsen Bayati, Nima Hamidi, Ramesh Johari, and Khashayar Khosravi. Unreasonable effectiveness
of greedy algorithms in multi-armed bandit with many arms. Advances in Neural Information
_Processing Systems, 33, 2020._

Yu-An Chung, Hung-Yi Lee, and James Glass. Supervised and unsupervised transfer learning for
question answering. arXiv preprint arXiv:1711.05345, 2017.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised
feature learning. In Proceedings of the fourteenth international conference on artificial intelli_gence and statistics, pp. 215–223. JMLR Workshop and Conference Proceedings, 2011._

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248–255. Ieee, 2009.

Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, and Peder Olsen. Improving simple models with confidence profiles. Advances in neural information processing systems, 2018.

Amit Dhurandhar, Karthikeyan Shanmugam, and Ronny Luss. Enhancing simple models by exploiting what they already know. International Conference on Machine Learning, 2020.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and
stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of
_machine learning research, 7(6), 2006._

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision,
pp. 1440–1448, 2015.

Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.
Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Con_ference on Computer Vision and Pattern Recognition, pp. 4805–4814, 2019._

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.

Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Girshick. Mask r-cnn. In Proceedings of the
_IEEE international conference on computer vision, pp. 2961–2969, 2017._

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
_preprint arXiv:1503.02531, 2015._

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by
reducing internal covariate shift. In International conference on machine learning, pp. 448–456.
PMLR, 2015.

Yunhun Jang, Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Learning what and where to transfer.
In International Conference on Machine Learning, pp. 3030–3039. PMLR, 2019.

Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via
attention-based feature matching. In Proceedings of the AAAI Conference on Artificial Intelli_gence, volume 35, pp. 7945–7952, 2021._


-----

Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for finegrained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Grained Visual
_Categorization (FGVC), volume 2. Citeseer, 2011._

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.

Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep
learning transfer using feature map with attention for convolutional networks. In International
_Conference on Learning Representations, 2018._

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis
_and machine intelligence, 40(12):2935–2947, 2017._

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic
segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 3431–3440, 2015.

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li,
Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised
pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181–
196, 2018.

Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. Question answering through transfer learning
from large fine-grained supervision data. arXiv preprint arXiv:1702.02171, 2017.

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008.11687, 2020.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge
_and data engineering, 22(10):1345–1359, 2009._

Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on
_Computer Vision and Pattern Recognition, pp. 413–420. IEEE, 2009._

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object
detection with region proposal networks. Advances in neural information processing systems, 28:
91–99, 2015.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.

Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of
non-linear functions for multi-task learning. In International Conference on Learning Represen_tations, 2018._

Jeongun Ryu, Jaewoong Shin, Hae Beom Lee, and Sung Ju Hwang. Metaperturb: Transferable
regularizer for heterogeneous tasks and architectures. Advances in neural information processing
_systems, 2020._

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh,
and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618–626,
2017.

Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features offthe-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on
_computer vision and pattern recognition workshops, pp. 806–813, 2014._

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image
recognition. arXiv preprint arXiv:1409.1556, 2014.

Suraj Srinivas and Franc¸ois Fleuret. Knowledge transfer with jacobian matching. In International
_Conference on Machine Learning, pp. 4723–4731. PMLR, 2018._


-----

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on
_computer vision, pp. 843–852, 2017._

Yun-Yun Tsai, Pin-Yu Chen, and Tsung-Yi Ho. Transfer learning without knowing: Reprogramming
black-box machine learning models with scarce data and limited resources. In International
_Conference on Machine Learning, pp. 9614–9624. PMLR, 2020._

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd
birds-200-2011 dataset. 2011.

Yijun Wang, Yingce Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and Tie-Yan Liu. Dual transfer
learning for neural machine translation with marginal distribution regularization. In Proceedings
_of the AAAI Conference on Artificial Intelligence, volume 32, 2018._

LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning
with convolutional networks. In International Conference on Machine Learning, pp. 2825–2834.
PMLR, 2018.

Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. In 2011 International
_conference on computer vision, pp. 1331–1338. IEEE, 2011._

Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928,
2016.

Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource
neural machine translation. arXiv preprint arXiv:1604.02201, 2016.


-----

A APPENDIX

A.1 TOY EXAMPLE

In this section, we simulate our experiment on a toy example. We compare our Auto-Transfer with
the other baselines: L2T-ww and Scratch. In this simulation, we consider Auto-Transfer with a fixed
(one-to-one) setup for simplicity in our experiment analysis.

We consider predicting a sine wave function (y = sin(x)) as our source task and a sinc function
(y = _[sin]x[(][x][)]_ ) as our target task. Clearly, the features from the pretrained source model will help the

target task in predicting the sinc function. Both the input data point x and the output value y are
one-dimensional vectors (din = dout = 1). We use a shallow linear network consists of 4 linear
blocks: f1 = Lin(din,h1)(x), f2 = Lin(h1,h2)(f1), f3 = Lin(h1,h2)(f2), out = Lin(h3,dout)(f3)
for a datapoint x. For source network, we set the hidden size to 64 (i.e., h1 = h2 = h3 = 64) and
16 for the target network. We sampled 30, 000 data points to generate training set (x,y) and 10, 000
test-set data points for the source network and (i.e., x is sampled from a Gaussian distribution and
_y = sin(x)). Similarly, we generated 1000 training examples and 800 test set examples for the_
target network. Both the source and the target networks are trained for E = 50 epochs.

0.60.4 - Sine (with noise)-Source Prediction 0.4 - Sinc (with noise)- Auto-Transfer

- L2T-ww

- Scratch

0.2 0.2 - Source Prediction

0.0 0.0

0.2

0.2

0.4

0.4

0.6

7.5 5.0 2.5 0.0 2.5 5.0 7.5 7.5 5.0 2.5 0.0 2.5 5.0 7.5


Figure 4: (Left) shows the test set data from the source task and the source models’ prediction. (Right) shows
the test-set predictions for target task data from Scratch, Source prediction, L2T-ww and Auto-Transfer with
the shallow linear network configuration [din = 1, h1 = 16, h2 = 16, h3 = 16, dout = 1].

Figure 4 (left) shows the source model prediction for the test data. Given the shallow linear network
with 64 hidden dimensions and 30, 000 training example, the source model perfectly predicts the
_sin(x) function. Figure 4 (right) shows the predictions from the scratch target model, source model,_
L2T-ww and Auto-Transfer for the target test data. We report the Auto-Transfer with fixed choice of

[(0,0),(1,1),(2,2), wtAdd] for this experiment. We can see that the Auto-Transfer accurately predicts
the target task even when there is a limited amount of labeled examples.

Our results show the test set loss for the target data is relatively less compared to the other baselines
(0.0030 MSE loss for Auto-Transfer vs 0.0033 and 0.125 MSE loss for the scratch and L2T-ww).
Figures 5 and 6 show The results on different network configurations and how the feature representations for Scratch, L2T-ww and Auto-Transfer changes over 50 training epochs.

A.2 REAL DATASETS

We evaluate the performance of Auto-Transfer on six benchmarks with different tasks: Stanford
Actions 40 dataset for action recognition, CUBS Birds 200 dataset for object recognition, Stanford
Dogs 120 for fine-grained object recognition, MIT Indoors 67 for scene classification, CIFAR 100
and STL-10 image recognition datasets.

**Stanford Actions 40. Stanford Actions 40 dataset contains images of humans performing 40 ac-**
tions. There are about 180-300 images per class. We do not use bounding box and other annotation
information for training. There are a total of 9,532 images, making it the smallest dataset in our
benchmark experiments.


-----

0.4 0.4

0.2 0.2

0.0 0.0

0.2 0.2

0.4 0.4

7.5 5.0 2.5 0.0 2.5 5.0 7.5 7.5 5.0 2.5 0.0 2.5 5.0 7.5


Figure 5: Test-set predictions for Scratch, Source prediction, L2T-ww and Auto-Transfer for the target task
data with the shallow linear network configurations left: [din = 1, h1 = 4, h2 = 4, h3 = 4, dout = 1], right:

[din = 1, h1 = 8, h2 = 8, h3 = 8, dout = 1]

1.0

0.4 0.5

0.0

0.2

0.5

0.0 1.0

1.5 -Scratch

0.2 -L2T-ww

2.0 -Auto-Transfer

0.4 2.5

3 2 1 0 1 2

7.5 5.0 2.5 0.0 2.5 5.0 7.5


Figure 6: Test-set predictions for Scratch, Source prediction, L2T-ww and Auto-Transfer for the target task
data with different choices from the routing function left: [(2,0),(-1,1),(-1,2), wAdd] right: show the feature
representations of a single data point (plotted over the 50 training epochs) extracted from the final layer of the
target network

**Caltech-UCSD Birds-200-2011.** CUB-200-2011 is a bird classification datset with 200 bird
species. Each species is associated with a wikipedia article and organized by scientific classification. Each image is annotated with bounding box, part location, and attribute labels. We use only
classification labels during training. There are a total of 11,788 images.

**Stanford Dogs 120. The Stanford Dogs dataset contains images of 120 breeds of dogs from around**
the world. There are exactly 100 examples per category in the training set. It is used for the task of
fine-grained image categorization. We do not use the bounding box annotations. There are a total of
20,580 images.

**MIT Indoors 67. MIT Indoors 67 is a scene classification task containing 67 indoor scene cate-**
gories, each of which consists of at most 80 images for training and 20 for testing. Indoor scene
recognition is challenging because spatial properties, background information and object characters
are expected to be extracted. There are 15,620 images in total.

**CIFAR 100. CIFAR 100 is a image recognition task containing 100 different classes with 600**
images in each class. There are 500 training images and 100 testing images per class. It is a subset
of tiny images datastet.

**STL 10. STL 10 is a image recognition task containing 10 classes. It is inspired by the CIFAR-10**
dataset but with some modifications. In particular, each class has fewer labeled training examples
than in CIFAR-10.


-----

A.3 EXPERIMENT DETAILS

For our experimental analysis in the main paper, we set the number of epochs for training to
_E = 200. The learning rate for SGD is set to 0.1 with momentum 0.9 and weight decay 0.001._
The learning rate for the ADAM is set to 0.001 with and weight decay of 0.001. We use Cosine
Annealing learning rate scheduler for both optimizers. The batch size for training is set to 64. Our
target networks were randomly initialized before training.

The target models were trained in parallel on two machines with the specifications shown in Table
2.

**Resource** **Setting**

CPU Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Memory 128GB
GPUs 1 x NVIDIA Tesla V100 16 GB
Disk 600GB
OS Ubuntu 18.04-64 Minimal for VSI.


Table 2: Resources used by Auto-Transfer

A.4 TRAINING AND TESTING PERFORMANCE

Figure 7: Above we see test accuracies as a function of training time (minutes) plotted for following architectures (i) Finetuning (ResNet18 - ResNet18), (ii) AutoTransfer (ResNet18 - ResNet18), (iii) AutoTransfer
(ResNet34 - ResNet18), denoted FT(18-18), AT(18-18), and AT(34-18), respectively. We significantly outperform finetuning in all datasets.

A.5 ADDITIONAL EXPERIMENTS ON LIMITED AMOUNTS OF DATA

To evaluate our Auto-Transfer method in data constrained scenario further, we train our AutoTransfer (route) method on the CUB200, Stanford Dogs and MIT67 datasets by limiting the number
of training samples (Figure 9). We vary the samples per class from 10% to 100% at 10% intervals.


-----

Table 3: Classification accuracy (%) of transfer learning for matched architectures ResNet18 - ResNet18 for
Auto-Transfer and Finetuning. Best results are bolded.

CUB200 Stanford Dogs MIT67 Stanford40

Finetune (R18 - R18) 42.96±1.45 53.02±3.57 47.93±3.66 34.40±5.94
AutoTransfer (R18 - R18) **66.97±1.38** **79.46±1.05** **69.54±2.49** **75.07±2.55**

Table 4: Average classification accuracy (%) and average inference times of transfer learning for time matched
architectures using ResNet18 - ResNet18 for Auto-Transfer and ResNet34 - ResNet34 for Finetuning.

|Col1|CUB200 Stanford Dogs MIT67 Stanford40|
|---|---|
||t (sec) % t % t % t %|
|Finetuning (R34 - R34) Auto-Transfer (R18 - R18) Auto-Transfer (R34 - R18)|12.88 37.13 12.66 52.26 12.22 44.37 14.0 31.12 14.46 64.37 13.83 77.07 14.26 67.89 15.28 69.02 18.55 71.84 18.27 85.09 18.62 69.76 19.20 79.74|



Table 5: Final source layer selected at 200th epoch for each target layer for 3 repetitions for Table 1 experiments.

|Col1|Selected source layer (run 1, run 2, run 3)|
|---|---|
|Target Layer|Layer 1 Layer 2 Layer 3 Layer 4|
|CUB200 Stanford Dogs MIT67 Stanford40|2, 2, 2 3, 2, 2 2, 1, 1 2, 4, 4 1, 1, 4 3, 3, 5 2, 3, 2 4, 5, 4 2, 4, 2 3, 1, 5 2, 3, 1 3, 3, 4 1, 2, 4 4, 3, 3 2, 3, 2 3, 4, 3|



Table 6: Transfer between Resnet model and VGG model: Classification accuracy (%) of transfer learning
from TinyImageNet to CIFAR100 and VGG9. ResNet32 and VGG9 are used as source and target networks
respectively. Best results are bolded and each experiment is repeated 3 times.

Source task TinyImageNet

Target task CIFAR100 STL-10

Scratch 67.69±0.22 65.18±0.91
Finetune 67.80±1.76 65.98±1.25
LwF 69.23±0.09 68.64±0.58
AT 67.54±0.40 74.19±0.22
LwF+AT 68.75±0.09 75.06±0.57
FM 69.97±0.24 76.38±0.88
L2T-ww 70.96±0.61 76.38±1.18

Auto-Transfer

-  full **72.48±0.42** 78.46±1.10

-  fixed 70.48±0.25 79.92±1.49

-  route 70.89±0.36 **82.09±0.29**


-----

Figure 8: Test accuracies as a function of inference time plotted for following architectures (i) Finetuning
(ResNet34 - ResNet34), (ii) AutoTransfer (ResNet18 - ResNet18), (iii) AutoTransfer (ResNet34 - ResNet18),
denoted FT(34-34), AT(18-18), and AT(34-18), respectively. Each circle represents a batch of 128 sample
images. We significantly outperform finetuning in all datasets.

Figure 9: Above we see test accuracies as a function of (target) training sample size for CUB200, Stanford
Dogs and MIT67 datasets. Each experiment is repeated 3 times.

A.6 ABLATION STUDIES

TRAINING THE NETWORK USING BANDIT SELECTED PAIRS

To evaluate the importance of training the target network with adversarial multi-armed bandit, we
retrained our target network with a fixed source-layer configuration selected at 200th epoch of previous best bandit based experiments. For Eg. in our best bandit based experiment for CUB200, the
source,target pairs were {(2,1), (3,2), (2,3), (2,4)}. As seen in Table 7, we find that this experiment
decreased performance in comparison to bandit based one in all target tasks. This confirms the need
for bandit based decision maker, that learns combination weights and pairs over training steps.

Table 7: Classification accuracy (%) of transfer learning ResNet34 to ResNet18 transfer where the source-target
layer pairs are fixed to Auto-Transfer (route) selected ones at 200th epoch from previous runs.

Task CUB200 Stanford Dogs MIT67 Stanford40

Auto-Transfer (fixed, retrain) 73.09 85.05 69.10 78.90
Auto-Transfer (route) **75.15** **86.40** **76.87** **80.68**

TRAINING THE NETWORK USING DIFFERENT AGGREGATION OPERATORS

To evaluate how different aggregation operators influence Auto-Transfer, we train Auto-Transfer
Route by fixing aggregation to 5 different operations. Identity (iden), Simple Addition (sAdd),
Weighted Addition (wtAdd), Linear Combination (LinComb) and Factored Reduction (FactRed).
Results for Stanford40 dataset is found in Table 8. We find that weighted addition performs the best.


-----

Table 8: Classification accuracy (%) of transfer learning ResNet34 to ResNet18 transfer where the aggregation
operator is fixed to Identity (iden), Simple Addition (sAdd), Weighted Addition (wtAdd), Linear Combination
(LinComb) and Factored Reduction (FactRed).

Iden SAdd WtAdd LinComb FactRed

Auto-Transfer (route) 37.56 77.78 **80.10** 76.6 76.66

A.7 VISUALIZING INTERMEDIATE REPRESENTATIONS

Figure 10: Example of learned intermediate representations for a bird image from CUB200 dataset. We plot
the first 36 features in each layer ( there are 64, 128, 256 and 512 features for layers 1 to 4). It is hard to draw
meaningful patterns by looking at intermediate representations, and hence we chose to investigate layer-wise
Grad-CAM images.

A.8 ADDITIONAL EXPLANATIONS USING GRAD-CAM

We here offer more examples of visual explanations of what is being transferred using Auto-Transfer
Route. The first example in Figure 11 is an image of cooking from the Stanford40 dataset. The
Scratch model incorrectly classifies the image as cutting (pi = 0.88, pc = 0.01) by paying attention
to only the cooking surface that looks like a table and person sitting down (typical for someone
cutting vegetables). On the other hand, our model correctly labels the image (pc = 0.99) by paying
attention to the wok and cooking utensils such as water pot, etc. We hypothesize that this surrounding information is provided by the source model which is useful in making the correct decision.

The second example in Figure 11 is from the Stanford Dogs dataset (Figure 11). The scratch model
fails to pay attention to relevant class information (dog) and labels a chihuahua as german sheperd
(pi = 0.23, pc = 0.0002) by focusing on the flower, while our method picks the correct label
(pc = 0.99). Bandid Auto-Transfer gets knowledge about the flower early on and then disregards this
knowledge before attending to relevant class information. Further examples of visual explanations
comparing to L2T-ww (Figure 12) and counter-examples where our method identifies the wrong
label (Figure 13) follow below. For these counter-examples we find that the task is typically hard.
For eg. playing violin vs playing guitar. And, the class probability of incorrect label is closer to that
of correct label, suggesting that our method was not confident in predicting wrong class.


-----

Figure 11: Layer-wise Grad-CAM images highlighting important pixels that correspond to predicted output
class. We show examples from Stanford40 and Stanford Dogs (ImageNet based transfer) where the independently trained scratch model predicted the input image incorrectly, but our bandit based auto-transfer method
predicted the right class for that image. Correctly predicted class is indicated in green text and incorrectly
classified class is indicated in red text. Class probability for these predictions is also provided.


-----

Figure 12: Layer-wise Grad-CAM images highlighting important pixels that correspond to predicted output
class. We show examples where the L2T-ww model predicted the input image incorrectly, but our bandit based
auto-transfer method predicted the right class for that image. Correctly predicted class is indicated in green text
and incorrectly classified class is indicated in red text. Class probability for these predictions is also provided.


-----

Figure 13: Layer-wise Grad-CAM images highlighting important pixels that correspond to predicted output
class. We show examples where the L2T-ww model predicted the input image correctly, but our bandit based
auto-transfer method predicted the wrong class for that image. Correctly predicted class is indicated in green
text and incorrectly classified class is indicated in red text. Class probability for these predictions is also
provided.


-----

