# LEARNING VERSATILE NEURAL ARCHITECTURES BY PROPAGATING NETWORK CODES

**Mingyu Ding[1], Yuqi Huo[2], Haoyu Lu[2], Linjie Yang[3], Zhe Wang[4], Zhiwu Lu[2], Jingdong Wang[5], Ping Luo[1]**

1University of Hong Kong, 2Gaoling School of Artificial Intelligence, Renmin University of China,
3ByteDance Inc., 4SenseTime Research, 5Baidu
mingyuding@hku.hk wangjingdong@outlook.com pluo@cs.hku.hk

ABSTRACT

This work explores how to design a single neural network capable of adapting to
multiple heterogeneous vision tasks, such as image segmentation, 3D detection,
and video recognition. This goal is challenging because both network architecture
search (NAS) spaces and methods in different tasks are inconsistent. We solve this
challenge from both sides. We first introduce a unified design space for multiple
tasks and build a multitask NAS benchmark (NAS-Bench-MR) on many widely
used datasets, including ImageNet, Cityscapes, KITTI, and HMDB51. We further
propose Network Coding Propagation (NCP), which back-propagates gradients of
neural predictors to directly update architecture codes along the desired gradient
directions to solve various tasks. In this way, optimal architecture configurations
can be found by NCP in our large search space in seconds.
Unlike prior arts of NAS that typically focus on a single task, NCP has several
unique benefits. (1) NCP transforms architecture optimization from data-driven to
architecture-driven, enabling joint search an architecture among multitasks with
different data distributions. (2) NCP learns from network codes but not original
data, enabling it to update the architecture efficiently across datasets. (3) In addition to our NAS-Bench-MR, NCP performs well on other NAS benchmarks, such
as NAS-Bench-201. (4) Thorough studies of NCP on inter-, cross-, and intra-tasks
highlight the importance of cross-task neural architecture design, i.e., multitask
neural architectures and architecture transferring between different tasks. Code is
[available at github.com/dingmyu/NCP](https://github.com/dingmyu/NCP) [1].

1 INTRODUCTION

Designing a single neural network architecture that adapts to multiple different tasks is challenging.
This is because different tasks, such as image segmentation in Cityscapes (Cordts et al., 2016) and
video recognition in HMDB51 (Kuehne et al., 2011), have different data distributions and require
different granularity of feature representations. For example, although the manually designed networks ResNet (He et al., 2016) and HRNet (Wang et al., 2020a) work well on certain tasks such as
image classification on ImageNet, they deteriorate in the other tasks. Intuitively, manually designing
a single neural architecture that is applicable in all these tasks is difficult.

Recently, neural architecture search (NAS) has achieved great success in searching network architectures automatically. However, existing NAS methods (Wu et al., 2019; Liang et al., 2019; Liu
et al., 2019b; Xie et al., 2018; Cai et al., 2020; Yu et al., 2020; Shaw et al., 2019b;a) typically search
on a single task. Though works (Ding et al., 2021; Duan et al., 2021; Zamir et al., 2018) designed
NAS algorithms or datasets that can be used for multiple tasks, they still search different architectures for different tasks, indicating that the costly searching procedure needs to be repeated many
times. The problem of learning versatile neural architectures capable of adapting to multiple different tasks remains unsolved, i.e., searching a multitask architecture or transferring architectures
between different tasks. In principle, it faces the challenge of task and dataset inconsistency.

Different tasks may require different granularity of feature representations, e.g., the segmentation
task requires more multi-scale features and low-level representations than classification. The key
to solving task inconsistency is to design a unified architecture search space for multiple tasks. In
contrast to most previous works that simply extend search space designed for image classification

[1Project page: https://network-propagation.github.io](https://network-propagation.github.io)


-----

Cls Predictor Seg Predictor Cls Predictor Gradient Seg Predictor

Acc FLOPs mIoU FLOPs Acc FLOPs mIoU FLOPs

Cls Network 0.25

Cluster

0.5

0.75

Seg Network

Cluster

Gradient Seg Predictor

Acc FLOPs mIoU FLOPs

Cls Network 0.25

Cluster

0.5

0.75

Forward Backward Seg Network

Cluster

Cls Predictor Seg Predictor

Acc FLOPs mIoU FLOPs

**(a) Structure adaptation across tasks** **(b) Joint propagation for multiple tasks**

Figure 1: NCP optimizes and propagates the network code in an architecture coding space to achieve
the target constraints with back-propagation on the neural predictors. (a) NCP searches for an optimal structure on classification, then adapts it to segmentation through the segmentation predictor.
(b) joint propagation for two tasks by accumulating gradients of two predictors.

for other tasks and build NAS benchmarks (Dong & Yang, 2020; Ying et al., 2019; Siems et al.,
2020) on small datasets (e.g., CIFAR10, ImageNet-16) with unrealistic settings, we design a multiresolution network space and build a multi-task practical NAS benchmark (NAS-Bench-MR) on
four challenging datasets including ImageNet-224 (Deng et al., 2009), Cityscapes (Cordts et al.,
2016), KITTI (Geiger et al., 2012), and HMDB51 (Kuehne et al., 2011). Inspired by HRNet (Wang
et al., 2020a), our network space is a multi-branch multi-resolution space that naturally contains
various granularities of representations for different tasks, e.g., high-resolution features (Wang et al.,
2020a) for segmentation while low-resolution ones for classification. NAS-Bench-MR closes the
gap between existing benchmarks and NAS in multi-task and real-world scenarios. It serves as an
important contribution of this work to facilitate future cross-task NAS research.

To solve the challenge of dataset inconsistency, in this work, we propose a novel predictorbased NAS algorithm, termed Network Coding Propagation (NCP), for finding versatile and tasktransferable architectures. NCP transforms data-oriented optimization into architecture-oriented by
learning to traverse the search space. It works as follows. We formulate all network hyperparameters into a coding space by representing each architecture hyper-parameter as a code, e.g., ‘3,2,64’
denotes there are 3 blocks and each block contains 2 residual blocks with a channel width of 64.
We then learn neural predictors to build the mapping between the network coding and its evaluation
metrics (e.g., Acc, mIoU, FLOPs) for each task. By setting high-desired accuracy of each task and
FLOPs as the target, we back-propagate gradients of the learned predictor to directly update values of network codes to achieve the target. In this way, good architectures can be found in several
forward-backward iterations in seconds, as shown in Fig. 1.

NCP has several appealing benefits: (1) NCP addresses the data mismatch problem in multi-task
learning by learning from network coding but not original data. (2) NCP works in large spaces in
seconds by back-propagating the neural predictor and traversing the search space along the gradient
direction. (3) NCP can use multiple neural predictors for architecture transferring across tasks, as
shown in Fig. 1(a), it adapts an architecture to a new task with only a few iterations. (4) In NCP,
the multi-task learning objective is transformed to gradient accumulation across multiple predictors,
as shown in Fig. 1(b), making NCP naturally applicable to various even conflicting objectives, such
as multi-task structure optimization, architecture transferring across tasks, and accuracy-efficiency
trade-off for specific computational budgets.

Our main contributions are three-fold. (1) We propose Network Coding Propagation (NCP), which
back-propagates the gradients of neural predictors to directly update architecture codes along desired
gradient directions for various objectives. (2) We build NAS-Bench-MR on four challenging datasets
under practical training settings for learning task-transferable architectures. We believe it will facilitate future NAS research, especially for multi-task NAS and architecture transferring across tasks.
(3) Extensive studies on inter-, intra-, cross-task generalizability show the effectiveness of NCP in
finding versatile and transferable architectures among different even conflicting objectives and tasks.

2 RELATED WORK

**Neural Architecture Search Spaces and Benchmarks. Most existing NAS spaces (Jin et al.,**
2019; Xu et al., 2019; Wu et al., 2019; Cai et al., 2019; Xie et al., 2018; Stamoulis et al., 2019;


-----

Table 1: Comparisons among five NAS benchmarks. Existing benchmarks are either built on small
datasets for image classification, or trained with a single simplified setting. In contrast, our NASBench-MR is built on four widely-used visual recognition tasks and various realistic settings. The
architectures in our NAS-Bench-MR are trained following the common practices in real-world scenarios, e.g., 512 × 1024 and 500 epochs on the CityScapes dataset (Cordts et al., 2016). It takes
about 400,000 GPU hours to build our benchmark using Nvidia V100 GPUs.

|Benchmarks|Datasets|Tasks|Scales|Epochs|Input Sizes|Settings per Task|
|---|---|---|---|---|---|---|
|NAS-Bench-101 (Ying et al., 2019) NAS-Bench-201 (Dong & Yang, 2020) NAS-Bench-301 (Siems et al., 2020) TransNAS-Bench-101 (Duan et al., 2021)|Cifar-10 ImageNet-16 Cifar-10 Taskonomy|1 1 1 7|108 104 1018 103|4/12/36/108 200 100 ≤30|32 × 32 16 × 16 32 × 32 256 × 256|different training epochs single setting single setting single setting|
|NAS-Bench-MR (Ours)|ImageNet, Cityscapes, KITTI, HMDB51|4|1023|≥100|224 × 224 512 × 1024|different image sizes, data scale, number of classes, epochs, pretraining|



Mei et al., 2020; Guo et al., 2020; Dai et al., 2020) are designed for image classification using
either a single-branch structure with a group of candidate operators in each layer or a repeated cell
structure, e.g., Darts-based (Liu et al., 2019b) and MobileNet-based (Sandler et al., 2018) search
spaces. Based on these spaces, several NAS benchmarks (Ying et al., 2019; Dong & Yang, 2020;
Siems et al., 2020; Duan et al., 2021) have been proposed to pre-evaluate the architectures. However,
the above search spaces and benchmarks are built either on proxy settings or small datasets, such
as Cifar-10 and ImageNet-16 (16 × 16), which is less suitable for other tasks that rely on multiscale information. For those tasks, some search space are explored in segmentation (Shaw et al.,
2019a; Liu et al., 2019a; Nekrasov et al., 2019; Lin et al., 2020; Chen et al., 2018) and object
detection (Chen et al., 2019; Ghiasi et al., 2019; Du et al., 2020; Wang et al., 2020b) by introducing
feature aggregation heads (e.g., ASPP (Chen et al., 2017)) for multi-scale information. Nevertheless,
the whole network is still organized in a chain-like single branch manner, resulting in sub-optimal
performance. Another relevant work to ours is NAS-Bench-NLP Klyuchnikov et al. (2020), which
constructs a benchmark with 14k trained architectures in search space of recurrent neural networks
on two language modeling datasets.

Compared to previous spaces, our multi-resolution search space, including searchable numbers of
resolutions/blocks/channels, is naturally designed for multiple vision tasks as it contains various
granularities of feature representations. Based on our search space, we build NAS-Bench-MR for
various vision tasks, including classification, segmentation, 3D detection, and video recognition.
Detailed comparisons of NAS benchmarks can be found in Tab. 1.

**Neural Architecture Search Methods. Generally, NAS trains numerous candidate architectures**
from a search space and evaluates their performance to find the optimal architecture, which is costly.
To reduce training costs, weight-sharing NAS methods (Liu et al., 2019b; Jin et al., 2019; Xu et al.,
2019; Cai et al., 2019; Guo et al., 2020; Li & Talwalkar, 2020) are proposed to jointly train a large
number of candidate networks within a super-network. Different searching strategies are employed
within this framework such as reinforcement learning (Pham et al., 2018), importance factor learning (Liu et al., 2019b; Cai et al., 2019; Stamoulis et al., 2019; Xu et al., 2019), path sampling (You
et al., 2020; Guo et al., 2020; Xie et al., 2018), and channel pruning (Mei et al., 2020; Yu & Huang,
2019). However, recent analysis (Sciuto et al., 2020; Wang et al., 2021) shows that the magnitude
of importance parameters in the weight-sharing NAS framework does not reflect the true ranking of
the final architectures. Without weight-sharing, hyperparameter optimization methods (Tan & Le,
2019; Radosavovic et al., 2020; Baker et al., 2017; Wen et al., 2020; Lu et al., 2019; Luo et al., 2020;
Chau et al., 2020; Yan et al., 2020) has shown its effectiveness by learning the relationship between
network hyperparameters and their performance. For example, RegNet (Radosavovic et al., 2020)
explains the widths and depths of good networks by a quantized linear function. Predictor-based
methods (Wen et al., 2020; Luo et al., 2020; Chau et al., 2020; Yan et al., 2020; Luo et al., 2018)
learn predictors, such as Gaussian process (Dai et al., 2019) and graph convolution networks (Wen
et al., 2020), to predict the performance of all candidate models in the search space. A subset of
models with high predicted accuracies is then trained for the final selection.

Our Network Coding Propagation (NCP) belongs to the predictor-based method, but is different
from existing methods in: (1) NCP searches in a large search space in seconds without evaluating
all candidate models by back-propagating the neural predictor and traversing the search space along
the gradient direction. (2) Benefiting from the gradient back-propagation in network coding space,
NCP is naturally applicable to various objectives across different tasks, such as multi-task structure


-----

|4x 8x 16x Parallel Module Fusion Module Parallel Module 32 Fusion Module 32 64 b=2, n=[1,1,2], c=[32,64,32] 3 3 3|Out-layer Stage4 32x Stage3 16x Codi Stage2 8x Editi Stage1 4x In-layer2 2x In-layer1|
|---|---|


Out-layer Forward Increase

Stage4 **pacc** LAcc **tacc**

Stage3

Decrease

Stage2Stage1 **pflops** LFLOPs **tflops**

In-layers Fixed

Number of channels (i, c, o)Number of blocks (b)Number of residual units (n) Backward **ptask-spec** Ltask-spec **ttask-spec**


Example of the Stage 3


Multi-resolution Architecture


27-dimensional Coding Neural Predictor Prediction Target


Figure 2: Overview of our Network Coding Propagation (NCP) framework. Each architecture in
our search space follows a multi-resolution paradigm, where each network contains four stages,
and each stage is composed of modularized blocks (a parallel module and a fusion module). The
example on the left shows the 3rd stage consists of b3 = 2 modularized blocks with three branches,
where each branch contains a number of n[i]3 [residual units with a channel number of][ c]3[i] [,][ i][ ∈{][1][,][ 2][,][ 3][}][.]
We first learn a neural predictor between an architecture coding and its evaluation metrics such as
accuracy and FLOPs. After that, we set optimization objectives and back-propagate the predictor to
edit the network architecture along the gradient directions.

optimization, architecture transferring, and accuracy-efficiency trade-offs. (3) Different from Luo
et al. (2018); Baker et al. (2017) jointly train an encoder, a performance predictor, and a decoder
to minimize the combination of performance prediction loss and structure reconstruction loss, we
learn and inverse the neural predictor directly in our coding space to make the gradient updating and
network editing explicit and transparent.

3 METHODOLOGY

NCP aims to customize specific network hyperparameters for different optimization objectives, such
as single- and multi-task learning and accuracy-efficiency trade-offs, in an architecture coding space.
An overview of NCP is shown in Fig. 2. In this section, we first discuss learning efficient models on
a single task with two strategies and then show that it can be easily extended to multi-task scenarios.
Lastly, we demonstrate our multi-resolution coding space and NAS-Bench-MR.

3.1 NETWORK CODING PROPAGATION

For each task, we learn a neural predictortruth evaluation metrics, such as accuracy g Facc and FLOPs(·) that projects an architecture code gflops. Given an initialized and normalized e to its ground
coding e, the predicted metrics and the loss to learn the neural predictor are represented by:
_pacc, pflops, ptask-spec = FW (e),_ (1)
_Lpredictor = L2(pacc, gacc) + L2(pflops, gflops) + ..._ (2)
where W is the weight of the neural predictor, which is fixed after learning. pacc, pflops, and ptask-spec
denote the predicted accuracy, FLOPs, and task-specific metrics (if any), L2 is the L2 norm. Note
that although the FLOPs can be directly calculated from network codes, we learn it to enable the
backward gradient upon FLOPs constraints for accuracy-efficiency trade-offs.

NCP edits the network coding e by back-propagating the learned neural predictor W to maximize/minimize the evaluation metrics (e.g., high accuracy and low FLOPs) along the gradient directions. According to the number of dimensions to be edited, two strategies are proposed to propagate
the network coding: continuous propagation and winner-takes-all propagation, where the first one
uses all the dimensions of the gradient to update e, while the latter uses only the one dimension with
the largest gradient. Fig. 3 visualizes the network update process of our two strategies.

**Continuous Propagation.** In continuous propagation, we set target metrics as an optimization
goal and use gradient descending to automatically update the network coding e. Taking accuracy
and FLOPs as an example, we set the target accuracy tacc and target FLOPs tflops for e and calculate
the loss between the prediction and the target by:
_Lpropagation = L1(pacc, tacc) + λL1(pflops, tflops)_ (3)
where L1 denotes the smooth L1 loss, λ is the coefficient to balance the trade-off between accuracy
and efficiency. The loss is propagated back through the fixed neural predictor FW (·), and the gradient on the coding e is calculated as _[∂L]∂e_ [(use][ ∇][e][ in following for simplicity). We update the coding]


-----

128

104

80

56

32


128

104

32

|Col1|St St St St|age1: N_bl age2: N_bl age3: N_bl age4: N_bl|ocks ocks ocks ocks|
|---|---|---|---|


20 40 60 8

In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels

128

104

80

56

|Col1|Col2|Col3|1|
|---|---|---|---|
|||||
|Stag Stag Stag|e3: N_cha e3: N_cha e3: N_cha|nnels in br nnels in br nnels in br|anch1 anch2 anch3|



20 40 60 8

4

2

1

20 40 60

|Stag Stag Stag|e3: N_unit e3: N_unit e3: N_unit|s in branch s in branch s in branch|1 2 3|Col5|
|---|---|---|---|---|
||||||
||||||


Iterations


77.5

75.0

60

40


77

76


20 40 60


20 40 60

Predicted mIoU (%)


20 40 60


Predicted mIoU (%)


40

0 20 40 60

GFLOPs

128

80

56

32

8 0 20 40 60

|Stage Stage|2: N_chann 2: N_chann|els in branch els in branch|
|---|---|---|
||||


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2

4

2

1 0 20 40 60

|Col1|St St St|age1: N_u age2: N_u age2: N_u|Col4|nits in branch nits in branch nits in branch|
|---|---|---|---|---|
||||||
||||||


Iterations

|Col1|In-layer Out-lay|s: N_chann er: N_chan|els1 nels|
|---|---|---|---|

|Stag Stag Stag|e2: N_blo e3: N_blo e4: N_blo|cks cks cks|Col4|
|---|---|---|---|

|In-layers: Out-layer|N_chan : N_cha|nels1 nnels|
|---|---|---|


20 40 60

|Col1|Col2|Col3|1|
|---|---|---|---|
|||||
|Stag Stag Stag|e3: N_cha e3: N_cha e3: N_cha|nnels in bra nnels in bra nnels in bra|nch1 nch2 nch3|


Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


20 40 60

20 40 60

|Col1|St St St|age3: N_ age3: N_ age3: N_|units in bran units in bran units in bran|Col5|ch1 ch2 ch3|
|---|---|---|---|---|---|
|||||||
|||||||


Iterations


8 0 20 40 60


20 40 60

|Stag|e4: N_cha|nnels in br|anch1|
|---|---|---|---|
|Stag Stag Stag|e4: N_cha e4: N_cha e4: N_cha|nnels in br nnels in br nnels in br|anch2 anch3 anch4|



20 40 60

20 40 60

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|St St St St|age4: N_u age4: N_u age4: N_u age4: N_u|nits in bra nits in bra nits in bra nits in bra|nch1 nch2 nch3 nch4|
|||||


Iterations


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


GFLOPs


128

104

80

56

32


128

104

80

56

32


80

56

32


56

32

|Stage Stage|2: N_chan 2: N_chan|nels in bra nels in bra|nch1 nch2|
|---|---|---|---|
|||||

|tage4: N_channe|els in bra|anch2|
|---|---|---|
|age4: N_channe age4: N_channe age4: N_channe|ls in bra ls in bra ls in bra|nch2 nch3 nch4|
||||


20 40 60

Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


20 40 60

|Stag Stag Stag|e1: N_unit e2: N_unit e2: N_unit|s in branch s in branch s in branch|1 1 2|Col5|
|---|---|---|---|---|
||||||
||||||


Iterations


20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Stage4: N_units i Stage4: N_units i Stage4: N_units i|n branc n branc n branc|h1 h2 h3|
|---|---|---|
|Stage4: N_units i|n branc|h4|
||||


Stage4: N_units in branch1
Stage4: N_units in branch2
Stage4: N_units in branch3
Stage4: N_units in branch4

Iterations


Figure 3: Visualization of the network propagation process of our two strategies (left: continuous;
right: winner-takes-all) for segmentation. We group 27 dimensions into eight subfigures.


**_e iteratively by e ←_** **_e −∇e. In this way, NCP can search for the most efficient model upon a_**
lower-bound accuracy and the most accurate model upon upper-bound FLOPs. In practice, we set
_tacc = pacc + 1 and tflops = pflops −_ 1 to find a model with a good efficiency-accuracy trade-off.

The propagation is stopped once the prediction reaches the target or after a certain number of iterations (e.g., 70). Finally, we round (the number of channels is a multiple of 8) and decode the coding
**_e to obtain the architecture optimized for the specific goal._**

**Winner-Takes-All Propagation. Editing different dimensions of the coding often have different**
degrees of impact on the network. For example, modifying the number of network blocks has a
greater impact than the number of channels, i.e., a block costs more computational resources than
several channels. In light of this, we select one dimension of the coding e with the best accuracyefficiency for propagation in each iteration, termed as winner-takes-all propagation.

We create a lookup table T (·) to obtain the corresponding FLOPs of any network coding. We then
calculate the relative increase ∆r in FLOPs as each dimension of the coding grows:

**_e[′]l_** [=][ e]l [+][ k]

(4)
∆rl = T (e[′]) −T (e)

where l denotes the index of the coding e; k = 8 if el represents the number of convolutional
channels, otherwise k = 1. We then use ∆r as a normalization term of backward gradients ∇e and
select the optimal dimension l of the normalized gradients ∇e/∆r:

arg max( **_e/∆r)_** if max( **_e/∆r) > 0_**

_l =_ _∇_ _∇_ (5)

(arg min( **_e/∆r)_** if max( **_e/∆r) < 0_**
_∇_ _∇_

iftaneously. While ifaccuracy under the unit FLOPs consumption. The coding max(∇e/∆r) > max( 0, modifying the value of∇e/∆r) < 0, we choose the dimension with the highest improvement in el may improve accuracy and decrease FLOPs simul- e is then updated as follows:

**_el_** **_el −_** _k_ if max(∇e/∆r) > 0 (6)
_←_ (el + k if max( **_e/∆r) < 0_**

_∇_

**Cross-task Learning.** Our NCP provides a natural way for cross-task learning by transforming different objectives across tasks into gradient accumulation operations directly in the architecture coding space, facilitating joint architecture search for multiple tasks and cross-task architecture
transferring, as shown in Fig. 1. For example, NCP enables joint search for classification and segmentation by accumulating the gradients ∇e[cls] and ∇e[seg] from the two predictors:

_∇e = ∇e[cls]_ + ∇e[seg] (7)

Note that different weights coefficients can be used to learn multi-task architectures with task preferences. In this work, we set equal weight coefficients for these tasks.

3.2 NAS-BENCH-MR

**Multi-Resolution Search Space.** Inspired by HRNet (Wang et al., 2020a; Ding et al., 2021), we
design a multi-branch multi-resolution search space that contains four branches and maintains both


-----

high-resolution and low-resolution representations throughout the network. As shown in Fig. 2, after
two convolutional layers decreasing the feature resolution to 1/4 (4×) of the image size, we start
with this high-resolution feature branch and gradually add lower-resolution branches with feature
fusions, and connect the multi-resolution branches in parallel. Multi-resolution features are resized
and concatenated for the final classification/regression without any additional heads.

The design space follows the following key characteristics. (1) Parallel modules: extend each branch
of the corresponding resolutions in parallel using residual units (He et al., 2016); (2) Fusion modules: repeatedly fuse the information across multiple resolutions and generate the new resolution
(only between two stages). For each output resolution, the corresponding input features of its neighboring resolutions are gathered by residual units (He et al., 2016) followed by element-wise addition
(upsampling is used for the low-to-high resolution feature transformation), e.g., the 8× output feature contains information of 4×, 8×, and 16× input features; (3) A modularized block is then formed
by a fusion module and a parallel module.

**Coding. We leverage the hyperparameter space of our multi-resolution network by projecting the**
depth and width values of all branches of each stage into a continuous-valued coding space. Formally, two 3 × 3 stride 2 convolutional layers with i1, i2 number of channels are used to obtain
the 4× high-resolution features at the beginning of the network. We then divide the network into
four stages with s = {1, 2, 3, 4} branches, respectively. Each stage contains bs modularized blocks,
where the fusion module of the first block is used for transition between two stages (e.g., from 2
branches to 3 branches). For the parallel module of the block, we set the number of residual units
and the number of convolutional channels to ns = [n[1]s[, . . ., n][s]s[]][ and][ c][s] [= [][c][1]s[, . . ., c][s]s[]][ respectively,]
where s denotes the s-th stage containing s branches. Since the fusion module always exists between two parallel modules, once the hyperparameters of the parallel modules are determined, the
corresponding number of channels of fusion modules will be set automatically to connect the two
parallel modules. The s-th stage comprising a plurality of same modular blocks is represented by:

_bs, n[1]s[, . . ., n][s]s[, c][1]s[, . . ., c][s]s_ (8)

where the four stages are represented by 3, 5, 7, and 9-dimensional codes, respectively. By the end
of the 4-th stage of the network, we use a 1 × 1 stride 1 convolutional layer with a channel number
of o1 to fuse the four resized feature representations with different resolutions. In general, the entire
architecture can be represented by a 27-dimensional continuous-valued code:

**_e = i1, i2, b1, n1, c1, . . ., b4, n4, c4, o1_** (9)

We then build our coding space by sampling and assigning values to the codes e. For each structure, we randomly choose b, n ∈ 1, 2, 3, 4, and i, c, o ∈{8, 16, 24, . . ., 128}, resulting in a 27dimensional coding. In this way, a fine-grained multi-resolution space that enables searching structures for various tasks with different preferences on the granularity of features is constructed.

**Dataset. We build our NAS-Bench-MR on four carefully selected visual recognition tasks that**
require different granularity of features: image classification on ImageNet (Deng et al., 2009), semantic segmentation on Cityscapes (Cordts et al., 2016), 3D object detection on KITTI (Geiger
et al., 2012), and video recognition on HMDB51 (Kuehne et al., 2011).


For classification, considering the diversity of the
number of classes and data scales in practical applications, we construct three subsets: ImageNet-50-1000
(Cls-A), ImageNet-50-100 (Cls-B), and ImageNet-101000 (Cls-C), where the first number indicates the
number of classes and the second one denotes the number of images per class. For segmentation, we follow HRNet (Wang et al., 2020a) by upsampling all
branches to the high-resolution one. For 3D detection,
we follow PointPillars (Lang et al., 2019) that first
converts 3D point points into bird’s-eye view (BEV)
featuremaps and then unitizes a 2D network. For
**video recognition, we replace the last 2D convolu-**
tional layer before the final classification layer of our
network with a 3D convolutional layer.



1.0

0.8

0.6

0.4

0.2

0.0


Cls-A

Cls-B

Cls-C

Cls-10c

Seg

Seg-4x

3dDet

Video

Video-p

|Cls-A|Cls-B|Cls-C|Cls-10c|Seg|Seg-4x|3dDet|Video|Video-p|
|---|---|---|---|---|---|---|---|---|
|1.000|0.300|0.318|0.517|0.639|0.643|0.494|0.204|0.285|
|0.300|1.000|0.519|0.818|0.185|0.368|0.200|0.647|0.625|
|0.318|0.519|1.000|0.538|0.223|0.389|0.237|0.367|0.375|
|0.517|0.818|0.538|1.000|0.398|0.513|0.342|0.632|0.618|
|0.639|0.185|0.223|0.398|1.000|0.631|0.409|-0.003|0.029|
|0.643|0.368|0.389|0.513|0.631|1.000|0.551|0.153|0.174|
|0.494|0.200|0.237|0.342|0.409|0.551|1.000|0.129|0.116|
|0.204|0.647|0.367|0.632|-0.003|0.153|0.129|1.000|0.730|
|0.285|0.625|0.375|0.618|0.029|0.174|0.116|0.730|1.000|


1.000 0.300 0.318 0.517 0.639 0.643 0.494 0.204 0.285

0.300 1.000 0.519 0.818 0.185 0.368 0.200 0.647 0.625

0.318 0.519 1.000 0.538 0.223 0.389 0.237 0.367 0.375

0.517 0.818 0.538 1.000 0.398 0.513 0.342 0.632 0.618

0.639 0.185 0.223 0.398 1.000 0.631 0.409 -0.003 0.029

0.643 0.368 0.389 0.513 0.631 1.000 0.551 0.153 0.174

0.494 0.200 0.237 0.342 0.409 0.551 1.000 0.129 0.116

0.204 0.647 0.367 0.632 -0.003 0.153 0.129 1.000 0.730

0.285 0.625 0.375 0.618 0.029 0.174 0.116 0.730 1.000


Figure 4: Spearman’s rank correlation of
9 subtasks in NAS-Bench-MR.


-----

Table 2: Performance comparison (%) on the Cityscapes validation set. All hyperparameter optimization methods are searched using our NAS-Bench-MR. All models are trained from scratch.

|FLOPs is measured using an input siz|ze of 512 × 1024. λ = 0.|.7, 0.3, 0.1 for ‘S|S’, ‘M’, and ‘L’.|
|---|---|---|---|
|Model|Type|Params FLOPs|mIoU mAcc aAcc|
|ResNet34-PSP (Zhao et al., 2017) ResNet50-PSP (Zhao et al., 2017) HRNet-W18 (Wang et al., 2020a) HRNet-W32 (Wang et al., 2020a)|manually-designed manually-designed manually-designed manually-designed|23.05M 193.12G 48.98M 356.91G 9.64M 37.01G 29.55M 90.55G|76.17 82.66 95.99 76.49 83.18 95.98 77.73 85.61 96.20 79.28 86.48 96.31|
|SqueezeNAS (Shaw et al., 2019a) Auto-DeepLab (Liu et al., 2019a)|NAS in SqueezeNAS space NAS in DeepLab space|3.00M 32.73G 10.15M 289.78G|75.19 − − 79.74 − −|
|NetAdapt (Yang et al., 2018) hyperparameter optimization 10.83M 42.49G 78.02 85.68 96.24 Random Search (Bergstra & Bengio, 2012) hyperparameter optimization 16.27M 130.50G 77.66 85.35 96.13 Neural Predictor (Wen et al., 2020) hyperparameter optimization 33.49M 182.60G 78.89 86.46 96.21||||
|NCP-Net-M-winner takes all NCP-Net-M-continuous NCP-Net-S-continuous NCP-Net-L-continuous|hyperparameter optimization hyperparameter optimization hyperparameter optimization hyperparameter optimization|7.18M 32.88G 8.39M 36.25G 6.04M 20.07G 37.70M 179.98G|77.91 85.69 96.23 78.36 86.24 96.29 76.98 85.14 96.07 80.05 87.12 96.49|



We also explore several proxy training settings, such as 10-epoch training used in RegNet (Radosavovic et al., 2020) (Cls-10c), reduced input resolution by four times (Seg-4x), and pretraining instead of training from scratch (Video-p). In summary, we randomly sample 2,500 structures
from our search space, train and evaluate these same architectures for those above nine different
tasks/settings, resulting in 22,500 trained models. We believe these well-designed and fully-trained
models serve as an important contribution of this work to facilitate future NAS research.

4 EXPERIMENTS

4.1 IMPLEMENTATION DETAILS

Taking a three-layer fully connected network as the neural predictor, we apply NCP to NAS-BenchMR and obtain NCP-Net. To train the neural predictor, 2000 and 500 structures in the benchmark
are used as the training and validation sets for each task. Unless specified, we use continuous
propagation with an initial code of {b, n = 2; c, i, o = 64} and λ = 0.5 for 70 iterations in all
experiments. The optimization goal is set to higher performance and lower FLOPs (tacc = pacc + 1,
_tflops = pflops −_ 1). More experiments and details can be found in Appendix.

4.2 BASELINE EVALUATIONS

We make comparisons among different search strategies on our multi-resolution space and segmentation benchmark in Tab. 2. For Random Search (Bergstra & Bengio, 2012) and Neural Predictor (Wen et al., 2020), we sample 100 and 10,000 network codes respectively and report the highest
result in top-10 models. For NetAdapt (Yang et al., 2018), we traverse each dimension of the initial
codes (increasing or decreasing a unit value) and adapt the codes by greedy strategy.

**Effectiveness of NCP. (1) NCP outperforms manually-designed networks, e.g., ResNet (He et al.,**
2016) and HRNet (Wang et al., 2020a), hyperparameter optimization methods (Bergstra & Bengio,
2012; Wen et al., 2020; Yang et al., 2018), and even well-designed weight-sharing NAS (Liu et al.,
2019a; Shaw et al., 2019a). (2) Compared to other hyperparameter optimization methods, NCP is
not sensitive to the randomness and noise in model selection by adopting an approximate gradient
direction. (3) NCP is much more efficient as it evaluates all dimensions of the codes with only one
back-propagation without top-K ranking. (4) NCP is able to search for optimal architectures under
different computing budgets by simply setting the coefficient λ.

**Effectiveness of our search space. (1) Random Search (Bergstra & Bengio, 2012) works well in**
our coding space, showing the effectiveness of our multi-resolution space. (2) Models in our search
space consist of only 3 × 3 convolution in the basic residual unit (He et al., 2016) without complex
operators, making it applicable to various computing platforms, such as CPU, GPU, and FPGA.

**Search strategies of NCP. Intuitively, the winner-take-all strategy is more like a greedy algorithm.**
It selects one coding dimension with the best accuracy-efficiency trade-off for propagation, resulting


-----

Table 3: Performance (%) on five datasets of the models searched on different optimization objectives (including single- and multi-task optimization) using our NCP, e.g.“Cls + Seg” denotes the
model is searched using “Cls-A” and “Seg” benchmarks, “Four Tasks” denotes the model is propagated by using the predictors trained on the all four benchmarks (Cls-A, Seg, 3dDet, Video). Note
that in addition to cross-task evaluation, we also show the generalizability of NAS by applying the
searched model to a new dataset, ADE20K, which is not used to train neural predictors. For clearer
comparisons, the FLOPs of all networks is measured using input size 128 × 128 under the segmentation task. The top-2 results are highlighted in bold.

|Method|Params FLOPs|Classification ImageNet-50-1000 top1 top5|Segmentation Cityscapes mIoU mAcc aAcc|Video HMDB51 top1 top5|3D Object Detection KITTI car-3D car-BEV ped-3D ped-BEV|Segmentation ADE20K mIoU mAcc aAcc|
|---|---|---|---|---|---|---|
|Cls Seg Video 3dDet|5.35M 2.05G 7.91M 0.90G 2.56M 0.74G 2.89M 1.21G|85.56 95.36 82.52 94.00 81.88 93.84 82.60 94.64|75.65 84.11 95.23 77.15 84.73 96.01 71.09 79.91 95.14 69.43 78.48 95.05|25.53 59.69 14.68 44.92 28.47 62.18 19.84 53.82|73.60 83.92 35.20 41.39 69.99 84.12 21.83 31.21 68.60 81.68 19.02 27.82 75.44 87.37 40.42 48.69|32.74 42.77 76.99 33.32 42.97 77.41 25.84 34.07 73.88 23.28 30.64 72.29|
|Cls + Seg Cls + Video Cls + 3dDet Seg + Video Seg + 3dDet Video + 3dDet|8.29M 1.70G 4.32M 1.60G 4.46M 1.81G 4.46M 1.16G 4.30M 0.91G 2.37M 0.85G|86.05 95.35 85.64 95.28 85.76 95.27 83.84 94.20 83.08 94.57 83.27 94.66|77.16 84.95 96.18 72.98 81.51 95.54 72.18 80.81 95.42 75.64 84.16 95.88 75.60 83.74 95.85 70.32 79.33 95.12|22.95 57.74 27.83 60.84 23.17 58.40 28.46 60.77 21.01 56.46 28.20 61.15|74.39 84.72 27.18 35.70 71.05 84.56 23.84 30.27 75.47 87.63 40.61 48.80 71.24 83.47 34.44 42.00 75.54 86.67 41.45 46.32 74.83 83.52 37.82 44.17|35.58 46.02 78.19 30.43 41.29 75.78 31.98 42.00 76.67 34.23 44.45 77.28 34.65 46.38 77.71 26.42 36.34 73.69|
|Four Tasks|4.40M 1.44G|84.36 95.04|75.49 83.86 95.83|25.09 58.36|71.90 84.29 27.27 34.34|34.53 45.36 77.45|



in efficient models. The continuous strategy updates all dimensions of the coding purely based on
gradients. Thus, it reaches better accuracy, though the generated model is not the most efficient.
The experiment verifies the above intuition. Unless specified, we use the continuous strategy for all
experiments. Fig. 3 shows the code editing process of two strategies.

4.3 INTER-TASK ARCHITECTURE SEARCH

We conduct multi-task architecture learning experiments on four basic benchmarks, including ClsA, Seg, Video, and 3dDet. We use NCP to search models on single (e.g., Cls) or multiple tasks (e.g.,
Cls + Seg) and evaluate their performance on five datasets, including a new dataset ADE20K (Zhou
et al., 2017) which is not included in NAS-Bench-MR. Quantitative comparisons are shown in Tab. 3.

**Optimization for various objectives. (1) NCP can customize good architectures for every single**
task, e.g., the model searched on Cls achieves a top-1 accuracy of 85.56%, better than the models
searched on the other three tasks. (2) NCP can learn architectures that achieve good performance
on multiple tasks by inverting their predictors simultaneously, e.g., the model searched on Cls +
Seg achieves a top1 accuracy of 86.05% and 77.16% mIoU. The joint optimization of all four tasks
achieved moderate results on each task. (3) NCP achieves good accuracy-efficiency trade-offs and
tunes a proper model size for each task automatically by using the FLOPs constraint.

**Relationship among tasks. Two tasks may help each other if they are highly related, and vice**
versa. For example, jointly optimizing Seg + Cls (correlation is 0.639) improves the performance
on both tasks with 17% fewer FLOPs than a single Cls model, while jointly optimizing Seg +
Video (correlation is -0.003) hinders the performance of both tasks. When optimizing Seg + 3dDet
(correlation is 0.409) simultaneously, the resulted network has better results in 3dDet but worse in
Seg, which means accurate semantic information is useful for 3D detection, while the object-level
localization may harm the performance of per-pixel classification. The above observations can be
verified by Spearman’s rank correlation coefficients in Fig. 4. This way, we can decide which tasks
are more suitable for joint optimization, such as CLs + Video, improving both tasks.

**Generalizability. The searched models work well on the new dataset ADE20K (Zhou et al., 2017).**
An interesting observation is that the searched models under multi-task objectives, e.g., Seg + Cls,
even conflicting tasks, e.g., Seg + Video, show better performance on ADE20K than those searching
on a single task, demonstrating the generalizability of NCP. We also adapt our segmentation model
(NCP-Net-L-continuous) in Tab. 2 to the COCO instance segmentation dataset. The model with only
360 GFLOPs (measured using 800x1280 resolution, Mask R-CNN head) achieves 47.2% bounding
box AP and 42.3% mask AP on the detection and instance segmentation tasks, respectively. With
fewer computational costs, our results outperform HRNet-W48 (46.1% for bbox AP and 41.0% for
mask AP) by a large margin.


-----

Table 4: Our architecture transfer results between four different tasks. We first find an optimal
architecture coding for each task and then use it as the initial coding to search other three tasks. “-F”
and “-T” denote the architecture finetuning and transferring results, respectively.

|Method|FLOPs-F|Classification ImageNet-50-1000 top1-F top1-T FLOPs-T|Col4|Segmentation Cityscapes mIoU-F mIoU-T FLOPs-T|Col6|Video Recognition HMDB51 top1-F top1-T FLOPs-T|Col8|3D Object Detection KITTI car-3D-F car-3D-T FLOPs-T|Col10|
|---|---|---|---|---|---|---|---|---|---|
|Classification Segmentation Video Recognition 3D Object Detection|2.05G 0.90G 0.74G 1.21G|85.56 82.52 81.88 82.60|– – 85.84 1.54G 86.03 1.86G 85.63 1.61G|75.42 77.15 71.09 69.43|78.27 1.10G – – 77.56 0.95G 77.25 1.04G|25.53 14.68 28.47 19.84|28.65 0.76G 28.44 0.75G – – 28.12 0.78G|73.60 69.99 68.60 75.44|75.59 1.30G 76.17 1.21G 75.78 1.17G – –|



4.4 CROSS-TASK ARCHITECTURE TRANSFERRING

We experiment on architecture transferring across four different tasks. We first find optimal network
codings for each task and then use it as the initial coding to search each of the other three tasks.

Comparison results can be found in Tab. 4. We see that: (1) Compared to searching a network from
scratch for one task (e.g., 77.15% on segmentation) or finetuning architectures trained on other tasks
(e.g., 75.42% by finetuning a classification model), our transferred architecture are always better
(e.g., 78.27% by transferring a classification architecture to segmentation).

(2) The architecture searched on classification achieves better transferable ability on segmentation
and video recognition, while the segmentation model is more suitable for transferring to 3D detection. It can be validated by Spearman’s rank correlation in Fig. 4.

(3) NCP is not sensitive to the initialized architecture coding. Models can be searched with different
initial codings and achieve good performance.

4.5 INTRA-TASK ARCHITECTURE SEARCH

To show NCP ’s ability to customize architectures according to different data scales and the number
of classes, we evaluate the searched architecture on the three subsets of the ImageNet dataset. From
Tab. 5, we observe that: (1) NCP can find optimal architectures for each subset, which is essential for
practical applications with different magnitudes of data. (2) We perform a joint architecture search
for all three subsets. The searched architecture (NCP-Net-ABC) shows surprisingly great results
on all three subsets with only 0.77G extra FLOPs and fewer parameters, demonstrating intra-task
generalizability of NCP.


Table 5: Single-crop top-1 error
rates (%) on the ImageNet validation
set. ‘ImNet-A’, ‘ImNet-B’, ‘ImNetC’ denote the predictor is trained
on benchmarks of ImageNet-501000, ImageNet-50-100, ImageNet10-1000, respectively. FLOPs is
measured using input size 224×224.
_λ = 0.7._ Top-2 results are highlighted in bold.

5 CONCLUSION


|Model|Params FLOPs|ImNet-A ImNet-B ImNet-C|
|---|---|---|
|ResNet50 ResNet101 HRNet-W32 HRNet-W48|23.61M 4.12G 42.60M 7.85G 39.29M 8.99G 75.52M 17.36G|83.76 50.16 86.00 84.32 51.92 86.00 84.00 52.00 83.80 84.52 52.88 86.60|
|NCP-Net-A NCP-Net-B NCP-Net-C NCP-Net-ABC|4.39M 5.95G 3.33M 3.55G 4.06M 5.51G 4.28M 6.72G|85.80 56.04 86.80 82.76 56.40 85.60 84.60 55.60 87.80 85.12 58.12 88.20|


This work provides an initial study of learning task-transferable architectures in the network coding
space. We propose an efficient NAS method, namely Network Coding Propagation (NCP), which
optimizes the network code to achieve the target constraints with back-propagation on neural predictors. In NCP, the multi-task learning objective is transformed to gradient accumulation across multiple predictors, making NCP naturally applicable to various objectives, such as multi-task structure
optimization, architecture transferring across tasks, and accuracy-efficiency trade-offs. To facilitate the research in designing versatile network architectures, we also build a comprehensive NAS
benchmark (NAS-Bench-MR) upon a multi-resolution network space on many challenging datasets,
enabling NAS methods to spot good architectures across multiple tasks, other than classification as
the main focus of prior works. We hope this work and models can advance future NAS research.


-----

ACKNOWLEDGEMENTS

We sincerely appreciate all reviewers’ efforts and constructive suggestions in improving our paper.
Ping Luo was supported by the General Research Fund of HK No.27208720 and HKU-TCL Joint
Research Center for Artificial Intelligence. Zhiwu Lu was supported by National Natural Science
Foundation of China (61976220).

REFERENCES

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture
search using performance prediction. arXiv preprint arXiv:1705.10823, 2017. 3, 4

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of
_machine learning research, 13(2), 2012. 7_

Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task
and hardware. In ICLR, 2019. 2, 3

Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one
network and specialize it for efficient deployment. In ICLR, 2020. 1

Thomas Chau, Łukasz Dudziak, Mohamed S Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D
Lane. Brp-nas: Prediction-based nas using gcns. arXiv preprint arXiv:2007.08668, 2020. 3

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and
fully connected crfs. IEEE TPAMI, 40(4):834–848, 2017. 3

Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff,
Hartwig Adam, and Jon Shlens. Searching for efficient multi-scale architectures for dense image
prediction. In NeurIPS, pp. 8699–8710, 2018. 3

Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew G Berneshawi, Huimin Ma, Sanja Fidler, and
Raquel Urtasun. 3d object proposals for accurate object class detection. In NeurIPS, pp. 424–432,
2015. 15

Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas:
Neural architecture search on object detection. arXiv preprint arXiv:1903.10979, 1(2):4–1, 2019.
3

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo
Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic
urban scene understanding. In CVPR, pp. 3213–3223, 2016. 1, 2, 3, 6, 14

Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan,
Yunqing Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through
platform-aware model adaptation. In CVPR, pp. 11398–11407, 2019. 3

Xiyang Dai, Dongdong Chen, Mengchen Liu, Yinpeng Chen, and Lu Yuan. Da-nas: Data adapted
pruning for efficient neural architecture search. arXiv preprint arXiv:2003.12563, 2020. 3

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR, pp. 248–255. Ieee, 2009. 2, 6, 14

Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning
depth-guided convolutions for monocular 3d object detection. In CVPR, 2020a. 15

Mingyu Ding, Zhe Wang, Bolei Zhou, Jianping Shi, Zhiwu Lu, and Ping Luo. Every frame counts:
joint learning of video segmentation and optical flow. In AAAI, volume 34, pp. 10713–10720,
2020b. 15

Mingyu Ding, Xiaochen Lian, Linjie Yang, Peng Wang, Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hrnas: Searching efficient high-resolution neural architectures with lightweight transformers. In
_CVPR, pp. 2982–2992, 2021. 1, 5_


-----

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture
search. In ICLR, 2020. 2, 3, 18

Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and
Xiaodan Song. Spinenet: Learning scale-permuted backbone for recognition and localization. In
_CVPR, pp. 11592–11601, 2020. 3_

Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li.
Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In CVPR, pp. 5251–5260, 2021. 1, 3

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti
vision benchmark suite. In CVPR, pp. 3354–3361, 2012. 2, 6, 15

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In CVPR, pp. 7036–7045, 2019. 3

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun.
Single path one-shot neural architecture search with uniform sampling. In ECCV, 2020. 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770–778, 2016. 1, 6, 7, 17, 18, 19

Xiaojie Jin, Jiang Wang, Joshua Slocum, Ming-Hsuan Yang, Shengyang Dai, Shuicheng Yan, and
Jiashi Feng. Rc-darts: Resource constrained differentiable architecture search. arXiv preprint
_arXiv:1912.12814, 2019. 2, 3_

Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and
Evgeny Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language
processing. arXiv preprint arXiv:2006.07116, 2020. 3

Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb:
A large video database for human motion recognition. In ICCV, pp. 2556–2563, 2011. 1, 2, 6, 15

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, pp. 12697–12705, 2019.
6, 15

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In
_Uncertainty in Artificial Intelligence, pp. 367–377. PMLR, 2020. 3_

Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and
Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv
_preprint arXiv:1909.06035, 2019. 1_

Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Xi Li, and Jianping Shi. Graph-guided architecture search for real-time semantic segmentation. In CVPR, pp. 4203–4212, 2020. 3

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li FeiFei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In
_CVPR, pp. 82–92, 2019a. 3, 7_

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In
_ICLR, 2019b. 1, 3, 18_

Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and
Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm.
In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419–427, 2019. 3

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization.
In Advances in neural information processing systems, pp. 7816–7827, 2018. 3, 4

Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture
search with gbdt. arXiv preprint arXiv:2007.04785, 2020. 3, 19, 21


-----

Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang.
Atomnas: Fine-grained end-to-end neural architecture search. In ICLR, 2020. 3

Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Fast neural architecture search of
compact semantic segmentation models via auxiliary cells. In CVPR, pp. 9126–9135, 2019. 3

Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture
search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018. 3

Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll´ar. Designing
network design spaces. In CVPR, pp. 10428–10436, 2020. 3, 7, 16

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510–4520, 2018. 3

Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating
the search phase of neural architecture search. ICLR, 2020. 3

Albert Shaw, Daniel Hunter, Forrest Landola, and Sammy Sidhu. Squeezenas: Fast neural architecture search for faster semantic segmentation. In ICCVW, 2019a. 1, 3, 7

Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. In NeurIPS,
pp. 11227–11237, 2019b. 1

Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nasbench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint
_arXiv:2008.09777, 2020. 2, 3_

Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu,
and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4
hours. In ECML-PKDD, pp. 481–497, 2019. 2, 3

Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural
networks. In ICML, pp. 6105–6114, 2019. 3

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu,
Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning
for visual recognition. IEEE TPAMI, 2020a. 1, 2, 5, 6, 7, 15, 17, 18, 19

Ning Wang, Yang Gao, Hao Chen, Peng Wang, Zhi Tian, Chunhua Shen, and Yanning Zhang. Nasfcos: Fast neural architecture search for object detection. In CVPR, pp. 11943–11951, 2020b.
3

Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking
architecture selection in differentiable nas. In ICLR, 2021. 3

Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural
predictor for neural architecture search. In ECCV, pp. 660–676. Springer, 2020. 3, 7, 19, 21

Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian,
Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via
differentiable neural architecture search. In CVPR, pp. 10734–10742, 2019. 1, 2

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search.
_arXiv preprint arXiv:1812.09926, 2018. 1, 2, 3_

Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pcdarts: Partial channel connections for memory-efficient differentiable architecture search. arXiv
_preprint arXiv:1907.05737, 2019. 2, 3_

Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? arXiv preprint arXiv:2006.06936, 2020. 3


-----

Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and
Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In
_ECCV, pp. 285–300, 2018. 7, 21_

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nasbench-101: Towards reproducible neural architecture search. In ICML, pp. 7105–7114, 2019. 2,
3, 18

Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas:
Towards fast one-shot nas with greedy supernet. In CVPR, pp. 1999–2008, 2020. 3

Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers.
_arXiv preprint arXiv:1903.11728, 2019. 3_

Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan,
Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In ECCV, 2020. 1

Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio
Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, pp. 3712–3722, 2018.
1

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing
network. In CVPR, pp. 2881–2890, 2017. 7, 15, 17

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene
parsing through ade20k dataset. In CVPR, pp. 633–641, 2017. 8, 18


-----

Table 6: Detailed statistics of our NAS-Bench-MR. NAS-Bench-MR contains 4 datasets and 9 settings. Considering the diversity and complexity of real-world applications (e.g.different scales/input
sizes of training data), we use a variety of challenging settings (e.g., full resolution and training
epochs) to ensure that the model is fully trained. For classification, we train models with different
numbers of classes, numbers of training samples, and training epochs. For semantic segmentation,
we train models under different input sizes. We also evaluated video action recognition models
under two settings: trained from scratch and pretrained with ImageNet-50-1000. † denotes each
sample contains multiple classes. ‡ denotes there are three basic classes (car, pedestrian, cyclist)
in KITTI, while for each object we also regress its 3D location (XYZ), dimensions (WHL), and
orientation (α). “N” – training from scratch. “Y” – training with the ImageNet-50-1000 pretrained
model. We also give the mean, std, and the validation L1 loss (%) of the neural predictor under the
main evaluation metric of each setting.

|Col1|Cls-50-1000|Cls-50-100|Cls-10-1000|Cls-10c|Seg|Seg-4x|3dDet|Video|Video-p|
|---|---|---|---|---|---|---|---|---|---|
|Dataset|ImNet-50-1000|ImNet-50-100|ImNet-10-1000|ImNet-50-1000|Cityscapes|resized Cityscapes|KITTI|HMDB51|HMDB51|
|Input Size|224 × 224|224 × 224|224 × 224|224 × 224|512 × 1024|128 × 256|432 × 496|112 × 112|112 × 112|
|Epochs|100|100|100|10|530|530|80|100|100|
|Number of classes|50|50|10|50|19|19|3‡|51|51|
|Number of training samples per class|1000|1000|1000|1000|2975†|2975†|3712†|≈70|≈70|
|Pretrained?|N|N|N|N|N|N|N|N|Y|
|Main metric Mean Standard Deviation Prediction Loss|top1 Acc 82.21 2.64 0.56|top1 Acc 48.59 3.69 1.07|top1 Acc 85.70 1.51 0.83|top1 Acc 57.07 4.51 1.05|mIoU 72.58 5.33 0.75|mIoU 62.21 3.37 0.88|car-3D AP 76.36 4.70 0.85|top1 Acc 19.56 3.69 1.07|top1 Acc 30.88 4.65 1.26|



A DETAILS OF NAS-BENCH-MR

In this section, we provide details of the datasets and settings used to build our NAS-Bench-MR. We
follow the common practice and realistic settings for training and evaluation in each task, making
the scientific insights generated by our benchmark easier to generalize to real-world scenarios.

A.1 DATASETS AND SETTINGS

We randomly sample 2,500 structures from our network coding space, and train and evaluate these
same architectures for each task and setting. Each architecture is represented by a 27-dimensional
continuous-valued code. Tab. 7 shows the representation of each dimension of the code. For 9
different datasets and proxy tasks, we train a total of 22,500 models. See below for details.

**ImageNet for Image Classification. The ILSVRC 2012 classification dataset (Deng et al., 2009)**
consists of 1,000 classes, with a number of 1.2 million training images and 50,000 validation images.

Considering the diversity of the number of classes and data scales in practical applications, and saving training time, we construct three subsets: ImageNet-50-1000, ImageNet-50-100, and ImageNet10-1000, where the first number indicates the number of classes and the second one denotes the
number of images per class.

In this work, we adopt an SGD optimizer with momentum 0.9 and weight decay 1e-4. The input
size is 224 × 224. The initial learning rate is set to 0.1 with a total batch size of 160 on 2 Tesla
V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0. We
adopt the basic data augmentation scheme to train the classification models, i.e., random resizing
and cropping, and random horizontal flipping (flip ratio 0.5), and use single-crop for evaluation.

We report the top-1 and top-5 accuracies as the evaluation metric on all three benchmark datasets.
We print the training log (top-1, top-5, loss, lr) every 20 iterations and evaluate the model every 5
epochs on the ImageNet validation set. We will release four checkpoints of 25, 50, 75, 100 epochs
with the optimizer data and all the training logs for each model.

**Cityscapes for Semantic Segmentation. The Cityscapes dataset (Cordts et al., 2016) contains high-**
quality pixel-level annotations of 5000 images with size 1024 × 2048 (2975, 500, and 1525 for the
training, validation, and test sets respectively) and about 20000 coarsely annotated training images.
Following the evaluation protocol (Cordts et al., 2016), 19 semantic labels are used for evaluation
without considering the void label.


-----

Table 7: Representations of our 27-dimensional coding.

|Component|Codes N N N blocks residual units channels|
|---|---|


|Input layers Stage 1 Stage 2 Stage 3 Stage 4 Output layer|b 1 b 2 b 3 b 4|n1 1 n1, n2 2 2 n1, n2, n3 3 3 3 n1, n2, n3, n4 4 4 4 4|i, i 1 2 c1 1 c1, c2 2 2 c1, c2, c3 3 3 3 c1, c2, c3, c4 4 4 4 4 o 1|
|---|---|---|---|



To study the effect of different image resolutions in segmentation, we conduct two benchmarks with
the input size of 512 × 1024 and 128 × 256, respectively. For the small-resolution setting, we
pre-resize the Cityscapes dataset to 256 × 512 before the data augmentation.

In this work, we use an SGD optimizer with momentum 0.9 and weight decay 4e-5. The initial
learning rate is set to 0.1 with a total batch size of 64 on 8 Tesla V100 GPUs for 25000 iterations
(about 537 epochs). Follows the common practice in (Zhao et al., 2017; Wang et al., 2020a; Ding
et al., 2020b), the learning rate and momentum follow the “poly” scheduler with power 0.9 and a
minimum learning rate of 1e-4. We use basic data augmentation, i.e., random resizing and cropping,
random horizontal flipping, and photometric distortion for training and single-crop testing with the
test size of 1024 × 2048 and 256 × 512 respectively for two settings.

We report the mean Intersection over Union (mIoU), mean (macro-averaged) Accuracy (mAcc),
and overall (micro-averaged) Accuracy (aAcc) as the evaluation metrics. We print the training log
(mAcc, loss, lr) every 50 iterations and evaluate the model every 5000 iterations on the Cityscapes
validation set. We will release five checkpoints of 5000, 10000, 15000, 20000, 25000 iterations with
the optimizer data and all the training logs for each model.

**KITTI for 3D Object Detection. The KITTI 3D object detection dataset (Geiger et al., 2012) is**
widely used for monocular and LiDAR-based 3D detection. It consists of 7,481 training images
and 7,518 test images as well as the corresponding point clouds and the calibration parameters,
comprising a total of 80,256 2D-3D labeled objects with three object classes: Car, Pedestrian, and
Cyclist. Each 3D ground truth box is assigned to one out of three difficulty classes (easy, moderate,
hard) according to the occlusion and truncation levels of objects.

In this work, we follow the train-val split (Chen et al., 2015; Ding et al., 2020a), which contains
3,712 training and 3,769 validation images. The overall framework is based on Pointpillars (Lang
et al., 2019). The input point points are projected into bird’s-eye view (BEV) feature maps by a
voxel feature encoder (VFE). The projected BEV feature maps (496 × 432) are then used as the
input of our 2D network for 3D/BEV detection.

Following (Lang et al., 2019), we set, pillar resolution: 0.16m, max number of pillars: 12000,
and max number of points per pillar: 100. We apply the same data augmentation, i.e., random
mirroring and flipping, global rotation and scaling, and global translation for 3D point clouds as in
Pointpillar (Lang et al., 2019). We use the one-cycle scheduler with an initial learning rate of 2e-3,
a minimum learning rate of 2e-4, and batch size 16 on 8 Tesla V100 GPUs for 80 epochs. We use
an AdamW optimizer with momentum 0.9 and weight decay 1e-2. At inference time, axis-aligned
non-maximum suppression (NMS) with an overlap threshold of 0.5 IoU is used for final selection.

We report standard average precision (AP) on each class as the evaluation metric. We will release
five checkpoints of 76, 77, 78, 79, 80 epochs with the optimizer data and detailed evaluation results
(Car/Pedestrian/Cyclist easy/moderate/hard 3D/BEV/Orientation detection AP) of each checkpoint
for each model. Due to the unstable training of the KITTI dataset, we provide the last five checkpoints for researchers to tune hyperparameters from different criteria, such as the single model best
performance, the average best performance, and the last epoch best performance.

**HMDB51 for Video recognition.** We train video recognition models on the HMDB51
dataset (Kuehne et al., 2011), consisting of 6,766 videos with 51 categories. We use the first training and validation split composing of 3570 and 1530 videos for evaluation, respectively. The video
containing less than 64 frames is filtered, resulting in 2649 samples for training.

Considering the difficulty of learning temporal information, in addition to training from scratch,
we also conduct training using models of ImageNet-50-1000 as pretraining. To model the temporal


-----

Correlation of epoch numbers


1.0

0.9

0.8

0.7

0.6

0.5


2500

2000

1500

1000

500


2500

1000

500

0

0 500 1000 1500 2000 2500

Car-3D
Car-BEV
Ped-3D
Ped-BEV

Architecture ranking under 3dDet metrics


0 500 1000 1500 2000 2500

mIoU
mAcc
aAcc

Architecture ranking under Seg metrics

|100|90|50|10|10-C|
|---|---|---|---|---|
|1.000|0.970|0.838|0.451|0.514|
|0.970|1.000|0.843|0.460|0.526|
|0.838|0.843|1.000|0.535|0.577|
|0.451|0.460|0.535|1.000|0.738|
|0.514|0.526|0.577|0.738|1.000|


1.000 0.970 0.838 0.451 0.514

0.970 1.000 0.843 0.460 0.526

0.838 0.843 1.000 0.535 0.577

0.451 0.460 0.535 1.000 0.738

0.514 0.526 0.577 0.738 1.000


(a) (b) (c)

Figure 5: (a) Spearman’s rank correlation of the different number of epochs (checkpoints of 10, 50,
90, 100 epochs during training on ImageNet-50-1000). ‘10-C’ denotes using the convergent learning
rate for 10 epochs, i.e., the proxy setting used in RegNet (Radosavovic et al., 2020). (b) Architecture
rankings under the evaluation metrics of semantic segmentation (mIoU, mAcc, aAcc). (c) Architecture rankings under the evaluation metrics of 3D object detection (car/pedestrian 3D/bird’s-eye view
detection AP).

information, we replace the last 2D convolutional layer before the final classification layer of HRNet
and its counterparts with a 3D convolutional layer.

In this work, the input size is set to 112 × 112. We adopt an Adam optimizer with momentum 0.9
and weight decay 1e-5. The initial learning rate is set to 0.01 with a total batch size of 80 on 4 Tesla
V100 GPUs for 100 epochs, and decays by cosine annealing with a minimum learning rate of 0. We
adopt random resizing and cropping, random brightness 0.5, random contrast 0.5, random saturation
0.5, random hue 0.25, and random grayscale 0.3 as data augmentation.

We report the top-1 and top-5 accuracies as the evaluation metric on both two settings. We print the
training log (top-1, top-5, loss, lr) every 10 iterations and evaluate the model every 10 epochs on the
validation set. We will release the last checkpoint and all the training logs for each model.

A.2 ANALYSIS OF NAS-BENCH-MR

We summarize detailed statistics of our NAS-Bench-MR in Tab. 6. We also give the mean, std, and
the validation L1 loss (%) of our neural predictors under the main evaluation metric of each setting.

From Spearman’s rank correlation of nine benchmarks in our main paper, we observe that:

(1) The four main tasks have different preferences for the architecture, and their correlation coefficients are between -0.003 (segmentation and video recognition) and 0.639 (segmentation and
classification). This may be because segmentation can be seen as per-pixel image classification.

(2) Different settings in the same task also have different preferences for the architecture, and their
correlation coefficients are between 0.3 (Cls-50-1000 and Cls-50-100) and 0.818 (Cls-50-100 and
Cls-10c). From Fig. 5 (a) we also see that different training epochs result in a significant performance change. In this way, the proxy 10-epoch training setting (Radosavovic et al., 2020) may not
generalize well to real-world scenarios.

(3) The correlation coefficient matrix is related to the model performance of multi-task joint optimization using NP. The higher the correlation coefficient of the two tasks, the greater the gain of
joint optimization. If the correlation coefficient of the two tasks is too low, joint optimization may
reduce the performance on both tasks.

(4) We show the architecture ranking under different metrics for different tasks in Fig. 5 (b) and (c).
We noticed that the three metrics (i.e., mIoU, mAcc, aAcc) in segmentation are positively correlated,
which means only one (e.g., mIoU) needs to be optimized to obtain a good model under all three
metrics. However, in the 3D object detection task, different metrics are not necessarily related,
which makes multi-objective optimization especially important.


-----

Table 8: Comparative results (%) of NCP on the Cityscapes validation set. The first two architectures are searched using neural predictors that are trained on the Cityscapes dataset with an input
size of 512×1024 (Seg) and the resized Cityscapes dataset with an input size of 128×256 (Seg-4x),
respectively. The last model is searched by joint optimization of both the two predictors. λ is set to
0.5 during the network codes propagation process. All models are trained from scratch. FLOPs is

|measured using 512 × 1024.|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
|Method|Params FLOPs|Cityscapes 512 × 1024 mIoU mACC aACC|Cityscapes 128 × 512 mIoU mACC aACC|ADE20K 512 × 512 mIoU mACC aACC|
|ResNet18-PSP (Zhao et al., 2017) ResNet34-PSP (Zhao et al., 2017) HRNet-W18s (Wang et al., 2020a) HRNet-W18 (Wang et al., 2020a)|12.94M 108.53G 23.05M 193.12G 5.48M 22.45G 9.64M 37.01G|73.21 79.86 95.51 76.17 82.66 95.99 76.21 84.43 95.86 77.73 85.61 96.20|62.66 70.17 93.25 64.17 72.37 93.48 64.49 73.58 93.71 65.19 74.85 93.66|33.45 40.95 77.78 32.78 41.23 77.52 34.39 44.65 77.60 34.84 45.27 77.80|
|NCP-Net-512 × 1024 (Fig. 19) NCP-Net-128 × 512 (Fig. 20) NCP-Net-Both (Fig. 21)|7.91M 29.34G 2.61M 31.37G 7.82M 50.90G|77.15 84.73 96.01 70.91 80.20 95.21 77.35 84.65 96.23|62.40 71.70 93.34 65.89 75.19 93.79 64.98 73.82 93.83|33.32 42.97 77.41 27.52 35.88 73.85 35.65 45.64 77.94|



Table 9: Video recognition results (%) of NCP on the HMDB51 dataset. The first two models are
searched using neural predictors that are trained on HMDB51 from scratch (Video) and using models
of ImageNet-50-1000 as pretraining (Video-p). The last model is searched by joint optimization
of both two predictors. We also show the classification accuracy of the pretrained model on the
ImageNet-50-1000 dataset (the column of “Cls-Pretraining”). Note that the last 2D convolutional
layer before the final classifier is replaced with a 3D convolutional layer (3 × 3 × 3) for all models

|to capture the temporal inform 112 × 112.|mation. λ is set to 0|0.5. FLOPs is c|× × calculated using|× g an input size of|
|---|---|---|---|---|
|Method|Params FLOPs|Video-Scratch top1 top5|Cls-Pretraining top1 top5|Video-Pretrained top1 top5|
|ResNet18 (He et al., 2016) ResNet34 (He et al., 2016) HRNet-W18s (Wang et al., 2020a) HRNet-W18 (Wang et al., 2020a)|12.10M 0.59G 22.21M 1.20G 14.35M 0.86G 20.05M 1.41G|21.50 54.93 19.89 52.94 24.55 55.87 24.11 55.33|82.72 93.88 83.40 94.28 83.04 95.16 83.48 94.04|27.84 60.50 31.22 63.53 30.96 61.21 32.48 64.30|
|NCP-Net-Scratch (Fig. 22) NCP-Net-Pretrained (Fig. 23) NCP-Net-Both (Fig. 24)|2.56M 0.69G 1.72M 1.00G 2.39M 1.16G|28.47 62.18 27.49 62.01 27.14 62.28|81.88 93.84 82.48 94.80 83.92 94.76|37.54 68.50 39.14 69.66 40.21 70.29|



B INTRA-TASK GENERALIZABILITY

In this section, we explore the effect of multiple proxy settings such as different input sizes (512 ×
1024 and 128 × 256), and pretraining strategies (pretraining from classification or training from the
scratch for video recognition) in designing network architectures. We search architectures under
single (e.g., 512 × 1024) or multiple (e.g., both resolutions) to validate the effectiveness of NCP for
different intra-task settings.

Tab. 8 and 9 show the intra-task cross-setting generalizability of our searched NCP-Net and some
manually-designed networks, such as ResNet (He et al., 2016) and HRNet (Wang et al., 2020a), on
the Cityscapes dataset and the HMDB51 dataset, respectively. We see that:

(1) Generally, the network searched on a specific setting performs the best under this setting but
poor under other settings, which means that NCP can optimize and customize model structures
for a specific setting. For example, Tab. 8 shows NCP customizing structures for different input
resolutions on Cityscapes, which is essential for practical applications in real-world scenarios.

(2) Benefiting from the high correlation between different settings of the same task, joint optimization of multiple settings within the same task (for both segmentation and video recognition) usually
has a positive effect, i.e., improving the performance on each setting, at the cost of the increasing
model size (larger FLOPs). However, different tasks may not necessarily correlate, e.g.segmentation
+ video recognition. This can be verified by Spearman’s correlation, as discussed in our main paper.


-----

(3) In Tab. 8, we apply the searched segmentation network to the ADE20K dataset (Zhou et al.,
2017) to show the generalizability of the searched architectures. The network trained with a large
input resolution (512×1024) has better performance than the network trained with a small resolution
(128 × 256).

Moreover, the jointly searched architecture using both resolutions shows better generalizability than
those two with a single objective. It also demonstrated that the network searched on our NAS-BenchMR has stronger transfer capability to real-world scenarios, compared to the previous benchmarks
such as (Dong & Yang, 2020; Ying et al., 2019) that uses a small input size.

(4) For both the segmentation and the video recognition tasks, our joint searched networks outperform manually-designed networks, such as HRNet (Wang et al., 2020a) and ResNet (He et al., 2016),
and achieve better generalizability to other settings and datasets, e.g., ADE20K.

C NCP ON NAS-BENCH-201


In this section, we apply our NCP to the NAS-Bench-201 benchmark (Dong & Yang, 2020) to show
its effectiveness. We first briefly introduce NAS-Bench-201 and then state the difference between
our NAS-Bench-MR and NAS-Bench-201. Lastly, we detail the experiment.

C.1 NAS-BENCH-201


NAS-Bench-201 employs a DARTS-like (Liu
et al., 2019b) search space including three
stacks of cells, connected by a residual
block (He et al., 2016). Each cell is stacked
_N = 5 times, with the number of output chan-_
nels as 16, 32, and 64 for the first, second,
and third stages, respectively. The intermediate
residual block is the basic residual block with
a stride of 2. There are 6 searching paths in
the space of NAS-Bench-201, where each path
contains 5 options: (1) zeroize, (2) skip connection, (3) 1-by-1 convolution, (4) 3-by-3 convolution, and (5) 3-by-3 average pooling layer,
resulting in 15,625 different models in total.

NAS-Bench-201 train these models on the
ImageNet-16 (with an input size of 16×16) and
Cifar10 (with an input size of 32 _×_ 32) datasets.
Since the ImageNet dataset is closer to practical
applications, we use the models in ImageNet16 as a comparison in the following section.

C.2 DIFFERENCES


46

44

Acc (%) 4240

2

0

80 100 120 140 160

FLOPs


Figure 6: Visualization of the architecture propagation process on the NAS-Bench-201 benchmark (Dong & Yang, 2020) (ImageNet-16). ⋆
represents the propagated model in each step.
NCP finds the optimal structure from a low-Acc
and high-FLOPs starting point with only 6 steps
by optimizing accuracy and FLOPs constraints simultaneously.


There are some differences between the NAS-Bench-201 benchmark and our NAS-Bench-MR.

(1) NAS-Bench-201 and NAS-Bench-MR are of different magnitudes (15,625 v.s. 10[23]). (2) The
number of channels/blocks/resolutions is fixed in NAS-Bench-201, while it is searchable in our
search space. In this way, our search space is more fine-grained and suitable for customizing to
tasks with different preferences (high- and low-level features, deeper and shallower networks, wider
and narrower networks, etc.). (3) The architecture in NAS-Bench-201 is represented as a one-hot
encoding (choose one of five operators). While in our NAS-Bench-MR, the continuous-valued code
is more appealing to the learning of the neural predictor. (4) NAS-Bench-201 is built based on a
single setting while NAS-Bench-MR contains multiple settings.

We then conduct experiments and show that despite the above many differences, NCP works well
on NAS-Bench-201, demonstrating the generalization ability of NCP to other benchmarks.


-----

86

NCP Net(Ours)

Neural Predictor

85

ResNet101

HRNet-W48

84 HRNet-W32

Acc (%) ResNet50

ResNet34 HRNet-W18

83 HRNet-W18s

ResNet18

82

0 2 4 6 8 10 12 14 16 18

FLOPs (G)

Image Classification

NCP Net(Ours)

78

Pointpillar

HRNet-W32

76

74 HRNet-W18 ResNet50

mIOU (%)

72

HRNet-W18s

70 ResNet34

ResNet18

0 20 40 60 80 100 120 140 160 180

FLOPs (G)

3D Detection

NCP Net L(Ours)

80 HRNet-W32 Auto-DeepLab

HRNet-W48

78 NCPHRNet-W18Net M(Ours)

NCP Net S(Ours) ResNet101-PSP

mIOU (%)76 HRNet-W18s

ResNet34-PSP ResNet50-PSP

74

ResNet18-PSP

72

0 100 200 300 400 500

FLOPs (G)

Semantic Segmentation

30 NCP Net M(Ours)

NCP Net S(Ours)

26 HRNet-W18s NCP Net L(Ours)

HRNet-W18

HRNet-W32

22 ResNet18

Acc (%)

ResNet34

HRNet-W48

18

ResNet50
ResNet101

14

0 1 2 3 4 5

FLOPs (G)

Video Recognition

Figure 7: Comparisons of the efficiency (i.e., FLOPs) and the performance (e.g., Acc, mIoU,
AP) on four computer vision tasks, i.e., image classification (ImageNet), semantic segmentation
(CityScapes), 3D detection (KITTI), and video recognition (HMDB51) between the proposed approach and existing methods. Each method is represented by a circle, whose size represents the
number of parameters. ⋆ represents the optimal model with both high performance and low FLOPs.
Our approach achieves superior performance compared to its counterparts on all four tasks.

C.3 EXPERIMENTS


In this work, we formulate each architecture in NAS-Bench-201 to a 6∗5 = 30-dimensional one-hot
encoding. We then use our continuous network propagation strategy to traverse architectures in the
space with higher-Acc and lower-FLOPs as optimization goals. Given an initial one-hot encoding,
our NCP treats it as a continuous-valued coding and utilizes the argmax operation to obtain an
edited one-hot coding after every several iterations.

In the experiment, we use the argmax operation after every 10 iterations, termed as a step. As shown
in Fig. 6, our NCP finds the optimal structure from a low-Acc and high-FLOPs starting point with
only 6 steps (60 iterations). The entire search process takes less than 10 seconds. Compared to other
neural predictor-based methods such as (Wen et al., 2020; Luo et al., 2020) that need to predict the
accuracy of a large number of architectures, NCP is more efficient.

From Fig. 6 we observe that from the second to fourth steps, the performance of the searched model
vibrates. This may be because of the instability caused by one-hot encoding. NCP finds an optimal
model with an accuracy of 46.8 and FLOPs of only 90.36 (the highest accuracy in NAS-Bench-201
is 47.33%), showing its effectiveness and generalizability.

D NCP ON DIFFERENT SINGLE TASKS


We also use NCP to find the optimal architecture on the four basic vision tasks, i.e., image classification, semantic segmentation, 3D detection, and video recognition. Quantitative comparisons
in Fig. 7 show that the architectures found by NCP outperform well-designed networks, such as
HRNet (Wang et al., 2020a) and ResNet (He et al., 2016).

We visualize the searched architectures by NCP in Fig. 8 to show it can customize different representations for different tasks (objectives). We see that:


-----

128

96
64
32

0

In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Cls


128

96
64
32

0

128

96
64
32


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Seg

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|Col14|Col15|Col16|Col17|Block ResUnit|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|



In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

3dDet


128

96
64
32

0

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Block ResUnit|
|---|---|---|---|---|---|---|---|---|---|---|---|


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Video


Figure 8: Visualization of the searched models by NCP for four different tasks (λ = 0.5). The
27-dimensional array in each row represents a network structure.

**Algorithm 1 The network propagation process.**


1. Learn a neural predictor FW (·) and fix it;
2. Initialize an architecture code e;
3. Set the target metrics, such as tacc and tflops;
**for each iteration do**

4. Re-set new target metrics (optional);
5. Forward FW (e) and calculate the loss;
6. Back-propagate and calculate the gradient ∆e;
7. Select an optimal dimension l (optional);
8. update e based on the gradient ∆e;

**end for**
9. Round and decode e to obtain the final architecture.


(1) Segmentation requires the most high-level information in the last two branches of the last stage,
and the video recognition model contains the least low-level semantics in the first two stages.

(2) The classification model contains more channels than other tasks, because the FLOPs constraint
for classification is the weakest, i.e., classification costs fewer FLOPs than segmentation under the
same architecture.

(3) The 3D detection model mainly utilizes the first two branches, which means it may rely more on
high-resolution representations.

E DETAILS OF NCP


In this section, we provide supplementary details and complexity analysis of our NCP. The algorithm
of Network Coding Propagation (NCP) is shown in Algorithm 1. Code is available.

E.1 IMPLEMENTATION DETAILS


In this work, for each task, 2000 and 500 structures in the benchmark are used as the training set and
validation set to train the neural predictor. The optimization goal is set to higher performance (main
evaluation metric of each task) and lower FLOPs.

Intuitively, large models can achieve moderate performance on multiple tasks by stacking redundant structures. On the contrary, the lightweight model tends to pay more attention to task-specific
structural design due to limited computing resources, making it more suitable for generalizability
evaluation. To this end, we set λ = 0.5 to obtain lightweight models unless specified.

We employ a three-layer fully connected network with a dropout ratio of 0.5 as the neural predictor.
The first layer is a shared layer for all metrics with a dimension of 256. The second layer is a
separated layer for each metric with a dimension of 128. Then for each evaluation metric, a 128by-1 fully-connected layer is used for final regression. During training, we use an Adam optimizer
with a weight decay of 1e-7. The initial learning rate is set to 0.01 with batch size 128 on a single
GPU for 200 epochs, and decays by the one cycle scheduler. The metric prediction is learned
using the smoothL1 loss. In the network propagation process, unless specified, we use continuous
propagation with an initial code of {b, n = 2; c, i, o = 64} and learning rate of 3 for 70 iterations in
all experiments.


-----

Table 10: Searching time of predictor-based searching methods. Our NCP is the fastest as it evaluates all dimensions of the code with only one back-propagation without top-K ranking. The time is
measured on a Tesla V100 GPU.

Model Searching Time Notes

Predict the validation metric of 10,000 random architeNeural Predictor Wen et al. (2020) _> 1 GPU day_
ctures and train the top-10 models for final evaluation.

Traverse each dimension of the code, predict its per-formance,
NetAdapt Yang et al. (2018) 5min
and then edit the dimension with the highest accuracy improvement.

Maximize the evaluation metrics along the gradient dirNCP (Ours) 10s (70 iterations)
ections by propagating architectures in the search space.

E.2 TIME COMPLEXITY

Tab. 10 shows the searching time of three predictor-based searching methods. NCP is the fastest as
it traverses all dimensions of the code with only one back-propagation without the top-K ranking.

Suffering from the random noise in random search and the neural predictor, existing methods (Wen
et al., 2020; Luo et al., 2020) often need to train the top-K models for final evaluation, which is
costly (e.g., training a segmentation model on Cityscapes costs 7 hours on 8 Tesla V100 GPUs).
NCP uses gradient directions as guidance to alleviate the randomness issue.

F VISUALIZATION AND ANALYSIS

F.1 ANALYSIS OF TASK TRANSFERRING

We visualize the network coding propagation process of our cross-task architecture transferring, e.g.,
an architecture transferring from the classification task to the segmentation task by using the optimal
coding on classification as initialization and using the neural predictor trained on segmentation for
optimization. The detailed transferring visualizations of every two of the four tasks are shown in
Fig. 9-12 with many interesting findings. For example, all networks in segmentation try to increase
the number of channels of the 4th branch of stage 4, while the networks in classification try to
decrease it; the networks in video recognition keep it at an intermediate value.

F.2 ANALYSIS OF SINGLE- AND MULTI-TASK SEARCHING

We visualize the searched architectures (Fig. 13-14) and the network propagation process of each
architecture (Fig. 19-24) in Tab. 8-9.

**Classification. Manually-designed classification models often use gradually decreasing resolution**
and gradually increasing the number of channels, because intuitively the learning of classification
task requires low-resolution high-level semantic information. However, under different data scales
and number of classes, the situation is different, as shown in Fig. 15-18. For example, by reducing the training samples from 1000 to 100 (from Fig. 15 to Fig. 16), the number of convolutional
channels and the number of residual units in the 1st branch of stage 4 are increased during propagation, showing the high-resolution low-level information is important when the training samples are
insufficient.

**Segmentation.** From Fig 19, Fig 20, and Fig 21 we observe the differences of the network editing
process of segmentation models. For example, compared to the two models optimized at a single
resolution, the model optimized at both input resolutions reduces the numbers of residual units of
the first two stages and increases the number of blocks of stage 2, resulting in more fusion modules
and fewer parallel units.

**Segmentation and Video Recognition.** From the visualization of found architecture codes in
Fig. 13 and 14, we can find, it is not always that the larger the model, the better the performance.
Different objectives result in different customized architecture codes. For example, segmentation
tends to select architectures with fewer input channels, while video recognition architectures often
have more output channels.


-----

76 4 128

74 0 20 Predicted mIoU (%)40 60 3 10480

21 0 20 40 GFLOPs60 21 0 20 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks40 60 56328 0 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels20 40 60

1281048056 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch2 1281048056 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch3 1281048056 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch4

32 32 32

8 0 20 40 60 8 0 20 40 60 8 0 20 40 60

43 43 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 43 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

21 0 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch220 Iterations40 60 21 0 Cls20 IterationsSeg40 60 21 0 20 Iterations40 60


272625 0 20 Predicted Acc (%)40 60 43 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 12810480

21 0 20 40 GFLOPs60 21 0 20 40 60 56328 0 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels20 40 60

1281048056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 1281048056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 1281048056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Cls20 IterationsVideo40 60 1 0 20 Iterations40 60


77.577.0 0 Predicted Car-3D AP (%)20 40 60 43 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 12810480

2.52.01.5 0 20 40 GFLOPs60 21 0 20 40 60 56328 0 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels20 40 60

128 128 128

104 104 104

8056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 8056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 8056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Cls20 Iterations3dDet40 60 1 0 20 Iterations40 60


Figure 9: Visualization of our network propagation process of the optimal model in classification
transferring to other three tasks (λ = 0.5).

|SSStttaaagggeee122::: NNN___uuu|nnniiitttsss iiinnn bbbrrraaannnccchhh112|
|---|---|

|Stage3: N_unit|s in branch3|
|---|---|

|SSSStttt|aaaaggggeeee4444:::: NNNN____uuuu|nnnniiiittttssss iiiinnnn bbbbrrrraaaannnncccchhhh|
|---|---|---|

|Stage2: N_u|nits in branch2|
|---|---|

|St|age3: N_u|nits in bra|nch3|
|---|---|---|---|

|SSSStttt|aaaaggggeeee4444:::: NNNN____uuuu|nnnniiiittttssss iiiinnnn bbbbrrrraaaannnncccchhhh1234|
|---|---|---|

|SSStttaaagggeee122::: NNN___uuu|nnniiitttsss iiinnn bbbrrraaannnccchhh112|
|---|---|

|St|age3: N_u|nits in branch3|
|---|---|---|

|SStt|aaggee44:: NN__uu|nniittss iinn bbrraanncchh|
|---|---|---|


85.082.5 0 20 Predicted Acc (%)40 60 43 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 12810480

21 0 20 40 GFLOPs60 21 0 20 40 60 56328 0 20 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels40 60

128 128 128

104 104 104

8056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 8056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 8056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Seg20 Iterations40Cls 60 1 0 20 Iterations40 60


252015 0 20 Predicted Acc (%)40 60 432 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 1281048056 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels

0.750.50 0 20 40 GFLOPs60 1 0 20 40 60 328 0 20 40 60

12810480 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch2 12810480 12810480

56328 0 20 40 60 56328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 56328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Seg20 IterationsVideo40 60 1 0 20 Iterations40 60


7876 0 20Predicted Car-3D AP (%)40 60 43 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 12810480 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels

1.251.00 GFLOPs 2 5632

0.75 0 20 40 60 1 0 20 40 60 8 0 20 40 60

1281048056 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch2 1281048056 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch3 1281048056 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch4

32 32 32

8 0 20 40 60 8 0 20 40 60 8 0 20 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Seg20 Iterations3dDet40 60 1 0 20 Iterations40 60


Figure 10: Visualization of our network propagation process of the optimal model in segmentation
transferring to other three tasks (λ = 0.5).

|SSStttaaagggeee122::: NNN___uuu|nnniiitttsss iiinnn bbbrrraaannnccchhh112|
|---|---|


|SSStttaaagggeee333::: NNN___uuunnniiittt|sss iiinnn bbbrrraaannnccchhh123|
|---|---|


|SSSSttttaaaagggg|eeee4444:::: NNNN____uuuunnnniiiitttt|ssss iiiinnnn bbbbrrrraaaannnncccchhhh1234|
|---|---|---|


|Stage2: N_u|nits in branch2|
|---|---|


|St|age3: N_u|nits in bra|nch3|
|---|---|---|---|


|SSSSttttaaaagggg|eeee4444:::: NNNN____uuuunnnniiiitttt|ssss iiiinnnn bbbbrrrraaaannnncccchhhh1234|
|---|---|---|


|Stage2: N_unit|s in branch2|
|---|---|


|Stag|e3: N_unit|s in branch3|
|---|---|---|


|SSSSttttaaaagggg|eeee4444:::: NNNN____uuuunnnniiiitttt|ssss iiiinnnn bbbbrrrraaaannnncccchhhh1234|
|---|---|---|


85.082.5321 00 2020 Predicted Acc (%)4040 GFLOPs6060 4321 0 20 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks40 60 1281048056328 0 20 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels40 60

128 128 128

104 104 104

8056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 8056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 8056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

4 4 4

32 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 32 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 32 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Video20 Iterations40 Cls60 1 0 20 Iterations40 60

75.072.51.51.0 00 2020 Predicted mIoU (%)4040 GFLOPs6060 4321 0 20 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks40 60 1281048056328 0 20 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels40 60

1281048056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 1281048056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 1281048056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Video20 Iterations40Seg60 1 0 20 Iterations40 60

1.251.007876 0 GFLOPs20Predicted Car-3D AP (%)40 60 432 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 128104805632 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels

0.75 0 20 40 60 1 0 20 40 60 8 0 20 40 60

128104 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch2 128104 128104

80 80 80

56328 0 20 40 60 56328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 56328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 Video20 Iterations3dDet40 60 1 0 20 Iterations40 60


Figure 11: Visualization of our network propagation process of the optimal model in video action
**recognition transferring to other three tasks (λ = 0.5).**




85.082.5 0 20 Predicted Acc (%)40 60 43 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 12810480

2.52.01.5 0 20 40 GFLOPs60 21 0 20 40 60 56328 0 20 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels40 60

128 128 128

1048056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 1048056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 1048056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

432 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 432 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 432 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

1 0 20 Iterations40 60 1 0 3dDet20 Iterations40 Cls60 1 0 20 Iterations40 60


75 4 128

1.51.070 00 2020 Predicted mIoU (%)4040 GFLOPs6060 321 0 20 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks40 60 1048056328 0 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels20 40 60

1281048056328 0 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch220 40 60 1281048056328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 1281048056328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

4321 0 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch220 Iterations40 60 4321 0 3dDetStage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch320 Iterations40 Seg60 4321 0 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch420 Iterations40 60


1.00272625 0 20 Predicted Acc (%)40 GFLOPs60 432 Stage1: N_blocksStage2: N_blocksStage3: N_blocksStage4: N_blocks 128104805632 In-layers: N_channels0In-layers: N_channels1Out-layer: N_channels

0.75 0 20 40 60 1 0 20 40 60 8 0 20 40 60

128104 Stage1: N_channels in branch1Stage2: N_channels in branch1Stage2: N_channels in branch2 128104 128104

80 80 80

56328 0 20 40 60 56328 0 Stage3: N_channels in branch1Stage3: N_channels in branch2Stage3: N_channels in branch320 40 60 56328 0 Stage4: N_channels in branch1Stage4: N_channels in branch2Stage4: N_channels in branch3Stage4: N_channels in branch420 40 60

43 Stage1: N_units in branch1Stage2: N_units in branch1Stage2: N_units in branch2 43 Stage3: N_units in branch1Stage3: N_units in branch2Stage3: N_units in branch3 43 Stage4: N_units in branch1Stage4: N_units in branch2Stage4: N_units in branch3Stage4: N_units in branch4

2 2 2

1 0 20 Iterations40 60 1 0 3dDet20 Iterations40Video60 1 0 20 Iterations40 60


Figure 12: Visualization of our network propagation process of the optimal model in 3D object
**detection transferring to other three tasks (λ = 0.5).**


-----

128

96
64

Channel 32

0

128

96
64

Channel 32


128

96
64

Channel 32

0

128

96
64

Channel 32

|Col1|Col2|Channel Block ResUnit|
|---|---|---|

|Col1|Col2|Col3|Col4|Channel Block ResUnit|
|---|---|---|---|---|


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Seg-512x1024

In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Seg-128x512


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Video-Scratch

In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Video-Pretrained


128

96
64

Channel 32


128

96
64

Channel 32

|Col1|Col2|Col3|Col4|Channel Block ResUnit|
|---|---|---|---|---|
||||||

|Col1|Col2|Col3|Col4|Col5|Channel Block ResUnit|
|---|---|---|---|---|---|


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Seg-Both

Figure 13: Visualization of the searched segmen**tation models in Tab. 8 by our NCP for intra-task**
generalizability (λ = 0.5). The 27-dimensional
array in each row represents a network structure.


In-layers Stage1 Stage2 Stage3 Stage4 Out-layer

Channel
Block
ResUnit

Video-Both

Figure 14: Visualization of the searched video
**recognition models in Tab. 9 by our NCP for**
intra-task generalizability (λ = 0.5). The 27dimensional array in each row represents a network structure.


128

104

80

56

32

8

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

32


128

104

80

56

32


86

84


55

50


20 40 60

Predicted Acc (%)


20 40 60


50 100 150 200

Predicted Acc (%)


50 100 150 200

|Col1|Stage2: N_block Stage3: N_block Stage4: N_block|
|---|---|

|In-layers: N_ch In-layers: N_ch|annels0 annels1|Col3|
|---|---|---|

|Col1|S S S|tage2: tage3: tage4:|N_blocks N_blocks N_blocks|
|---|---|---|---|

|In-lay In-lay Out-l|ers: N_c ers: N_c ayer: N_c|hannels0 hannels1 hannels|
|---|---|---|


20 40 60

|Col1|Col2|
|---|---|



20 40 60


20 40 60

|Stage4: N_chan Stage4: N_chan Stage4: N_chan Stage4: N_chan|nels in bra nels in bra nels in bra nels in bra|nch1 nch2 nch3 nch4|
|---|---|---|


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


20 40 60


50 100 150 200

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56


50 100 150 200

|Stage4: N_c|hannels i|n branch1|
|---|---|---|



50 100 150 200


GFLOPs


GFLOPs


128

104

80

56

32


128

104

80

56

32


128

104

|Stage1: N_chan Stage2: N_chan Stage2: N_chan|nels in nels in nels in|branch1 branch1 branch2|
|---|---|---|



32

50 100 150 200 8


128

104

80

56

32

|Stage1: N_chan Stage2: N_chan Stage2: N_chan|nels in branch1 nels in branch1 nels in branch2|
|---|---|

|Col1|Col2|Col3|Col4|
|---|---|---|---|


20 40 60

|Col1|Col2|
|---|---|
|Stage1: N_u Stage2: N_u Stage2: N_u|nits in branch1 nits in branch1 nits in branch2|
|||


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


20 40 60

Iterations


50 100 150 200


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


|Stage3: N_un Stage3: N_un Stage3: N_un|its in branch1 its in branch2 its in branch3|
|---|---|
|||
|||


20 40 60

Stage3: N_units in branch1
Stage3: N_units in branch2
Stage3: N_units in branch3

Iterations


|Stage4: N_u Stage4: N_u Stage4: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|
|Stage4: N_u|nits in bra|nch4|
||||


20 40 60

Iterations


50 100 150 200

|Col1|Col2|Col3|
|---|---|---|
|Stage1: N_u Stage2: N_u Stage2: N_u|nits in nits in nits in|branch1 branch1 branch2|
||||


Iterations


50 100 150 200

|St St St|age3: N_ age3: N_ age3: N_|units in units in units in|branch1 branch2 branch3|
|---|---|---|---|
|||||
|||||


Iterations


50 100 150 200

|Stage4: N Stage4: N Stage4: N|_units in _units in _units in|branch1 branch2 branch3|
|---|---|---|
|Stage4: N|_units in|branch4|
||||


Iterations


Figure 15: Visualization of our network propagation process of “NCP-Net-A” (λ = 0.7) for classification.


Figure 16: Visualization of our network propagation process of “NCP-Net-B” (λ = 0.7) for
classification.


128

104

80

56

32


128

104

80

56

32

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

32


85

84

83


88

86


50 100 150 200

|Col1|Col2|GFLOPs|
|---|---|---|


Predicted Acc (%)


50 100 150 200


20 40 60

|S S S|tage2: tage3: tage4:|N_blocks N_blocks N_blocks|
|---|---|---|
||||

|In-lay In-lay Out-l|ers: N_c ers: N_c ayer: N_c|hannels0 hannels1 hannels|
|---|---|---|
||||

|Col1|GFLOP|
|---|---|

|Col1|S S S|tage2: N_blocks tage3: N_blocks tage4: N_blocks|
|---|---|---|
||||

|In-layer In-layer Out-lay|s: N_chan s: N_chan er: N_chan|nels0 nels1 nels|
|---|---|---|
||||


Predicted Acc (%)


50 100 150 200

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

32


50 100 150 200

|Stage4: N_ch Stage4: N_ch Stage4: N_ch Stage4: N_ch|annels in annels in annels in annels in|branch1 branch2 branch3 branch4|
|---|---|---|



50 100 150 200


20 40 60

|Col1|Col2|
|---|---|



20 40 60


20 40 60

|Col1|Col2|Col3|
|---|---|---|



20 40 60


20 40 60

|Stage4: N_cha|nnels in br|anch1|
|---|---|---|



20 40 60


128

104

80

56

32


128

104

80

56

|Stage1: N_cha|nnels in|branch|
|---|---|---|



50 100 150 200


128

104

80

56

32


128

104

80

56

32

|Stage3: N_cha Stage3: N_cha Stage3: N_cha|nnels in nnels in nnels in|branch1 branch2 branch3|
|---|---|---|


50 100 150 200


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Stage1: N_u Stage2: N_u Stage2: N_u|nits in branch1 nits in branch1 nits in branch2|
|---|---|
|||
|||


Iterations


20 40 60

|Stage Stage Stage|3: N_un 3: N_un 3: N_un|its in branch1 its in branch2 its in branch3|
|---|---|---|
||||
||||


Iterations


20 40 60

|Stage4: N_u Stage4: N_u Stage4: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|
|Stage4: N_u|nits in bra|nch4|
||||


Iterations


50 100 150 200

|Col1|Col2|Col3|
|---|---|---|
|Stage1: N_u Stage2: N_u Stage2: N_u|nits in nits in nits in|branch1 branch1 branch2|
||||


Iterations


50 100 150 200

|Stage3: N_ Stage3: N_ Stage3: N_|units in units in units in|branch1 branch2 branch3|
|---|---|---|
||||
||||


Iterations


50 100 150 200

|Stage4: N Stage4: N Stage4: N|_units in _units in _units in|branch1 branch2 branch3|
|---|---|---|
|Stage4: N|_units in|branch4|
||||


Iterations


Figure 17: Visualization of our network propagation process of “NCP-Net-C” (λ = 0.7) for
classification.


Figure 18: Visualization of our network propagation process of “NCP-Net-ABC” (λ = 0.7) for
classification.


-----

77.5

75.0

2

1

128

104

80

56

32


128

104

80

56

|Col1|St St St|age1: N_bl age2: N_bl age3: N_bl|ocks ocks ocks|
|---|---|---|---|



32

8

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

|Col1|Col2|Col3|Col4|
|---|---|---|---|


32


128

104

80

56

|St St St St|age1: N_bl age2: N_bl age3: N_bl age4: N_bl|ocks ocks ocks ocks|
|---|---|---|



32

8

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

|Stage3: N_chan Stage3: N_chan Stage3: N_chan|nels in bra nels in bra nels in bra|nch1 nch2 nch3|
|---|---|---|


32


66

64

2

1

128

104

80

56

32


20 40 60

Predicted mIoU (%)


20 40 60

|Col1|Col2|Col3|
|---|---|---|


GFLOPs


20 40 60


20 40 60

Predicted mIoU (%)


20 40 60

|Stage Stage Stage|1: N_chann 2: N_chann 2: N_chann|els in bra els in bra els in bra|nch1 nch1 nch2|
|---|---|---|---|


GFLOPs


20 40 60

|S S S|tage1: N_u tage2: N_u tage2: N_u|nits in bra nits in bra nits in bra|nch1 nch1 nch2|
|---|---|---|---|



20 40 60

Iterations

|In-layer In-layer Out-lay|s: N_chan s: N_chan er: N_chan|nels0 nels1 nels|
|---|---|---|


20 40 60

20 40 60

Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


20 40 60

|St St St|age3: N_u age3: N_u age3: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|---|


Iterations


20 40 60

|Stage4: N_cha|nnels in br|anch1|
|---|---|---|


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Stage4: N_u Stage4: N_u Stage4: N_u Stage4: N_u|nits in bra nits in bra nits in bra nits in bra|nch1 nch2 nch3 nch4|
|---|---|---|


Iterations


20 40 60

20 40 60

|Stage3: N_u Stage3: N_u Stage3: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|


Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


20 40 60

Iterations


20 40 60

|Stage Stage Stage Stage|4: N_chan 4: N_chan 4: N_chan 4: N_chan|nels in bra nels in bra nels in bra nels in bra|nch1 nch2 nch3 nch4|
|---|---|---|---|


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


20 40 60

|Stag Stag|e4: N_unit e4: N_unit|s in branc s in branc|h3 h4|
|---|---|---|---|


Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

Stage4: N_units in branch1
Stage4: N_units in branch2
Stage4: N_units in branch3
Stage4: N_units in branch4

Iterations


128

104

80

56

32


128

104

80

56

32


Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2


20 40 60

|Col1|Col2|Col3|
|---|---|---|


Stage1: N_units in branch1
Stage2: N_units in branch1
Stage2: N_units in branch2

Iterations


Figure 19: Visualization of our network propagation process of “NCP-Net-512 × 1024” in Tab. 8
(λ = 0.5).

4 128


Figure 20: Visualization of our network propagation process of “NCP-Net-128 × 512” (λ = 0.5)
in Tab. 8 for segmentation.


128

104

80

56

32

8

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

32


25.0

22.5

0 20 40 60

Predicted Acc (%)

2

1

0 20 40 60

GFLOPs

128

80

56

32

|Stage Stage Stage|1: N_chann 2: N_chann 2: N_chann|els in bra els in bra els in bra|nch1 nch1 nch2|
|---|---|---|---|
|||||


8 0 20 40 60

4

2

1 0 20 40 60

|S S S|tage1: N_u tage2: N_u tage2: N_u|nits in bra nits in bra nits in bra|nch1 nch1 nch2|
|---|---|---|---|
|||||
|||||


Iterations


76

74 0 20 40 60

Predicted mIoU (%)

2.0

0 20 40 60

GFLOPs

128

104

80

56

|Col1|Col2|1|
|---|---|---|
||||
|Stage1: N_chan Stage2: N_chan|nels in br nels in br|anch1 anch1|



8

0 20 40 60

Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2

4

2

1 0 20 40 60

|Stage1: N_u Stage2: N_u Stage2: N_u|nits in bra nits in bra nits in bra|nch1 nch1 nch2|
|---|---|---|
||||
||||


Iterations


104

80

56

32

8

Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

32


20 40 60

|Stage3 Stage3 Stage3|: N_chan : N_chan : N_chan|nels in bra nels in bra nels in bra|nch1 nch2 nch3|
|---|---|---|---|
|||||



20 40 60

Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


20 40 60

|Stage Stage Stage|3: N_unit 3: N_unit 3: N_unit|s in branch s in branch s in branch|1 2 3|Col5|
|---|---|---|---|---|
||||||
||||||


Iterations


20 40 60

|Col1|Col2|Col3|
|---|---|---|
|Stage4: N_chan Stage4: N_chan Stage4: N_chan|nels in br nels in br nels in br|anch1 anch2 anch3|


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Stage4: N_unit Stage4: N_unit Stage4: N_unit|s in branch s in branch s in branch|1 2 3|Col4|
|---|---|---|---|
|Stage4: N_unit|s in branch|4||
|||||


Stage4: N_units in branch1
Stage4: N_units in branch2
Stage4: N_units in branch3
Stage4: N_units in branch4

Iterations


20 40 60

|Col1|Col2|Col3|
|---|---|---|
|Stage3: N_cha Stage3: N_cha|nnels in br nnels in br|anch1 anch2|



20 40 60

Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


20 40 60

|Stage3: N_u Stage3: N_u Stage3: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|
||||
||||


Iterations


20 40 60

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|Stag Stag Stag|e4: N_cha e4: N_cha e4: N_cha|nnels in br nnels in br nnels in br|anch1 anch2 anch3|


In-layers: N_channels0
In-layers: N_channels1
Out-layer: N_channels


20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|St St St St|age4: N_u age4: N_u age4: N_u age4: N_u|nits in bra nits in bra nits in bra nits in bra|nch1 nch2 nch3 nch4|
|||||


Iterations


128

104

80

56

32


128

104

80

56

32


Figure 21: Visualization of our network propagation process of “NCP-Net-Both” (λ = 0.5) in
Tab. 8 for segmentation.


Figure 22: Visualization of our network propagation process of “NCP-Net-Scratch” in Tab. 9
(λ = 0.5).


27.5

25.0

22.5

2

1

128

104

80

56

32


128

104

80

56

32

8

|Col1|St St St|age2: N_bl age3: N_bl age4: N_bl|ocks ocks ocks|
|---|---|---|---|
|||||


Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

|Stage Stage Stage|3: N_chan 3: N_chan 3: N_chan|nels in bra nels in bra nels in bra|nch1 nch2 nch3|
|---|---|---|---|


32


128

104

80

56

32

8

|St St St|age2: N_bl age3: N_bl age4: N_bl|ocks ocks ocks|
|---|---|---|
||||


Stage1: N_blocks
Stage2: N_blocks
Stage3: N_blocks
Stage4: N_blocks


128

104

80

56

|Col1|Col2|Col3|
|---|---|---|


32


35

0 20 40 60

Predicted Acc (%)

2

1

0 20 40 60

|Col1|GF|LOPs|
|---|---|---|


128

104

|Stage1: N_chann Stage2: N_chann Stage2: N_chann|els in bra els in bra els in bra|nch1 nch1 nch2|
|---|---|---|



32

8

0 20 40 60

Stage1: N_channels in branch1
Stage2: N_channels in branch1
Stage2: N_channels in branch2

4

2

1 0 20 40 60

|Stage1: N_u Stage2: N_u Stage2: N_u|nits in bra nits in bra nits in bra|nch1 nch1 nch2|
|---|---|---|
||||


Iterations


20 40 60

Predicted Acc (%)


20 40 60

|Col1|Col2|GF|LOPs|
|---|---|---|---|


|Stage Stage Stage|1: N_chann 2: N_chann 2: N_chann|els in bra els in bra els in bra|nch1 nch1 nch2|
|---|---|---|---|



20 40 60

20 40 60

|Stag Stag Stag|e1: N_unit e2: N_unit e2: N_unit|s in branch s in branch s in branch|1 1 2|
|---|---|---|---|
|||||


Iterations


20 40 60

Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3

20 40 60

20 40 60

|St|age3: N_u|nits in bra|nch1|
|---|---|---|---|
|St St|age3: N_u age3: N_u|nits in bra nits in bra|nch2 nch3|


Iterations


20 40 60

|In-layer In-layer Out-lay|s: N_chan s: N_chan er: N_chan|nels0 nels1 nels|
|---|---|---|
||||


|Stage4: N_chan Stage4: N_chan Stage4: N_chan Stage4: N_chan|nels in bra nels in bra nels in bra nels in bra|nch1 nch2 nch3 nch4|
|---|---|---|



20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|Stage4: N_unit Stage4: N_unit Stage4: N_unit Stage4: N_unit|s in branc s in branc s in branc s in branc|h1 h2 h3 h4|Col4|
|---|---|---|---|
|||||


Stage4: N_units in branch1
Stage4: N_units in branch2
Stage4: N_units in branch3
Stage4: N_units in branch4

Iterations


20 40 60

20 40 60

Stage3: N_channels in branch1
Stage3: N_channels in branch2
Stage3: N_channels in branch3


20 40 60

|Stage3: N_u Stage3: N_u Stage3: N_u|nits in bra nits in bra nits in bra|nch1 nch2 nch3|
|---|---|---|
||||


Iterations


20 40 60

|Col1|In-layer In-layer Out-lay|s: N_chann s: N_chann er: N_chan|els0 els1 nels|
|---|---|---|---|
|||||


|Stag|e4: N_cha|nnels in br|anch1|
|---|---|---|---|



20 40 60

Stage4: N_channels in branch1
Stage4: N_channels in branch2
Stage4: N_channels in branch3
Stage4: N_channels in branch4


20 40 60

|St|age4: N_u|nits in bra|nch1|
|---|---|---|---|
|St St St|age4: N_u age4: N_u age4: N_u|nits in bra nits in bra nits in bra|nch2 nch3 nch4|


Iterations


128

104

80

56

32


128

104

80

56

32


Figure 24: Visualization of our network propagation process of “NCP-Net-Both” (λ = 0.5) in
Tab. 9 for video recognition.


Figure 23: Visualization of our network propagation process of “NCP-Net-Pretrained” (λ = 0.5)
in Tab. 9 for video recognition.


-----

