# OMNI-DIMENSIONAL DYNAMIC CONVOLUTION

**Chao Li[1][∗], Aojun Zhou[2], Anbang Yao[1][†]**

1Intel Labs China, 2CUHK-SenseTime Joint Lab, The Chinese University of Hong Kong
chao.li3@intel.com, aojun.zhou@gmail.com, anbang.yao@intel.com

ABSTRACT

Learning a single static convolutional kernel [1] in each convolutional layer is the
common training paradigm of modern Convolutional Neural Networks (CNNs).
Instead, recent research in dynamic convolution shows that learning a linear combination of n convolutional kernels weighted with their input-dependent attentions
can significantly improve the accuracy of light-weight CNNs, while maintaining
efficient inference. However, we observe that existing works endow convolutional
kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present
Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research. ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to
learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement
of regular convolutions, ODConv can be plugged into many CNN architectures.
Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77%∼5.71%|1.86%∼3.72% absolute
top-1 improvements to MobivleNetV2|ResNet family on the ImageNet dataset.
Intriguingly, thanks to its improved feature learning ability, ODConv with even
one single kernel can compete with or outperform existing dynamic convolution
counterparts with multiple kernels, substantially reducing extra parameters. Furthermore, ODConv is also superior to other attention modules for modulating the
output features or the convolutional weights. Code and models will be available
[at https://github.com/OSVAI/ODConv.](https://github.com/OSVAI/ODConv)

1 INTRODUCTION

In the past decade, we have witnessed the tremendous success of deep Convolutional Neural Networks (CNNs) in many computer vision applications (Krizhevsky et al., 2012; Girshick et al., 2014;
Long et al., 2015; He et al., 2017). The most common way of constructing a deep CNN is to stack
a number of convolutional layers as well as other basic layers organized with the predefined feature
connection topology. Along with great advances in CNN architecture design by manual engineering (Krizhevsky et al., 2012; He et al., 2016; Howard et al., 2017) and automatic searching (Zoph
& Le, 2017; Pham et al., 2018; Howard et al., 2019), lots of prevailing classification backbones
have been presented. Recent works (Wang et al., 2017; Hu et al., 2018b; Park et al., 2018; Woo
et al., 2018; Yang et al., 2019; Chen et al., 2020) show that incorporating attention mechanisms into
convolutional blocks can further push the performance boundaries of modern CNNs, and thus it has
attracted great research interest in the deep learning community.

The well-known SENet (Hu et al., 2018b) uses an attention module consisting of squeeze and excitation operations to adaptively recalibrate the output features of convolutional layers, strengthening

_∗This work was done when Chao Li was an intern at Intel Labs China, supervised by Anbang Yao who_
proposed the original idea and led the writing of the paper. _[†]_ Corresponding author.
1Here, we follow the definitions in (Yang et al., 2019; Chen et al., 2020) where a convolutional kernel refers
to the filter set of a convolutional layer.


-----

the representation power of a CNN via encouraging informative feature channels while suppressing
less important ones. Numerous attentive feature recalibration variants (Woo et al., 2018; Park et al.,
2018; Hu et al., 2018a) have been proposed since then. In Lin et al. (2020) and Quader et al. (2020),
two attention extensions to modulate the convolutional weights instead of the output features are also presented. Unlike the aforementioned methods in which the number of convolutional parameters
of a target network is fixed, dynamic convolution, which applies the attention mechanism over n
additive convolutional kernels to increase the size and the capacity of a network while maintaining
efficient inference, has recently become popular in optimizing efficient CNNs. This line of research
is pioneered by Conditionally Parameterized Convolutions (CondConv) (Yang et al., 2019) and Dynamic Convolution (DyConv) (Chen et al., 2020) whose basic ideas are the same. Generally, unlike a
regular convolutional layer which applies the same (i.e., static) convolutional kernel to all input samples, a dynamic convolutional layer learns a linear combination of n convolutional kernels weighted
with their attentions conditioned on the input features. Despite significant accuracy improvements
for light-weight CNNs, dynamic convolution designed in this way has two limitations. Firstly, the
main limitation lies in the attention mechanism design. The dynamic property of CondConv and
DyConv comes from computing convolutional kernels as a function of the input features. However,
we observe that they endow the dynamic property to convolutional kernels through one dimension
(regarding the convolutional kernel number) of the kernel space while the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each
convolutional kernel) are overlooked. As a result, the weights of each convolutional kernel share
the same attention scalar for a given input, limiting their abilities to capture rich contextual cues.
That is, the potentials of dynamic convolutional property have not been fully explored by existing
works, and thus they leave considerable room for improving the model performance. Secondly, at a
convolutional layer, replacing regular convolution by dynamic convolution increases the number of
convolutional parameters by n times. When applying dynamic convolution to a lot of convolutional
layers, it will heavily increase the model size. To handle this limitation, Li et al. (2021) proposes a
dynamic convolution decomposition method which can get more compact yet competitive models.
_Instead, in this paper we address both of the above limitations in a new perspective: formulating a_
_more diverse and effective attention mechanism and inserting it into the convolutional kernel space._

Our core contribution is a more generalized yet elegant dynamic convolution design called Omnidimensional Dynamic Convolution (ODConv). Unlike existing works discussed above, at any convolutional layer, ODConv leverages a novel multi-dimensional attention mechanism to learn four
types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner. We show that these four types of attentions learnt by our ODConv are complementary
to each other, and progressively applying them to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN. Consequently, ODConv with even one single kernel can compete with or outperform existing dynamic
convolution counterparts with multiple kernels, substantially reducing extra parameters.

As a drop-in design, ODConv can be used to replace regular convolutions in many CNN architectures. It strikes a better tradeoff between model accuracy and efficiency compared to existing
dynamic convolution designs, as validated by extensive experiments on the large-scale ImageNet
classification dataset (Russakovsky et al., 2015) with various prevailing CNN backbones. ODConv also shows better recognition performance under similar model complexities when compared to
other state-of-the-art attention methods for output feature recalibration (Woo et al., 2018; Hu et al.,
2018b; Wang et al., 2020; Lin et al., 2020) or for convolutional weight modification (Ma et al., 2020;
Lin et al., 2020; Quader et al., 2020). Furthermore, the performance improvements by ODConv for
the pre-trained classification models can transfer well to downstream tasks such as object detection
on the MS-COCO dataset (Lin et al., 2014), validating its promising generalization ability.

2 RELATED WORK

**Deep CNN Architectures. AlexNet (Krizhevsky et al., 2012) ignited the surge of deep CNNs**
by winning the ImageNet classification challenge 2012. Since then, lots of well-known CNN architectures such as VGGNet (Simonyan & Zisserman, 2015), InceptionNet (Szegedy et al., 2015),
ResNet (He et al., 2016), DenseNet (Huang et al., 2017) and ResNeXt (Xie et al., 2017) have been
proposed, which are designed to be much deeper and have more sophisticated connection topologies
compared with AlexNet. To ease the deployment of inference models on resource-limited platforms,


-----

MobileNets (Howard et al., 2017; Sandler et al., 2018) and ShuffleNet (Zhang et al., 2018b; Ma et al.,
2018) are presented. All the aforementioned CNNs are manually designed. Recently, researchers
have also made great efforts (Zoph & Le, 2017; Pham et al., 2018; Howard et al., 2019) to automate
the network design process. Our ODConv could be potentially used to boost their performance.

**Attentive Feature Recalibration. Designing attentive feature recalibration modules to improve**
the performance of a CNN has been widely studied in recent years. Wang et al. (2017) proposes
a specialized attention module consisting of a trunk branch and a mask branch, and inserts it into
the intermediate stages of deep residual networks. SENet (Hu et al., 2018b) uses a seminal channel
attention module termed Squeeze-and-Excitation (SE) to exploit the interdependencies between the
channels of convolutional features. Many subsequent works improve SE from different aspects, following its two-stage design (i.e., feature aggregation and feature recalibration). BAM (Park et al.,
2018) and CBAM (Woo et al., 2018) combine the channel attention module with the spatial attention
module. Misra et al. (2021) presents an attention module having three branches conditioned on the
features rotated along three different dimensions. GE (Hu et al., 2018a) introduces a gather operator to extract better global context from a large spatial extent. To enhance the feature aggregation
capability, SRM (Lee et al., 2019) replaces the global average by the channel-wise mean and standard deviation. SKNets (Li et al., 2019) add an attention design over two branches with different
sized convolutions to fuse multi-scale feature outputs. ECA (Wang et al., 2020) provides a more
efficient channel attention design using cheaper 1D convolutions to replace the first fully connected
layer of SE. Instead of recalibrating the output convolutional features by attention modules, dynamic
convolution methods apply attention mechanisms to a linear combination of n convolutional kernels.

**Dynamic Weight Networks. Making the weights of a neural network to be sample-adaptive via**
dynamic mechanisms has shown great potentials for boosting model capacity and generalization.
Hypernetworks (Ha et al., 2017) use a small network called hypernetwork to generate the weights
for a larger recurrent network called main network. MetaNet (Munkhdalai & Yu, 2017) adopts a
meta learning model to parameterize the task-adaptive network for rapid generalization across a sequence of tasks. Jaderberg et al. (2015) proposes a Spatial Transformer module conditioned on the
learnt features to predict the parametric transformation, and applies it to align the distorted input
image. Dynamic Filter Network (Jia et al., 2016) uses a filter generation network to produce filters
conditioned on an input, and processes another input with the generated filters. DynamoNet (Diba
et al., 2019) uses dynamically generated motion filters to handle the action recognition problem.
Kernel Prediction Networks (Bako et al., 2017; Mildenhall et al., 2018) leverage a CNN architecture
to predict spatially varying kernels used for video denoising. WeightNet (Ma et al., 2020) appends a
grouped fully connected layer to the attention feature vector of an SE block, generating the weights
of a CNN used for image recognition. Lin et al. (2020) modifies the weights of convolutional layers
with a gated module under the guidance of global context, while Quader et al. (2020) directly uses
either an SE block or a simple activation function conditioned on the magnitudes of convolutional
weights to modify the weights themselves. Our ODConv aims to address the limitations of recently proposed dynamic convolution (Yang et al., 2019; Chen et al., 2020) which differs from these
methods both in focus and formulation, see the Introduction and Method sections for details.

3 METHOD

In this section, we first make a review of dynamic convolution via a general formulation. Then, we
describe the formulation of our ODConv, clarify its properties and detail its implementation.

3.1 REVIEW OF DYNAMIC CONVOLUTION

**Basic concept. A regular convolutional layer has a single static convolutional kernel which is ap-**
plied to all input samples. For a dynamic convolutional layer, it uses a linear combination of n
convolutional kernels weighted dynamically with an attention mechanism, making convolution operations be input-dependent. Mathematically, dynamic convolution operations can be defined as

_y = (αw1W1 + ... + αwnWn) ∗_ _x,_ (1)

where x ∈ R[h][×][w][×][c][in] and y ∈ R[h][×][w][×][c][out] denote the input features and the output features (having
_cin/cout channels with the height h and the width w), respectively; Wi denotes the i[th]_ convolutional
kernel consisting of cout filters Wi[m] R[k][×][k][×][c][in] _, m = 1, ..., cout; αwi_ R is the attention scalar
_∈_ _∈_


-----

|Sig|moid|
|---|---|
|F|C|

|Sig|moid|
|---|---|
|F|C|

|Sig|moid|
|---|---|
|F|C|

|Col1|AP 𝑾𝟏 𝜶𝒘𝟏 C 𝑾𝟐 . 𝜶𝒘𝟐 LU … … 𝑾𝒏 𝜶𝒘𝒏 Softmax FC|Col3|Col4|Col5|
|---|---|---|---|---|
|G|AP|𝑾𝟏 𝑾𝟐 … 𝑾𝒏|||
||||||
|F|C||||
||||||
|Re|LU||||
||||||
||||Soft|max|
||||F|C|
||||||

|Col1|AP 𝑾𝟏 𝜶𝒔𝟏 𝜶𝒄𝟏 𝜶𝒇𝟏 𝜶𝒘𝟏 C 𝑾𝟐 . 𝜶𝒔𝟐 . 𝜶𝒄𝟐 .. 𝜶𝒇𝟐 .. 𝜶𝒘𝟐 LU … … … … … 𝑾𝒏 𝜶𝒔𝒏 𝜶𝒄𝒏 𝜶𝒇𝒏 𝜶𝒘𝒏 Sigmoid Sigmoid Sigmoid Softmax FC FC FC FC|Col3|Col4|Col5|
|---|---|---|---|---|
|G|AP|𝑾𝟏 𝑾𝟐 … 𝑾𝒏|||
||||||
|F|C||||
||||||
|Re|LU||||
||||||
||||Soft|max|
||||F|C|
||||||


**+**


(a) (b)

Figure 1: A schematic comparison of (a) DyConv (CondConv uses GAP+FC+Sigmoid) and (b)
ODConv. Unlike CondConv and DyConv which compute a single attention scalar αwi for the convolutional kernel Wi, ODConv leverages a novel multi-dimensional attention mechanism to compute
four types of attentions αsi, αci, αfi and αwi for Wi along all four dimensions of the kernel space
in a parallel manner. Their formulations and implementations are clarified in the Method section.

for weighting Wi, which is computed by an attention function πwi(x) conditioned on the input
features; ∗ denotes the convolution operation. For conciseness, here we omit the bias term.

**CondConv vs. DyConv. Although the concept of dynamic convolution defined in Eq. 1 is proposed**
separately in CondConv (Yang et al., 2019) and DyConv (Chen et al., 2020), their implementations
are different, mainly in the structure of πwi(x) to compute αwi, the model training strategy, and the
layer locations to apply dynamic convolutions. Specifically, both methods choose the modified SE
structure for πwi(x), and CondConv uses a Sigmoid function while DyConv uses a Softmax function
as the activation function to compute αwi. DyConv adopts a temperature annealing strategy in the
training process to suppress the near one-hot output of the Softmax function. For all their tested CNN
architectures, CondConv replaces the convolutional layers in the final several blocks (e.g., 6 for the
MobileNetV2 backbones and 3 for the ResNet backbones) and the last fully connected layer, while
DyConv replaces all convolutional layers except the first layer. These implementation differences
lead to different results in model accuracy, size and efficiency for CondConv and DyConv.

**Limitation Discussions. According to Eq. 1, dynamic convolution has two basic components: the**
convolutional kernels _W1, ...Wn_, and the attention function πwi(x) to compute their attention
_{_ _}_
scalars _αw1, ...αwn_ . Given n convolutional kernels, the corresponding kernel space has four di_{_ _}_
mensions regarding the spatial kernel size k×k, the input channel number cin and the output channel
number cout for each convolutional kernel, and the convolutional kernel number n. However, for
CondConv and DyConv, we can observe that πwi(x) allocates a single attention scalar αwi to the
convolutional kernel Wi, meaning that all its cout filters Wi[m] R[k][×][k][×][c][in] _, m = 1, ..., cout have the_
_∈_
same attention value for the input x. In other words, the spatial dimension, the input channel dimension and the output channel dimension for the convolutional kernel Wi are ignored by CondConv
and DyConv. This leads to a coarse exploitation of the kernel space when they design their attention
mechanisms for endowing n convolutional kernels with the dynamic property. This may also be
one of the reasons why CondConv and DyConv show much lower performance gains to relatively
larger CNNs compared to efficient ones. Besides, compared to a regular convolutional layer, a dynamic convolutional layer increases the number of convolutional parameters by n times (although
the increase of Multiply-Adds (MAdds) is marginal due to the additive property of n convolutional
kernels). Typically, CondConv sets n = 8 and DyConv sets n = 4. Therefore, it will heavily increase the model size when applying dynamic convolution to a lot of convolutional layers. However,
we empirically find that removing the attention mechanism from CondConv|DyConv (i.e., setting
_αwi = 1) almost diminishes the accuracy boosts for prevailing CNN backbones on the ImageNet_
dataset close to zero. For instance, on ResNet18, the top-1 gain averaged over 3 runs decreases from
1.74%|2.51% to 0.08%|0.14% when removing the attention mechanism from CondConv|DyConv.
_These observations indicate that the attention mechanism design plays the key role in dynamic con-_
_volution, and a more effective design may strike a good balance between model accuracy and size._

3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION

In light of the above discussions, our ODConv introduces a multi-dimensional attention mechanism
with a parallel strategy to learn diverse attentions for convolutional kernels along all four dimensions
of the kernel space. Fig. 1 provides a schematic comparison of CondConv, DyConv and ODConv.


-----

(a) 𝑾𝒎𝒊 𝜶𝒔𝒊 𝜶𝒔𝒊 [⊙] 𝑾𝒎𝒊 (b) 𝑾𝒎𝒊 𝜶𝒄𝒊 𝜶𝒄𝒊 [⊙] 𝑾𝒎𝒊

𝑘 𝑘 𝑐𝑖𝑛 . 𝑘 𝑘 1 𝑘 𝑘 𝑐𝑖𝑛 𝑘 𝑘 𝑐𝑖𝑛 . 1 1 𝑐𝑖𝑛 𝑘 𝑘 𝑐𝑖𝑛

(c) 𝑾𝒊 𝜶𝒇𝒊 𝜶𝒇𝒊 [⊙] 𝑾𝒊 (d)

[𝑾𝟏 … 𝑾𝒏] 𝜶𝒘 𝜶𝒘[⊙] [𝑾𝟏 … 𝑾𝒏]

𝑘 1 𝑘

𝑘… 𝑐𝑖𝑛 𝑐𝑜𝑢𝑡 . …1 1 𝑐𝑜𝑢𝑡 𝑘… 𝑐𝑖𝑛 𝑐𝑜𝑢𝑡 𝑐𝑜𝑢𝑡 𝑘 …𝑘 𝑐𝑖𝑛…𝑘 …𝑘 𝑐𝑖𝑛. 1 … 1 𝑘 …𝑘 𝑐𝑖𝑛…𝑘 …𝑘 𝑐𝑖𝑛

1 1 1 1

𝑘 𝑘 𝑘 𝑘

𝑘 𝑘 𝑐𝑖𝑛 1 1 1 𝑘 𝑘 𝑐𝑖𝑛 𝑘 𝑐𝑖𝑛𝑛 𝑘 𝑐𝑖𝑛 𝑛 𝑘 𝑐𝑖𝑛𝑛 𝑘 𝑐𝑖𝑛


Figure 2: Illustration of multiplying four types of attentions in ODConv to convolutional kernels
progressively. (a) Location-wise multiplication operations along the spatial dimension, (b) channelwise multiplication operations along the input channel dimension, (c) filter-wise multiplication operations along the output channel dimension, and (d) kernel-wise multiplication operations along the
kernel dimension of the convolutional kernel space. Notations are clarified in the Method section.

**Formulation of ODConv. Following the notations in Eq. 1, ODConv can be defined as**

_y = (αw1 ⊙_ _αf_ 1 ⊙ _αc1 ⊙_ _αs1 ⊙_ _W1 + ... + αwn ⊙_ _αfn ⊙_ _αcn ⊙_ _αsn ⊙_ _Wn) ∗_ _x,_ (2)

to that in Eq. 1;tions, which are computed along the spatial dimension, the input channel dimension and the outputwhere αwi ∈ R α denotes the attention scalar for the convolutional kernelsi ∈ R[k][×][k], αci ∈ R[c][in] and αfi ∈ R[c][out] denote three newly introduced atten- Wi, which is the same
channel dimension of the kernel space for the convolutional kernel Wi, respectively; denotes the
_⊙_
multiplication operations along different dimensions of the kernel space. Here, αsi, αci, αfi and
_αwi are computed with a multi-head attention module πi(x) which will be clarified later._

**A Deep Understanding of ODConv. In ODConv, for the convolutional kernel Wi: (1) αsi assigns**
different attention scalars to convolutional parameters (per filter) at k _k spatial locations; (2) αci_
_×_
assigns different attention scalars to cin channels of each convolutional filter Wi[m][; (3)][ α][fi][ assign-]
s different attention scalars to cout convolutional filters; (4) αwi assigns an attention scalar to the
whole convolutional kernel. Fig. 2 illustrates the process of multiplying these four types of attentions to n convolutional kernels. In principle, these four types of attentions are complementary to
_each other, and progressively multiplying them to the convolutional kernel Wi in the location-wise,_
_channel-wise, filter-wise and kernel-wise orders makes convolution operations be different w.r.t. all_
_spatial locations, all input channels, all filters and all kernels for the input x, providing a perfor-_
_mance guarantee to capture rich context cues. Therefore, ODConv can significantly strengthen the_
feature extraction ability of basic convolution operations of a CNN. Moreover, ODConv with one
single convolutional kernel can compete with or outperform standard CondConv and DyConv, introducing substantially fewer extra parameters to the final models. Extensive experiments are provided
to validate these advantages. By comparing Eq. 1 and Eq. 2, we can clearly see that ODConv is a
more generalized dynamic convolution. Moreover, when setting n = 1 and all components of αs1,
_αc1 and αw1 to 1, ODConv with only filter-wise attention αf_ 1 will be reduced into: applying an SE
variant conditioned on the input features to the convolutional filters, then followed by convolution
operations (note the original SE (Hu et al., 2018b) is conditioned on the output features, and is used
to recalibrate the output features themselves). Such an SE variant is a special case of ODConv.

**Implementation. For ODConv, a critical question is how to compute four types of attentions αsi,**
_αci, αfi and αwi for the convolutional kernel Wi. Following CondConv and DyConv, we also use_
a SE-typed attention module (Hu et al., 2018b) but with multiple heads as πi(x) to compute them,
whose structure is shown in Fig. 1. Specifically, the input x is squeezed into a feature vector with the
length of cin by channel-wise Global Average Pooling (GAP) operations first. Subsequently, there is
a Fully Connected (FC) layer and four head branches. A Rectified Linear Unit (ReLU) (Krizhevsky
et al., 2012) comes after the FC layer. The FC layer maps the squeezed feature vector to a lower
dimensional space with the reduction ratio r (according to the ablative experiments, we set r = 1/16
in all main experiments, avoiding high model complexity). For four head branches, each has an FC
layer with the output size of k _k, cin_ 1, cout 1 and n 1, and a Softmax or Sigmoid function
to generate the normalized attentions × _α ×si, αci, α ×fi and αwi ×, respectively. We adopt the temperature_
annealing strategy proposed in DyConv to facilitate the training process. For easy implementation,
_we apply ODConv to all convolutional layers except the first layer of each CNN architecture tested_


-----

_in our main experiments just like DyConv, and share αsi, αci and αfi to all convolutional kernels._
In the Experiments section, a comparison of the inference speed for different dynamic convolution
methods is provided. In the Appendix, we further provide a computational cost analysis of ODConv
and an ablation study of applying ODConv to different layer locations.

4 EXPERIMENTS

In this section, we provide comprehensive experiments on two large-scale image recognition datasets
with different CNN architectures to validate the effectiveness of ODConv, compare its performance
with many attention based methods, and study the design of ODConv from different aspects.

4.1 IMAGE CLASSIFICATION ON IMAGENET

Our main experiments are performed on the ImageNet dataset (Russakovsky et al., 2015). It has over
1.2 million images for training and 50,000 images for validation, including 1,000 object classes.

**CNN Backbones. We use MobileNetV2 (Sandler et al., 2018) and ResNet (He et al., 2016) families**
for experiments, covering both light-weight CNN architectures and larger ones. Specifically, we
choose ResNet18, ResNet50, ResNet101, and MobileNetV2 (1.0×, 0.75×, 0.5×) as the backbones.

**Experimental Setup. In the experiments, we consider existing dynamic convolution methods in-**
cluding CondConv (Yang et al., 2019), DyConv (Chen et al., 2020) and DCD (Li et al., 2021) as
the key reference methods for comparisons on all CNN backbones. On the ResNet backbones, we
also compare our ODConv with many state-of-the-art methods using attention modules: (1) for output feature recalibration including SE (Hu et al., 2018b), CBAM (Woo et al., 2018) and ECA (Wang
et al., 2020), and (2) for convolutional weight modification including CGC (Lin et al., 2020), WeightNet (Ma et al., 2020) and WE (Quader et al., 2020). For fair comparisons, we use public codes of
_these methods unless otherwise stated, and adopt the popular training and test settings used in the_
_community for implementing the experiments. The models trained by all methods use the same set-_
_tings including the batch size, the number of training epochs, the learning rate schedule, the weight_
_decay, the momentum and the data processing pipeline. Moreover, we do not use advanced training_
_tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016), aiming to_
_have clean performance comparisons. Experimental details are described in the Appendix._

Table 1: Results comparison on the ImageNet validation set with the MobileNetV2 (1.0×, 0.75×,
0.5×) backbones trained for 150 epochs. For our ODConv, we set r = 1/16. Best results are bolded.

|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|
|MobileNetV2 (1.0×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|3.50M 22.88M 12.40M 5.72M 4.94M 11.52M|300.8M 318.1M 317.1M 318.4M 311.8M 327.1M|71.65 74.13 (↑2.48) 74.94 (↑3.29) 74.18 (↑2.53) 74.84 (↑3.19) 75.42 (↑3.77)|90.22 91.67 (↑1.45) 91.83 (↑1.61) 91.72 (↑1.50) 92.13 (↑1.91) 92.18 (↑1.96)|
|MobileNetV2 (0.75×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|2.64M 17.51M 7.95M 4.08M 3.51M 7.50M|209.1M 223.9M 220.1M 222.9M 217.1M 226.3M|69.18 71.79 (↑2.61) 72.75 (↑3.57) 71.92 (↑2.74) 72.43 (↑3.25) 73.81 (↑4.63)|88.82 90.17 (↑1.35) 90.93 (↑2.11) 90.20 (↑1.38) 90.82 (↑2.00) 91.33 (↑2.51)|
|MobileNetV2 (0.5×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|2.00M 13.61M 4.57M 3.06M 2.43M 4.44M|97.1M 110.0M 103.2M 105.6M 101.8M 106.4M|64.30 67.24 (↑2.94) 69.05 (↑4.75) 69.32 (↑5.02) 68.26 (↑3.96) 70.01 (↑5.71)|85.21 87.51 (↑2.30) 88.37 (↑3.16) 88.44 (↑3.23) 87.98 (↑2.77) 89.01 (↑3.80)|



**Results Comparison on MobileNets. Table 1 shows the results comparison on the MobileNetV2**
(1.0×, 0.75×, 0.5×) backbones. As CondConv and DyConv are primarily proposed to improve
the performance of efficient CNNs, they all bring promising top-1 gains to the light-weight MobileNetV2 (1.0×, 0.75×, 0.5×) backbones. Comparatively, our ODConv (1×) with one single
convolutional kernel performs better than CondConv (8×) with 8 convolutional kernels, and its performance is also on par with DyConv (4×) with 4 convolutional kernels. Note that these competitive
results of ODConv (1×) are obtained with significantly lower numbers of extra parameters, validating that our ODConv can strike a better tradeoff between model accuracy and size. Besides, ODConv
(1×) also performs better than DCD in most cases. ODConv (4×) always achieves the best results


-----

on all backbones. The results comparison with an increase number of training epochs (from 150 to
300) can be found in the Appendix, from which we can observe similar performance trends.

**Results Comparison on ResNets. Table 2 shows the results comparison on the ResNet18 and**
ResNet50 backbones which are much larger than the MobileNetV2 backbones. We can obtain the
following observations: (1) On the ResNet18 backbone, dynamic convolution methods (CondConv,
DyConv, DCD and our ODConv) and convolutional weight modification methods (CGC, WeightNet
and WE) mostly show better performance than output feature recalibration methods (SE, CBAM
and ECA), although they all use attention mechanisms. Comparatively, our ODConv (1×) with one
single convolutional kernel outperforms the other methods both in model accuracy and size, bringing
2.85% top-1 gain to the baseline model. ODConv (4×) gets the best results, yielding a top-1 gain of
3.72%; (2) However, on the larger ResNet50 backbone, CondConv, DyConv and DCD show worse
results than most of the other methods even though they have significantly increased numbers of
parameters. Due to the increased parameter redundancy, adding more parameters to larger networks
tends to be less effective in improving model accuracy, compared to small networks. Thanks to
the proposed multi-dimensional attention mechanism, ODConv can address this problem better,
achieving superior performance both in model accuracy and size for larger backbones as well as
light-weight ones. To further validate the performance of ODConv on very deep and large networks,
we apply ODConv to the ResNet101 backbone, and the results comparison is provided in Table 3.
Again, ODConv shows very promising results, yielding 1.57% top-1 gain with ODConv (1×).

Table 2: Results comparison on the ImageNet validation set with the ResNet18 and ResNet50 backbones trained for 100 epochs. For our ODConv, we set r = 1/16. * denotes the results are from the
paper of WE (Quader et al., 2020) as its code is not publicly available. Best results are bolded.

|Network|ResNet18|Col3|Col4|Col5|ResNet50|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|Baseline|11.69M|1.814G|70.25|89.38|25.56M|3.858G|76.23|93.01|
|+ CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|81.35M 45.47M 14.70M 11.94M 44.90M|1.894G 1.861G 1.841G 1.838G 1.916G|71.99 (↑1.74) 72.76 (↑2.51) 72.33 (↑2.08) 73.10 (↑2.85) 73.97 (↑3.72)|90.27 (↑0.89) 90.79 (↑1.41) 90.65 (↑1.27) 91.10 (↑1.72) 91.35 (↑1.97)|129.86M 100.88M 29.84M 28.64M 90.67M|3.978G 3.965G 3.944G 3.916G 4.080G|76.70 (↑0.47) 76.82 (↑0.59) 76.92 (↑0.69) 77.96 (↑1.73) 78.52 (↑2.29)|93.12 (↑0.11) 93.16 (↑0.15) 93.46 (↑0.45) 93.84 (↑0.83) 94.01 (↑1.00)|
|+ SE + CBAM + ECA|11.78M 11.78M 11.69M|1.816G 1.818G 1.816G|70.98 (↑0.73) 71.01 (↑0.76) 70.60 (↑0.35)|90.03 (↑0.65) 89.85 (↑0.47) 89.68 (↑0.30)|28.07M 28.07M 25.56M|3.872G 3.886G 3.870G|77.31 (↑1.08) 77.46 (↑1.23) 77.34 (↑1.11)|93.63 (↑0.62) 93.59 (↑0.58) 93.64 (↑0.63)|
|+ CGC + WeightNet + WE (*)|11.69M 11.93M 11.90M|1.827G 1.826G 1.820G|71.60 (↑1.35) 71.56 (↑1.31) 71.00 (↑0.75)|90.35 (↑0.97) 90.38 (↑1.00) 90.00 (↑0.62)|25.59M 30.38M 28.10M|3.877G 3.885G 3.860G|76.79 (↑0.56) 77.51 (↑1.28) 77.10 (↑0.87)|93.37 (↑0.36) 93.69 (↑0.68) 93.50 (↑0.49)|



Table 3: Results comparison on the ImageNet validation set with the ResNet101 backbone trained
for 100 epochs. For our ODConv, we set r = 1/16. Best results are bolded.

|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|
|ResNet101|44.55M|7.570G|77.41|93.67|
|+ SE + CBAM + ECA + ODConv (1×) + ODConv (2×)|49.29M 49.30M 44.55M 50.82M 90.44M|7.593G 7.617G 7.590G 7.675G 7.802G|78.42 (↑1.01) 78.50 (↑1.09) 78.60 (↑1.19) 78.98 (↑1.57) 79.27 (↑1.86)|94.15 (↑0.48) 94.20 (↑0.53) 94.34 (↑0.67) 94.38 (↑0.71) 94.47 (↑0.80)|



4.2 OBJECT DETECTION ON MS-COCO

Given a backbone model pre-trained on the ImageNet classification dataset, a critical question is
whether the performance improvement by ODConv can be transferred to downstream tasks or not.
To explore it, we next perform comparative experiments on the object detection track of MS-COCO
dataset (Lin et al., 2014). The 2017 version of MS-COCO dataset contains 118,000 training images
and 5,000 validation images with 80 object classes.

**Experimental Setup. We use the popular MMDetection toolbox (Chen et al., 2019) for experiments**
with the pre-trained ResNet50 and MobileNetV2 (1.0×) models as the backbones for the detector.
We select the mainstream Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017)
detectors with Feature Pyramid Networks (FPNs) (Lin et al., 2017) as the necks to build the basic
object detection systems. For a neat comparison, the convolutional layers in the FPN necks still
use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter


-----

Table 4: Results comparison on the MS-COCO 2017 validation set. Regarding Params or MAdds,
the number in the bracket is for the pre-trained backbone models excluding the last fully connected
layer, which is almost the same to that shown in Table 1 and Table 2, while the other number is for
the whole object detector. Best results are bolded.

|Backbone Models|Detectors|AP(%)|AP50(%)|AP75(%)|APS(%)|APM(%)|APL(%)|Params|MAdds|
|---|---|---|---|---|---|---|---|---|---|
|ResNet50 + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|Faster R-CNN|37.2 38.1 38.3 38.1 39.0 39.2|57.8 58.9 59.7 59.3 60.5 60.7|40.4 41.5 41.6 41.3 42.3 42.6|21.5 22.4 22.6 21.9 23.4 23.1|40.6 42.1 42.3 42.0 42.3 42.6|48.0 48.7 49.4 49.5 50.5 51.0|43.80M (23.51M) 133.75M (113.46M) 119.12M (98.83M) 48.08M (27.79M) 46.88M (26.59M) 108.91M (88.62M)|207.07G (76.50G) 207.08G (76.51G) 207.23G (76.66G) 207.20G (76.63G) 207.18G (76.61G) 207.42G (76.85G)|
|MobileNetV2 (1.0×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)||31.3 33.7 34.5 33.3 34.3 35.1|51.1 54.9 55.6 53.0 55.6 56.7|33.1 35.6 36.5 35.1 36.5 37.0|17.4 19.3 19.8 19.9 20.7 20.6|33.5 36.4 37.3 36.1 37.3 38.0|41.2 43.7 44.7 43.2 44.5 45.2|21.13M (2.22M) 31.54M (12.63M) 30.02M (11.12M) 23.34M (4.44M) 22.56M (3.66M) 29.14M (10.24M)|122.58G (24.45G) 122.59G (24.46G) 123.01G (24.88G) 123.01G (24.88G) 123.00G (24.87G) 123.02G (24.89G)|
|ResNet50 + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|Mask R-CNN|38.0 38.8 39.2 38.8 39.9 40.1|58.6 59.3 60.3 59.8 61.2 61.5|41.5 42.3 42.5 42.2 43.5 43.6|21.6 22.5 23.0 23.1 23.6 24.0|41.5 42.5 42.9 42.7 43.8 43.6|49.2 50.3 51.4 49.8 52.3 52.3|46.45M (23.51M) 136.4M (113.46M) 121.77M (98.83M) 50.73M (27.79M) 49.53M (26.59M) 111.56M (88.62M)|260.14G (76.50G) 260.15G (76.51G) 260.30G (76.66G) 260.27G (76.63G) 260.25G (76.61G) 260.49G (76.85G)|
|MobileNetV2 (1.0×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)||32.2 34.4 35.2 34.3 35.0 35.8|52.1 55.4 56.2 54.9 56.1 57.0|34.2 36.6 37.5 36.6 37.3 38.1|18.4 19.8 20.7 20.6 19.9 20.5|34.4 36.9 38.0 37.1 37.7 38.5|42.4 44.6 45.5 44.8 46.2 45.9|23.78M (2.22M) 34.19M (12.63M) 32.68M (11.12M) 26.00M (4.44M) 25.22M (3.66M) 31.80M (10.24M)|175.66G (24.45G) 175.67G (24.46G) 176.09G (24.88G) 176.09G (24.88G) 176.08G (24.87G) 176.10G (24.89G)|



settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively.
Experimental details are described in the Appendix.

**Results Comparison. From the results shown in Table 4, we can observe similar performance**
improvement trends as on the ImageNet dataset. For Faster R-CNN|Mask R-CNN with the pretrained ResNet50 backbone models, CondConv (8×) and DyConv (4×) show an AP improvement
of 0.9%|0.8% and 1.1%|1.2% to the baseline model respectively, while our method performs much
better, e.g., ODConv (1×) with one single convolutional kernel even shows an AP improvement of
1.8%|1.9%, respectively. With the pre-trained MobileNetV2 (1.0×) backbone models, our ODConv
(1×) performs obviously better than CondConv (8×), and its performance is on par with that of
DyConv (4×) as their AP gap is only 0.2% for both two detectors, striking a better model accuracy
and efficiency tradeoff. Similar boosts to AP scores for small, medium and large objects are also
obtained by three methods on both detectors. ODConv (4×) always achieves the best AP scores.

4.3 ABLATION STUDIES

Finally, we conduct a lot of ablative experiments on the ImageNet dataset, in order to have a better
analysis of our ODConv.

Table 5: Results comparison of the ResNet18 models based on ODConv with different settings of
the reduction ratio r. All models are trained on the ImageNet dataset. Best results are bolded.

|Models|r|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|---|
|ResNet18|-|11.69M|1.814G|70.25|89.38|
|+ ODConv (1×)|1/4 1/8 1/16|12.58M 12.15M 11.94M|1.839G 1.838G 1.838G|73.41 73.11 73.10|91.29 91.10 91.10|



Table 6: Results comparison of the ResNet18 models based on ODConv with different numbers of
convolutional kernels n. All models are trained on the ImageNet dataset. Best results are bolded.

|Models|n|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|---|
|ResNet18|-|11.69M|1.814G|70.25|89.38|
|+ ODConv (r = 1/16)|1× 2× 3× 4× 8×|11.94M 22.93M 33.92M 44.90M 88.84M|1.838G 1.872G 1.894G 1.916G 2.006G|73.10 73.59 73.77 73.97 74.08|91.10 91.08 91.35 91.35 91.44|



**Reduction Ratio Selection. Our first set of ablative experiments is for the selection of the reduction**
ratio r used in the attention structure πi(x). From the results shown in Table 5, we can find that under
_r = 1/4, r = 1/8 and r = 1/16, ODConv consistently obtains large performance improvements to_
the baseline ResNet18 model (2.85∼3.16% top-1 gain), and the extra MAdds are negligible. Comparatively, ODConv with r = 1/16 strikes the best tradeoff between model accuracy and efficiency,
and thus we choose it as our default setting.


-----

Table 7: Investigating the complementarity of four types of attentions proposed in ODConv. In the
experiments, when αwi is used, we set n = 4, and otherwise n = 1. For the optimal comparison, we

Best results are bolded.

|r setting reported are bolded.|d in T|Table|5. A|All Re|esNet18|8 models|s are trained|on the Image|
|---|---|---|---|---|---|---|---|---|
|Models|αsi|αci|αfi|αwi|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|ResNet18|-|-|-|-|11.69M|1.814G|70.25|89.38|
|+ ODConv (r = 1/4)|✓ - - ✓ ✓ - ✓ ✓|- ✓ - ✓ - ✓ ✓ ✓|- - ✓ - ✓ ✓ ✓ ✓|- - - - - - - ✓|11.98M 12.26M 12.28M 12.27M 12.29M 12.57M 12.58M 45.54M|1.827G 1.827G 1.827G 1.827G 1.827G 1.829G 1.839G 1.953G|72.42 72.07 71.46 73.13 72.80 72.20 73.41 74.33|90.76 90.59 90.43 91.14 90.99 90.67 91.29 91.53|



Table 8: Comparison of the inference speed (frames per second) for different dynamic convolution
methods. All pre-trained models are tested on an NVIDIA TITAN X GPU (with batch size 200) and
a single core of Intel E5-2683 v3 CPU (with batch size 1) separately, and the input image size is

Baseline model 652.2 6.4 Baseline model 1452.3 17.3
+ CondConv 425.1 4.0 + CondConv 1076.9 14.8
+ DyConv 326.6 3.8 + DyConv 918.2 11.9
+ DCD 400.0 3.6 + DCD 875.7 9.5
+ ODConv (+ ODConv (14××)) 293.9152.7 3.92.5 + ODConv (+ ODConv (14××)) 1029.2608.3 12.811.2

|× 224 pixels. ResNet50|Speed on GPU|Speed on CPU|
|---|---|---|
|ResNet50|Speed on GPU|Speed on CPU|
|Baseline model + CondConv + DyConv + DCD + ODConv (1×) + ODConv (4×)|652.2 425.1 326.6 400.0 293.9 152.7|6.4 4.0 3.8 3.6 3.9 2.5|


|MobileNetV2 (1.0×)|Speed on GPU|Speed on CPU|
|---|---|---|
|Baseline model + CondConv + DyConv + DCD + ODConv (1×) + ODConv (4×)|1452.3 1076.9 918.2 875.7 1029.2 608.3|17.3 14.8 11.9 9.5 12.8 11.2|



**Convolutional Kernel Number. Accordingly to the main experiments discussed beforehand, OD-**
Conv can strike a better tradeoff between model accuracy and size, compared to existing dynamic
convolution methods. For a better understanding of this advantage, we next perform the second set
of ablative experiments on the ImageNet dataset, training the ResNet18 backbone based on ODConv
with different settings of the convolutional kernel number n. Table 6 shows the results. It can be
seen that ODConv with one single convolutional kernel brings near 3.0% top-1 gain to the ResNet18
baseline, and the gain tends to be saturated when the number of convolutional kernels is set to 8.

**Four Types of Attentions. Note that our ODConv has four types of convolutional kernel attentions**
_αsi, αci, αfi and αwi computed along all four dimensions of the kernel space, respectively. Then, we_
perform another set of ablative experiments on the ResNet18 backbone with different combinations
of them to investigate their dependencies. Results are summarized in Table 7, from which the strong
complementarity of four types of attentions αsi, αci, αfi and αwi can be clearly observed.

**Inference Speed. Besides the size and the MAdds for a CNN model, the inference speed at runtime**
is very important in the practical model deployment. Table 8 provides a comparison of the inference
speed for different dynamic convolution methods (both on a GPU and a CPU). It can be seen that
the models trained with our ODConv (1×) are faster than the counterparts trained with DyConv and
DCD on a CPU. Comparatively, the models trained with CondConv show the fastest run-time speed
both on a GPU and a CPU, this is because CondConv is merely added to the final several blocks and
the last fully connected layer as discussed in the Method section. When adding ODConv (1×) to the
same layer locations as for CondConv, we can obtain more accurate and efficient models.

**More Ablations. In the Appendix, we provide more ablative experiments to study: (1) the effects**
of applying ODConv to different layer locations; (2) the importance of the temperature annealing
strategy; (3) the choices of the activation functions for πi(x); (4) the influence of the attention
sharing strategy; (5) the stability of the model training process; (6) other potentials of ODConv.

5 CONCLUSION

In this paper, we present a new dynamic convolution design called Omni-dimensional Dynamic
Convolution (ODConv) to promote the representation power of deep CNNs. ODConv leverages a
multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels
along all four dimensions of the kernel space in a parallel manner, and progressively applying these
attentions to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN. Experimental results on the ImageNet and
MS-COCO datasets with various prevailing CNN architectures validate its superior performance.


-----

REFERENCES

Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan NovDoll´ak, Alex Harvill, Pradeep
Sen, Tony DeRose, and Fabrice Rousselle. Kernel-predicting convolutional networks for denoising monte carlo renderings. In Siggraph, 2017.

Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Gradcam++: Improved visual explanations for deep convolutional networks. In WACV, 2018.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen
Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie
Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli
Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and
benchmark. arXiv preprint arXiv:1906.07155, 2019.

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic
convolution: Attention over convolution kernels. In CVPR, 2020.

Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and
motion network. In ICCV, 2019.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.

David Ha, Andrew M. Dai, and Quoc V. Le. Hyperneteworks. In ICLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.

Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Girshick. Mask r-cnn. In ICCV, 2017.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun
Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching
for mobilenetv3. In ICCV, 2019.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand,
Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for
mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.

Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature
context in convolutional neural networks. In NeurIPS, 2018a.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, 2018b.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected
convolutional networks. In CVPR, 2017.

Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer
networks. In NIPS, 2015.

Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In
_NIPS, 2016._

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.

HyunJae Lee, Hyo-Eun Kim, and Hyeonseob Nam. Srm: A style-based recalibration module for
convolutional neural networks. In ICCV, 2019.

Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, 2019.

Yunsheng Li, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Ye Yu, Yuan Lu,
Zicheng Liu, Mei Chen, and Nuno Vasconcelos. Revisiting dynamic convolution via matrix decomposition. In ICLR, 2021.


-----

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Doll´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.

Tsung-Yi Lin, Piotr Doll´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.
Feature pyramid networks for object detection. In CVPR, 2017.

Xudong Lin, Lin Ma, Wei Liu, and Shih-Fu Chang. Context-gated convolution. In ECCV, 2020.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic
segmentation. In CVPR, 2015.

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for
efficient cnn architecture design. In ECCV, 2018.

Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space
of weight networks. In ECCV, 2020.

Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll.
Burst denoising with kernel prediction networks. In CVPR, 2018.

Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou. Rotate to attend: Convolutional triplet attention module. In WACV, 2021.

Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017.

Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Bam: Bottleneck attention
module. In BMVC, 2018.

Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture
search via parameter sharing. In ICML, 2018.

Niamul Quader, Md Mafijul Islam Bhuiyan, Juwei Lu, Peng Dai, and Wei Li. Weight excitation:
Built-in attention mechanisms in convolutional neural networks. In ECCV, 2020.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object
detection with region proposal networks. In NIPS, 2015.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng
Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei
Li. Imagenet large scale visual recognition challenge. IJCV, 2015.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image
recognition. In ICLR, 2015.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In
_CVPR, 2015._

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking
the inception architecture for computer vision. In CVPR, 2016.

Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang,
and Xiaoou Tang. Residual attention network for image classification. In CVPR, 2017.

Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net:
Efficient channel attention for deep convolutional neural networks. In CVPR, 2020.

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block
attention module. In ECCV, 2018.

Saining Xie, Ross Girshick, Piotr Doll´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.


-----

Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. In NeurIPS, 2019.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical
risk minimization. In ICLR, 2018a.

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient
convolutional neural network for mobile devices. In CVPR, 2018b.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR,
2017.


-----

A APPENDIX

In this section, we describe the supplementary materials including: (1) a computational cost analysis of ODConv; (2) implementation details for the experiments on the ImageNet and MS-COCO
datasets; (3) more ablative experiments on the ImageNet dataset; (4) illustrative training and validation curves to compare the stability of the model training process with different dynamic convolution
methods; (5) more experiments to study other potentials of ODConv.

A.1 COMPUTATIONAL COST OF ODCONV

For a convolutional layer with ODConv, its computational cost can be easily calculated according to
the design and the multiplication process of four types of attentions shown in Fig. 1 and Fig. 2, respectively. Specifically, following the notations defined in the Method section of the main paper, the
extra MAdds of ODConv (1×) over regular convolution (which has hwk[2]cincout MAdds, without
consideration of the bias term) can be calculated as


_MAdds = hwcin +_ _[c][in][(2][c][in][ +][ c][out][ +][ k][2][)]_


+ k[2]cin(1 + 2cout). (3)


For ODConv (n×), the extra MAdds over regular convolution can be calculated as


_MAdds = hwcin +_ _[c][in][(2][c][in][ +][ c][out][ +][ k][2][ +][ n][)]_


+ k[2]cin(1 + cout + 2ncout). (4)


Compared to hwk[2]cincout MAdds for regular convolution, the extra MAdds by ODConv are small.

A.2 EXPERIMENTAL DETAILS

**Experimental Details on ImageNet. Recall that we use MobileNetV2 (Sandler et al., 2018) and**
ResNet (He et al., 2016) families for experiments on the ImageNet dataset, covering both lightweight CNN architectures and larger ones. For fair comparisons, we adopt the popular training
settings used in the community to train the respective backbone models with all methods. Specifically, for ResNet18, ResNet50 and ResNet101, all models are trained with SGD for 100 epochs. We
set the batch size as 256, the weight decay as 0.0001 and the momentum as 0.9. The learning rate
starts at 0.1, and is divided by 10 every 30 epochs. Following DyConv (Chen et al., 2020), for our
ODConv, we also use dropout rate of 0.1 for ResNet18. We use dropout rate of 0.2 for ResNet50
and ResNet101 for our ODConv. For MobileNetV2 (1.0×, 0.75×, 0.5×), all models are trained with
SGD for 150 epochs (we also have the experiments for training all models with 300 epochs in this
Appendix). We set the batch size as 256, the weight decay as 0.00004 and the momentum as 0.9. The
learning rate starts at 0.05, and is scheduled to arrive at zero within a single cosine cycle. Following
DyConv, for our ODConv, we also use dropout rate of 0.2 for MobileNetV2 (1.0×) and dropout
rate of (0.1, 0) for MobileNetV2 (0.75×, 0.5×). Regarding the temperature annealing strategy used
for DyConv and ODConv, the temperature reduces from 30 to 1 linearly in the first 10 epochs for
all models. All experiments are performed on the servers having 8 GPUs. We follow the standard
protocols to train and evaluate each CNN backbone. For training, images are resized to 256 × 256
first, and then 224 × 224 crops are randomly sampled from the resized images or their horizontal
flips normalized with the per-channel mean and standard deviation values. For evaluation, we report
top-1 and top-5 recognition rates using the center image crops.

**Experimental Details on MS-COCO. Recall that we use the popular MMDetection toolbox (Chen**
et al., 2019) for experiments on the MS-COCO dataset with the pre-trained ResNet50 and MobileNetV2 (1.0×) models as the backbones for the detector. We select the mainstream Faster RCNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) detectors with Feature Pyramid Networks (FPNs) (Lin et al., 2017) as the necks to build the basic object detection systems. For a neat
and fair comparison, the convolutional layers in the FPN necks still use regular convolutions, and we
maintain the same data preparation pipeline and hyperparameter settings for all pre-trained models
built with CondConv, DyConv and our ODConv, respectively. All experiments are performed on
a server having 8 GPUs with a mini-batch size of 2 images per GPU. We finetune these detectors
on the MS-COCO training set following the 1× learning rate schedule, which indicates a total of
12 epochs with the learning rate divided by 10 at the 8[th] epoch and the 11[th] epoch, respectively.


-----

Following DyConv, the temperature annealing strategy is not used in the downstream experiments
on the MS-COCO dataset, as it is prone to worse results. In the validation, we report the standard
Average Precision (AP) under IOU thresholds ranging from 0.5 to 0.95 with an increment of 0.05.
We also keep AP scores for small, medium and large objects.

A.3 MORE ABLATIVE EXPERIMENTS ON IMAGENET

Besides the ablative experiments described in the main paper, here we provide more ablative experiments, for a better understanding of our ODConv.

**Effects of Applying ODConv to Different Layer Locations. In the Experiments section of the**
main paper, we show that applying dynamic convolution to less layers leads to a faster runtime speed
compared to applying it to more layers. For easy implementation, we follow DyConv and apply
ODConv to all convolutional layers except the first layer of each CNN architecture tested in our main
experiments. However, it is necessary to study the effects of applying ODConv to different layer
locations. The performance comparison of adding ODConv to different layers of the MobileNetV2
(0.5×) backbone is given in Table 9, showing that our default setting (i.e., adding ODConv to all
convolutional layers except the first layer) has the largest gain. This indicates that adding ODConv
to more convolutional layers tends to have a model with higher recognition accuracy.

Table 9: Results comparison of adding ODConv to different layers of the MobileNetV2 (0.5×)
backbone. All models are trained on the ImageNet dataset for 150 epochs, and we set r = 1/16 and
_n = 1. For the inference speed, all pre-trained models are tested on an NVIDIA TITAN X GPU_
(with batch size 200) and a single core of Intel E5-2683 v3 CPU (with batch size 1) separately, and

|he input image size is 224 × 224 pix|xels. Be|est resul|lts are bolde|ed.|Col6|Col7|
|---|---|---|---|---|---|---|
|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|Speed on GPU|Speed on CPU|
|MobileNetV2 (0.5×)|2.00M|97.1M|64.30|85.21|2840.9|30.1|
|+ ODConv (to all conv layers except the first layer) + ODConv (to all 1×1 conv layers) + ODConv (to all 3×3 conv layers) + ODConv (to last 6 residual blocks)|2.43M 2.26M 2.17M 2.31M|101.8M 100.2M 99.1M 99.1M|68.26 67.43 67.54 67.29|87.98 87.41 87.55 87.39|1837.4 2083.3 2064.9 1879.5|18.3 27.1 27.4 19.6|



**Importance of Temperature Annealing Strategy. Following DyConv, we also use the temperature**
annealing strategy to facilitate the model training process of our ODConv. To study its effect, we
perform a set of ablative experiments on the ImageNet dataset, training the ResNet18 backbone
based on ODConv with or without using the temperature annealing strategy. From the results shown
in Table 10, we can find that the temperature annealing strategy is really important, which brings
1.06%|1.04% top-1 improvement to the ResNet18 model based on ODConv (1×)|ODConv (4×).

Table 10: Analyzing the effect of the temperature annealing when training ResNet18 with ODConv.

|ed on the Imag|geNet dataset, and w|we set r = 1/|/4. Best resul|
|---|---|---|---|
|Models|Temperature Annealing|Top-1 Acc (%)|Top-5 Acc (%)|
|ResNet18|-|70.25|89.38|
|+ ODConv (1×)|✓ -|73.41 72.35|91.29 90.65|
|+ ODConv (4×)|✓ -|74.33 73.29|91.53 90.95|



**Choices of the Activation Functions. Recall that in the structure of πi(x) for our ODConv, we**
use two different activation functions (either a Sigmoid function or a Softmax function) to compute
four types of attentions αsi, αci, αfi and αwi. It is also critical to compare the performance of
different activation functions for ODConv. Since the activation function choices for the channel
dimension and the convolutional kernel dimension have been thoroughly discussed in the papers
of SE and DyConv respectively, we adopt their suggestions for three related attentions defined in
our ODConv. Here we perform another set of ablative experiments, focusing on the choices of the
activation function for the spatial dimension. With the ImageNet dataset, we use ResNet18 as the test
backbone network to explore the effects of different activation functions for computing the attention
of ODConv along the spatial dimension of the kernel space. Table 11 summarizes the results, from
which we can see that the Sigmoid function performs better than the Softmax function, and thus we
use the Sigmoid function to compute the attention scalars along the spatial dimension for ODConv.

**Attention Sharing Strategy. As we discussed in the Method section, we share three attentions αsi,**
_αci and αfi to all convolutional kernels for the easy implementation as well as for more efficient_


-----

Table 11: Results comparison of ODConv with different activation functions. All models are trained

|taset, and we se|et r = 1/4. Best r|results are bold|ded.|
|---|---|---|---|
|Models|Activation Function|Top-1 Acc (%)|Top-5 Acc (%)|
|ResNet18|-|70.25|89.38|
|+ ODConv (1×)|Sigmoid Softmax|73.41 73.23|91.29 91.19|
|+ ODConv (4×)|Sigmoid Softmax|74.33 73.97|91.53 91.50|



Table 12: Results comparison of ODConv with or without using the attention sharing strategy. All
models are trained on the ImageNet dataset. Best results are bolded.

|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|
|ResNet18|11.69M|1.814G|70.25|89.38|
|+ ODConv w/ attention sharing (r = 1/16, 4×) + ODConv w/o attention sharing (r = 1/16, 4×)|44.90M 45.43M|1.916G 1.950G|73.97 74.16|91.35 91.47|



training. Actually, training separate αsi, αci and αfi for each additive convolutional kernel will
bring further improved model accuracy, as can be seen from the ablative results in Table 12. That is,
when preferring to get more accurate models, the attention sharing strategy would be removed.

**Training MobileNetV2 Backbones for 300 Epochs. Recall that in the Experiments section of the**
main paper, we compare our ODConv with CondConv, DyConv and DCD on MobileNetV2 (1.0×,
0.75×, 0.5×), following the popular training settings used in the community, where all models
are trained for 150 epochs. To validate the effectiveness of our ODConv further, we also conduct
another set of experiments using an increased number of training epochs (300 instead of 150) as
adopted in DyConv (Chen et al., 2020). All the other training settings remain the same to those for
the experiments described in the main paper. For a clean comparison of all methods, we do not use
advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al.,
2016). Results are summarized in Table 13 where performance gains show a similar trend as those
in Table 1 for the model training with 150 epochs. Results in Fig. 3 further show that our ODConv
gets a better trade-off between model accuracy and size even on the light-weight MobileNetV2
backbones compared to CondConv and DyConv.

Table 13: Results comparison on the ImageNet validation set with MobileNetV2 (1.0×, 0.75×,
0.5×) as the backbones. All models are trained for 300 epochs. Best results are bolded.

|Models|Params|MAdds|Top-1 Acc (%)|
|---|---|---|---|
|MobileNetV2 (1.0×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|3.50M 22.88M 12.40M 5.72M 4.94M 11.52M|300.8M 318.1M 317.1M 318.4M 311.8M 327.1M|72.07 74.31(↑2.24) 75.08(↑3.01) 74.48(↑2.41) 75.13(↑3.06) 75.68(↑3.61)|
|MobileNetV2 (0.75×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|2.64M 17.51M 7.95M 4.08M 3.51M 7.50M|209.1M 223.9M 220.1M 222.9M 217.1M 226.3M|69.76 72.18(↑2.42) 73.48(↑3.72) 72.39(↑2.63) 73.03(↑3.27) 74.45(↑4.69)|
|MobileNetV2 (0.5×) + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|2.00M 13.61M 4.57M 3.06M 2.43M 4.44M|97.1M 110.0M 103.2M 105.6M 101.8M 106.4M|65.10 68.45(↑3.35) 69.83(↑4.73) 70.08(↑4.98) 69.16(↑4.06) 70.87(↑5.77)|



A.4 ILLUSTRATION OF MODEL TRAINING AND VALIDATION CURVES

Fig. 4 illustrates the training and validation accuracy curves of the ResNet18 models trained on the
ImageNet dataset with CondConv, DyConv, ODConv (1×) and ODConv (4×), respectively. We can
see that our ODConv shows consistent high top-1 gains throughout the training process compared
to the other three dynamic convolution counterparts.


-----

Figure 3: Comparison of model accuracy and size for the pre-trained MobileNetV2 models based
on different dynamic convolution methods. All models are trained for 150 epochs on the ImageNet
dataset. It can be seen that our ODConv (1×) makes a better accuracy and size tradeoff for the lightweight MobileNetV2 backbones compared to CondConv and DyConv. On the larger ResNet18 and
ResNet50 backbones, even better results are obtained by ODConv, as can be seen from the results
shown in Table 2 of the main paper.

Figure 4: Curves of top-1 training accuracy (dashed line) and validation accuracy (solid line) of
the ResNet18 models trained on the ImageNet dataset with CondConv (8×), DyConv (4×), our
ODConv (1×) and ODConv (4×), respectively. Comparatively, our ODConv (1×) outperforms
both CondConv and DyConv yet has only 14.68%|26.26% parameters of the model trained with
CondConv|DyConv. Our ODConv (4×) converges with the best validation accuracy, which outperforms CondConv|DyConv by 1.98%|1.21% top-1 gain with fewer parameters.

A.5 MORE EXPERIMENTS FOR STUDYING OTHER POTENTIALS OF ODCONV

In this section, we provide a lot of extra experiments conducted for studying other potentials of
ODConv.


-----

**Performance Comparison on A CNN Backbone Added with the SE Module. Recall that SE (Hu**
et al., 2018b) performs attention on the output features of a convolutional layer, while our ODConv
performs attention on the convolutional kernels, and thus they are likely to be complimentary. To
validate this, we perform another set of experiments on the ImageNet dataset with ResNet18 backbone. In the experiments, we use the ResNet18+SE variant reported in Table 2 as the backbone, and
train this backbone with CondConv, DyConv, DCD and our ODConv separately, adopting the same
training settings used for the ResNet18 backbone. Results are shown in Table 14. It can be seen that
our ODConv brings significantly large performance improvement to the ResNet18 backbone already
incorporating the SE module, showing its great combination potential.

Table 14: Results comparison on the ImageNet validation set with the ResNet18+SE backbone

|00 epochs. For our OD|DConv, we|set r = 1|1/16. Best results|s are bolded.|
|---|---|---|---|---|
|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|ResNet18|11.69M|1.814G|70.25|89.38|
|+ SE + SE & ODConv(4×)|11.78G 44.99M|1.816G 1.918G|70.98 (↑0.73) 74.04 (↑3.79)|90.03 (↑0.65) 91.38 (↑2.00)|



**Performance Boost Using Heavy Data Augmentations and a Longer Training Schedule. Recall**
that in the main paper, we do not use advanced training tricks such as mixup (Zhang et al., 2018a)
and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons. To
better explore the potential of our ODConv, it would be necessary to have a study about whether
ODConv can also work well when using heavy data augmentations and a longer training schedule.
To this end, we perform a set of experiments on the ImageNet dataset with the ResNet18 backbone trained with ODConv (4×). In the experiments, we set r = 1/16 for our method, first train
ResNet18 with label smoothing, mixup and (label smoothing+mixup) separately for 100 epochs (the
other training settings are the same to those used for Table 2), and then train ResNet18 with (label
smoothing+mixup) for 120 epochs instead of 100 epochs. Detailed results are summarized in Table 15. It can be seen that our ODConv works well when using aggressive data augmentations and a
longer training schedule, showing further improved performance.

Table 15: Results comparison of the ResNet18 models based on ODConv with aggressive data
augmentations and a longer training schedule. All models are trained on the ImageNet dataset. Best

|ts are bolded.|Col2|Col3|
|---|---|---|
|Models|Top-1 Acc (%)|Top-5 Acc (%)|
|ResNet18 + ODConv (r = 1/16, 4×)|73.97|91.35|
|+ Label Smoothing + Mixup + Label Smoothing & Mixup + Label Smoothing & Mixup & Longer Training Schedule|74.15 (↑0.18) 74.05 (↑0.08) 74.18 (↑0.21) 74.59 (↑0.62)|91.42 (↑0.07) 91.63 (↑0.28) 91.68 (↑0.33) 91.74 (↑0.39)|



**Feature Pooling Strategy. Note that when computing four types of attentions in our ODConv, the**
inputs features are always globally averaged, reducing their spatial dimension to one pixel. According to the first three results of Table 7 in the main paper, the spatial convolutional kernel attention
_αsi brings the largest gain to the baseline model compared to the other two attentions (αci and αfi_
computed along the dimension of the input channels and the output channels, respectively). Intuitively, at the first glance, it may easily lead to a counterintuitive feeling as the input features used to
predict αsi (as well as the other three attentions) do not include spatial information after the global
average pooling. However, it should be noted that our ODConv uses the attentions generated from
the input features to modify the convolutional kernels, which is quite different from popular selfcalibration attention mechanisms (e.g., SE (Hu et al., 2018b) and CBAM (Woo et al., 2018) use
the channel attentions generated from the output features of a convolutional layer to recalibrate the
output features themselves). As our ODConv is not a self-calibration attention module, reducing
the spatial dimension of the input features to one pixel (but the length of the reduced vector is large
enough) will not affect its promising performance. To validate this, we conduct another experiment
on the ImageNet dataset with the ResNet18 backbone. In the experiment, for αsi, we pool the input
features to have a size of 3×3 instead of 1×1, and show the results in Table 16. It can be seen that
preserving more spatial information only brings 0.06%|0.02% gain to top-1|top-5 accuracy, which
proves the effectiveness of our current design to a large degree.


-----

Table 16: Comparison of the feature pooling strategy in ODConv using different spatial sizes for the
reduced features. All ResNet18 models are trained on the ImageNet dataset. Best results are bolded.

|Models|Params|MAdds|Top-1 Acc (%)|Top-5 Acc (%)|
|---|---|---|---|---|
|ResNet18|11.69M|1.814G|70.25|89.38|
|+ ODConv (r = 1/4, 1×, pooling to 1×1) + ODConv (r = 1/4, 1×, pooling to 3×3)|12.58M 14.85M|1.839G 1.842G|73.41 (↑3.16) 73.47 (↑3.22)|91.29 (↑1.91) 91.31 (↑1.93)|



**Comparison of Training Cost. Recall that in the main paper, we provide a comparison of the**
inference speed for different dynamic convolution methods. Here, we further perform a set of experiments to compare the training cost of our ODConv, CondConv, DyConv and DCD. Specifically,
experiments are performed on the ImageNet dataset with the ResNet50 and MobileNetV2 backbones, using the same training settings as in the main paper. Regarding the training cost, we report
results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of hours
for the whole training) in Table 17. Regarding different dynamic convolution methods, it can be seen
that the training cost trend is somewhat similar to that for the runtime inference speed reported in Table 8. Generally, the training cost for dynamic convolution methods is obviously heavier (about 2×
to 4×) than that for the baseline model, which is mainly due to the time-intensive back-propagation
process for additive convolutional kernels weighted with input-dependent attentions.

Table 17: Comparison of the training cost for different dynamic convolution methods. All models
are trained on the ImageNet dataset using the server with 8 NVIDIA TITAN X GPUs. We report
results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of
hours for the whole training).

|Network|ResNet50|Col3|Col4|MobileNetV2 (1.0×)|Col6|Col7|
|---|---|---|---|---|---|---|
|Models|Batch Cost (second)|Epoch Cost (minute)|Total Cost (hour)|Batch Cost (second)|Epoch Cost (minute)|Total Cost (hour)|
|Baseline + CondConv (8×) + DyConv (4×) + DCD + ODConv (1×) + ODConv (4×)|0.216 0.429 0.569 0.414 0.513 0.896|18.4 36.4 48.0 35.1 43.4 75.5|31.7 62.8 82.3 60.4 73.5 129.3|0.113 0.175 0.310 0.282 0.301 0.402|9.7 15.1 26.4 24.0 25.6 34.0|25.3 39.3 67.9 61.9 65.7 86.8|



**A Deep Understanding of Learnt Attention Values. Note that our ODConv leverages a multi-**
dimensional attention mechanism with a parallel strategy to learn four types of attentions αsi, αci,
_αfi, αwi for convolutional kernels along all four dimensions of the kernel space at any convolution-_
al layer. In principle, progressively multiplying these four types of attentions to the convolutional
kernel in the location-wise, channel-wise, filter-wise and kernel-wise orders makes convolution operations be different w.r.t. all spatial locations, all input channels, all filters and all kernels for each
input sample, providing a performance guarantee to capture rich context cues. To have a better understanding of this powerful attention mechanism, it is necessary to study the learnt attention values.
To this end, we use the well-trained ResNet18 models reported in Table 7 and all 50,000 images in
the ImageNet validation set, and conduct a set of experiments to analyze the learnt attention values for αsi, αci, αfi, αwi and their full combination by providing lots of visualization examples
(obtained with the Grad-CAM++ method (Chattopadhyay et al., 2018)) and the statistical distributions across different layers. The detailed computational process and results are shown in Fig. 5 and
Fig. 6. We can get the following observations: (1) Four types of attentions αsi, αci, αfi and αwi are
complementary to each other by visualization examples, which echoes the conclusion from Table 7.
The most salient attention will combat the failure cases of the other three attentions, helping the
model to produce accurate predications; (2) All four types of attentions demonstrate varying attention value distribution trends across different layers of the trained model, showing their capability
to capture rich context cues; (3) Each attention has its own value distribution trend, which is more
diverse for αsi and αwi than for αci and αfi, showing αsi and αwi maybe more important to some
degree. Note that the dimension of attentions αsi, αci, αfi and αwi is quite different, e.g., 9 for αsi
with 3 × 3 convolutional kernels, 4 for αwi with n = 4, and typically several hundred for αci and
_αfi. This is another perspective why they are different and complementary in their design nature_
regarding all four dimensions of the convolutional kernel space for any convolutional layer.

**Limitations of ODConv. On the one side, according to the results shown in Table 1, Table 2 and**
Table 8, although our ODConv shows obviously better performance under the similar model size


-----

GT Image 𝛼𝑠𝑖 𝛼𝑐𝑖 𝛼𝑓𝑖 𝛼𝑤𝑖 𝑎𝑙𝑙


GT Image 𝛼𝑠𝑖 𝛼𝑐𝑖 𝛼𝑓𝑖 𝛼𝑤𝑖 𝑎𝑙𝑙

bow tie

grille

jean

(b)

GT Image 𝛼𝑠𝑖 𝛼𝑐𝑖 𝛼𝑓𝑖 𝛼𝑤𝑖 𝑎𝑙𝑙

crate

guillotine

score
board

(d)


anemone

fish

picket

fence

patio


(a)

GT Image 𝛼𝑠𝑖 𝛼𝑐𝑖 𝛼𝑓𝑖 𝛼𝑤𝑖 𝑎𝑙𝑙

band aid

CD player


plate rack

(c)


GT Image 𝛼𝑠𝑖 𝛼𝑐𝑖 𝛼𝑓𝑖 𝛼𝑤𝑖 𝑎𝑙𝑙


mobile

home

drumstick

geyser

(e)

Figure 5: Comparison of illustrative visualization results with Grad-CAM++ (Chattopadhyay et al.,
2018). Results are obtained from the pre-trained ResNet18 models (reported in Table 7) with αsi,
_αci, αfi, αwi, and all of them (i.e., ODConv (4×)), separately. (a) The model with αsi and the_
model with all four attentions make the right predications, while the other three models with one
single attention fail. (b) The model with αci and the model with all four attentions make the right
predications, while the other three models with one single attention fail. (c) The model with αfi and
the model with all four attentions make the right predications, while the other three models with one
single attention fail. (d) The model with αwi and the model with all four attentions make the right
predications, while the other three models with one single attention fail. (e) Only the model with all
four attentions makes the right predications, while the other four models with one single attention
fail. These visualization results further backup the conclusion observed from Table 7. Best viewed
with zoom-in.

(e.g., ODConv (4×) vs. DyConv (4×)), it leads to slightly increased FLOPs and introduces extra
latency to the runtime inference speed, as the proposed four types of convolutional kernel attentions
_αsi, αci, αfi and αwi introduce a bit more learnable parameters. Under the similar model size, the_
training cost of our ODConv is also heavier than reference methods (e.g., ODConv (4×) vs. DyConv
(4×)), according to the results shown in Table 17. On the other side, although we provide a wide
range of ablative studies to analyze the effect of different hyperparameters of our ODConv using the
ResNet18 backbone with the ImageNet dataset, and apply the resulting combination of the hyperparameters to all backbone networks, it is not the optimal setting to different backbone networks. This
suggests the way to reduce the computational cost of our method, finding a proper combination to
constrain the final model to the target speed and accuracy for a particular circumstance. Besides, the
potential of applying ODConv to more deep and large backbones beyond ResNet101 has not been
explored due to the constraint of our available computational resource.


-----

α𝑠𝑖

(a)

2

5

8

α𝑤𝑖

(b)

|0|1|2|
|---|---|---|
|3|4|5|
|6|7|8|


(d)


α𝑐𝑖

(c)


α𝑓𝑖


Figure 6: Comparison of the statistical distributions of the learnt attention values regarding αsi,
_αci, αfi and αwi across different layers of the pre-trained ResNet18 model with ODConv (4×)._
We run the model on all 50,000 images from the ImageNet validation dataset to first collect learnt
attention values for αsi, αci, αfi and αwi separately, and then compute the layer-wise statistical
distribution for each of them over all image samples. (a) For αsi, we show the mean attention value
for each weight of 3 3 convolutional kernels. (b) For αwi, we show the mean attention value for
_×_
each of 4 convolutional kernels. (c) For αci, we show the statistical distribution of the mean attention
value over all input channels corresponding to 4 convolutional kernels. (d) For αfi, we show the
statistical distribution of the mean attention value over all convolutional filters corresponding to 4
convolutional kernels. Best viewed with zoom-in.


-----

