Under review as a conference paper at ICLR 2022
BLUR IS AN ENSEMBLE:
SPATIAL SMOOTHINGS TO IMPROVE
ACCURACY, UNCERTAINTY, AND ROBUSTNESS
Anonymous authors
Paper under double-blind review
ABSTRACT
Bayesian neural networks (BNNs) have shown success in the areas of uncertainty
1
estimation and robustness. However, a crucial challenge prohibits their use in
2
practice. Bayesian NNs require a large number of predictions to produce reliable
3
results, leading to a signiﬁcant increase in computational cost. To alleviate this
4
issue, we propose spatial smoothing, a method that ensembles neighboring feature
5
map points of CNNs. By simply adding a few blur layers to the models, we
6
empirically show that spatial smoothing improves accuracy, uncertainty estimation,
7
and robustness of BNNs across a whole range of ensemble sizes. In particular,
8
BNNs incorporating spatial smoothing achieve high predictive performance merely
9
with a handful of ensembles. Moreover, this method also can be applied to canonical
10
deterministic neural networks to improve the performances. A number of evidences
11
suggest that the improvements can be attributed to the stabilized feature maps
12
and the ﬂattening of the loss landscape. In addition, we provide a fundamental
13
explanation for prior works—namely, global average pooling, pre-activation, and
14
ReLU6—by addressing them as special cases of spatial smoothing. These not
15
only enhance accuracy, but also improve uncertainty estimation and robustness by
16
making the loss landscape smoother in the same manner as spatial smoothing.
17
1
INTRODUCTION
18
70
75
80
1.8
2.0
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
VGG-16
VGG-19
67.5
70.0
72.5
75.0
77.5
80.0
Accuracy (%)
0.8
1.0
1.2
NLL
VGG-16
VGG-19
ResNet-18
ResNet-18
ResNet-50
ResNet-50
ResNeXt-50
Figure 1: Spatial smoothing improves
both accuracy and uncertainty (NLL).
Smooth means spatial smoothing. Down-
ward from left to the right (↘) means bet-
ter accuracy and uncertainty.
In a real-world environment where many unexpected
19
events occur, machine learning systems cannot be guar-
20
anteed to always produce accurate predictions. In or-
21
der to handle this issue, we make system decisions
22
more reliable by considering estimated uncertainties,
23
in addition to predictions. Uncertainty quantiﬁcation is
24
particularly crucial in building a trustworthy system in
25
the ﬁeld of safety-critical applications, including med-
26
ical analysis and autonomous vehicle control. However,
27
canonical deep neural networks (NNs)—or determinis-
28
tic NNs—cannot produce reliable estimations of uncer-
29
tainties (Guo et al., 2017), and their accuracy is often
30
severely compromised by natural data corruptions from
31
noise, blur, and weather changes (Engstrom et al., 2019;
32
Azulay & Weiss, 2019).
33
Bayesian neural networks (BNNs), such as Monte Carlo
34
(MC) dropout (Gal & Ghahramani, 2016), provide a
35
probabilistic representation of NN weights. They com-
36
bine a number of models selected based on weight prob-
37
ability to make predictions of desired results. Thanks to
38
this feature, BNNs have been widely used in the areas of uncertainty estimation (Kendall & Gal, 2017)
39
and robustness (Ovadia et al., 2019). They are also promising in other ﬁelds like out-of-distribution
40
detection (Malinin & Gales, 2018) and meta-learning (Yoon et al., 2018).
41
1
Under review as a conference paper at ICLR 2022
Prediction
Importance
Input
t=0
t=-1
t=-2
Temporal smoothing
✕
✕
✕
✕
π0
x0
x-1
x-2
π-1
π-2
p0
p-1
p-2
Temporally proximate 
data stream
Spatial smoothing
✕
✕
p0
p1
p2
p3
x0
x2
x1
x3
π0
π2
π1
π3
Spatially proximate 
data points
BNN
✕
✕
✕
✕
N
/
1
N
/
1
N
/
1
x0
p0
p1
p2
Observed 
data
Figure 2: Comparison of three different Bayesian neural network inferences: canonical BNN
inference, temporal smoothing (Park et al., 2021), and spatial smoothing (ours). In this ﬁgure, x0 is
observed data, pi is predictions p(y|x0, wi) or p(y|xi, wi), πi is importances π(xi|x0), and N is
ensemble size.
Nevertheless, there remains a signiﬁcant challenge that prohibits their use in practice. BNNs require
42
an ensemble size of up to ﬁfty to achieve high predictive performance, which results in a ﬁftyfold
43
increase in computational cost (Kendall & Gal, 2017; Loquercio et al., 2020). Therefore, if BNNs
44
can achieve high predictive performance merely with a handful of ensembles, they could be applied
45
to a much wider range of areas.
46
1.1
PRELIMINARY
47
We would ﬁrst like to discuss BNN inference in detail, then move on to Vector-Quantized BNN
48
(VQ-BNN) inference (Park et al., 2021), an efﬁcient approximated BNN inference.
49
BNN inference.
Suppose we have access to posterior probability of NN weight p(w|D) for training
dataset D. The predictive result of BNN is given by the following predictive distribution:
p(y|x0, D) =
Z
p(y|x0, w) p(w|D) dw
(1)
where x0 is observed input data vector, y is output vector, and p(y|x, w) is the probabilistic prediction
parameterized by the result of NN for an input x and weight w. In most cases, the integral cannot be
solved analytically. Thus, we use the MC estimator to approximate it as follows:
p(y|x0, D) ≃
N−1
X
i=0
1
N p(y|x0, wi)
(2)
where wi ∼p(w|D) and N is the number of the samples. The equation indicates that BNN inference
50
is ensemble average of NN predictions for one observed data point as shown on the left of Fig. 2.
51
Using N neural networks in the ensemble would requires N times more computational complexity
52
than one NN execution.
53
2
Under review as a conference paper at ICLR 2022
100
101
Ensemble Size
0.80
0.90
1.00
NLL
100
101
Ensemble Size
76.0
78.0
Accuracy (%)
100
101
Ensemble Size
4.0
6.0
8.0
ECE (%)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure 3: Spatial smoothing improves both accuracy and uncertainty across a whole range of
ensemble sizes. We report the predictive performance of ResNet-18 on CIFAR-100.
Data-complemented BNN inference.
To reduce the computational cost of BNN inference, VQ-
BNN (Park et al., 2021) executes NN for an observed data only once and complements the result
with previously calculated predictions for other data. If we have access to previous predictions, the
computational performance of VQ-BNN becomes comparable to that of one NN execution. To be
speciﬁc, VQ-BNN inference is:
p(y|x0, D) ≃
N−1
X
i=0
π(xi|x0) p(y|xi, wi)
(3)
where π(xi|x0) is the importance of data xi with respect to the observed data x0, and it is deﬁned as a
54
similarity between xi and x0. p(y|x0, w0) is the newly calculated prediction, and {p(y|x1, w1), · · · }
55
are previously calculated predictions. To accurately infer the results, the previous predictions should
56
consist of predictions for “data similar to the observed data”.
57
Thanks to the temporal consistency of real-world data streams, aggregating predictions for similar
58
data in data streams is straightforward. Since temporally proximate data sequences tend to be similar,
59
we can memorize recent predictions and calculates their average using exponentially decreasing
60
importance. In other words, VQ-BNN inference for data streams is simply temporal smoothing of
61
recent predictions as shown in the middle of Fig. 2.
62
VQ-BNN has two limitations, although it may be a promising approach to obtain reliable results
63
in an efﬁcient way. First, it was only applicable to data streams such as video sequences. Applying
64
VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.
65
Second, Park et al. (2021) used VQ-BNN only in the testing phase, not in the training phase. We ﬁnd
66
that ensembling predictions for similar data helps in NN training by smoothing the loss landscape.
67
1.2
MAIN CONTRIBUTION
68
1
Spatially neighboring points in visual imagery tend to be similar, as do feature maps of convolu-
69
tional neural networks (CNNs). By exploiting this spatial consistency, we propose spatial smoothing
70
as a method of ensembling nearby feature maps to improve the efﬁciency of ensemble size in BNN
71
inference. The right side of Fig. 2 visualizes spatial smoothing aggregating neighboring feature maps.
72
2
We empirically demonstrate that spatial smoothing improves the efﬁciency in vision tasks, such
73
as image classiﬁcation on CIFAR (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015),
74
without any additional training parameters. Figure 3 shows that negative log-likelihood (NLL) of
75
“MC dropout + spatial smoothing” with an ensemble size of two is comparable to that of vanilla MC
76
dropout with an ensemble size of ﬁfty. We also demonstrate that spatial smoothing improves accuracy,
77
uncertainty, and robustness all at the same time. Figure 1 shows that spatial smoothing improves both
78
the accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size of
79
ﬁfty on CIFAR-100.
80
3
Global average pooling (GAP) (Lin et al., 2014; Zhou et al., 2016), pre-activation (He et al.,
81
2016b), and ReLU6 (Krizhevsky & Hinton, 2010; Sandler et al., 2018) have been widely used in vision
82
tasks. However, their motives are largely justiﬁed by the experiments. We provide an explanation for
83
these methods by addressing them as special cases of spatial smoothing. Experiments support the
84
claim by showing that the methods improve not only accuracy but also uncertainty and robustness.
85
3
Under review as a conference paper at ICLR 2022
2
PROBABILISTIC SPATIAL SMOOTHING
86
Stage i
Stage i+1
H✕W✕C
H/2✕W/2✕2C
H✕W✕C
Stage i
Stage i+1
Smooth
Prob
(tanh-ReLU)
Blur
(AvgPool)
H/2✕W/2✕2C
H✕W✕C
Figure 4: Stages of CNNs such as ResNet (left) and the
stages incorporating spatial smoothing layer (right).
To improve the computational perfor-
87
mance of BNN inference, VQ-BNN
88
(Park et al., 2021) executes NN pre-
89
diction only once and complements
90
the result with previously calculated
91
predictions. The key to the success
92
of this approach largely depends on
93
the collection of previous predictions
94
for proximate data. Gathering tempo-
95
rally proximate data and their predic-
96
tions from data streams is easy be-
97
cause recent data and predictions can
98
be aggregated using temporal consis-
99
tency. On the other hand, gathering
100
time-independent proximate data, e.g. images, is more difﬁcult because they lack such consistency.
101
2.1
MODULE ARCHITECTURE FOR ENSEMBLING NEIGHBORING FEATURE MAP POINTS
102
So instead of temporal consistency, we use spatial consistency—where neighboring pixels of images
103
are similar—for real-world images. Under this hypothesis, we take the feature maps as predictions
104
and aggregate neighboring feature maps.
105
Most CNN architectures, including ResNet, consist of multiple stages that begin with increasing the
number of channels while reducing the spatial dimension of the input volume. We decompose an
entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows:
p(zi+1|zi, D) =
Z
p(zi+1|zi, wi) p(wi|D) dwi
(4)
where zi is input volume of the i-th stage, and the ﬁrst and the last volume are input data and
106
output. wi and p(wi|D) are NN weight in the i-th stage and its probability. p(zi+1|zi, wi) is output
107
probability of zi+1 with respect to the input volume zi. To derive the probability from the output
108
feature map, we transform each point of the feature map into a Bernoulli distribution. To do so, a
109
composition of tanh and ReLU, a function from value of range [−∞, ∞] into probability, is added
110
after each stage. Put shortly, we use neural networks for point-wise binary feature classiﬁcation.
111
Since Eq. (4) is a kind of BNN inference, it can be approximated using Eq. (3). In other words, each
112
stage predicts feature map points only once and complements predictions with similar feature maps.
113
Under spatial consistency, it averages probabilities of spatially neighboring feature map points, which
114
is well known as blur operation in image processing. For the sake of implementation simplicity,
115
average pooling with a kernel size of 2 and a stride of 1 is used as a box blur. This operation ensembles
116
four neighboring probabilities with the same importances.
117
In summary, as shown in Fig. 4, we propose the following probabilistic spatial smoothing layer:
Smooth(z) = Blur ◦Prob (z)
(5)
where Prob(·) is a point-wise function from a feature map to probability, and Blur(·) is importance-
118
weighted average for ensembling spatially neighboring probabilities from feature maps. Smooth layer
119
is added after each stage. Prob and Blur are further elaborated below.
120
Prob: Feature map to probability.
Prob is a function that transforms a real-valued feature map
into probability. We use tanh–ReLU composition for this purpose. However, tanh is commonly
known to suffer from the vanishing gradient problem. To alleviate this issue, we propose the following
temperature-scaled tanh:
tanhτ(z) = τ tanh (z/τ)
(6)
where τ is a hyperparameter called temperature. τ is 1 in conventional tanh and ∞in identity
121
function. tanhτ imposes an upper bound on a value, but does not limit the upper bound to 1.
122
4
Under review as a conference paper at ICLR 2022
c1 s1
c2
s2
c3
s3 c4 s4
Blocks
0.000
0.025
0.050
0.075
0.100
Feature map variance
(a) Model uncertainty
c1 s1
c2
s2
c3
s3 c4 s4
Blocks
0.00
0.05
0.10
0.15
0.20
Feature map variance
(b) Data uncertainty
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
MC dropout
MC dropout + Smooth
Figure 5: Spatial smoothing layers reduce feature map variances, suggesting that they ensemble
feature map points. We provide standard deviation of feature maps by block depth with ResNet-50 on
CIFAR-100. c1 to c4 and s1 to s4 each stand for stages and spatial smoothing layers, respectively.
Model uncertainty is represented by the average standard deviation of several feature maps obtained
from multiple NN executions. Data uncertainty is represented by the standard deviation of feature
map points obtained from one NN execution.
An unnormalized probability, ranging from 0 to τ, is allowed as the output of Prob. Then, thanks to
the linearity of integration, we obtain an unnormalized predictive distribution accordingly. Taking
this into account, we propose the following Prob:
Prob(z) = ReLU ◦tanhτ(z)
(7)
where τ > 1. We empirically determine τ to minimize NLL, a metric that measures both accuracy
123
and uncertainty. See Fig. B.3 for more detailed ablation studies. In addition, we expect upper-bounded
124
functions, e.g., ReLU6(z) = ReLU ◦min(z, 6) and feature map scaling z/τ with τ > 1 which
125
is BatchNorm, to be able to replace tanhτ in Prob; and as expected, these alternatives improve
126
uncertainty estimation in addition to accuracy. See Appendix C.2 and Appendix C.3 for detailed
127
discussions on activation (ReLU ◦BatchNorm) and ReLU6 as Prob.
128
Blur: Averaging neighboring probabilities.
Blur averages the probabilities from feature maps.
We primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation
of Blur for the sake of simplicity. Nevertheless, we could generalize Blur by using the following
depth-wise convolution, which acts on each input channel separately, with non-trainable kernel
K =
1
||k||2
1
k ⊗k⊤
(8)
where k is a 1D matrix, e.g., k ∈{(1) , (1, 1) , (1, 2, 1) , (1, 4, 6, 4, 1)}. Different ks derive different
129
importances for neighboring feature maps. We empirically show that most Blurs improve the
130
predictive performance and that optimal K varies by model. For more ablation studies, see Table B.2.
131
2.2
HOW DOES SPATIAL SMOOTHING HELP OPTIMIZATION?
132
We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.
133
Feature map variance.
BNNs have two types of uncertainties: One is model uncertainty and the
134
other is data uncertainty (Park et al., 2021). These randomnesses increase the variance of the feature
135
maps. To demonstrate that spatial smoothing is an ensemble, we use the following proposition:
136
Proposition 1. Ensembles reduce the variance of predictions.
137
We omit the proof since it is straightforward. In our context, predictions are output feature maps of a
138
stage. We investigate model and data uncertainties of the predictions along NN layers to show that
139
spatial smoothing reduces the randomnesses and ensembles feature maps. Figure 5 shows the model
140
uncertainty and data uncertainty of Bayesian ResNet including MC dropout layers. In this ﬁgure, the
141
5
Under review as a conference paper at ICLR 2022
1.0 π
0.0 π
w
(a) Frequency mask
0.0
0.5
1.0
Frequency
2.0
4.0
6.0
Log amplitude
(b) Log amplitude at c1
0.2
0.5
0.8
Noise frequency
60
65
70
Accuracy (%)
(c) Robustness for noise frequency
0.0
0.5
1.0
Frequency
0.0
1.0
2.0
Log amplitude
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure 6: MC dropout adds high-frequency noises, and spatial smoothing ﬁlters high-frequency
signals. In these experiments, we use ResNet-50 for ImageNet. Left: Frequency mask Mf with
w = 0.1π. Middle: Diagonal components of Fourier transformed feature maps at the end of the
stage 1. Right: The accuracy against frequency-based random noise. ResNets are vulnerable to
high-frequency noises. Spatial smoothing improves the robustness against high-frequency noises.
uncertainty of MC dropout’s feature map only accumulates, and almost monotonically increases in
142
every NN layer. In contrast, the uncertainty of “MC dropout + spatial smoothing”’s feature map is
143
signiﬁcantly decreases at the end of stages, suggesting that the smoothing layers ensemble the feature
144
map. In other words, they make the feature map more accurate and stabilized input volumes for the
145
next stages. In addition, consistently, the spatial smoothing layer close to the last layer signiﬁcantly
146
improves performance because it reduces the uncertainty of predictions largely. See Fig. B.5 for more
147
detailed results. Deterministic NNs do not have model uncertainty but data uncertainty. Therefore,
148
spatial smoothing improves the performance of deterministic NNs as well as Bayesian NNs.
149
Fourier analysis.
We also analyze spatial smoothing through the lens of Fourier transform:
150
Proposition 2. Ensembles ﬁlter high-frequency signals.
151
The proof is provided in Eqs. (16) to (17). Figure 6b shows the 2D Fourier transformed output
152
feature map at the end of the stage 1. This ﬁgure reveals that MC dropout almost does not affect
153
low-frequency (< 0.3π) ranges, and it adds high-frequency (≥0.3π) noises. Since spatial smoothing
154
is a low-pass ﬁlter, it effectively ﬁlters high-frequency signals, including the noises caused by MC
155
dropout.
156
We also ﬁnd that CNNs are particularly vulnerable to high-frequency noises. To demonstrate this
157
claim, following Shao et al. (2021), we measure accuracy with respect to data with frequency-based
158
random noise xnoise = x0 +F−1 (F(δ) ⊙Mf), where x0 is clean data, F(·) and F−1(·) are Fourier
159
transform and inverse Fourier transform, δ is random noise, and Mf is frequency mask as shown
160
in Fig. 6a. Figure 6c exhibits the results. In sum, high-frequency noises, including those caused by
161
MC dropout, signiﬁcantly impair accuracy. Spatial smoothing improves the robustness by effectively
162
removing high-frequency noises.
163
Loss landscape.
Lastly, we show that the randomness hinders NN training as follows:
164
Proposition 3. Randomness of predictions sharpens the loss landscape, and ensembles ﬂatten it.
165
The proof is provided in Eqs. (18) to (25). Since a sharp loss function disturbs NN optimization
166
(Keskar et al., 2017; Santurkar et al., 2018; Foret et al., 2020), reducing the uncertainty helps NN
167
learn strong representations. For example, training phase NN ensemble averages out the randomness,
168
and it ﬂattens the loss function. In consequence, an ensemble of BNN outputs in training phase
169
signiﬁcantly improves the predictive performance. See Fig. D.4 for numerical results. However, we
170
do not use training phase ensemble because it signiﬁcantly increases the training time. Instead, we
171
use spatial smoothing as a method that ensembles feature maps without sacriﬁcing training time.
172
We visualizes the loss landscapes (Li et al., 2018), the contours of NLL on training dataset. Figure 8b
173
shows that the loss landscapes of MC dropout ﬂuctuate and have irregular surfaces due to the
174
6
Under review as a conference paper at ICLR 2022
(a) MLP classiﬁer
(b) GAP classiﬁer (vanilla)
(c) GAP classiﬁer + Smooth
Figure 8: Both GAP and spatial smoothing smoothen the loss landscapes. To demonstrate this,
we present the loss landscape visualizations of ResNet-18 models with MC dropout on CIFAR-100.
0
500
1000
Max Eigenvlaue
Density
MLP
GAP
GAP + Smooth
Figure 7: Both GAP and spa-
tial smoothing suppress large
Hessian eigenvalue outliers,
i.e., they ﬂatten the loss land-
scapes. Compare with Fig. 8.
randomness. As Li et al. (2018); Foret et al. (2020) pointed out,
175
this may lead to poor generalization and predictive performance.
176
Spatial smoothing reduces randomness as discussed above, and
177
spatial smoothing aids in optimization by stabilizing and ﬂattening
178
the loss landscape of BNN as shown in Fig. 8c.
179
Furthermore, we use Hessian to quantitatively represent the sharp-
180
ness of the loss landscapes. Figure 7 shows the Hessian max eigen-
181
value spectra of the models in Fig. 8 with a batch size of 128, which
182
reveals that spatial smoothing reduces the magnitude of Hessian
183
eigenvalues and suppresses outliers. Since large Hessian eigenval-
184
ues disturb NN training (Ghorbani et al., 2019), we come to the
185
same conclusion that spatial smoothing helps NN optimization. See
186
Appendix C.1 for a more detailed description of the conﬁgurations
187
of the Hessian max eigenvalue spectra. In addition, from these
188
observations, we propose the conjecture that the ﬂatter the loss
189
landscape, the better the uncertainty estimation, and vice versa.
190
2.3
REVISITING GLOBAL AVERAGE POOLING
191
Table 1: MLP does not overﬁt the
training dataset. We report train-
ing NLL (NLLtrain) and testing NLL
(NLLtest) of ResNet-50 on CIFAR-100.
CLASSIFIER
NLLtrain
NLLtest
GAP
0.0061
0.822
MLP
0.0071
1.029
The success of GAP classiﬁer in image classiﬁcation is
192
indisputable. The initial motivation and the most widely
193
accepted explanation for this success is that GAP prevents
194
overﬁtting by using far fewer parameters than multi-layer
195
perceptron (MLP) (Lin et al., 2014). However, we discover
196
that the explanation is poorly supported. We compares
197
GAP with other classiﬁers including MLP. Contrary to
198
popular belief, Table 1 suggests that MLP does not overﬁt
199
the training dataset. MLP underﬁts or gives comparable
200
performance to GAP on the training dataset. On the test
201
dataset, GAP provides better results compared with MLP. See Table C.1 for more detailed results.
202
Our argument is that GAP is an extreme case of spatial smoothing. In other words, GAP is successful
203
because it ensembles feature maps and smoothens the loss landscape to help optimization. To support
204
this claim, we visualizes the loss landscape of MLP as shown in Fig. 8a. It is chaotic compared to
205
that of GAP as shown in Fig. 8b. Hessian shows the consistent results as demonstrated by Fig. 7.
206
3
EXPERIMENTS
207
This section presents two experiments. The ﬁrst experiment is image classiﬁcation through which
208
we show that spatial smoothing not only improves the ensemble efﬁciency, but also the accuracy,
209
uncertainty, and robustness of both deterministic NN and MC dropout. The second experiment is
210
semantic segmentation on data streams through which we show that spatial smoothing and temporal
211
smoothing (Park et al., 2021) are complementary. See Appendix A for more detailed conﬁgurations.
212
7
Under review as a conference paper at ICLR 2022
100
101
Ensemble Size
0.92
0.94
0.96
NLL
100
101
Ensemble Size
75.5
76.0
76.5
77.0
Accuracy (%)
100
101
Ensemble Size
2.0
2.5
3.0
3.5
ECE (%)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure 9: Spatial smoothing also improves predictive performance on large datasets. We report
predictive performance of ResNet-50 on ImageNet.
Three metrics are measured in these experiments: NLL (↓1), accuracy (↑), and expected calibration
213
error (ECE, ↓) (Guo et al., 2017). NLL represents both accuracy and uncertainty, and is the most
214
widely used as a proper scoring rule. ECE measures discrepancy between accuracy and conﬁdence.
215
3.1
IMAGE CLASSIFICATION
216
This section mainly discuss ResNet (He et al., 2016a). Table E.1 also discuss other settings that
217
show the same trend: e.g., VGG (Simonyan & Zisserman, 2015), ResNeXt (Xie et al., 2017),
218
and pre-activation models (He et al., 2016a). Spatial smoothing also improves deep ensemble
219
(Lakshminarayanan et al., 2017), another non-Bayesian probabilistic NN method. See Fig. E.1.
220
Performance.
Fig. 3 and Fig. 9 show the predictive performances of ResNet-18 on CIFAR-100
221
and ResNet-50 on ImageNet, respectively. The results indicate that spatial smoothing improves both
222
accuracy and uncertainty in many respects. Let us be more speciﬁc. First, spatial smoothing improves
223
the efﬁciency of ensemble size. In these examples, the NLL of “MC dropout + spatial smoothing”
224
with an ensemble size of 2 is comparable to or even better than that of MC dropout with an ensemble
225
size of 50. In other words, “MC dropout + spatial smoothing” is 25× faster than MC dropout with
226
a similar predictive performance. Second, the predictive performance of “MC dropout + spatial
227
smoothing” is better than that of MC dropout, at an ensemble size of 50. Third, spatial smoothing
228
improves the predictive performance of deterministic NN, as well as MC dropout.
229
100
101
Ensemble Size
60
65
70
mCNLL (%)
MC dropout
MC dropout + Smooth
Deterministic
Deterministic + Smooth
Figure 10: Spatial smoothing
improves the robustness. See
Fig. E.3 for more details.
Robustness.
To evaluate robustness against data corruption, we
230
measure predictive performance of ResNet-18 on CIFAR-100-
231
C (Hendrycks & Dietterich, 2019). This dataset consists of data
232
corrupted by 15 different types, each with 5 levels of intensity
233
each. We use mean corruption NLL (mCNLL, ↓), the averages
234
of NLL over intensities and corruption types, to summarize the
235
performance of corrupted data in a single value. See Eq. (32) for
236
a more rigorous deﬁnition. Figure 10 shows that spatial smoothing
237
not only improves the efﬁciency but also corruption robustness
238
across a whole range of ensemble size. See Fig. E.3 for more
239
details. Spatial smoothing also improves adversarial robustness
240
and perturbation consistency (↑) (Hendrycks & Dietterich, 2019;
241
Zhang, 2019a), shift-transformation invariance. See Table E.2,
242
Table E.3, and Fig. E.4 for more details.
243
3.2
SEMANTIC SEGMENTATION
244
Table 2 summarizes the result of semantic segmentation on CamVid dataset (Brostow et al., 2008)
245
that consists of real-world 360×480 pixels videos. The table shows that spatial smoothing improves
246
predictive performance, which is consistent with the image classiﬁcation experiment. Moreover, the
247
result reveals that spatial smoothing and temporal smoothing (Park et al., 2021) are complementary.
248
See Table E.4 for more results.
249
1We use arrows to indicate which direction is better.
8
Under review as a conference paper at ICLR 2022
Table 2: Spatial smoothing and temporal smoothing are complementary. We provide predictive
performance of MC dropout in semantic segmentation. SPAT and TEMP each stand for spatial
smoothing and temporal smoothing. ACC and CONS stand for accuracy and consistency. The numbers
in brackets denote the performance improvements over the baseline.
SPAT
TEMP
NLL
ACC
(%)
ECE
(%)
CONS
(%)
·
·
0.298 (-0.000)
92.5 (+0.0)
4.20 (-0.00)
95.4 (+0.0)
✓
·
0.284 (-0.014)
92.6 (+0.1)
3.96 (-0.24)
95.6 (+0.2)
·
✓
0.273 (-0.025)
92.6 (+0.1)
3.23 (-0.97)
96.4 (+1.0)
✓
✓
0.260 (-0.038)
92.6 (+0.1)
2.71 (-1.49)
96.5 (+1.1)
4
RELATED WORK
250
Spatial smoothing can be compared with prior works in the following areas.
251
Anti-aliased CNNs.
Local means (Zhang, 2019a; Zou et al., 2020; Vasconcelos et al., 2020; Sinha
252
et al., 2020) were introduced for the shift-invariance of deterministic CNNs in image classiﬁcation.
253
They were motivated to prevent the aliasing effect of subsampling. Although the local ﬁltering can
254
result in a loss of information, Zhang (2019a) experimentally observed an increase in accuracy that
255
was beyond expectation. We provide a fundamental explanation for this phenomenon: Local means
256
are a spatial ensemble. An ensemble not only improves accuracy, but also uncertainty and robustness
257
of deterministic and Bayesian NNs. In Fig. F.1, we also show that the predictive performance
258
improvement is not due to anti-aliasing of local mean. See Appendix F for more discussion on local
259
means. For a discussion on non-local means (Wang et al., 2018) and self-attention (Dosovitskiy et al.,
260
2021), see Section 5.
261
Sampling-free BNNs.
Sampling-free BNNs (Hernández-Lobato & Adams, 2015; Wang et al.,
262
2016; Wu et al., 2019) predict results based on a single or couple of NN executions. To this end, it is
263
assumed that posterior and feature maps follow Gaussian distributions. However, the discrepancy
264
between reality and assumption accumulates in every NN layer. Consequently, to the best of our
265
knowledge, most of the sampling-free BNNs could only be applied to shallow models, such as LeNet,
266
and were tested on small datasets. Postels et al. (2019) applied sampling-free BNNs to SegNet;
267
nonetheless, Park et al. (2021) argued that they do not predict well-calibrated results.
268
Efﬁcient deep ensembles.
Deep ensemble (Lakshminarayanan et al., 2017; Fort et al., 2019) is
269
another probabilistic NN approach for predicting reliable results. BatchEnsemble (Wen et al., 2020;
270
Dusenberry et al., 2020) ensembles over a low-rank subspace to make deep ensemble more efﬁcient.
271
Depth uncertainty network (Antoran et al., 2020) aggregates feature maps from different depths of
272
a single NN to predict results efﬁciently. Despite being robust against data corruption, it provides
273
weaker predictive performance compared to deterministic NN and MC dropout.
274
5
DISCUSSION
275
We propose spatial smoothing, a simple yet efﬁcient module to improve BNN. Three different per-
276
spectives, namely, feature map variance, Fourier analysis, and loss landscape, suggest that spatial
277
smoothing ensembles feature maps. The limitation of spatial smoothing is that designing its compo-
278
nents requires inductive bias. In other words, the optimal shape of the blur kernel is model-dependent.
279
We believe this problem can be solved by introducing self-attention (Vaswani et al., 2017). Self-
280
attentions for computer vision (Dosovitskiy et al., 2021; Touvron et al., 2021; Carion et al., 2020)
281
can be deemed as trainable importance-weighted ensembles of feature maps. The observation that
282
Transformers are more robust than expected (Bhojanapalli et al., 2021; Shao et al., 2021) supports this
283
claim. Therefore, using self-attentions to generalize spatial smoothing would be a promising future
284
work because it not only expands our work, but also helps deepen our understanding of self-attention.
285
9
Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
286
To ensure reproducibility, we provide comprehensive resources, such as code and experimental details.
287
The codebase will be released as open source under the Apache License 2.0. See the supplemental
288
material for the code. Appendix A provides the speciﬁcations of all models used in this work. Detailed
289
experimental setup including hyperparameters and ablation study are also available in Appendix A
290
and Appendix B. De-facto image datasets are used for all experiments as described in Appendix A.
291
REFERENCES
292
Javier Antoran, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural
293
networks. Advances in Neural Information Processing Systems, 2020.
294
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small
295
image transformations? Journal of Machine Learning Research, 2019.
296
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and
297
Andreas Veit. Understanding robustness of transformers for image classiﬁcation. In Proceedings
298
of the IEEE/CVF International Conference on Computer Vision, 2021.
299
Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog-
300
nition using structure from motion point clouds. In European Conference on Computer Vision.
301
Springer, 2008.
302
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey
303
Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer
304
Vision. Springer, 2020.
305
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
306
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image
307
is worth 16x16 words: Transformers for image recognition at scale. In International Conference
308
on Learning Representations, 2021.
309
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji
310
Lakshminarayanan, and Dustin Tran. Efﬁcient and scalable bayesian neural nets with rank-1
311
factors. In International Conference on Machine Learning. PMLR, 2020.
312
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring
313
the landscape of spatial robustness. In International Conference on Machine Learning. PMLR,
314
2019.
315
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization
316
for efﬁciently improving generalization. In International Conference on Learning Representations,
317
2020.
318
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec-
319
tive. arXiv preprint arXiv:1912.02757, 2019.
320
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training batchnorm and only batchnorm:
321
On the expressive power of random features in cnns. In International Conference on Learning
322
Representations, 2021.
323
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model
324
uncertainty in deep learning. In International Conference on Machine Learning. PMLR, 2016.
325
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and
326
Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves
327
accuracy and robustness. In International Conference on Learning Representations, 2019.
328
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization
329
via hessian eigenvalue density. In International Conference on Machine Learning. PMLR, 2019.
330
10
Under review as a conference paper at ICLR 2022
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
331
examples. International Conference on Learning Representations, 2015.
332
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola,
333
Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet
334
in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
335
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural
336
networks. In International Conference on Machine Learning. PMLR, 2017.
337
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
338
recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
339
2016a.
340
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual
341
networks. In European Conference on Computer Vision. Springer, 2016b.
342
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common
343
corruptions and perturbations. In International Conference on Learning Representations, 2019.
344
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning
345
of bayesian neural networks. In International Conference on Machine Learning. PMLR, 2015.
346
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoeﬂer, and Daniel Soudry. Augment
347
your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF
348
Conference on Computer Vision and Pattern Recognition, 2020.
349
A Kendall, V Badrinarayanan, and R Cipolla. Bayesian segnet: Model uncertainty in deep convolu-
350
tional encoder-decoder architectures for scene understanding. In BMVC, 2017.
351
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer
352
vision? Advances in Neural Information Processing Systems, 2017.
353
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter
354
Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In
355
International Conference on Learning Representations, 2017.
356
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished
357
manuscript, 2010.
358
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
359
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
360
uncertainty estimation using deep ensembles. In Advances in Neural Information Processing
361
Systems, 2017.
362
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape
363
of neural nets. In Advances in Neural Information Processing Systems, 2018.
364
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In International Conference on
365
Learning Representations, 2014.
366
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty
367
estimation in deep learning. IEEE Robotics and Automation Letters, 2020.
368
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
369
Towards deep learning models resistant to adversarial attacks. In International Conference on
370
Learning Representations, 2018.
371
A Malinin and M Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural
372
Information Processing Systems. Curran Associates, Inc., 2018.
373
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua
374
Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty?
375
evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing
376
Systems, 2019.
377
11
Under review as a conference paper at ICLR 2022
Namuk Park, Taekyu Lee, and Songkuk Kim. Vector quantized bayesian neural network inference
378
for data streams. In AAAI Conference on Artiﬁcial Intelligence, 2021.
379
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
380
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style,
381
high-performance deep learning library. Advances in Neural Information Processing Systems,
382
2019.
383
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling-
384
free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of
385
the IEEE/CVF International Conference on Computer Vision, 2019.
386
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical
387
image segmentation. In International Conference on Medical image computing and computer-
388
assisted intervention. Springer, 2015.
389
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,
390
Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet
391
Large Scale Visual Recognition Challenge. International Journal of Computer Vision, pp. 211–252,
392
2015.
393
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo-
394
bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on
395
Computer Vision and Pattern Recognition, 2018.
396
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal-
397
ization help optimization? Advances in Neural Information Processing Systems, 2018.
398
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness
399
of visual transformers. arXiv preprint arXiv:2103.15670, 2021.
400
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image
401
recognition. In International Conference on Learning Representations, 2015.
402
Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing. Advances in Neural
403
Information Processing Systems, 2020.
404
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
405
Jégou. Training data-efﬁcient image transformers & distillation through attention. In International
406
Conference on Machine Learning. PMLR, 2021.
407
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, and Ross Goroshin.
408
An effective anti-aliasing approach for residual networks. arXiv preprint arXiv:2011.10675, 2020.
409
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
410
Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information
411
Processing Systems, 2017.
412
Hao Wang, Xingjian Shi, and Dit-Yan Yeung. Natural-parameter networks: A class of probabilistic
413
neural networks. Advances in Neural Information Processing Systems, 2016.
414
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In
415
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
416
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efﬁcient
417
ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
418
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández-Lobato,
419
and Alexander L Gaunt. Deterministic variational inference for robust bayesian neural networks.
420
In International Conference on Learning Representations, 2019.
421
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual
422
transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer
423
Vision and Pattern Recognition, 2017.
424
12
Under review as a conference paper at ICLR 2022
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks
425
through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data).
426
IEEE, 2020.
427
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn.
428
Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems,
429
2018.
430
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
431
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.
432
Theoretically principled trade-off between robustness and accuracy. In International Conference
433
on Machine Learning. PMLR, 2019.
434
Richard Zhang. Making convolutional networks shift-invariant again. In International Conference on
435
Machine Learning. PMLR, 2019a.
436
Richard Zhang. Ofﬁcial meta-review of making convolutional networks shift-invariant again, 2019b.
437
URL https://openreview.net/forum?id=SklVEnR5K7&noteId=rklZnFS-gN.
438
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep
439
features for discriminative localization. In Proceedings of the IEEE Conference on Computer
440
Vision and Pattern Recognition, 2016.
441
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets.
442
In BMVC, 2020.
443
13
Under review as a conference paper at ICLR 2022
10
20
30
40
50
MC dropout rate (%)
0.8
0.9
1.0
1.1
NLL
10
20
30
40
50
MC dropout rate (%)
74
76
78
Accuracy (%)
10
20
30
40
50
MC dropout rate (%)
2.5
5.0
7.5
10.0
ECE (%)
20
40
MC dropout rate (%)
0.8
0.9
1.0
1.1
NLL
MC dropout (N=1)
MC dropout (N=50)
MC dropout + Smooth (N=1)
MC dropout + Smooth (N=50)
Figure A.1: Spatial smoothing improves predictive performance at all dropout rates. As the
dropout rate increases, both accuracy and ECE decrease. The performance is optimized when
accuracy and uncertainty are balanced.
A
EXPERIMENTAL SETUP AND DATASETS
444
We obtain the main experimental results with the Intel Xeon W-2123 Processor, 32GB memory, and
445
a single GeForce RTX 2080 Ti for CIFAR (Krizhevsky et al., 2009) and CamVid (Brostow et al.,
446
2008). For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core
447
Processor, 256GB memory, and four GeForce RTX 2080 Ti. We conduct ablation studies with four
448
Intel Intel Broadwell CPUs, 15GB memory, and a single NVIDIA T4. Models are implemented
449
in PyTorch(Paszke et al., 2019). The detailed conﬁgurations of image classiﬁcation and semantic
450
segmentation are as follows.
451
A.1
IMAGE CLASSIFICATION
452
We use VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016a), pre-activation ResNet (He
453
et al., 2016a), and ResNeXt (Xie et al., 2017) in image classiﬁcation. According to the structure
454
suggested by Zagoruyko & Komodakis (2016), each block of Bayesian NNs contains one MC dropout
455
layer.
456
NNs are trained using categorical cross-entropy loss and SGD optimizer with initial learning rate of
457
0.1, momentum of 0.9, and weight decay of 5 × 10−4. We also use multi-step learning rate scheduler
458
with milestones at 60, 130, and 160, and gamma of 0.2 on CIFAR, and with milestones at 30, 60,
459
and 80, and gamma of 0.2 on ImageNet. We train NNs for 200 epochs with batch size of 128 on
460
CIFAR, and for 90 epochs with batch size of 256 on ImageNet. We start training with gradual warmup
461
(Goyal et al., 2017) for 1 epoch on CIFAR. Basic data augmentations, namely random cropping and
462
horizontal ﬂipping, are used. One exception is the training of ResNeXt on ImageNet. In this case, we
463
use the batch size of 128 and learning rate of 0.05 because of memory limitation.
464
We use hyperparameters that minimizes NLL of ResNet. Table A.1 provides hyperparameters for
465
deterministic and Bayesian NNs. For fair comparison, models with and without spatial smoothing
466
share hyperparameters such as MC dropout rate. However, Fig. A.1 shows that spatial smoothing
467
improves predictive performance of ResNet-18 at all dropout rates on CIFAR-100. The default
468
ensemble size of MC dropout is 50. We report averages of three evaluations, and error bars in ﬁgures
469
represent min and max values. Standard deviations are omitted from tables for better visualization.
470
See the source code released on GitHub for other details.
471
A.2
SEMANTIC SEGMENTATION
472
We use U-Net (Ronneberger et al., 2015) in semantic segmentation. Following Bayesian SegNet
473
(Kendall et al., 2017), Bayesian U-Net contains six MC dropout layers. We add spatial smoothing
474
before each subsampling layer in U-Net encoder. We use 5 previous predictions and decay rate of
475
e−0.8 for temporal smoothing.
476
14
Under review as a conference paper at ICLR 2022
Table A.1: Hyperparameters of models for image classiﬁcation.
DATASET
MODEL
MC DROPOUT RATE
(%)
|k|
TEMPERATURE
CIFAR-10
& CIFAR-100
VGG
·
·
·
30
·
·
·
2
10
30
2
10
ResNet
·
·
·
30
·
·
·
2
10
30
2
10
Preact-ResNet
·
·
·
30
·
·
·
2
10
30
2
10
ResNeXt
·
·
·
30
·
·
·
2
10
30
2
10
ImageNet
ResNet
·
·
·
5
·
·
·
2
10
5
2
10
ResNeXt
·
·
·
5
·
·
·
2
10
5
2
10
CamVid consists of 720×960 pixels road scene video sequences. We resize the image bilinearly to
477
360×480 pixels. We use a list reduced to 11 labels by following previous works, e.g. (Kendall & Gal,
478
2017).
479
NNs are trained using categorical cross-entropy loss and Adam optimizer with initial learning rate of
480
0.001 and β1 of 0.9, and β2 of 0.999. We train NN for 130 epoch with batch size of 3. The learning
481
rate decreases to 0.0002 at the 100 epoch. Random cropping and horizontal ﬂipping are used for
482
data augmentation. Median frequency balancing is used to mitigate dataset imbalance. Other details
483
follow Park et al. (2021).
484
B
ABLATION STUDY
485
0.0
2.5
5.0
7.5
10.0
0
2
4
6
8
10
ReLU
tanh
ReLU6
Constant
BatchNorm
Figure B.1: Upper-bounded functions as
a candidates of Prob.
The probabilistic spatial smoothing proposed in this
486
paper consists of two components: Prob and Blur. This
487
section explores several candidates for each component
488
and their properties.
489
B.1
Prob: FEATURE MAPS TO PROBABILITIES
490
We deﬁne Prob as a composition of an upper-
491
bounded function and ReLU, a function that imposes
492
the lower bound of zero. Fig. B.1 shows widely used
493
upper-bounded functions: tanhτ(x) = τ tanh(x/τ),
494
ReLU6(x) = min(max(x, 6), 0), and constant scaling
495
which is x/τ.
496
Table B.1 shows the predictive performance improve-
497
ment by Prob with various upper-bounded functions on
498
15
Under review as a conference paper at ICLR 2022
Table B.1: We use tanh as the default for Prob based on the predictive performance of MC dropout
for CIFAR-100 with various Probs.
MODEL
SMOOTH
NLL
ACC
(%)
ECE
(%)
VGG-16
·
1.133 (-0.000)
68.8 (+0.0)
3.66 (+0.00)
ReLU ◦tanh
1.064 (-0.069)
70.4 (+1.6)
2.99 (-0.67)
ReLU ◦ReLU6
1.093 (-0.040)
69.8 (+1.0)
4.26 (+0.60)
ReLU ◦Constant
0.995 (-0.138)
72.5 (+3.7)
2.11 (-1.55)
Blur
0.985 (-0.000)
72.4 (+0.0)
1.77 (+0.00)
Blur ◦ReLU ◦tanh
0.984 (-0.001)
72.7 (+0.3)
2.07 (+0.30)
Blur ◦ReLU ◦ReLU6
0.982 (-0.003)
72.5 (+0.1)
1.84 (+0.07)
Blur ◦ReLU ◦Constant
0.991 (+0.005)
72.9 (+0.5)
1.03 (-0.74)
VGG-19
·
1.215 (-0.000)
67.3 (+0.0)
6.37 (+0.00)
ReLU ◦tanh
1.131 (-0.084)
69.2 (+1.9)
5.23 (-1.14)
ReLU ◦ReLU6
1.166 (-0.049)
68.3 (+1.0)
6.44 (-0.06)
ReLU ◦Constant
0.997 (-0.218)
72.5 (+5.2)
1.09 (-5.29)
Blur
1.039 (-0.000)
71.1 (+0.0)
3.12 (+0.00)
Blur ◦ReLU ◦tanh
1.034 (-0.005)
71.3 (+0.2)
3.31 (+0.19)
Blur ◦ReLU ◦ReLU6
1.038 (-0.002)
71.3 (+0.2)
3.84 (+0.72)
Blur ◦ReLU ◦Constant
0.995 (-0.045)
72.3 (+1.2)
1.41 (-1.71)
ResNet-18
·
0.848 (-0.000)
77.3 (+0.0)
3.01 (+0.00)
ReLU ◦tanh
0.838 (-0.010)
77.7 (+0.4)
2.92 (-0.08)
ReLU ◦ReLU6
0.844 (-0.004)
77.4 (+0.1)
2.74 (-0.27)
ReLU ◦Constant
0.825 (-0.023)
77.7 (+0.4)
1.87 (-1.14)
Blur
0.806 (-0.000)
78.6 (+0.0)
2.56 (+0.00)
Blur ◦ReLU ◦tanh
0.801 (-0.005)
78.9 (+0.3)
2.56 (-0.01)
Blur ◦ReLU ◦ReLU6
0.805 (-0.001)
78.9 (+0.2)
2.59 (+0.03)
Blur ◦ReLU ◦Constant
0.811 (+0.005)
78.5 (-0.2)
1.84 (-0.72)
ResNet-50
·
0.822 (-0.000)
79.1 (+0.0)
6.63 (+0.00)
ReLU ◦tanh
0.812 (-0.010)
79.3 (+0.2)
6.74 (+0.11)
ReLU ◦ReLU6
0.799 (-0.023)
79.4 (+0.3)
6.71 (+0.08)
ReLU ◦Constant
0.788 (-0.034)
79.6 (+0.5)
5.22 (-1.41)
Blur
0.798 (-0.000)
80.0 (+0.0)
7.21 (+0.00)
Blur ◦ReLU ◦tanh
0.800 (+0.002)
80.1 (+0.1)
7.25 (+0.04)
Blur ◦ReLU ◦ReLU6
0.800 (+0.002)
80.2 (+0.2)
7.30 (+0.09)
Blur ◦ReLU ◦Constant
0.779 (-0.019)
80.4 (+0.4)
5.81 (-1.40)
16
Under review as a conference paper at ICLR 2022
0.0
2.5
5.0
7.5
10.0
0.0
2.0
4.0
6.0
8.0
10.0
0.0
2.5
5.0
7.5
10.0
0.00
0.25
0.50
0.75
1.00
0.0
2.5
5.0
7.5
10.0
0
2
4
6
8
10
ReLU
= 0.5
= 1
= 2
= 5
= 10
Figure B.2: Temperature-scaled tanhs (left) and their ﬁrst derivatives (right) for different tem-
peratures.
100
101
102
Temperature
0.8
0.9
1.0
NLL
100
101
102
Temperature
76
77
78
79
Accuracy (%)
100
101
102
Temperature
3.0
4.0
5.0
6.0
ECE (%)
100
101
102
Temperature
0.8
0.9
1.0
NLL
MC dropout + Smooth (N=1)
MC dropout + Smooth (N=50)
Figure B.3: The temperature controls the trade-off between accuracy and uncertainty. The
accuracy increases as the temperature increases, but predictions become more overconﬁdent.
CIFAR-100. In this experiment, we use models with MC dropout, and τ = 5 for constant scaling. The
499
results indicate that upper-bounded functions with ReLU tend to improve accuracy and uncertainty
500
at the same time. In addition, they show that Prob and Blur are complementary. The best results
501
are obtained when using both Prob and Blur. For the main experiments, we use the composition
502
of tanhτ and ReLU as Prob, because the hyperparameter of constant scaling is highly dependent on
503
dataset and model.
504
Temperature.
The characteristics of temperature-scaled tanh depends on τ. Figure B.2 plots
505
tanhτ and their ﬁrst derivatives with various temperatures. As shown in this ﬁgure, tanhτ has a
506
couple of useful properties. First, tanhτ has an upper bound of τ. Second, the ﬁrst derivative of
507
tanhτ at x = 0 does not depend on τ.
508
Fig. B.3 shows the predictive performance of ResNet-18 with MC dropout and spatial smoothing for
509
the temperature on CIFAR-100. In this ﬁgure, the accuracy increases as the temperature increases. In
510
terms of ECE, NN predicts more underconﬁdent results as τ decreases. It is a misinterpretation that
511
the result is overconﬁdent at low τ because ECE is high. By deﬁnition, ECE relies on the absolute
512
value of the difference between conﬁdence and accuracy. In this example, at low τ, the accuracy is
513
greater than the conﬁdence, which leads to a high ECE. Moreover, at τ = 0.2, ECE with N = 50 is
514
greater than that with N = 1, which means that the result is severely underconﬁdent. NLL, a metric
515
representing both accuracy and uncertainty, is minimized when the accuracy and the uncertainty are
516
balanced. In conclusion, we set the default value of τ to 10.
517
17
Under review as a conference paper at ICLR 2022
1.00
(a) k = (1)
0.25
0.25
0.25
0.25
(b) k = (1, 1)
0.06
0.12
0.06
0.12
0.25
0.12
0.06
0.12
0.06
(c) k = (1, 2, 1)
0.00
0.02
0.02
0.02
0.00
0.02
0.06
0.09
0.06
0.02
0.02
0.09
0.14
0.09
0.02
0.02
0.06
0.09
0.06
0.02
0.00
0.02
0.02
0.02
0.00
(d) k = (1, 4, 6, 4, 1)
Figure B.4: Kernels for Blur. Brighter background indicates higher importance.
Table B.2: The optimal shape of the blur kernel is model-dependent. We measure the predictive
performance of MC dropout using spatial smoothing with various size of Blur kernels on CIFAR-100.
MODEL
|k|
NLL
ACC
(%)
ECE
(%)
VGG-16
1
1.087 (-0.000)
69.8 (+0.0)
3.43 (-0.00)
2
1.034 (-0.053)
71.4 (+1.6)
1.06 (-2.37)
3
0.986 (-0.101)
72.7 (+2.9)
1.03 (-2.40)
5
1.018 (-0.069)
72.0 (+2.2)
1.32 (-2.11)
VGG-19
1
1.096 (-0.000)
69.8 (+0.0)
4.74 (-0.00)
2
1.071 (-0.025)
70.4 (+0.6)
2.15 (-2.59)
3
1.026 (-0.070)
71.9 (+2.1)
2.56 (-2.18)
5
1.032 (-0.064)
71.6 (+1.8)
2.16 (-2.58)
ResNet-18
1
0.840 (-0.000)
77.6 (+0.0)
2.63 (-0.00)
2
0.801 (-0.039)
78.9 (+1.4)
2.56 (-0.07)
3
0.822 (-0.018)
78.7 (+1.1)
2.86 (-0.23)
5
0.837 (-0.003)
78.4 (+0.8)
3.05 (-0.42)
ResNet-50
1
0.814 (-0.000)
79.5 (+0.0)
6.56 (-0.00)
2
0.806 (-0.008)
80.0 (+0.5)
7.35 (+0.79)
3
0.796 (-0.019)
79.9 (+0.4)
7.38 (+0.82)
5
0.816 (+0.001)
79.4 (-0.1)
7.38 (+0.82)
18
Under review as a conference paper at ICLR 2022
100
101
Ensemble Size
0.85
0.90
0.95
1.00
NLL
100
101
Ensemble Size
75
76
77
78
Accuracy (%)
100
101
Ensemble Size
2.0
4.0
6.0
8.0
ECE (%)
100
101
Ensemble Size
0.9
1.0
NLL
None
s1
s2
s3
s4
Figure B.5: Spatial smoothing close to the last layer (s3) signiﬁcantly improves performance.
We report predictive performance of ResNet-18 with one spatial smoothing after each stage on
CIFAR-100. None indicates vanilla MC dropout.
B.2
Blur: AVERAGING NEIGHBORING PROBABILITIES
518
Blur is a depth-wise convolution with a kernel. The kernel given by Eq. (8) is derived from various
519
ks such as k ∈{(1) , (1, 1) , (1, 2, 1) , (1, 4, 6, 4, 1)}. In these examples, if |k| is 1, Blur is identity.
520
If |k| is 2, Blur is a box blur, which is used in the main experiments. If |k| is 3 or 5, Blur is an
521
approximated Gaussian blur.
522
Table B.2 shows predictive performance of models using spatial smoothing with the kernels on
523
CIFAR-100. This results show that most kernels improve both accuracy and uncertainty. However,
524
the most effective kernel size depends on the model.
525
B.3
POSITION OF SPATIAL SMOOTHING.
526
As shown in Fig. 5, the magnitude of uncertainty tends to increase as the depth increases. Therefore,
527
we expect that spatial smoothing close to the output layer will mainly drive performance improvement.
528
We investigate the predictive performance of models with MC dropout using only one spatial smooth-
529
ing layer. Figure B.5 shows the predictive performance of ResNet-18 with one spatial smoothing after
530
each stage on CIFAR-100. The results suggest that spatial smoothing after s3 is the most important
531
for improving performance. Surprisingly, spatial smoothing after s4 is the least important. This is
532
because GAP, the most extreme case of spatial smoothing, already exists there.
533
C
REVISITING PRIOR WORKS
534
As mentioned in Section 2, prior works—namely, GAP, pre-activation, and ReLU6—are spacial cases
535
of spatial smoothing. This section discusses them in detail.
536
C.1
GLOBAL AVERAGE POOLING
537
The composition of GAP and a fully connected layer is the most popular classiﬁer in classiﬁcation
538
tasks. The original motivation and the most widely accepted explanation for the success is that GAP
539
classiﬁer prevents overﬁtting because it uses signiﬁcantly fewer parameters than MLP (Lin et al.,
540
2014). To verify this claim, we measure the predictive performance of MLP, GAP, and global max
541
pooling (GMaxP), a classiﬁer that uses the same number of parameters as GAP, on training dataset.
542
Predictive performance.
Table C.1 shows the experimental results on the training and the test
543
dataset of CIFAR-100, suggesting that the explanation is poorly supported. On both the training
544
and the test dataset, most predictive performance of MLP is worse than that of GAP. It is a counter-
545
intuitive result meaning that MLP do not overﬁt the training dataset. In addition, the performance
546
improvement by GAP is remarkable in VGG, which has irregular loss landscape. The predictive
547
19
Under review as a conference paper at ICLR 2022
Table C.1: MLP classiﬁer does not overﬁt training dataset, i.e., GAP does not regularize NNs. We
provide predictive performance of MC dropout with various classiﬁers on CIFAR-100. ERR is error.
MODEL
CLASSIFIER
TRAIN
TEST
NLL
ERR
(%)
ECE
(%)
NLL
ACC
(%)
ECE
(%)
VGG-16
GAP
0.0852
0.461
6.75
1.030
72.3
3.24
MLP
0.5492
13.1
13.8
1.133
68.8
3.66
GMaxP
0.0846
0.470
6.67
1.050
72.2
3.60
GMedP
0.0867
0.501
6.80
1.042
72.2
3.35
VGG-19
GAP
0.1825
2.50
10.4
1.035
71.9
1.46
MLP
0.7144
17.7
14.8
1.215
67.3
6.37
GMaxP
0.1939
2.85
10.6
1.063
71.5
2.10
GMedP
0.1938
2.80
10.6
1.051
71.7
1.70
ResNet-18
GAP
0.0124
0.0287
1.19
0.841
77.5
2.92
MLP
0.0076
0.0347
7.22
1.040
74.8
9.55
GMaxP
0.0113
0.0233
1.41
0.905
76.3
5.23
GMedP
0.0156
0.0347
1.46
0.889
76.4
5.03
ResNet-50
GAP
0.0061
0.0220
0.48
0.822
79.1
6.63
MLP
0.0071
0.0370
8.53
1.029
76.9
11.8
GMaxP
0.0074
0.0313
1.09
0.887
77.2
5.67
GMedP
0.0053
0.0287
0.47
0.849
78.5
6.29
Test
1
2
3
4
5
Intensity
2
4
6
8
10
NLL
Test
1
2
3
4
5
Intensity
25
50
75
Accuracy (%)
Test
1
2
3
4
5
Intensity
0
20
40
60
ECE (%)
0
1
2
3
4
5
Intensity
2
4
6
8
10
NLL
GAP
MLP
GMaxP
GMedP
100
101
Ensemble Size
60
80
100
mCNLL (%)
100
101
Ensemble Size
86
88
90
92
mCE (%)
100
101
Ensemble Size
40
60
80
100
mCECE (%)
100
101
Ensemble Size
60
80
100
mCNLL (%)
GAP
MLP
GMaxP
GMedP
Figure C.1: GAP classiﬁer improves not only the predictive performance on clean dataset but
also the robustness. We measure the predictive performance of ResNet-18 using MC dropout with
classiﬁers on CIFAR-100-C.
20
Under review as a conference paper at ICLR 2022
(a) MLP classiﬁer
(b) GAP classiﬁer
(c) GAP classiﬁer + Smooth
Figure C.2: GAP and spatial smoothing ﬂatten the loss landscapes. We visualize the loss landscape
sequences of ResNet-18 with MC dropout on CIFAR-100. Although each sequence shares the bases,
it ﬂuctuates due to the randomness of the MC dropout.
performance of GMaxP is better than that of MLP, but worse than that of GAP. This shows that using
548
fewer parameters partially helps to improve predictive performance; however, it is insufﬁcient to
549
explain the predictive performance improvement by GAP. Finally, global median pooling (GMedP)
550
provides better predictive performance than GMaxP. It implies that using other noise reduction
551
methods instead of average pooling helps to improve predictive performance.
552
Robustness.
To evaluate the robustness of the classiﬁers, we measure the predictive performance of
553
ResNet-18 using MC dropout with the classiﬁers on CIFAR-100-C. Figure C.1 shows the experimental
554
results. This ﬁgure suggests that MLP is not robust against data corruption, as we would expect. In
555
terms of accuracy, the robustness of GMaxP and GMedP is relatively comparable to that of GAP;
556
however, in terms of uncertainty, GAP is the most robust. These are consistent results with other
557
spatial smoothing experiments.
558
Loss landscape visualization.
To understand the mechanism of GAP performance improvement,
559
we investigate the loss landscape. Figure C.2 shows the loss landscape sequences of ResNet with
560
MC dropout. In this ﬁgure, each sequence shares the bases, but they ﬂuctuate due to the randomness
561
of the MC dropout. Figure C.2a is the loss landscape of the model using MLP classiﬁer instead of
562
GAP classiﬁer. The loss landscape is chaotic and irregular, resulting in hindering and destabilizing
563
NN optimization. Fig. C.2b is loss landscape sequence of ResNet with GAP classiﬁer. Since GAP
564
ensembles all of the feature map points at the last stage, it ﬂattens and stabilizes the loss landscape.
565
Likewise, as shown in Fig. C.2c, spatial smoothing layers at the end of all stages also ﬂattens and
566
stabilizes the loss landscape.
567
Hessian eigenvalue spectra.
To evaluate the smoothness of the loss landscapes quantitatively,
568
we also investigate their Hessians at the optimized weights. In particular, we calculate Hessian
569
eigenvalue spectra (Ghorbani et al., 2019), distributions of Hessian eigenvalues, to show how spatial
570
smoothing helps NN optimization. To this end, we try to use stochastic Lanczos quadrature algorithm
571
21
Under review as a conference paper at ICLR 2022
implemented by Yao et al. (2020). However, the problem is that the model with MLP classiﬁer
572
requires a lot of memory while the algorithm is memory inefﬁcient.
573
In the training phase, we calculate the mean gradients with respect to mini-batches, rather than the
574
entire dataset. Therefore, it may be reasonable to investigate the properties of the Hessian “mini-
575
batch-wisely”. For that purpose, we propose a method, Hessian max eigenvalue spectra, that evaluates
576
the distribution of “Hessian’s maximum eigenvalues for one mini-batch”. We use power iteration to
577
produce only the greatest eigenvalue of the Hessian. This algorithm is easy to implement and requires
578
signiﬁcantly less memory and computational cost, compared with stochastic Lanczos quadrature
579
with respect to entire dataset. With this method, we can investigate the Hessian of NNs with MLP
580
classiﬁers, which would require a lot of GPU memory.
581
0
10
20
Eigenvlaue
Log density
GAP
GAP + Smooth
Figure C.3: Spatial smoothing
suppress eigenvalue outliers.
We provide Hessian eigenvalue
spectra of ResNet-18 with MC
dropout on CIFAR-100. See also
Fig. 7.
Figure 7 shows the Hessian max eigenvalue spectra of GAP classi-
582
ﬁer models with and without spatial smoothing layers. As Li et al.
583
(2018); Foret et al. (2020) and Appendix D.3 pointed out, Hessian
584
eigenvalue outliers disturb NN training. This ﬁgure explicitly show
585
that the GAP and spatial smoothing reduce the magnitude of the
586
Hessian eigenvalues and suppress the outliers, which leads to the
587
same result as the previous visualizations: GAP as well as spatial
588
smoothing smoothen the loss landscape. In conclusion, averag-
589
ing feature map points tends to help neural network optimization
590
by smoothing, ﬂattening, and stabilizing the loss landscape. We
591
observe a similar phenomenon for deterministic NNs. We also
592
evaluate the Hesse eigenvalue spectrum as shown in Fig. C.3, and
593
it leads to the same conclusion.
594
In these experiments, we use MLP incorporating dropout layers
595
with a rate of 50% as the classiﬁer. Since the dropout is one of
596
the factors that makes MLP underﬁt the training dataset, we also
597
evaluate MLP using dropouts with a rate of 0%. Nevertheless, the
598
results still shows that the predictive performance of MLP is worse
599
than that of GAP on the training dataset. Moreover, it severely degrades predictive performance of
600
ResNet on the test dataset.
601
C.2
PRE-ACTIVATION
602
He et al. (2016b) experimentally showed that the pre-activation arrangement, in which the activation
603
ReLU ◦BatchNorm is placed before the convolution, improves the accuracy of ResNet. Since γs of
604
most BatchNorms in CNNs are near-zero (Frankle et al., 2021), BatchNorms reduce the magnitude
605
of feature maps. As shown in Fig. B.1, constant scaling is a non-trainable BatchNorm with no
606
bias, and it also reduces the magnitude of feature map. In Table B.1, we show that constant scaling
607
improves predictive performance. Considering the similarity between Prob with constant scaling and
608
conventional activation, i.e., the similarity between ReLU◦ConstantScaling and ReLU◦BatchNorm,
609
we ﬁnd that the pre-activation arrangement improves uncertainty as well as accuracy, because
610
convolutions act as a Blur.
611
To show this, we change the post-activation of all layers to pre-activation, and measure the predictive
612
performance. For ResNet, we follow the original paper by He et al. (2016b). Table C.2 shows
613
the predictive performance of models with pre-activation. The results suggests that pre-activation
614
improves both accuracy and uncertainty in most cases. For deterministic VGG-19, pre-activation
615
signiﬁcantly degrades accuracy but improves NLL. In conclusion, they imply that pre-activation is a
616
special case of spatial smoothing.
617
Santurkar et al. (2018) argued that BatchNorm helps in optimization by ﬂattening the loss landscape.
618
We show that spatial smoothing ﬂattens and smoothens the loss landscape, which is a consistent
619
explanation. It will be interesting to investigate if BatchNorm helps in ensembling feature maps.
620
C.3
RELU6
621
ReLU6 was experimentally introduced to improve predictive performance (Krizhevsky & Hinton,
622
2010). Sandler et al. (2018) used “ReLU6 as the non linearity because of its robustness when used
623
22
Under review as a conference paper at ICLR 2022
Table C.2: Pre-activation arrangement improves uncertainty as well as accuracy. We measure
the predictive performance of models with pre-activation arrangement on CIFAR-100.
MODEL
MC DROPOUT
PRE-ACT
NLL
ACC
(%)
ECE
(%)
VGG-16
·
·
2.047 (-0.000)
71.6 (+0.0)
19.2 (-0.0)
·
✓
1.827 (-0.219)
72.5 (+0.9)
19.8 (+0.6)
✓
·
1.133 (-0.000)
68.8 (+0.0)
3.66 (-0.00)
✓
✓
1.036 (-0.096)
71.7 (+2.9)
3.55 (-0.11)
VGG-19
·
·
2.016 (-0.000)
67.6 (+0.0)
21.2 (-0.0)
·
✓
1.799 (-0.217)
64.4 (-3.2)
17.2 (-4.0)
✓
·
1.215 (-0.000)
67.3 (+0.0)
6.37 (-0.00)
✓
✓
1.084 (-0.131)
70.1 (+3.7)
4.23 (-2.14)
ResNet-18
·
·
0.983 (-0.000)
77.1 (+0.0)
7.75 (-0.00)
·
✓
0.934 (-0.049)
77.6 (+0.5)
8.04 (+0.29)
✓
·
0.937 (-0.000)
76.9 (+0.0)
5.11 (-0.00)
✓
✓
0.872 (-0.065)
77.6 (+0.7)
5.53 (+0.42)
ResNet-50
·
·
0.880 (-0.000)
79.0 (+0.0)
8.35 (-0.00)
·
✓
0.870 (-0.010)
79.4 (+0.4)
8.27 (-0.08)
✓
·
0.831 (-0.000)
78.6 (+0.0)
6.06 (-0.00)
✓
✓
0.819 (-0.012)
79.5 (+0.9)
6.29 (+0.23)
with low-precision computation”. In Table B.1, we show that ReLU6s at the end of stages helps to
624
ensemble spatial information by transforming the feature map to Bernoulli distributions. Since spatial
625
smoothing improves robustness against data corruption, it seems reasonable that ReLU6 is robust to
626
low-precision computation. A more abundant investigation into this topic is promising future works.
627
We measure the predictive performance of NNs using all activations as ReLU6 instead of ReLU.
628
However, in contrast to the results in Table B.1, the results are not consistent. We speculate that the
629
reason is that a lot of ReLU6s overly regularize NNs.
630
D
EXTENDED ANALYSIS OF HOW SPATIAL SMOOTHING WORKS
631
This section provides further explanation of the analysis in Section 2.2.
632
D.1
NEIGHBORING FEATURE MAPS IN CNNS ARE SIMILAR
633
This work exploits the spatial consistency of feature maps, i.e., neighboring feature maps in CNNs
634
are similar. Below, we theoretically and empirically prove the spatial consistency. Moreover, this
635
spatial consistency of feature maps holds even if the input data is spatially inconsistent.
636
Consider a single-layer convolutional neural network with one channel:
yi = [w ∗x]i
(9)
=
k
X
l=1
wlxi−l+1
(10)
23
Under review as a conference paper at ICLR 2022
(a) single-layer CNN
(b) ﬁve-layer CNN with ReLU
Figure D.1: Neighboring feature map points in CNNs are similar, even if input values are iid.
We provide covariances of feature map points with respect to the center feature map (in the red
square). Input values are Gaussian random noise. Left: A single convolutional layer correlates the
target feature map with another feature map that is 3 pixels away, since the kernel size is 3×3. Right:
A deep CNN more strongly correlates neighboring feature maps.
where ∗is convolution operator with a kernel of size k, y is feature map output, w is kernel weight,
and x is input random variable. Then, the covariance of two neighboring feature maps is:
Cov(yi, yi+1) = Cov(
k
X
l=1
wlxi−l+1,
k
X
m=1
wmxi−m+2)
(11)
=
k
X
l=1
k
X
m=1
wlwm Cov(xi−l+1, xi−m+2)
(12)
=
k−1
X
l=1
wlwl+1 σ2(xi−l+2) + · · ·
(13)
where σ2(xi−l+1) is the variance of xi−l+1. Therefore, Cov(yi, yi+1) is non-zero for randomly
637
initialized weights. If x is iid, i.e., Cov(xi, xj) = δijσ2(xi) where δij is the Kronecker delta, the
638
remainders in Eq. (13) vanish.
639
For example, the covariance of two neighboring feature map points in a CNN with a kernel size of 3
is:
Cov(y1, y2) = w1w1 Cov(x1, x2) + w1w2 Cov(x1, x3) + w1w3 Cov(x1, x4)
+ w2w1 Cov(x2, x2) + w2w2 Cov(x2, x3) + w2w3 Cov(x2, x4)
+ w3w1 Cov(x3, x2) + w3w2 Cov(x3, x3) + w3w3 Cov(x3, x4)
(14)
When xi is iid, the covariance is:
Cov(y1, y2) = w1w2 σ2(x2) + w2w3 σ2(x3)
(15)
Since it is non-zero, the neighboring feature maps y1 and y2 are correlated.
640
Experiment.
To demonstrate the spatial consistency of feature maps empirically, we provide
641
feature map covariances of randomly initialized single-layer CNN and ﬁve-layer CNN with ReLU
642
non-linearity. In this experiment, the input values are Gaussian random noises. As shown in Fig. D.1a,
643
one convolutional layer correlates neighboring feature map points. Fig. D.1b shows that multiple
644
convolutional layers correlate one feature map with distant feature maps. Moreover, the feature maps
645
in deep CNNs have a stronger relationship with neighboring feature maps.
646
D.2
ENSEMBLE FILTERS HIGH-FREQUENCY SIGNALS
647
Following the notation of Eq. (3), the ensemble is convolution of importance π and prediction p:
π ∗p
(16)
24
Under review as a conference paper at ICLR 2022
where πi,j = π(xi|xj) and pi = p(y|xi, wi). To show that this ensemble is low-pass ﬁlter, we
apply the convolution N times:
π ∗· · · ∗π
|
{z
}
N times
∗p
(17)
Since π is probability, i.e., P
i πi,j = 1, π ∗· · · ∗π is the probability for the sum of N random
648
variables from π, i.e, φ + · · · + φ ∼π ∗· · · ∗π where φ ∼π. By deﬁnition, an operator is
649
low-pass ﬁlter if and only if the high frequency component vanishes when the operator is applied
650
inﬁnitely. Therefore, ensemble with π is low-pass ﬁlter because Var(φ + · · · + φ) = N Var(φ) and
651
F [π ∗· · · ∗π ∗p] = F [π ∗· · · ∗π] F [p] where F is Fourier transform.
652
Experiment.
Since blur ﬁlter is low-pass ﬁlter, probabilistic spatial smoothing is also low-pass
653
ﬁlter. In Section 2.2, at the end of the stage 1, we show that MC dropout adds high-frequency noise to
654
feature maps, and spatial smoothing effectively removes it. As shown in Fig. D.2, we observe the
655
same phenomena at other stages.
656
In addition, Fig. 6c shows that CNNs are vulnerable to high-frequency random noise. Interestingly, it
657
also shows that CNNs are robust against noise with frequencies from 0.6π to 0.8π, corresponding to
658
approximately 3 pixel periods. Since the receptive ﬁelds of convolutions are 3×3, the noise with a
659
period smaller than the size is averaged out by convolutions. For the same reason, convolutions are
660
particularly vulnerable against the noise with a frequency of 0.3π, corresponding to a period of 6
661
pixel.
662
D.3
RANDOMNESS SHARPENS LOSS LANDSCAPE, AND ENSEMBLE SMOOTHENS IT
663
Ws show that the randomness of BNNs hinder and destabilize NN training because it causes the loss
664
landscape and its gradient to ﬂuctuate from moment to moment. In other words, the randomness,
665
such as dropout, sharpens the loss landscape.
666
To show the claim theoretically, we use Foret et al. (2020)’s deﬁnition of sharpness with respect to
training dataset D:
sharpnessρ = max
δw≤ρ LD(w + δw) −LD(w)
(18)
where LD is NLL loss, w is NN weight, δw is small weight perturbation, and ρ is neighborhood
667
radius. Therefore, as dropout rate—and the magnitude of δw—increases, the sharpness increases.
668
We next calculate the sharpness more rigorously. Let pi ∈(0, 1] be a conﬁdence of one NN prediction,
and ¯
p(N) be a conﬁdence of N ensemble, i.e., ¯
p(N) =
1
N
PN
i=1 pi. Then, the variance of the NLL
loss is:
V [L] = V
"
1
|D|
X
D
−log ¯
p(N)
#
(19)
=
1
|D|V
h
−log ¯
p(N)i
(20)
≃
1
|D|V

−log µ +

1 −¯
p(N)
µ

(21)
=
1
|D|V

−¯
p(N)
µ

(22)
= 1
N
V [pi]
µ2|D|
(23)
= 1
N
σ2
pred
µ2|D|
(24)
where µ = ¯
p(∞) and σ2
pred is predictive variance of conﬁdence. We use the formula V
h
1
N
PN
i=1 ξ
i
=
1
N V [ξ] for arbitrary random variable ξ, and we take the ﬁrst-order Taylor expansion with an assump-
25
Under review as a conference paper at ICLR 2022
(a) Deterministic
(b) Deterministic + Smooth
(c) MC dropout
(d) MC dropout + Smooth
0.0
0.5
1.0
Frequency
2.0
4.0
6.0
Log amplitude
0.0
0.5
1.0
Frequency
1.0
2.0
3.0
4.0
Log amplitude
0.0
0.5
1.0
Frequency
0.5
1.0
1.5
2.0
Log amplitude
0.0
0.5
1.0
Frequency
1.0
2.0
Log amplitude
0.0
0.5
1.0
Frequency
0.0
1.0
2.0
Log amplitude
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
(e) Diagonal components
Figure D.2: Spatial smoothing ﬁlters high-frequency signals including MC dropout noise. We
present average feature maps of ResNet-50 on ImageNet in frequency space by using Fourier
transform. Each column corresponds to feature maps at stage 1 to 4.
26
Under review as a conference paper at ICLR 2022
0
10
20
Eigenvlaue
Log density
0
200
400
600
Max Eigenvlaue
Density
0
10
20
Eigenvlaue
Density
0 %
10 %
20 %
30 %
Figure D.3: Randomness due to MC dropout sharpens the loss function. We provide Hessian
eigenvalue (left) and Hessian max eigenvalue spectra (right) of ResNet-18 on CIFAR-100.
tion ¯
p(N) ≃µ in Eq. (21). Therefore, the approximated sharpness is:
sharpness2
ρ ≃1
N
σ2
pred
µ2|D|
(25)
In conclusion, the variance of NLL, (the square of) the sharpness, is proportional to the variance of
669
predictions σ2
pred and inversely proportional to the ensemble size N. As the ensemble size increases
670
in the training phase, the loss landscape becomes smoother. Flat loss landscape results in better
671
predictive performance and generalization (Foret et al., 2020).
672
Here, we only consider model uncertainty for the sake of simplicity. Extending the formulations to
data uncertainty is straightforward. The predictive distribution of data-complemented BNN inference
(Park et al., 2021) is:
p(y|S, D) =
Z
p(y|x, w)p(x|S)p(w|D)dxdw
(26)
=
Z
p(y|z)p(z|S, D)dz
(27)
where S is proximate data distribution, z = (x, w), and p(z|S, D) = p(x|S) p(w|D). This equation
673
clearly shows that w and x are symmetric. Therefore, we obtain the formulas including both model
674
and data uncertainty by replacing w with joint random variable of x and w, i.e. w →z = (w, x).
675
Experiment.
Above, we claim two statements. First, the higher the dropout rate, the sharper the
676
loss landscape. Second, the variance of the loss is inversely proportional to the ensemble size.
677
To demonstrate the former claim quantitatively, we compare the Hessian eigenvalue spectra and the
678
Hessian max eigenvalue spectra of MC dropout with various dropout rates. In these experiments,
679
we use ensemble size of one for MC dropout. For detailed explanation of Hessian max eigenvalue
680
spectrum, see Appendix C.1.
681
Fig. D.3 represents the spectra, which reveals that as the randomness of the model increases, the
682
number of Hessian eigenvalue outliers increases. Since outliers are detrimental to the optimization
683
process (Ghorbani et al., 2019), dropout disturb NN optimization.
684
To show the latter claim, we evaluate the variance of NLL loss for ensemble size Ntrain as shown in
685
Fig. D.4a. As we would expect, the variance of the NLL loss—the sharpness of the loss landscape—is
686
inversely proportional to the ensemble size for large Ntrain.
687
D.4
TRAINING PHASE ENSEMBLE LEADS TO BETTER PERFORMANCE
688
Appendix D.3 raises an immediate question: Is there a performance difference between ‘training
689
with prediction ensemble’ and ‘training with a low MC dropout rate, instead of no ensemble’? Note
690
27
Under review as a conference paper at ICLR 2022
100
101
Ntrain
5.0e-08
1.0e-07
NLLtrain variance
(a) V [L] for ensemble size on training dataset
100
101
Ntest
0.90
1.00
NLLtest
Ntrain = 1
Ntrain = 2
Ntrain = 3
(b) NLL for ensemble size on test dataset
Figure D.4: Training phase ensemble helps NN learn strong representation. Left: The variance
of NLL (V [L]) on training dataset is inversely proportional to the ensemble size for large Ntrain. See
Eq. (24). Right: Training phase ensemble improves the predictive performance on test dataset.
that both methods reduce the sharpness of the loss landscape. This section answers the question
691
by providing theoretical and experimental explanations that the ensemble in the training phase can
692
improve predictive performance.
693
According to Gal & Ghahramani (2016), the total predictive variance (in regression tasks) is:
σ2
pred = σ2
model + σ2
sample
(28)
where σ2
model is model precision and σ2
sample is NN prediction variance. Therefore, the model precision
is the lower bound of the predictive variance, i.e.:
σ2
pred ≥σ2
model
(29)
The model precision depends only on the model architecture. For example, in the case of MC dropout,
σ2
model is proportional to the dropout rate (Gal & Ghahramani, 2016) as follows:
σ2
model ∝dropout rate
(30)
These suggest that model precision dominate predictive variance if the MC dropout rate is large
694
enough, i.e., even if the number of ensembles is increased in the training phase, the predictive variance
695
is almost the same. In contrast, decreasing the MC dropout rate reduces prediction diversity, and it
696
obviously leads to performance degradation. Therefore, in the training phase, it is better to ensemble
697
predictions than to lower the MC dropout rate. We believe that the training phase ensemble is strongly
698
correlated with Batch Augmentation (Hoffer et al., 2020). We leave concrete analysis for future work.
699
Experiment.
The experiments below support the theoretical analysis. We train MC dropout by
700
using training-phase ensemble method with various ensemble sizes Ntrain.
701
As we would expect, Fig. D.4b shows that training phase ensemble signiﬁcantly improves the
702
predictive performance. In this experiment, we use MC dropout rate of 30%. As shown in Fig. A.1, it
703
provides the best predictive performance. We use ensemble size Ntest = 50 in test phase.
704
We also measure the predictive variances of NLL. The predictive variances of the model with
705
Ntrain = 1 and with Ntrain = 3 are V [L] = 0.0169 and V [L] = 0.0179, respectively. Since the
706
predictive variances of the two models are almost the same, we infer that there exists a lower bound.
707
E
EXTENDED INFORMATIONS OF EXPERIMENTS
708
This section provides additional information on the experiments in Section 3.
709
E.1
IMAGE CLASSIFICATION
710
We present numerical comparisons in the image classiﬁcation experiment and discuss the results in
711
detail.
712
28
Under review as a conference paper at ICLR 2022
Table E.1: Spatial smoothing improves both accuracy and uncertainty at the same time. Predic-
tive performance of models with spatial smoothing in image classiﬁcation on CIFAR-10, CIFAR-100,
and ImageNet.
MODEL &
DATASET
MC DROPOUT
SMOOTH
NLL
ACC
(%)
ECE
(%)
VGG-19 &
CIFAR-10
·
·
0.401 (-0.000)
93.1 (+0.0)
3.80 (-0.00)
·
✓
0.376 (-0.002)
93.2 (+0.1)
5.49 (+1.69)
✓
·
0.238 (-0.000)
92.6 (+0.0)
3.55 (-0.00)
✓
✓
0.197 (-0.041)
93.3 (+0.7)
0.68 (-2.86)
ResNet-18 &
CIFAR-10
·
·
0.182 (-0.000)
95.2 (+0.0)
2.75 (-0.00)
·
✓
0.173 (-0.009)
95.4 (+0.2)
2.31 (-0.44)
✓
·
0.157 (-0.000)
95.2 (+0.0)
1.14 (-0.00)
✓
✓
0.144 (-0.014)
95.5 (+0.2)
1.04 (-0.10)
VGG-16 &
CIFAR-100
·
·
2.047 (-0.000)
71.6 (+0.0)
19.2 (-0.0)
·
✓
1.878 (-0.169)
72.2 (+0.6)
20.5 (+1.3)
✓
·
1.133 (-0.000)
68.8 (+0.0)
3.66 (-0.00)
✓
✓
1.034 (-0.099)
71.4 (+2.6)
1.06 (-2.60)
VGG-19 &
CIFAR-100
·
·
2.016 (-0.000)
67.6 (+0.0)
21.2 (-0.0)
·
✓
1.851 (-0.165)
71.7 (+4.0)
20.2 (-1.0)
✓
·
1.215 (-0.000)
67.3 (+0.0)
6.37 (-0.00)
✓
✓
1.071 (-0.144)
70.4 (+3.0)
2.15 (-4.22)
ResNet-18 &
CIFAR-100
·
·
0.886 (-0.000)
77.9 (+0.0)
4.97 (-0.00)
·
✓
0.863 (-0.023)
78.9 (+1.0)
4.40 (-0.57)
✓
·
0.848 (-0.000)
77.3 (+0.0)
3.01 (-0.00)
✓
✓
0.801 (-0.047)
78.9 (+1.6)
2.56 (-0.45)
ResNet-50 &
CIFAR-100
·
·
0.835 (-0.000)
79.9 (+0.0)
8.88 (-0.00)
·
✓
0.834 (-0.002)
80.7 (+0.8)
9.29 (+0.42)
✓
·
0.822 (-0.000)
79.1 (+0.0)
6.63 (-0.00)
✓
✓
0.800 (-0.022)
80.1 (+1.0)
7.25 (+0.62)
ResNeXt-50 &
CIFAR-100
·
·
0.804 (-0.000)
80.6 (+0.0)
8.23 (-0.00)
·
✓
0.825 (+0.022)
80.8 (+0.3)
9.41 (+1.18)
✓
·
0.762 (-0.000)
80.5 (+0.0)
5.67 (-0.00)
✓
✓
0.759 (-0.002)
80.7 (+0.2)
6.62 (+0.94)
ResNet-18 &
ImageNet
·
·
1.210 (-0.000)
70.3 (+0.0)
1.62 (-0.00)
·
✓
1.183 (-0.027)
70.6 (+0.3)
1.22 (-0.40)
✓
·
1.215 (-0.000)
70.0 (+0.0)
1.39 (-0.00)
✓
✓
1.190 (-0.032)
70.6 (+0.6)
2.25 (+0.86)
ResNet-50 &
ImageNet
·
·
0.949 (-0.000)
76.0 (+0.0)
2.97 (-0.00)
·
✓
0.916 (-0.033)
76.9 (+0.9)
3.46 (+0.49)
✓
·
0.945 (-0.000)
76.0 (+0.0)
1.89 (-0.00)
✓
✓
0.905 (-0.040)
77.0 (+1.0)
2.49 (+0.60)
ResNeXt-50 &
ImageNet
·
·
0.919 (-0.000)
77.7 (+0.0)
3.63 (-0.00)
·
✓
0.907 (-0.012)
78.0 (+0.3)
4.60 (+0.97)
✓
·
0.895 (-0.000)
77.7 (+0.0)
2.53 (-0.00)
✓
✓
0.887 (-0.008)
78.1 (+0.4)
3.28 (+0.75)
29
Under review as a conference paper at ICLR 2022
100
101
Ensemble Size
0.20
0.30
0.40
NLL
100
101
Ensemble Size
90.0
92.0
94.0
Accuracy (%)
100
101
Ensemble Size
2.0
4.0
6.0
ECE (%)
(a) VGG-19 on CIFAR-10
100
101
Ensemble Size
0.15
0.20
NLL
100
101
Ensemble Size
95.0
96.0
Accuracy (%)
100
101
Ensemble Size
1.0
2.0
3.0
ECE (%)
(b) ResNet-18 on CIFAR-10
100
101
Ensemble Size
0.80
0.90
1.00
NLL
100
101
Ensemble Size
76.0
78.0
80.0
82.0
Accuracy (%)
100
101
Ensemble Size
4.0
6.0
8.0
ECE (%)
(c) ResNet-18 on CIFAR-100
100
101
Ensemble Size
0.80
1.00
NLL
100
101
Ensemble Size
78.0
80.0
82.0
Accuracy (%)
100
101
Ensemble Size
2.5
5.0
7.5
10.0
ECE (%)
(d) ResNet-50 on CIFAR-100
100
101
Ensemble Size
0.80
0.85
0.90
0.95
NLL
100
101
Ensemble Size
76.0
78.0
Accuracy (%)
100
101
Ensemble Size
1.0
2.0
3.0
ECE (%)
(e) ResNet-50 on ImageNet
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deep ensemble
Deep ensemble + Smooth
MC dropout
MC dropout + Smooth
Figure E.1: Spatial smoothing improves both accuracy and uncertainty across a whole range of
ensemble sizes.
30
Under review as a conference paper at ICLR 2022
Test
1
2
3
4
5
Intensity
2.0
4.0
6.0
NLL
Test
1
2
3
4
5
Intensity
25
50
75
Accuracy (%)
Test
1
2
3
4
5
Intensity
 0
20
40
ECE (%)
0
1
2
3
4
5
Intensity
0
10
20
ECE (%)
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
100
101
Ensemble Size
60
65
70
mCNLL (%)
100
101
Ensemble Size
85.0
87.5
90.0
mCE (%)
100
101
Ensemble Size
40
60
mCECE (%)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure E.3: Spatial smoothing improves corruption robustness. We measure the predictive per-
formance of ResNet-18 on CIFAR-100-C. In the top row, we use an ensemble size of ﬁfty for MC
dropout with and without spatial smoothing.
Computational performance.
The throughput of MC dropout and “MC dropout + spatial smooth-
713
ing” is 755 and 675 image/sec, respectively, in training phase on ImageNet. As mentioned in lines
714
Section 3.1, NLL of “MC dropout + spatial smoothing” with ensemble size of 2 is comparable
715
to or even better than that of MC dropout with ensemble size of 50. Therefore, “MC dropout +
716
spatial smoothing” is 22× faster than MC dropout with similar predictive performance, in terms of
717
throughput.
718
40
60
80
100
Confidence (%)
40
60
80
100
Accuracy (%)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure E.2: Spatial smoothing cal-
ibrates predictions. We present re-
liability diagram of ResNet-18 on
CIFAR-100.
Predictive performance on test dataset.
Fig. E.2 repre-
719
sents the reliability diagram of ResNet-18 on CIFAR-100,
720
which shows that spatial smoothing improves the uncertainty
721
of both deterministic and Bayesian NNs. Numerical compar-
722
isons are provided below.
723
Table E.1 shows the predictive performance of various deter-
724
ministic and Bayesian NNs with and without spatial smooth-
725
ing on CIFAR-10, CIFAR-100, and ImageNet. This table
726
suggests the following: First, spatial smoothing improves
727
both accuracy and uncertainty in most cases. In particular, it
728
improves the predictive performance of all models with MC
729
dropouts. Second, spatial smoothing signiﬁcantly improves
730
the predictive performance of VGG compared with ResNet.
731
VGG has a chaotic loss landscape, which results in poor pre-
732
dictive performance (Li et al., 2018), and spatial smoothing
733
smoothens its loss landscape effectively. Third, as the depth
734
increases, the performance improvement decreases. Deeper
735
NNs provide more overconﬁdent results (Guo et al., 2017),
736
but the number of spatial smoothing layers calibrating uncer-
737
tainty is ﬁxed. Last, the performance improvement of ResNeXt, which includes an ensemble in its
738
internal structure, is relatively marginal.
739
Fig. E.1 shows predictive performance of MC dropout and deep ensemble for ensemble size. A
740
deep ensemble with an ensemble size of 1 is a deterministic NN. This ﬁgure shows that spatial
741
smoothing improves efﬁciency of ensemble size and the predictive performance at ensemble size of
742
31
Under review as a conference paper at ICLR 2022
Table E.2: Spatial smoothing improves adversarial robustness. We measure the accuracy (ACC)
and the Attack Success Rate (ASR) of ResNet-50 against adversarial attacks on ImageNet.
ATTACK
MC DROPOUT
SMOOTH
ACC
(%)
ASR
(%)
FGSM
·
·
28.3 (+0.0)
62.9 (-0.0)
·
✓
30.3 (+2.0)
60.5 (-2.4)
✓
·
30.3 (+0.0)
59.8 (-0.0)
✓
✓
32.6 (+2.3)
57.4 (-2.4)
PGD
·
·
7.5 (+0.0)
90.1 (-0.0)
·
✓
9.0 (+1.4)
88.2 (-1.9)
✓
·
12.2 (+0.0)
83.7 (-0.0)
✓
✓
13.7 (+1.5)
82.1 (-1.6)
50. In addition, spatial smoothing stabilizes NN training. It reduces the variance of the performance,
743
especially in VGG.
744
A peculiarity of the results on ImageNet is that spatial smoothing degrades ECE of ResNet-50. It
745
is because spatial smoothing signiﬁcantly improves the accuracy in this case, and there tends to be
746
a trade-off between accuracy and ECE, e.g. as shown in (Guo et al., 2017), Fig. A.1, and Fig. B.3.
747
Instead, spatial smoothing shows the improvement in NLL, another uncertainty metric.
748
Predictive performance on training datasets.
Note that spatial smoothing helps NN learn strong
749
representations. In other words, spatial smoothing does not regularize NNs. For example, NLL
750
ResNet-18 with MC dropout on CIFAR-100 training dataset is 2.20 × 10−2. The NLL of the ResNet
751
with spatial smoothing is 1.94 × 10−2. In conclusion, spatial smoothing reduces the training loss.
752
Corruption robustness.
We measure predictive performance on CIFAR-100-C (Hendrycks &
753
Dietterich, 2019) in order to evaluate the robustness of the models against 5 intensities and 15 types
754
of data corruption. The top row of Fig. E.3 shows the results as a box plot. The box plot shows the
755
median, interquartile range (IQR), minimum, and maximum of predictive performance for types.
756
They reveal that spatial smoothing improves predictive performance for corrupted data. In particular,
757
spatial smoothing undoubtedly helps in predicting reliable uncertainty.
758
To summarize the performance of corrupted data in a single value, Hendrycks & Dietterich (2019)
introduced a corruption error (CE) for quantitative comparison. CEf
c , which is CE for corruption type
c and model f, is as follows:
CEf
c =
 5
X
i=1
Ef
i,c
!   5
X
i=1
EAlexNet
i,c
!
(31)
where Ef
i,c is top-1 error of f for corruption type c and intensity i, and EAlexNet
i,c
is the error of AlexNet.
Mean CE or mCE summarizes CEf
c by averaging them over 15 corruption types such as Gaussian
noise, brightness, and show. Likewise, to evaluate robustness in terms of uncertainty, we introduce
corruption NLL (CNLL, ↓) and corruption ECE (CECE, ↓) as follows:
CNLLf
c =
 5
X
i=1
NLLf
i,c
!   5
X
i=1
NLLAlexNet
i,c
!
(32)
and
CECEf
c =
 5
X
i=1
ECEf
i,c
!   5
X
i=1
ECEAlexNet
i,c
!
(33)
32
Under review as a conference paper at ICLR 2022
Table E.3: Spatial smoothing improves the consistency, robustness against shift-perturbation.
We measure the consistency of ResNet-18 on CIFAR-10-P. Deterministic NN with N = 5 means
deep ensemble.
MC DROPOUT
SMOOTH
N
CONS
(%)
CEC
(×10−2)
·
·
1
97.9 (+0.0)
1.03 (-0.00)
·
✓
1
98.2 (+0.3)
1.16 (+0.13)
·
·
5
98.7 (+0.0)
1.22 (-0.00)
·
✓
5
98.9 (+0.2)
1.33 (+0.11)
✓
·
50
98.2 (+0.0)
1.29 (-0.00)
✓
✓
50
98.4 (+0.2)
1.34 (+0.05)
where NLLf
i,c and ECEf
i,c are NLL and ECE of f for c and i, respectively. mCNLL and mCECE
759
are averages over corruption types. Experimental results show that spatial smoothing improves the
760
robustness against data corruption. See Fig. E.3 for the results.
761
The bottom row of Fig. E.3 shows mCNLL, mCE, and mCECE for ensemble size. They indicates that
762
spatial smoothing improves not only the efﬁciency but corruption robustness across a whole range of
763
ensemble size.
764
Adversarial robustness.
We show that spatial smoothing also improves adversarial robustness.
765
First, we measure the robustness, in terms of accuracy and attack success rate (ASR), of ResNet-
766
50 on ImageNet against popular adversarial attacks, namely FGSM (Goodfellow et al., 2015) and
767
PGD (Madry et al., 2018). Table E.2 indicate that both MC dropout and spatial smoothing improve
768
robustness against adversarial attacks.
769
Next, we ﬁnd out how spatial smoothing improves adversarial robustness. To this end, similar
770
to Section 2.2, we measure the accuracy on the test datasets with frequency-based adversarial
771
perturbations. In this experiment, we use FGSM attack. This experimental result shows that spatial
772
smoothing is particularly robust against high frequency (≥0.3π) adversarial attacks. This is because
773
spatial smoothing is a low-pass ﬁlter, as we mentioned in Section 2.2. Since the ResNet is vulnerable
774
−6 −4 −2
0
2
4
6
8
0
25
50
75
100
Relative confidence (%)
Translation (px)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Deterministic
Deterministic + Smooth
MC dropout
MC dropout + Smooth
Figure E.4: Spatial smoothing improves the
conﬁdence when the predictions are incorrect.
We deﬁne relative conﬁdence (See Eq. (36)),
and measure the metric of ResNet-18 on CIFAR-
10-P.
against high frequency adversarial attack, an ef-
775
fective defense of spatial smoothing against high
776
frequency attacks signiﬁcantly improves the robust-
777
ness.
778
Consistency.
To evaluate the translation invari-
ance of models, we use consistency (Hendrycks &
Dietterich, 2019; Zhang, 2019a), a metric represent-
ing translation consistency for shift-translated data
sequences S = {x1, · · · , xM+1}, as follows:
Consistency = 1
M
M
X
i=1
1(g(xi) = g(xi+1))
(34)
where g(x) = arg max p(y|x, D). Table E.3 pro-
779
vides consistency of ResNet-18 on CIFAR-10-P
780
(Hendrycks & Dietterich, 2019). The results shows
781
that MC dropout and deep ensemble improve con-
782
sistency, and spatial smoothing improves consis-
783
tency of both deterministic and Bayesian NNs.
784
Prior works (Zhang, 2019a; Azulay & Weiss, 2019)
investigated the ﬂuctuation of predictive conﬁdence
33
Under review as a conference paper at ICLR 2022
Table E.4: Spatial smoothing and temporal smoothing are complementary. We provide predictive
performance of MC dropout in semantic segmentation on CamVid for each method. SPAT and TEMP
each stand for spatial smoothing and temporal smoothing. CONS stands for consistency.
MC DROPOUT
SPAT
TEMP
N
NLL
ACC
(%)
ECE
(%)
CONS
(%)
·
·
·
1
0.354 (+0.000)
92.3 (+0.0)
4.95 (+0.00)
95.1 (+0.0)
·
✓
·
1
0.318 (+0.036)
92.4 (+0.1)
4.54 (+0.41)
95.5 (+0.4)
·
·
✓
1
0.290 (+0.064)
92.5 (+0.2)
3.18 (+1.77)
96.3 (+1.2)
·
✓
✓
1
0.278 (+0.076)
92.5 (+0.2)
3.03 (+1.92)
96.6 (+1.5)
✓
·
·
50
0.298 (+0.000)
92.5 (+0.0)
4.20 (+0.00)
95.4 (+0.0)
✓
✓
·
50
0.284 (+0.014)
92.6 (+0.1)
3.96 (+0.24)
95.6 (+0.2)
✓
·
✓
1
0.273 (+0.025)
92.6 (+0.1)
3.23 (+0.97)
96.4 (+1.0)
✓
✓
✓
1
0.260 (+0.038)
92.6 (+0.1)
2.71 (+1.49)
96.5 (+1.1)
on shift-translated data sequence. However, surprisingly, we ﬁnd that conﬁdence ﬂuctuation has
little to do with consistency. To demonstrate this claim, we introduce cross-entropy consistency
(CEC, ↓), a metric that represents the ﬂuctuation of conﬁdence on a shift-translated data sequence
S = {x1, · · · , xM+1}, as follows:
CEC = −1
M
M
X
i=1
f(xi) · log(f(xi+1))
(35)
where f(x) = p(y|x, D). In Table E.3, high consistency does not mean low CEC; conversely, high
785
consistency tends to be high CEC. Canonical NNs predict overconﬁdent probabilities, and their
786
conﬁdence sometimes changes drastically from near-zero to near-one. Correspondingly, it results in
787
low consistency but low CEC. On the contrary, well-calibrated NNs such as MC dropout provide
788
conﬁdence that oscillates between zero and one, which results in high CEC.
789
To represent the NN reliability properly, we propose relative conﬁdence (↑) as follows:
Relative conﬁdence = p(ytrue|x, D)

max p(y|x, D)
(36)
where max p(y|x, D) is conﬁdence of predictive result and p(ytrue|x, D) is probability of the result
790
for true label. It is 1 when NN classiﬁes the image correctly, and less than 1 when NN classiﬁes it
791
incorrectly. Therefore, relative conﬁdence is a metric that indicates the overconﬁdence of a prediction
792
when NN’s prediction is incorrect.
793
Figure E.4 shows a qualitative example of consistency on CIFAR-10-P by using relative conﬁdence.
794
This ﬁgure suggests that spatial smoothing improves consistency of both deterministic and Bayesian
795
NN.
796
E.2
SEMANTIC SEGMENTATION
797
Table E.4 shows the performance of U-Net on the CamVid dataset. This table indicates that spatial
798
smoothing improves accuracy, uncertainty, and consistency of deterministic and Bayesian NNs.
799
This is consistent with the results in image classiﬁcation. In addition, temporal smoothing leads
800
to signiﬁcant improvement in efﬁciency of ensemble size, accuracy, uncertainty, and consistency
801
by exploiting temporal information. Moreover, temporal smoothing requires only one ensemble to
802
achieve high predictive performance, since it cooperates with the temporally previous predictions. We
803
obtain the best predictive and computational performance by using both temporal smoothing and
804
spatial smoothing.
805
34
Under review as a conference paper at ICLR 2022
F
COMPARISON WITH ANTI-ALIASED CNN
806
As we mentioned in Section 4, local means (Blur), also known as anti-aliased CNN (Zhang, 2019a),
807
improve accuracy. Nevertheless, our work (Prob + Blur) has novelties in three respects: different
808
motivation, improved uncertainty estimation, and analysis of how spatial smoothing works.
809
Different motivation.
The motivation of local means was to mitigate the aliasing effect of subsam-
810
pling and to improve shift invariance. In contrast, our spatial smoothing is introduced to aggregate
811
and ensemble nearby feature map points.
812
Improved uncertainty estimation.
We demonstrate that spatial smoothing improves not only
813
accuracy, but also uncertainty estimation and robustness against natural corruptions and adversarial
814
attacks all at the same time. Moreover, we show that spatial smoothing signiﬁcantly enhances the
815
performance of MC dropout. Since there typically tends to be a trade-off between accuracy and
816
“uncertainty + robustness”—e.g. as shown in (Guo et al., 2017; Zhang et al., 2019; Geirhos et al.,
817
2019; Zhang, 2019b), Fig. A.1, and Fig. B.3—in NN modeling, we believe our simple yet effective
818
method makes major inroads into the uncertainty quantiﬁcation and generalization.
819
Analysis of how spatial smoothing improves performance.
We ﬁnd that the predictive perfor-
820
mance improvement is not due to the anti-aliasing effect of local means.
821
• Prob + Blur—our probabilistic spatial smoothing—improves the performance of pre-
822
activation CNNs, but Blur alone—local mean or anti-aliased CNN—does not. In fact,
823
contrary to (Zhang, 2019a), local mean degrades the predictive performance since it results
824
in loss of information. It suggests that Prob plays an key role in prediction. For more details,
825
see Appendix F.1.
826
• Although the local ﬁltering can result in loss of information, Zhang (2019a) experimentally
827
observed an increase in both shift-invariance (as expected) and accuracy (which was be-
828
yond expectation). However, “there exist a fundamental trade-off between ‘shift-invariance
829
plus anti-aliasing’ and performance” (Zhang, 2019b). Moreover, it is difﬁcult to relate
830
anti-aliasing to improved uncertainty and robustness. Zhang (2019a) did not provide an
831
explanation for these phenomena. As discussed in Appendix E.1, spatial smoothing helps
832
NNs learn strong representations, not regularizes NNs.
833
• Spatial smoothing is, surprisingly, robust against blur corruptions.
834
We analyze how spatial smoothing improves predictive performance, by using loss landscape vi-
835
sualization, Hessian eigenvalue spectra, and Fourier analysis. These analyzes draw the following
836
conclusions:
837
• Loss landscape visualization: Spatial smoothing stabilizes loss landscape ﬂuc-
838
tuations, caused by e.g. MC dropout. This results in stabilizing NN training
839
and improving performance as well as generalization. See Figs. 8 and C.2.
840
See
also
code/resources/losslandscapes/resnet_mcdo_18.gif
and
841
code/resources/losslandscapes/resnet_mcdo_smoothing_18.gif
in
the
842
supplementary material.
843
• Hessian eigenvalue spectra: Spatial smoothing suppresses outliers of Hessian eigenvalues,
844
which disrupt NN training. See Figs. 7 and C.3.
845
• Fourier analysis: Spatial smoothing effectively removes high frequency signals, including
846
noise due to MC dropout. We also show that CNNs are vulnerable to high frequency noise
847
and high frequency adversarial attacks. See Figs. 6 and D.2.
848
We also provide theoretical analysis of how spatial smoothing works. We prove that dropout sharpens
849
the loss landscape, and ensemble smoothens it. Since the spatial smoothing is a spatial ensemble, it
850
signiﬁcantly enhances the performance of MC dropout. See Appendix D.3 for more details. Further-
851
more, we also show that training-phase ensemble signiﬁcantly improves the predictive performance
852
because it smoothens the loss landscape without loss of prediction diversity. Therefore, the spatial
853
smoothing, which ensembles feature map points at training time, improves the performance effectively.
854
See Appendix D.4.
855
35
Under review as a conference paper at ICLR 2022
F.1
Prob PLAYS AN IMPORTANT ROLE IN SPATIAL SMOOTHING
856
As discussed in Section 2.1, we take the perspective that each point in feature map is a prediction for
857
binary classiﬁcation by deriving the Bernoulli distributions from the feature map by using Prob. It is
858
in contrast to previous works known as sampling-free BNNs (Hernández-Lobato & Adams, 2015;
859
Wang et al., 2016; Wu et al., 2019) attempting to approximate the distribution of feature map with
860
one Gaussian distribution. We do not use any assumptions on the distribution of feature map, and
861
exactly represent the Bernoulli distributions and their averages. However, sampling-free BNNs are
862
error-prone because there is no guarantee that feature maps will follow a Gaussian distribution.
863
This Prob plays an important role in spatial smoothing. CNNs such as VGG, ResNet, and ResNeXt
generally use post-activation arrangement. In other words, their stages end with BatchNorm and
ReLU. Therefore, spatial smoothing layers Smooth(z) = Blur ◦Prob(z) in CNNs cooperates with
BatchNorm and ReLU as follows:
Prob(z) = ReLU ◦tanhτ ◦ReLU ◦BatchNorm (z)
(37)
= ReLU ◦tanhτ ◦BatchNorm (z)
(38)
since ReLU and tanhτ are commutative, and ReLU ◦ReLU is ReLU. This Prob is trainable and is a
864
general form of Eq. (7). If we only use Blur as spatial smoothing, the activations BatchNorm–ReLU
865
play the role of Prob.
866
100
101
Ensemble Size
1.00
1.20
1.40
1.60
1.80
NLL
100
101
Ensemble Size
1.00
1.20
1.40
1.60
1.80
NLL
Deterministic
Deterministic + Blur
Deterministic + Prob + Blur
MC dropout
MC dropout + Blur
MC dropout + Prob + Blur
Figure F.1: Blur alone harms the predic-
tive performance, although Prob + Blur im-
proves it. We provide NLL of pre-activation
VGG-16 on CIFAR-100.
In order to analyze the roles of Prob and Blur
867
more precisely, we measure the predictive perfor-
868
mance of the model that does not use the post-
869
activation. Figure F.1 shows NLL of pre-activation
870
VGG-16 on CIFAR-100. The result shows that
871
Blur with Prob improves the performance, but
872
Blur alone does not. In fact, contrary to (Zhang,
873
2019a), blur degrades the predictive performance
874
since it results in loss of information. We also
875
measure the performance of VGG-19, ResNet-18,
876
ResNet-50, and BlurPool (Zhang, 2019a) with pre-
877
activation, and observe the same phenomenon. In
878
addition, BatchNorm–ReLU in front of GAP signif-
879
icantly improves the performance of pre-activation
880
ResNet.
881
As mentioned in Appendix C.2, pre-activation is
882
a special case of spatial smoothing. Therefore, the
883
performance improvement of pre-activation by spa-
884
tial smoothing is marginal compared to that of post-
885
activation.
886
36
