## STOCHASTIC DEEP NETWORKS WITH LINEAR COM### PETING UNITS FOR MODEL-AGNOSTIC META## LEARNING

**Anonymous authors**
Paper under double-blind review

ABSTRACT

This work addresses meta-learning (ML) by considering deep networks with
stochastic local winner-takes-all (LWTA) activations. This type of network units
result in sparse representations from each model layer, as the units are organized
into blocks where only one unit generates a non-zero output. The main operating principle of the introduced units lies on stochastic principles, as the network
performs posterior sampling over competing units to select the winner. Therefore,
the proposed networks are explicitly designed to extract input data representations of sparse stochastic nature, as opposed to the currently standard deterministic representation paradigm. We posit that these modeling principles, inspired
from Bayesian statistics, yield representations of stronger generalization capacity;
this is of immense importance in the case of ML, which is the focus of this work.
As we experimentally show, our approach produces state-of-the-art predictive accuracy on standard few-shot image classification benchmarks. This improvement
comes with an immensely reduced network sized required for achieving this accuracy; this amounts to a parameter reduction by one order of magnitude on average
compared to the current state-of-the-art.

1 INTRODUCTION

When we train machine learning models on problems with limited amounts of training data, we
cannot usually get good predictive performance (Lai, 2019; Sculley et al., 2015). This comes in
contrast to the human ability to quickly derive information from a range of different tasks, and then
adapt to a new task with limited available new examples (K¨uhl et al., 2020; Castro et al., 2008).
In essence, this capability of the mind of learning how to learn (Black et al., 2006) has inspired
researchers to investigate the concept of Meta-Learning (ML) (Lake et al., 2017; Vilalta & Drissi,
2002; Wang et al., 2016; Zoph & Le, 2017; Flennerhag et al., 2020; Lake et al., 2015; HutsebautBuysse et al., 2019).

There is a large variety of deep learning methods for ML (Finn et al., 2019; Baik et al., 2020;
Andrychowicz et al., 2016). Specifically, Finn et al. (2017) presented the Model-Agnostic Meta_Learning (MAML) algorithm that enables tuning the parameters of a trained network to quickly_
learn a new task with only a few gradient updates. To get away with the entailed second-order computations, which are expensive, several researchers proposed appropriate first-order approximations
for MAML; such works are the First-Order MAML (FOMAML) of Finn et al. (2017) and the Rep_tile algorithm of Nichol et al. (2018). Recently, several researchers have also considered Bayesian_
inference-driven methods for deep learning ML, extending upon older works built for conventional
machine learning approaches (Yoon et al., 2018; Finn et al., 2018; Ravi & Beatson, 2019; Patacchiola et al., 2020; Zou & Lu, 2020; Grant et al., 2018; Chen et al., 2020).

This work proposes a different regard toward improving generalization capacity for deep networks
in the context of ML. Specifically, our proposed approach relies on the following main concepts:

-  The concept of sparse learned representations. For the first time in the literature of
deep network-driven ML, we employ a mechanism that inherently learns to extract sparse
data representations. This consists in replacing standard unit nonlinearities (e.g., ReLU)


-----

with a unit competition mechanism. Specifically, (linear) units are organized into blocks.
Presented with some input, the units within a block engage in a competition process with
only one winner. The outputs of all units except for the winner are zeroed out; the output
of the winner retains its computed value (local winner takes-all, LWTA, architecture).

-  The concept of stochastic representations. We establish a stochastic formulation for the
previously described competition process. Specifically, we postulate that, within a block of
competing units, winner is selected via sampling from an appropriate Categorical posterior.
The corresponding winning probability of each unit is proportional to its linear computation
(thus depending on the layer input). Via this competition process, we yield stochastic
representations from network layers, that is representations that may change each time we
present to the network layer exactly the same input.

Based on the results from existing approaches, we posit that the proposed treatment of the ML problem, which combines learned representation sparsity and stochasticity, will be extremely beneficial
to the deep learning community. We dub our approach Stochastic LWTA for ML (StochLWTA-ML).

We perform a variational Bayes treatment of the proposed model. We opt for a full Bayesian treatment, by also handling network weights as latent variables. That is, we elect to impose an appropriate prior over the network weights and fit approximate (variational) posteriors. We evaluate our
approach on a number of standard benchmarks in the field, namely Omniglot (Lake et al., 2017),
Mini-Imagenet (Vinyals et al., 2016) and CIFAR-100 (Krizhevsky, 2009). We show that our approach offers a variety of advantages over the current state-of-the-art methods, namely: (i) incurring
reduced predictive error rate compared to the currently state-of-the-art methods in the field; (ii)
obtaining this performance with networks that comprise one order of magnitude less trainable parameters, and therefore give rise to better computational efficiency and imposed memory footprint.

The remainder of this paper is organized as follows: In Section 2, we briefly review related work.
Section 3 introduces our approach and provides the related training and prediction algorithms. In
Section 4, we perform a thorough experimental evaluation of StochLWTA-ML, and compare our
findings to the current state-of-the-art. In the final Section 5, we end up with the conclusions of our
work, and suggest lines of further research.

2 RELATED WORK

2.1 LWTA LAYERS IN DEEP LEARNING

LWTA layers are not new in the field of deep learning; see, e.g., Srivastava et al. (2013). Although
not much work has been pursued along these lines, the recent works of Panousis et al. (2019; 2021)
and Voskou et al. (2021) have spurred some fresh interest in the field. These works have presented
alternative implementations of the basic concepts of LWTA units in the context of diverse deep
network architectures. Specifically, Panousis et al. (2019) propose a stochastic LWTA formulation
which is founded upon the Indian Buffet process (IBP) prior, borrowed from nonparametric statistics; they use this architecture to effect data-driven network compression. In their follow-up paper
(Panousis et al., 2021), they exploit the same technique to train adversarially-robust deep networks.
On the other hand, Voskou et al. (2021) consider a different incarnation of stochastic LWTA architectures, which relies on sampling the winner from a Categorical posterior, driven from the layer
input. This layer architecture is used to replace dense ReLU layers in Transformer networks; it is
then shown to yield important benefits in a Sign-Language Translation benchmark.

This paper is different from the previous works in various ways: (i) stochasticity does not stem
from utilization of the IBP; we rather adopt an approach similar to Voskou et al. (2021); (ii) we do
not use the proposed architecture as a replacement for a specific type of layer in a greater network
architecture (Transformer) that remains largely unchanged; instead, we build completely new networks using these layers; (iii) we perform a full Bayesian treatment, by treating network weights
as random variables; we do not employ this construction as a means of compressing the weights at
prediction time, contrary to Panousis et al. (2019), Voskou et al. (2021); instead, we perform weight
sampling at prediction time as a means of improving accuracy; and (iv) for the first time, we examine how these principles perform in the context of deep network-driven ML. Note that, apart from
LWTA architectures, other data-driven sparsity models have also been proposed recently, e.g. Lee


-----

et al. (2018), Kessler et al. (2021). However, none of these have been developed or evaluated in the
context of an ML setting.

2.2 MODEL-AGNOSTIC META-LEARNING

ML (Schmidhuber, 1987), also referred to as learning to learn (Thrun & Pratt, 1998), has the goal
of obtaining machine learning models that can learn new tasks with limited availability of data, and
via only a few gradient steps, by capitalizing upon information stemming from previously learned
tasks. In this context, Finn et al. (2017) suggested a model-agnostic algorithm for ML, that can
be applied to any model trained via gradient descent. The introduced MAML algorithm initializes
model parameters in a way that can be quickly adapted to several types of new tasks.

Let us consider a model with parameters θ and a parametric form fθ. When the model is adapting
to an unseen task Ti, sampled from the distribution over tasks P (T ), MAML initially runs few steps
of inner-loop gradient descent that yields the task-specific parameter set

**_θi[′]_** [=][ θ][ −] _[α][∇][θ][L][T]i_ [(][f][θ][)] (1)

where α is the step size hyperparameter and LTi denotes the loss on the task Ti.

Subsequently, training proceeds to optimize the function fθi[′] [with respect to the model parameters]
**_θ. Assuming that the batch of tasks has size M_**, we can define the targeted meta-objective as:


_LTi_ (fθi[′] [) =] _LTi_ (fθ−α∇θ _LTi_ (fθ )). (2)
_i=1_ _i=1_

X X


_Lmeta(θ) =_


Optimization of this meta-objective over θ yields the outer-loop update:


**_θ ←_** **_θ −_** _β∇θ_ _LTi_ (fθi[′] [)] (3)

_i=1_

X

where β stands for the outer-loop learning rate.


Finally, as MAML involves expensive computations stemming from the second-order updates of
Eqs. (2) and (3), Finn et al. (2017) developed a first-order approximation that reduces the outer-loop
update (3) to:


_∇θi[′]_ _[L][T][i]_ [(][f][θ]i[′] [)] (4)
_i=1_

X


**_θ ←_** **_θ −_** _β_


In other words, FOMAML computes the gradients with respect to the updated parameter values θi[′][,]
but omits the gradients of θi[′] [with respect to][ θ][.]

In a different fashion, the Reptile algorithm of Nichol et al. (2018) disposes the costly secondorder updates of MAML by: (i) applying some inner-loop gradient descent steps using (1); and (ii)
suggesting a new outer-loop that simply subtracts the parameters, θ, from the updates θi[′] [(instead of]
computing derivatives):
**_θ_** **_θ + β(θi[′]_** (5)
_←_ _[−]_ **_[θ][)][.]_**

3 PROPOSED APPROACH

3.1 ARCHITECTURE

Let us denote as x ∈ R[I] an input vector presented to a dense ReLU layer of a conventional deep
neural network, with corresponding weights matrix W ∈ R[I][×][O]. The output of the layer is the
vector y ∈ R[O] and is fed to the subsequent layer. In our approach, a ReLU unit is replaced by J
competing linear units, organized in one block; in the following, we denote with R the number of
blocks in a layer. The input x is now presented to each block through weights that are organized
into a three-dimensional matrix W ∈ R[I][×][R][×][J] . Then, the j-th competing unit within r-th block
computes the sum _i=1[(][w][i,r,j][)][ ·][ x][i][. Competition means that, of the][ J][ units in the block, one]_
unit (the ”winner”) will present its linear computation to the next layer; the rest will present zero

[P][I]


-----

values. Traditionally in the literature, the winner unit is selected to be the unit with greatest linear
computation. Recently, stochastic competition principles have been considered, e.g. Panousis et al.
(2019; 2021), Voskou et al. (2021).

Let us denote asand is sparse, since all units except for one, in each block, yield zero values. Let us introduce the y ∈ R[R][·][J] the output of an LWTA layer; this is composed of R subvectors yr ∈ R[J]
discrete latent indicator vector ξ ∈ one hot(J)[R] to denote the winner units in the R blocks that
constitute a considered stochastic LWTA layer. This vector comprises R component subvectors,
where each component entails one non-zero value at the index position that corresponds to the
winner unit of the respective LWTA block. On this basis, the output y of the stochastic LWTA
layer’s (r · j)-th component yr,j is defined as:


_i=1(wi,r,j) · xi ∈_ R (6)

X


**_yr,j = ξr,j_**


where we denote asξ. **_ξr,j the j-th component of ξr, and ξr ∈_** one hot(J) holds the r-th subvector of

In Eq. (6), we postulate that the latent winner indicator variables are drawn from a Categorical
distribution which is proportional to the intermediate linear computation that each unit performs.
Therefore, the stronger the linearity the higher the chance of the unit winning the stochastic competition within its block. In detail, we postulate that, a posteriori, the winner distributions yield:


**_ξr_** softmax(
# |



[wi,r,j][J]j=1
_i=1_ _[·][ x][i][)]_

X


(7)


_q(ξr) = Categorical_


where [wi,r,j][J]j=1 [denotes the vector concatenation of the set][ {][w][i,r,j][}]j[J]=1[. A graphical illustration of]
the proposed stochastic architecture is provided in Fig. 1.

As a network composed of such (StochLWTA) layers entails latent variables ξ, we need to perform
a Bayesian network treatment to perform effective parameter training. We opt for a (approximate)
stochastic gradient variational Bayes treatment (Kingma & Welling, 2014), for scalability purposes.
This means that the used objective function takes the form of an evidence lower-bound (ELBO)
objective, as we describe next. In our work, we take one step further: we also elect to infer a
posterior density over the network weights W, as opposed to obtaining point-estimates. This results
in a second source of stochasticity for our approach, which may further increase its generalization
capacity under uncertain conditions arising from limited training data availability. Note that the
use of these posteriors is totally different from Panousis et al. (2019) and Voskou et al. (2021):
therein, posterior variance is used for compressing posterior mean bit-precision; then, predictions
are performed using only the compressed posterior mean. Instead, in our work we sample multiple
times from the trained weight posterior and perform model averaging (in a Bayesian sense), as a
means of increasing generalization capacity.

We postulate:
_q(vec(W )) = N_ (vec(W )|µ, diag(σ[2])) (8)

where ψ ≜ _{µ, σ[2]} are the means and variances of the Gaussian weight posteriors, respectively._

3.2 A MODEL-AGNOSTIC ML ALGORITHM

Let us define an ML problem where we are given a training dataset D of tasks, T, that are governed
by a distribution P (T ). We consider an N -way, K-shot learning setting, where each task contains
_K labelled examples for each of N available classes._

Our approach entails parameter sets θ which coincide with the hyperparameter sets ψ = {µ, σ[2]}
of the weights W ∈ R[I][×][R][×][J] ; these sets ψ = {µ, σ[2]} are the target of our MAML-type training
algorithm.

Therefore, to perform model training, we have to first initialize the trainable parameters µ and σ[2]
across layers. To this end, one can appropriately exploit popular initialization schemes, such as


-----

Figure 1: A zoomed-in graphical illustration of the r-th block of a stochastic LWTA layer. Input
**_x = [x1, x2, . . ., xI_** ] is presented to each unit in the block. Assume that the index of the winner unit
is j0. Then, the output of the block is a vector with a single non-zero value at index j0.

Glorot Uniform (Glorot & Bengio, 2010). Then, our approach entails an inner-outer loop scheme,
in a vein similar to existing approaches, but with some crucial differences:

1. The stochastic nature of the postulated weights, W, results in the updates taking place over
the posterior means, µ and variances, σ[2].

2. The stochastic nature of both the inferred representations and the network weights themselves implies that proper training must rely on optimization of the ELBO function of
the network. Let us consider an inner-loop dealing with task Ti _P_ (T ), with data
_∼_
_Di = (Xi, Yi) ⊂_ _D. Let CE(Yi, fψ(Xi; ξ[ˆ],_ **_W[ˆ]_** )) be the categorical cross-entropy between
the data labels Yi and the class probabilities fψ(Xi; ξ[ˆ], **_W[ˆ]_** ) generated by the penultimate
Softmax layer. Then, we yield:

_LTi_ (ψ) = CE(Yi, fψ(Xi; ξ[ˆ], **_W[ˆ]_** )) _KL[ q(ξ)_ _p(ξ) ]_ _KL[ q(W )_ _p(W ) ]_ (9)
_−_ _−_ _||_ _−_ _||_

for task Ti with dataset Di which is dealt with on the i-th training iteration.

Here, for simplicity and without harming generality, we consider that the weights prior p(vec(W ))
is a Gaussian distribution N (0, I) and the latent variables prior p(ξ) is a symmetric Categorical
distribution Categorical(1/J). In addition, in our notation we stress that the output fψ(Xi; ξ[ˆ], **_W[ˆ]_** )
depends on the winner selection process, which is stochastic, and the outcomes of sampling the
network weights. Specifically, in our work we perform Monte-Carlo (MC) sampling using one
reparameterized sample of the corresponding latent variables.

Let **_ξ[¯] be the unnormalized probabilities of the Categorical distribution q(ξ) (Eq. (7)). The sampled_**
instances of ξ, **_ξ[ˆ]r,j, are expressed as (Maddison et al., 2017):_**

**_ξˆr,j = Softmax((log ξ[¯]r,j + gr,j)/τ_** ), ∀r = 1, . . ., R, j = 1, . . ., J (10)

wherecontrols how closely the Categorical distribution is approximated by this continuous relaxation. gr,j = − log( − log Ur,j), Ur,j ∼ Uniform(0, 1), and τ ∈ (0, ∞) is a temperature factor that

Similarly, the Gaussian weights yield: ˆwt,r,j = µt,r,j + σt,r,jϵ,ˆ and ˆϵ ∼ _N_ (0, 1).

On this basis, the KL divergences in Eq. (9) become:


_KL[ q(ξr,j || p(ξr,j) ] = Eq(ξr,j_ )[log q(ξr,j) − log p(ξr,j)] (11)

log q(ξ[ˆ]r,j) log p(ξ[ˆ]r,j), _r, j_
_≈_ _−_ _∀_

and
_KL[ q(wt,r,j) || p(wt,r,j) ] = Eq(wt,r,j_ )[log q(wt,r,j) − log p(wt,r,j)] (12)

log q( ˆwt,r,j) log p( ˆwt,r,j), _t = 1, . . ., I, r, j_
_≈_ _−_ _∀_


-----

Hence, the ELBO becomes:

_LTi_ (ψ) = CE(Yi, fψ(Xi; ξ[ˆ], **_W[ˆ]_** ))
_−_ _−_


log q( ξ[ˆ]r,j) log p( ξ[ˆ]r,j)
_−_


(log q( ˆwt,r,j) log p( ˆwt,r,j))
_−_
_t,r,j_

X

(13)


_r,j_


Therefore, we establish a MAML-type algorithm, where: (i) the used networks comprise blocks of stochastic
LWTA units visually depicted in Fig. 1; (ii) the trainable parameters are the means µ and variances σ[2] of the
synaptic weights; and (iii) the objective function of the inner-loop process is given in Eq. (13).

We summarize our training algorithm in Alg. 1.

**Algorithm 1: Model training with StochLWTA-ML**

**Require: P** (T ): distribution over tasks
Initialize ψ := {µ, σ[2]}
Define outer-step size β and inner learning rate α
**for i = 1,2, . . . do**

**Inner training loop:**
Sample task Ti ∼ _P_ (T ), where Ti contains data Di = (Xi, Yi) ⊂ _D_
Compute LTi (ψ) using Eq. (13)
Compute adapted parameters with SGD: ψi[′] [=][ ψ][ −] _[α][∇][ψ][L][T]i_ [(][f][ψ][)]
**Outer training loop:**
Derive ψ **_ψ + β(ψi[′]_**
_←_ _[−]_ **_[ψ][)]_**
**end**

3.3 PREDICTION ALGORITHM

At prediction time, we draw a set of B samples of the Gaussian connection weights from the trained posteriors
_N_ (µ, σ[2]). Then, we select the winning units in each block of the network by similarly sampling from the
posteriors q(ξ). This results in a set of B output logits of the network, which we average to obtain the final
predictive outcome:


_fψ(Xi; ξ[˜],_ **_W[˜]_** ) ≈ _B[1]_


_fψ(Xi; ξ[˜]s,_ **_W[˜]_** _s)_ (14)

_s=1_

X


where **_ξ[˜]s and_** **_W[˜]_** _s are sampled directly from the posteriors q(ξ) and q(W ), respectively._

This concludes the formulation of the proposed model-agnostic ML approach.

**Algorithm 2: Prediction with StochLWTA-ML**

**Require: Learned parameters ψ = {µ, σ[2]}, input data X** _[′]_

Sample W ∼ _q(vec(W )) = N_ (vec(W )|µ, diag(σ[2]))
Sample ξ ∼ _q(ξ) defined in Eq. (7), for w = ψ and (xi = x[′]i[)][ ∈]_ _[X]_ _[′]_

Compute output logits, given the sampled values ξ and ψ
**Repeat B times**
Use Eq. (14) to average over the resulting set of B logits and derive the final prediction.

4 EXPERIMENTS

4.1 EXPERIMENTAL SETUP

We evaluate StochLWTA-ML on Omniglot, Mini-Imagenet and CIFAR-100 which are popular few-shot image
classification datasets, and compare its performance to state-of-the-art prior results. After thorough exploration
on the number of LWTA layers as well as the number of blocks for each layer and the competing units per
block, we end up with using networks comprising 2 layers with 16 blocks and 2 competing units per block
on the former layer, and 8 blocks with 2 units per block on the latter. The penultimate network layer is a
Softmax. Weight mean initialization, as well as point-estimate initialization for our competitors, is performed
via Glorot Uniform. Weight log-variance initialization is performed via Glorot Normal, by sampling from
_N_ (0.0005, 0.01). The Gumbel-Softmax relaxation temperature is set to τ = 0.67.


-----

In the inner-loop updates, we use the Stochastic Gradient Descent (SGD) (Robbins, 2007) optimizer with a
learning rate of 0.003. For the outer-loop, we use SGD with a linear annealed outer step size to 0, and an initial
value of 0.25. Additionally, all the experiments were ran with task batch size of 50 for both training and testing
mode. Prediction is carried out averaging over B = 4 output logits.

The results presented in Sections 4.2 and 4.3 stand for the average performance over three runs with different
random seeds. Note that each experiment consists of different number of iterations for training, depending on
its convergence speed. The code was implemented in Tensorflow (Abadi et al., 2016).

4.2 RESULTS

In Table 1, we show how StochLWTA-ML performs on Omniglot 20-way, Mini-Imagenet 5-way and CIFAR100 5-way few-shot settings. We compare our findings to state-of-the-art ML algorithms such as LLAMA
(Grant et al., 2018) and PLATIPUS (Finn et al., 2018) as reported in Gordon et al. (2018), Amortized Bayesian
Meta-Learning (ABML) (Ravi & Beatson, 2019), MAML, FOMAML (Finn et al., 2017), Reptile (Nichol
et al., 2018) and others. Using the original architectures with the same hyperparameters and data preprocessing
as in Finn et al. (2017), we have also locally reproduced ABML, BMAML (with 5 particles), PLATIPUS,
MAML, FOMAML and Reptile (dubbed ”local” in Table 1). For completeness sake, we also compare our
findings to other state-of-the-art ML models as reported in Finn et al. (2017), including Matching Nets (Santoro
et al., 2016) and LSTM Meta-Learner (Ravi & Larochelle, 2017). As we observe, our method outperforms the
existing state-of-the-art in both the 1-shot and 5-shot settings.

Table 1: N-way K-shot (%) classification accuracies on Omniglot, Mini-Imagenet and CIFAR-100




|Col1|Omniglot 20-way|Mini-Imagenet 5-way|CIFAR-100 5-way|
|---|---|---|---|
|Algorithm|1-shot 5-shot|1-shot 5-shot|1-shot 5-shot|
|Matching Nets LSTM Meta-Learner MAML FOMAML Reptile PredCP (Nalisnick et al., 2021) Neural Statistician (Edwards & Storkey, 2016) mAP-SSVM (Triantafillou et al., 2017) LLAMA PLATIPUS GEM-BML+ (Zou & Lu, 2020) DKT (Patacchiola et al., 2020) ABML BMAML (with 5 particles) (Yoon et al., 2018)|93.80 98.50 - - 95.80 98.90 - - 88.14 96.65 - - 93.20 98.10 95.20 98.60 - - - - 96.24 98.94 - - - - - -|43.56 55.31 43.44 60.60 48.70 63.11 48.07 63.15 47.07 62.74 49.30 61.90 - - 50.32 63.94 49.40 - 50.13 - 50.03 - 49.73 64.00 45.00 - 53.80 -|- - - - - - - - - - - - - - - - - - - - - - - - 49.50 - -|
|ABML (local) BMAML (local) PLATIPUS (local) MAML (local) FOMAML (local) Reptile (local)|90.21 93.39 96.92 98.11 94.35 98.30 95.48 98.61 94.92 98.12 87.98 96.36|44.23 52.12 53.10 64.80 49.97 63.13 48.60 63.01 47.93 63.10 46.97 62.53|49.23 53.60 52.60 65.80 51.14 63.61 50.67 62.89 49.13 63.80 48.19 63.45|
|StochLWTA-ML|97.79 98.97|54.11 66.70|54.60 66.73|


4.3 ABLATION STUDY

4.3.1 DOES STOCHASTIC COMPETITION CONTRIBUTE TO CLASSIFICATION ACCURACY?

To check whether the accuracy improvements stem from the LWTA-induced sparsity or the proposed stochastic competition concept, we evaluate both our approach as well as MAML, FOMAML, ABML, BMAML and
PLATIPUS, considering both ”deterministic LWTA” and ”stochastic LWTA” setups; deterministic LWTA networks have been adopted from Srivastava et al. (2013). As we see in Table 2, replacing ReLU with deterministic
LWTA yields negligible differences. On the other hand, stochastic LWTA units yield a clear improvement in
all cases. This improvement becomes even more important in the case of our approach, where we sample from
stochastic weights.


-----

Table 2: Mini-Imagenet 5-way Few-Shot ablation study (% accuracy)

|Algorithm|Network type|1-shot|5-shot|
|---|---|---|---|
|MAML (local)|deterministic LWTA stochastic LWTA|48.88 49.61|63.15 64.03|
|FOMAML (local)|deterministic LWTA stochastic LWTA|48.11 49.24|63.54 64.54|
|ABML (local)|deterministic LWTA stochastic LWTA|44.31 45.11|52.27 53.31|
|BMAML (local)|deterministic LWTA stochastic LWTA|53.12 53.50|64.84 65.31|
|PLATIPUS (local)|deterministic LWTA stochastic LWTA|49.99 51.06|63.21 64.18|
|StochLWTA-ML|deterministic LWTA stochastic LWTA|53.12 54.11|64.93 66.70|



4.3.2 EFFECT OF BLOCK SIZE J

As it is presented in Table 3, increasing the number of competing units per block to J = 4 or J = 8 does not
notably improve the results of our approach. On the contrary, it increases the number of trained parameters,
thus leading to higher network computational complexity. This corroborates our initial choice of using J = 2
competing units per block in our approach.

Table 3: Effect of block size J in StochLWTA-ML’s classification (%) accuracy

|Col1|Omniglot 20-way|Mini-Imagenet 5-way|CIFAR-100 5-way|
|---|---|---|---|
|Number of units|1-shot 5-shot|1-shot 5-shot|1-shot 5-shot|
|J = 2 J = 4 J = 8|97.79 98.97 96.33 98.55 95.38 98.83|54.11 66.70 53.99 66.65 53.70 67.08|54.60 66.73 54.51 66.13 54.45 66.18|



4.3.3 HOW DOES THE NUMBER OF SAMPLES AT PREDICTION TIME AFFECT ACCURACY?

We scrutinize the effect of the number of drawn samples, B, on StochLWTA-ML’s predictive accuracy. To this
end, we repeat our experiments using B = 10 logits sets. In Table 4, we provide the comparative outcomes
concerning the two sample size configurations of B = 4 and B = 10. As we observe, an increase in prediction
sample size, B, yields a slight accuracy increase. However, the aforementioned increase in sample size imposes
some computational overhead, which we elaborate upon in the following Section. We argue that this overhead
might not be worth it for the slight performance increase reported in Table 4. More information on the effect of
sample size in our approach’s predictive performance are provided in the Supplementary.

Table 4: Effect of sample size B in StochLWTA-ML’s classification (%) accuracy

|Col1|Omniglot 20-way|Mini-Imagenet 5-way|CIFAR-100 5-way|
|---|---|---|---|
|Number of samples|1-shot 5-shot|1-shot 5-shot|1-shot 5-shot|
|B = 4 B = 10|97.79 98.97 96.91 99.23|54.11 66.70 54.89 67.22|54.60 66.73 55.18 66.15|



4.3.4 IS THERE A COMPUTATIONAL TIME TRADE-OFF FOR THE INCREASED ACCURACY?

It is also important to investigate whether our approach represents a trade-off between accuracy and computational time compared to our competitors. To facilitate this investigation, in Table 5 we provide training
iteration wall-clock times for our approach and the existing locally reproduced state-of-the-art, as well as the
total number of iterations each model needs to achieve the reported performance of Table 1. It appears that our
methodology takes 77% less training time than the less efficient algorithms ABML, BMAML, PLATIPUS, and
is comparable to other approaches. This happens because our approach yields the reported state-of-the-art performance by employing a network architecture (that is, number of LWTA layers, as well as number of blocks
and block size on each layer) that result in a total number of trainable parameters that is one order of magnitude
_less on average than the best performing baseline methods. This can be seen in the last three columns of Table_


-----

5 (dubbed DA, DB and DC for Omniglot, Mini-Imagenet and CIFAR-100 respectively). In addition, training
for our approach converges fast.

The situation changes when it comes to prediction: our approach imposes a slight computational time overhead
compared to MAML, FOMAML and Reptile, but still much less than the time-consuming PLATIPUS. BMAML
and ABML. This is a rather negligible increase when we are dealing with a low number of drawn samples, B,
but increases as we increase B. The provided results suggest that a selection of B = 4 represents a favorable
accuracy/prediction wall-clock time for our approach.

Table 5: Performance comparison: average wall-clock time (in msecs), training iterations for each
locally reproduced method and number of baselines’ trainable parameters over the considered
datasets of Table 1

|Algorithm|Training|Prediction|Number of training iterations|DA parameters|DB parameters|DC parameters|
|---|---|---|---|---|---|---|
|PLATIPUS (local) BMAML (local) ABML (local) MAML (local) FOMAML (local) Reptile (local)|1603.39 1450.31 678.48 288.25 284.49 284.30|602.77 514.43 265.78 103.28 102.34 102.27|333600 301800 138000 60000 60000 60000|560025 560025 224010 112005 112005 113221|615395 615395 246158 123079 123079 124613|580440 580440 232176 116088 116088 117463|
|StochLWTA-ML|282.90|113.44 (B = 4) 121.87 (B = 10)|60000|54549|60112|56745|



Finally, we provide an example of how training for our approach converges, and how this compares to the
alternatives. We illustrate our outcomes on the Omniglot 20-way 1-shot benchmark; similar outcomes have
been observed in the rest of the considered datasets. Fig. 2(a) compares StochLWTA-ML with prior traditional
ML methods: MAML, FOMAML and Reptile. It becomes apparent that our approach converges equally fast
to these competitors. Further, Fig. 2(b) compares StochLWTA-ML with the probabilistic ML models ABML,
BMAML, PLATIPUS. Since, as we see in the ablation study of Section 4.3.4, these methods are quite timeconsuming and less efficient regarding to memory consumption, StochLWTA-ML gives rise to an easier time
training MAML based probabilistic model.

(a) (b)

Figure 2: ML algorithms’ training convergence comparison

5 CONCLUSION

In this paper, we proposed a sparse and stochastic network paradigm for ML, with novel network design principles compared to currently used model-agnostic ML models. We introduced stochastic LWTA activations in
the context of a variational Bayesian treatment that gave rise to a doubly-stochastic ML framework, bearing
the promise of stronger generalization capacity. We evaluated our approach using standard benchmarks in the
field, and showed that it outperformed the state-of-the-art in terms of both predictive accuracy and computational costs. The results have provided strong empirical evidence supporting our claims. In the future, we plan
to study the effect of StochLWTA-ML in areas related to ML, such as Continual Learning (Javed & White,
2019) and Reinforcement Learning (Zhu et al., 2020).


-----

REFERENCES

Mart´ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado,
Andy Davis, Jeffrey Dean, and Matthieu Devin. Tensorflow: Large-scale machine learning on heterogeneous
distributed systems. pp. 265–283. In Proceedings of the 12th USENIX conference on Operating Systems
_Design and Implementation, 2016._

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan
Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Neural
_Information Processing Systems, 2016._

Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-learning with
adaptive hyperparameters. In Neural Information Processing Systems, 2020.

Paul Black, Robert McCormick, Mary James, and David Pedderd. Learning how to learn and assessment for
learning: a theoretical inquiry. Research Papers in Education, 21:119–132, 2006.

Rui Castro, Charles Kalish, Robert Nowak, Ruichen Qian, Tim Rogers, and Jerry Zhu. Human active learning.
In Neural Information Processing Systems, 2008.

Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnaud Doucet, David Budden, Matthew Hoffman, and
Nando de Freitas. Modular meta-learning with shrinkage. In Neural Information Processing Systems, 2020.

Harrison Edwards and Amos Storkey. Towards a neural statistician. 2016. URL arXivpreprintarXiv:
1606.02185.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep
networks. In International Conference on Machine Learning, 2017.

Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Neural Informa_tion Processing Systems, 2018._

Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International
_Conference on Machine Learning, 2019._

Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Metalearning with warped gradient descent. In International Conference on Learning Representations, 2020.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks.
_Journal of Machine Learning Research 9, pp. 249–256, 2010._

Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta-learning
probabilistic inference for prediction. In International Conference on Learning Representations, 2018.

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based
meta-learning as hierarchical bayes. In International Conference on Learning Representations, 2018.

Matthias Hutsebaut-Buysse, Kevin Mets, and Steven Latr´e. Fast task-adaptation for tasks labeled using natural
[language in reinforcement learning. 2019. URL https://arxiv.org/abs/1910.04040.](https://arxiv.org/abs/1910.04040)

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing
internal covariate shift. In International Conference on Machine Learning, 2015.

K. Javed and M White. Meta-learning representations for continual learning. pp. 1818–1828. In Neural Infor_mation Processing Systems, 2019._

Samuel Kessler, Vu Nguyen, Stefan Zohren, and Stephen Roberts. Hierarchical indian buffet neural networks
for bayesian continual learning. In UAI, 2021.

D. P. Kingma and M Welling. Auto-encoding variational bayes. In International Conference on Learning
_Representations, 2014._

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Niklas K¨uhl, Marc Goutier, Lucas Baier, Clemens Wolff, and Dominik Martin. Human vs. supervised machine
[learning: Who learns patterns faster? 2020. URL https://arxiv.org/abs/2012.03661.](https://arxiv.org/abs/2012.03661)

Yunfei Lai. A comparison of traditional machine learning and deep learning in image recognition. Journal of
_Physics: Conference Series, 1314, 2019._


-----

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through
probabilistic program induction. Science, 350(6266):1332–1338, 2015.

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that
learn and think like people. Behavioral and Brain Sciences, 40, 2017.

Juho Lee, Saehoon Kim, Jaehong Yoon, Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Adaptive network
sparsification with dependent variational beta-bernoulli dropout. 2018. URL arXivpreprintarXiv:
1805.10896.

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of
discrete random variables. In International Conference on Learning Representations, 2017.

Eric Nalisnick, Jonathan Gordon, and Jos´e Miguel Hern´andez-Lobato. Predictive complexity priors. In Neural
_Information Processing Systems, 2021._

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. 2018. URL

[https://arxiv.org/abs/1803.02999.](https://arxiv.org/abs/1803.02999)

Konstantinos Panousis, Sotirios Chatzis, Antonios Alexos, and Sergios Theodoridis. Local competition and
stochasticity for adversarial robustness in deep learning. In International Conference on Artificial Intelli_gence and Statistics, 2021._

Konstantinos P. Panousis, Sotirios Chatzis, and Sergios Theodoridis. Nonparametric bayesian deep networks
with local competition. In International Conference on Machine Learning, 2019.

Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O’Boyle, and Amos Storkey. Bayesian
meta-learning for the few-shot setting via deep kernels. In Neural Information Processing Systems, 2020.

Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning
_Representations, 2019._

Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference
_on Learning Representations, 2017._

H. Robbins. A stochastic approximation method. Annals of Mathematical Statistics, 2007.

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning
with memory-augmented neural networks. In International Conference on Machine Learning, 2016.

Jurgen Schmidhuber. On learning how to learn: The meta-meta-... hook., evolutionary principles in selfreferential learning. Master’s thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.

D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, and Todd Phillips. Hidden technical debt in machine
learning systems. In Neural Information Processing Systems, 2015.

Rupesh Kumar Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and J¨urgen Schmidhube.
Compete to compute. In Neural Information Processing Systems, 2013.

S. Thrun and L. (eds.). Pratt. Learning to Learn. Kluwer Academic Publishers, USA, 1998. ISBN 0792380479.

E. Triantafillou, R. Zemel, and R. Urtasun. Few-shot learning through an information retrieval lens. In Neural
_Information Processing Systems, 2017._

Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence
_Review, 18(2):77–95, 2002._

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Neural Information Processing Systems, 2016.

Andreas Voskou, Konstantinos Panousis, Dimitrios Kosmopoulos, Dimitris Metaxas, and Sotirios Chatzis.
Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. In
_International Conference on Computer Vision, 2021._

J. X Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z Leibo, R. Munos, C. Blundell, D. Kumaran, and
M. Botvinick. Learning to reinforcement learn. In International Conference on Machine Learning, 2016.

J. Yoon, T. Kim, O. Dia, O. Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. In Neural
_Information Processing Systems, 2018._


-----

Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and
Sergey Levine. The ingredients of real-world robotic reinforcement learning. In International Conference
_on Learning Representations, 2020._

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Confer_ence on Machine Learning, 2017._

Yayi Zou and Xiaoqi Lu. Gradient-em bayesian meta-learning. In Neural Information Processing Systems,
2020.

A FURTHER DETAILS ON DATASETS EXPERIMENTAL FORMATION

Omniglot is a dataset of 1623 characters from different alphabets, containing 20 examples per character scaled
down to 28x28 grayscale pixels. The ratio between training and testings sets is 3:2, so after shuffling the
character classes we randomly choose the first 974 classes for training and the remaining are left for testing. As
for the Mini-Imagenet dataset, it has color images of size 84x84 and contains 100 classes with 600 examples
from the ImageNet dataset. We randomly choose 45000 examples for the training phase and the rest constitute
the testing population. The CIFAR-100 dataset consists of color images of size 32x32 and contains 100 classes
with 600 images per class. We randomly choose 500 images per class for training and the rest 100 images per
class constitute the testing population.

B ADDITIONAL EXPERIMENTS

In Table B1 we provide the results of the ablation study of Section 4.3.1 on the Omniglot 20-way dataset. The
full-fledged StochLWTA-ML approach yields again better predictive performance compared to the alternative
variants of ML algorithms.

Table B1: Omniglot 20-way Few-Shot ablation study (% accuracy)

|Algorithm|Network type|1-shot|5-shot|
|---|---|---|---|
|MAML (local)|deterministic LWTA stochastic LWTA|95.52 95.91|98.15 98.78|
|FOMAML (local)|deterministic LWTA stochastic LWTA|95.01 95.80|98.18 98.41|
|ABML (local)|deterministic LWTA stochastic LWTA|90.30 91.21|93.64 93.91|
|BMAML (local)|deterministic LWTA stochastic LWTA|96.96 97.11|98.21 98.30|
|PLATIPUS (local)|deterministic LWTA stochastic LWTA|94.48 95.13|98.31 98.56|
|StochLWTA-ML|deterministic LWTA stochastic LWTA|96.95 97.79|98.63 98.97|



C FEW-SHOT CLASSIFICATION NETWORK ARCHITECTURES

For the local replicates of prior ML algorithms in the experiments of our work, we follow the same architecture
for the deep neural network as the one used by Vinyals et al. (2016). For Omniglot, the network is composed of
4 convolutional layers with 64 filters, 3 x 3 convolutions and 2 x 2 strides, followed by a Batch Normalization
layer (Ioffe & Szegedy, 2015) and the final values of each layer are processed by an activation function. For
both Mini-Imagenet and CIFAR-100, we use 4 convolutional layers with 32 filters to reduce overfitting like
Ravi & Larochelle (2017), 3 x 3 convolutions followed by Batch Normalization layer and 2 × 2 max-pooling
layer with the values of each layer finally passed again through an activation block. The activation functions
used for experiments of main paper’s Tables 1 and 5 is ReLU, and LWTA for experiments of main paper’s Table
2 and Supplementary’s Table B1.


-----

D HOW DOES THE TASK BATCH SIZE AFFECT STOCHLWTA-ML’S
PERFORMANCE?

In Fig. 3 and 4, we illustrate the performance of our model on the Mini-Imagenet 5-way 1-shot setting with
different task batch sizes. As we see, our model performs optimally with task batch size of 50 for both training
and testing phase. Using this value for batch size, we noticed that the classification accuracy as well as training
time per iteration were optimal. This outcome was also observed in the rest of the considered datasets’ settings.

Figure 3: The effect of task batch size in StochLWTA’s predictive accuracy

Figure 4: The effect of task batch size in StochLWTA’s training time per iteration (in msecs)


-----

E HOW DOES THE SAMPLE SIZE B AT PREDICTION TIME AFFECT
STOCHLWTA-ML’S ACCURACY?

As we observe in Fig. 5, an increase in sample size, B, does not always yield an accuracy increase. We finally
choose B = 4, since at that point our model achieves its best predictive performance at most of the experiments.
In this figure, we illustrate our outcomes on Omniglot 20-way 1-shot, Mini-Imagenet 5-way 1-shot and CIFAR100 5-way 5-shot settings. Our approach has shown similar behaviour to the rest of experimental settings,
leading us to the same choice for the number of sample size B.

Figure 5: The effect of sample size B in StochLWTA-ML’s classification (%) accuracy

F WHAT PARAMETERS DO WE COUNT FOR THE OUTCOMES OF MAIN
PAPER’S TABLE 5?

The included parameters of each baseline for the outcomes of Table 5 are:

-  PLATIPUS: Θ = {µθ, σθ[2][,][ v][θ][,][ γ][p][,][ γ][q][}]

-  BMAML: Θ = {θ[m]}m[5] =1[, for using 5 particles]

-  ABML: θ = {µθ, σθ[2][}]

-  MAML: θ = **_µθ_**
_{_ _}_

-  FOMAML: θ = **_µθ_**
_{_ _}_

-  Reptile: θ = **_µθ_**
_{_ _}_

-  StochLWTA-ML: θ = {µθ, σθ[2][}]


-----

