# CSQ: CENTERED SYMMETRIC QUANTIZATION FOR EXTREMELY LOW BIT NEURAL NETWORKS

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Recent advances in quantized neural networks (QNNs) are closing the performance gap with the full precision neural networks. However at very low precision
(i.e., ≤ 3-bits), QNNs often still suffer significant performance degradation. The
conventional uniform symmetric quantization scheme allocates unequal numbers
of positive and negative quantization levels. We show that this asymmetry in the
number of positive and negative quantization levels can result in significant quantization error and performance degradation at low precision. We propose and analyze a quantizer called centered symmetric quantizer (CSQ), which preserves the
symmetry of latent distribution by providing equal representations to the negative
and positive sides of the distribution. We also propose a novel method to efficiently map CSQ to binarized neural network hardware using bitwise operations.
Our analyses and experimental results using state-of-the-art quantization methods on ImageNet and CIFAR-10 show the importance of using CSQ for weight
in place of the conventional quantization scheme at extremely low-bit precision
(2∼3 bits).

1 INTRODUCTION

Quantized neural networks (QNNs) (Krishnamoorthi, 2018; Esser et al., 2019; Lee et al., 2021; Li
et al., 2021; Nagel et al., 2020) can reduce both computational complexity and memory requirement
quite effectively over the full-precision (i.e., floating-point) version, and hence are commonly used
for deployment. Recent state-of-the-art QNNs (Esser et al., 2019; Lee et al., 2021; Li et al., 2019) can
achieve near-full-precision accuracy even at 3-bit for ImageNet classification. However, most of the
existing quantization method target 3-bit or higher. At 2-bit there is not much room for optimization,
even with non-uniform step-size quantization (Li et al., 2019).

While some previous work, e.g. (Li et al., 2019), treats 2-bit as ternary ({−1, 0, 1}), it wastes one
quantization level. But to make the most of the 2-bit precision, the conventional 2-bit quantizer
allocates two vs. one quantization levels to the negative side, causing a severe imbalance. As an
alternative to the conventional quantizer, we propose (zero-)centered symmetric quantizer (CSQ),
which is most effective at low precision (2∼3-bit) though the method itself is applicable to any
precision. Through a set of analyses and experiments we show that CSQ improves QNN performance
over the conventional linear quantizer (CLQ) in quantization-aware training (QAT).[1]

We note that the particular quantizer function we propose has been used in some previous work (Choi
et al., 2018a; Boo et al., 2021; Gong et al., 2019; Lee et al., 2021; Chen et al., 2021). However, no
previous work has proposed it explicitly nor provided any analysis on the effectiveness of such a
scheme (see Section 2.3). Moreover, while low-precision CLQ can be efficiently implemented in
hardware via multiple invocations of XNOR-popcount such as on BNN (Binarized Neural Network)
hardware, doing the same for CSQ poses new challenges that have never been addressed.

In this paper we make the following contributions.

-  We propose CSQ that uses perfectly symmetrical quantization levels with uniform step size.
We provide analytical and empirical evidence showing that using CSQ for weight instead
of CLQ improves performance of low-bit QNNs (2∼3-bit).

1We use CSQ (and CLQ) to refer to both a quantizer and a quantization method.


-----

-  We propose a binary coding scheme and a mapping method for CSQ, which allows for an
efficient hardware implementation of QNNs using bitwise operations on BNN hardware.

2 BACKGROUND AND RELATED WORK

2.1 QUANTIZATION PRIMER

A quantizer is a function from a real number x to a discrete value or an integer xQ. To train a network
while simulating the effect of quantizer, one often maps the discrete value xQ back to a real value ˜x.
Instead of quantizing the entire range of inputs, clipping off the extreme values and only mapping
the mid-range values uniformly is often more efficient. A generalized version of a quantizer used in
Neural Networks can be defined as:

_x_
_x˜ = clip(_ _, L, U_ ) _s_ (1)

_s_ _·_

j m

where L and U (the lower and upper bound) are the minimum and maximum integer values that
_xQ can take, and clip(x, a, b) = min(max(x, a), b). Quantization using the above quantizer in (1)_
is called uniform step-size quantization, also called linear quantization. We do not consider nonuniform step-size quantization in this paper.

Based on the input range (i.e., whether input is signed vs. un-signed), quantizers are classified as
symmetric vs. asymmetric quantizers. For the linear quantizer in (1), symmetric and asymmetric
quantizers can be defined as follows, where b is the number of bits used for representation.

Symmetric quantizer (for signed input): _L = −2[b][−][1],_ _U = 2[b][−][1]_ _−_ 1 (2)

Asymmetric quantizer (for un-signed input): _L = 0,_ _U = 2[b]_ _−_ 1 (3)

From here on we will refer to quantization method defined by (1) and (2) as the Conventional Linear
Quantizer (CLQ).

2.2 PREVIOUS WORK ON DNN QUANTIZATION

Most of the previous works on DNN quantization (Jung et al., 2019; Choi et al., 2018b; Esser
et al., 2019) use essentially the same linear quantizer, but each work proposes a slightly different
way to optimize quantizer functions. Often the differences come from different formulations of
the quantizer function. For instance, QIL (Jung et al., 2019) optimizes quantization boundary, PACT
(Choi et al., 2018b) optimizes a clipping parameter to train the quantized activations, and LSQ(Esser
et al., 2019) optimizes the step size.We are not aware of any previous work explicitly proposing
a new set of quantization levels for uniform quantization. Though the formulation of a quantizer
function varies, in the end they all assume the same integer hardware that expects 2’s complement
number representation.

2.3 SIMILAR APPROACHES

The idea of using a zero centered quantizer is also seen in (Choi et al., 2018a; Boo et al., 2021; Gong
et al., 2019; Lee et al., 2021; Chen et al., 2021). All of these works present a novel training method
for their quantizer. However, none of these works analyze zero-centered quantization or present
it as a source of performance gain. Furthermore, to the best of our knowledge, no previous work
has attempted to address the realization of zero-centered quantization on hardware. Realizing zerocentered quantization on hardware can be a challenging task, because it introduces new quantization
points which can not be represented using 2’s complement number representation.

3 CENTERED SYMMETRIC QUANTIZATION (CSQ)

3.1 EXACT ZERO REPRESENTATION VS. PERFECT SYMMETRY

Any linear quantization to b-bit precision results in 2[b] quantization levels. Therefore, inclusion of
zero among the quantization points, naturally results in an asymmetry in the number of positive and


-----

Table 1: Quantization levels by representation (2-bit example)

_∗Note: ESQ requires more than 2 bits._

|Case|Quantization levels|Representation|
|---|---|---|
|1 2 3 4 5 6 7|2, 1, 0, 1 {− − } 1, 0, 1, 2 {− } 1, 0, 1 {− } 2, 1, 0, 1, 2 {− − } 2, 1, 1, 2 {− − } 1.5, 0.5, 0.5, 1.5 {− − } 3, 1, 1, 3 {− − }|CLQ CLQ-alternative RSQ (Ternary) ESQ∗ NSQ CSQ CSQ (scaled by 2)|



negative quantization levels. While exact representation of zero may be important due to common
operations like zero padding (Krishnamoorthi, 2018), it is also true that the asymmetry among positive and negative quantization levels grows large as precision becomes low. Thus one of our aims in
this paper is to explore the trade-off between exact zero representation and perfect symmetry in the
context of weight quantization of neural networks.

3.2 ALTERNATIVES

From the perspective of quantization levels, one can consider the following quantization schemes
(see Table 1). Conventional Linear Quantization (CLQ) uses the quantization levels represented
by 2’s complement, which is also used by most of the previous uniform step-size quantizers, e.g.
(1). Alternatively, the asymmetry between the positive and negative sides can be reversed; i.e., by
setting L = −2[b][−][1] + 1 and U = 2[b][−][1]. Reduced Symmetric Quantization (RSQ) uses one less
quantization level, thereby achieving both exact zero representation and perfect symmetry; i.e., L =
_−2[b][−][1]_ +1 and U = 2[b][−][1] _−_ 1. This scheme wastes one quantization level and is expected to result in
inferior performance. Extended Symmetric Quantization (ESQ), on the other hand, uses one more
quantization level to achieve zero representation and symmetry; i.e., L = −2[b][−][1] and U = 2[b][−][1].
However, since ESQ requires more than b-bit precision, it is not feasible for practical deployment,
but included here for comparison. Non-uniform Symmetric Quantization (NSQ) achieves both
exact zero representation and perfect symmetry with exactly 2[b] quantization levels, but the step size
is not uniform, leading to a completely different quantizer function and QAT methods. Since NSQ
is non-uniform quantization, we do not consider it for the scope of this paper. Centered Symmetric
**Quantization (CSQ) stipulates uniform step size and perfect symmetry between the positive and**
negative sides, while compromising on the exact representation of zero. It can also be represented
using integers only by scaling all quantization levels by 2 as shown in the last row of the table. For
the remainder of the paper we focus on CSQ and CLQ, as they are the most practical.

3.3 QUANTIZER FUNCTION FOR CSQ

We define the quantizer for CSQ as follows:

_v_
_v˙ =_ 0.5 (4)

_s_ [+ 0][.][5] _−_

_v¯ = clip ( ˙j_ _v, −mQ, Q)_ (5)
_vˆ = ¯v × s_ (6)

where v is any input value, s is the step size, and Q = 2[b][−][1] _−_ 0.5 with b being the quantization precision (i.e., the number of bits). As usual, ⌊·⌉ is the round operation, and clip(x, a, b) =
min(max(x, a), b). Here ¯v is not an integer. Nevertheless, it is an exact value that can be represented by the b-bit CSQ format. Moreover, our proposed CSQ format permits efficient hardware
and software realizations (see Section 5). Therefore ¯v represents the value that is computed by b-bit
hardware. Finally, ˆv is the scaled-back version of ¯v, defined and used for the purpose of training.
The proposed quantizer provides equal representation for the positive and negative sides of the input
distribution. Figure 1a shows the 2-bit quantizer functions for conventional linear quantization and
our CSQ. Figure 1b shares the gradient for step size which is a quantization parameter. Detailed
training method is described in Appendix A.


-----

(a) Quantization of 2-bit signed data using
conventional linear quantization (CLQ) vs.
CSQ (ours).


(b) The gradient of the CSQ quantization
output w.r.t. step size.


Figure 1: CSQ quantizer and gradient.

Table 2: List of values in the product term (2-bit example, unsigned activation)

|Col1|W-A: CLQ -CLQ s u|W-A: CSQ-CLQ u|
|---|---|---|
|W (signed) A (unsigned)|{−2, −1, 0, 1} {0, 1, 2, 3}|{−1.5, −0.5, 0.5, 1.5} {0, 1, 2, 3}|
|W · A|{−6, −4, −3, −2, 1, 0, 1, 2, 3}|{−4.5, −3, −1.5, −1, −0.5, 0, 0.5, 1, 1.5, 3, 4.5}|


W-A: CLQs-CLQu W-A: CSQ-CLQu

W (signed) _{−2, −1, 0, 1}_ _{−1.5, −0.5, 0.5, 1.5}_

A (unsigned) _{0, 1, 2, 3}_ _{0, 1, 2, 3}_

_W · A_ _{−6, −4, −3, −2, 1, 0, 1, 2, 3}_ _{−4.5, −3, −1.5, −1, −0.5, 0, 0.5, 1, 1.5, 3, 4.5}_


4 ANALYSIS OF CSQ

4.1 IMPROVED REPRESENTATIONAL CAPACITY OF CSQ

Even though both CSQ and CLQ have the same number of quantization levels, and therefore the
same representational capacity on the operand level, multiplication result, or the product of weight
and activation, may have different representational capacity depending on the choice of quantizer.
We compare the representational capacity of the product of weight and activation, when using CSQ
vs. CLQ for weight. For activation quantization we consider unsigned CLQ (denoted by CLQu) for
unsigned activation, and either CSQ or signed CLQ (denoted by CLQs) for signed activation. We
start by defining the range of quantization levels of each scheme:

CLQu 0, 1, _, 2[b]_ 1 (7)
_∼{_ _· · ·_ _−_ _}_

CLQs 2[b][−][1], 2[b][−][1] + 1, _, 2[b][−][1]_ 1 (8)
_∼{−_ _−_ _· · ·_ _−_ _}_

CSQ ∼{−2[b][−][1] + 0.5, −2[b][−][1] + 1.5, · · ·, 2[b][−][1] _−_ 0.5} (9)
where b is the precision. It should be noted that CSQ can only be signed. Table 2 shows the range of
weights, activations and their product for 2-bit precision using different quantization methods.

Table 3 shows that using CSQ for weight quantization almost always increases the representational
capacity of quantized multiplication. The only exception is 2-bit precision for signed activation
where it remains the same. Given that improving representational capacity is crucial for increasing
the overall performance of QNNs (Liu et al., 2018), our analysis suggests that CSQ has a definite
advantage over CLQ for quantizing weights. Furthermore, the distribution of product resembles
non-uniform quantization which provides higher accuracy. For more details refer to Appendix C.

In the case of signed activation, our result in Table 3 suggests that using CLQs for activation and
CSQ for weight provides the highest representational capacity.

4.2 CSQ AND CLQ AS DIFFERENT ZERO-POINTS OF AFFINE QUANTIZATION

In this section we analyze and compare CSQ and CLQ using a more general framework of affine
quantization, which is defined as follows (Krishnamoorthi, 2018):

_x_
_xint =_ + z, _x¯ = clip(xint, 0, 2[b]_ 1), _xˆ = s(¯x_ _z)_ (10)

_s_ _−_ _−_

j m


-----

Table 3: Number of unique values in the product term

|#bits|Unsigned Activation (W-A) CLQ -CLQ CSQ-CLQ s u u|Signed Activation (W-A) CLQ -CLQ CSQ-CLQ CSQ-CSQ s s s|
|---|---|---|
|2-bit 3-bit 4-bit|9 11 (+22.2%) 35 43 (+22.9%) 120 155 (+29.2%)|6 9 (+50.0%) 6 (0.0%) 18 31 (+72.2%) 20 (+11.0%) 60 105 (+75.0%) 66 (+10.0%)|


Unsigned Activation (W-A) Signed Activation (W-A)

#bits CLQs-CLQu CSQ-CLQu CLQs-CLQs CSQ-CLQs CSQ-CSQ

2-bit 9 11 (+22.2%) 6 9 (+50.0%) 6 (0.0%)

3-bit 35 43 (+22.9%) 18 31 (+72.2%) 20 (+11.0%)

4-bit 120 155 (+29.2%) 60 105 (+75.0%) 66 (+10.0%)


0 1 ½ 2 3

|Col1|Col2|Col3|x|Col5|Col6|
|---|---|---|---|---|---|
||o o|||ox|o|


(= zCSQ) (= zCLQ)

Figure 2: Zero-point for CLQ and CSQ at 2-bit precision. Affine quantizer allows integer zero-point
only (shown in green circles). In relaxed affine quantizer, zero-point can take any real value.

where z is called zero-point, and must be an integer since the other terms on both sides are all integers. However if we relax the zero-point to be any real value, CLQ and CSQ can both be considered
as a special case of (relaxed) affine quantizer. Then the zero-point for CLQ (zCLQ) and CSQ (zCSQ)
can be defined as follows:
_zCLQ = 2[b][−][1],_ _zCSQ = 2[b][−][1]_ _−_ 0.5 (11)

_x_
For instance, at b = 2, the term _s_ has the range of 2, 1, 0, 1 in (signed) CLQ. Thus to satisfy

_{−_ _−_ _}_
(10), we have zCLQ = 2.

 

We can consider four cases depending on the restriction on zero-point (ZP): (i) ZP can take any real
value (i.e., relaxed affine quantizer), (ii) ZP can take any integer value (i.e., affine quantizer), (iii)
ZP is fixed to zCLQ, and (iv) ZP is fixed to zCSQ. Among these, CLQ is a special case of integer
zero-point (affine quantizer) while CSQ is not (see Figure 2). At the same time, there is a hardware
overhead for (i) and (ii), though integer zero-point can be implemented more efficiently than realvalue zero-point. We show in Section 5 that CSQ can be implemented as efficiently as CLQ.

The above view provides us with a methodology to somewhat objectively compare CSQ and CLQ
by the distance of the optimal real-value ZP (z[∗]) to zCSQ vs. zCLQ; of which we present our results
in Section 6.5. Also, it follows from (11) that the difference between zCLQ and zCSQ, relative to the
entire quantization range, diminishes exponentially as b increases (see Appendix H), suggesting that
the performance difference between CLQ and CSQ will be negligible for large b.

5 EFFICIENT REALIZATION OF CSQ

The high efficiency of BNN hardware comes from the bit-parallel processing (i.e., bitwise XNOR)
of large arrays, which can be extended to efficient processing of very low-precision QNNs by interchanging the order of summation between precision and vector dimensions as observed by Zhou
et al. (2016).

The challenge is how to extend the bitwise-operation based inner-product computation method to
CSQ. Since there are multiple combinations, here we show the result for the case of a CLQu vector
**x and a CSQ vector v, each of which is N** -dimensional and n-bit.

An n-bit binary number an−1an−2...a0 (n ≥ 2) represents the following value: Ab = _i=0_ _[a][i][2][i][.]_
We define the CSQ number format as exemplified in Table 4, which leads to the following equalities
(the second equality is not obvious, but is correct).

[P][n][−][1]

_n−1_ _n−1_

_Acsq =_ _ai2[i]_ _−_ (2[n] _−_ 1)/2 = (−1)[a][i][+1]2[i][−][1] (12)

_i=0_ _i=0_

X X

To avoid dealing with fractional numbers, let us use the 2× scaled version of CSQ (see Table 1).
Then,

_n−1_

_Acsq2 =_ (−1)[a][i][+1]2[i], (13)

_i=0_

X


-----

Table 4: Comparison of number representations: CLQ vs. CSQ (2-bit example)

|2-bit binary|CLQ s|CLQ u|CSQ|
|---|---|---|---|
|00 01 10 11|0 1 2 − 1 −|0 1 2 3|1.5 − 0.5 − 0.5 1.5|


2-bit binary CLQs CLQu CSQ

00 0 0 _−1.5_

01 1 1 _−0.5_

10 _−2_ 2 0.5

11 _−1_ 3 1.5


which has a very similar mathematical structure as a CLQ number, allowing us to use the same trick
of changing the order of precision (n) and vector dimension (N ) as in the inner-product computation
of two CLQ numbers. Finally, we arrive at the following inner-product computation method (for
more detail, see Appendix I):

**v · x = (vH · xH << 2) + (vH · xL << 1) + (vL · xH << 1) + vL · xL** (14)

which is shown for the 2-bit case (n = 2). xH and xL (similarly for vH and vL) are the N - 
dimensional bit-vectors of x containing only the higher and lower bits only, respectively, and <<
is the bitwise shift-left operation. Each product on the right-hand side of (14) can be computed on
BNN hardware in a single cycle. Thus v · x can be computed in four cycles, using an additional
adder/accumulator. It is worth mentioning that the same method as illustrated in (14) is also used
when computing the inner-product of two CLQ vectors.

Now for the inner-product of two bit-vectors (e.g. vH **xH** ) we can use the same structure of a
bitwise operation followed by popcount, with a slight variation. ·

**vcsq2 · xclqu = 2 · popcount(AND(vcsq2, xclqu** )) − popcount(xclqu ) (15)

ements of the CLQ bit-vector must be ignored. Note that BNN hardware,We subtract popcount(xclqu), since the bits of the CSQ bit-vector corresponding to the zero el- e.g. (Umuroglu et al.,
2017), relies on the same method for the inner-product computation of two bit-vectors, except that
we use AND instead of XNOR and to subtract a popcount value, we need an additional popcount
operation. However, accelerating QNNs with the CLQ format also requires AND operations in the
exact same manner, and the additional popcount operation does not increase the hardware cost. Thus
the bit-vector-level complexity of using CSQ is nearly the same as that of using CLQ. Since there
is no distinction at the vector level, the inner-product operation with CSQ can be implemented as
efficiently as with CLQ.

6 EXPERIMENTS

6.1 EXPERIMENTAL SETUP

We conduct experiments on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al.,
2015) using ResNet-18, ResNet-20, ResNet-34 (He et al., 2016) and MobileNet-v2 (Sandler et al.,
2018). All quantized models on ImageNet are initialized with the weights of pretrained full-precision
model of the same network. The first and last layers are kept at 8-bit precision. Other than convolution and fully connected layers, all the other layers, e.g. batch norm, are kept in full precision.

For ImageNet experiments we use the training recipe of Esser et al. (2019). We use stochastic gradient descent (SGD) optimizer, with 0.9 momentum, cosine learning rate decay (Loshchilov & Hutter,
2016) without restarts, and the initial learning rate of 0.01. Weight decay is 0.25 × 10[−][4] for 2bit, 0.5 × 10[−][4] for 3-bit, and 10[−][4] for 4-bit quantization. The quantized models are fine-tuned for
90 epochs. For ResNet-18 ImageNet experiments, we train the full-precision model ourselves from
scratch, using the same experimental setup except 0.1 initial learning rate and 10[−][4] weight decay.

For CIFAR-10 experiments the weights for quantized models are trained from scratch. We use the
same setting as with ImageNet except the following: the initial learning rate is 0.1, weight decay is
10[−][4], and each model is trained for 300 epochs. All experiments on CIFAR-10 are conducted using
ResNet-20 network. All ImageNet and CIFAR-10 models were implemented in PyTorch.


-----

Table 5: Comparison of CLQ and CSQ (ours) on ResNet-20 for CIFAR-10.


Top-1 Accuracy @ Precision

**Network** ResNet-20
_Full Precision: 91.22_

**Precision (W/A)** 2/2 3/3 4/4

CLQ (LSQ) 89.09±0.11 90.70±0.18 **91.07±0.13**
CSQ (ours) **89.46±0.225** **90.80±0.12** 90.96±0.25

Table 6: Comparison of CLQ, CSQ and other quantization-aware training methods on ImageNet.
The CLQ case also represents LSQ (Esser et al., 2019). MobileNet-v2 2-bit case did not converge.


Top-1 Accuracy @ Precision

**Network** ResNet-18 ResNet-34 MobileNet-v2
_Full Precision: 70.58_ _Full Precision: 73.31_ _Full Precision: 71.88_

**Precision (W/A)** 2/2 3/3 4/4 2/2 3/3 4/4 2/2 3/3 4/4

PACT 64.40 68.10 69.20 -  -  -  -  -  - 
LQ-Nets 64.90 68.20 69.30 69.80 71.90 -  -  -  - 
QIL 65.70 69.20 70.10 70.60 73.10 73.70 -  -  - 
CLQ (LSQ) 66.59 69.38 70.52 70.56 73.21 73.82 -  60.41 66.82
LSQ+ 66.80 69.40 **70.80** -  -  -  -  -  - 
CSQ (ours) **66.92** **69.48** 70.63 **70.82** **73.29** **74.01** -  **60.89** **66.98**

6.2 CIFAR-10 RESULT

We compare our method with CLQ using a state-of-the-art QAT method by Esser et al. (2019). Each
case is repeated five times; mean ± std. dev. is reported for each case. The results are summarized in
Table 5, which confirms that our proposed method performs significantly better than CLQ at 2-bit,
while at 3- and 4-bit, there is little difference in performance between CLQ and CSQ.

6.3 IMAGENET RESULT

We compare CLQ vs. CSQ on ImageNet classification using the standard models such as ResNet and
MobileNet. We also compare our method with PACT (Choi et al., 2018b), LQ-Nets (Zhang et al.,
2018), QIL (Jung et al., 2019) and LSQ+ (Bhalgat et al., 2020). Since we use the training method
by LSQ (Esser et al., 2019), the CLQ case also represents the previous work (LSQ). However,
LSQ use pre-activation ResNet (He et al., 2016) which has higher performance than the standard
ResNet architecture, and their trained models are not available. Therefore for a fair comparison, we
have implemented LSQ ourselves, and use it as the baseline. For CSQ we use the same model and
replace only the weight quantizer with CSQ, except in the first and last layers which are quantized
to 8-bit CLQ. LSQ+ (Bhalgat et al., 2020) uses affine quantizer for activation that allows floatingpoint zero-point. However, ResNet and MobileNet use ReLU activation which results in unsigned
activation which can not fully utilize the capability of affine quantizer. Therefore, we do not use
affine quantizer for activations in our experiments. For comparison with LSQ+ we use their reported
results. Lee et al. (2021) is another recent work that presents a training method for QNNs, that
outperforms LSQ. However, they already use a quantizer than results in zero-centered quantization
levels, similar to the one proposed in this paper. Therefore, we do not compare our results with them.

The results are summarized in Table 6 which shows that CSQ outperforms CLQ for all cases. We can
observe that the performance gain by our method diminishes as precision increases. At 2- and 3-bit
precision CSQ outperforms LSQ+ as well. We attribute this gain in performance to the improved
representational capacity of CSQ, discussed in Section 4.1, and reduced quantization error shown in
Section 6.4.

6.4 QUANTIZATION ERROR USING LSQ

We study the quantization error using CLQ and CSQ. We use full precision weights trained on
ImageNet, which are also used to initialize weights for quantization-aware training. We compare


-----

20

10

0

ERROR REDUCTION BY CSQ 10

0 5 10 15

LAYERS

(a) 2-bit precision


15

10

5

0

5

ERROR REDUCTION BY CSQ

10

0 5 10 15

LAYERS

(b) 3-bit precision


8

6

4

2

0

2

ERROR REDUCTION BY CSQ 4

6

0 5 10 15

LAYERS

(c) 4-bit precision


Figure 3: Percentage error reduction using CSQ. The results are shown for ResNet-18 trained on
ImageNet.

25

20

2 2 15

Frequency Frequency Frequency

10


1.2 1.4 1.6 1.8 2.0

|Col1|CLQ CSQ Affine Zero Point|CLQ CSQ Affine Zero Point|
|---|---|---|


CLQ
CSQ
Affine Zero Point

Zero Points

(a) 2-bit precision


3.4 3.6 3.8 4.0

|Col1|CLQ CSQ Affine Zero Point|CLQ CSQ Affine Zero Point|
|---|---|---|


CLQ
CSQ
Affine Zero Point

Zero Points

(b) 3-bit precision


7.5 7.6 7.7 7.8 7.9 8.0

|Col1|CLQ CSQ Affine Zero Point|CLQ CSQ Affine Zero Point|
|---|---|---|


CLQ
CSQ
Affine Zero Point

Zero Points

(c) 4-bit precision


Figure 4: Distribution of optimal zero-point values of relaxed affine quantizer, across layers. The
graphs also show the zero-points for CSQ and CLQ, represented as vertical lines.

mean square error (w _wq)[2]_ between CLQ and CSQ, where w is the full precision weight and wq
_⟨_ _−_ _⟩_
is the quantized weight. For each experiment we find a step size (s) that minimizes each error metric,
by an exhaustive search. We report percentage error reduction by CSQ for each layer. which we
define as: error reduction(%) = [(][E][CLQ]ECLQ[−][E][CSQ] [)] 100, where ECLQ and ECSQ are the quantization

_×_
errors using CLQ and CSQ, respectively. The results are presented in Figure 3.

It can be seen that CSQ reduces the overall quantization error compared to CLQ. At 2-bit precision
CSQ provides a clear advantage over CLQ. At 3-bit precision CSQ is still significantly better than
CLQ but the improvement in quantization error using CSQ is less comapred to 2-bit. Finally at 4-bit
precision the improvement using CSQ seems much smaller than 2 and 3-bit precision. Additionally
CLQ provides significantly better results in some layers compared to CSQ. Based on these results
we can conclude that CSQ provides significant advanatge at 2-bit precision, which diminishes as we
increase the precision.


6.5 ANALYZING CSQ USING AFFINE QUANTIZER

To analyze the effectiveness of CSQ, we train ResNet-20 for 2, 3 and 4-bit precision on CIFAR10, using affine quantization defined in (10). The latent weights for quantized model, are initialized
using pre-trained full precision weights. We initialize and train the step size using LSQ (Esser et al.,
2019), similar to experiments in Section 6.2 and Section 6.3. Zero-point is relaxed to be a real value
and initialized to zCLQ from (11). We choose the initialization biased towards CLQ to show that
zero-point converges closer to CSQ irrespective of initialization.

The results are shown in Figure 4. It can be seen that despite our initialization being biased towards
CLQ, the zero-point distribution is always closer to CSQ than CLQ. Considering that the zero point
distribution moves away from CSQ as the precision increases, the results suggest that there is an
inverse relation between the effectiveness of CSQ and precision of the network, which is also consistent with our analytical result in Section 4.2, that CSQ is most effective at low precision (i.e.,
2-bit) and loses effectiveness as we increase precision. The result in Figure 4 also indicates that
CSQ provides significantly better approximation of affine quantization than CLQ.


-----

Table 7: Matrix multiplication runtime (size: 16384×16384) on Nvidia RTX 2080 TI

Kernel Runtime (ms) Relative Speed

cuBLAS 579.81 ± 9.16 1
1-bit 85.44 ± 11.99 6.8
2-bit CSQ 197.85 ± 6.82 2.9
2-bit CLQ 197.85 ± 10.20 2.9

6.6 GPU IMPLEMENTATION RESULTS

To see the speedup on GPU, we have implemented three custom kernels that use concatenation and
bitwise operations to compute matrix multiplication (see Appendix I.2). We compare these custom
kernels with cuBLAS. The first kernel is 1-bit CSQ, which is the same as 1-bit BNN. The second
and third kernels are 2-bit, each using CSQ and CLQ.

The result is summarized in Table 7, which is the average of five runs. We observe exactly the same
speed between CSQ and CLQ, which demonstrates that our CSQ is as efficient as the conventional
binary representation on GPU. Note that apart from the main computation kernel, our CSQ is exactly
the same as CLQ (such as memory requirement) and therefore has the same overall runtime as CLQ.
Also our 2-bit CLQ is 2.9 times faster than cuBLAS while being only 2.3 times slower than 1-bit.

6.7 DISCUSSION

We can see from experimental results and analyses that CSQ provides significant advantage at 2-bit
precision. It is also evident that CSQ loses significance as we increase precision. The question is,
**“what is the exact precision where CSQ loses its superiority to CLQ ?”. Experimental results and**
quantization error experiments in Section 6.4 demonstrate that superiority of CSQ is significantly
reduced at 4-bit precision. However representational capacity analysis in Section 4.1 and affine
quantization experiments in Section 6.5 suggest that CSQ provides significant advantage at 4-bit
precision as well.

We know that 4-bit precision does not suffer from performance degradation using QuantizationAware Training (QAT). In fact with current state-of-the-art 4-bit precision achieves similar or even
higher accuracy than full-precision networks for ResNet. This can also be seen in our experimental
results in Table 6. In other words, for ResNet-18 4-bit precision can provide more than sufficient
representation irrespective of quantization method and lack of representation capacity does not remain a bottleneck in performance. That is why at 4-bit precision we observe very similar results
whether we use CSQ or CLQ. Based on this we conclude that CSQ achieves its limit at 4-bit precision for ResNet and beyond 4-bit precision CLQ and CSQ should perform similarly. While the exact
precision threshold cannot be generalized to other networks or datasets (e.g. CSQ outperforms CLQ
on ResNet-18 and ResNet-34 on ImageNet but does not provide the same superiority on ResNet-20
on CIFAR-10), the criteria to find the threshold are generalizable; that is, CSQ is superior while
representation capacity is the bottleneck.

Finally, though we have used LSQ only (Esser et al., 2019) as the quantization method in our experiments, our quantizer can be applied with any quantization method. In Appendix D we present our
PTQ (post-training quantization) result using another state-of-the-art quantization method (Li et al.,
2021).

7 CONCLUSION

In this paper we provided an in-depth analysis of CSQ for extreme low-bit quantization, which is
both completely symmetric around zero and trainable using existing linear quantization methods.
Our analyses and experimental results using state-of-the-art quantization methods with CIFAR-10
and ImageNet datasets show that a simple change of quantization levels can result in significant
performance improvement for extremely low bit quantized neural networks (≤ 4 bits). We also show
that CSQ can be realized efficiently on BNN hardware and GPUs. Considering there are very few
previous works targeting 2-bit network performance, CSQ can be a very useful tool for optimizing
extreme low-precision neural networks for deployment.


-----

REFERENCES

Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving
low-bit quantization through learnable offsets and better initialization. In Proceedings of the
_IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 696–697,_
2020.

Yoonho Boo, Sungho Shin, Jungwook Choi, and Wonyong Sung. Stochastic precision ensemble:
self-knowledge distillation for quantized deep neural networks. In Proceedings of the AAAI Con_ference on Artificial Intelligence, volume 35, pp. 6794–6802, 2021._

Peng Chen, Jing Liu, Bohan Zhuang, Mingkui Tan, and Chunhua Shen. Aqd: Towards accurate
quantized object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and
_Pattern Recognition, pp. 104–113, 2021._

Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint arXiv:1807.06964, 2018a.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural
networks. arXiv preprint arXiv:1805.06085, 2018b.

Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. In International Conference on Learning Repre_sentations, 2019._

Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and
Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852–4861,
2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
770–778, 2016.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
_preprint arXiv:1503.02531, 2015._

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized
neural networks. In Proceedings of the 30th International Conference on Neural Information
_Processing Systems, pp. 4114–4122, 2016._

Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju
Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization
intervals with task loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern
_Recognition, pp. 4350–4359, 2019._

Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A
whitepaper. arXiv preprint arXiv:1806.08342, 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.

E. H. Lee, D. Miyashita, E. Chai, B. Murmann, and S. S. Wong. Lognet: Energy-efficient neural networks using logarithmic computation. In 2017 IEEE International Conference on Acous_tics, Speech and Signal Processing (ICASSP), pp. 5900–5904, 2017. doi: 10.1109/ICASSP.2017._
7953288.

Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradient
scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni_tion, pp. 6448–6457, 2021._


-----

Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient nonuniform discretization for neural networks. In International Conference on Learning Representa_tions, 2019._

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi
Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint
_arXiv:2102.05426, 2021._

Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net:
Enhancing the performance of 1-bit cnns with improved representational capability and advanced
training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp.
722–737, 2018.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
_preprint arXiv:1608.03983, 2016._

Joel Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1):7–12,
1960.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or
down? adaptive rounding for post-training quantization. In International Conference on Machine
_Learning, pp. 7197–7206. PMLR, 2020._

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,
Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet. International journal of
_computer vision, 115(3):211–252, 2015._

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on
_computer vision and pattern recognition, pp. 4510–4520, 2018._

Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus
Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network inference.
In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate
_Arrays, pp. 65–74, 2017._

Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for
highly accurate and compact deep neural networks. In Proceedings of the European conference
_on computer vision (ECCV), pp. 365–382, 2018._

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. _arXiv preprint_
_arXiv:1606.06160, 2016._


-----

A OUR TRAINING METHOD

We have employed the method presented in Esser et al. (2019) to train the quantization parameters.
To optimize the step size s using gradient descent, we use the following derivative formula.

_v/s + ˙v_ if _Q < ˙v < Q_

_∂vˆ_ _−_ _−_

_Q_ if ˙v _Q_ (16)

_∂s_ [=] − _≤−_

Q if ˙v _Q_

_≥_

Then computing the loss gradient w.r.t. step size is straightforward. Similar to gradient scaling in



(Esser et al., 2019), the gradient of step size is scaled by factor g = 1/[√]NW 2[p], where NW is the
number of weight parameters. The weights are initialized as 2⟨|v|⟩/[√]Q, where ⟨.⟩ represents the
notation for mean of a distribution. Figure 1b shares the gradient of step size parameter at 2-bit
precision. Our training method is based on LSQ (Esser et al., 2019) but it should be noted that CSQ
can be used with any training method.


LIMITAIONS OF AFFINE QUANTIZER


We have defined affine quantizer by Krishnamoorthi (2018) in (10). Another variant of affine quantizer has also been proposed by Bhalgat et al. (2020):

_x_ _z_
_x¯ =_ clip _−_ _, n, p_ _,_ _xˆ = s_ ¯x + z (17)

_s_ _·_

  

where n and p are clipping intervals. Bhalgat et al. (2020) does not restrict their zero-point to integer
values. This approach can lead to significant hardware overhead. To tackle this problem they propose
the affine quantizer for activations only, in which case zero-point can be implemented simply as a
bias term, since

_wˆ · ˆx = ( ¯w · sw) (¯x · sx + z) = ¯wxs¯_ _wsx + zsw ¯w_ (18)
bias

and ¯w is known in advance. This resolves the hardware overhead problem of affine quantization

| {z }

for activation. However, the same method cannot be applied to weight because it would involve
pre-computing ¯x in advance, which is impossible.

Affine quantization for weights still results in some hardware overhead as the the zero-point of
weights quantizer can not be implemented as the bias. This can be shown as:

_wˆxˆ = ( ¯w × sw + zw) (¯x × sx + zx) = ¯wxs¯_ _wsx + zxsw ¯w + zwzx_ + zwsxx¯ (19)
bias overhead

This is because ¯x can not be pre-computed as it will change with each input. Similarly weight

| {z } | {z }

quantization overhead is shown by (Krishnamoorthi, 2018) as:

_y(k, l, n) = swsx conv ( ¯w(k, l, m; n) −_ _zw, ¯x(k, l, m) −_ _zx)_ (20)

_K−1_ _K−1_ _N_ _−1_

_y(k, l, n) = conv ( ¯w(k, l, m; n), ¯x(k, l, m))_ _zw_ _x¯(k, l, m)_ (21)
_−_

_k=0_ _l=0_ _m=0_

X X X


_K−1_

_k=0_

X


_K−1_

_l=0_

X


_N_ _−1_

_w¯(k, l, m; n) + zxzw_ (22)
_m=0_

X


_zx_
_−_


Considering these limitations of affine quantizer we have proposed CSQ as an approximation of
affine quantizer which can be efficiently realized on hardware.

C MORE ON REPRESENTATIONAL CAPACITY

In Section 4.1 we have shown that CSQ improves the representational capacity compared to CLQ.
We have considered using signed, as well as unsigned activations. The most common activation


-----

(a) 2-bit precision (b) 3-bit precision (c) 4-bit precision

Figure 5: Distribution of quantized product values when using CSQ vs. CLQs for weight, and CLQu
for unsigned activation.


(a) 2-bit precision with signed activations


(b) 3-bit precision with signed activations


(c) 4-bit precision with signed activations


Figure 6: Representational capacity of quantized product with signed activations, using different
quantization methods for 2, 3 and 4-bit precision.

function is ReLU, which produces unsigned activations. In case of signed activations such as PReLU
and h-swish, affine quantization (Krishnamoorthi, 2018; Bhalgat et al., 2020) provides the best representation. However, affine quantizer can also be represented by unsigned CLQ before applying
zero-points. Therefore unsigned CLQ for activations encapsulates affine quantizer as well. We also
compare the representational capacity of quantized operation using symmetric activation range. Such
an activation is unusual as the activation function introduces some non-linearity. However, it is possible in some cases e.g output of Tanh activation. In this case the activation is quantized using signed
CLQ or CSQ. We show the normalized distribution of product with unsigned activations in Figure 5.
It can be observed that the distribution of product values when using CSQ for weight is perfectly
symmetric, and densely populated in the middle of the output range while being sparse near the
boundary. This bell-shaped distribution better resembles layer activations, which is often exploited
by non-uniform quantization schemes (Li et al., 2019; Lee et al., 2017) to achieve higher quantization performance. On the other hand, the product values when using CLQ for weight, are more
uniformly distributed for the entire output range, with an exceptional outlier in the negative range
and no matching positive outlier, introducing asymmetry to the output as well.

Figure 6 shows the normalized distribution of product with signed activation. It can be seen that using
CSQ for weights and CLQ for activations provides significantly higher representational capacity
compared to using CLQ for weights or CSQ for activations.

D POST TRAINING QUANTIZATION RESULTS (COMPARISON WITH BRECQ)

BRECQ (Li et al., 2021) uses affine quantizer with integer zero-point, which extends the linear
quantizer in a different direction than our CSQ. Thus it is very interesting to see how our method
compares with BRECQ. Note that integer zero-point requires some additional hardware overhead
whereas CSQ does not. At the same time, the two schemes are orthogonal in the sense that one
can combine both schemes, viz. integer zero-point and CSQ, so that zero-point take any integer or
half-integer value, at the cost of some hardware overhead. Here we compare only CSQ vs. BRECQ,


-----

Table 8: Comparison of CSQ with affine quantizer using BRECQ on ImageNet.


Accuracy

**Network** **Method** 2/32 3/32 4/32

ResNet-18 _Full Precision: 71.08_
BRECQ 66.60 **69.82** **70.64**
BRECQ-CSQ (ours) **66.93** 69.81 _70.62_

ResNet-50 _Full Precision: 77.00_
BRECQ 72.16 **75.68** **76.41**
BRECQ-CSQ (ours) **72.24** 75.52 76.35

but not the combination. Since we have proposed CSQ for weight quantization, we only quantize
the weights in BRECQ experiments.

To implement CSQ we simply fix the zero point value to CSQ which is shown as ZCSQ in (11).
The shared results may be different from reported results. The results have been reproduced using
the officially shared code. We share our results on ResNet-18 and ResNet-34. We used channelwise quantization and all the hyper-parameters are same as shown in Li et al. (2021). Table 8 shows
the experimental results using BRECQ on ImageNet data. It can be seen that at 2-bit precision
CSQ outperforms affine quantizer. This is especially interesting because unlike affine quantization,
CSQ does not incur any hardware overhead. At 3 and 4-bit precision, affine quantizer gives better
performance. However, CSQ also provides competitive results. This shows that CSQ is a very strong
approximation of (relaxed) affine quantizer.

E ADDITIONAL EXPERIMENTS USING KNOWLEDGE DISTILLATION

To further demonstrate the generalization-ability of CSQ, we conduct some experiments with knowledge distillation (Hinton et al., 2015) loss. We use LSQ (Esser et al., 2019) for training the quantization parameters and show that CSQ performs superior to CLQ at low precision even when we use
advanced training methods such as knowledge distillation over state-of-the-art quantization-aware
training method.

The experimental methodology and training setups are the same as described in Section 6.1, except that we use a weight decay of 0.25e-4 and knowledge distillation loss for all experiments. For
knowledge distillation loss we set temperature as 1 and give equal weight to the standard loss and
the distillation loss following (Esser et al., 2019). Our experimental results in Table 9 show that at
extremely low precision, i.e., at 2- and 3-bit, CSQ outperforms CLQ. This is consistent with experimental results in Table 6. We have already discussed in Section 6.7 why CSQ may lose its advantage
at 4-bit precision.

F WHY CSQ IS ALWAYS BETTER THAN CLQ AT 2-BIT

In the weight quantization error experiment of Section 6.4, optimizing for minimum quantization
error faces the challenge of maximizing the utility of limited quantization levels. The utility is max
### -1½ -½ ½ 1½

|Col1|P(0 ≤|
|---|---|


### P (0 ≤ X

 X


Figure 7: With 2-bit CSQ, quantization levels (shown on the x-axis) can resemble Gaussian distribution while, at the same time, real-valued weight data are also uniformly mapped to them (scale
parameter s is omitted for brevity).


-----

Table 9: Comparison of CSQ vs. CLQ using knowledge distillation training on ImageNet.


Top-1 Accuracy @ Precision

**Network** ResNet-18
_Full Precision: 70.52_

**Precision (W/A)** 2/2 3/3 4/4

CLQ (LSQ) + Knowledge Distillation 66.99 69.77 **70.63**
CSQ + Knowledge Distillation (ours) **67.24** **69.90** 70.56

(a) CLQ 2-bit (b) CSQ 2-bit

Figure 8: How real-valued weight data are mapped to CLQ vs. CSQ after training ResNet-18 on
ImageNet.

imized (i) if each quantization level has equal number of real values mapped to it, in the same way
as the Lloyd-Max quantization (Max, 1960) is optimal. Also it can be helped a lot (ii) if the quantization levels have the same distribution as the underlying data. In other words, for the first objective
the underlying data (when mapped to quantization levels) should be distributed as uniformly as possible, while for the second objective the distribution of quantization levels should resemble that of
the underlying data (e.g., Gaussian). This usually creates conflicting requirements, but not in the
case of 2-bit CSQ. At 2-bit, CSQ has only three quantization thresholds, {−1, 0, 1}, and therefore
can satisfy both requirements simultaneously: (i) real-valued weight data are uniformly distributed
across quantization levels, and at the same time, (ii) quantization levels follow Gaussian (or any
symmetric) distribution, as illustrated in Figure 7. Note that this is not possible for 2-bit CLQ, as in
other precision for either CLQ or CSQ, which explains why CSQ always shows better performance
than CLQ at 2-bit. From the figure, the scale parameter can be determined as s = Φ[−][1](0.75), where
Φ is the CDF of standard normal distribution, since P (0 ≤ _X < s) = 0.25 when X ∼_ _N_ (0, 1)).
Also our experimental results confirm that indeed real-valued weight data are mapped uniformly to
2-bit CSQ quantization levels as shown in Figure 8.

G EXHAUSTIVE SEARCH METHOD FOR QUANTIZATION ERROR
EXPERIMENTS

We present quantization error experiments in Section 6.4. To find the step size that minimizes the
Quantization Error we use an exhaustive search method. Our exhaustive search method is very similar to the method used by Esser et al. (2019) for their quantization error experiments. The exhaustive
search goes as follows. First, we initialize step sizes as:

_s0 =_ (23)

2[⟨|][p][w][|⟩]1
_−_

where w represents full precision weights,⟨.⟩ represents the mean operation, and p is the bit-width.
Then for the search space S = 0.01s0, 0.02s0, 0.03s0, · · ·, 5s0, we exhaustively find the value
_{_ _}_
of s ∈ _S that minimizes the target quantization error metric. This helps us find the minimum quan-_


-----

tization error using CLQ and CSQ, for any given bit-width. Experimental results and analysis has
been presented in Section 6.4. https://www.overleaf.com/project/608a0e19c95387366f0de2f6

H MORE ABOUT ZERO-POINT

The difference between zCLQ and zCSQ is constant at 0.5 (see (11)) while the possible range of values that can be taken by the quantizer grows exponentially with b. Thus we can give the percentage
difference between zero-points of CLQ vs. CSQ relative to the entire quantization range by:

1

_D =_ _[z][CLQ][ −]_ _[z][CSQ]_ 100 = (24)

2[b] _×_ 2[b][+1][ ×][ 100]

This equation implies that D decreases as we increase the precision. For example, at 2-bit precision,
_D is 12.5% but at 4-bit precision, D reduces to mere 3.125%. This shows that at higher precision_
the difference between CSQ and CLQ becomes negligible compared to the entire distribution range
and they should provide similar performance. This is consistent with our experimental results in
Section 6.

H.1 ZERO REPRESENTATION

CSQ does not provide an exact representation for zero. Instead, zero or values slightly less than zero
are rounded to −0.5 while values slightly greater than zero are rounded to 0.5 (before scale factor).
Despite this it has been shown in Section 6.4 that CSQ can reduce quantization error compared with
CLQ. Furthermore, our experimental results have also shown superior performance with CSQ. This
serves as an evidence that exact zero representation is not critical for weight quantization, especially
at low precision (< 4 bits).

H.2 ZERO PADDING

Zero padding is needed for activation only, not for weight. In case, if zero padding were used for
weight as well, we can simulate the exact effect of rounding zeros during QAT, and adjust weight
accordingly. So it would not contribute to any performance degradation.

Now, vector and tensor processors (e.g. TPU) must process an array of values together, and may “fill”
some elements with zeros as needed. This zero filling is needed for both weight and activation. For
CSQ weight, inexact zero representation may introduce an error or discrepancy between algorithm
and realization. This error can be eliminated or minimized, depending on hardware dataflow, by
resetting corresponding activation values to zeroes or filling with both +0.5 and −0.5.

H.3 WHY PROPOSE CSQ FOR WEIGHT ONLY

CSQ can be used for signed activation. However, activation frequently involves zero padding. Therefore, activation quantization strictly demands exact zero representation to support zero padding.
Furthermore Section 4.1 shows that when using CSQ for weight, using CLQs for activation is much
better in terms of representational capacity than using CSQ for both weight and activation. To summarize, using CLQ for activation is not only important to support zero padding, but also for best
performance. Thus we recommend that using CSQ for weight quantization and CLQ for activation
quantization, which provides the best performance with no hardware overhead.

I EFFICIENT HARDWARE REALIZATION OF CSQ

I.1 DIGITAL HARDWARE IMPLEMENTATION

CSQ can be efficiently implemented in hardware and software. To avoid using fractional numbers, in
this section we assume the 2× scaled version of CSQ (see Table 1). We can think of CSQ as an n-bit
extension of BNN encoding. We will show in the section below how we use the XNOR-popcount
based inner-product method of BNNs to realize the product of two CSQ vectors. Conventional BNN
hardware takes two 1-bit CSQ bit-vectors as an input. Assuming two N -element bit-vectors v and


-----

**x, the inner-product is calculated as follows:**

**vcsq · xcsq = 2 · popcount(XNOR(vcsq, xcsq)) −** _N_ (25)

In the case of the product between CSQ and unsigned CLQ, we should ignore the values of CSQ
when CLQu is 0. Therefore, we subtract the number of CSQ values after the popcount operation,
which is identical to popcount(xclqu ).

**vcsq · xclqu = 2 · popcount(AND(vcsq, xclqu** )) − popcount(xclqu ) (26)

We now extend (26) to the multi-bit case so that the inner-product between CSQ and unsigned CLQ
vectors can be performed efficiently. Consider a 2-bit CSQ vector and a 2-bit unsigned CLQ vector
of N elements.

**vcsq =** _a[N]1_ _[−][1]a[N]0_ _[−][1]_ _a[N]1_ _[−][2]a[N]0_ _[−][2]_ _..._ _a[0]1[a]0[0]_ _,_ **xclqu =** _b[N]1_ _[−][1]b[N]0_ _[−][1]_ _b[N]1_ _[−][2]b[N]0_ _[−][2]_ _..._ _b[0]1[b]0[0]_

(27)

   

Each vector can be split into two bit-vectors by dividing the lower bits and the higher bits of each
element.

**vHcsq =** _a[N]1_ _[−][1]_ _a[N]1_ _[−][2]_ _..._ _a[0]1_ _,_ **vLcsq =** _a[N]0_ _[−][1]_ _a[N]0_ _[−][2]_ _..._ _a[0]0_ (28)

**xHclqu =** _b[N]1_ _[−][1]_ _b[N]1_ _[−][2]_ _..._ _b[0]1_ _,_ **xLclqu =** _b[N]0_ _[−][1]_ _b[N]0_ _[−][2]_ _..._ _b[0]0_ (29)

The product of v and x can be computed as follows ( << is the bitwise shift-left operation): 

**v · x = (vH · xH << 2) + (vH · xL << 1) + (vL · xH << 1) + vL · xL** (30)

Each product on the right-hand side of (30) can be computed on the proposed hardware in a single
cycle. Thus v · x can be computed in four cycles, using an additional adder/accumulator.

I.2 FAST GPU IMPLEMENTATION

It is possible to speed up BNNs on GPU by using bitwise operations (Hubara et al., 2016), through a
custom GPU kernel to boost matrix multiplication on GPU. We have implemented a custom kernel
to compute the inner product of two (N -dimensional) vectors of n-bit CSQ numbers.

**a =** _a[N]n_ _[−]1[1][a]n[N]_ _[−]2[1]_ 0 _a[N]n_ _[−]1[2][a]n[N]_ _[−]2[2]_ 0 _a[0]n_ 1[a][0]n 2 0 (31)
_−_ _−_ _[· · ·][ a][N]_ _[−][1]_ _−_ _−_ _[· · ·][ a][N]_ _[−][2]_ _· · ·_ _−_ _−_ _[· · ·][ a][0]_

**b =** _b[N]n_ _[−]1[1][b]n[N]_ _[−]2[1]_ 0 _b[N]n_ _[−]1[2][b]n[N]_ _[−]2[2]_ 0 _b[0]n_ 1[b][0]n 2 0  (32)
_−_ _−_ _[· · ·][ b][N]_ _[−][1]_ _−_ _−_ _[· · ·][ b][N]_ _[−][2]_ _· · ·_ _−_ _−_ _[· · ·][ b][0]_

The kernel first concatenates the most significant bits into a single _N_ -bit number (if N > 32, we
break it to a multiple of 32). Repeating this for all n bits, each vector is converted to an n-element
array.
**ac =** _a[N]n_ _[−]1[1][a]n[N]_ _[−]1[2][...a]n[0]_ 1 _a[N]n_ _[−]2[1][a]n[N]_ _[−]2[2]_ _n_ 2 _a[N]0_ _[−][1]a[N]0_ _[−][2]_ _a[0]0_ (33)
_−_ _−_ _−_ _−_ _−_ _[· · ·][ a][0]_ _−_ _· · ·_ _· · ·_

**bc =**  _b[N]n_ _[−]1[1][b]n[N]_ _[−]1[2]_ _n_ 1 _b[N]n_ _[−]2[1][b][N]n_ _[−]2[2]_ _n_ 2 _b[N]0_ _[−][1]b[N]0_ _[−][2]_ _b[0]0_  (34)
_−_ _−_ _[· · ·][ b][0]_ _−_ _−_ _−_ _[· · ·][ b][0]_ _−_ _· · ·_ _· · ·_
 

Then we perform the XNOR-popcount operation n[2] times, accumulating its result.


_s_ _s +_ popcount(XNOR(ac[i], bc[j])) << (i + j) (35)
_←_ _{_ _}_


-----

