# SYNTHESISING AUDIO ADVERSARIAL EXAMPLES FOR AUTOMATIC SPEECH RECOGNITION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Adversarial examples in automatic speech recognition (ASR) are naturally
sounded by humans yet capable of fooling well trained ASR models to transcribe incorrectly. Existing audio adversarial examples are typically constructed
by adding constrained perturbations on benign audio inputs. Such attacks are
therefore generated with an audio dependent assumption. For the first time, we
propose the Speech Synthesising based Attack (SSA), a novel threat model that
constructs audio adversarial examples entirely from scratch, i.e., without depending on any existing audio) to fool cutting-edge ASR models. To this end, we introduce a conditional variational auto-encoder (CVAE) as the speech synthesiser.
Meanwhile, an adaptive sign gradient descent algorithm is proposed to solve the
adversarial audio synthesis task. Experiments on three datasets (i.e., Audio Mnist,
Common Voice, and Librispeech) show that our method could synthesise audio
adversarial examples that are naturally sounded but misleading the start-of-theart ASR models. The project webpage containing generated audio demos is at
[https://sites.google.com/view/ssa-asr/home.](https://sites.google.com/view/ssa-asr/home)

1 INTRODUCTION

Deep neural network (DNN) based models have documented many success stories in various domains, such as reinforcement learning (Silver et al., 2017), image classification (Deng et al., 2009),
and automatic speech recognition (ASR) (Chan et al., 2016). However, DNN models are found to be
vulnerable to adversarial attacks (Goodfellow et al., 2015), viz., the slightly perturbed input would
cause severe errors or performance drops of well trained DNN models. This paper mainly focuses
on the ASR domain, where many voice assistant systems could be hijacked or controlled by the
audio adversarial examples constructed by an attacker.

To investigate the threat of audio adversarial examples, many different approaches (Carlini & Wagner, 2018; Yuan et al., 2018; Yang et al., 2019; Qin et al., 2019) have been developed. In general,
existing approaches assume that the semantics (for human beings) of adversarial audio can be preserved as long as the distance between the adversarial audio and the benign audio is restricted properly. In particular, a shared design principle is to add constrained adversarial perturbations (i.e., as
imperceptible as possible) on benign audios yet with the goal of fooling ASR models significantly.
For instance, Qin et al. (2019) introduced a psychoacoustic rule of auditory masking to only add
perturbation on a benign audio where the noise is hard to be heard. Therefore, existing approaches
can be deemed as audio dependent attack (ADA), viz., the audio adversarial examples have to be
constructed depending on some benign audios as shown in Figure 1 (a). However, in real cases, the
human speaker and/or the benign audio may not be available or accessible. Moreover, ADA relies
on an imperceptible perturbation principle, viz., the added perturbation must be restricted enough to
avoid being perceived by human beings.

In contrast, this paper proposes the audio independent attack (AIA) as shown in Figure 1 (b) that
sheds light on a more general principle of adversarial attacks, viz., any audio that deceives ASR
_models yet fails to deceive human beings would cause security issues in speech recognition. Alter-_
natively stated, an adversarial audio does not have to be constructed based on an existing benign
audio as ADA does. Our AIA thus enables a novel threat model that constructs audio adversarial
examples completely from scratch instead of adding perturbations on existing benign audios. Particularly, mounted on recent advances in neural speech synthesis (Tan et al., 2021), we can directly


-----

_"True"_

_"True"_

Benigh audio Adversarial audio ASR model Adversarial audio

Neural Speech Synthesis

Perturbation

**(a) Audio dependent attack** _"False"_ **(b) Audio independent attack**


Figure 1: Audio dependent attack versus audio independent attack.

synthesise the adversarial audio that conserves the desired semantic content but deceives the ASR
model to predict incorrect or even targeted transcriptions.

With this goal in mind, we propose the Speech
Synthesising based Attack (SSA) as shown in "Send a greating email to TOM" "Transfer one million
dollars to Jerry"
Figure 2, where a conditional variational autoencoder CVAE is incorporated to synthesise Transcription
audio waveform x that is connected to an ASR Text Encoder Decoder
model thereafter. The basic philosophy of SSA
is: the audio style vector z in CVAE controls
the pitches and rhythms of synthesised wave- Duration Projection
forms that may cause the ASR model to transcribe incorrectly or even as a target; thus some

|Text Encoder Duration Projection Predictor|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||Text Encoder||
|||||
|||Projection||
||tor|||

particular synthesised waveforms would become adversarial examples. In particular, fol- Audio Style
lowing (Kim et al., 2021), we first train CVAE Vector ASR Model
by adopting a variational inference augmented Flow
with normalizing flows and an adversarial training process, so as to synthesise natural sound- Decoder Raw Waveform
ing audios. Once the CVAE is well trained,
given a sequence of conditional text that represents the ground truth audio semantics, we can
optimize z to get a particular z that can de_∗_ Figure 2: The general structure of SSA.

Transcription

Text Encoder Decoder

Duration Projection

Predictor

Audio Style

Vector ASR Model

Flow

Decoder Raw Waveform

ceive an advanced ASR model through a regularized connectionist temporal classification loss. In this regard, we formulate the adversarial
example synthesising task as a gradient sign based optimization problem. More importantly, to
solve the optimization efficiently, we design an adaptive learning rate decay based on the annealing
mechanism, which can automatically adjust the step size during optimization. Our contributions are
summarized as:



-  For the first time, we propose an audio independent adversarial attack as a novel threat model in
ASR, which constructs adversarial example completely from scratch without depending on any
existing speaker/audio. This sheds light on a more general principle of adversarial attacks, viz.,
**_any audio that deceives ASR models yet fails to deceive human beings would cause security issues_**
in speech recognition.

-  We develop SSA that adopts CVAE as the speech synthesiser. To efficiently synthesise audio
adversarial examples, we establish an adaptive sign gradient descent algorithm via designing an
annealing mechanism inspired learning rate decay.

-  Extensive experiments across three datasets (i.e., Audio Mnist (Becker et al., 2018), Common
Voice (Ardila et al., 2019), and Librispeech (Panayotov et al., 2015)) based on DeepSpeech model
(Amodei et al., 2016) show that our SSA can effectively generate audio adversarial examples.

2 RELATED WORK

In ASR adversarial attacks, Carlini & Wagner (2018) were among the first ones to showcase that
slight perturbations on audio can easily fool a start-of-the-art ASR model to transcribe the perturbed
audio into any target sentence. Later on, Yuan et al. (2018) demonstrated that such adversarial audio
perturbation can be embedded into a song. Yang et al. (2019) analyzed that the temporal dependency
could promote the discriminative power against adversarial examples in ASR. Moreover, Khare et al.


-----

(2019) and Taori et al. (2019) found that similar audio adversarial perturbations could be generated
using black-box optimization algorithms (e.g., genetic algorithm (Mitchell, 1998)). To generate
the audio adversarial examples faster and improve the attack efficiency, Liu et al. (2020) proposed
to adaptively rectify the weights of audio perturbations on different positions. To make the audio
perturbation more imperceptible, Qin et al. (2019) introduced a psychoacoustic principle of auditory
masking to smartly add perturbations on frames where noises are hard to be perceived. Xie et al.
(2021) utilized generative adversarial network (GAN) to generate adversarial examples on speech
domain, where GAN is to learn the distribution of predefined adversarial perturbations. Thus the
adversarial examples are still generated depending on the benign audios.

In summary, previous adversarial attacks on ASR are audio-dependent, viz., the audio adversarial
examples must be generated based on corresponding benign audios. Although there are studies, e.g.,
(Song et al., 2018; Wang et al., 2019), on generating adversarial examples from scratch, but most
of them focus on the image domain. Carlini & Wagner (2018) studied an audio attack that starts
from non-speech (e.g., a piece of classic music), while existing audios are stilled required to mount
perturbations on. Roy et al. (2018) and Zhang et al. (2017) proposed to modulate voice commands
on ultrasonic carriers (e.g., frequency > 20 kHz) to achieve inaudibility, where the scenario is an
adversary that stands on the road and silently controls the voice command assistant systems. This
therefore is different from our scope of generating natural sounding adversarial audios. To the best
our knowledge, our work is the first attempt to generate audio adversarial examples from scratch
without utilizing benign audios in ASR.

3 BACKGROUND

**Automatic Speech Recognition (ASR). In this paper, we focus on the speech-to-text tasks that are**
based on neural ASR models. Following previous studies (Esmaeilpour et al., 2021; Liu et al.,
2020; Yakura & Sakuma, 2019; Carlini & Wagner, 2018), we stick our attacks on the Deep**Speech (Amodei et al., 2016), a state-of-the-art ASR model based on Connectionist Temporal Clas-**
sification (CTC) method (Graves et al., 2006). DeepSpeech uses CTC as the input aduios and the
corresponding transcriptions are unaligned, viz., the exact position of each word in the audio sample
is unknown. To enable efficient supervised training, a transcription is first enumerated to obtain all
alignments. Hence, the CTC loss is minimized to maximize the probabilities over all alignments.

**CTC Loss. Let X be the audio input domain and Y be the text output domain with dimension |Y|.**
The ASR model is donoted as F : X _[N]_ _→Y_ _[N]_ _[·|Y|], which takes a N frames x ∈X as input and_
outputs a probability distribution F(x) over the output domain. Given y as a phrase (i.e., a sequence
of characters), we define a token sequence π being reducible to y if the two operations, namely,
removing sequentially duplicated tokens, and deleting all blank tokens on π could produce y. We
further denote π as an alignment of y if π can be reduced to y and the length of π equals to the
length of prediction F(x). Let Φ(y) be the alignment set obtained from targeted transcription y
using dynamic programming (Graves et al., 2006). Accordingly, the CTC loss can be formulated as,

_N_

_Lctc(x, y) = −log_ _π∈Φ(y)_ _i_ _[F]_ [(][x][)]π[i] _i_  _,_ (1)
X Y

where (x)[i]πi [is the probability of the token][ π][i] [on the][ i][th] [frame. To get the transcription in inference,]
_F_
a decoder D (e.g., greedy decoding or beam search decoding) is required. Thereby, if an ASR
model is well trained, the transcription would satisfy y = D(F(x)) = f (x), where f (·) is a merged
denotation of F and D.

**White-Box Assumption. Following most of previous studies, we use the similar white-box assump-**
tion, viz., the parameter and structure of ASR model is known. Investigating the black-box attack is
another direction. Moreover, there are many ways (Oh et al., 2019; Zanella-Beguelin et al., 2021;
Qin et al., 2019) to convert a black-box model to a white-box one.

**Targeted Attack. Compared with the untargeted attack that only maximizes the word error rate**
(WER), the targeted attack is a more challenging task since it requires not only the audio perturbation
imperceptible, but also the adversarial audio being transcribed to a specified target phrase. Our
main focus is on the targeted scenario. Without specification, the adversarial example in following
sections refers to the targeted one.


-----

**Speech Synthesise (End-To-End Text-To-Speech). Text-to-speech (TTS) model synthesises wave-**
forms given text phrases as semantic contents. For efficiently synthesising adversarial audios in our
SSA, we turn to the end-to-end TTS model that could easily utilize modern parallel processors for
faster synthesis speed. In particular, we utilize a conditional variational autoencoder (Kim et al.,
2021) based TTS model to generate natural sounding audios, which is explained in Section 4.2.

4 METHODS

4.1 PROBLEM SETTINGS

Given a well trained ASR model f (·), the objective of adversarial attack is to construct an audio
waveform x ∈X that is naturally sounded yet able to deceive f (·) in predicting incorrect/targeted
transcriptions. Suppose o : X →Y is an oracle that takes an audio waveform x as input and
outputs the ground truth transcription yo = o(x), where Y is the set of all text transcriptions under
consideration. Moreover, compared to the untargeted attack that only introduces spelling errors, we
focus on the more challenging targeted attack, viz., f (x) = y where y ∈Y is the expected target
transcription by an attacker.

In previous studies, the adversarial audio x is constructed by adding perturbation δ on a benign audio
_xo, viz., x = xo + δ; thus being dependent on xo. Mounted on these notations, we give a formal_
definition of the previous audio dependent attack as follows.
**Definition 1 (Audio Dependent Attack - ADA). Given a benign audio xo and its oracle transcription**
∆
_yo = o(xo), the corresponding adversarial example can be defined as any audio x, viz., x ∈Aδ_ =
_{xo + δ ∈X|∃δ, M(δ) ≤_ _ϵ_ _[∩]_ _[o][(][x][o][ +][ δ][) =][ y][o][ =][ o][(][x][o][)]_ _[∩]_ _[f]_ [(][x][o][ +][ δ][) =][ y][t] _[∩]_ _[f]_ [(][x][o][)][ ̸][=][ y][t] _[∩]_ _[y][o][ ̸][=][ y][t]_

where ( ) is a distance measurement (e.g., matrix norm); ϵ is a small positive constant; and yt
_M_ _·_
indicates the targeted transcription of an attacker.

| {z } | {z } | {z } | {z } | {z }[}][,]

From Definition 1 the adversarial audio x is directly built on a benign audio xo. Moreover, to
guarantee x being acoustically realistic or natural, the efforts mainly focus on forcing M(δ) ≤ _ϵ_
with the goal of restricting x to be close to xo. However, in some cases, the benign audio xo may not
be available. For instance, an attacker want to deceive a voice commander when no human speaking
happens nearby. Moreover, ADA relies on an imperceptible perturbation principle, viz., the added
perturbation δ must be small enough to avoid being percepted by human beings. In contrast, a more
general scenario would be audio independent attack that is defined as follows.
**Definition 2 (Audio Independent Attack - AIA).** Given a conditional text (i.e., _yo),_
∆
an audio independent adversarial example can be any element from _a_ = _x_
_A_ _{_ _∈_
_o(x) = yo_
_X|_ _[∩]_ _[f]_ [(][x][) =][ y][t] _[∩]_ _[y][o][ ̸][=][ y][t]_

rectly conveys its semantic content yo; and f (x) = yt means the ASR model f (·) is successfully
fooled to output the targeted transcription yt.

| {z } | {z } | {z }[}][,][ where][ o][(][x][) =][ y][o][ indicates that the synthesised audio][ x][ cor-]

Such AIA sheds light on a more general principle of adversarial attacks, viz., any audio that deceives
_ASR models yet fails to deceive humans would cause security issues in speech recognition. In other_
words, the adversarial audios are not necessarily constructed via adding perturbations. Instead, they
can be directly synthesized with the goal of preserving the desired semantic content (i.e. o(x) = yo),
while simultaneously deceiving the ASR model to predict incorrect or even targeted transcriptions
(i.e. f (x) = yt). This motivates us to leverage the powerful generative model in TTS area to
construct such adversarial audio x. Next, we will introduce the speech synthesising based attack.

4.2 SPEECH SYNTHESISING BASED ATTACK

The overall structure of the speech synthesis based attack (SSA) is depicted in Figure 2.

**Conditional Variational Autoencoder based Speech Synthesis. The key for synthesising natural**
sounding adversarial audio is to model the TTS mapping. In practice, we can select different types of
TTS models (Tan et al., 2021) to generate natural sounding audios. We choose the recent conditional
variational autoencoder (CVAE) based TTS model (Kim et al., 2021) due to its rich variety in audio
generation, viz., a text input can be spoken in multiple ways with different pitches and rhythms.


-----

**Algorithm 1: SSA Algorithm**

**Input: yo – the ground truth text; yt – the target for attack; G(·) – the generative model for speech**
synthesise; F (·) – the ASR model; α0 – the initial learning rate; dα – the learning rate decay ratio;
_λ – the weight of Lreg(z); Im – the maximum iteration number; Dls(·) – Levenshtein distance._

**1 Initialization: audio style vector z ∼N** (0, 1); the patience p = 0; record best loss L∗ = +∞;

**23 for iCalculate the SSA loss ∈** **N+ ∩** _i < Im do_ _Lssa(z, yo, yt) in Eq. (3);_

**4** **if Lssa(z, yo, yt) <= L∗** **then**

**5** _L∗_ = Lssa(z, yo, yt), p = 0

**6** **else**

**7** _p ←_ _p + 1_

**8** **if p >= pm then**

**9** _α_ _dα_ _α, p = 0_
_←_ _·_

**10** Do back-propogation to get _[∂][L][ssa][(]∂z[z,y][o][,y][t][)]_ and update z following Eq. (5);

**11** **if Dls(f** (G(z, yo)), yt) == 0 then

**12** **Return the successful audio adversarial example x∗** = G(z, yo)


**13 Return the current best audio adversarial example x = G(z, yo)**

This accordingly enlarges the sample space, and enhances the possibility of constructing successful
adversarial examples. In specific, such CVAE based TTS model (Kim et al., 2021) is based on three
components: 1) a conditional VAE formulation; 2) an alignment estimation derived from variational
inference; 3) an adversarial training for improving the synthesis quality. The architecture of the
CVAE model in inference is shown in Figure 2, where the corresponding training loss and settings
can refer to (Kim et al., 2021).

In particular, given a conditional text ctext = yo (where yo conveys the ground truth semantic
content), the TTS model aims to build the mapping G(·), viz.,

_x =_ (z, ctext), z (0, 1), (2)
_G_ _∼N_

where z is a normally distributed vector that controls the audio styles with different pitches and
_rhythms. Therefore, with different z, there are different ways to speak the content in ctext. This_
aligns to the fact, viz., even human beings always pronounce same words differently from time
to time. Note that the audio x generated by CVAE model G(·) in (Kim et al., 2021) has been
tested to be natural sounding using the mean opinion score obtained from Amazon Mechanical Turk
(www.mturk.com). The following goal thus becomes figuring out a particular z that can fool an
ASR model F(·).

**Speech Synthesising based Attack (SSA) Formulation. We focus on the targeted attack, viz.,**
deceiving the ASR model ( ) to predict a target phrase yt. Given the speech synthesiser (z, ctext)
_F_ _·_ _G_
in Eq. (2), our SSA is formulated as finding a particular z∗ during synthesising audio x that can
enable the targeted attack. To this end, a loss function Lssa is designed to construct natural sounding
yet fooling enough x. Accordingly, the optimization of z can be defined as

_z∗_ = arg minz _Lssa(z, yo, yt) = arg minz_ _Lctc(G(z, yo), yt) + λLreg(z),_ (3)

where yo is the phrase that the CVAE model wants to synthesise; yt is the targeted phrase that the
ASR model is fooled to predict; and _reg(z) is the regularization loss controlled by λ._ To be
_L_
specific, _reg(z) is designed to boost the generated audio to be naturally sounded. Based on (Kim_
_L_
et al., 2021), z is sampled from a normal distribution. To preserve this property, we design it as:

_reg(z) = ϕ(µ(z)) + ϕ(δ(z)_ 1), (4)
_L_ _−_

where ϕ(·) is an absolute value function; µ(z) and δ(z) are the mean and variance of z, respectively.
To detemine whether the generated audio x = (z, yo) can enable a successful targeted attack (i.e.,
_G_
_f_ (x) == yt), we involve the Levenshtein distance (Yujian & Bo, 2007). During the optimization of
_ssa(z, yo, yt), if Dls(f_ ( (z, yo)), yt) = 0, the targeted attack is successful.
_L_ _G_

**Adaptive Sign Gradient Descent. To solve the optimization problem in Eq. (3), we propose an**
adaptive sign gradient descent algorithm whereby an adaptive learning rate decay mechanism is


-----

designed in the sign gradient descent optimization. Typically, the sign gradient descent is frequently
adopted in adversarial attack algorithms, e.g., fast gradient sign method (Goodfellow et al., 2015)
and projected gradient descent (Madry et al., 2018). Moreover, Balles & Hennig (2018) theoretically
proved the benefits of using gradient sign as the optimization direction. In our SSA, the sign gradient
descent based optimization is given by,

_z_ _z + α_ _[∂][L][ssa][(][z, y][o][, y][t][)]_ _,_ (5)
_←_ _∂z_

where α is the learning rate, which significantly impacts the convergence of the SSA optimization.
Inspired by the heuristics of annealing (Bertsimas & Tsitsiklis, 1993), we carefully design an adaptive learning rate decay mechanism as below.
**Definition 3 (Annealing based adaptive learning rate decay). For every pm steps where the attack**
loss _ssa(z, yo, yt) has no improvement, the learning rate α will decay as α := dα_ _α, where_
_dα_ _L[0, 1] is the decay ratio._ _·_
_∈_

Definition 3 tells that if a specific α gets stuck for particular pm steps with regard to Lssa(·), α
should decay to perform a more local search. The overall process of SSA is shown in Algorithm 1.

5 EXPERIMENTS AND RESULTS

5.1 EXPERIMENT SETTINGS

**Datasets. In our experiments[1], we use three datasets, i.e., Audio Mnist (Becker et al., 2018), Com-**
mon Voice (Ardila et al., 2019), and Librispeech (Panayotov et al., 2015). However, different from
previous studies using both the waveform and text label, we only utilize the text information due to
the audio independent property of our SSA. In particular, the Audio Mnist contains the text digits
(i.e., from “ZERO” to ”NINE”). When building the targeted attack pairs, we choose one text digit
(e.g.,“ZERO”) as yo and enumerate the rest digits (i.e., from “ONE” to “NINE”) as its attack target
_yt. For Common Voice and Librispeech, we randomly sample 1000 text labels first. Those sampled_
texts are filtered and clustered based on their text length. Thereafter, 100 texts are sampled out from
the filtered 1000 samples as the candidates of conditional text yo. The corresponding target text yt is
sampled from their matching clusters. The length comparison between yo and yt on the two datasets
are shown in Figures (10-11) in the appendix, where we notice that yo and yt are aligned with a
similar length. More analysis can refer to appendix 7.1. Our constructed datasets[2] are released to
benefit future research on speech synthesise/generative model related attacks.

**Model Settings. In our SSA paradigm, the CVAE model and DeepSpeech are utilized as speech**
synthesiser and speech recogniser, respectively. The CVAE model follows the setting of VITS (Kim
et al., 2021) during inference. In particular, we utilize the CVAE model trained on VCTK dataset
(Veaux et al., 2017), which can be obtained from VITS[3]. The standard deviation of the input noise
for the stochastic duration predictor is set to be 0, thus making it a deterministic one. In addition,
a scaling factor for z is applied. The DeepSpeech model follows the Pytorch implementation in the
adversarial robustness toolbox (Nicolae et al., 2018).

**Evaluation Metrics. Two evaluation metrics, i.e., the word error rate (WER) and success rate (SR),**
are adopted under the targeted attack setting. In particular, they are defined as,

_WER =_ _[S][ +][ D][ +][ I]_ _, SR =_ _[N][s]_ (6)

_Nw_ _Na_


where S, D and I indicate the number of subsitutions, deletions and insertions of words respectively;
_Nw is the total number of words; Ns is the number of successful adversarial example (i.e., f_ (x) =
_yt); and Na is the total number of audios synthesised. Note that SR is same as the sentence-level_
accuracy that is used in previous studies (Qin et al., 2019). In general, larger values of WER and SR
indicate a stronger attack algorithm.

**SSA Optimization Settings. The learning rate α decays in updating z as highlighted in Definition**
3 and Algorithm 1. In particular, α0 (i.e., initialization of learning rate) is searched in the range of

1Our code will be released upon acceptance.
2
[https://drive.google.com/file/d/1EHXRlWrlMXr6qu8WYtjt8-pRzw-VfCau/view?usp=sharing](https://drive.google.com/file/d/1EHXRlWrlMXr6qu8WYtjt8-pRzw-VfCau/view?usp=sharing)
3
[https://github.com/jaywalnut310/vits](https://github.com/jaywalnut310/vits)


-----

Table 1: Targeted results of SSA on the three datasets and comparisons with baselines.

|Attack algorithms|Col2|WER (%)|SR (%)|
|---|---|---|---|
|Baselines|DeepSpeech (No Attack)|7.55|NA|
||C&W (Carlini & Wagner, 2018) Y&S (Yakura & Sakuma, 2019) GAA (Taori et al., 2019) MOOA (Khare et al., 2019) Metamorph (Chen et al., 2020) CIPMA (Esmaeilpour et al., 2021)|78.94 ± 2.01 80.28 ± 3.14 65.80 ± 2.55 68.06 ± 2.71 72.48 ± 1.06 88.19 ± 3.15|30.74 ± 3.16 35.49 ± 0.28 48.35 ± 3.38 47.01 ± 1.42 45.84 ± 4.71 21.69 ± 3.09|
|SSA|SSA-Audio Mnist SSA-Common Voice SSA-Librispeech|100.00 ± 0.00 106.27 ± 18.03 100.71 ± 13.21|100.00 ± 0.00 96.00 ± 8.00 83.19 ± 16.33|
||SSA-Average (Common Voice & Librispeech)|103.49 ± 16.05|89.60 ± 14.37|



Table 2: The MOS comparison between original and SSA synthesised audios.

|Type of audios|MOS|
|---|---|
|Before attack (original synthesised audios) After attack (SSA synthesised audios)|4.09 0.10 ± 3.39 0.29 ±|



[0.01, 0.09] stepped by 0.01; the patience pm ranges from 50 to 400 stepped by 50; the learning rate
decay ratio dα is searched in {0, 5, 0.6, 0.7}; moreover, the regularization weight λ is searched in
_{0, 50, 100, 150, 200}. The maximum iteration number Im is 8000, where any case without finding_
a successful attack within Im budget is deemed as failed.

5.2 TARGETED ATTACK PERFORMANCE

We first show that our proposed SSA is capable of efficiently constructing audio adversarial examples, viz., significantly outperforming existing baselines in Table 1. In particular, the baselines include DeepSpeech[4], C&W attack (Carlini & Wagner, 2018), Y&S attack (Yakura & Sakuma, 2019),
GAA attack (Taori et al., 2019), MOOA attach (Khare et al., 2019), Metamorph attack (Chen et al.,
2020), and CIPMA attack (Esmaeilpour et al., 2021). The results from the baselines are reported
in (Esmaeilpour et al., 2021), which are averaged results over Common Voice and Librispeech. Our
SSA is evaluated on three datasets (i.e., Audio Mnist, Common Voice, and Librispeech), where an
averaged result over Common Voice and Librispeech is calculated for a fair comparison. All the
results of our SSAs are based on targeted attack settings.

In general, we can see that our SSA achieves remarkably better performance on both WER and SR.
Specifically, (1) SSA-Average achieves 103.49% and 89.60% regarding WER and SR, respectively,
which outperforms the best results obtained by CIPMA on WER (i.e. 88.19%) and GAA on SR
(i.e. 48.35%); (2) both SSA-Common Voice and SSA-Librispeech dramatically outperform the best
results obtained from the baselines as well. Especially for SSA-Common Voice, both of its WER (i.e.
106.27%) and SR (i.e. 96.00%) are the best across all comparisons; (3) SSA-Librispeech behaves
worse than SSA-Common Voice with regards to WER and SR, which is mainly attributed to the
difference of conditional text lengths between the two datasets; and (4) our proposed SSA realizes
100% w.r.t. both WER and SR on Audio Mnist, which can serve as a strong baseline for future study
on the audio adversarial attack. We suggest the reader to listen to our synthesised adversarial audios
that are available at our webpage[5].

5.3 QUALITY EVALUATION OF SYNTHESISED AUDIOS

Although the regularization loss _reg(z) in our SSA is designed to force our synthesised audios_
_L_
to be independently and identically distributed w.r.t. the audio generated in VITS [1], we still find
some synthesised audios not natural sounding enough. Therefore, we further evaluate the quality
of synthesised adversarial audios by mean opinion score (MOS) tests. Specifically, we invited 20
participants to rate on 50 sample pairs, where each sample pair includes an original synthesised
audio by CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each
participant will listen to these 100 audio samples that are randomly shuffled. The naturalness rating

4
[https://github.com/Picovoice/speech-to-text-benchmark#mozilla-deepspeech](https://github.com/Picovoice/speech-to-text-benchmark#mozilla-deepspeech)
5
[https://sites.google.com/view/ssa-asr/home](https://sites.google.com/view/ssa-asr/home)


-----

100

90

80

70

60

50

40



50 100 150 200 250 300 350 400

28 48 62 67 75 80 79 82

54 66 78 86 88 92 95 92

54 79 90 91 94 98 99 98

62 79 94 95 97 97 98 98

64 86 93 93 97 99 100 98

69 86 97 97 97 99 98 100

73 90 93 95 99 99 99 98

72 91 97 100 99 99 99 100

69 95 96 98 98 97 100 98

Patience



pm = 50 pm = 100 pm = 150 pm = 200 pm = 250 pm = 300 pm = 350 pm = 400

100

80

60

40

Success rate (%) 20

0 α0 = 0.01 α0 = 0.02 α0 = 0.03 α0 = 0.04 α0 = 0.05 α0 = 0.06 α0 = 0.07 α0 = 0.08 α0 = 0.09

Learing rate initialization


Figure 3: The results of SR with different α0 and pm on Audio Mnist.

59.0077.00 63.0083.00 71.0082.00 63.0083.00 80 100

85.00 86.00 90.00 88.00 60 80

90.00 89.00 92.00 89.00 60

89.00 92.00 86.00 91.00 40 40


50 100 150 200 250 300 350 400

1.00 6.00 12.00 18.00 21.00 23.00 24.00 27.00

7.00 23.00 41.00 50.00 59.00 63.00 71.00 63.00

12.00 50.00 65.00 73.00 77.00 83.00 82.00 83.00

23.00 61.00 77.00 83.00 85.00 86.00 90.00 88.00

35.00 70.00 82.00 84.00 90.00 89.00 92.00 89.00

39.00 77.00 86.00 88.00 89.00 92.00 86.00 91.00

47.00 79.00 86.00 92.00 91.00 93.00 92.00 93.00

53.00 76.00 90.00 89.00 92.00 91.00 95.00 93.00

55.00 83.00 88.00 91.00 92.00 94.00 93.00 96.00

Patience



pm = 50 pm = 100 pm = 150 pm = 200 pm = 250 pm = 300 pm = 350 pm = 400

100

80

60

40

Success rate (%) 20

0 α0 = 0.01 α0 = 0.02 α0 = 0.03 α0 = 0.04 α0 = 0.05 α0 = 0.06 α0 = 0.07 α0 = 0.08 α0 = 0.09

Learing rate initialization


Figure 4: The results of SR with different α0 and pm on Common Voice.

score is scaled from 1 to 5. The MOS results are shown in Table 2, where we can observe a slightly
worse MOS score of our SSA synthesised audios (i.e., 3.39 ± 0.29) compared with that of original
synthesised ones (i.e., 4.09 ± 0.10). This again indicates that there exists distortions in the synthesised audios by SSA. However, we further note that the difference of MOS scores between the two
types of synthesised audios is not significant, which indicates that the distortions are still subjectively acceptable. Future works can focus on how to eliminate such distortions in the adversarial
audio generation, such as redesigning the regularization loss _reg(z) in Eq (3)._
_L_


5.4 HYPERPARAMETER ANALYSIS ON THE ADAPTIVE SIGN GRADIENT DECENT

In the adaptive sign gradient decent, the initial learning rate α0 and the patience pm are two important
hyperparameters. Their impacts regarding SR on Audio Mnist and Common Voice datasets are
displayed in Figure 3 and 4, respectively. Moreover, we also analyze their impacts on Common
Voice w.r.t. WER. Several interesting findings are noted as below.

**Analysis on SR. (1) Generally, the SR first climbs up as α0 increases and then keeps stable with**
further increase of α0. For instance, a stable SR is achieve with α = 0.03 on Audio Mnist. Similar
trends are also held with pm. (2) On both Audio Mnist and Common Voice, pm needs to be set as
a mild value (e.g., pm 300, 350 ) in order to obtain a promising SR. This indicates that if the
learning rate α decays too fast, the sign gradient descent optimization in Algorithm 1 may get stuck. ∈{ _}_
**(3) Compared the results in Figures (3-4), the SR from Common Voice is usually smaller than that**
from Audio Mnist even with the same settings on α0 and pm. This is mainly because the audio
synthesised in Common Voice is clearly longer than that from Audio Mnist.

**Analysis on WER. Fig-**

102.40 103.74 104.40 104.18 104.11 104.83 104.48 104.16

104.24 104.26 105.17 105.82 105.66 105.66 105.45 105.68

104.26 105.43 105.62 106.02 106.02 106.02 105.85 106.02

104.76 105.90 105.82 105.91 106.13 106.02 105.91 106.13

105.09 105.74 105.85 106.13 106.02 106.00 106.13 106.13

105.64 106.02 105.91 105.96 106.02 106.13 106.02 106.27

105.09 106.13 106.02 106.13 106.13 106.13 106.02 106.13

105.46 106.13 106.13 106.13 106.13 106.13 106.13 106.13

105.63 106.02 106.13 106.13 106.13 106.02 106.13 106.13

ferent lines (i.e., with dif- Patience Patience
ferent α0), we can only ob- Figure 5: WER with different α0 and pm on Common Voice.

α0 = 0.01 α0 = 0.04 α0 = 0.07

α0 = 0.02 α0 = 0.05 α0 = 0.08

α0 = 0.03 α0 = 0.06 α0 = 0.09

106.0

105.5

105.0

104.5

104.0

103.5

Word error rate (%)

103.0

102.5

50 100 150 200 250 300 350 400

Patience

serve a slight change on WER with α0 0.04. This leads to a similar conclusion with the analysis
on SR, viz., proper settings of α0 and p ≥m can easily synthesise harmful adversarial audios via SSA.


-----

600

500

400

300

200

100




|Col1|case-1 cas case-39 cas|e-66 e-75|
|---|---|---|
||case-59 cas|e-80|
||||
||||
||||
||||


1000 2000 3000 4000 5000 6000 7000 8000

case-1 case-66
case-39 case-75
case-59 case-80

Iteration steps

(a) CTC loss convergence


(b) Levenshtein distance convergence


60 case-1case-39 case-66case-75

50 case-59 case-80

40

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps


(c) Text length comparison (d) Learning rate dynamics

Conditional text Target text

70

60

50

40

Text Length30

20

10

0

1 39 59 66 75 80

Case

case-1 case-59 case-75
case-39 case-66 case-80

0.06

0.05

0.04

0.03

Learning rate

0.02

0.01

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

Figure 6: Convergence analyses on selected representative cases on Common Voice.


(a) Original audio (Carlini & Wagner, 2018)


(b) Attacked audio (Carlini & Wagner, 2018)


(c) SSA generated audio


0.2

0.0

0.2

Waveform strength

0.4

0 10000 20000 30000 40000 50000

Frame number


Perturbation
Original

0.2

0.0

0.2

Waveform strength

0.4

0 10000 20000 30000 40000 50000

Frame number


1.00

0.75

0.50

0.25

0.00

0.25

Waveform strength

0.50

0.75

0 10000 20000 30000 40000

Frame number


Figure 7: The comparison on waveforms between the audio dependent attack and our SSA.

5.5 CONVERGENCE ANALYSIS


One natural question is, given a particular setting on α0 and pm, how does the process of adaptive
_sign gradient descent look like? We choose the best setting according to the results in Figure 4, viz.,_
_α0 = 0.06 and pm = 400, and select six representative evaluated cases to show the convergence of_
the CTC loss, Levenshtein distance on Common Voice in Figures 6(a-b), respectively. In addition,
to assist the analysis, the correspond text length comparison and learning rate decays are provided
in Figures 6(c-d). Due to space limitation, the overall convergence process across 100 samples
related to CTC loss and Levenshtein distance and learning rate decays are respectively shown in
Figures (12-14) in the appendix. Some interesting observations are noted as follows.

First, from Figure 6(a) and Figure 12, we can see that the CTC loss of most cases quickly converges
to a small value close to 0. The convergence speed is especially fast at the beginning of the optimization. Such a phenomenon is also observed in other studies (Amodei et al., 2016) related to CTC
loss based training. Second, in Figure 6(b), the Levenshtein distance on most cases except case 1
converges to 0, indicating a successful targeted attack. Third, on some cases, e.g., case 1 in Figure 6(a), although the convergence curve does not stop before reaching the maximum iteration step
8000, it can be deemed as an approximately successful attack as reflected by the small CTC loss in
Figure 6(a) and Levenshtein distance in Figure 6(b). Fourth, the additional text length comparison
in Figure 6(c) shows that the text length generally reflects the difficulty of generating a successful
attack, viz., a longer text usually requires more iterations in both CTC loss and Levenshtein distance
convergences. Lastly, the learning rate dynamics in Figure 6(d) also showcase that for a harder problem (i.e., with a larger text length), the learning rate decays more times to exploit a solution close to
a successful attack. More results and analyses can refer to Appendix 7.2.


-----

5.6 WAVEFORM PATTERN ANALYSIS

To showcase that our SSA is a more general attack compared with audio dependent attacks, we further analyse the waveform pattern of both attacks as shown in Figure 7. In particular, Figure 7(a)
and (b) respectively depict the original audio and the corresponding adversarially perturbed audio,
where we can easily observe that the attacked audio needs to be restricted to only add minor perturbations. In contrast, the adversarial audio constructed by our SSA as shown in Figure 7(c) is free of
such restriction, viz., the waveform can be significantly different. In sum, our SSA can be deemed
as an audio independent attack, which brings in more threat to ASR models in the wild. Further
analyses towards the audio style vector z before and after attack of SSA are shown in Appendix 7.3.
In addition, we also evaluate the tranferability of the adversarial audios w.r.t another ASR model
(i.e., ESPnet (Watanabe et al., 2018)) as shown in Appendix 7.4.

6 CONCLUSION

This paper investigates the audio adversarial attack for ASR models. Existing attack algorithms are
based on an audio dependent assumption, viz., adding constrained perturbations on benign audio
inputs. In contrast, we propose SSA, a novel threat model that constructs audio adversarial examples entirely from scratch, viz, without depending on any existing audio to fool cutting-edge ASR
models. To this end, we propose to use a conditional variational auto-encoder (CVAE) as the speech
synthesiser. Accordingly, the adversarial audio synthesising task is formulated as an optimization
problem via searching in the hidden space of CVAE. Meanwhile, an adaptive sign gradient descent
algorithm is further devised to solve the SSA optimization problem. Experiments on three datasets
show that our proposed SSA can synthesise audios that are naturally sounded but deceive start-ofthe-art ASR models. In our experiments, we also find that some synthesised adversarial audios do
not sound as natural as those without any manipulation on z, which thus needs future efforts to
enhance the quality of adversarial audio synthesis.

REFERENCES

Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl
Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: Endto-end speech recognition in english and mandarin. In Proceedings of the 33th International
_Conference on Machine Learning (ICML), pp. 173–182. PMLR, 2016._

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer,
Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A
massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019.

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic
gradients. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp.
404–413. PMLR, 2018.

S¨oren Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert M¨uller, and Wojciech
Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv
_preprint arXiv:1807.03418, 2018._

Dimitris Bertsimas and John Tsitsiklis. Simulated annealing. Statistical Science, 8(1):10–15, 1993.

Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-totext. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 1–7. IEEE, 2018.

William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural
network for large vocabulary conversational speech recognition. In 2016 IEEE International
_Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960–4964. IEEE, 2016._

Tao Chen, Longfei Shangguan, Zhenjiang Li, and Kyle Jamieson. Metamorph: Injecting inaudible commands into over-the-air voice controlled systems. In Network and Distributed Systems
_Security (NDSS) Symposium, 2020._


-----

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), pp. 248–255. Ieee, 2009._

Mohammad Esmaeilpour, Patrick Cardinal, and Alessandro Lameiras Koerich. Towards robust
speech-to-text adversarial attack. arXiv preprint arXiv:2103.08095, 2021.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
examples. arXiv preprint arXiv:1412.6572, 2015.

Alex Graves, Santiago Fern´andez, Faustino Gomez, and J¨urgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In
_Proceedings of the 23rd International Conference on Machine Learning (ICML), pp. 369–376,_
2006.

Shreya Khare, Rahul Aralikatte, and Senthil Mani. Adversarial black-box attacks on automatic
speech recognition systems using multi-objective evolutionary optimization. Conference of the
_International Speech Communication Association (INTERSPEECH), 2019._

Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. Proceedings of the 38th International Conference on
_Machine Learning (ICML), 2021._

Xiaolei Liu, Kun Wan, Yufei Ding, Xiaosong Zhang, and Qingxin Zhu. Weighted-sampling audio
adversarial example attack. In Proceedings of the AAAI Conference on Artificial Intelligence
_(AAAI), volume 34, pp. 4908–4915, 2020._

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. In International Conference on
_Learning Representations (ICLR), 2018._

Melanie Mitchell. An introduction to genetic algorithms. MIT Press, 1998.

Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy,
and Ben Edwards. Adversarial robustness toolbox v1.2.0. _CoRR, 1807.01069, 2018._ URL
[https://arxiv.org/pdf/1807.01069.](https://arxiv.org/pdf/1807.01069)

Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural
networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 121–
144. Springer, 2019.

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus
based on public domain audio books. In 2015 IEEE international Conference on Acoustics,
_Speech and Signal Processing (ICASSP), pp. 5206–5210. IEEE, 2015._

Yao Qin, Nicholas Carlini, Garrison Cottrell, Ian Goodfellow, and Colin Raffel. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In International Con_ference on Machine Learning (ICML), pp. 5231–5240. PMLR, 2019._

Nirupam Roy, Sheng Shen, Haitham Hassanieh, and Romit Roy Choudhury. Inaudible voice commands: The long-range attack and defense. In 15th {USENIX} Symposium on Networked Systems
_Design and Implementation ({NSDI} 18), pp. 547–560, 2018._

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez,
Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go
without human knowledge. Nature, 550(7676):354–359, 2017.

Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. Advances in Neural Information Processing Systems (NeurIPS),
31:8312–8323, 2018.

Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. A survey on neural speech synthesis. arXiv
_e-prints, pp. arXiv–2106, 2021._


-----

Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. Targeted adversarial examples
for black box audio systems. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 15–20.
IEEE, 2019.

Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vctk corpus: English multispeaker corpus for cstr voice cloning toolkit. 2017.

Xiaosen Wang, Kun He, Chuanbiao Song, Liwei Wang, and John E Hopcroft. At-gan: An adversarial generator model for non-constrained adversarial examples. arXiv preprint arXiv:1904.07793,
2019.

Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno,
Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Pro_ceedings of Interspeech, pp. 2207–2211, 2018._ doi: 10.21437/Interspeech.2018-1456. URL
[http://dx.doi.org/10.21437/Interspeech.2018-1456.](http://dx.doi.org/10.21437/Interspeech.2018-1456)

Yi Xie, Zhuohang Li, Cong Shi, Jian Liu, Yingying Chen, and Bo Yuan. Enabling fast and universal
audio adversarial attack using generative model. In Proceedings of the AAAI Conference on
_Artificial Intelligence (AAAI), volume 35, pp. 14129–14137, 2021._

Hiromu Yakura and Jun Sakuma. Robust audio adversarial example for a physical attack. Proceed_ings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI), pp._
5334–5341, 2019.

Zhuolin Yang, Pin Yu Chen, Bo Li, and Dawn Song. Characterizing audio adversarial examples
using temporal dependency. In 7th International Conference on Learning Representations (ICLR),
2019.

Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang,
Heqing Huang, Xiaofeng Wang, and Carl A Gunter. Commandersong: A systematic approach
for practical adversarial voice recognition. In 27th {USENIX} Security Symposium ({USENIX}
_Security 18), pp. 49–64, 2018._

Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE Transactions on Pattern
_Analysis and Machine Intelligence (TPAMI), 29(6):1091–1095, 2007._

Santiago Zanella-Beguelin, Shruti Tople, Andrew Paverd, and Boris K¨opf. Grey-box extraction of
natural language models. In International Conference on Machine Learning (ICML), pp. 12278–
12286. PMLR, 2021.

Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang, Taimin Zhang, and Wenyuan Xu. Dolphinattack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC Conference on
_Computer and Communications Security, pp. 103–117, 2017._

Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on
deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent
_Systems and Technology (TIST), 11(3):1–41, 2020._


-----

7 APPENDIX

7.1 DATASET ANALYSIS

Figure 9 shows the length distribution of 1000 samples on Common Voice and Librispeech. Note
that, to better sample the adversarial attack target, we filter out data points that have too small
counts. From the selected 1000 samples, we generate our targeted attack dataset, viz., matching a
target that has a similar length with the conditional text. Figures (10-11) illustrate the comparison
of the text length between conditional text yo and target text yt on Common Voice and Librispeech,
respectively. In general, the length of yo and yt are similar. Moreover, in both datasets, the text
length has a huge variance, e.g., case 3 versus 30 in Librispeech. Compared Common Voice to
Librispeech, the length of many cases on Librispeech is much larger than the maximum length (i.e.,
100) on Common Voice.

7.2 ADDITIONAL ANALYSIS ON THE CONVERGENCE OF CTC LOSS AND LEVENSHTEIN
DISTANCE AND LEARNING RATE DECAY

The CTC loss convergence processes across the 100 samples on Common Voice are shown in Figure 12. Moreover, we further analyse the convergence of Levenshtein distance Dls( ) (Khare et al.,

_·_
2019) in Figure 13, in order to depict how does our proposed SSA succeeds. Note that the Levenshtein distance is calculated as Dls(f ( (z, yo)), yt), which indicates the distance between the
_G_
transcription on the current synthesised audio x and the target transcription yt. We also analyse the
dynamics of α in Figure 14.

From Figure 12, we can see that the CTC loss of most cases quickly converges to a value close to 0.
The corresponding Levenshtein distance in Figure 13 exactly converges to 0 on these cases, suggesting successful attacks on most cases. For the cases that do not stop the optimization before reaching
the maximum iteration step 8000, both the CTC loss and Levenshtein distance end with small values, which can be deemed as approximately successful attacks based on the related studies (Zhang
et al., 2020) on adversarial attack in natural language processing. From Figure 14, generally different cases have their own learning rate schedule. On most cases, the learning rate needs to decay
for at least 2 times. Only on some particular cases (e.g., case 2), directly using the initialized α
without decay can find a successful adversarial example. This implies that our designed adaptive
sign gradient descent algorithm enables the learning rate α to dynamically change according to the
optimized loss.

7.3 ANALYSES ON THE AUDIO STYLE VECTOR z

We further compare the original audio style vector z (i.e., sampled from the normal speech synthesis)
and the adversarial z (i.e., optimized by our SSA loss) on three cases as shown in Figure 8. For better
visualization, we only plot partial of vector z. For instance, z is reshaped from a (192 × 231) matrix
with dimension determined by the conditional text in Figure 2, while we only plot (2 × 231) of
them as shown in Figure 8 (a). In general, Figure 8 shows that the original z and adversarial z
are significantly different. Namely, although both original and adversarial z fluctuate around the
mean 0 and share a comparable variance, their specific values for each dimension are quite different.
This indicates that our SSA has more flexibility of searching for a successful attack. In contrast,
the optimization space of previous audio dependent attacks is restricted around the original audio
waveform by a norm bound as shown in Figure 7 (b).

7.4 ANALYSES OF ATTACK TRANSFER

Our SSA is designed to be ASR model dependent. In specific, the adversarial audios are synthesized
based on Deep Speech. Therefore, as expected, these adversarial examples should pose limited threat
to other ASR models. To validate such a hypothesis, we mount the successfully synthesised audio
attacks on ESPnet (Watanabe et al., 2018) (i.e., an attention-based encoder-decoder network). In
doing so, we randomly sample 30 successfully synthesised attacks (i.e., based on Deep Speech),
input them to the ESPnet and calculate the levenstein distance (LD) with respect to the target text,
where LD = 0 indicates a successful targeted attack. Results show that the success rate and LD of


-----

Table 3: The human speech recognition evaluations on the original and SSA synthesised audios.


|pe of audios|Human Translation W|
|---|---|
|fore attack (original synthesised a|udios) 18.52 7.46% ±|
|er attack (SSA synthesised audio|s) 22.30 5.05% ±|


these transferred attacks on ESPnet are 0% and 40.97 ± 15.34, respectively. This suggests that the
adversarial audios generated by SSA are hardly transferable to a different ASR model.


7.5 HUMAN SPEECH RECOGNITION (SR) EVALUATION

The human speech recognition (SR) evaluation has been conducted. Specifically, we invite 5 participants to listen to 50 sample pairs, where each sample pair includes an original synthesised audio by
CVAE and its corresponding synthesised adversarial audio optimized by our SSA. Each participant
then writes down the corresponding translation text. The WER is calculated between the human
translation and ground truth text. The averaged WER from the human evaluation is shown in Table
3, which indicates that the human translation performance is only slightly impacted compared to the
original synthesised audios.


|4|Col2|Col3|
|---|---|---|
|3 2 1 0|||
||||
||||
||||
|1|||
|2|||
|3||Original Adversarial|


100 200 300 400

4

3

2

1

0

1

2

3 Original

Adversarial

4

Dimension of audio style vector z

(a) Case 1.


|3 2 1 0 1|Col2|Col3|
|---|---|---|
||||
||||
||||
|2|||
|3|||
|||Original Adversarial|


25 50 75 100 125 150 175 200

3

2

1

0

1

2

3 Original

Adversarial

4

Dimension of audio style vector z

(b) Case 2.


|Col1|Original|Col3|
|---|---|---|
|3 2 1 0|Original Adversarial||
||||
||||
||||
|1|||
|2|||
|3|||


100 200 300 400

4 Original

3 Adversarial

2

1

0

1

2

3

4

Dimension of audio style vector z

(c) Case 3.


Figure 8: Comparison of z from the normal speech synthesis and the one optimized by our SSA
loss.


-----

The distribution of the evalated data from Librispeech The distribution of the evalated data from Librispeech

175 160

150 140

120

125

100

100

Count Count 80

75

60

50 40

25 20

0 0

[0, 10) [10, 20) [20, 30) [30, 40) [40, 50) [50, 60) [60, 70) [70, 80) [80, 90) [90, 100) [0, 20) [20, 40) [40, 60) [60, 80) [80, 100) [100, 120) [120, 140) [140, 160) [160, 180) [180, 200) [200, 220) [220, 240) [240, 260) [260, 280) [280, 300)

Length range Length range

(b) Length distribution on Librispeech.


(a) Length distribution on Common Voice.


Figure 9: The length distribution of 1000 samples on Common Voice and Librispeech


100 Conditional text length The length comparison between conditional text and target text

Target text length

80

60

Length

40

20

0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Data index


Figure 10: The length comparison between condition text yo and target text yt on Common Voice


The length comparison between conditional text and target text

Conditional text length

250 Target text length

200

Length150

100

50

0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

Data index


Figure 11: The length comparison between condition text yo and target text yt on Librispeech


-----

600 case-10 case-15

case-11 case-16

500 case-12 case-17

case-13 case-18

400 case-14 case-19

300

CTC loss

200

100

0

0 1000 2000 3000 4000 5000

Iteration steps

600 case-30 case-35

case-31 case-36

500 case-32 case-37

case-33 case-38

400 case-34 case-39

300

CTC loss

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

800 case-50 case-55

700 case-51 case-56

case-52 case-57

600 case-53 case-58

500 case-54 case-59

400

CTC loss

300

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

700 case-70 case-75

case-71 case-76

600 case-72 case-77

500 case-73 case-78

case-74 case-79

400

CTC loss300

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

700 case-90 case-95

case-91 case-96

600 case-92 case-97

500 case-93 case-98

case-94 case-99

400

CTC loss300

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

Figure 12: The CTC loss convergence of SSA on Common Voice.


600 case-0 case-5

case-1 case-6

500 case-2 case-7

case-3 case-8

400 case-4 case-9

300

CTC loss

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

case-20 case-25

600 case-21 case-26

500 case-22 case-27

case-23 case-28

400 case-24 case-29

300

CTC loss

200

100

0

0 1000 2000 3000 4000

Iteration steps

700 case-40 case-45

600 case-41 case-46

case-42 case-47

500 case-43 case-48

case-44 case-49

400

CTC loss300

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

600 case-60 case-65

case-61 case-66

500 case-62 case-67

case-63 case-68

400 case-64 case-69

300

CTC loss

200

100

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

case-80 case-85

400 case-81 case-86

case-82 case-87
case-83 case-88

300 case-84 case-89

CTC loss200

100

0

0 1000 2000 3000 4000 5000

Iteration steps


-----

case-10 case-15

60 case-11 case-16

50 case-12 case-17

case-13 case-18

40 case-14 case-19

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000

Iteration steps

70 case-30 case-35

60 case-31 case-36

case-32 case-37

50 case-33 case-38

case-34 case-39

40

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

80 case-50 case-55

70 case-51 case-56

case-52 case-57

60 case-53 case-58

50 case-54 case-59

40

30

20

Levenshtein distance

10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

70 case-70 case-75

case-71 case-76

60 case-72 case-77

50 case-73 case-78

case-74 case-79

40

30

20

Levenshtein distance

10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

80 case-90 case-95

70 case-91 case-96

case-92 case-97

60 case-93 case-98

50 case-94 case-99

40

30

20

Levenshtein distance

10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

Figure 13: The Levenshtein distance convergence of SSA on Common Voice.


60 case-0 case-5

case-1 case-6

50 case-2 case-7

case-3 case-8

40 case-4 case-9

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

60 case-20 case-25

case-21 case-26

50 case-22 case-27

case-23 case-28

40 case-24 case-29

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000

Iteration steps

70 case-40 case-45

case-41 case-46

60 case-42 case-47

50 case-43 case-48

case-44 case-49

40

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

70 case-60 case-65

60 case-61 case-66

case-62 case-67

50 case-63 case-68

case-64 case-69

40

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps

case-80 case-85

50 case-81 case-86

case-82 case-87

40 case-83 case-88

case-84 case-89

30

20

Levenshtein distance10

0

0 1000 2000 3000 4000 5000

Iteration steps


-----

0.020

|case-10 case-12 case-14 case-16 case-18 case-11 case-13 case-15 case-17 case-19|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|0.060 0.055 0.050 0.045 loss 0.040 CTC 0.035 0.030 0.025|||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||


1000 2000 3000 4000 5000

Iteration steps


case-0 case-2 case-4 case-6 case-8

case-1 case-3 case-5 case-7 case-9

0.06

0.05

0.04

0.03

CTC loss

0.02

0.01

0 1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps











0.020

|case-20 case-22 case-24 case-26 case-28|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|
|---|---|---|---|---|---|---|---|---|---|---|
|case-20 case-22 case-24 case-26 case-28 case-21 case-23 case-25 case-27 case-29|||||||||||
|0.060 0.055 0.050 0.045 loss 0.040 CTC 0.035 0.030 0.025|||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||

|case-30 case-32 case-34 case-36 case-38|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|case-30 case-32 case-34 case-36 case-38 case-31 case-33 case-35 case-37 case-39|||||||||
|0.06 0.05 0.04 loss CTC 0.03 0.02 0.01|||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||


1000 2000 3000 4000

Iteration steps


1000 2000 3000 4000 5000 6000 7000 8000

Iteration steps






0






1000 2000 3000 4000 5000 6000 7000 8000

|case-40 case-42 case-44 case-46 case-48 case-41 case-43 case-45 case-47 case-49|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|
|---|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 loss 0.04 CTC 0.03 0.02|||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||


Iteration steps


1000 2000 3000 4000 5000 6000 7000 8000

|case-50 case-52 case-54 case-56 case-58 case-51 case-53 case-55 case-57 case-59|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 0.04 loss CTC 0.03 0.02 0.01||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||


Iteration steps


0










1000 2000 3000 4000 5000 6000 7000 8000

|case-60 case-62 case-64 case-66 case-68 case-61 case-63 case-65 case-67 case-69|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|
|---|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 loss 0.04 CTC 0.03 0.02|||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||


Iteration steps


1000 2000 3000 4000 5000 6000 7000 8000

|case-70 case-72 case-74 case-76 case-78 case-71 case-73 case-75 case-77 case-79|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 0.04 loss 0.03 CTC 0.02 0.01||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||


Iteration steps











1000 2000 3000 4000 5000

|case-80 case-82 case-84 case-86 case-88 case-81 case-83 case-85 case-87 case-89|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 loss 0.04 CTC 0.03 0.02||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||


Iteration steps


1000 2000 3000 4000 5000 6000 7000 8000

|case-90 case-92 case-94 case-96 case-98 case-91 case-93 case-95 case-97 case-99|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|
|---|---|---|---|---|---|---|---|---|---|---|
|0.06 0.05 loss 0.04 CTC 0.03 0.02|||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||
||||||||||||


Iteration steps


Figure 14: The learning rate decay of SSA on Common Voice.


-----

