# DPP-TTS: DIVERSIFYING PROSODIC FEATURES OF
## SPEECH VIA DETERMINANTAL POINT PROCESSES

**Anonymous authors**
Paper under double-blind review

ABSTRACT

With the rapid advancement in deep generative models, recent neural text-tospeech models have succeeded in synthesizing human-like speech, even in an endto-end manner. However, many synthesized samples often have a monotonous
speaking style or simply follow the speaking style of their ground-truth samples.
Although there have been many proposed methods to increase the diversity of
prosody in speech, increasing prosody variance in speech often hurts the naturalness of speech. Recently, Determinantal point processes (DPPs) have shown
remarkable results for modeling diversity in a wide range of machine learning
tasks. However, their application in speech synthesis has not been explored. To
enhance the expressiveness of speech, we propose DPP-TTS: a text-to-speech
model based on a determinantal point process to diversify prosody in speech. The
extent of prosody diversity can be easily controlled by adjusting parameters in our
model. We demonstrate that DPP-TTS generates more expressive samples than
baselines in the side-by-side comparison test while not harming the naturalness of
the speech.

1 INTRODUCTION

In the past few years, text-to-speech models have made a lot of progress in synthesizing human-like
speech (Shen et al., 2018; Ping et al., 2018; Li et al., 2019; Ren et al., 2019). Furthermore, in the
latest studies, several text-to-speech models made high-quality speech samples even in the end-toend setting without a two-stage synthesis process (Donahue et al., 2021; Kim et al., 2021). Based on
these technical developments, text-to-speech models are now able to generate high-fidelity speech.

However, human speech contains many prosodic features like intonation, stress, and rhythm beyond
textual features. Therefore, it is crucial to generate speech samples with rich prosody. There have
been many attempts to synthesize speech with rich prosodic features. To incorporate rich prosodic
features into speech, reference acoustic samples like mel-spectrograms are processed through reference encoder (Skerry-Ryan et al., 2018; Wang et al., 2018) or text-to-speech models are conditioned
on prosodic features like duration, pitch and energy then these features are predicted or manually
controlled at inference (Ren et al., 2021; La´ncucki, 2021). However, these methods have drawbacks
that the bottleneck dimension should be carefully tuned for desirable results, or the prosody predictor
just learns averaged prosodic features of training sets.

Meanwhile, generative models like VAEs and flow models (Hsu et al., 2019; Sun et al., 2020; Vall´esP´erez et al., 2021; Lee et al., 2021; Valle et al., 2021) have been recently used for speech synthesis
and their latent spaces are manipulated for generating more expressive speech. Controlling the
amount of variation in the speech is achieved by generating samples in Gaussian prior with an adequate temperature, however, it has two major drawbacks. First, generating latent samples with high
variance often make generated samples to be unstable in terms of the naturalness and intelligibility
of speech. Second, sampling in the latent space might only cover major modes learned during the
training stage and only monotonous prosodic patterns can occur in some segments of speech.

Determinantal point processes (DPPs) have shown great results for modeling diversity in various
machine learning tasks and their applications. Its uses include text summarization (Cho et al., 2019),
recommendation systems (Gartrell et al., 2021), multi-modal output generation (Elfeki et al., 2019),
diverse trajectory forecasting (Yuan & Kitani, 2020), and machine translation (Meister et al., 2021).
Determinantal point process (DPPs) offers us an efficient subset selection method by considering


-----

both the quality and diversity of items within a set. Specifically, items within a set are sampled
according to DPPs kernel which reflects the quality and diversity of items in the ground set. In some
cases, we may need to incorporate the conditional information into the sampling process. To this
end, conditional DPP (Kulesza & Taskar, 2012) has been widely used for sampling subset by setting
the kernel conditioned on the specific information.

To use conditional DPP for sampling prosodic features of speech, two main issues should be considered. First, prosodic features are usually varying in length. Second, it is ambiguous what the
ground set is in this problem. In this work, we resolve these issues by using a new similarity metric
between prosodic features and adding a prosodic diversifying module (PDM) into the framework.
Specifically, PDM generates candidates of prosodic features of speech by mapping latent codes
from normal distribution to a new latent space and these candidates are used as the ground set in the
sampling process. In addition, the similarity between two prosodic features is evaluated using soft
dynamic time warping discrepancy (Soft-DTW) (Cuturi & Blondel, 2017) which enables to evaluate
the similarity between features in different lengths.

The kernel matrix of conditional DPP is learned during the training by parameters of PDM getting
updated. Specifically, parameters of the prosody diversifying module (PDM) are updated in the
training stage with conditional maximum induced cardinality (MIC) objective which is adapted from
MIC objective introduced in Gillenwater et al. (2018). We formally introduce the conditional MIC
objective and its derivative for clarity in Section 4.

To implement the aforementioned DPP, a stochastic duration predictor and pitch predictor are introduced in this work. Both the duration and pitch predictor are built upon normalizing flows trained
with the maximum likelihood (MLE) objective. At inference, the prosody predictor maps latent
codes from a latent space to feature spaces with the inverse flow.

In experiments, we compare DPP-TTS with the state-of-the-art models including VITS (Kim et al.,
2021) and Flowtron (Valle et al., 2021) in terms of prosody diversity and speech quality. The results
demonstrate that our model generates speech with richer prosody than baselines while maintaining
speech naturalness. We also demonstrate that our DPP-TTS can be used in real-time applications by
evaluating the inference speed of our model.

In summary, our contributions follow:

-  We propose a novel method for diversifying prosodic features of speech based on conditional DPPs by considering prosodic features of context words as conditions.

-  To learn the kernel matrix of conditional DPPs, we propose to train prosody diversifying
module (PDM) with the conditional maximum induced cardinality (MIC) objective.

-  Experiments on the side-by-side comparison and the MOS test verify that our model outperforms the two baseline models in terms of prosodic diversity while maintaining the
naturalness of speech.

2 BACKGROUND

2.1 DETERMINANTAL POINT PROCESSES

DPPs encourage diversity within a set by discouraging sampling similar items within the ground set.
Formally, point process P is called a determinantal point process when Y is a random subset drawn
according to P, we have, for every A ⊆Y,

(A **_Y )_** det(KA), (1)
_P_ _⊆_ _∝_

where K is a positive definite matrix whose eigenvalues are all between 0 and 1 and KA is a positive
a definite matrix indexed by elements in A. The marginal probability of including two elements ei
the extent of negative correlation between itemand ej is KiiKjj − _Kij[2]_ [=][ p][(][e][i][ ∈] **_[Y][ )][p][(][e][j][ ∈]_** _i[Y] and[ )][ −] j[K]. More frequently, DPPs are defined byij[2]_ [. Therefore, the value of][ K]ij[2] [models]
L-ensemble through real and symmetric matrix L instead of the marginal kernel K:

det(LY )
_L(Y = Y ) =_ (2)
_P_ det(L + I) _[,]_


-----

where det(L + I) in the denominator acts as a normalization constant. The marginal kernel K and
kernel L has the following relation:

**_K = L(L + I)[−][1]_** = I − (L + I)[−][1] (3)

To model diversity between items, the DPP kernel L is usually constructed as a symmetric similarity
matrix, where Sij represents the similarity between two items xi and xj. In Kulesza & Taskar
(2010), authors proposed decomposing the kernel L as a a Gram matrix incorporating a quality
vector to weigh each item according to its quality :


_P(J ⊆_ _Y ) ∝_ det(φ(J)[T]φ(J))


_q[2](ei) = Diag(q)_ **_S_** Diag(q), (4)
_·_ _·_
_eYi∈J_


wherereal positive semidefinite matrix. φi ∈ R[D]; D ≤ _N and ||φi||2 = 1. In this manner, the similarity matrix S is guaranteed to be_

2.2 CONDITIONAL DETERMINANTAL POINT PROCESSES

If DPPs are used for diversifying prosodic features by setting targets of DPPs as prosodic features
of sentences, it would result in diversity among generated samples. However, there still can be
monotonous patterns in each generated speech. To resolve this issue, it is required to diversify
prosodic features accounting into their neighbor prosodic features. Therefore, we need conditional
DPPs to take into account neighboring prosodic features (contexts).

By setting conditions of point process P, DPPs can be extended to conditional DPPs, For a subset
_B ⊆_ _Y not intersecting with A we have_

(Y = A _B_ _A_ **_Y ) =_** = [det][(][L][A][∪][B][)] (5)
_P_ _∪_ _|_ _⊆_ _[P][(][Y]([ =]A_ _[ A][ ∪]Y )[B][)]_ det(L + I ¯A[)] _[,]_

_P_ _⊆_

where I ¯A [is the matrix with ones in the diagonal entries indexed by elements of][ Y −] _[A][ and zeros]_
elsewhere. In Borodin & Rains (2004), authors showed that this conditional distribution is again a
DPP, with a kernel given by
**_L[A]_** = ([(L + I ¯A[)][−][1][]][ ¯]A[)][−][1][ −] **_[I][.]_** (6)
In conditional DPPs, items in the ground set Y are sampled according to kernel given contexts. In
this work, contexts of target words are used as conditions for conditional DPPs.


2.3 SOFT DYNAMIC TIME WARPING

To build the kernel of conditional DPPs, we need a measure of similarity between two temporal
signals. Simple Euclidean distance is not applicable because two time signals often vary in their
lengths. In this work, soft dynamic time warping (Soft DTW) discrepancy which allows measuring
the similarity of shape between time series of different lengths is adapted to calculate the similarity
between prosodic sequences. DTW computes the best possible alignment between two temporal
signals. First, for the length of n and m signals, DTW computes n by m pairwise distance matrix
between these points with a specific metric (e.g., Euclidean distance, L1 distance). After that, this
matrix is used to solve the dynamic program using Bellman’s recursion with a quadratic O(mn)
cost.

Unfortunately, vanilla DTW cost is not easy to optimize because it only considers a single alignment
between two temporal signals. Cuturi & Blondel (2017) proposed differentiable Soft-DTW which
considers all possible alignments between two temporal series. Soft-DTW is differentiable in all of
its arguments with quadratic cost. Formally, Soft-DTW is defined as follows:

**dtwγ(x, y) := min[γ]{⟨A, δ(x, y)⟩, A ∈An,m},** (7)

where An,m denotes all possible alignments between x and y and min[γ] is defined as

min[γ]{a1, ..., an} := (−minγ logi≤nai,i=1 _[e]_ _−γai,_ ifotherwiseγ = 0 (8)

A small magnitude of γ reflects the true discrepancy of two temporal signals, however, optimization
becomes unstable. [P][n]


-----

(b) Variance adaptor at inference
of DPP-TTS


(a) Base text-to-speech model training


Figure 1: Diagrams describing (a): training of base text-to-speech model and (b) variance adaptor
at the inference of DPP-TTS after PDM is added.

3 DPP-TTS

Our model DPP-TTS is composed of a Seq2Seq module for generating the mel-spectrogram, a
prosody predictor for predicting duration and pitch sequences, and a prosody diversifying module
(PDM). At the first stage, the base TTS model which consists of the Seq2Seq module and the prosody
predictor is trained as shown in Figure 1a. Once the base TTS is trained, PDM is inserted in front
of the prosody predictor and trained with the method which will be described in detail in Section 4.
We describe the main modules of DPP-TTS and their roles in the following subsection.

3.1 MAIN MODULES OF DPP-TTS

**Seq2Seq module** The role of the Seq2seq module is generating mel-spectrograms from phoneme
sequences. The module is adapted from FastSpeech2 (Ren et al., 2021) with some modifications.
The model consists of four main parts: a phoneme prenet, a phoneme encoder, a variance adaptor, and a mel-spectrogram decoder. In the phoneme encoder, phoneme sequences are processed
through a stack of feed-forward transformer blocks with a relative positional representation (Shaw
et al., 2018). In variance adaptor at training, pitch embeddings and energy embeddings are added to
encoded hidden representations and then hidden representations are expanded according to groundtruth duration labels [1]. At inference, these prosodic features are provided from predictions of the
prosody predictor. Finally, expanded representations are processed through a stack of feed-forward
transformer blocks and mel-spectrograms with 80 channels are generated after the linear projection. The Seq2Seq module is trained to minimize L1 distance between the predicted and target
mel-spectrogram.

**Prosody predictor** In FastSpeech2, the variance adaptor consists of deterministic predictors for
predicting prosodic features. However, a deterministic prosodic predictor is not expressive enough
to learn the speaking style of a person. For diverse rhythm and pitch, a stochastic duration predictor
and pitch predictor are built upon normalizing flows. Specifically, the stochastic duration predictor
estimates the distribution of phoneme duration and the stochastic pitch predictor estimates the distribution of phoneme pitch from the hidden sequence. At the training stage, the prosody predictor
learns the mapping from the distribution of prosodic features to normal distribution. At inference,
it predicts the phoneme-level duration or pitch by reversing the learned flows. In addition, it also

1Ground-truth labels are obtained via monotonic alignment search (Kim et al., 2020) between the phonemes
and mel-spectrogram.


-----

serves as the density estimator for prosodic features during the training of PDM which will be described in detail in Section 4. The prosody predictor is trained to maximize a variational lower bound
of the likelihood of the phoneme duration or pitch. More details regarding the prosody predictor are
in Appendix A.

**PDM** Although the stochastic duration and pitch predictor are trained to generate a speech with
diverse rhythm and pitch, the prosody predictor may favors major modes and it leads to the
monotonous prosodic pattern in the speech. For more expressive speech modeling, PDM is added in
front of the prosody predictor as shown in Figure 1b. Its role is to guide latent codes from a standard
normal distribution to another distribution for diverse prosodic features of speech. This module is
trained with an objective based on conditional DPPs which is described in Section 2.2. At inference
of DPP-TTS, multiple prosodic candidates are generated by PDM and the prosodic feature of speech
is selected via MAP inference.

(a) Segmentation of an input text (b) Training procedure of PDM

Figure 2: Diagrams describing (a): segmentation of an input text for generating target and context
indices and (b) training procedure of PDM.

4 DIVERSIFYING PROSODIC FEATURES OF SPEECH WITH PDM

In this section, we explain the methodology for training the prosody diversifying module (PDM).
The training process mainly consists of three steps: segmentation of an input text, generation of
prosodic feature candidates, building DPP kernel. We also explain the conditional maximum induced cardinality (MIC) objective for training PDM. The overall training procedure is depicted in
Figure 2b.

4.1 SEGMENTATION OF AN INPUT TEXT

In this stage, targets in input text for diversification of prosody are chosen. The scale of each target
is important for the desired goal. Phoneme-level targets would result in too dynamic prosody of
generated speech. In contrast, sentence-level targets may result in a monotonous pattern in each
speech corresponding to each sentence. Therefore, we choose a few words as each target in the
input text. However, naively choosing random words in the input text is not desirable because
prosodic boundaries and syntax of the sentence are closely related (Cole et al., 2010). We choose
the noun phrases in the input text as each target using Spacy [2] library to take account into the
syntactic structure of the input text. Neighboring words with the same number of words as the target
are selected as the left of right context. For example, as shown in Figure 2a, if the target is an
Atreides, then way for and to take are selected as the left and right contexts respectively.
Indices of targets and contexts are stored after the segmentation.

2https://github.com/explosion/spaCy


-----

4.2 GENERATION OF PROSODIC FEATURE CANDIDATES

In this stage, multiple prosodic candidates are generated for the ground set of DPP as shown in
Figure 2b. First, an input text is fed into the pre-trained phoneme prenet and encoder and a hidden sequence is generated. Second, the pre-trained prosody predictor conditioned on the hidden
sequence predicts the whole prosodic features of the input text from the latent codes samples from
a normal distribution. Third, new nc latent codes from a normal distribution are fed into PDM, then
the prosody predictor conditioned on hidden sequence corresponds to the target and its neighboring
context predicts new nc prosodic features. Finally, newly generated target prosodic sequences and
previously predicted prosodic sequences are mixed, then nc prosodic candidates are generated with
the target and context entangled.

4.3 CONSTRUCTION OF DPP KERNEL

Generated candidates are split into left and right context
_dL, dR and n targets d1, d2, ...dn, then the ground set for_
DPP is constructed merging the targets and contexts as
shown in Figure 3. The kernel of conditional DPP is built
incorporating both diversity and quality of the ground set.
As mentioned in Section 2.1, the kernel of DPP can be decomposed as L = Diag(q) · S · Diag(q), where q denotes
the quality vectors of predicted features and S denotes the
similarity matrix.

**Similarity metric** Target sequences and context sequences often vary in lengths, therefore the Euclidean distance is not applicable to calculate the similarity between
two sequences. The similarity metric is calculated as follows based on Soft DTW:

**_Si,j = exp(−k · dtw[D]γ_** [(][d][i][,][ d][j][))] (9)

**dtw[D]γ** [denotes soft-DTW discrepancy with a metric][ D]
and smoothing parameter γ. When the metric D is chosen
as the L1 distance D(x, y) = _i_

the similarity matrix is positive semi-definitesian D(x, y) = ||x − **_y||2[2]_** [+ log(2][|][ −][x][i][ −][exp(][y][i][−||][|][ or half gaus-][x][ −](Cuturi[y][||]2[2][))][,] Figure 3: Construction of DPP kernelwith nc = 4

[P]

et al., 2007; Blondel et al., 2021). In this work, L1 distance is used as the metric of Soft-DTW so that S to be positive semi-definite.

**Quality metric** Based on the estimated density of predicted features, quality scores are calculated.
The density values of predicted features are calculated with importance sampling using the prosody
predictor: p(x; θ) _i=1_ _qp((zxi,zxi;;φθ))_ [. In the experiment, using raw density value resulted in not]
_≈_ [P][N] _|_
enough diverse sample, therefore quality values are not penalized if the density value is greater than
a specific threshold. With log likelihood π(x) = log p(x), the quality score is defined as

_w_ if π(x) >= k
**_q(x) =_** (10)
_w_ exp(π(x) _k)_ otherwise
 _·_ _−_

The threshold value k was set as the average density of the dataset in the experiment. Finally, the
kernel of conditional DPPs is defines as L = diag(q) · S · diag(q).

4.4 OBJECTIVE FUNCTION

We need a measure of diversity with respect to kernel L to train the PDM. One straightforward
choice is maximum likelihood (MLE) objective, log PL(Y ) = log det(L) − log det(LY + I).
However, there are some cases where almost identical prosodic features are predicted. This leads
the objective value to be close to zero which makes the training process unstable. Instead, maximum
induced cardinality (MIC) (Gillenwater et al., 2018) objective which is defined as EY **_L[_** **_Y_** ] can
_∼P_ _|_ _|_
be an alternative. It does not suffer from training instability since identical items in the ground


-----

set Y are allowed. In this work, conditional MIC objective is used as setting contexts dL, dR as
the condition. The objective function with respect to the ground set [dL, dR, d1, d2, ..., dN ] and its
derivative are as follows:
**Proposition 1 (MIC objective of conditional DPPs).** _With respect to the ground set_

[dL, dR, d1, d2, ..., dN ], the MIC objective of conditional DPPs and its derivative are as follows:

_′_

**_LMIC = tr(I_** [(Lθ + I ¯A[)][−][1][]][ ¯]A[)][, ∂][L][MIC] = ((L + I ¯A[)][−][1][I][ ¯]A[(][L][ +][ I][ ¯]A[)][−][1][)][T][L] (θ), (11)
_−_ _∂θ_

_where A denotes the set of contexts (dL, dR),_ _A[¯] denotes the complement of the set A and θ is the_
_parameter of PDM._

Detailed proof is presented in Appendix B. At inference, for predicting a single prosodic feature, MAP inference is performed across sets with just a single item as follows: **_x[∗]_** =
arg maxx _A¯_ [log det(][L] **_x_** _A[)][. The detailed procedure of training PDM and the inference of DPP-]_
_∈_ _{_ _}∪_
TTS are in Appendix C.

5 EXPERIMENTS

In this section, we first explain the experimental setup and then human evaluation results and further
evaluation results that are conducted to evaluate the proposed DPP-TTS. Audio samples used for the
evaluation are in the supplementary material.

5.1 EXPERIMENTAL SETUP

**Dataset and preprocessing** We conducted experiments on the LJ Speech dataset which consists
of audio clips with approximately 24 hours lengths. We split audio samples into 12500/100/500
samples for the training, validation, and test set. Audio samples with 22kHz sampling rate are
transformed into 80 bands mel-spectrograms through the Short-time Fourier transform (STFT) with
1024 window size and 256 hop length. International Phonetic Alphabet (IPA) sequences are used
as input for phoneme encoder. Text sequences are converted to IPA phoneme sequences using
Phonemizer[3] software. Following Kim et al. (2020), the converted sequences are interspersed
with blank tokens, which represent the transition from one phoneme to another phoneme.

**Training Setup** Both the basic TTS and PDM are trained using the AdamW optimizer (Loshchilov
& Hutter, 2019) with β1 = 0.8, β2 = 0.99 and λ = 0.01. The initial learning rate is 2 _×_ 10[−][4] for the
basic TTS and 1 × 10[−][5] for the PDM and the exponential learning decay is used for both of them.
Basic TTS is trained with 16 batch size for 200k steps on 2 NVIDIA RTX 3080 GPUs. PDM is
trained with 8 batch size for 5k steps on same devices. In both the duration and pitch diversification
modeling, the quality weight of the model is set as w = 10.

**Baseline** We compare our model with two state-of-the-art models in terms of naturalness and
prosody diversity. The first one is VITS (Kim et al., 2021) which is an end-to-end TTS model
based on conditional VAE and normalizing flows. In all experiments, the temperature σ of the
duration predictor in VITS was set 0.8. The second one is Flowtron (Valle et al., 2021) which is
an autoregressive flow-based TTS model. In experiments, the temperature σ of Flowtron was set
to 0.6. For our DPP-TTS and Flowtron, HiFi-GAN (Kong et al., 2020) is used as the vocoder for
synthesizing waveforms from the mel-spectrograms.

5.2 HUMAN EVALUATION RESULTS

**Side by side evaluation** First, we conduct a side-by-side evaluation of the prosody of a sentence.
In this experiment, texts from test samples from LJSpeech are synthesized. Target phrases extracted
from the sentence are processed by PDM and pitch diversity modeling is used in DPP-TTS. Via
Amazon Mechanical Turk (AMT), we assign five testers living in the United States to a pair of
audio samples (i.e., DPP-TTS and a baseline), and ask them to listen to audio samples and choose
among three options: A: sample A has more varied prosody than sample B, Same: sample A and B
are equally varied in prosody, B: the opposite of option (1).

3https://github.com/bootphon/phonemizer


-----

In addition, we also conduct a side-by-side evaluation of prosody in a paragraph that is not included
in the LJSpeech dataset since a monotonous pattern is prominent in long texts rather than a single
sentence. We prepare two DPP-TTS versions: (1): DPP-TTS1: a model that has only duration
diversifying module and DPP-TTS2: a model that has both duration and pitch diversifying modules.
In this experiment, only VITS is compared as the baseline. Likewise the evaluation of a sentence,
listeners are asked to choose among the three options.


Results are shown in Table 1, 2. In the sideby-side evaluation of prosody in a sentence,
DPP-TTS outperforms VITS and Flowtron.
In the side-by-side evaluation of prosody in
a paragraph, both the DPP-TTS1 and DPPTTS2 outperform the baseline.


Table 1: Side-by-side comparison on the LJSpeech
dataset.

**Model A/B** **A** **Same** **B**

DPP-TTS/Flowtron 55.3% 12.7% 32.0%
DPP-TTS/VITS 40.0% 32.0% 28.0%


**Naturalness test** In this experiment, we

Table 2: Side-by-Side comparison on the paragraph.

measure the Mean-Opinion-Score (MOS)
for audio samples evaluated in the side-by
**Model A/B** **A** **Same** **B**

side evaluation. Likewise in the previous

DPP-TTS1/VITS 56.25% 6.25% 37.5%

test, five testers are assigned to each audio

DPP-TTS2/VITS 50.0% 12.5% 37.5%

sample. Testers are asked to give a score between 1 to 5 on a 9-scale based on the sam
Table 3: Mos evaluation results with 95% confidence

ple’s naturalness. We used the same qual
intervals.

ity weight value w = 10 for DPP-TTS as
in the side-by-side comparison test. MOS
**Model** **MOS-S** **MOS-P**

**S denotes the quality of audio from the**

VITS 4.08 0.53 3.06 0.52

LJSpeech dataset and MOS-P denotes the _±_ _±_

Flowtron 3.81 0.72 - 

quality of audio from the paragraph. The _±_

DPP-TTS 3.92 0.58 3.13 0.54

MOS result is shown in Table 3. In MOS- _±_ _±_

Ground-truth 4.46 0.51 - 

S, our model gets lower MOS than VITS, _±_
it yet still maintain a MOS of 3.92 and outperforms Flowtron. In MOS-P, our model outperforms
VITS. A significant reduction of MOS in MOS-P compared to MOS-S indicates that it’s challenging
to maintain the naturalness in long texts.

5.3 ADDITIONAL EVALUATION

In this section, we conduct additional evaluations for our DPP-TTS and VITS [4]. We use the following
metrics for the evaluation : (1) σframe: frame-level standard deviation of the pitch in a speech. All
pitch values are normalized since speech samples have different absolute pitch values. Standard
deviations are averaged over 50 generated samples. DIO algorithm (Morise et al., 2009) is used for
pitch extraction. (2): σphoneme: phoneme-level standard deviation of duration or pitch in a speech.
Likewise, the standard deviation values are averaged over 50 generated samples. (3): Determinant:
a determinant of the matrix with respect to a feature set J generated by multiple samples is used
to evaluate the diversity of phoneme-level prosodic features of multiple generated samples. Cosine
similarity is used as a metric between two sequences. Each set is constructed from 10 generated
samples. (4): Inference time: the inference time for synthesizing one-second length waveform is
calculated in the text-to-speech model. The inference speed is evaluated on Intel(R) Core(TM) i77800X CPU and a single NVIDIA RTX 3080 GPU. Computation time is averaged over 100 forward
passes.

|Model|Pitch σframe Pitch σphoneme Pitch Determinant Duration σphoneme Duration Determinant Inference Time(s)|
|---|---|
|DPP-TTS VITS|0.012 0.014 2.92 × 10−14 1.81 7.21 × 10−7 4.4 × 10−2 0.010 0.011 5.58 × 10−17 0.22 6.68 × 10−7 1.3 × 10−2|



Table 4: Additional evaluation results. Numbers in bold denote the better result.

Table 4 shows the result of σframe, σphoneme, determinant and inference time. At both frame and
phoneme levels, the standard deviation of the pitch in DPP-TTS results in higher values than the
baseline. In addition, the standard deviation of duration at phoneme-level in DPP-TTS also results
in a higher value than the baseline. It demonstrates that DPP-TTS generates a speech with more

4Hyperparameter settings remain same as the Section 5.2.


-----

dynamic pitch and rhythm than the baseline. The determinants of duration and pitch sets of DPPTTS outperform the baseline. It shows that DPP-TTS generates more diverse samples in terms of
prosody than baseline. Finally, DPP-TTS results in 0.044 seconds of inference speed. Although its
inference speed is slower than the baseline, our model is applicable in practice since the inference
speed of our model is 22.7x faster than real-time.

5.4 MODEL ANALYSIS

**Ablation study** We conduct ablation studies to verify the effectiveness of PDM, the results are
shown in Table 5. In the side-by-side comparison, 34% more listeners choose the option that DPPTTS has more varied prosody compared to DPP-TTS without PDM. It demonstrates that PDM
contributes to prosody diversification significantly. DPP-TTS also achieves a higher phoneme-level
standard deviation of pitch and duration than the baseline. Although there is a quality degradation
(-0.26 MOS) compared to the baseline, DPP-TTS still achieve a MOS of 3.92.

|Model|Side-by-Side MOS Pitch σphoneme Duration σphoneme Comparison|
|---|---|
|DPP-TTS DPP-TTS w/o PDM|58.5% 3.92±0.58 0.014 1.81 17.0% 24.5% 4.18 ± 0.49 0.01 1.76|



Table 5: Comparison in the ablation studies.

**Adjusting the extent of variation** To study the impact of quality weight w, we plot (normalized)
pitch and log duration values at phoneme-level with different values of quality weight. As expected,
for a large magnitude of quality weight, the model learns diversifying prosodic features with small
deviations from contexts. In contrast, for a small magnitude of quality weight, the model diversifies
prosodic features with large deviations from contexts. We also found that a small magnitude of
quality weight leads to faster convergence of the model but it easily hurts the naturalness of generated
samples. Figures are shows in Appendix E.

**Effect of number of candidates on diver-**

**Number of candidates** **Pitch determinant**

**sity** In this experiment, we study the effect of the number of candidates on the _nc = 4_ 1.69 × 10[−][15]
prosody diversity in generated samples. For _nc = 8_ 4.56 × 10[−][15]
a different number of candidates nc = _nc = 12_ **7.8** **10[−][15]**

**_×_**

4, 8, 12, DPP-TTS is trained up to 4k steps
and all other hyperparameters remain un- Table 6: Pitch determinant values with different numchanged. After training, we calculate pitch ber of candidates
determinants over generated samples for each setup. The result is reported in Table 6. We observe that using a large number of candidates results in more diverse samples although only a single
prosody item is selected at MAP inference. We also find that using a large number of candidates has
a regularization effect for training. Using a small number of candidates generates unstable speech
samples in terms of naturalness, in contrast, using a large number of candidates results in more stable
speech samples.

6 CONCLUSION

In this paper, we have proposed a novel prosody diversifying method based on conditional determinantal point processes (DPPs). To learn the kernel matrix of conditional DPPs, conditional determinantal point process (DPPs) is parameterized additional prosody diversifying module (PDM). To
build the kernel, Soft dynamic time warping is adopted to measure the extent of similarity between
two prosodic features, and the kernel of conditional DPPs is learned with conditional maximum
induced cardinality (MIC) objective. In the experiment, we demonstrated that our new prosody
diversifying method generates more dynamic samples than the baseline while maintaining the naturalness. In addition, we demonstrate that our model can be used in real-time applications by showing
the inference speed of our model.


-----

REFERENCES

Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.06450,
2016.

Mathieu Blondel, A. Mensch, and Jean-Philippe Vert. Differentiable divergences between time
series. In AISTATS, 2021.

Alexei Borodin and Eric M. Rains. Eynard–mehta theorem, schur process, and their pfaffian analogs.
_Journal of Statistical Physics, 121:291–317, 2004._

Jianfei Chen, Cheng Lu, Biqi Chenli, J. Zhu, and Tian Tian. Vflow: More expressive generative
flows with variational data augmentation. ArXiv, abs/2002.09741, 2020.

Sangwoo Cho, Chen Li, Dong Yu, H. Foroosh, and Fei Liu. Multi-document summarization with
determinantal point processes and contextualized representations. ArXiv, abs/1910.11411, 2019.

Jennifer S. Cole, Yoonsook Mo, and Soondo Baek. The role of syntactic structure in guiding prosody
perception with ordinary listeners and everyday speech. Language and Cognitive Processes, 25:
1141 – 1177, 2010.

Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In
_ICML, 2017._

Marco Cuturi, Jean-Philippe Vert, Øystein Birkenes, and T. Matsui. A kernel for time series based
on global alignments. 2007 IEEE International Conference on Acoustics, Speech and Signal
_Processing - ICASSP ’07, 2:II–413–II–416, 2007._

Jeff Donahue, S. Dieleman, Mikolaj Binkowski, Erich Elsen, and K. Simonyan. End-to-end adversarial text-to-speech. ArXiv, abs/2006.03575, 2021.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In
_NeurIPS, 2019._

M. Elfeki, C. Couprie, M. Rivi`ere, and Mohamed Elhoseiny. Gdpp: Learning diverse generations
using determinantal point process. In ICML, 2019.

Mike Gartrell, Insu Han, Elvis Dohmatob, Jennifer Gillenwater, and Victor-Emmanuel Brunel.
Scalable learning and map inference for nonsymmetric determinantal point processes. ArXiv,
abs/2006.09862, 2021.

Jennifer Gillenwater, A. Kulesza, Sergei Vassilvitskii, and Zelda E. Mariet. Maximizing induced
cardinality under a determinantal point process. In NeurIPS, 2018.

Jonathan Ho, Xi Chen, A. Srinivas, Yan Duan, and P. Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. ArXiv, abs/1902.00275,
2019.

Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia,
Z. Chen, Jonathan Shen, P. Nguyen, and Ruoming Pang. Hierarchical generative modeling for
controllable speech synthesis. ArXiv, abs/1810.07217, 2019.

Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for
text-to-speech via monotonic alignment search. ArXiv, abs/2005.11129, 2020.

Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial
learning for end-to-end text-to-speech. ArXiv, abs/2106.06103, 2021.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for
efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F.
Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.
17022–17033. Curran Associates, Inc., 2020. [URL https://proceedings.neurips.](https://proceedings.neurips.cc/paper/2020/file/c5d736809766d46260d816d8dbc9eb44-Paper.pdf)
[cc/paper/2020/file/c5d736809766d46260d816d8dbc9eb44-Paper.pdf.](https://proceedings.neurips.cc/paper/2020/file/c5d736809766d46260d816d8dbc9eb44-Paper.pdf)


-----

A. Kulesza and B. Taskar. Determinantal point processes for machine learning. Found. Trends Mach.
_Learn., 5:123–286, 2012._

Alex Kulesza and Ben Taskar. Structured determinantal point processes. In NIPS, 2010.

Adrian La´ncucki. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP, 2021.

Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for nonautoregressive text-to-speech. In ICLR, 2021.

N. Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In AAAI, 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer_[ence on Learning Representations, 2019. URL https://openreview.net/forum?id=](https://openreview.net/forum?id=Bkg6RiCqY7)_
[Bkg6RiCqY7.](https://openreview.net/forum?id=Bkg6RiCqY7)

Clara Meister, Martina Forster, and Ryan Cotterell. Determinantal beam search. In ACL/IJCNLP,
2021.

Diganta Misra. Mish: A self regularized non-monotonic neural activation function. _ArXiv,_
abs/1908.08681, 2019.

Masanori Morise, Hideki Kawahara, and Haruhiro Katayose. Fast and reliable f0 estimation method
based on the period extraction of vocal fold vibration of singing voice and speech. 2009.

Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan[¨]
Raiman, and John Miller. Deep voice 3: Scaling text-to-speech with convolutional sequence
learning. arXiv: Sound, 2018.

Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech:
Fast, robust and controllable text to speech. In NeurIPS, 2019.

Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2:
Fast and high-quality end-to-end text to speech. ArXiv, abs/2006.04558, 2021.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, 2018.

Jonathan Shen, Ruoming Pang, Ron J. Weiss, M. Schuster, Navdeep Jaitly, Zongheng Yang, Z. Chen,
Yu Zhang, Yuxuan Wang, R. Skerry-Ryan, R. Saurous, Yannis Agiomyrgiannakis, and Yonghui
Wu. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. 2018 IEEE
_International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779–4783,_
2018.

R. Skerry-Ryan, Eric Battenberg, Y. Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss,
R. Clark, and R. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis
with tacotron. ArXiv, abs/1803.09047, 2018.

G. Sun, Y. Zhang, Ron J. Weiss, Yuanbin Cao, H. Zen, and Yonghui Wu. Fully-hierarchical finegrained prosody modeling for interpretable speech synthesis. ICASSP 2020 - 2020 IEEE Interna_tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6264–6268, 2020._

Rafael Valle, Kevin J. Shih, R. Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flowbased generative network for text-to-speech synthesis. ArXiv, abs/2005.05957, 2021.

Iv´an Vall´es-P´erez, Julian Roth, Grzegorz Beringer, R. Barra-Chicote, and J. Droppo. Improving multi-speaker tts prosody variance with a residual encoder and normalizing flows. ArXiv,
abs/2106.05762, 2021.

Yuxuan Wang, Daisy Stanton, Yu Zhang, R. Skerry-Ryan, Eric Battenberg, Joel Shor, Y. Xiao, Fei
Ren, Ye Jia, and R. Saurous. Style tokens: Unsupervised style modeling, control and transfer in
end-to-end speech synthesis. In ICML, 2018.


-----

Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn M. Fung, Yin Li, and
Vikas Singh. Nystr¨omformer: A nystr¨om-based algorithm for approximating self-attention. In
_AAAI, 2021._

Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. ArXiv,
abs/1907.04967, 2020.


-----

A DETAILS OF PROSODY PREDICTOR AND PDM

(a) Duration predictor (b) Coupling block (c) PDM block

Figure 4: Diagrams describing (a): the duration predictor, (b): Coupling block in normalizng flows
and (c): Prosody diversifying module

A.1 PROSODY PREDICTOR

**Training** It is hard to use the maximum likelihood objective directly to train duration predictor
because the duration of each phoneme is 1) a discrete integer and a scalar, which hinders expressive
transformation because invertibility should remain in normalizing flow. To train duration predictor,
duration values are extended to continuous values using variational dequantization (Ho et al., 2019)
and are augmented with extra dimensions using variational data augmentation (Chen et al., 2020).
Specifically, duration sequence d becomes continuous as d − _u where u’s value is restricted to [0, 1)_
and augmented as [d − _u, v] with a extra random variable v. Two random variables u and v are_
sampled through approximate posterior qφ(u, v _d, htext). The ELBO can be calculated as follows:_
_|_

log pθ(d _htext)_ Eqφ(u,v _d,htext)[log_ _[p][θ][(][d][ −]_ _[u, v][|][h][text][)]_ (12)
_|_ _≥_ _|_ _qφ(u, v_ _d, htext) []]_

_|_

Like the duration predictor, the ELBO for pitch predictor can be calculated as follows:

log pθ(p _htext)_ Eqφ(v _p,htext)[log_ _[p][θ][(][p, v][|][h][text][)]_ (13)
_|_ _≥_ _|_ _qφ(v_ _p, htext)_ []][,]

_|_

where p denotes the pitch sequences. Both the duration and pitch predictor are trained on negative
lower bound of likelihood.

**Architecture** The pitch predictor and duration predictor share identical architecture except for the
additional random variable u. We will introduce the architecture of the duration predictor whose
diagram is shown in Figure 4a. Duration predictor consists of condition encoder for hidden sequence, posterior encoder for the duration sequence, and the main flow blocks g(·). Specifically,
posterior encoder maps latent codes from normal distribution to random variable [u, v] and flow g(·)
maps [d − _u, v] to normal distribution. Figure 4b shows the coupling of normalizing flows. First,_
input x is split into [x0, x1] and then x0 is processed by a 1x1 convolution block. The output of the
convolution block is processed by dilated convolution and Nystromer block and their outputs are
concatenated. The concatenated output is processed by LayerNorm, Mish activation, and Convolution block. Finally, spline flows (Durkan et al., 2019) are parameterized by the output and then the
output of spline flows y and x0 are concatenated.

A.2 PDM

The architecture of PDM is shown in Figure 4c. First, the hidden sequence is processed by convolution block followed by the dilated convolution block conditioned on hidden sequences, Layernorm,


-----

and 1x1 convolution block. After that, noise from a normal distribution is processed by 1x1 convolution followed by dilated convolution block, LayerNorm, and Mish activation.

As the prosody pattern of human speech is correlated with the local context and global context of the
text, we design the architecture of the model to encode both the local and global features. Specifically, dilated convolution blocks are stacked to encode local features and a transformer block is used
to encode global features in the coupling layers of normalizing flows. However, using the vanilla
transformer to encode the global features in the coupling layer requires too large computational
complexity. Therefore, Nystr¨omformer (Xiong et al., 2021) which is a Nystr¨om-Based algorithm
for approximating self-attention is used to encode global context for more efficient memory usage
in the coupling layer of normalizing flow. Encoded local features and global features are concatenated and processed through LayerNorm (Ba et al., 2016), Mish activation (Misra, 2019) and a 1x1
convolution module.

Following the duration predictor model in Kim et al. (2021), variational dequantization (Ho et al.,
2019) is used for the duration predictor since the phoneme duration is a discrete integer. In addition,
variational augmentation (Chen et al., 2020) is used to expand channel dimensions for expressive
flows in both the duration and pitch predictor. The stochastic and pitch predictor are trained by
maximizing their variational lower bounds (ELBO). Details of the duration and pitch predictor are
described in Appendix A.

B PROOF OF Proposition 1

**Objective** From equation [45] in (Kulesza & Taskar, 2012), the marginal kernel of conditional
DPPs given the appearance of set A has a following form:

**_K[A]_** = I [(L + I ¯A[)][−][1][]][ ¯]A (14)
_−_

In addition, from equation [34] in Kulesza & Taskar (2012) the expected cardinality of Y given the
marginal kernel K is:


_λn_ (15)

_λn + 1 [= tr(][K][)]_


E[|Y |] =


_n=1_


From equation 14, 15, the expected cardinality of conditional DPPs given the appearance of set A
is:

E[|Y |] = tr(K[A]) = tr(I − [(L + I ¯A[)][−][1][]][ ¯]A[)] (16)

**Derivative** For the proof, we will start with a following lemma:

**Lemma 1. Given a matrix E and non-singular matrix A, following equation holds:**

_∂_

(17)
_∂A_ [tr(][E][T][A][−][1][E][) =][ −][(][A][−][1][EE][T][A][−][1][)][T]


_Proof. First, consider_


_∂_

_∂Aij_ [t][r][(][E][T][A][−][1][E][)][. Since][ trace][ and derivative operator are interchangeable,]

_∂_ _∂_

tr(E[T]A[−][1]E) = tr( (E[T]A[−][1]E))
_∂Aij_ _∂Aij_ (18)

= tr(E[T]A[−][1][ ∂A] _A[−][1]E)_
_−_ _∂Aij_


-----

By setting _∂A∂Aij_ [=][ E][ij][ where][ E][ij][ denotes the matrix whose][ (][i, j][)][ component is][ 1][ and][ 0][ elsewhere]

and C = −E[T]A[−][1]E[ij]A[−][1]E,

tr(E[T]A[−][1]E[ij]A[−][1]E) = _Ci[′]_ _i[′]_
_−_

_i[′]_

X


(E[T]A[−][1])ii[′] _k1_ _Ek[ij]1k2_ [(][A][−][1][E][)]k2i[′]
_k2_

X


= −

= −

= −


_i[′]_


_k1_


(19)

(20)


(E[T]A[−][1])i[′] _i(A[−][1]E)ji[′] =_
_i[′]_

X

(A[−][T]E)ii[′] (E[T]A[−][T])i[′] _j = −(A[−][T]EE[T]A[−][T])ij_
_i[′]_

X


= (A[−][1]EE[T]A[−][1])[T]ij
_−_

=⇒ _∂A∂ij_ [tr(][E][T][A][−][1][E][) =][ −][(][A][−][1][EE][T][A][−][1][)]ij[T] [.]

Now, with respect to set a A whose cardinality is p and matrix L ∈ R[(][p][+][q][)][×][(][p][+][q][)]

_∂_

_A[)][−][1][]][ ¯]A[)]_
_∂θ_ [tr(][I][ −] [[(][L][ +][ I][ ¯]

= _A[)][−][1][]][ ¯]A[)]_
_−_ _∂θ[∂]_ [tr([(][L][ +][ I][ ¯]


= _A[)][−][1][E][)][,]_
_−_ _∂θ[∂]_ [tr(][E][T][(][L][ +][ I][ ¯]

_∈_ R[(][p][+][q][)][×][q]. Then by Lemma 1.


**0**
where E denotes
**_Iq_**



_′_
_A[)][−][1][E][) = ((][L][ +][ I][ ¯]A[)][−][1][EE][T][(][L][ +][ I][ ¯]A[)][−][1][)][T][L]_ (θ)

_−_ _∂θ[∂]_ [tr(][E][T][(][L][ +][ I][ ¯] (21)

_′_
= ((L + I ¯A[)][−][1][I][q][(][L][ +][ I][ ¯]A[)][−][1][)][T][L] (θ)

The proof of Proposition 1 is now finished.


-----

C ALGORITHMS FOR PDM TRAINING AND INFERENCE OF DPP-TTS

C.1 PDM TRAINING

**Algorithm 1 Training of PDM**

**Require: TextEncoder f** (·), PDM parameterized θ, a prosody predictor (trained flow) g(·), number
of candidates nc, noise scale ϵ

1: while not converged do
2: _htext_ _f_ (text)

3: Split h ←text into [htarget, hcontext]

4:5: Sample latent codeGet prosodic features of contexts: zcontext ∈ R[T] dwith noise scalecontext _g[−][1]( ϵhcontext, zcontext)_

6: Get quality of contexts: qcontext = Density estimation ← (dcontext)

7: Sample latent codes ztarget with noise scale ϵ

8: Get latent codes after PDM: ztarget ← PDM(ztarget) ∈ R[n][c][×][T]

9: Get nc prosodic features of targets: (d[1]target[, d]target[2] _[, ...d]target[n][c]_ [)]

10: Get quality of targets: qtarget = Density estimation(dtarget)

11: Concatenate contexts and targets: [qcontext, qtarget], [dcontext, dtarget]

12: Build the kernel of conditional DPPs:
**_L_** Build kernel([qcontext, qtarget], [dContext, dtarget])
_←_

14:13: UpdateCalculate the loss function: θ with the gradient _LLdiversitydiversity ←−tr(I −_ [(L + IA¯ [)][−][1][]]A[¯] [)]
_∇_

15: end while


C.2 INFERENCE OF DPP-TTS

**Algorithm 2 Inference of DPP-TTS**

**Require: TextEncoder f** (·), Decoder h(·), PDM, a prosody predictor g(·), noise scale ϵ

1: htext _f_ (text)
2: Split h ←text into [htarget, hcontext]
3:4: Sample latent code Get prosodic features of contexts: zcontext ∈ R[rmT] dcontextwith noise scaleg[−][1](hcontext ϵ _, zcontext)_
5: Get quality of contexts: qcontext = Density estimation ← (dcontext)
6: Sample latent codes ztarget with noise scale ϵ
7: Get latent codes after PDM: ztarget ← PDM(ztarget) ∈ R[n][c][×][T]

8: Get nc prosodic features of targets: (d[1]target[, d][2]target[, ...d]target[n][c] [)]

9: Get quality of targets: qtarget = Density estimation(dtarget)

10: Concatenate contexts and targets: [qcontext, qtarget], [dcontext, dtarget]
11: Build the kernel of conditional DPPs:
**_L_** Build kernel([qcontext, qtarget], [dContext, dtarget])
_←_

12: Perform the MAP inference d[∗] _←_ arg maxd logdet(Ld∪dcontext )
13: Synthesize wavs with prosodic features: y _h(htext, d[∗])_
_←_


-----

D SAMPLE PARAGRAPH FOR THE SIDE-BY-SIDE COMPARISON TEST

Known individually and collectively as Shai-Hulud, the sandworms are
these supermassive beings that plow through the deserts of Arrakis,
consuming everything that dares venture unprepared into their territory. The worms are what make harvesting spice so difficult because
they tend to eat whatever tools off-worlders use to mine it. They are
also sacred to the Fremen, who seem to know ways to navigate around
them, and, somehow, they’re linked to the creation of spice. Think of
them as big honking metaphors for the sublime powers of nature that
loom beyond human understanding, like a desert full of Moby Dicks

E ADJUSTING THE EXTENT OF VARIATION


(b) Pitch plot
with w= 2


(c) Log duration plot
with w=10


(d) Log duration plot
with w=2


(a) Pitch plot
with w = 10


Figure 5: Pitch and log duration plots with different values of quality weight. Green plots indicate
the prosody prediction before MAP inference of DPP and red plots indicate the prosody prediction
after MAP inference of DPP.


-----

