# PEARL: DATA SYNTHESIS VIA PRIVATE EMBEDDINGS
## AND ADVERSARIAL RECONSTRUCTION LEARNING

**Seng Pei Liew, Tsubasa Takahashi, Michihiko Ueno**
LINE Corporation
{sengpei.liew,tsubasa.takahashi,michihiko.ueno}@linecorp.com

ABSTRACT

We propose a new framework of synthesizing data using deep generative models in
a differentially private manner. Within our framework, sensitive data are sanitized
with rigorous privacy guarantees in a one-shot fashion, such that training deep
generative models is possible without re-using the original data. Hence, no extra
privacy costs or model constraints are incurred, in contrast to popular gradient
sanitization approaches, which, among other issues, cause degradation in privacy
guarantees as the training iteration increases. We demonstrate a realization of
our framework by making use of the characteristic function and an adversarial
re-weighting objective, which are of independent interest as well. Our proposal
has theoretical guarantees of performance, and empirical evaluations on multiple
datasets show that our approach outperforms other methods at reasonable levels of
privacy.

1 INTRODUCTION

Synthesizing data under differential privacy (DP) (Dwork (2006; 2011); Dwork et al. (2014)) enables
us to share the synthetic data and generative model with rigorous privacy guarantees. Particularly,
DP approaches of data synthesis involving the use of deep generative models have received attention
lately (Takagi et al. (2021); Xie et al. (2018); Torkzadehmahani et al. (2019); Frigerio et al. (2019);
Jordon et al. (2019); Chen et al. (2020); Harder et al. (2021)).

Typically, the training of such models utilizes gradient sanitization techniques (Abadi et al. (2016))
that add noises to the gradient updates to preserve privacy. While such methods are conducive to
deep learning, due to composability, each access to data leads to degradation in privacy guarantees,
and as a result, the training iteration is limited by the privacy budget. Recently, Harder et al. (2021)
has proposed DP-MERF, which first represents the sensitive data as random features in a DP manner
and then learns a generator by minimizing the discrepancy between the (fixed) representation and
generated data points. DP-MERF can iterate the learning process of the generator without further
consuming the privacy budget; however, it is limited in the learning and generalization capabilities
due to its fixed representation. In this work, we seek a strategy of training deep generative models
_privately that is able to resolve the aforementioned shortcomings, and is practical in terms of privacy_
_(e.g., usable image data at ϵ ≃_ 1.)

We propose a private learning framework called PEARL (Private Embeddings and Adversarial
Reconstruction Learning). In this framework, we have i) no limitation in learning iterations, and
ii) well-reconstruction capability. Towards those preferable properties, our framework first obtains
(1) informative embedding of sensitive data and (2) auxiliary information (e.g., hyperparameters)
useful for training, both in a differentially private manner, then (3) the generative model is trained
implicitly like GANs via the private embedding and auiliary information, where the learning is based
on a stochastic procedure that generates data, and (4) a critic distinguishing between the real and
generated data. The overview of PEARL is illustrated in Fig. 1.

As a concrete realization of PEARL, We first identify that the characteristic function (CF) representation of data can be sanitized as the private embedding of PEARL. Consequently, it is possible to train
deep generative models using an appropriately defined metric measuring the discrepancy between the
real (but sanitized) and generated data distribution based on the CF without re-using the original data.
As will be explained in detail in later Sections, the generative modelling approach using CFs also


-----

(3)




Generator


|Adv. Recon. Learner|Synthesized 1 Synthesized 2 … Synthesized k Critic (4)|Synthesized 1|
|---|---|---|
|||Synthesized 2|
|||Critic|


(1) Privately

Embedded
1

Privately
Embedded
2

Sensitive …
Data Privately

Embedded
k

Aux

(2)

DP flow (one-shot)

Data flow (one-shot)

Training loop

Figure 1: PEARL is a private learning framework that has i) no limitation in learning iterations, and
ii) well-reconstruction capability. Towards those preferable properties, our framework first obtains (1)
private embedding and (2) auxiliary information from the sensitive data, then (3) trains a generator
while (4) optimizing a critic to distinguish between the real and generated data.

involves sampling “frequencies” from an ad hoc distribution, to project the data to the embedding. It
is desirable to optimize the sampling distribution to better represent the data as an embedding, but
the naive way of optimizing it would require re-accessing the data via sampling, coming at a cost of
privacy budget. Henceforth, we also propose to incorporate a privacy-preserving critic to optimize the
sampling strategy, which, through re-weighting, chooses the best representation from a fixed samples
of frequencies without extra privacy costs.

To this end, we propose the following minimax optimization training objective:


_k_

_ω(ti)_

inf ΦPr (ti) ΦQθ (ti) _._ (1)
_θ∈Θ_ _ω[sup]∈Ω_ _i=1_ _ω0(ti)_ _−_ [b]

X

See later parts for notations and details. Theoretically, we show that our proposed objective hase [2]
properties similar to those that are suited to training GANs, i.e., continuity and differentiability
of the generator’s parameters, and continuity in weak topology. We also prove the consistency of
our privacy-preserving sampling strategy at the asymptotic limit of infinite sampling. Empirical
evaluations show that PEARL is able to high-quality synthetic data at reasonable privacy levels.

**Related works. Traditional methods of synthesizing data are mainly concerned with discrete data or**
data preprocessed to the discrete form (Zhang et al. (2017); Qardaji et al. (2014); He et al. (2015);
Chen et al. (2015); Cai et al. (2021); Zhang et al. (2021)), whereas we are interested in more general
methods involving continuous data. Deep generative models under the DP setting are suitable for
this type of tasks (Takagi et al. (2021); Xie et al. (2018); Torkzadehmahani et al. (2019); Frigerio
et al. (2019); Jordon et al. (2019); Chen et al. (2020); Harder et al. (2021)). The private training
of deep generative models is usually performed using gradient sanitization methods. An exception
is DP-MERF (Harder et al. (2021)), which is closest to our work. There, random features used
to approximate the maximum mean discrepancy (MMD) objective are privatized and utilized for
training a generator. PEARL, which, as a realization, uses CFs, may be viewed as a generalization of
DP-MERF. Additionally, PEARL has several distinctive features which are lacking in DP-MERF.
The first lies in the introduction of a privacy-preserving critic, which leads to an improvement of
performance. The second is the private selection of the parameter of the sampling distribution, which
is also shown to be vital. Moreover, DP-MERF uses non-characteristic kernels when treating tabular
data, in contrast to ours, which is characteristic and has guarantees in convergence. We finally note
that generative models using CFs but only non-privately have been explored before (Ansari et al.
(2020); Li et al. (2020)) .

**Contributions. Our contribution in this paper is three-fold: (i) We propose a general framework**
called PEARL, where, unlike gradient sanitization methods, the generator training process and
iteration are unconstrained; reliance on ad-hoc (non-private) hyperparameter tuning is reduced by
extracting hyperparameters (auxiliary information) privately. (ii) We demonstrate a realization of our
framework by making use of the characteristic function and an adversarial re-weighting objective.
(iii) Our proposal has theoretical guarantees of performance, and empirical evaluations show that our
approach outperforms competitors at reasonable levels of privacy (ϵ ≃ 1).


-----

2 PRELIMINARIES

This Section gives a brief review of essential preliminaries about differential privacy, characteristic
function and the related notations.

2.1 DIFFERENTIAL PRIVACY

**Definition 1 ((ϵ, δ)-Differential Privacy). Given privacy parameters ϵ ≥** 0 and δ ≥ 0, a randomized
_mechanism, M : D →R with domain D and range R satisfies (ϵ, δ)-differential privacy (DP) if for_
_any two adjacent inputs d, d[′]_ _∈D and for any subset of outputs S ⊆R, the following holds:_
Pr[M(d) ∈ _S] ≤_ _e[ϵ]_ _· Pr[M(d[′]) ∈_ _S] + δ._ (2)

We next consider concrete ways of sanitizing certain outputs with DP. A typical paradigm of DP is
applying the randomized mechanism, M, to a certain deterministic function f : D → R such that the
output of f is DP. The noise magnitude added by M is determined by the sensitivity of f, defined as
∆d[′]fare any adjacent pairs of dataset. Laplacian and Gaussian mechanisms are the standard randomized = supd,d′∈D ∥f (d) − _f_ (d[′])∥, where || · || is a norm function defined on f ’s output domain. d and
mechanisms. We primarily utilize the Gaussian mechanism in this paper (Dwork et al. (2014)):
**Definition 2 (Gaussian Mechanism). Let f : X →** R be an arbitrary function with sensitivity ∆f _._
_The Gaussian Mechanism,_ _σ, parameterized by σ, adds noise to the output of f as follows:_
_M_
_σ(x) = f_ (x) + (0, σ[2]I). (3)
_M_ _N_
One of the most important properties of DP relevant to our work is the post-processing theorem
(Dwork et al. (2014)):
**Theorem 1 (Post-processing Theorem). Let M : D →R be (ϵ, δ)-DP and let f : R →R[′]** _be an_
_arbitrary randomized function. Then, f ◦M : D →R[′]_ _is (ϵ, δ)-DP._

It ensures that the DP-sanitized data can be re-used without further consuming privacy budgets.

2.2 CHARACTERISTIC FUNCTIONS

Characteristic function (CF) is widely utilized in statistics and probability theory, and perhaps is best
known to be used to prove the central limit theorem (Williams (1991)). The definition is as follows.
**Definition 3 (Characteristic Function). Given a random variable X ⊆** R[d] _and P as the probability_
_measure associated with it, and t ∈_ R[d], the corresponding characteristic function (CF) is given by

ΦP(t) = Ex∼P[e[i][t][·][x]] = (4)

R[d][ e][i][t][·][x][d][P][.]

Z

Here, i is the imaginary number. From the signal processing point of view, this mathematical
operation is equivalent to the Fourier transformation, and ΦP(t) is the Fourier transform at frequency
**t. It is noted that we deal with the discrete approximation of CFs in practice. That is, given a dataset**
with n i.i.d. samples, {xj}j[n]=1 [from][ P][, the empirical CF is written as][ b]ΦP(t) = _n[1]_ _ni=j_ _[e][i][t][·][x][j]_ [. We]

next introduce characteristic function distance (CFD) (Heathcote (1972); Chwialkowski et al. (2015)):
P
**Definition 4 (Characteristic Function Distance). Given two distributions P and Q of random variables**
_residing in R[d], and ω a sampling distribution on t ∈_ R[d], the squared characteristic function distance
_(CFD) between P and Q is computed as:_

_C[2](P, Q) = Et∼ω(t)[_ ΦP(t) − ΦQ(t) ] = ΦP(t) − ΦQ(t) _ω(t)dt._ (5)

R[d]

Z

**Notations.** Let us make a short note on the notations before continuing. Let[2] [2] _k be the num-_
ber of t drawn from ω and P be the probability measure of a random variable. We group
the CFs associated to P of different frequencies, (Φ[b] P(t1), . . ., ΦP(tk))[⊤] more compactly as
**_φP(x). To make the dependence of_** **_φP(x) on the sampled data explicit, we also use the fol-_**

lowing notation: **_φP(x) =_** _n[1]_ _nj=1_ **_φP(xj). We notice that_** **_φP(x[b])_** 2 _km=1_ ΦP(tm) =

b [b] _∥_ [b] _∥_ _≡_ _[|][b]_ _|[2]_

_k_ _k_
_m=1_ _l=1_ _[e][i][t][m][·][x][l]_ _[/n][|][2][ ≤]P_ _m=1_ _l=1_ [1][/n][|][2][ =] _√k, where the norm is taken over theqP_

_[|][ P][n]_ [b] [b] _[|][ P][n]_

(complex) frequency space. With a slight abuse of notation, we abbreviateqP qP **_φP as_** **_φ when there is no_**
ambiguity in the underlying probability measure associated with the CF.

[b] [b]


-----

3 GENERATIVE MODEL OF PEARL

Let us describe generative modelling under the PEARL framework. The realization is summarized by
the following Proposition:
**Proposition 1. Let the real data distribution be Pr, and the output distribution of an implicit**
_generative model Gθ by Qθ. Let n be the total number of real data instances and consider releasing k_
_CFs from the dataset. Then, Gθ trained to optimize the empirical CFD, min_ (Pr, Qθ) with the CF
_θ_ Θ _C[2]_

_sanitized according to the Gaussian mechanism (Defn. 3) with sensitivity∈ 2√k/n satisfies (ϵ, δ)-DP,_

b

_where σ ≥_ 2 log(1.25/δ)/ϵ.
p

_Proof. In the following we give a detailed explanation of the above Proposition. The first step of_
PEARL is projecting the data to the CF as in Eq. 4, where the number of embeddings is the number
of frequency drawn from ω(t).

We note that CF has several attractive properties. CF is uniformly continuous and bounded, as can be
seen from its expression in Eq. 4. Unlike the density function, the CF of a random variable always
exists, and the uniqueness theorem implies that two distributions are identical if and only if the CFs
of the random variables are equal (Lukacs (1972)).

The CF is sanitized with DP by applying the Gaussian mechanism (Defn. 3) to _φ(x):_

**_φ(x) =_** **_φ(x) +_** (0, ∆[2] (6)
_N_ **_φ(x)[σ][2][I][)][,]_** [b]

where we write the sanitized CF ase **_φ(x); ∆[b]_** **_φ(x)_** [denotes the sensitivity of the CF,]b _[ σ][ denotes the noise]_
scale which is determined by the privacy budget, (ϵ, δ).
b

[e]

The calculation of sensitivity is tractable (no ad-hoc clipping commonly required by gradient sanitization methods) Without loss of generality, consider two neighboring datasets of size n where only the
last instance differs (xn ̸= x[′]n[). The sensitivity of][ b]φ(x) may then be calculated as

∆φ(x) [= max], _n1_ _n_ **_φ(xj) −_** _n[1]_ _n_ **_φ(x[′]j[)]_** = maxxn,x[′]n _n[1]_ **_φ(xn) −_** _n[1]_ **_φ(x[′]n[)]_** 2 [=][ 2]√nk _[,]_
_D_ _D[′]_ _j=1_ _j=1_
b X X 2

where we have used triangle inequality andb b **_φ(_** ) 2 _√k. It can be seen that the sensitivity is[b]_ [b]

proportional to the square root of the number of embeddings (releasing more embeddings requires ∥ [b] _·_ _∥_ _≤_
more DP noises to be added to **_φ) but inversely proportional to the dataset size, which is important_**
for controlling the magnitude of noise injection at practical privacy levels, as will be discussed in
later Sections.

[b]

We would like to train a generative model where the CFs constructed from the generated data
distribution, Y ⊆ R[d], matches those (sanitized) from the real data distribution, X ⊆ R[d]. A natural
way of achieving this is via implicit generative modelling (MacKay (1995); Goodfellow et al. (2014)).
We introduce a generative model parametrized by θ, Gθ : Z → R[d], which takes a low-dimensional
latent vector z ∈Z sampled from a pre-determined distribution (e.g., Gaussian distribution) as the
input.

In order to quantify the discrepancy between the real and generated data distribution, we use the CFD
defined in Eq. 5. Empirically, when a finite number of frequencies, k, are sampled from ω, C[2](P, Q)
is approximated by

_k_

2

(P, Q) = [1] ΦP(ti) ΦQ(ti) **_φP(x)_** **_φQ(x)_** (7)
_C[2]_ _k_ _−_ [b] _≡_ _−_ [b] 2 _[,]_

_i=1_

X

where ΦP(t) and ΦbQ(t) are the empirical CFs evaluated from i.i.d. samples of distributionsb [2] [b] P and Q
respectively. The training objective of the generator is to find the optimal θ ∈ Θ that minimizes the
empirical CFD:[b] minθ [b]Θ _C[2](Pr, Qθ). It can be shown via uniqueness theorem that as long as ω resides in_
_∈_

R[d], (P, Q) = 0 P = Q (Sriperumbudur et al. (2011)). This makes CFD an ideal distance
_C_ _⇐⇒b_
metric for training the generator.


-----

**Optimization procedure. The generator parameter, θ, is updated as follows.** **_φPr_** (x) is first sanitized
to obtain **_φPr_** (x), as in Eq. 6. This is performed only for once (one-shot). Then, at each iteration, m
samples of z are drawn to calculate **_φQθ_** (x). Gradient updates on θ are performed by minimizing the[b]

CFD, **_φP[e]r_** (x) **_φQθ_** (x)
_−_ [b] 2[.]

[b]

We note that only the first term,[e] **_φPr_** (x), has access to the real data distribution,, of which privacy
_X_
is of concern. Then, by Thm. 1, Gθ trained with respect to **_φPr_** (x) is DP. Furthermore, unlike gradient
sanitization methods, the training of[e] _Gθ is not affected by network structure or training iterations._
Once the sanitized CF is released, there is no additional constraints due to privacy on the training

[e]
procedure.

**DP release of auxiliary information. Auxiliary information, e.g., hyperparameters regarding to the**
dataset useful for generating data with better quality may be obtained under DP. In our case, one may
extract auxiliary information privately from the dataset to select good parameters for the sampling
distribution ω(t). This will be discussed in more detail in the next Sections.

Another example is the modelling of tabular data. Continuous columns of tabular data consisting of
multiple modes may be modelled using Gaussian mixture models (GMMs) (Xu et al. (2019)). GMMs
are trainable under DP using a DP version of the expectation-maximization algorithm (Park et al.
(2017)). The total privacy budget due to multiple releases of information is accounted for using the
Rényi DP composition. See App. A for the definition of Rényi DP.

4 ADVERSARIAL RECONSTRUCTION LEARNING

This Section is devoted to proposing a privacy-preserving critic for optimizing ω(t), while giving
provable guarantees of performance.

Back to Eq. 5 and Eq. 7, we note that choosing a “good” ω(t) or a “good” set of sampled frequencies
is vital at helping to discriminate between P and Q. For example, if the difference between P and Q
lies in the high-frequency region, one should choose to use t with large values to, in the language of
two-sample testing, improve the test power.

**Adversarial objective. If the resulting empirical CFD remains small due to under-optimized ω(t),**
while the two distributions still differ significantly, the generator cannot be optimally trained to
generate high-quality samples resembling the real data. Hence, we consider training the generator by,
in addition to minimizing the empirical CFD, maximizing the empirical CFD using an adversarial
_objective which acts as a critic, where the empirical CFD is maximized by finding the best sampling_
distribution. We consider a training objective of the form

2
inf _ω_ (Pr, Qθ), (8)
_θ∈Θ_ _ω[sup]∈Ω_ _C_

where Ω is some set of probability distribution space of which the sampling distributionc _ω lives in._

**Privacy-preserving optimization. It is intractable to directly optimize ω in the integral form as in**
Eq. 5. In this work, we opt for a privacy-preserving one-shot sampling strategy where once the
private data is released (by sampling from ω), optimization is performed without further spending the
privacy budget. We believe that such a new formulation of distribution learning is of independent
interest as well.

**Effectiveness of the critic. To further motivate why it is preferable to introduce an adversarial**
objective as in Eq. 8, we present a simple demonstration through the lens of two-sample testing
(Chwialkowski et al. (2015); Jitkrittum et al. (2016)) using synthetic data generated from Gaussian
distributions. We generate two unit-variance multivariate Gaussian distributions P, Q, where all
dimensions but one have zero mean. We conduct a two-sample test using the CF to distinguish
between the two distributions, which gets more difficult as the dimensionality increases. We test
if the null hypothesis where the samples are drawn from the same distribution is rejected. Higher
rejection rate indicates better test power.

Note that the first dimension (where the distribution differs) of the frequency used to construct CFs
is the most discriminative dimension for distinguishing the distributions. We consider three sets of


-----

frequencies: “unoptimized”, “normal”, and “optimized”, where the set of “unoptimized” frequencies
is optimized with re-weighting. More details can be found in App. C.


Fig. 2 shows the hypothesis rejection rate versus
the number of dimensions for the three cases
considered above. As can be observed, the “optimized” case gives overall the best test power.
While this experiment is somewhat contrived,
it can be understood that although both “unoptimized” and “optimized” sets of frequencies
contain the discriminative t0, the re-weighting
procedure of selecting the most discriminative
CF improves the test power significantly. Even
without re-sampling from a “better” ω(t), reweighting the existing frequencies can improve
the test power. The fact that re-weighting can improve the test power is crucial privacy-wise because the altenative method, re-sampling, causes
degradation in privacy.


1.0

0.8

0.6

0.4

0.2

0.0

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
|||||||||
|||||||||
|||||||||
|||||||||
||Unop Norm Opti|timized al mized||||||
|||||||||


20 25 30 35 40 45 50


Dimensions

Figure 2: Increased test power upon optimization
(green) in two-sample test.


**Proposal. Recall that the empirical CFD,** (Pr, Qθ) = _k[1]_ _ki=1_ ΦPr (ti) ΦQθ (ti), is obtained
_C[2]_ _−_ [b]

by drawing k frequencies from a base distribution ω0. Our idea is to find a (weighted) set of
frequencies that gives the best test power from the drawn set. We propose Eq. 1 as the optimization[b] P e [2]
objective, restated below:


_k_

_ω(ti)_

inf ΦPr (ti) ΦQθ (ti) _._
_θ∈Θ_ _ω[sup]∈Ω_ _i=1_ _ω0(ti)_ _−_ [b]

X

Note that the generator trained with this objective still satisfies DP as given in Prop. 1 due to Thm. 1.e [2]
The following Lemma ensures that the discrete approximation of the inner maximization of Eq. 1
approaches the population optimum as the number of sampling frequency increases (k →∞):
**Lemma 1. Let ω0 be any probability distribution defined on R[d], and let f : R[d]** _→R[′]_ _be any function._
_Also let t ∈R[d]_ _and ω[∗]_ _be the maximal distribution of ω with respect to Eω[f_ (t)] ≡ _f_ (t)ω(t)dt.
_Assume that the empirical approximation_ Eω[f (t)] Eω[f (t)] at the asymptotic limit for any ω.
_→_ R
_Then,_ Eω0 [f (t) _[ω]ω[∗]0([(]t[t])[)]_ []][ →] [E][ω][∗] [[][f] [(][t][)]][ at the asymptotic limit as well.]

[b]

The proof is in App. B, and is based on importance sampling. Empirically, we find that treating ω(ti)

[b]

as a free parameter and optimizing it directly does not lead to improvement in performance. This may
be due to the optimization procedure focusing too much on uninformative frequencies that contain
merely noises due to DP or sampling. We perform parametric optimization instead, that is, e.g., we
perform optimization with respect to {µ, σ} if ω is of the Gaussian form, N (µ, σ[2]).

**Performance guarantees. Let us discuss the theoretical properties of Eq. 8. The objective defined**
in Eq. 8 shares beneficial properties similar to those required to train good GANs, first formulated
in (Arjovsky et al. (2017)). First, the generator learns from a distance continuous and differentiable
almost everywhere within the generator’s parameters. Second, the distance is continuous in weak
topology, and thus provides informative feedback to the generator (different from, e.g., the JensenShannon divergence, which does not satisfy this property). We make assumptions similar to those
given in (Arjovsky et al. (2017)), and state the first theorem as follows.
**Theorem 2. Assume that Gθ(z) is locally Lipschitz with respect to (θ, z); there exists L((θ, z) satis-**
_fying Ez [L(θ, z)] <_ _; and supω_ Ω Eω [ **t** ] < _for all t. Then, the function supω_ Ω _ω[(][P][r][,][ Q][θ][)]_
_∞_ _∈_ _|_ _|_ _∞_ _∈_ _C[2]_
_is continuous in θ ∈_ Θ everywhere, and differentiable in θ ∈ Θ almost everywhere.

Note that the local Lipschitz assumptions are satisfied by commonly used neural network components,
such as fully connected layers and ReLU activations. The continuity and differentiability conditions
with respect to θ stated above allow Gθ to be trained via gradient descent. The theorem related to
continuity in weak topology is the following:
**Theorem 3. Let P be a distribution on X and (Pn)n∈N be a sequence of distributions on X** _. Under_
_the assumption supω_ Ω Eω(t) [ **t** ] < _, the function supω_ Ω _ω[(][P][n][,][ P][)][ is continuous in the]_
_∈_ _∥_ _∥_ _∞_ _∈_ _C[2]_


-----

_D_ _D_ _D_
_weak topology, i.e., if Pn_ P, then supω Ω _ω[(][P][n][,][ P][)]_ 0, where _denotes convergence in_
_−→_ _∈_ _C[2]_ _−→_ _−→_
_distribution._

Weakness is desirable as the easier (weaker) it is for the distributions to converge, the easier it will be
for the model to learn to fit the data distribution. The core ideas used to prove both of these theorems
are the fact that the difference of the CFs (which is of the form e[ia]) can be bounded as follows:
_|e[ia]_ _−_ _e[ib]| ≤|a −_ _b|, and showing that the function is locally Lipschitz, which ensures the desired_
properties of continuity and differentiability. See App. B for the full proofs.

5 EXPERIMENTS

5.1 EXPERIMENTAL SETUP

To test the efficacy of PEARL, we perform empirical evaluations on three datasets, namely MNIST
(LeCun et al. (2010)), Fashion-MNIST (Xiao et al. (2017)) and Adult (Asuncion & Newman (2007)).
Detailed setups are available in App. H.

**Training Procedure. As our training involves minimax optimization (Eq. 1), we perform gradient**
descent updates based on the minimization and maximization objectives alternately. We use a zeromean diagonal standard-deviation Gaussian distribution, N (0, diag(σ[2]) as the sampling distribution,
_ω. Maximization is performed with respect to σ. Let us give a high-level description of the_
full training procedure: draw k frequencies from ω, calculate the CFs with them and perform
DP sanitization, train a generator with the sanitized CFs using the minimax optimization method.
The pseudo-code of the full algorithm is presented in App. E. We further give the computational
complexity analysis of our algorithm in App. F.

**DP release of auxiliary information. We also note that applying maximization (re-weighting) on**
a randomly selected set of frequencies would not work well. We initially fix the inverse standard
deviation of the base distribution, ω0, to be the DP estimate of the mean of the pairwise distance of
the data, to obtain (privately) a spectrum of frequencies that overlaps with the "good" frequencies.
This is motivated by the median heuristic (Garreau et al. (2017)). Mean is estimated instead of median
as its sensitivity is more tractable when considering neighboring datasets. We obtain ∆= 2√d/n

as its sensitivity, where d and n are the data dimension and the number data instances respectively.
See App. D for the derivation. [1] Using this DP estimate, we also investigate as a ablation study the
generator performance with minimization objective only (w/o critic). The full PEARL framework
includes the use of the critic to find the "best" frequencies among the selected spectrum to distinguish
the distributions (and overall perform a minimax optimization).

**Evaluation Metrics. In the main text and App. I, we show qualitative results, i.e., synthetic images**
(image dataset) and histograms (tabular dataset) generated with PEARL. Furthermore, for image
datasets, the Fréchet Inception Distance (FID) (Heusel et al. (2017)) and Kernel Inception Distance
(KID) (Bi´nkowski et al. (2018)) are used to evaluate the quantitative performance. For tabular data,
we use the synthetic data as the training data of 10 scikit-learn classifiers (Pedregosa et al. (2011))
and evaluate the classifiers’ performances on real test data. The performance indicates how well
synthetic data generalize to the real data distribution and how useful synthetic data are in machine
learning tasks. ROC (area under the receiver operating characteristics curve) and PRC (area under
the precision recall curve) are the evaluation metrics. Definitions of FID and KID are in App. G.

5.2 EVALUATION DETAILS

**MNIST and Fashion-MNIST. Privacy budget is allocated equally between the sanitization of CFs**
and the release of auxiliary information. [2] On a single GPU (Tesla V100-PCIE-32GB), training
MNIST (with 100 epochs) requires less than 10 minutes.

1We note that DP-MERF also uses a sampling distribution. Since privacy budget is not allocated for calculating the parameters of the sampling distribution there, we fix its sampling distribution to be2We expect that CF sanitization requires more privacy allocation as it is involved in doing the heavy lifting N (0, diag(1)).
of training the model. We find that allocating around 80-90% of the budget to CF sanitization can increase the
performance by around 15%. Nevertheless, we utilize an equal share of privacy budget for fairness reasons, as
tuning the budget allocation would require re-using the private data and this can lead to privacy leakage.


-----

(a) ϵ = 0.2 (MNIST) (b) ϵ = 1 (MNIST) (c) ϵ = 10 (MNIST) (d) ϵ = ∞ (MNIST)

(e) ϵ = 0.2 (FMNIST) (f) ϵ = 1 (FMNIST) (g) ϵ = 10 (FMNIST) (h) ϵ = ∞ (FMNIST)

Figure 3: Generated MNIST and Fashion-MNIST samples for various values of ϵ and δ = 10[−][5].

Datasets Metrics DPGAN DPCGAN DP-MERF **Ours (w/o critic)** **Ours**

MNIST FID 22.1 ± 1.01 17.3 ± 2.90 49.9 ± 0.22 3.79 ± 0.06 **3.52 ± 0.06**
KID (×10[3]) 573 ± 41.2 286 ± 69.5 148 ± 46.2 77.8 ± 9.88 **70.5 ± 10.3**

Fashion-MNIST FID 16.6 ± 1.34 14.61 ± 1.75 37.0 ± 0.15 1.99 ± 0.04 **1.92 ± 0.04**
KID (×10[3]) 535 ± 61.5 425 ± 64.2 1220 ± 36.1 **24.0 ± 6.90** 26.9 ± 6.80

Table 1: FID and KID (lower is better) on image datasets at (ϵ, δ) = (1, 10[−][5]).

Some of the generated images of ours and baseline models are shown in Fig. 3, and more (including
baselines’) can be found in App. I. At the non-private limit, the image quality is worse than other
popular non-private approaches such as GANs due to two reasons. First, projecting data to a lower
dimension causes information loss. Second, our architecture does not have a discriminator-like
network as in the vanilla GAN framework. However, we notice that the quality of the images does
not drop much as the privacy level increases (except at ϵ = 0.2, where the quality starts to degrade
visibly) because the noise added to the CFs is small as it scales inversely proportional to the total
sample size. It also indicates that our approach works particularly well at practical levels of privacy.

We now provide quantitative results by performing evaluation at (ϵ, δ) = (1, 10[−][5]). Note that we
focus on this high privacy region despite the fact that many previous studies experimented with
_ϵ ≃_ 10, because recent analyses showed that realistic attacks can lead to privacy violations at low
privacy (Jagielski et al. (2020); Nasr et al. (2021)). Particularly, the claim “values of ϵ that offer
no meaningful theoretical guarantees” (ϵ ≫ 1) can be chosen in practice has been “refuted” in
general (Nasr et al. (2021)). This motivates us to perform evaluation at ϵ with meaningful theoretical
guarantees (ϵ ≃ 1).

At this privacy region, we compare PEARL with models using gradient sanitization and DP-MERF
(Harder et al. (2021)) as other methods do not produce usable images at single-digit ϵ. [3] We run
the experiment five times (with different random seeds each time), and for each time, 60k samples
are generated to evaluate the FID and KID. In Table 1, the FID and KID (average and error) of
DP-MERF, DPGAN, PEARL without critic, and PEARL are shown. It can be seen that PEARL
outperforms DP-MERF significantly, and the critic leads to improvement in the scores.

3In the gradient sanitized generative model literature, GS-WGAN (Chen et al. (2020)) is known to be the
state-of-the-art, but we are unable to train a stable model at ϵ = 1. Thus, we make comparisons with other
standard methods, namely DPGAN (Xie et al. (2018)) and DPCGAN (Torkzadehmahani et al. (2019)).


-----

Real data DP-MERF Ours


ROC PRC ROC PRC ROC PRC

LR 0.788 0.681 0.661 ± 0.059 0.542 ± 0.041 **0.752 ± 0.009** **0.641 ± 0.015**
Gaussian NB 0.629 0.511 0.587 ± 0.079 0.491 ± 0.06 **0.661 ± 0.036** **0.537 ± 0.028**
Bernoulli NB 0.769 0.651 0.588 ± 0.056 0.488 ± 0.04 **0.763 ± 0.008** **0.644 ± 0.009**
Linear SVM 0.781 0.671 0.568 ± 0.091 0.489 ± 0.067 **0.752 ± 0.009** **0.640 ± 0.015**
Decision Tree 0.759 0.646 **0.696 ± 0.081** 0.576 ± 0.063 0.675 ± 0.03 **0.582 ± 0.028**
LDA 0.782 0.670 0.634 ± 0.060 0.541 ± 0.048 **0.755 ± 0.005** **0.640 ± 0.007**
Adaboost 0.789 0.682 0.642 ± 0.097 0.546 ± 0.071 **0.709 ± 0.031** **0.628 ± 0.024**
Bagging 0.772 0.667 0.659 ± 0.06 0.538 ± 0.042 **0.687 ± 0.041** **0.601 ± 0.039**
GBM 0.800 0.695 0.706 ± 0.069 0.586 ± 0.047 **0.709 ± 0.029** **0.635 ± 0.025**
MLP 0.776 0.660 0.667 ± 0.088 0.558 ± 0.063 **0.744 ± 0.012** **0.635 ± 0.015**

**Average** 0.765 0.654 0.641 ± 0.044 0.536 ± 0.034 **0.721 ± 0.035** **0.618 ± 0.033**


Table 2: Quantitative results for the Adult dataset evaluated at (ϵ, δ) = (1, 10[−][5]).


10

10


10

10





10

10

10


10


10


10



Real
Ours
DP-MERF


|Col1|Real|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
||Real Ours DP-MERF|||||
|||||||
|||||||
|||||||
|1.0 2.0 “mari|3.0 4.0 category tal-sta|5.0 tus|”|6.0||


10 [2]

count

10 [3] Real

DP­MERF
Ours

20 30 40 50 60 70 80 90

value

(e) “age”


|Col1|Col2|Col3|Real|Col5|Col6|Col7|Col8|
|---|---|---|---|---|---|---|---|
||||Real Ours DP-MER|||||
|||||||||
|.0 c) “|1.0 r|el|2.0 category ation|3.0 sh|i|4.0 p|5. ” Real DP­MER Ours|


10 [4] RealDP­MERF

Ours

10 [5]

count 10 [6]

10 [7]

0 20000 40000 60000 80000 100000

value

(g) “capital-gain”

|Col1|Real|Col3|Col4|Col5|
|---|---|---|---|---|
||Real Ours DP-MERF||||
||||||
||||||
|0.0 1.0 (b|2.0 3.0 ) “|4.0 5.0 6.0 7.0 8 categor occup|.0 9.0 10.011.012.0 y ation” Real DP­M Ours|13.014 ERF|

|Col1|Col2|Real|Col4|Col5|Col6|Real|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|||Real Ours DP-MER||||Real Ours DP-MER|||
||||||||||
||||||||||
|0.0||(|d|1.0 ) “|2.0 category race”|3.0|Re DP Ou|4.0 al ­MER rs|


Real

10 [3] DP­MERFOurs

count 10 [4]

10 [5]

10 [6]

0 500 1000 1500 2000 2500 3000 3500 4000

value


10 [5] Real

DP­MERF

10 [6] Ours

count 10 [7]

10 [8]

0.0 0.2 0.4 0.6value 0.8 1.0 1.2 1e6


(f) “capital-loss”


(h) “fnlwgt”


Figure 4: Histogram plots for the the Adult dataset. Evaluation is performed at (ϵ, δ) = (1, 10[−][5]).

**Adult. The Adult dataset consists of continuous and categorical features. As data pre-processing,**
continuous features are scaled to [0, 1]. We also compare our results with DP-MERF, as DP-MERF
performs best among other existing methods (e.g., DPCGAN achieves ROC and PRC around 0.5;
see Table 4.) Again, auxiliary information and CFs share equal budget of privacy, and we perform
evaluation at (ϵ, δ) = (1, 10[−][5]). 11k synthetic samples are generated for evaluation.

Histogram plots of the attributes comparing the real and synthetic data is shown in Fig. 4. As can be
observed in the Figure, PEARL is able to model the real data better than DP-MERF, e.g., covering
more modes in categorical variables with less discrepancy in the frequency of each mode. PEARL is
also able to better model the descending trend of the continuous variable “age”. The average ROC
and PRC scores (average and error) are shown in Table 2. ROC and PRC scores based on PEARL’s
training data are shown to be closer to those based on real training data compared to DP-MERF. More
histogram plots in higher resolution are available in App. I. Also, in App. I, we present experimental
results of another tabular dataset (Credit), as well as evaluations with other metrics/tasks using the
synthetic data. PEARL still performs favorably under these different conditions. Overall, we have
demonstrated that PEARL is able to produce high-quality synthetic data at practical privacy levels.


6 CONCLUSION

We have developed a DP framework to synthesize data with deep generative models. Our approach
provides synthetic samples at practical privacy levels, and sidesteps difficulties encountered in
gradient sanitization methods. While we have limited ourselves to characteristic functions, it is
interesting to adopt and adapt other paradigms to the PEARL framework as a future direction.


-----

REFERENCES

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and
Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC
_Conference on Computer and Communications Security, pp. 308–318. ACM, 2016._

Abdul Fatir Ansari, Jonathan Scarlett, and Harold Soh. A characteristic function approach to deep
implicit generative modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision
_and Pattern Recognition, pp. 7478–7487, 2020._

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks.
In International conference on machine learning, pp. 214–223. PMLR, 2017.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007.

Mikołaj Bi´nkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD
GANs. arXiv preprint arXiv:1801.01401, 2018.

Kuntai Cai, Xiaoyu Lei, Jianxin Wei, and Xiaokui Xiao. Data synthesis via differentially private
markov random fields. Proceedings of the VLDB Endowment, 14(11):2190–2202, 2021.

Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitized approach
for learning differentially private generators. Advances in Neural Information Processing Systems,
33:12673–12684, 2020.

Rui Chen, Qian Xiao, Yu Zhang, and Jianliang Xu. Differentially private high-dimensional data
publication via sampling-based inference. In Proceedings of the 21th ACM SIGKDD International
_Conference on Knowledge Discovery and Data Mining, pp. 129–138, 2015._

Kacper P Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample
testing with analytic representations of probability measures. In Advances in Neural Information
_Processing Systems, pp. 1981–1989, 2015._

Andrea Dal Pozzolo, Olivier Caelen, Reid A Johnson, and Gianluca Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE Symposium Series on
_Computational Intelligence, pp. 159–166. IEEE, 2015._

Cynthia Dwork. Differential privacy. In Proceedings of the 33rd international conference on
_Automata, Languages and Programming-Volume Part II, pp. 1–12, 2006._

Cynthia Dwork. A firm foundation for private data analysis. Communications of the ACM, 54(1):
86–95, 2011.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends
_Theor. Comput. Sci., 9(3-4):211–407, 2014._

Herbert Federer. Geometric measure theory. Springer, 2014.

Lorenzo Frigerio, Anderson Santana de Oliveira, Laurent Gomez, and Patrick Duverger. Differentially
private generative adversarial networks for time series, continuous, and discrete open data. In IFIP
_International Conference on ICT Systems Security and Privacy Protection, pp. 151–164. Springer,_
2019.

Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median
heuristic. arXiv preprint arXiv:1707.07269, 2017.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural
_Information Processing Systems, pp. 2672–2680, 2014._

Frederik Harder, Kamil Adamczewski, and Mijung Park. Dp-merf: Differentially private mean
embeddings with randomfeatures for practical privacy-preserving data generation. In International
_conference on artificial intelligence and statistics, pp. 1819–1827. PMLR, 2021._


-----

Xi He, Graham Cormode, Ashwin Machanavajjhala, Cecilia M Procopiuc, and Divesh Srivastava.
Dpt: differentially private trajectory synthesis using hierarchical reference systems. Proceedings
_of the VLDB Endowment, 8(11):1154–1165, 2015._

CE Heathcote. A test of goodness of fit for symmetric random variables. Australian Journal of
_Statistics, 14(2):172–181, 1972._

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans
trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural
_information processing systems, 30, 2017._

Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine
learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:
22205–22216, 2020.

Wittawat Jitkrittum, Zoltán Szabó, Kacper P. Chwialkowski, and Arthur Gretton. Interpretable
distribution features with maximum testing power. In Advances in Neural Information Processing
_Systems, pp. 181–189, 2016._

James Jordon, Jinsung Yoon, and Mihaela van der Schaar. PATE-GAN: generating synthetic data
with differential privacy guarantees. In International Conference on Learning Representations,
_ICLR, 2019._

Achim Klenke. Probability theory: a comprehensive course. Springer Science & Business Media,
2013.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online].
_Available: http://yann.lecun.com/exdb/mnist, 2, 2010._

Shengxi Li, Zeyang Yu, Min Xiang, and Danilo P. Mandic. Reciprocal adversarial learning via
characteristic functions. In Advances in Neural Information Processing Systems, 2020.

Eugene Lukacs. A survey of the theory of characteristic functions. Advances in Applied Probability,
4(1):1–37, 1972.

David J.C MacKay. Bayesian neural networks and density networks. Nuclear Instruments and
_Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated_
_Equipment, pp. 73–80, 1995._

Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations
_Symposium (CSF), pp. 263–275. IEEE, 2017._

Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary
instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium
_on Security and Privacy (SP), pp. 866–882. IEEE, 2021._

Mijung Park, James R. Foulds, Kamalika Choudhary, and Max Welling. DP-EM: differentially
private expectation maximization. In Proceedings of the 20th International Conference on Artificial
_Intelligence and Statistics, pp. 896–904, 2017._

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier
Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn:
Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.

Wahbeh Qardaji, Weining Yang, and Ninghui Li. Priview: practical differentially private release of
marginal contingency tables. In Proceedings of the 2014 ACM SIGMOD international conference
_on Management of data, pp. 1435–1446, 2014._

Bharath K Sriperumbudur, Kenji Fukumizu, and Gert RG Lanckriet. Universality, characteristic
kernels and rkhs embedding of measures. Journal of Machine Learning Research, 12(7), 2011.

Shun Takagi, Tsubasa Takahashi, Yang Cao, and Masatoshi Yoshikawa. P3gm: Private highdimensional data release via privacy preserving phased generative model. In 2021 IEEE 37th
_International Conference on Data Engineering (ICDE), pp. 169–180. IEEE, 2021._


-----

Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private
synthetic data and label generation. In The IEEE Conference on Computer Vision and Pattern
_Recognition (CVPR) Workshops, June 2019._

Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential
privacy and analytical moments accountant. In The 22nd International Conference on Artificial
_Intelligence and Statistics, pp. 1226–1235. PMLR, 2019._

David Williams. Probability with martingales. Cambridge University Press, 1991.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking
machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.

Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative
adversarial network. arXiv preprint arXiv:1802.06739, 2018.

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular
data using conditional gan. In Advances in Neural Information Processing Systems, volume 32,
2019.

Jun Zhang, Graham Cormode, Cecilia M Procopiuc, Divesh Srivastava, and Xiaokui Xiao. Privbayes:
Private data release via bayesian networks. ACM Transactions on Database Systems (TODS), 42
(4):1–41, 2017.

Zhikun Zhang, Tianhao Wang, Ninghui Li, Jean Honorio, Michael Backes, Shibo He, Jiming Chen,
and Yang Zhang. PrivSyn: Differentially private data synthesis. In 30th USENIX Security
_Symposium (USENIX Security 21), pp. 929–946, 2021._


-----

A ADDITIONAL DEFINITIONS AND PREVIOUS RESULTS

**Definition 5 (Rényi differential privacy). A randomized mechanism M is said to satisfy ε-Rényi**
_differential privacy of order λ, when_

_λ_ 1[#]

1 _Pr[_ (d) = x] _−_
_Dλ(_ (d) (d[′])) = _M_ _ε_
_M_ _∥M_ _λ_ 1 [log][ E][x][∼M][(][d][)] _Pr[_ (d[′]) = x] _≤_

_−_ " _M_ 

_is satisfied for any adjacent datasets d and d[′]. Here, Dλ(P_ _∥Q) =_ _λ−1_ 1 [log][ E][x][∼][Q][[(][P] [(][x][)][/Q][(][x][))][λ][]][ is]

_the Rényi divergence. Furthermore, a ε-RDP mechanism of order λ is also (ε +_ [log 1]λ 1[/δ] _[, δ][)][-DP.]_

_−_

Next, we note that the Gaussian mechanism is (λ, _[λ][∆]2σ[2][2][f][ )][ 2]_ [-RDP (Mironov (2017)). The particular]

advantage of using RDP is that it gives a convenient way of tracking the privacy costs when a
sequence of mechanisms is applied. More precisely, the following theorem holds (Mironov (2017)):
**Theorem 4 (RDP Composition). For a sequence of mechanisms M1, ..., Mk s.t. Mi is (λ, εi)-RDP**
_∀i, the composition M1 ◦_ _... ◦_ _Mk is (λ,_ _i_ _[ε][i][)][-RDP.]_

We use the autodp package to keep track of the privacy budget (Wang et al. (2019)).

[P]


B PROOFS

B.1 PROOF OF THM. 2

_Proof. Let Pr denote the real data distribution, Qθ denote the data distribution generated by Gθ using_
a latent vector, z sampled from a pre-determined distribution, and ω a sampling distribution. | · |
denotes the modulus of ·. Then, the CFD between the distributions is

_Cω[2]_ [(][P][r][,][ Q][θ][) =][ E]t∼ω(t) _|ΦPr_ (t) − ΦQθ (t)|[2][i] _,_
h

where ΦP(t) = Ex∼P _e[i][t][·][x][]_ is the CF of P. For notational brevity, we write ΦPr (t) as Φr(t), and
ΦQθ (t) as Φθ(t) in the following.


We consider the optimized CFD, supω∈ΩCω[2] [(][P][r][,][ Q][θ][)][, and would like to show that it is locally Lips-]

chitz, [4] which subsequently means that it is continuous. Since the Radamacher’s theorem (Federer
(2014)) implies that any locally Lipschitz function is differentiable almost everywhere, the claim of
differentiability is also justified once the Lipschitz locality is proven.

We first note that the difference of two maximal functions’ values is smaller or equal to the maximal
difference of the two functions. Then, for any θ and θ[′],
_ωsup∈ΩCω[2]_ [(][P][r][,][ Q][θ][)][ −] _ω[sup]∈ΩCω[2]_ [(][P][r][,][ Q][θ][′] [)] _≤_ _ωsup∈Ω_ _Cω[2]_ [(][P][r][,][ Q][θ][)][ −C]ω[2] [(][P][r][,][ Q][θ][′] [)] _._ (9)

We note that the absolute difference of any two complex values represented in terms of its real-number
amplitude (A, B) and phase (α, β) satisfies |Ae[iα] _−_ _Be[iβ]| ≤|A| + |B|. Writing the maximal ω as_
_ω[∗], the RHS of Eq. 9 can be written as_
_Cω[2]_ _[∗]_ [(][P][r][,][ Q][θ][)][ −C]ω[2] _[∗]_ [(][P][r][,][ Q][θ][′] [)] (10)

= Et∼ω∗(t) [(Cω∗ (Pr, Qθ) + Cω∗ (Pr, Qθ′ )) (Cω∗ (Pr, Qθ) −Cω∗ (Pr, Qθ′ ))]

= Et∼ω∗(t) [(|Φr(t) − Φθ(t)| + |Φr(t) − Φθ[′] (t)|)(|Φr(t) − Φθ(t)| −|Φr(t) − Φθ[′] (t)|)]

_≤_ Et∼ω∗(t) [(2|Φr(t)| + |Φθ(t)| + |Φθ′ (t)|)(|Φr(t) − Φθ(t)| −|Φr(t) − Φθ′ (t)|)]

(a)
_≤_ 4Et∼ω∗(t) [|Φr(t) − Φθ(t)| −|Φr(t) − Φθ[′] (t)|]

(b)
_≤_ 4Et∼ω∗(t) [|Φθ(t) − Φθ′ (t)|],

4A function f is locally Lipschitz if there exist constants δ ≥ 0 and M ≥ 0 such that |x − _y| < δ →_
_|f_ (x) − _f_ (y)| ≤ _M · |x −_ _y| for all x, y._


-----

where we have used (a) |ΦP| ≤ 1, (b) triangle inequality. By interpreting a complex number as
a vector on a 2D plane, and using trigonometric arguments, one can deduce that |e[ia] _−_ _e[ib]| =_
2 sin(|a − _b|/2) ≤|a −_ _b|. Then,_

4Et∼ω∗(t) [|Φθ(t) − Φθ′ (t)|] = 4Et∼ω∗(t) _|Ez[e[i][t][·][G][θ][(][z][)]] −_ Ez[e[i][t][·][G][θ][′] [(][z][)]]| (11)

(c) h i
_≤_ 4Et∼ω∗(t) [Ez[|t · Gθ(z) − **t · Gθ′** (z)|]]

(d)
_≤_ 4Et∼ω∗(t)Ez [|t| · |Gθ(z) − _Gθ′_ (z)|] .

In (c), we have also used the Jensen inequality. In (d), the Cauchy-Schwarz inequality has been
applied. As we are assuming that Gθ is a L(θ, z)-Lipschitz function, we have

Et∼ω∗(t)Ez [|t| · |Gθ(z) − _Gθ′_ (z)|] ≤ 4Et∼ω∗(t)[|t|] · Ez[L(θ, z)] · |θ − _θ[′]|._ (12)

Since we are also assuming that Et∼ω∗(t)[|t|], Ez[L(θ, z)] ≤∞, we have shown that supω∈ΩCω is

locally Lipschitz, as required. It is therefore continuous and differentiable almost everywhere, as
discussed above.

B.2 PROOF OF THM. 3

_Proof. We denote xn ∼_ Pn and x ∼ P, and ω[∗] the maximal function of ω. Notice that

_ω[∗]_ [(][P][n][,][ P][) =][ E]t _ω[∗](t)_ Exn _e[i][t][·][x][n]_ [] Ex _e[i][t][·][x][]_
_C[2]_ _∼_ _−_

= Et _ω∗(t)_ hExn e[i][t][·][x][n] [] Ex e[i][t][·][x][] Exn _e[i][t][·][x][n]_ [] Ex _e[i][t][·][x][]_
_∼_ _−_ _·[2][i]_ _−_

(a)

    

_≤_ 2 Et∼ω∗(t) Exn _e[i][t][·][x][n]_ [] _−_ Ex _e[i][t][·][x][]_ []

(b)

  

_≤_ 2 Et∼ω∗(t) Exn,x _e[i][t][·][x][n] −_ _e[i][t][·][x]_ []

(c)



_≤_ 2 Et∼ω∗(t) [|t|] Exn,x [|xn − **x|] .** [] (13)

Here, (a) uses |Ae[iα] _−_ _Be[iβ]| ≤|A| + |B| as argued above Eq. 10; (b) uses Jensen inequality; (c)_
uses the argumentconvergence equivalence (Klenke (2013)), the RHS of Eq. 13 approaches zero as |e[ia] _−_ _e[ib]| = 2 sin(|a −_ _b|/2) ≤|a −_ _b|, given above Eq. 11. Then, by weak Pn_ _−D→_ P, hence
proving the theorem.

B.3 PROOF OF LEMMA. 1

_Proof. Recall that for any two distributions, ω(t), ω0(t) and any function f_ (t),

Eω[f (t)] = _f_ (t)ω(t)dt
Z


_f_ (t) _[ω][(][t][)]_

_ω0(t)_ _[ω][0][(][t][)][d][t]_


= Eω0 [f (t) _[ω][(][t][)]_

_ω0(t)_ []][.]

Hence, Eω(t)[f (t)] = Eω0 [f (t) _ω[ω]0[(]([t]t[)])_ []][. Let][ ω][∗] [be the maximal probability distribution. It is then]

clear that Eω[f (t)] Eω[f (t)] implies Eω0 [f (t) _[ω]ω[∗]0([(]t[t])[)]_ []][ →] [E][ω][∗] [[][f] [(][t][)]][, as desired.]

C EXPERIMENTAL[b] _→_ SETUP OF T[b]WO-SAMPLE TESTING ON SYNTHETIC DATA

Let us describe in more detail the experiment presented in Sec. 4. Data are generated from two
unit-variance multivariate Gaussian distributions P, Q, where all dimensions but one have zero mean
(P ∼N (0d, Id), Q ∼N ((1, 0, . . ., 0)[⊤], Id)). We wish to conduct a two-sample test using the CF to


-----

distinguish between the two distributions, which gets more difficult as the dimensionality increases.
We test if the null hypothesis where the samples are drawn from the same distribution is rejected.

Three sets of frequencies are considered. The number of frequncies in each set is set to 20. The
first set is an “unoptimized” set of frequencies. The first dimension of all but one frequency
has the value of zero. Other dimensions have values generated randomly from a zero-mean unitvariance multivariate Gaussian distribution. We denote the frequency with non-zero value in the
first dimension by t0 without loss of generality. A “normal” set of frequencies is also considered
for comparison, where the frequencies of all dimensions are sampled randomly from a multivariate
Gaussian distributions. Finally, we consider an “optimized” set of frequencies, where from the
“unoptimized” set of frequencies, only t0 is selected to be used for two-sample testing. In other words,
we re-weight the set of frequencies such that all but t0 has zero weight. 1,000 samples are generated
from each of P and Q. We repeat the problem for 100 trials to obtain the rejection rate (and repeat
the whole experiment 5 times to get the error bar).

D DP ESTIMATE OF THE MEAN OF PAIRWISE DISTANCE

Median heuristic is applied widely in kernel methods applications to determine the bandwidth of the
radial basis function (RBF) kernels (Garreau et al. (2017)). The bandwidth is taken to be the median
of all pairwise distances of data samples. Here we give a DP estimation of the mean instead as the
calculation of mean is more tractable. Let x be samples of a certain data distribution of dimension d
and assume that the values lie in [0, 1][d]. Given n samples, there is a total of n(n − 1)/2 pairwise
distance pairs. Then, the mean of the pairwise distance of samples is


_Dn(x) =_


**xi** **xj** 2.

Xi≠ _j_ _∥_ _−_ _∥_


_n(n −_ 1)


where ∥· ∥2 indicates the Euclidean norm.

Consider a pair of neighboring datasets, _,_ . Without loss of generality, let xn = x[′]n [and][ x][i] [=][ x][′]i
_D_ _D[′]_ _̸_
for i = n. Then, the sensitivity of Dn(x) is
_̸_


**x[′]i** _j[∥][2]_

Xi≠ _j_ _∥_ _[−]_ **[x][′]**


∆Dn(x) = maxD,D[′]


**xi** **xj** 2

Xi≠ _j_ _∥_ _−_ _∥_ _−_


_n(n −_ 1)


_n(n −_ 1)


_n−1_

**xi** **xn** 2
_i=1_ _∥_ _−_ _∥_ _−_

X


_n−1_

**xi** **x[′]n[∥][2]**
_i=1_ _∥_ _−_

X


(a)
=

(b)


_n(n_ 1) [max],
_−_ _D_ _D[′]_


(b)
=

_n(n −_ 1) _[·][ (][n][ −]_ [1)]

= [2]√d

_n [.]_


In (a), we cancel out all terms unrelated toandmechanism as in Eq. 6 to obtain the DP estimate of the mean of the pairwise distance of samples. ∥xi − **x[′]n[∥][2]** [lie in][ [0][,] _√d]. After obtaining the sensitivity, one can then applies the Gaussian xn or x[′]n[. In (b), we use the fact that][ ∥][x][i]_ _[−]_ **[x][n][∥][2]**

E TRAINING ALGORITHM

The pseudo-code of the proposed training algorithm is given in Algorithm 1.

F COMPUTATIONAL COMPLEXITY ANALYSIS

PEARL is trained based on deep learning methods, which is efficient computation-wise using a
GPU. Nevertheless, let us give a computational complexity analysis in a traditional sense by ignoring
(partially) the parallel computing capability.


-----

**Algorithm 1: PEARL Training**

**Input: Sensitive data {x}i[n]=1[, differential privacy noise scale][ σ][DP][, number of frequencies][ k][,]**
base sampling distribution variance σ0, training iterations T, learning rates ηC and
_ηG, number of generator iterations per critic iteration ngen, batch size B, latent_
distribution Pz

**Output: Differentially private generator Gθ**

**1 Obtain auxiliary information (e.g., base sampling distribution variance σ0);**


**2 Sample frequencies {t}i[k]=1** [with][ t][ ∼N] [(][0][,][ diag][(][σ][0][))][;]

**34 for iΦ inP( {ti1) =, ..., kn[1]** _} donj=1_ _[e][i][t][i][·][x][j]_

**5** Φb P(ti) = ΦPP(ti) + N (0, ∆[2]φ(x)[σ]DP[2] _[I][)]_

**6 end**

b

**7 Accumulate privacy coste** [b] _ϵ;_


**8** _φ(x) ←_ (Φ[e] P(t1), . . ., ΦP(tk))[⊤]

**9 Initialize generator Gθ, sampling distribution variance σ ;**

**10 for step in** 1, ..., T **do**

[e] _{_ _}_ [e]

**11** **for t in {1, ..., ngen} do**

**12** Sample batch {zi}i[B]=1 [with][ z][i][ ∼] _[P][z][;]_

**1314** **for iΦ inQ( {t1i) =, ..., kB[1]} doBj=1** _[e][i][t][i][·][G][θ][(][z][j]_ [)]

**15** **end**

P

**16** _φ(z)b ←_ (Φ[b] Q(t1), . . .,2 ΦQ(tk))[⊤]

**17** _θ_ _θ_ _ηG_ _θ_ [c]ω (φ(x), _φ(z))_

**18** **endb ←** _−_ _· ∇_ _C_ [b]

**19** Sample batch {zi}i[B]=1 [with]e[ z][i][ ∼][b][P][z][;]

**2021** **for iΦ inQ( {t1i) =, ..., kB[1]} doBj=1** _[e][i][t][i][·][G][θ][(][z][j]_ [)]

**22** **end**

P

**23** _φ(z)b ←_ (Φ[b] Q(t1), . . .,2ΦQ(tk))[⊤]

**24** **_σ_** **_σ + ηC_** **_σ_** [c]ω (φ(x), _φ(z))_

**25 endb ←** _· ∇_ _C_ [b]

**26 Return Gθ** e [b]

**Time complexity. We sample k frequencies and calculate its inner product with data points (of**
dimension d), make a summation with respect to n data points to build k CFs. Since inner product
takes d steps, summation n steps, and we do it for k times, the time complexity of generating CFs is
_nkd. We additionally train the generative model, requiring ttr. Note that unlike graphical models_
like PrivBayes, which gradually add nodes to the Bayesian network, our method can be parallelized
efficiently with GPU. As we sample n records from the generative model during data generation,
the time complexity of dataset generation is n times the inference time, tinf . The total training and
inference time is O(nkd + ntinf + ttr).

**Space complexity. We need to store the k CF values during training. We also need to store the**
generative model with m parameters. The space complexity is O(k + m).

G EVALUATION METRICS

We utilize evaluation metrics commonly used to evaluate GAN’s performance, namely Fréchet
Inception Distance (FID) (Heusel et al. (2017)) and Kernel Inception Distance (KID) (Bi´nkowski
et al. (2018)). FID corresponds to computing the Fréchet distance between the Gaussian fits of the
Inception features obtained from real and fake distributions. KID is the calculation of the MMD of
the Inception features between real and fake distributions using a polynomial kernel of degree 3.


-----

The precision definitions are as follows. Let **x[r]** _i_ _i=1_ [be samples from the real data distribution][ P][r]
_{_ _}[n]_
and {x[θ] _i}i[m]=1_ [be samples from the generated data distribution][ Q]θ[. The corresponding feature vectors]
extracted from a pre-trained network (LeNet in our case) are {z[r] _i}i[n]=1_ [and][ {][z][θi][}]i[m]=1 [respectively.]
The FID and KID are defined as

FID(Pr, Qθ) = _µr_ _µθ_ 2 [+ Tr(Σ][r] [+ Σ][θ] (14)
_∥_ _−_ _∥[2]n_ _n_ _[−]_ [2(Σ][r][Σ][θ][)][1][/][2][)][,]


KID(Pr, Qθ) =


_κ(z[r]i_ _[,][ z]j[r][)]_


_κ(z[θ]i_ _[,][ z]j[θ][)]_ (15)
 


_n(n −_ 1)


_i=1_


_j=1,j≠_ _i_


_m(m −_ 1)


_i=1_


_j=1,j≠_ _i_


_κ(z[r]i_ _[,][ z]j[θ][)]_


_mn_


_i=1_


_j=1_


where (µr, Σr) and (µθ, Σθ) are the sample mean & covariance matrix of the inception features of
the real and generated data distributions, and κ is a polynomial kernel of degree 3:

3

1
_κ(x, y) =_ _,_ (16)

_d_ **[x][ ·][ y][ + 1]**

 

where d is the dimensionality of the feature vectors. We compute FID with 10 bootstrap resamplings
and KID by sampling 100 elements without replacement from the whole generated dataset.

H IMPLEMENTATION DETAILS

**Datasets. For MNIST and Fashion-MNIST, we use the default train subset of the torchvision**
5 library for training the generator, and the default subset for evaluation. For Adult, we follow the
preprocessing procedure in Harder et al. (2021) to make the dataset more balanced by downsampling
the class with the most number of samples.

**Neural networks. The generator for image datasets has the following network architecture:**

-  fc → bn → fc → bn → upsamp → relu → upconv → sigmoid,

where fc, bn, upsamp, relu, upconv, sigmoid refers to fully connected, batch normalization, 2D
bilinear upsampling, ReLU, up-convolution, and Sigmoid layers respectively.

For tabular dataset, we use the following architecture:

-  fc → bn → relu → fc → bn → relu → fc → tanh/softmax,

where tanh and softmax are the Tanh and softmax layers respectively. Network output corresponding
to the continuous attribute is passed through the Tanh layer, whereas network output corresponding
to the categorical attribute is passed through the softmax layer, where the category with the highest
value from the softmax output is set to be the generated value of the categorical attribute. We train
both networks conditioned on the class labels. For DP-MERF, we use the same architectures for fair
comparisons.

**Hyperparameters. We use Adam optimizer with learning rates of 0.01 for both the minimization**
and maximization objectives. Batch size is 100 (1,100) for the image datasets (tabular dataset). The
number of frequencies is set to 1,000 (3,000) for MNIST and tabular datasets (Fashion-MNIST).
The training iterations are 6,000, 3,000, and 8,000 for MNIST, Fashion-MNIST, and tabular datasets
respectively.

[5https://pytorch.org/vision/stable/index.html](https://pytorch.org/vision/stable/index.html)


-----

Datasets Methods ROC PRC Range Marginal MMD

Adult DP-MERF 0.64 0.54 0.10 1.37 2 × 10[−][3]

**Ours** **0.72** **0.62** **0.06** **0.71** **1 × 10[−][7]**

Credit DP-MERF 0.66 0.26 0.049 0.95 2 × 10[−][5]

**Ours** **0.84** **0.68** **0.026** **0.72** **1 × 10[−][8]**

Table 3: Evaluation with ROC, PRC (higher is better), range query, marginal release and MMD
(lower is better) on tabular datasets at (ϵ, δ) = (1, 10[−][5]).


Real data DPCGAN Ours

ROC PRC ROC PRC ROC PRC

LR 0.788 0.681 0.501 ± 0.001 0.484 ± 0.001 **0.752 ± 0.009** **0.641 ± 0.015**
Gaussian NB 0.629 0.511 0.499 ± 0.005 0.483 ± 0.003 **0.661 ± 0.036** **0.537 ± 0.028**
Bernoulli NB 0.769 0.651 0.482 ± 0.159 0.498 ± 0.091 **0.763 ± 0.008** **0.644 ± 0.009**
Linear SVM 0.781 0.671 0.501 ± 0.001 0.484 ± 0.001 **0.752 ± 0.009** **0.640 ± 0.015**
Decision Tree 0.759 0.646 0.599 ± 0.078 0.546 ± 0.056 **0.675 ± 0.03** **0.582 ± 0.028**
LDA 0.782 0.670 0.501 ± 0.002 0.484 ± 0.002 **0.755 ± 0.005** **0.640 ± 0.007**
Adaboost 0.789 0.682 0.523 ± 0.065 0.5 ± 0.036 **0.709 ± 0.031** **0.628 ± 0.024**
Bagging 0.772 0.667 0.546 ± 0.082 0.516 ± 0.058 **0.687 ± 0.041** **0.601 ± 0.039**
GBM 0.800 0.695 0.544 ± 0.101 0.518 ± 0.066 **0.709 ± 0.029** **0.635 ± 0.025**
MLP 0.776 0.660 0.503 ± 0.004 0.486 ± 0.004 **0.744 ± 0.012** **0.635 ± 0.015**

Table 4: Quantitative results for the Adult dataset evaluated at (ϵ, δ) = (1, 10[−][5]).

I ADDITIONAL RESULTS

I.1 ADDITIONAL TASKS AND DATASET FOR TABULAR DATA

We here present more results relevant to PEARL’s generative modeling of tabular data. We first
compare PEARL with DPCGAN in Table 4 using the same classifiers as in the main text.

Three additional metrics/tasks are utilized: Range query. 1000 range queries containing three
attributes are sampled randomly. The average l1 error between the original and synthetic datasets is
evaluated. Marginal release. All 2-way marginals are calculated and the average l1 error is evaluated.
**MMD. The max mean discrepancy (MMD) between the original and synthetic dataset is evaluated.**
We use the MMD because it is a powerful measure that could in principle detect any discrepancy
between two distributions. A Gaussian kernel is used in the evaluation.

Additionally, we perform evaluation on another tabular dataset: Credit (Dal Pozzolo et al. (2015)).
Our results comparing with DP-MERF as shown in Table 3 demonstrate that PEARL is competitive
with various tasks and datasets.

I.2 MNIST AND FASHION-MNIST IMAGES

Fig. 5 and Fig. 6 show more enlarged images of MNIST and Fashion-MNIST at various level of ϵ
generated with PEARL. We also show images generated by PEARL but without critic, and DPGAN.
Note that we are unable to generate meaningful images from DP-MERF after taking privacy into
account appropriately (see Sec. 5).

I.3 ADULT HISTOGRAMS

We show more histogram plots for continuous and categorical attributes comparing our method with
DP-MERF in Fig. 9 and Fig. 10.


-----

(a) ϵ = ∞ (b) ϵ = 10

(c) ϵ = 1 (d) ϵ = 0.2

Figure 5: Additional generated MNIST images at various ϵ.


-----

(a) ϵ = ∞ (b) ϵ = 10

(c) ϵ = 1 (d) ϵ = 0.2

Figure 6: Additional generated Fashion-MNIST images at various ϵ.

(a) ϵ = 1, MNIST (DPGAN) (b) ϵ = 1, FMNIST(DPGAN)

Figure 7: Images generated by DPGAN at ϵ = 1.

(a) ϵ = 1, MNIST (Ours w/o critic) (b) ϵ = 1, FMNIST(Ours w/o critic)

Figure 8: Images generated by our proposal (PEARL) but without critic at ϵ = 1.


-----

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|
|---|---|---|---|---|---|---|
||||||||
||||||||
||||||||
||||||||
||20 40 60 80||||||

|Ours|Col2|Col3|
|---|---|---|
|rea syn|l thetic||
||||
||||
|60 80|||


DP­MERF Ours

real
synthetic

10 [2]

10 [2]

10 [3]

10 [3]

20 40 60 80 20 40 60 80


(a) Age

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
|0.|00 0.25||0.50 0.75 1.00|||||1.2 1e|


|Ours|Col2|Col3|
|---|---|---|
|real syntheti||c|
||||
||||
||||
|.50 0.75 1.00|1. 1|25 e6|


DP­MERF Ours

10 [5] 10 [5] real

synthetic

10 [6] 10 [6]

10 [7] 10 [7]

10 [8] 10 [8]

0.00 0.25 0.50 0.75 1.00 1.25 0.00 0.25 0.50 0.75 1.00 1.25

1e6 1e6


(b) fnlwgt

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|0||20000||400|0|0 60000 80000 10|||00|


|Ours|Col2|Col3|
|---|---|---|
|real|||
|synthetic|||
||||
||||
||||
||||
||||
|00 60000 80000 1||000|


DP­MERF Ours

real

10 [4] 10 [4] synthetic

10 [5] 10 [5]

10 [6] 10 [6]

10 [7] 10 [7]

0 20000 40000 60000 80000 100000 0 20000 40000 60000 80000 100000


(c) Capital Gain

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|0|||1000 2000 300|||||0|40|


|Ours|Col2|
|---|---|
|real synthetic||
|||
|||
|||
|||
|2000 3000|40|


DP­MERF Ours

real
synthetic

10 [3] 10 [3]

10 [4] 10 [4]

10 [5] 10 [5]

10 [6] 10 [6]

0 1000 2000 3000 4000 0 1000 2000 3000 4000


(d) Capital loss

Figure 9: Histograms of various continuous attributes of Adult dataset comparing real and synthetic


-----

10[4]

10[3]


10[4]

10[3]



10[2]

10[1]


10[2]

10[3]


Real
Ours
DP-MERF



10[4]

10[3]


10[2]

10[1]


10[4]

10[3]


6 × 10[3]

4 × 10[3]


3 × 10[3]


10[2]

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Real Ours DP-MERF|Col11|Col12|
|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||
|||||||||||||
|||||||||||||
|.0 1.0|2.0 3|.0 4 (|.0 5 a)|.0 E|6.0 7. ca du|0 8 teg ca|.0 9 ory tio|.0 10.01 n|1.012.01|3.01|4.015|
||||||||||Re Ou DP|al rs -ME|RF|
|||||||||||||
|||||||||||||
|||||||||||||
||1.0|2 (|.0 c)|3. W|0 ca or|4.0 teg k|ory cla|5.0 ss|6.0 7|.0|8.0|
||||||||||Re O DP|al urs -M|ERF|
|||||||||||||
|||||||||||||

|Col1|Real Ours DP-MERF|Col3|Col4|Col5|Col6|Col7|Col8|Real Ours DP-MERF|Col10|Col11|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||
||||||||||||||
||||||||||||||
|0.0 1|.0 2|.0 3|.0 4 (b|.0 5.0 ) O|6.0 7 cate ccu|.0 8 gory pat|.0 9.0 ion|10.01|1.012|.01|3.014.||
||||||||||Re Ou DP|al rs -M|ERF||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
|0.|0||1.0 (d)|Re|2.0 cate lati|gory on|3.0 ship|4|.0||5.0||
||||||||||Re Ou DP|al rs -M|ER|F|
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||


0.0 1.0

category

(e) Sex


0.0 1.0 2.0 3.0 4.0

category

(f) Race


Figure 10: Histograms of various categorical attributes of Adult dataset comparing real and synthetic
data.


-----

