# WHEN, WHY, AND WHICH PRETRAINED GANS ARE USEFUL?


**Timofey Grigoryev** _[∗]_
Yandex
grigorev.ta@phystech.edu

**Artem Babenko**
Yandex
artem.babenko@phystech.edu


**Andrey Voynov** _[∗]_
Yandex
an.voynov@yandex.ru


ABSTRACT

The literature has proposed several methods to finetune pretrained GANs on new
datasets, which typically results in higher performance compared to training from
scratch, especially in the limited-data regime. However, despite the apparent empirical benefits of GAN pretraining, its inner mechanisms were not analyzed indepth, and understanding of its role is not entirely clear. Moreover, the essential
practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do
not have rigorous grounding and are typically determined by trial and error.
This work aims to dissect the process of GAN finetuning. First, we show that
initializing the GAN training process by a pretrained checkpoint primarily affects
the model’s coverage rather than the fidelity of individual samples. Second, we
explicitly describe how pretrained generators and discriminators contribute to the
finetuning process and explain the previous evidence on the importance of pretraining both of them. Finally, as an immediate practical benefit of our analysis,
we describe a simple recipe to choose an appropriate GAN checkpoint that is the
most suitable for finetuning to a particular target task. Importantly, for most of the
target tasks, Imagenet-pretrained GAN, despite having poor visual quality, appears
to be an excellent starting point for finetuning, resembling the typical pretraining
scenario of discriminative computer vision models.

1 INTRODUCTION

These days, generative adversarial networks (GANs) (Goodfellow et al., 2014) can successfully approximate the high-dimensional distributions of real images. The exceptional quality of the state-ofthe-art GANs (Karras et al., 2020b; Brock et al., 2019) makes them a key ingredient in applications,
including semantic editing (Isola et al., 2017; Zhu et al., 2018; Shen et al., 2020; Voynov & Babenko,
2020), image processing (Pan et al., 2020; Ledig et al., 2017; Menon et al., 2020), video generation
(Wang et al., 2018a), producing high-quality synthetics (Zhang et al., 2021; Voynov et al., 2020).

To extend the success of GANs to the limited-data regime, it is common to use pretraining, i.e., to
initialize the optimization process by the GAN checkpoint pretrained on some large dataset. A line
of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang
et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and
report significant advantages compared to training from scratch both in terms of generative quality
and convergence speed. However, the empirical success of GAN pretraining was not investigated
in-depth, and its reasons are not entirely understood. From the practical standpoint, it is unclear how
to choose a proper pretrained checkpoint or if one should initialize both generator and discriminator
or only one of them. To the best of our knowledge, the only work that systematically studies the
benefits of pretraining is Wang et al. (2018b). However, the experiments in (Wang et al., 2018b)
were performed with currently outdated models, and we observed that some conclusions from Wang

_∗Indicates equal contribution_


-----

et al. (2018b) are not confirmed for modern architectures like StyleGAN2 (Karras et al., 2020b). In
particular, unlike the prior results, it appears that for state-of-the-art GANs, it is beneficial to transfer
from sparse and diverse sources rather than dense and less diverse ones.

In this work, we thoroughly investigate the process of GAN finetuning. First, we demonstrate that
starting the GAN training from the pretrained checkpoint can significantly influence the diversity
of the finetuned model, while the fidelity of individual samples is less affected. Second, we dissect
the mechanisms of how pretrained generators and discriminators contribute to the higher coverage of finetuned GANs. In a nutshell, we show that a proper pretrained generator produces samples in the neighborhood of many modes of the target distribution, while a proper pretrained discriminator serves as a gradient field that guides the samples to the closest mode, which together
result in a smaller risk of mode missing. This result explains the evidence from the literature
that it is beneficial to initialize both generator and discriminator when finetuning GANs. Finally,
we investigate different ways to choose a suitable pretrained GAN checkpoint for a given target
dataset. Interestingly, for most of the tasks, Imagenet-pretrained models appear to be the optimal
initializers, which mirrors the pretraining of discriminative models, where Imagenet-based initialization is de-facto standard (Donahue et al., 2014; Long et al., 2015; He et al., 2020; Chen et al.,
2020a). Our conclusions are confirmed by experiments with the state-of-the-art StyleGAN2 (Karras
et al., 2020b), chosen due to its practical importance and a variety of open-sourced checkpoints,
which can be used as pretrained sources. The code and pretrained models are available online at
[https://github.com/yandex-research/gan-transfer](https://github.com/yandex-research/gan-transfer)

The main contributions of our analysis are the following:

1. We show that initializing the GAN training by the pretrained checkpoint can significantly
affect the coverage and has much less influence on the realism of individual samples.

2. We explain why it is important to initialize both generator and discriminator by describing
their roles in the finetuning process.

3. We describe a simple automatic approach to choose a pretrained checkpoint that is the most
suitable for a given generation task.

2 ANALYSIS

This section aims to explain the success of the GAN finetuning process compared to training from
scratch. First, we formulate the understanding of this process speculatively and then confirm this
understanding by experiments on synthetic data and real images.

2.1 HIGH-LEVEL INTUITION
Let us consider a pretrained generator G and discriminator D that are used to initialize the GAN
training on a new data from a distribution ptarget. Throughout the paper, we show that a discriminator
initialization is “responsible for” an initial gradient field, and a generator initialization is “responsible for” a target data modes coverage. Figure 1 illustrates the overall idea with different initialization
patterns. Intuitively, the proper discriminator initialization guarantees that generated samples will
move towards “correct” data regions. On the other hand, the proper pretrained generator guarantees
that the samples will be sufficiently diverse at the initialization, and once guided by this vector field,
they will cover all target distribution. Below, we confirm the validity of this intuition.

2.2 SYNTHETIC EXPERIMENT
We start by considering the simplest synthetic data presented on Figure 2. Our goal is to train a GAN
on the target distribution, a mixture of ten Gaussians arranged in a circle. We explore three options
to initialize the GAN training process. First, we start from random initialization. The second and the
third options initialize training by GANs pretrained on the two different source distributions. The
first source distribution corresponds to a wide ring around the target points, having high coverage
and low precision w.r.t. target data. The second source distribution is formed by three Gaussians that
share their centers with three target ones but have a slightly higher variance. This source distribution
has high precision and relatively low coverage w.r.t. target data. Then we train two source GANs
from scratch to fit the first and the second source distributions and employ these checkpoints to
initialize GAN training on the target data. The results of GAN training for the three options are
presented on Figure 2, which shows the advantage of pretraining from the more diverse model,


-----

Figure 1: Different G/D initialization patterns: the red dots denote pretrained generator samples,
the arrows denote a pretrained discriminator gradient field, the blue distribution is the target. From
left to right: bad discriminators will lead good initial samples out of the target distribution; bad
generators will drop some of the modes even being guided by good discriminators; both proper
_G/D serve as an optimal initialization for transfer to a new task._

|Source-I Source-II Target|Source-I Source-II Target|Col3|
|---|---|---|


30 Source-I init Source-II init 30 Random init

20 Source-ISource-II 3020 Source-IGenerated 10 Source-IIGenerated 20 20 20

10 Target 10 5 10 10 10

0 0 0 0 0 0

3.9 16.5 12.8

_−10_ _−10_ _−5_ _−10_ _−10_ _−10_

_−20_ _−20_ _−10_ _−20_ _−20_ _−20_

_−30_ _−30_ _−20_ _−10_ 0 10 20 30 _−30_ _−20_ _−10_ 0 10 20 30 _−20_ _−15_ _−10_ _−5_ 0 _−20_ _−10_ 0 10 20 _−20_ _−10_ 0 10 20 _−30_ _−20_ _−10_ 0 10 20


Figure 2: Impact of GAN pretraining for synthetic data. 1) source and target distributions. 2_3) GANs pretrained on two source distributions. 4-6): GANs trained on the target distribution,_
initialized by the two source checkpoints and randomly. Each plot also reports the Wasserstein-1
distance between the generated and the target distributions.

which results in a higher number of covered modes. The details of the generation of the synthetic
are provided in the appendix.

**Dissecting the contributions from G and D. Here, we continue with the synthetic example from**
above and take a closer look at the roles that the pretrained generator and discriminator play when
finetuning GANs. Our goal is to highlight the importance of (1) the initial coverage of the target
distribution by the pretrained generator and (2) the quality of the gradient field from the pretrained
discriminator. We quantify the former by the established recall measure (Kynk¨a¨anniemi et al., 2019)
computed in the two-dimensional dataspace with k=5 for 1000 randomly picked samples from the
target distribution and the same number of samples produced by the pretrained generator. To evaluate
the quality of the discriminator gradient field, we use a protocol described in (Sinha et al., 2020).
Namely, we assume that the “golden” ground truth gradients would guide each sample towards
the closest Gaussian center from the target distribution. Then we compute the similarity between
the vector field _xD provided by the pretrained discriminator and the vector field of “golden”_
_∇_
gradients. Specifically, we evaluate the cosine similarity between these vector fields, computed for
the generated samples.

Given these two measures, we consider a series of different starting generator/discriminator checkpoints (Gi, Di), i = 1, . . ., N . The details on the choice of the starting checkpoints are provided
in the appendix. Then we use each pair (Gi, Di) as initialization of GAN training on the target
distribution of ten Gaussians described above. Additionally, for all starting Gi/Di, we evaluate the
recall and the discriminator gradients field similarity to the “golden” gradients. The overall quality
of GAN finetuning is measured as the Wasserstein-1 distance between the target distribution and
the distribution produced by the finetuned generator. The scatter plots of recall, the similarity of
gradient fields, and Wasserstein-1 distance are provided in Figure 3. As can be seen, both the recall and gradient similarity have significant negative correlations with the W1-distance between the
ground-truth distribution and the distribution of the finetuned GAN. Furthermore, for the same level
of recall, the higher values of the gradient similarity correspond to lower Wasserstein distances. Alternatively, for the same value of gradient similarity, higher recall of the source generator typically
corresponds to the lower Wasserstein distance. We also note that the role of the pretrained generator
is more important since, for high recall values, the influence from the discriminator is not significant
(see Figure 3, left).


-----

This synthetic experiment does not rigorously prove the existence of a causal relationship between
the recall or gradient similarity and the quality of the finetuned GANs since it demonstrates only
correlations of them. However, in the experimental section, we show that these correlations can
be successfully exploited to choose the optimal pretraining checkpoint, even for the state-of-the-art
GAN architectures.




16 16

1.0

14 14 14

0.8

12 12 12

0.6

10 10 10

Recall 0.4 -distance18 -distance18 8 -distance1

_W_ _W_ _W_

0.2 6 6 6

0.0 4 4 4

_−1.0_ _−0.8_ _−0.6_ _−0.4_ _−0.2_ 0.0 0.2 _−1.0_ _−0.8_ _−0.6_ _−0.4_ _−0.2_ 0.0 0.2 0.0 0.2 0.4 0.6 0.8 1.0

_∇D similarity_ _∇D similarity_ Recall


Figure 3: Scatter plots of the pretrained generator quality (Recall) and the pretrained discriminator
quality ( _D similarity) vs the quality of finetuned GAN (W1-distance). Each point represents_
_∇_
a result of GAN finetuning, which started from a particular pair of pretrained discriminator and
generator. The color indicates the W1-distance between the final generator distribution and the
target distribution. The Pearson correlation of the final W1-distance is equal 0.84 for the Recall,
_−_
and −0.73 for the gradient similarity.

3 LARGE-SCALE EXPERIMENTS

3.1 EXPLORING PRETRAINING FOR STYLEGAN2
In this section, we confirm the conclusions from the previous sections experimentally with the stateof-the-art StyleGAN2 architecture (Karras et al., 2020b). If not stated otherwise, we always work
with the image resolution 256 × 256.

**Datasets. We work with six standard datasets established in the GAN literature. We also include**
two datasets of satellite images to investigate the pretraining behavior beyond the domain of natural
images. As potential pretrained sources, we use the StyleGAN2 models trained on these datasets.
Table 1 reports the list of datasets and the FID values (Heusel et al., 2017) of the source checkpoints.
We also experimented with four smaller datasets to verify our conclusions in the medium-shot and
few-shot regimes. The details on the datasets are provided in the appendix.

**Experimental setup. Here, we describe the details of our experimental protocol for both the pre-**
training of the source checkpoints and the subsequent training on the target datasets. We always
use the official PyTorch implementation of StyleGAN2-ADA (Karras et al., 2020a) provided by the
authors[1]. We use the “stylegan2” configuration in the ADA implementation with the default hyperparameters (same for all datasets). Training is performed on eight Tesla V100 GPUs and takes
approximately three hours per 1M real images shown to the discriminator.

**Pretraining of source checkpoints. We pretrain one checkpoint on the Imagenet for 50M real**
images shown to the discriminator and seven checkpoints on other source datasets from Table 1
for 25M images. A larger number of optimization steps for the Imagenet is used since this dataset
is more challenging and requires more training epochs to converge. For the large LSUN datasets
(Cat, Dog, Bedroom), we use 10[6] first images to preserve memory. For Satellite-Landscapes, we
use ADA due to its smaller size. Then, we always use checkpoints with the best FID for further
transferring to target datasets for each source dataset.

**Training on target datasets. For each source checkpoint, we perform transfer learning to all**
datasets from Table 1. We use the default transfer learning settings from the StyleGAN2-ADA
implementation (faster adaptive data augmentation (ADA) adjustment rate, if applicable, and no
_Gema warmup). ADA is disabled for the datasets containing more than 50K images and enabled_
for others with default hyperparameters. In these experiments, we train for 25M real images shown

[1https://github.com/NVlabs/stylegan2-ada-pytorch](https://github.com/NVlabs/stylegan2-ada-pytorch)


-----

to the discriminator. Each transfer experiment is performed with three independent runs, and the
metrics are reported for the run corresponding to the median best FID (Heusel et al., 2017).

**Metrics. In the experiments, we evaluate the** Dataset Number of images FID
performance via the four following metrics. Datasets for pretraining
(1) Frechet Inception Distance (FID) (Heusel
et al., 2017), which quantifies the discrepancy
between the distributions of real and fake im- LSUN-Cat 1 000 000 7.8
ages, represented by deep embeddings. Both LSUN-Dog 1 000 000 15.0
distributions are approximated by Gaussians,
and the Wasserstein distance between them is
computed. (2) Precision (Kynk¨a¨anniemi et al., LSUN-Church 126 227 3.2
2019), which measures the realism of fake im- Satellite-Landscapes 2 608 26.6
ages, assuming that the visual quality of a particular fake is high if it belongs to the neighborhood of some real images in the embedding FFHQ 70 000 5.5
space. (3) Recall (Kynk¨a¨anniemi et al., 2019), Additional target datasets
which quantifies GAN diversity, measuring the
rate of real images that belong to the neighborhood of some fake images in the embedding Grumpy Cat 100 —
space. (4) Convergence rate equals a number Flowers 8 189 —
of real images that were shown to the discriminator at the moment when the generator FID
for the first time exceeded the optimal FID by at BreCaHAD 3 253 —
most 5%. Intuitively, this metric quantifies how

|Dataset|Number of images|FID|
|---|---|---|
|Datasets for pretraining|||
|Imagenet|1 281 137|49.8|
|LSUN-Cat|1 000 000|7.8|
|LSUN-Dog|1 000 000|15.0|
|LSUN-Bedroom|1 000 000|3.3|
|LSUN-Church|126 227|3.2|
|Satellite-Landscapes|2 608|26.6|
|Satellite-Buildings|280 741|12.4|
|FFHQ|70 000|5.5|
|Additional target datasets|||
|CIFAR-10|50 000|—|
|Grumpy Cat|100|—|
|Flowers|8 189|—|
|Simpsons|41 866|—|
|BreCaHAD|3 253|—|


Table 1: The datasets used in our experiments. All

fast the learning process reaches a plateau. FID
is computed based on the image embeddings images are resized to 256 × 256 resolution. The

last column reports the FID values of the source

extracted by the InceptionV3 model[2]. Precision

checkpoints trained with the random initialization.

and Recall use the embeddings provided by the
VGG-16 model[3]. Precision and Recall are always computed with k=5 neighbors. For FID calculation, we always use all real images and 50K
generated samples. For Precision/Recall calculation, we use the first 200K real images (or less, if
the real dataset is smaller) and 50K generated samples.

**Results. The metric values for all datasets are reported in Table 2, where each cell corresponds to**
a particular source-target pair. For the best (in terms of FID) checkpoint obtained for each sourcetarget transfer, we report the FID value (top row in each cell), Precision and Recall (the second and
the third rows in each cell), and the convergence rate measured in millions of images (bottom row in
each cell). We highlight the sources that provide the best FID for each target dataset or differ from
the best one by at most 5%. We additionally present the curves of FID, Precision, and Recall values
for several target datasets on Figure 9 and Figure 10 in the appendix.

We describe the key observations from Table 2 below:

-  In terms of FID, a pretraining based on a diverse source (e.g., Imagenet or LSUN Dog) is
superior to training from scratch on all datasets in our experiments.

-  The choice of the source checkpoint significantly influences the coverage of the finetuned
model, and the Recall values vary considerably for different sources, especially for smaller
target datasets. For instance, on the Flowers dataset, their variability exceeds ten percent.
In contrast, the Precision values are less affected by pretraining, and their typical variability
is about 2−3%. Figure 4 reports the standard deviations of Precision/Recall computed over
different sources and highlights that Recall has higher variability compared to Precision,
despite the latter having higher absolute values.

-  Pretraining considerably speeds up the optimization compared to the training from scratch.

[2https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt)
[metrics/inception-2015-12-05.pt](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/inception-2015-12-05.pt)
[3https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/vgg16.pt)
[metrics/vgg16.pt](https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/metrics/vgg16.pt)


-----

|Col1|FFH Q|L.Bedroom|L.Cat|L.Church|L.D og|S.Buildings|S.Landscapes|Im agenet|From scratch|
|---|---|---|---|---|---|---|---|---|---|









|F P Q FFH R C|Col2|5.7 0.782 0.417 5|5.52 0.793 0.419 8|6.08 0.767 0.455 7|5.18 0.793 0.459 11|8.49 0.798 0.318 14|6.57 0.797 0.355 16|4.87 0.784 0.457 12|5.52 0.795 0.472 13|
|---|---|---|---|---|---|---|---|---|---|
|F L.Bedroom P R C|2.71 0.664 0.469 24||3.01 0.651 0.447 25|2.8 0.663 0.485 25|2.63 0.657 0.471 25|3.77 0.667 0.314 21|4.3 0.643 0.344 24|2.57 0.679 0.475 25|3.3 ±0.2 0.668 0.459 23|
|F P L.Cat R C|7.62 0.68 0.402 22|7.68 0.68 0.381 25||7.72 0.669 0.402 25|6.65 0.687 0.409 18|9.09 0.703 0.273 25|9.95 0.67 0.303 25|7.12 0.688 0.39 21|7.85 0.684 0.368 20|
|F L.Church P R C|3.09 0.689 0.54 23|3.11 0.7 0.497 22|3.28 0.69 0.496 23||2.97 0.677 0.543 21|3.99 0.692 0.414 22|6.79 0.63 0.322 5|3.0 0.699 0.528 23|3.16 0.682 0.554 25|
|F P og L.D R C|14.8 0.74 0.363 22|15.1 0.747 0.334 22|13.9 0.757 0.359 24|15.6 0.738 0.36 25||18.4 0.753 0.237 25|18.4 0.754 0.256 25|14.4 0.743 0.365 24|15.02 0.758 0.349 24|
|F S.Buildings P R C|11.1 0.347 0.549 24|11.5 0.337 0.53 20|11.77 0.316 0.52 21|11.2 0.34 0.555 16|12.1 0.333 0.526 25||16.7 0.281 0.413 25|10.72 0.319 0.574 25|12.36 0.348 0.507 19|
|S.Landscapes F P R C|25.3 0.756 0.249 23|26.1 0.762 0.191 21|24.3 0.762 0.2 18|25.3 0.73 0.291 5|23.8 0.759 0.282 8|28.2 0.769 0.136 14||21.0 0.719 0.393 2|26.6 0.737 0.214 25|
|F R-10 P CIFA R C|8.6 ±0.5 0.79 0.493 15|8.29 0.764 0.45 11|7.6 0.75 0.479 5|8.62 0.759 0.502 8|7.11 0.769 0.525 8|10.4 0.783 0.397 25|9.22 0.759 0.401 19|6.2 ±0.5 0.761 0.559 3|9.33 0.781 0.455 20|
|F ers P Flow R C|9.47 0.786 0.251 22|9.79 0.776 0.215 16|9.4 0.795 0.194 15|9.88 0.773 0.226 25|8.88 0.773 0.269 6|11.8 0.772 0.14 20|9.07 0.809 0.153 21|8.31 0.773 0.282 9|10.73 0.78 0.271 21|
|F Cat P py R rum C G|11.9 0.999 0.06 24|12.3 0.996 0.02 25|14.0 0.997 0.03 24|16.1 0.995 0.0217 25|12.6 0.998 0.04 25|28.5 0.868 0.01 1|16.3 0.999 0.045 25|14.7 0.999 0.05 25|15.34 0.997 0.0175 25|
|F psons P R Sim C|7.87 0.432 0.406 24|8.0 ±0.4 0.423 0.349 25|8.12 0.431 0.353 24|7.93 0.436 0.384 21|7.67 0.416 0.395 22|10.0 0.404 0.219 24|9.98 0.418 0.173 25|8.28 0.427 0.364 25|8.42 0.42 0.335 25|
|F D A P BreCaH R C|26.31 0.694 0.385 3|23.12 0.679 0.417 1|24.80 0.696 0.462 1|25.40 0.709 0.412 4|25.36 0.692 0.439 1|23.81 0.730 0.337 2|21.84 0.712 0.473 3|22.73 0.703 0.483 1|23.72 0.705 0.434 11|


Table 2: Metrics computed for the best-FID checkpoint for different source and target datasets. Each
row corresponds to a particular target dataset, and each column corresponds to a particular source
model used to initialize the training. For each target dataset, we highlight (by orange) the sources
that provide the smallest FID or which FID differs from the best one by at most 5%. In each cell, we
report from to bottom: FID, Precision, Recall, and convergence rate measured in millions of images
(lower is better). In purpose to make the table easier to read, we report std only once it exceeds 5%
which happens rarely. The typical values vary around 0.1.


-----

0.06

0.04


0.02


0.00

Precision std
Recall std

FFHQ L.Bedroom L.Cat L.Church L.Dog S.BuildingsS.LandscapesCIFAR-10 FlowersGrumpy CatSimpsonsBreCaHAD

Figure 4: Standard deviations of Precision/Recall values for each target dataset computed over different sources. Due to the symmetric nature of the quantities, the table reports the standard deviation
for precision and recall computed over an equal number of real and generated samples (minimum
between a dataset size and 50K)

Overall, despite having poor quality (FID=49.8), the Imagenet-pretrained unconditional StyleGAN2
model appears to be a superior GAN initialization that typically leads to more efficient optimization
compared to alternatives. This result contradicts the observations in (Wang et al., 2018b) showing
that it is beneficial to transfer from dense and less diverse sources rather than sparse and diverse ones,
like Imagenet. We attribute this inconsistency to the fact that (Wang et al., 2018b) experimented with
the WGAN-GP models, which are significantly inferior to the current state-of-the-art ones.


3.2 ANALYSIS

In this section, we perform several additional experiments that illustrate the benefits of pretraining.

**Pretraining improves the mode coverage for real data. Here, we consider the Flowers dataset and**
assume that each of its 102 labeled classes corresponds to different distribution modes. To assign the
generator samples to the closest mode, we train a 102-way flowers classifier via finetuning the linear
head of the Imagenet-pretrained ResNet-50 on real labeled images from Flowers. Then we apply this
classifier to generated images from eleven consecutive generator snapshots from the GAN training
process on the interval from 0 to 200 kimgs taken every 20 kimgs. This pipeline allows for tracking
the number of covered and missed modes during the training process. Figure 5 (left) demonstrates
how the number of covered modes changes when GAN is trained from scratch or the checkpoints
pretrained on FFHQ and Imagenet. In this experiment, we consider a mode being “covered” if it
contains at least ten samples from the generated dataset of size 10 000. One can see that Imagenet,
being the most diverse source, initially covers more modes of the target data and faster discovers the
others. FFHQ also provides coverage improvement but misses more modes compared to Imagenet
even after training for 200 kimgs. With random initialization, the training process covers only a
third of modes after training for 200 kimgs . On the right of Figure 5, we show samples drawn for
the mode, which is poorly covered by the GAN trained from FFHQ initialization and well-covered
by its Imagenet counterpart.


100

80


60

40


20


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|Col11|Col12|Col13|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||Ima FFH Ran|gene Q-in dom|t-init it -init||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||
||||||||||||||


0 20 40 60 80 100 120 140 160 180 200

Imagenet-init
FFHQ-init
Random-init

1K images

Figure 5: Left: Number of modes covered by the generator snapshots during the training process
from three different initializations. Right: samples of the 65-th class of the Flowers dataset, which is
well-covered by the GAN trained from the Imagenet initialization and poorly covered by the GAN
trained from the FFHQ initialization. Top: real images; Middle: FFHQ; Bottom: Imagenet.


-----

0.8 FFHQ-init

Imagenet-init
Random-init

0.6

LPIPS 0.4
∆

0.2

0 1 2 3 4 5

1M images


Figure 6: Evolution of the generated samples with different source initializations. Left: average
LPIPS-distance between images generated by the consecutive generator snapshots for the same latent code. Right: images generated with the same latent code evolving during training: top row:
start from FFHQ, middle row: start from Imagenet, bottom row: start from random initialization.

Figure 7: Left: distribution of samples trajectories lengths with Flowers as a target dataset. Right:

0.30 Imagenet-init 0.7
FFHQ-init

0.25 Random-init 0.6

0.20 0.5

0.15

0.4

0.10 Imagenet-init

0.05 Class change probability 0.3 FFHQ-init

Random-init

0.2

0.00

10 15 20 25 30 0 10 20 30 40 50

LPIPS-trajectory length 1K images

generated class change probability for individual latents during the training.

**Pretraining provides more gradual image evolution. The observations above imply that it is ben-**
eficial to initialize training by the checkpoint with a higher recall so that the target data is originally
better “covered” by the source model. We conjecture that transferring from a model with higher
recall makes it easier to cover separate modes in the target distribution since, in this case, generated
samples can slowly drift to the closest samples of the target domain without abrupt changes to cover
previously missing modes. To validate this intuition, we consider a fixed batch of 64 random latent codes z and a sequence of the generator states G1, . . ., GN obtained during the training. Then
we quantify the difference between consecutive images computed as the perceptual LPIPS distance
Zhang et al. (2018) LPIPS(Gi(z), Gi+1(z)). Figure 6 shows the dynamics of the distances for
Flowers as the target dataset and Imagenet, FFHQ, and random initializations. Since the Imagenet
source initially has higher coverage of the target data, its samples need to transform less, which
results in higher performance and faster convergence.

Figure 6 indicates more gradual sample evolution when GAN training starts from a pretraining
checkpoint. Here we additionally report the distributions of samples’ trajectories’ lengths quantified
by LPIPS. Namely, for a fixed z and a sequence of the generator snapshots G1, . . ., GN obtained
during training, we calculate the length of the trajectory as a sum _i_ [LPIPS(][G][i][(][z][)][, G][i][+1][(][z][))][.]

Figure 7 (left) presents the length distributions for three initializations and Flowers as the target
dataset.

[P]

Finally, to track the dynamics of mode coverage of the target dataset, we obtain the class assignments
of the generated samples G1(z), . . ., GN (z) with a classifier pretrained on the Flowers dataset.
Then for the samples Gi(z), Gi+1(z) generated with the consequent checkpoints, we calculate the
probability that the sample changes its class assignment by averaging over 256 latent codes. That
is, we evaluate the probability that a flower class of a sample Gi(z) differs from a class of a sample
_Gi+1(z). The probabilities of the class change for different source checkpoints are presented in_
Figure 7, right. Importantly, training from pretrained sources demonstrates higher class persistence


-----

|Inverted Domain|Best FID|LPIPS-error|
|---|---|---|
|CelebA-HQ CelebA-HQ|5.35 4.86|0.22 0.22|
|FFHQ FFHQ|5.35 4.86|0.25 0.25|


**GAN Model (source, target)** **Inverted Domain** **Best FID** LPIPS-error _F_ **-error**

s: Random, t: FFHQ CelebA-HQ 5.35 0.22 0.186
s: Imagenet, t: FFHQ CelebA-HQ 4.86 0.22 **0.174**

s: Random, t: FFHQ FFHQ 5.35 0.25 0.180
s: Imagenet, t: FFHQ FFHQ 4.86 0.25 **0.168**

s: Random, t: L.Bedroom L.Bedroom 2.97 0.47 0.123
s: Imagenet, t: L.Bedroom L.Bedroom 2.56 **0.44** **0.115**

Table 3: Reconstruction errors for GAN models with different source and target datasets.

of individual samples. This indicates that the Imagenet-pretrained generator initially covers the
target dataset well enough and requires fewer mode-changing sample hops during training.

**Pretraining is beneficial for downstream tasks. Here, we focus on the task of inverting a real**
image given a pretrained generator, which is necessary for semantic editing. We employ the recent
GAN inversion approach (Tov et al., 2021) and train the encoders that map real images into the
latent space of generators approximating FFHQ and Bedroom distributions. For both FFHQ and
Bedroom, we consider the best generators that were trained from scratch and the Imagenet-based
pretraining. Table 3 reports the reconstruction errors quantified by the LPIPS measure and F **-error.**
The details of the metrics computation are provided in the appendix. Overall, Table 3 confirms that
higher GAN recall provided by pretraining allows for the more accurate inversion of real images.

4 CHOOSING PROPER PRETRAINED CHECKPOINT

This section describes a simple recipe to select the most appropriate pretrained checkpoint to initialize GAN training for a particular target dataset. To this end, we consider a set of natural proxy metrics that quantify the similarity between two distributions. Each metric is computed in two regimes.
In the first regime, we measure the distance between the source dataset, consisting of real images
used to pretrain the GAN checkpoint, and the target dataset of real images. In the second regime,
we use the generated images from pretrained checkpoints instead of the source dataset. The second
regime is more practical since it does not require the source dataset. As natural proxy metrics, we
consider FID, KID (Bi´nkowski et al., 2018), Precision, and Recall measures.

To estimate the reliability of each metric, we
calculate the number of target datasets for
which this metric does not correctly predict
the optimal starting checkpoint. We consider Real Source 3 5 11 **2**
a starting checkpoint optimal if it provides the Generated Source 3 3 7 3
lowest FID score or its FID score differs from

|Regime/Metric|FID|KID|Precision|Recall|
|---|---|---|---|---|
|Real Source|3|5|11|2|
|Generated Source|3|3|7|3|

the lowest by most 5%. The quality for all met- Table 4: The number of target datasets for which
rics is presented in Table 4, which shows that the metrics fail to identify the best source (with up
FID or Recall can be used as a rough guide to to 5% best FID deviation).
select a pretrained source in both regimes. On
the other hand, Precision is entirely unreliable.
This observation is consistent with our findings from Section 2 that imply that Recall can serve as a
predictive measure of finetuning quality.

5 CONCLUSION

Transferring pretrained models to new datasets and tasks is a workhorse of modern ML. In this paper,
we investigate its success in the context of GAN finetuning. First, we demonstrate that transfer
from pretrained checkpoints can improve the model coverage, which is crucial for GANs exhibiting
mode-seeking behavior. Second, we explain that it is beneficial to use both pretrained generators
and discriminators for optimal finetuning performance. This implies that the GAN studies should
open-source discriminator checkpoints as well rather than the generators only. Finally, we show that
the recall measure can guide the choice of a checkpoint for transfer and highlight the advantages
of Imagenet-based pretraining, which is not currently common in the GAN community. We opensource the StyleGAN2 checkpoints pretrained on the Imagenet of different resolutions for reuse in
future research.


-----

REFERENCES

Mikołaj Bi´nkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying
[MMD GANs. In International Conference on Learning Representations, 2018. URL https:](https://openreview.net/forum?id=r1lUOzWCW)
[//openreview.net/forum?id=r1lUOzWCW.](https://openreview.net/forum?id=r1lUOzWCW)

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural
image synthesis. In International Conference on Learning Representations, 2019.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In International conference on machine learning,
pp. 1597–1607. PMLR, 2020a.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor
Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter_national conference on machine learning, pp. 647–655. PMLR, 2014._

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor_mation processing systems, 2014._

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on
_Computer Vision and Pattern Recognition, pp. 9729–9738, 2020._

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances
_in Neural Information Processing Systems, pp. 6626–6637, 2017._

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with
conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and
_pattern recognition, 2017._

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training
generative adversarial networks with limited data. NeurIPS, 2020a.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on
_Computer Vision and Pattern Recognition, pp. 8110–8119, 2020b._

Tuomas Kynk¨a¨anniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved
precision and recall metric for assessing generative models. In Advances in Neural Information
_Processing Systems, pp. 3929–3938, 2019._

Christian Ledig, Lucas Theis, Ferenc Husz´ar, Jose Caballero, Andrew Cunningham, Alejandro
Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE
_conference on computer vision and pattern recognition, 2017._

Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic
weight consolidation. arXiv preprint arXiv:2012.02780, 2020.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic
segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 3431–3440, 2015.

Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Selfsupervised photo upsampling via latent space exploration of generative models. In Proceedings
_of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2437–2445, 2020._


-----

Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning
gans. arXiv preprint arXiv:2002.10964, 2020.

Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics
adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2750–2758, 2019.

Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting
deep generative prior for versatile image restoration and manipulation. In European Conference
_on Computer Vision, pp. 262–277. Springer, 2020._

Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for
semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and
_Pattern Recognition, pp. 9243–9252, 2020._

Samarth Sinha, Zhengli Zhao, Anirudh Goyal ALIAS PARTH GOYAL, Colin A Raffel, and Augustus Odena. Top-k training of gans: Improving gan performance by throwing away bad
samples. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad_vances in Neural Information Processing Systems, volume 33, pp. 14638–14649. Curran As-_
[sociates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/](https://proceedings.neurips.cc/paper/2020/file/a851bd0d418b13310dd1e5e3ac7318ab-Paper.pdf)
[a851bd0d418b13310dd1e5e3ac7318ab-Paper.pdf.](https://proceedings.neurips.cc/paper/2020/file/a851bd0d418b13310dd1e5e3ac7318ab-Paper.pdf)

Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder
for stylegan image manipulation. arXiv preprint arXiv:2102.02766, 2021.

Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan
latent space. In International Conference on Machine Learning, pp. 9786–9796. PMLR, 2020.

Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big gans are watching you: Towards unsupervised object segmentation with off-the-shelf generative models. arXiv preprint
_arXiv:2006.04988, 2020._

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems, 2018a.

Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the
_European Conference on Computer Vision (ECCV), pp. 218–234, 2018b._

Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost
van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 9332–9341, 2020.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable
effectiveness of deep features as a perceptual metric. In CVPR, 2018.

Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio
Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort.
_arXiv preprint arXiv:2104.06490, 2021._

Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for limited-data
generation. ICML, 2020.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation
using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference
_on computer vision, 2018._


-----

6 APPENDIX

DATASETS
Here we provide the details for the used datasets. Table 5 reports the size, original image resolution
(which was always resized to 256 × 256 in our experiments), number of samples used for training,
and URL for each of the datasets. In Tables 6, 7, 8, 9 we report pairwise distances between source
and target datasets for different metrics. Figure 8 illustrates samples from each dataset. As for
BreCaHAD, we generate a dataset of 256 × 256 crops of the original dataset images with the code
provided in StyleGAN-ADA repository.

|Dataset|Size|Original Resolution|Samples Used|
|---|---|---|---|
|CIFAR-104|50 000|32 32 ×|50 000|
|FFHQ5|70 000|1024 1024 ×|70 000|
|Flowers6|8 189|varies|8 189|
|Grumpy-Cat7|100|256 256 ×|100|
|Imagenet8|1 281 137|varies|1 281 137|
|LSUN Bedroom9|3 033 042|256 256 ×|1 000 000|
|LSUN Cat9|1 657 266|256 256 ×|1 000 000|
|LSUN Church9|126 227|256 256 ×|126 227|
|LSUN Dog9|5 054 817|256 256 ×|1 000 000|
|Satellite-Buildings10|280 741|300 300 ×|280 741|
|Satellite-Landscapes11|2 608|1800 1200 ×|2 608|
|Simpsons12|41 866|varies|41 866|
|BreCaHAD13|3 253|256 256 ×|3 253|



Table 5: Datasets information.

LEARNING CURVES

On Figure 9 and Figure 10 we present the learning curves from Table 2 in the main text. To make
the plots readable, for each target dataset, we report only the curves corresponding to training from
scratch, training from the Imagenet checkpoint, and from two checkpoints that perform best among
the rest as a representative subset of sources.

SYNTHETIC DATA DETAILS

Here we provide the details for the experiment described in Section 2.2. The synthetic target data is
formed by 10 Gaussians with centers on the circle of radius 20 and σ = 0.25. Source-I (blue) is a
distribution formed as a sum of a uniform distribution on a zero-centered circle of a radius 20 and
the zero-centered Gaussians with σ = 4. Source-II (green) is formed by 3 Gaussians with centers
that coincide with the consequent centers of three Gaussians of the original data and σ = 0.5. We
use the standard GAN loss (Goodfellow et al., 2014) and perform 5000 generator training steps
with 4 discriminator steps for every generator step. We use batch size 64 and Adam optimizers
with learning rate 0.0002 and β1, β2 = 0.5, 0.999. The generator has a 64-dimensional latent space

[4https://www.cs.toronto.edu/˜kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html)
[5https://github.com/NVlabs/ffhq-dataset](https://github.com/NVlabs/ffhq-dataset)
[6https://www.robots.ox.ac.uk/˜vgg/data/flowers/102/index.html](https://www.robots.ox.ac.uk/~vgg/data/flowers/102/index.html)
[7https://hanlab.mit.edu/projects/data-efficient-gans/datasets/](https://hanlab.mit.edu/projects/data-efficient-gans/datasets/)
[8https://image-net.org/index.php](https://image-net.org/index.php)
[9https://www.yf.io/p/lsun](https://www.yf.io/p/lsun)
[10https://www.aicrowd.com/challenges/mapping-challenge-old](https://www.aicrowd.com/challenges/mapping-challenge-old)
[11https://earthview.withgoogle.com](https://earthview.withgoogle.com)
[12https://www.kaggle.com/c/cmx-simpsons/data](https://www.kaggle.com/c/cmx-simpsons/data)
[13https://figshare.com/articles/dataset/BreCaHAD_A_Dataset_for_Breast_](https://figshare.com/articles/dataset/BreCaHAD_A_Dataset_for_Breast_Cancer_Histopathological_Annotation_and_Diagnosis/7379186)
[Cancer_Histopathological_Annotation_and_Diagnosis/7379186](https://figshare.com/articles/dataset/BreCaHAD_A_Dataset_for_Breast_Cancer_Histopathological_Annotation_and_Diagnosis/7379186)


-----

Figure 8: Samples for each of the target and source datasets.


-----

|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|


|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|
|F|0|244.0|194.8|240.5|178.8|256.1|233.3|150.8|
|L.B|244.0|0|165.0|182.8|162.4|233.3|236.7|143.4|
|L.Ca|194.8|165.0|0|200.8|97.6|206.9|185.9|104.1|
|L.Ch|240.5|182.8|200.8|0|167.0|199.8|232.5|140.4|
|L.Dog|178.8|162.4|97.6|167.0|0|200.0|182.3|63.9|
|S.B|256.1|233.3|206.9|199.8|200.0|0|172.2|177.5|
|S.L|233.3|236.7|185.9|232.5|182.3|172.2|0|145.3|
|C|197.2|188.1|120.9|192.3|102.2|202.1|185.3|85.4|
|Fl|257.7|254.7|235.4|243.8|215.9|285.4|261.4|192.8|
|GC|293.1|260.8|188.4|259.2|259.3|341.4|334.5|264.4|
|S|252.5|225.2|199.4|218.8|195.9|217.7|244.3|167.6|
|BCH|347.8|345.7|319.7|356.0|303.8|351.2|245.4|280.4|


Table 6: FID distances between source and target datasets. Underlined cell in a row corresponds
to a source domain that is closest to a fixed target. Datasets names are shortened as: L.Bdr
(LSUN Bedroom), L.Cat (LSUN Cat), L.Chr (LSUN Church), L.Dog (LSUN Dog), S.Bld (Satellite Buildings), S.Lnd (Satellite Landscapes), Imgn (Imagenet), C-10 (CIFAR-10), Flw (Flowers),
**GC (Grumpy Cat), S (Simpsons), BCH (BreCaHAD).**

|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|


|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|
|F|0|0.237|0.169|0.213|0.116|0.230|0.165|0.116|
|L.B|0.237|0|0.161|0.193|0.124|0.249|0.200|0.126|
|L.Ca|0.168|0.161|0|0.185|0.080|0.189|0.129|0.105|
|L.Ch|0.213|0.193|0.185|0|0.114|0.202|0.185|0.096|
|L.Dog|0.116|0.125|0.079|0.113|0|0.155|0.095|0.027|
|S.B|0.229|0.248|0.189|0.202|0.156|0|0.129|0.179|
|S.L|0.165|0.200|0.130|0.185|0.095|0.129|0|0.109|
|C|0.137|0.149|0.092|0.144|0.048|0.170|0.117|0.060|
|Fl|0.227|0.260|0.211|0.230|0.157|0.277|0.212|0.153|
|GC|0.260|0.283|0.113|0.276|0.196|0.332|0.249|0.195|
|S|0.265|0.276|0.215|0.244|0.178|0.247|0.227|0.179|
|BCH|0.335|0.374|0.316|0.375|0.267|0.349|0.205|0.273|



Table 7: KID distances between source and target datasets computed. Highlighted cell in a row
corresponds to a source domain that is closest to a fixed target.

and consists of six consequent linear layers, all but the last followed by batch-norms and ReLUactivations. The intermediate layers’ sizes are 64, 128, 128, 128, 64. The discriminator is formed by
a sequence of five linear layers, each but the last followed by the ReLU-activation. The intermediate
layers’ sizes are 64, 128, 128, 64.

The starting checkpoints for the Dissecting Contributions experiments are taken from the intermediate checkpoints of the GAN training for Source-I. We take every 50-th checkpoint, gathering 100
in total. We perform fine-tuning to the target distribution with the same parameters as above except
the number of steps equals 1000.

LONGER TRAINING

In this series of experiments, we run GAN training for a source checkpoint being either Imagenetpretrained or randomly initialized for two times higher number of steps (50 million real images
shown to the discriminator). The results are presented in Table 10. Generally, Imagenet-pretraining
almost always either improves GAN quality or performs equally to the random initialization while
speeding up convergence by a large margin.


-----

|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|


|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|
|F|1|0.000|0.014|0.000|0.057|0.001|0.000|0.005|
|L.B|0.333|1|0.333|0.235|0.337|0.307|0.058|0.021|
|L.Ca|0.448|0.598|1|0.253|0.384|0.619|0.229|0.094|
|L.Ch|0.027|0.050|0.007|1|0.058|0.208|0.016|0.003|
|L.Dog|0.539|0.679|0.591|0.350|1|0.726|0.265|0.144|
|S.B|0.000|0.000|0.000|0.000|0.000|1|0.000|0.000|
|S.L|0.007|0.014|0.007|0.042|0.002|0.705|1|0.016|
|C|0.000|0.000|0.000|0.000|0.000|0.001|0.000|0.000|
|Fl|0.006|0.000|0.001|0.000|0.000|0.002|0.012|0.003|
|GC|0.000|0.000|0.001|0.000|0.000|0.000|0.000|0.000|
|S|0.000|0.000|0.000|0.012|0.000|0.046|0.000|0.000|
|BCH|0.000|0.000|0.000|0.000|0.000|0.000|0.000|0.000|


Table 8: The Precision values computed for the targets datasets w.r.t. the source datasets.

|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|


|Col1|F|L.B|L.Ca|L.Ch|L.Dog|S.B|S.L|I|
|---|---|---|---|---|---|---|---|---|
|F|1|0.333|0.448|0.027|0.539|0.000|0.007|0.737|
|L.B|0.000|1|0.598|0.050|0.679|0.000|0.014|0.124|
|L.Ca|0.014|0.333|1|0.007|0.591|0.000|0.007|0.218|
|L.Ch|0.000|0.235|0.253|1|0.350|0.000|0.042|0.303|
|L.Dog|0.057|0.337|0.384|0.058|1|0.000|0.002|0.325|
|S.B|0.001|0.307|0.619|0.208|0.726|1|0.705|0.533|
|S.L|0.000|0.058|0.229|0.016|0.265|0.000|1|0.378|
|C|0.001|0.053|0.240|0.006|0.340|0.000|0.003|0.718|
|Fl|0.001|0.183|0.249|0.010|0.410|0.000|0.017|0.708|
|GC|0.000|0.020|0.790|0.000|0.970|0.000|0.000|0.000|
|S|0.013|0.324|0.328|0.060|0.379|0.000|0.045|0.294|
|BCH|0.001|0.203|0.428|0.053|0.546|0.006|0.553|0.789|



Table 9: The Recall values computed for the targets datasets w.r.t. the source datasets.


-----

|Dataset|From Scratch Step (M) FID Precision Recall|Col3|Col4|Col5|Imagenet pretraining Step (M) FID Precision Recall|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
|L.Bedroom|50|2.50|0.663|0.485|50|2.33|0.691|0.483|
|L.Cat|42|6.87|0.686|0.394|48|6.35|0.712|0.385|
|L.Church|36|3.01|0.705|0.547|12|3.00|0.693|0.523|
|L.Dog|40|12.7|0.751|0.384|45|12.8|0.753|0.382|
|S.Buildings|35|11.9|0.363|0.498|14|10.9|0.304|0.591|
|S.Landscapes|25|27.4|0.737|0.214|1|21.1|0.721|0.393|


Table 10: Number of real images shown to the discriminator (step) for the checkpoint with the best
FID value, this value and corresponding precison and recall values for long-term trainings with two
initialization options.

|Source Model|FID|Precision|Recall|Steps to Convergance|
|---|---|---|---|---|


|Source Model|FID|Precision|Recall|Steps to Convergance|
|---|---|---|---|---|
|Imagenet|8.31|0.77|0.28|9|
|Imagenet (half)|8.54|0.81|0.22|25|
|FFHQ|9.47|0.79|0.25|22|
|FFHQ (half)|9.5|0.77|0.27|25|



Table 11: Finetuning to Flowers from a converged source checkpoint and from a checkpoint that
passes two times fewer steps.

TRANSFER FROM AN EARLIER EPOCH

This experiment verifies if it is important to transfer from a well-converged nearly-optimal source
checkpoint, or it is sufficient to start from a roughly stabilized checkpoint from the intermediate step
of the optimization process. To address the question, we perform a series of additional experiments
with Imagenet and FFHQ as source domains, and Flowers as a target domain. As pretrained checkpoints, we consider the best-FID checkpoint and a checkpoint that passed two times fewer steps. The
results for these runs are presented in Table 11. Overall, the choice between two options has only a
marginal impact on the transfer quality, and one can use the source checkpoint from the middle of
training to initialize the finetuning process.

DETAILS OF EXPERIMENTS ON THE GAN INVERSION

We take the e4e generator inversion approach proposed by (Tov et al., 2021) and train an encoder that
maps real data to the GAN latent space. This scheme is known to be capable of mapping real images
to the GAN latent space preserving all generator properties such as latent attributes manipulations.
We follow the original author’s implementation and train an independent encoder model for each
generator. For a generator G we receive an encoder E which is trained to satisfy G(E(x))=x
for each real data sample x. We evaluate the encoders with the average LPIPS-distance (Zhang
et al., 2018) between a test set real samples and their inversions equal Ex _ptestLPIPS(x, G(E(x)))._
_∼_
We also report the average distance between an original image and its reconstruction features of a
pretrained features extractor F which is equal Ex _ptest_ _F_ (x) _F_ (G(E(x))) 2. The lower these
_∼_ _∥_ _−_ _∥_
quantities –, the better reconstruction quality is. Following (Tov et al., 2021), for FFHQ-target
generators, we train the encoder on the FFHQ dataset and evaluate it on the Celeba-HQ dataset and
on FFHQ itself. As for LSUN-Bedroom, we split the original data into a train and a test subset in
the proportion 9 : 1 and train e4e on the train set and evaluate on the test set. As the feature extractor
_F_, for FFHQ we use a Face-ID pretrained model, same as in (Tov et al., 2021), and MoCo-v2 (Chen
et al., 2020b) model for LSUN-Bedroom.


-----

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||


FFHQ

14 1.0 1.0

12 0.8 0.8

10 0.6 0.6

FID

8 0.4 Recall 0.4

Precision

6 0.2 0.2

4 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

LSUN-Bedroom

7 1.0 1.0

6 0.8 0.8

5 0.6 0.6

FID

4 0.4 Recall 0.4

Precision

3 0.2 0.2

2 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

LSUN-Cat

15 1.0 1.0

13 0.8 0.8

11 0.6 0.6

FID

9 0.4 Recall 0.4

Precision

7 0.2 0.2

5 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

LSUN-Church

7 1.0 1.0

6 0.8 0.8

5 0.6 0.6

FID

4 0.4 Recall 0.4

Precision

3 0.2 0.2

2 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

LSUN-Dog

30 1.0 1.0

26 0.8 0.8

22 0.6 0.6

FID

18 0.4 Recall 0.4

Precision

14 0.2 0.2

10 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

Satellite-Buildings

30 1.0 1.0

26 0.8 0.8

22 0.6 0.6

FID

18 0.4 Recall 0.4

Precision

14 0.2 0.2

10 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

From Scratch FFHQ LSUN-Cat LSUN-Dog
Imagenet LSUN-Church

Figure 9: Learning curves for different target and sources datasets, part 1.


-----

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||
|5 10 15 20 2 1M images|||||
||||||
||||||
||||||
||||||
||||||


Satellite-Landscapes

40 1.0 1.0

36 0.8 0.8

32 0.6 0.6

FID

28 0.4 Recall 0.4

Precision

24 0.2 0.2

20 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

CIFAR-10

15 1.0 1.0

13 0.8 0.8

11 0.6 0.6

FID

9 0.4 Recall 0.4

Precision

7 0.2 0.2

5 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

Flowers

15 1.0 1.0

13 0.8 0.8

11 0.6 0.6

FID

9 0.4 Recall 0.4

Precision

7 0.2 0.2

5 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

Grumpy-Cat

30 1.0 1.0

26 0.8 0.8

22 0.6 0.6

FID

18 0.4 Recall 0.4

Precision

14 0.2 0.2

10 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

Simpsons

16 1.0 1.0

14 0.8 0.8

12 0.6 0.6

FID

10 0.4 Recall 0.4

Precision

8 0.2 0.2

6 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

BreCaHAD

40 1.0 1.0

36 0.8 0.8

32 0.6 0.6

FID

28 0.4 Recall 0.4

Precision

24 0.2 0.2

20 0.0 0.0

0 5 10 15 20 25 0 5 10 15 20 25 0 5 10 15 20 25

1M images 1M images 1M images

From Scratch Satellite-Landscapes FFHQ LSUN-Dog
Imagenet LSUN-Bedroom LSUN-Cat

Figure 10: Learning curves for different target and sources datasets, part 2.


-----

