# ENSEMBLES AND COCKTAILS: ROBUST FINETUNING
## FOR NATURAL LANGUAGE GENERATION

**Anonymous authors**
Paper under double-blind review


ABSTRACT


When finetuning a pretrained language model for natural language generation
tasks, one is currently faced with a tradeoff. Lightweight finetuning (e.g., prefixtuning, adapters), which freezes all or most of the parameters of the pretrained
model, has been shown to achieve stronger out-of-distribution (OOD) performance
than full finetuning, which tunes all of the parameters. However, lightweight
finetuning can underperform full finetuning in-distribution (ID). In this work,
we present methods to combine the benefits of full and lightweight finetuning,
achieving strong performance both ID and OOD. First, we show that an ensemble
of the lightweight and full finetuning models achieves the best of both worlds:
performance matching the better of full and lightweight finetuning, both ID and
OOD. Second, we show that we can achieve similar improvements using a single
model instead of two with our proposed cocktail finetuning, which augments full
finetuning via distillation from a lightweight model. Finally, we provide some
explanatory theory in a multiclass logistic regression setting with a large number
of classes, describing how distillation on ID data can transfer the OOD behavior of
one model to another.

INTRODUCTION


When finetuning a pretrained language model for
natural language generation tasks like summarization (Narayanan et al., 2018) and table-to-text (Gardent et al., 2017), one is currently faced with a
tradeoff. One can achieve strong in-distribution
(ID) performance or strong out-of-distribution (OOD)
performance—but not both—by choosing one of two
families of finetuning approaches. Finetuning all parameters of the pretrained language model achieves
strong ID performance (Howard & Ruder, 2018). We
call this full finetuning. Alternatively, freezing most
of the pretrained parameters during finetuning, in e.g.,
adapters (Rebuffi et al., 2017; Houlsby et al., 2019a),
prefix-tuning (Li & Liang, 2021; Lester et al., 2021)
and bitfit (Zaken et al., 2021), achieves stronger OOD
performance than full finetuning (Li & Liang, 2021;
Lester et al., 2021), but worse ID performance. We
call these methods lightweight finetuning.



Ensembles and
Cocktail Finetuning:

Lightweight Best of both worlds
Finetuning

Weight

λ=1

Zero-sum
tradeoff

performance`

λ=0

Out-of-distribution

Full Finetuning

In-distribution
performance


performance than full finetuning (Li & Liang, 2021; Figure 1: Instead of a zero-sum tradeoff
Lester et al., 2021), but worse ID performance. We between the ID and OOD performance of
call these methods lightweight finetuning. lightweight and full finetuning, ensembling

and cocktail finetuning (purple square) ap
In this work, we are the first to demonstrate that this

proximately achieve the best performance of

tradeoff is not necessary in natural language genera
each, both ID and OOD.

tion: we can achieve the best of full and lightweight
finetuning both ID and OOD. We test this across three
natural language generation tasks in English: summarization (Narayanan et al., 2018), table-to-text
(Gardent et al., 2017), and open-domain question answering (Roberts et al., 2020; Lee et al., 2019a),
and two lightweight finetuning methods: adapters and prefix-tuning. Achieving the best of both
worlds is important since across our 6 settings (2 lightweight finetuning methods and 3 datasets),


-----

full finetuning significantly outperforms lightweight finetuning ID in 5 settings, and lightweight
finetuning significantly outperforms full finetuning OOD in 6 settings.

We present two methods for achieving the best of full and lightweight finetuning. We first show that
a weighted ensemble of one full finetuning model and one lightweight finetuning model achieves
performance comparable to the best of the two, both ID and OOD. To avoid running two models at
once, we propose cocktail finetuning: first train a lightweight finetuning model, and then train a full
finetuning model on a mixture of student-teacher distillation loss (Bucila et al., 2006; Hinton et al.,˘
2015) (with the lightweight model as teacher) and cross-entropy loss, on the original ID data.[1] The
ensemble weight (resp. the cocktail mixture loss weight) is chosen to optimize for ID performance,
and we observe that the resultant model achieves comparable OOD performance to lightweight
finetuning.

Our work contributes to an ongoing realization in machine learning: robust models that deviate
minimally from a pretrained model can be gainfully combined with a model adapted more precisely
to a specific dataset. Most saliently, in concurrent work, Wortsman et al. (2021) find that zeroshot image recognition through CLIP (Radford et al., 2021) can be combined with a finetuned
network to perform well ID and OOD. Our work demonstrates that similar results hold true in natural
language generation as well. We draw from simple, well-known methods: ensembles, student-teacher
distillation (Abnar et al., 2020; Hinton et al., 2015)—but our application to achieving the best of ID
and OOD performance in natural language generation is novel.

To better understand how cocktail finetuning can transfer the out-of-distribution behavior of the
lightweight finetuning model when distilling only on in-distribution data, we theoretically study
multiclass logistic regression with a large number of output classes, viewing language generation as
a classification problem over a large output space. We study a distribution shift setting where not
all classes are seen in the ID training data (as is true in language generation due to Zipf’s law (Zipf,
1949)), and getting good OOD accuracy requires generalizing to unseen outputs. First, we prove that
a model trained with gradient descent cannot distinguish the unseen output classes, leading to poor
OOD accuracy. Second, we prove that distilling a teacher model with good OOD accuracy on only
ID training data can transfer the OOD behavior of the teacher to the student, such that the student
and teacher make the same prediction on any new input. As a result, the student also has good OOD
accuracy.

In summary, we present two simple methods—ensembling and cocktail finetuning—for achieving
strong ID and OOD natural language generation with only the original ID data, achieving the “best of
both worlds” of full and lightweight finetuning methods.

2 RELATED WORK

**Lightweight finetuning.** Recently, as pretrained models scale up in the number of parameters,
_lightweight finetuning techniques have been developed to avoid updating all of the parameters. These_
techniques update a subset of the parameters (Zaken et al., 2021), introduce a small number of new
parameters between layers of the model (called “adapters”) (Rebuffi et al., 2017; Houlsby et al.,
2019b; Pfeiffer et al., 2020), or learn auxiliary inputs that are prepended to the models inputs (called
“prompts”) (Shin et al., 2020; Li & Liang, 2021; Qin & Eisner, 2021; Liu et al., 2021b; Logan IV
et al., 2021; and see Liu et al., 2021a for a survey). These techniques empirically have the benefit
of being able to achieve strong out-of-distribution extrapolation performance at the cost of worse
in-distribution performance compared to full finetuning models both in classification and generation
settings (Li & Liang, 2021; Lester et al., 2021; Zhou et al., 2021; Utama et al., 2021). In our work,
we aim to overcome this trade-off by proposing techniques for achieving the best of both worlds,
with a focus on language generation.

**Student-teacher distillation.** Bucilua et al. (2006) introduce distillation for compressing largeˇ
ensembles into smaller, faster, more efficient models, and Hinton et al. (2015) generalize their method
and call it knowledge distillation. Kasai et al. (2021) distill Transformers into RNNs for linear-time
inference. Abnar et al. (2020) show that knowledge distillation can transfer the inductive bias of one
architecture to another: for example, distilling an LSTM to a Transformer leads to a Transformer with

1We use the term cocktail as it is a refreshing mixture involving distillation.


-----

representations that are more similar to independently trained LSTMs than independently trained
Transformers. The efficiacy of our proposed cocktail finetuning is surprising even in context of these
results, as it empirically transfers just the strong OOD performance of the teacher lightweight model
to the student, while not transferring the weak ID performance.

**Robust finetuning in vision.** In concurrent work, Wortsman et al. (2021) propose a similar approach to ours, but focus on image classification: they combine a full finetuning model with a
lightweight finetuning (linear probe) model by ensembling the two models in weight space, and show
that their approach maintains both good ID and OOD performance. It is yet surprising that in the
structured prediction setting of natural language generation, we further show that we can achieve high
OOD performance even when weight-space ensembling is impossible (for instance, since there are
weights not held in common between the full finetuning model and an optimized prompt). Moreover,
we further this line of work by providing a theoretical study into how distillation can transfer OOD
behavior while only using ID data.

**Self-training for robustness.** In adversarial robustness, standard models achieve good accuracy
on clean examples but are not robust to adversarial examples, while adversarially trained models
are more robust but are less accurate on clean examples. Raghunathan et al. (2020) show that the
benefits of these two models can be combined using robust self-training (RST) (Carmon et al., 2019;
Uesato et al., 2019; Najafi et al., 2019) on unlabeled data to improve clean and robust accuracy.
Similarly, Khani & Liang (2021) show that RST can be used to combine models with spurious
features removed (which are more robust) and without removing spurious features (which are more
accurate). In general OOD robustness, In-N-Out (Xie et al., 2021) uses self-training to transfer the
robustness benefits of a model pre-trained with auxiliary self-supervision to a model trained with
extra auxiliary features. Berthelot et al. (2021) use unlabeled data and a consistency regularization
loss for domain adaptation. While many of these methods combine the benefits of two models or use
a self-training/distillation loss to improve robustness, cocktail finetuning differs in that it performs
distillation only on ID training data without any additional ID or OOD unlabeled examples.

3 PROBLEM STATEMENT

3.1 SETUP

We consider a conditional generation task from an input space X (e.g., a news article) to an output
space Y (e.g., a summary of the article), where both spaces consist of sequences of tokens from a
shared vocabulary V.

**Data.** Let Pid and Pood denote the distribution of (x, y) pairs in-distribution and out-of-distribution,
respectively, where x ∈X and y ∈Y. In summarization, Pid may consist of articles about world
and business news while Pood consists of health and technology news. The training data Dtrain and
validation data Dval consist of n and nval in-distribution (x, y) pairs respectively, drawn from Pid.

**Model and metrics.** Let qγ(y | x) = _j=1_ _[q][γ][(][y][j][ |][ x, y][<j][)][ be a pretrained neural autoregressive]_
sequence model with parameters γ. Finetuning (full or lightweight) produces a finetuned model
_pθ(y_ _x) from qγ, where θ denotes the set of trainable parameters. In full finetuning, θ is the same set_

[Q][len][(][y][)]
_|_
of parameters as γ; in lightweight finetuning, θ often refers to the parameters of inserted modules (e.g.,
adapters modules or prefixes). The final predictor is fθ(x) which approximates arg maxy pθ(y | x)
through, e.g., beam search. The goal is to learn parameters θ such that the predictor fθ obtains good
ID and OOD performance:

_Sid(fθ) = Ex,y_ _Pid_ [score(y, fθ(x))] (1)
_∼_
_Sood(fθ) = Ex,y_ _Pood[score(y, fθ(x))]_ (2)
_∼_

where score may be defined differently for each task (e.g., exact match score or BLEU (Papineni
et al., 2002b)).


-----

WebNLG ID (WebNLG ID (xy)) John Madin was born in Birmingham (with Andrew Mitchell as a key leader) and became an architect, designing 103 Colmore Row.(103 Colmore Row | architect | John Madin), (John Madin | birthPlace | Birmingham) (Birmingham | leaderName | Andrew Mitchell)

WebNLG OOD (WebNLG OOD (xy)) Albennie Jones, born in Errata, Mississippi, is a performer of rhythm and blues, of which disco is a derivative.[Albennie Jones | genre | Rhythm and blues] [Albennie Jones | birthPlace | Errata, Mississippi] [Rhythm and blues | derivative | Disco]

XSUM ID (x) The country’s consumer watchdog has taken Apple to court for false advertising because the tablet computer does not work on Australia’s 4G
network.Apple’s lawyers said they were willing to publish a clarification.However the company does not accept that it misled customers.The
Australian Competition and Consumer Commission (ACCC) said on Tuesday: [. . . 170 words] corrective advertising and refunds to consumers.
On its website, Apple does state that 4G LTE is only supported on selected networks in the US and Canada.
XSUM ID (y) US technology firm Apple has offered to refund Australian customers who felt misled about the 4G capabilities of the new iPad.

XSUM OOD (x) The 200ft (65m) structure, boasting panoramic views of London, is part of a £260m revamp of the world-famous art museum.It is being billed as
the UK’s most important new cultural building since the British Library.More than half of the solo displays are dedicated to women artists.At
Tuesday’s launch event, [... 627 words] Later this year Tate will launch Tate Exchange, an ”open experiment” occupying an entire floor of the
new Switch House building, that will enable 50 invited organisations from the across the UK to display their work.
XSUM OOD (y) Tate Modern has unveiled its new extension, a pyramid-like tower housing cavernous gallery spaces, ahead of its official opening.

OpenQA ID (x) medical term for the cause of a disease
OpenQA ID (y) etiology

OpenQA OOD (x) who is the newly elected governor of california?
OpenQA OOD (y) Jerry Brown

Table 1: Examples of in-distribution and out-of-distribution inputs and outputs from the WebNLG,
XSUM and OpenQA tasks, respectively. In WebNLG, the distribution shift is due to non-overlapping
topics; in XSUM, the distribution shift is due to non-overlapping news categories; in OpenQA the
distribution shift is due to different data collection processes.

3.2 BASELINE FINE-TUNING METHODS

**Full finetuning has been the defacto approach to adapt pretrained language models for downstream**
generation tasks (Lewis et al., 2020; Kale & Rastogi, 2020). Under this framework, the model pθ is
initialized with its pretrained parameters γ, and we optimize the following log-likelihood objective.

_L(θ) = E(x,y)∼Dtrain_ [− log pθ(y | x)] (3)

where the set of trainable parameters θ refer to the set of pretrained parameters γ.

**Lightweight finetuning is another family of adaptation approaches, which freezes most of the**
pretrained parameters and augments the model with small trainable modules. Lightweight finetuning
shares the same training objective as full finetuning, but the trainable parameters θ are different.
Within the lightweight finetuning family, we focus on adapter-tuning (Houlsby et al., 2019a) and
prefix-tuning (Li & Liang, 2021). Both approaches keep the pretrained parameters intact. Adaptertuning inserts task-specific MLP layers between each layer of the pretrained language models, and the
trainable parameter θ consists of these inserted MLP parameters. Prefix-tuning prepends a sequence
of trainable, task-specific prefix vectors to the input, and θ consists of these prefix parameters.[2]

4 EXPERIMENT SETUP

4.1 DATASETS AND METRICS

**Table-to-text.** We evaluate on the WebNLG dataset (Gardent et al., 2017), where the input x is
a sequence of (entity, predicate, entity) triples and the output y is a natural language description
that covers the input information. Each example in this dataset is labeled with one of 14 topic
(e.g., airports, comic characters). We select 9 topics as in-distribution and the remaining 5 topics as
out-of-distribution. We evaluate the performance using the official evaluation scripts and report the
BLEU scores (Papineni et al., 2002a).

**Summarization.** We use the XSUM (Narayan et al., 2018) dataset, which is an abstractive summarization dataset on BBC news articles. Each example is labeled with a news category. We select
world, UK and business categories to be in-distribution and let the remaining news categories such as
health and technology to be out-of-distribution. We report the ROUGE-2 scores (Lin, 2004).

**Open-domain, closed-book QA (Open-QA).** We use the open-domain variant (Lee et al., 2019b)
of the Natural Questions dataset (Kwiatkowski et al., 2019) as the in-distribution data and the Web
Questions dataset (Berant et al., 2013) as the out-of-distribution data. The distribution shift is induced
by different data collection process and their year of collection (roughly 2018 versus 2013). We

2For specifics of these methods, see their respective papers.


-----

report the exact match accuracy. Note that this task, unlike most QA (e.g., SQuAD (Rajpurkar et al.,
2016)), does not provide systems with a text from which to extract answers. Roberts et al. (2020)
demonstrated that one could finetune an LM to generate answers from knowledge stored during
pretraining.

4.2 MODEL ARCHITECTURES AND HYPERPARAMETERS

We use GPT-2-medium (Radford et al., 2019) as the pretrained model for table-to-text and opendomain QA tasks. GPT-2 is an autoregressive language model; therefore, we parametrize pθ(y _x)_
_|_
by concatenating x, a separator token, and y (e.g., [x; SEP; y]). We define the loss function to sum
over the token losses that correspond to y.

We use BART-large (Lewis et al., 2020) as the pretrained model for the summarization task. BART
uses an encoder-decoder architecture and parametrizes pθ(y _x) by first feeding x to the encoder,_
_|_
and decoding y conditioned on the encoder representation.

For open-domain closed-book QA, this is to our knowledge the first time GPT-2 has been finetuned
for the task. We used separate development OOD data to test a handful (<10) of hyperparameter
settings to determine whether we’d observe a similar ID/OOD tradeoff to the other tasks. We note
this because, in general, one cannot do hyperparameter optimization on OOD data; we did this to
provide another testbed for our methods.

Our implementation is based on the Hugging Face Transformers package (Wolf et al., 2020). We use
most of the hyperparameters suggested by the Hugging Face default setup. The hyperparameters we
tune include the number of epochs, batch size, learning rate, and prefix length (for prefix-tuning), all
reported in Table 3 in the appendix.

5 RESULTS OF FULL AND LIGHTWEIGHT FINETUNING

There is a trade-off in ID/OOD performance when choosing between full finetuning and lightweight
finetuning: full finetuning consistently outperforms in-distribution, whereas lightweight finetuning
consistently outperforms out-of-distribution (Li & Liang, 2021; Lester et al., 2021). To quantify this
trade-off, we examine the generation performance of both methods on ID and OOD data.

As shown in Table 2, when evaluating in-distribution, full finetuning achieves significantly stronger
results than lightweight finetuning in five out of six settings (across 3 datasets and two lightweight
finetuning methods. Only prefix-tuning on WebNLG achieves comparable accuracy to full finetuning.
When evaluating out-of-distribution, lightweight finetuning achieves significantly stronger results
than full finetuning in five of six settings. Only adapter-tuning on XSUM fails to outperform full
finetuning.[3]

6 COMBINING MODELS WITH A WEIGHTED ENSEMBLE

As we observe in Section 5, full finetuning models attain better ID performance but worse OOD
performance, whereas lightweight finetuning models attain better OOD performance at the cost of
good ID performance. Is it possible to achieve both good ID and OOD performance simultaneously?
In this section, we show that a simple ensemble of the two methods accomplishes this.

First, we train a lightweight finetuning model p[(][ℓ][)] and a full finetuning model p[(][f)] on Dtrain. At
decoding time, we combine the prediction of the p[(][ℓ][)] and p[(][f)] by interpolating them at each token.

_p[(Ens)](yt_ _x, y<t) = λ_ _p[(][ℓ][)](yt_ _x, y<t) + (1_ _λ)_ _p[(][f)](yt_ _x, y<t)_ (4)
_|_ _·_ _|_ _−_ _·_ _|_

At each time step t, both models output a next token distribution conditioned on the common history
_y<t, and the two distributions are interpolated with a mixture weight λ. The output is generated by_
beam searching the ensemble distribution p[(Ens)](yt _x, y<t)._
_|_

3Though it is worth noting again that for open-domain QA, we performed a small amount of manual,
exploratory hyperparameter optimization to optimize the ID of full finetuning and the OOD of lightweight
finetuning.


-----

WebNLG (BLEU) XSUM (ROUGE-2) OpenQA (EM)
ID OOD ID OOD ID OOD

Prefix-Tuning


Full fine-tuning (FT) 63.25 ± 0.42 30.39 ± 0.64 21.22 ± 0.11 15.47 ± 0.09 16.8 ± 0.1 9.8 ± 0.2
Prefix 63.18 ± 0.37 43.75 ± 0.33 19.75 ± 0.07 16.27 ± 0.03 8.2 ± 0.4 11.0 ± 0.1

Ensemble (Prefix) 65.04 ± 0.26 44.80 ± 0.37 21.99 ± 0.04 16.50 ± 0.03 17.2 ± 0.3 11.4 ± 0.1
Cocktail (Prefix) 65.35 ± 0.31 43.96 ± 0.53 21.80 ± 0.20 16.20 ± 0.09 17.8 ± 0.4 10.6 ± 0.2

Adapter-Tuning

Full fine-tuning (FT) 63.25 ± 0.42 30.39 ± 0.64 21.22 ± 0.11 15.47 ± 0.09 16.8 ± 0.1 9.8 ± 0.2
Adapters 60.43 ± 0.24 48.04 ± 0.45 19.04 ± 1.06 15.63 ± 0.59 9.7 ± 0.4 11.1 ± 0.4

Ensemble (Adapters) 64.57 ± 0.17 46.78 ± 0.13 21.72 ± 0.35 16.54 ± 0.19 17.3 ± 0.2 11.4 ± 0.2
Cocktail (Adapters) 64.61 ± 0.17 44.91 ± 0.18 21.08 ± 0.19 15.73 ± 0.08 17.8 ± 0.2 10.6 ± 0.2

Table 2: Performance of full finetuning, lightweight finetuning, our ensemble, and our cocktail
finetuning model, for prefix-tuning (first half) and for adapter-tuning (second half). Cells are
highlighted blue if they are statistically significantly better than lightweight finetuning (for ID
columns) or full finetuning (for OOD columns) and red otherwise.

6.1 CHOOSING THE MIXTURE WEIGHT λ

We start with a set of _λi_ _i=1_ [where][ λ][i] [0, 1]. For example we use λ
_{_ _}[K]_ _∈_ _∈_
_{0., 0.1, 0.25, 0.5, 0.75, 0.9, 1.0} in this experiment. We choose the mixture weight λi that attains_
the best score on Dval.

This method of choosing λ is useful because (1) it crucially does not use OOD data, which is typically
unaccessible at model selection time, and (2) it guarantees that ID performance is not sacrificed,
because we can match ID performance by setting λ = 0.

6.2 RESULTS

As shown in Table 2, the ensemble matches and slightly outperforms both the ID performance skyline
(achieved by full finetuning) and the OOD performance skyline (achieved by lightweight finetuning).
Across all datasets, ensembling achieves significantly improved performance over full finetuning
OOD, and lightweight finetuning ID. Further, in eleven out of twelve settings (across 3 datasets, 2
methods, and both ID and OOD), ensembling does not perform significantly worse than the better of
full and lightweight finetuning.

6.3 RUNTIME AND MEMORY

Despite its ability to combine the best of both worlds, ensembling is expensive at inference time
because two models are run simultaneously, doubling the amount of GPU memory required. Furthermore, two forward passes are required, doubling the runtime for decoding.[4] For deployment
efficiency, it’s better to have a single model at inference time.

7 COCKTAIL FINETUNING: COMBINING WITH DISTILLATION

The goal of cocktail finetuning is to approximate the ensemble in Section 6 with a single model.[5]

7.1 METHOD

Train a lightweight finetuning model p[(][ℓ][)]. Fix mixture weight λ ∈ [0, 1]. Train all parameters of the
pretrained model qγ(y _x) on the following objective. First, define a token-level student-teacher_
_|_

4The two forward passes are parallelizable, but at the cost of another 2x GPU memory.
5One might wish to approximate the two with a single lightweight model, a direction we explored but
eventually discarded due to the difficulty of approximating a full finetuning model with a lightweight finetuning
model. Anecdotally, this is evidence towards the speculation of Lester et al. (2021), that lightweight finetuning
is limited in its ability to alter a pretrained LM.


-----

input output

full prefix cocktail

In-distribution (NQ) medical term for the cause of a disease etiology cause of a disease etiology

Out-of-distribution (WQ) who is the newly elected governor of california? John H. Cox Jerry Brown Jerry Brown


Figure 2: Examples in the open-domain closed-book QA (wherein no text is given from which to
extract the answer). The full finetuning model performs well in-distribution but not out-of-distribution,
the lightweight finetuning model performs well out-of-distribution but not in-distribution, and the
cocktail finetuning model performs well on both. The distribution shift (Natural Questions to
WebQuestions) is due to temporal and data collection differences.

distillation loss at token j:


distillp(ℓ) (x, y, j) = KL _p[(][ℓ][)](yj_ _x, y<j)_ _pθ(yj_ _x, y<j))_ (5)
_|_ _∥_ _|_
 

The objective is the sum over the length of y of the λ-weighted mixture of distillation and loglikelihood loss:


len(y)

_λdistillp(ℓ)_ (x, y, j) (1 _λ) log pθ(yj_ _x, y<j)_ (6)

 _j=1_ _−_ _−_ _|_ 
X


_Lλ(θ) = E(x,y)∼Dtrain_


Let fθ,λ be the predictor resulting from optimizing on Lλ(θ) and estimate the ID score Sid(fθ,λ)
on Dval. Perform this across a set of mixture weights {λi}i. The cocktail finetuning model is the
predictor fθ,λi that maximizes the in-distribution score Sid(fθ, λi) across the mixture weights λi.[6]

7.2 RESULTS


As shown in Table 2, cocktail finetuning matches and slightly outperforms the ID performance skyline
(achieved by full finetuning) and achieves comparable to or slightly worse than the OOD performance
skyline (achieved by lightweight finetuning). Across all datasets, cocktail finetuning achieves
significantly improved performance over full finetuning OOD, and over lightweight finetuning ID.
Further, in seven out of twelve settings (across 3 datasets, 2 methods, and both ID and OOD), cocktail
finetuning does not perform significantly worse than the better of full and lightweight finetuning.

8 ANALYSIS OF THE WEIGHTING OF FULL AND LIGHTWEIGHT FINETUNING


**Trade-off curves.** Our ensembling and cock- XSUM Distillation: Full FT Teacher

17.0

tail finetuning methods are useful in practice = 0
because one can choose the mixture coefficient 16.5 = 0.1
_λ without OOD data. Now, we explore how_ 16.0 = 0.5
ID and OOD performance vary as a function of = 0.75
_λ. Figure 3 shows results across five random_ 15.5 = 0.9

OOD ROUGE2

seeds of ensembling and cocktail finetuning on 15.0 = 1.0
our three datasets, each dot colored according

14.5

to its λ value. In each plot, we see a smooth 19.5 20.0 20.5 21.0 21.5 22.0 22.5

|Col1|= 0 = 0.1 = 0.25 = 0.5 = 0.75 = 0.9 = 1.0|
|---|---|


ID ROUGE2

tradeoff curve, in which a range of λ improve
on the OOD, ID, or both of each constituent

Figure 4: Ablation on XSUM in which a full

model. Interestingly, in two of three tasks, there

finetuning model is distilled into a full finetuning

is effectively only one pareto-optimal λ. This

model. No benefits are observed.

means the λ is best for both for ID and OOD
performance, which is why optimizing for ID performance also leads to good OOD performance.

6For cocktail finetuning, we use the hyperparameters reported for full finetuning; no hyperparameter optimization was performed.


-----

WebNLG: Ensemble (prefix) XSUM: Ensemble (prefix) QA: Ensemble (prefix)

46 17.0

44 16.8 12.5

42 16.6 12.0

40 11.5

16.4

38 lambda 16.2 lambda 11.0 lambda

OOD performance3634 10.90.750.50.25 OOD performance16.015.8 10.90.750.50.25 OOD performance10.510.0 00.10.250.50.75

32 0.1 0.1 0.9

0 15.6 0 9.5 1

62.5 63.0 63.5 64.0 64.5 65.0 65.5 20.0 20.5 21.0 21.5 22.0 8 10 12 14 16 18

ID performance ID performance ID performance

464442 WebNLG: cocktail (prefix) 16.5016.25 lambda00.10.250.5XSUM: cocktail (prefix) 12.011.5 QA: cocktail (prefix)

16.00 0.75

40 0.9

38 15.75 1prefix 11.0

OOD performance3634 lambda00.10.250.5 OOD performance15.5015.25 OOD performance10.5 lambda00.10.250.5

32 0.750.9 15.00 10.0 0.750.9

1 14.75 1

30 prefix 9.5 prefix

62.5 63.0 63.5 64.0 64.5 65.0 65.5 19.0 19.5 20.0 20.5 21.0 21.5 22.0 8 10 12 14 16 18

ID performance ID performance ID performance


Figure 3: For WebNLG and XSUM, ensembles and cocktails both achieve the best of both prefixtuning and full finetuning for some λ; for OpenQA, one can achieve approximately the best of
both.

**Ablations.** One hypothesis to explain the strong performance of ensembles is the known benefits
of ensembling two models trained with different random seeds. Likewise for cocktail finetuning, one
could point to the effectiveness of born-again networks (Furlanello et al., 2018): distilling a teacher
model to an identically parameterized student with the same model architecture. As an ablation to test
this hypothesis, we distill from a full finetuning model into another full finetuning model, again using
_λ to mix the distillation loss with the label loss. The results, in Figure 4, show that this distillation_
does not improve over a single full finetuning model, for any of the λ.

9 WHY DOES DISTILLATION ON ID DATA IMPROVE OOD?

Prior works have used unlabeled OOD data and self-training (similar to distillation) to transfer good
OOD performance from another model (Raghunathan et al., 2020; Xie et al., 2021). In contrast,
cocktail finetuning performs distillation only on ID data. In this section, we provide an explanation
for why this can occur in multiclass logistic regression with a large number of classes.

**Prediction problem and distribution shift.** Motivated by the large output space in language
generation, we consider a prediction problem from inputs X ⊆ R[d] to labels Y = {1, . . ., k} where k
is large. The labeled training data is {(xi, yi)}i[n]=1[. For convenience, we also let][ y][i][ be the one-hot]
vector of length k denoting the correct class. The labels in the training data do not cover all the
possible labels, and we let the subset of seen labels be S.

In this problem, the class marginals of the ID and OOD distributions Pid and Pood are different. To
have good OOD accuracy, the model must generalize to unseen classes.

**Model and losses.** We consider training a multi-class logistic regression model using full batch
gradient descent where the learned parameters are θ ∈ R[k][×][d]. In full fine-tuning, we run gradient
descent on the negative log-likelihood.

In cocktail finetuning, we also add a distillation loss using soft labels (a probability vector) from
the prefix-tuned model, which we denote p[x]θ˜ [=][ p]θ[˜][(][· |][ x][)][ ∈] [R][k][ with parameters][ ˜]θ, given an input x.
Similarly, the predicted class probability vector of the cocktail model given an input x is denoted


-----

_p[x]θ_ [=][ p][θ][(][· |][ x][)][ ∈] [R][k][. In this setting, the cocktail loss (Equation 6) with parameter][ λ][ ∈] [[0][,][ 1]][ is]


((1 − _λ)yi + λp[x]θ˜[i]_ [)][⊤] [log][ p]θ[x][i] _[.]_ (7)
_i=1_

X


_ℓct(θ) =_


The cocktail loss encourages the model to learn to output a mixture of the hard label and the soft
label on the training data. The final score for the classification task is computed with respect to the
accuracy metric, where score(y, ˆy) = 1 _ℓ0_ 1(y, ˆy) and ℓ0 1 is the 0-1 loss.
_−_ _−_ _−_

**Poor OOD performance with full finetuning.** Our first result shows that when training a logistic
regression model using gradient descent (initialized at 0) on the only labeled training set (as in full
finetuning), the learned parameters θj for an unseen label j /∈S are all the same.

**Proposition 1. Let θ[(][t][)]** _be the parameters at iteration t of gradient descent and let θ[(0)]_ = 0. At any
_iteration t and for any unseen label index j /∈S, θj = gt where gt is the same for all j /∈S._

As a result, the model cannot distinguish between any of the unseen classes, resulting in poor
OOD performance. Suppose that the model outputs a fixed, arbitrary unseen class j ∈S when the
probabilities of the unseen classes are the largest. Then, assuming that the OOD distribution Pood is
such that the probability of drawing an example from an unseen class (excluding j) Pood(y /∈S ∪{j})
is at least c ∈ [0, 1], then the OOD accuracy is bounded as

_Sood ≤_ 1 − _Pood(y /∈S ∪{j}) ≤_ 1 − _c_ (8)

This OOD accuracy can be low when c is large, which can occur with very large output spaces.

**Disillation transfers OOD behavior through soft logits.** We investigate how distillation on ID
training data can affect OOD behavior. In the following, let X ∈ R[n][×][d] be the training data matrix.

**Proposition 2. Assume the data matrix X has full column rank. Let θ be the minimizer of the cocktail**
_loss when λ = 1, subject to the constraint that 1[⊤]_ exp(θX) = α1[⊤] exp(θX[˜] ) for some constant
_α > 0, such that the model learns the same normalizing constants (up to scaling) as the prefix-tuned_
_model on the training data only. Then_

_pθ(· | x) = pθ˜[(][· |][ x][)]_ (9)

_for any new input x, such that the cocktail model has the same OOD accuracy as the prefix model._

Therefore, if the prefix-tuning model has high OOD accuracy, then Sood is high. The proof proceeds
by showing that the extra information in the soft logits can help the model recover the parameters of
the prefix-tuned model (up to scaling), which in turn inherits its OOD behavior. Thus, in generation
problems where the output space is large, distillation on soft logits can transfer information about
OOD behavior even through ID data.

10 CONCLUSION

Considered together, our work and the work of Wortsman et al. (2021) provide strong cross-modal
evidence that simple ensembles of models that have minimally deviated from pretraining (for them,
zero-shot; for us, lightweight finetuning models) with full finetuning models provides strong ID
and OOD accuracy. The reasons for this “best of both worlds” behavior are still poorly understood,
motivating future work. Since ensemble models effectively weight by model confidence (though
modulated by λ), one explanation may be that lightweight finetuning models are less confident than
full finetuning models in-distribution, yet more confident out-of-distribution.

In summary, we provide simple, effective methods for achieving the best of lightweight and full
finetuning—strong OOD and ID performance—in language generation. Our cocktail finetuning
provides system builders with the opportunity to optimize for in-distribution performance (in choosing
lambda), often beating full finetuning, while resting assured that the out-of-distribution performance
(i.e., what they observe with real users) will likely be higher than had they performed full finetuning.


-----

ETHICS STATEMENT

The methods presented in this work have the potential to improve a wide range of natural language
generation tasks. Natural language generation has known dual-use issues, like scaling misinformation
generation and generating naturalistic text in scams. The WebNLG XSUM, Natural Questions, and
WebQuestions datasets we evaluate on contain personal information (though public and published)
information about real people and organizations.

REPRODUCIBILITY STATEMENT

All code (including experiment scripts and plot creation scripts) will be released upon publication.
All datasets used in this paper are publicly available and will be linked in our released code. All
reported numbers represent the average of results from 5 random seeds.

REFERENCES

Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through
knowledge distillation. arXiv preprint arXiv:2006.00555, 2020.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from
question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural
_Language Processing, pp. 1533–1544, Seattle, Washington, USA, October 2013. Association for_
[Computational Linguistics. URL https://aclanthology.org/D13-1160.](https://aclanthology.org/D13-1160)

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch:
A unified approach to semi-supervised learning and domain adaptation. CoRR, abs/2106.04732,
[2021. URL https://arxiv.org/abs/2106.04732.](https://arxiv.org/abs/2106.04732)

Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In˘ _International_
_Conference on Knowledge Discovery and Data Mining (KDD), 2006._

Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Inˇ _Proceedings_
_of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp._
535–541, 2006.

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled
data improves adversarial robustness. In Advances in Neural Information Processing Systems
_(NeurIPS), 2019._

Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.
Born again neural networks. In International Conference on Machine Learning, pp. 1607–1616.
PMLR, 2018.

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG
challenge: Generating text from RDF data. In Proceedings of the 10th International Conference
_on Natural Language Generation, pp. 124–133, Santiago de Compostela, Spain, September_
[2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-3518. URL https:](https://www.aclweb.org/anthology/W17-3518)
[//www.aclweb.org/anthology/W17-3518.](https://www.aclweb.org/anthology/W17-3518)

Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In
_NIPS Deep Learning and Representation Learning Workshop, 2015._

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe,
Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning
for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th
_International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning_
_[Research, pp. 2790–2799. PMLR, 09–15 Jun 2019a. URL http://proceedings.mlr.](http://proceedings.mlr.press/v97/houlsby19a.html)_
[press/v97/houlsby19a.html.](http://proceedings.mlr.press/v97/houlsby19a.html)

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea
Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP.
_arXiv, 2019b._


-----

Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In
_Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume_
_1: Long Papers), pp. 328–339, Melbourne, Australia, July 2018. Association for Computational_
[Linguistics. doi: 10.18653/v1/P18-1031. URL https://aclanthology.org/P18-1031.](https://aclanthology.org/P18-1031)

Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. In Proceedings of
_the 13th International Conference on Natural Language Generation, pp. 97–102, 2020._

Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao,
Weizhu Chen, and Noah A Smith. Finetuning pretrained transformers into rnns. arXiv preprint
_arXiv:2103.13076, 2021._

Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups
disproportionately. In ACM Conference on Fairness, Accountability, and Transparency (FAccT),
2021.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris
Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N.
Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
Natural questions: a benchmark for question answering research. Transactions of the Association
_of Computational Linguistics, 2019._

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open
domain question answering. In Association for Computational Linguistics (ACL), 2019a.

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.
Efficient exploration via state marginal matching. arXiv, 2019b.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt
tuning. 2021.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and comprehension. In Proceedings of the 58th
_Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, Online, July_
2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.703. URL
[https://www.aclweb.org/anthology/2020.acl-main.703.](https://www.aclweb.org/anthology/2020.acl-main.703)

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In
_Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the_
_11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),_
pp. 4582–4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/
[v1/2021.acl-long.353. URL https://aclanthology.org/2021.acl-long.353.](https://aclanthology.org/2021.acl-long.353)

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization
_Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics._
[URL https://www.aclweb.org/anthology/W04-1013.](https://www.aclweb.org/anthology/W04-1013)

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.
Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language
processing. ArXiv, abs/2107.13586, 2021a.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt
understands, too. arXiv preprint arXiv:2103.10385, 2021b.

Robert L Logan IV, Ivana Balazeviˇ c, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian´
Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models.
_arXiv preprint arXiv:2106.13353, 2021._

Amir Najafi, Shin ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial
perturbations in learning from incomplete data. In Advances in Neural Information Processing
_Systems (NeurIPS), 2019._


-----

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary!
Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018
_Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018._

Deepak Narayanan, Keshav Santhanam, Amar Phanishayee, and Matei Zaharia. Accelerating Deep
Learning Workloads through Efficient Multi-Model Execution. In NeurIPS Workshop on Systems
_for Machine Learning, pp. 20, 2018._

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for
_Computational Linguistics, ACL ’02, pp. 311–318, Stroudsburg, PA, USA, 2002a. Association_
[for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://doi.org/10.](https://doi.org/10.3115/1073083.1073135)
[3115/1073083.1073135.](https://doi.org/10.3115/1073083.1073135)

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic
evaluation of machine translation. In Association for Computational Linguistics (ACL), 2002b.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckl¨ e, Kyunghyun Cho, and Iryna Gurevych. Adapter-´
fusion: Non-destructive task composition for transfer learning, 2020.

Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts.
In Proceedings of the 2021 Conference of the North American Chapter of the Association for
_Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 5203–5212, Online,_
[June 2021. URL http://cs.jhu.edu/˜jason/papers/#qin-eisner-2021.](http://cs.jhu.edu/~jason/papers/#qin-eisner-2021)

A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models
are unsupervised multitask learners. 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,
Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision, 2021.

Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding
and mitigating the tradeoff between robustness and accuracy. In International Conference on
_Machine Learning (ICML), 2020._

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for
machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP),
2016.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with
residual adapters. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan,
and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 506–
[516. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/](https://proceedings.neurips.cc/paper/2017/file/e7b24b112a44fdd9ee93bdf998c6ca0e-Paper.pdf)
[2017/file/e7b24b112a44fdd9ee93bdf998c6ca0e-Paper.pdf.](https://proceedings.neurips.cc/paper/2017/file/e7b24b112a44fdd9ee93bdf998c6ca0e-Paper.pdf)

Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the
parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in
_Natural Language Processing (EMNLP), pp. 5418–5426, 2020._

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts,
2020.

Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and
Pushmeet Kohli. Are labels required for improving adversarial robustness? In Advances in Neural
_Information Processing Systems (NeurIPS), 2019._

Prasetya Ajie Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. Avoiding inference
heuristics in few-shot prompt-based finetuning, 2021.


-----

WebNLG (BLEU) XSUM (ROUGE-2) OpenQA (EM)
epoch lr batch size other epoch lr batch size other epoch lr batch size other

PrefixAdaptersFull fine-tuning (FT) 555 555 ∗ ∗ ∗ 101010[−][−][−][5][5][5] 556 d=200l=20- 1555 855 ∗ ∗ ∗ 101010[−][−][−][5][5][5] 64205 d=200l=40- 226 555 ∗ ∗ ∗ 101010[−][−][−][4][5][5] 161616 d=200l=20
Table 3: Hyperparameters. d refers to the dimension of the adapter MLP’s middle layer, and l refers
to the length of the prefix vector in prefix-tuning.

WebNLG (BLEU) XSUM (ROUGE-2) OpenQA (EM)

Ensemble (Adapters) 0.5 0.5 0.5
Ensemble (Prefix) 0.5 0.25 0.5

Cocktail (Adapters) 0.5 0.25 0.25
Cocktail (Prefix) 0.5 0.25 0.25

Table 4: The selected values of λ for the ensembles and cocktail experiments.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von´
Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama
Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language
processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro_cessing: System Demonstrations, pp. 38–45, Online, October 2020. Association for Computational_
[Linguistics. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.](https://www.aclweb.org/anthology/2020.emnlp-demos.6)

Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi,
Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv
_preprint arXiv:2109.01903, 2021._

Sang Michael Xie, Ananya Kumar, Robert Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. InN-out: Pre-training and self-training using auxiliary information for out-of-distribution robustness.
In International Conference on Learning Representations (ICLR), 2021.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient finetuning for transformer-based masked language-models. CoRR, abs/2106.10199, 2021. URL
[https://arxiv.org/abs/2106.10199.](https://arxiv.org/abs/2106.10199)

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for visionlanguage models, 2021.

George Kingsley Zipf. Human behavior and the principle of least effort. 1949.

A HYPERPARAMETERS AND DATASET DETAILS

training val test (id) test (ood)

WebNLG 18K 2.2K 1.0K 0.9 K
XSUM 129K 7k 7k 20k
OpenQA 88K 1.8K 1.8K 2k

Table 5: Number of examples in each dataset.

A.1 HYPERPARAMETERS

Relevant hyperparameters beyond the defaults of Huggingface Transformers Wolf et al. (2020) are
reported in Table 3.


-----

WebNLG: Ensemble (adapter) XSUM: Ensemble (adapter) QA: Ensemble (adapter)

46 lambda 13.0 lambda

44 16.5 10.90.75 12.5 00.10.25

42 0.50.25 12.0 0.50.75

16.0 0.1 0.9

40 0 11.5 1

38

lambda 15.5 11.0

OOD performance3634 10.90.750.50.25 OOD performance15.0 OOD performance10.510.0

32 0.1

0 9.5

62.5 63.0 63.5 64.0 64.5 65.0 65.5 18.5 19.0 19.5 20.0 20.5 21.0 21.5 22.0 10 12 14 16 18

ID performance ID performance ID performance

WebNLG: cocktail (adapter) XSUM: cocktail (adapter) QA: cocktail (adapter)

47.5 16.2 lambda0 12.0

45.042.5 16.015.815.6 0.10.250.50.750.91 11.511.0

40.0 adapter

lambda 15.4 lambda

37.5 00.1 15.2 10.5 00.1

OOD performance35.032.5 0.250.50.750.91 OOD performance15.014.8 OOD performance10.0 0.250.50.750.91

30.0 adapter 14.6 9.5 adapter

60 61 62 63 64 65 18.5 19.0 19.5 20.0 20.5 21.0 21.5 10 12 14 16 18

ID performance ID performance ID performance


Figure 5: For WebNLG and XSUM, Ensembles and cocktails both achieve the best both prefix and
full finetuning for some λ; for OpenQA, one can achieve approximately the best of both.

A.2 DATASET DETAILS


Table 5 shows the number of examples in each dataset split.

For the WebNLG dataset, the ID topics include {Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam, City, and WrittenWork} and the OOD topics include {Athlete,
Artist, MeanOfTransportation, CelestialBody, Politician}.

For the XSUM dataset, the ID topics include {uk, business, world } news and the OOD topics
includes { entertainment, election, technology, science, health, education, explainers, live, ukpolitics, world-us-canada, world-europe, uk-england, uk-scotland, world-asia, science-environment,
uk-scotland-scotland-politics, disability}.


ADDITIONAL RESULTS


Due to space constraints, we did not include in Figure 3 the tradeoff plots using adapters lightweight
finetuning method (instead reporting only those for prefix-tuning.) The plots for adapters are provided
in Figure 5.

C ANALYSIS


We restate and expand upon some of the setup for the analysis here.

**Prediction problem.** We consider a prediction problem from inputs X ⊆ R[d] to labels Y =
_{1, . . ., k} where k is large. The labeled training data is {(xi, yi)}i[n]=1[. For convenience, we also let]_
_yi be the one-hot vector of length k denoting the correct class, and will make this clear. The labels in_
the training data do not cover all the possible labels, and we let the subset of seen labels be S.

**Model.** We consider training a multi-class logistic regression model using full batch gradient
descent. The learned parameters are θ ∈ R[k][×][d] and the model prediction given input x is

_θj[⊤][x]_

_fθ(x) = arg max_ _pθ(x)j = arg max_ (10)
_j_ _j_ _j[′][ θ]j[⊤][′]_ _[x]_

P


-----

where θj is the j-th row of θ. In standard fine-tuning, we run gradient descent on the negative
log-likelihood

_n_

_ℓ(A) = −_ _yi[⊤]_ [log][ p]θ[x][i] (11)

_i=1_

X

where yi ∈ R[k] is a one-hot vector of the class label.

**Distillation loss.** In the Cocktail model, we also add a distillation loss using soft labels from
the prefix-tuned model, which we denote pθ˜ [with parameters][ ˜]θ. The Cocktail loss with parameter
_λ ∈_ [0, 1] is


(1 − _λ)yi[⊤]_ [log][ p]θ[x][i] [+][ λ][(][p][x]θ˜[i] [)][⊤] [log][ p]θ[x][i] (12)
 


_ℓct(A) =_


_i=1_


= ((1 − _λ)yi + λp[x]θ˜[i]_ [)][⊤] [log][ p]θ[x][i] (13)

_i=1_

X

The Cocktail loss encourages the model to learn to output a mixture of the hard label and the soft
label on the training data.

C.1 PROOF OF PROPOSITION 1

_Proof. By induction on t. By zero initialization, the statement holds with g0 = 0._

Suppose for iterations up to t, we have θj[(][t][)] = gt for all j / . We focus on some j / WLOG. The
_∈S_ _∈S_

gradient for θj[(][t][)] is


_∇θj(t)_ _[ℓ][(][θ][(][t][)][) =][ ∇][θ]j[(][t][)]_


_yi[⊤]_ [log][ p]θ[x][i]
_i=1_

X


(14)


_n_ exp((θj[(][t][)][)][⊤][x][i][)]

_−yijxi +_ _xi_ (15)
_i=1_ _j[′][ exp((][θ]j[(][t][′][ )][)]_ _[⊤][x][i][)]_

X

P


exp((θj[(][t][)][)][⊤][x][i][)]

_xi_ (16)
_j[′][ exp((][θ]j[(][t][′][ )][)]_ _[⊤][x][i][)]_

exp(gt[⊤][x][i][)]

_xi_ (17)
_j[′][ exp((][θ]j[(][t][′][ )][)]_ _[⊤][x][i][)]_


_i=1_

_n_

_i=1_

X


since yij = 0 for all i as j is unseen. This gradient does not vary with j, which shows the result.

C.2 PROOF OF PROPOSITION 2

_Proof.Yθ˜_ [be the class probabilities of the prefix-tuned model on training data. With] Let Yθ ∈ R[n][×][k] denote the predicted class probabilities of the model on the training data, and[ λ][ = 1][, the gradient of]
the loss is


_−p[x]θ˜[i]_ _[x]i[⊤]_ [+][ p]θ[x][i] _[x]i[⊤]_ (18)
_i=1_

X


_∇θℓct(θ) =_


= −Yθ˜[X][ +][ Y][θ][X] (19)
Since θ is the minimizer, this gradient is 0 and thus
_Yθ˜[X][ =][ Y][θ][X][ =]_ _Yθ˜_ [=][ Y][θ] (20)
_⇒_

for X with full column rank (right-invertible). We let the right-inverse of X be X _[†]. This implies that_
for all i,

exp(θx[˜] _i)_ exp(θxi)

= (21)
_j_ [exp(˜]θj[⊤][x][i][)] _j_ [exp(][θ]j[⊤][x][i][)] _[.]_

P P


-----

In matrix form, this is equivalent to

exp(θX[˜] ) · diag(1[⊤] exp(θX[˜] ))[−][1] = exp(θX) · diag(1[⊤] exp(θX))[−][1] (22)

=⇒ _θX[˜]_ _−_ **1(log 1[⊤]** exp(θX[˜] ))[⊤] = θX − **1(log 1[⊤]** exp(θX))[⊤] (23)

=⇒ _θ = θ[˜] + 1_ log 1[⊤] exp(θX) log 1[⊤] exp(θX[˜] ) _X_ _[†]._ (24)
h i


On a new input x,

_θx = θx[˜]_ + 1 log 1[⊤] exp(θX) log 1[⊤] exp(θX[˜] ) _X_ _[†]x_ (25)
h i

= θx[˜] + (log(α) + 1[⊤]X _[†]x)1_ (26)

by using the assumption on normalizing constants. Therefore the probability of the j-th class under
the model is


exp(θj[⊤][x][)]
(p[x]θ [)][j] [=] (27)

_j[′][ exp(][θ]j[⊤][′]_ _[x][)]_

Pexp(θ[˜]j[⊤][x][) exp(log(][α][) +][ 1][⊤][X] _[†][x][)]_

= (28)

_j[′][ exp(˜]θj[⊤][′]_ _[x][) exp(log(][α][) +][ 1][⊤][X]_ _[†][x][)]_

Pexp(θ[˜]j[⊤][x][)]

= = (p[x]θ˜[)][j] (29)

_j[′][ exp(˜]θj[⊤][′]_ _[x][)]_

P


-----

