# CONTEXTUALIZED SCENE IMAGINATION FOR GENERATIVE COMMONSENSE REASONING

**Peifeng Wang[1][,][3],** **Jonathan Zamora[2][∗],** **Junfeng Liu[1][∗],**
**Filip Ilievski[3],** **Muhao Chen[1][,][3],** **Xiang Ren[1][,][3]**

1Department of Computer Science, University of Southern California
2Department of Computer Science, University of California, San Diego
3Information Sciences Institute, University of Southern California
_{peifengw,liujunfe,muhaoche,xiangren}@usc.edu,_
jzamoraa@ucsd.edu, ilievski@isi.edu

ABSTRACT

Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative
commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural
text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to
their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning about unseen concept compositions.
In this paper, we propose an Imagine-and-Verbalize (I&V) method, which learns
to imagine a relational scene knowledge graph (SKG) with relations between the
input concepts, and leverage the SKG as a constraint when generating a plausible
scene description. We collect and harmonize a set of knowledge resources from
different domains and modalities, providing a rich auxiliary supervision signal for
I&V. The experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks,
while enabling the model to learn well from fewer task examples and generate
SKGs that make common sense to human annotators [1].

1 INTRODUCTION

Humans describe everyday scenes in natural language based on their understanding of common
concepts encountered in their environment (Tincoff & Jusczyk, 1999). Analogously, the task of
_generative commonsense reasoning (GCSR) asks machines to generate a description of everyday_
situations based on a set of concepts and an initial context (Liu et al., 2020; Li et al., 2021). For
example, given concept words {dog, frisbee, catch, throw}, a machine is expected to generate a
plausible description, e.g., “A man throws a frisbee and his dog catches it in the air”. Machines
with GCSR skills would communicate fluidly with humans, e.g., when summarizing a document by
preserving its key details (Sha, 2020), composing a creative story according to a set of clues (Yao
et al., 2019), and generating a conversation reply that includes specified keywords (Mou et al., 2016).

GCSR poses three unique challenges for automatic text generation methods. To depict plausible
scenes when composing sentences, machines require commonsense knowledge to reason about the
relations between concepts and the affordances of objects (e.g., “dog” performs the action “catch”
but not the action “throw”). Moreover, machines require a compositional generalization ability (Keysers et al., 2019), i.e., the ability to judge the plausibility of a new concept composition that has not
been observed during training, and to identify concepts related to the scene that are not explicitly
provided (e.g., “person” to perform “throw” in the above example).

GCSR can be directly attempted by fine-tuning pre-trained text-to-text language models (LMs) (Raffel et al., 2019; Radford et al., 2019). While pre-trained LMs capture certain encyclopedic knowl
_∗Equal contributions_
[1Code and data are available at https://github.com/wangpf3/imagine-and-verbalize.](https://github.com/wangpf3/imagine-and-verbalize)


-----

**Verbalization Module**

A woman throws a
frisbee and a dog catches
it in the air.


|Input|Concepts: {dog, frisbee, throw, catch}|
|---|---|
|Output|“A woman throws a frisbee and a dog catches it.”|



-  Task 2: Concept2Story

|Input|Context: “My sister arrived early to help me with the family BBQ.” Concepts1: {everyone, arrive} ... ConceptsK: {day, end, fireworks, shoot}|
|---|---|
|Output|“Everyone else arrived soon after... We ended the day shooting off some fireworks.”|


a) Captions

A woman riding a bicycle
along the river.

b) Stories

… When their wagon breaks
down, they make their way
through the woods to a small
hut.

c) Scene graphs

guitar

playing

fedora

person wearing

**Imagination Module (Our focus)**

dog ARG0

catch

woman frisbee ARG1

ARG1

ARG0

throw

Scene Knowledge Graph

Learning to Learning to
imagine verbalize


Figure 1: Overview of the proposed I&V method: (1) We leverage SKGs for unifying scene knowledge from different resources. (2) We pre-train a contextualized imagination module to construct
an SKG for a set of concepts, based on the collected SKG instances. (3) At inference time, our
verbalization module realizes the generated SKG into natural language.

edge mentioned in text corpora (e.g., Wikipedia) (Petroni et al., 2019) and can combine concepts in
novel ways, they may generate grammatically fluent but implausible sentences that conflict with human common sense (Lin et al., 2020). This is because LMs have no intrinsic mechanism to reason
over high-level relations between concepts Zhou et al. (2020). To close the knowledge gap, recent work augment LM input with knowledge graph triples (e.g., (dog, CapableOf, catch)) retrieved
from ConceptNet (Liu et al., 2020; Li et al., 2020), or prototype sentences that cover input concepts
retrieved from external text corpora (Fan et al., 2020; Wang et al., 2021). However, despite the input
augmentation, GCSR skills are implicitly learned based on the concept-text pairs in the training data,
without explicit supervision. While some recent work propose content planning in story generation
in the form of plots or scripts (Yao et al., 2019; Fan et al., 2019), only the narrative order of concepts
are planned in those methods instead of their plausible roles and relations. Given the complexity of
the GCSR task, machines need a direct mechanism to create a high-level relational representation of
the provided concepts, which would allow them to judge the plausibility of their combination.

In this paper, we propose to model an explicit scene imagination step which constructs a structured representation of a plausible scene based on input concepts and initial context. The scene
imagination module formalizes the background knowledge required for the reasoning through a
contextualized relational graph, called scene knowledge graph (SKG). An SKG allows us to collect
and harmonize diverse commonsense knowledge across resources and modalities into a comprehensive SKG distribution (see Figure 1 for an illustration). We develop an imagine-and-verbalize
framework: an imagination module learns to construct a contextualized SKG from input concepts
and context by pretraining over a large amount of external SKGs; a verbalization module learns to
faithfully realize the imagined SKG into natural language by training over downstream datasets. By
learning from a large number of diverse SKGs, our method is able to capture plausible relations
between concepts. By integrating these SKGs with LMs, the imagination module is able to compose new objects in novel ways, and identify implicit concepts for a scene. Imagine-and-verbalize
decomposes the challenging scene description task into two realistic tasks for which a wealth of
training data can be collected, simultaneously enabling for effective and explainable GCSR.

We experiment with two GCSR tasks and three scene graph resources, observing consistently better
or competitive performance to SotA baselines. We find that (1) SKGs extracted from visual captions and story datasets are more helpful than other resources; (2) our model can learn faster (with
less training data) with the help of scene imagination; and (3) the imagination module with a larger
backbone LM demonstrates larger capacity in encoding commonsense knowledge. Our human evaluation study on the generated imagination indicates that these SKGs capture common sense and that
the verbalization module generates the text by following the guidance of the imagination.

2 METHOD

Formally, in GCSR, we consider a list of concepts sets {x[1], x[2], ..., x[K]} and a textual context c ∈C
as input. Each concept set x[i] is unordered and consists of multiple concept words _xj_ . A concept
_{_ _}_


-----

word xj (or concept for brevity) is a commonly seen object (nouns such as “dog” or “frisbee”)
or commonly performed action (verbs such as “ ∈X _throw” or “catch”). The goal of GCSR is to generate_
_K sentences {y[1], y[2], ..., y[K]}, each describing a plausible situation following human common sense_
for a concept set x[i]. The i-th sentence y[i] _⊂Y should be generated using all concepts in x[i]._

We consider two variants of GCSR: 1) concepts-to-sentence generation (Lin et al., 2020), where no
context is given (i.e., c is empty) and only one concept set is provided (K = 1); and 2) concepts-to_story generation task, where c is the leading sentence of a multi-sentence story and more than one_
concept sets are provided, each corresponding to one sentence to be generated (K > 1). Both tasks
are evaluated by comparing the machine-generated text with human-generated (gold) references.

2.1 THE IMAGINE-AND-VERBALIZE APPROACH

Pre-trained LMs struggle with learning a generalizable mapping from concepts to plausible sentences solely based on the training data. Augmenting concepts with external knowledge to form the
_′_ _′_
input X and fine-tuning a pretrained LM to model P (Y|C, X ) (Liu et al., 2020; Fan et al., 2020;′
Li et al., 2021) alleviates this issue partially, while still learning a direct mapping of {C, X _} →Y._
In this work (Figure 1), we decompose the GCSR task into two sub-tasks, namely contextualized
scene generation (imagination) and scene-aware text generation (verbalization):


_P_ (Y|C, X _, Z)P_ (Z|C, X ), (1)


_P_ (Y|C, X ) =


where Z denotes the scene representation for the given concepts and context.

The contextualized scene imagination module P (Z|C, X ) aims to construct a multi-relational graph
representation Z (scene knowledge graph, or SKG) that describes a plausible scene that involves all
input concepts and corresponds to the provided context. To learn this module, we collect a diverse
set of SKG instances from different resources and modalities to form a comprehensive distribution
of scenes (§2.2). The imagination module is pre-trained over the collected scene instances and learns
to generate SKGs depicting plausible day-to-day situation. The imagination module is based on a
neural architecture, which enables it to generate concept compositions that might not have been
observed during training (§2.3).[2] We leverage the contextualized SKG for text generation with a
verbalization module P (Y|C, X _, Z) which takes the context, concepts, and the generated SKG as_
input, and composes a grammatical and plausible scene description in natural language (§2.4).

To perform GCSR, where one or multiple concept sets are given, we apply the imagination module to
sample z[i]. Since the marginalization over Z is generally intractable due to the complex structure of
the SKGs, we only sample the most probable scene representation z[∗][i] that maximizes P (z[i]|c[′], x[i]),
where c[′] includes the given context c and the previously generated y[j], (j < i) . We then apply the
verbalization module to generate one sentence at a time by sampling from P (y[i]|c[′], x[i], z[i][∗]). Multiple
sentences are generated by iteratively applying the imagination and verbalization modules.

2.2 IMAGINATION VIA GENERATING SKG

**Imagination through SKGs We adopt the term “scene graph” from the computer vision com-**
munity, and we generalize it to a novel relational schema that represents knowledge from multiple
modalities. Our SKG is defined as a relational graph G = (E, R) that organizes a set of concepts in
a coherent scene. The node set E of the graph includes both given and implicit concepts, while each
relation (edge type) r ∈R denotes how two concepts should be related. We follow the Abstract
Meaning Representation (AMR) (Banarescu et al., 2013) schema to consider the core relations between two concepts, which corresponds to the commonsense knowledge required by GCSR. Table 7
in the appendix illustrates a few representative relations and their examples.

**Collecting Diverse SKGs We consider two complementary modalities, text and vision, as some**
concepts and relationships are more likely to occur in one modality versus another. (1) Textual
_Modality: According to pragmatic principles of human language, people generally leave out ex-_
pected details about common scenes (Grice, 1975). For this reason, we extract SKGs from visual
captions and narrative stories, in which human annotators are asked to explicitly describe scenes that

2The imagination module can be further fine-tuned over the downstream datasets.


-----

_Gold SKGs from external resources_
for continual pretraining

task dataset for fine-tuning


Graph linearization

“throw <:ARG0> woman <SEP> throw <:ARG1> frisbee …”

“Context: People are playing on the grass. Concepts:

|Col1|Col2|
|---|---|
|Transformer||
|||

dog <SEP> frisbee <SEP> catch <SEP> throw”


) Silver SKGs from


Randomly ordered concepts


Figure 2: Continual pretraining and fine-tuning of the imagination module to output a linearized
SKG based on a sequential input (context and concepts).

may happen using descriptive language as shown in Figure 1(a,b). To extract an SKG out of these
textual signals, we adopt the AMR parsing tool to transform each sentence into an AMR graph. This
process yields a single SKG per sentence. For the story SKGs, we also keep the sentences (up to 256
tokens) that precede the sentence that corresponds to the SKG, as context c. (2) Visual Modality:
Image captions focus on salient information and may not capture all useful visual signals. Thus,
we also capture the scene structures directly from images, by using VisualGenome (Krishna et al.,
2016), a large-scale scene graph dataset annotated by humans. To adopt a unified SKG schema, we
manually map the relations in scene graphs from VisualGenome to the ones used in textual SKGs.
A full set of mapping rules can be found in the Appendix (A.1). The statistics of the SKGs collected
from each resource/modality are summarized in Table 1. We note that visual scene graphs may be
biased towards knowledge about spatial relationships and object affordance, which further motivates
our decision to extract SKGs from multiple modalities.

2.3 LEARNING THE SCENE IMAGINATION MODULE


We describe how we pre-train the scene imagination
model using multimodal SKG examples collected from
diverse sources, and how we fine-tune the imagination
module to downstream datasets.


Table 1: Statistics of the SKG instances collected from different resources.

**Knowledge source** **# SKGs** **# Concepts**


A straightforward way to construct a SKG is to retrieve Caption-AMRStory-AMR 584,252927,163 41,27222,961
instances that contains all the given concepts from the VG-SceneGraph 292,596 41,629
collected SKGs. However, performance of such method All 1,792,941 84,835
is limited by the coverage of the SKG collection and will
fail when encountering novel concept composition. We
propose to model P (Z|C, X ) with a neural graph generator. Inspired by previous work on (conditional) graph generation (You et al., 2018), we formulate SKG construction as an auto-regressive
sequence generation task, where a linearized SKG is generated sequentially conditioned on the context, input concepts, and the graph sequence generated so far. Since the sequence generation problem
can be efficiently tackled by pre-trained auto-regressive LMs (e.g., GPT-2 Radford et al. (2019)), we
adopt LMs as the backbone of our imagination module (Bosselut et al., 2019; Wang et al., 2020).

**Linearized SKG Generation To form training instances for the imagination module, we treat the**
nodes in an SKG instance as input concepts and the linearized SKG as the target output (Figure 2).
The input concepts are concatenated into a sequence x = [x1, x2, ..., xn], preceded by the context
**c[′]** _∈C. When c[′]_ is not given, we prepend the word “none” to the concept sequence. To linearize
an AMR-based SKG into a sequence z = [z1, z2, ..., zm], we adopt the PENMAN serialization
format (Goodman, 2020) which converts AMR into a spanning tree over the graph. This format is
shown to be more suitable than other linearization strategies like depth-first-search (DFS) in enabling
LMs to learn the graph structure (Mager et al., 2020).

During training, we randomize the order of the concepts at every training step such that the graph
generator learns to be invariant to concept order (Zhang et al., 2019). For each training instance, we
randomly discard a small subset of the SKG nodes (concepts) in each training epoch. This simulates
the scenario in which a subset of the concepts that constitute a scene will be given, thus teaching the
model to infer implicit concepts for completing a plausible scene.


-----

**Continual-Pretraining and Fine-tuning With both the input concepts (plus context) and the output**
graph linearized as sequences based on the collected SKG instances, we continually pretrain an autoregressive LM to generate z = Transformer(c[′], x). The training objective is to maximize P (Z|C, X )
by minimizing the negative log-likelihood:

_t=m_


log P (zt _z<t, c[′], x)._ (2)
_|_
_t=1_

X


_Limagine = −_


Our pre-trained imagination module generates an SKG on the fly, and it can be further fine-tuned
on downstream datasets, when their distributions of context and concepts are different from the
pretraining data (see Figure 2 for illustration). Since downstream datasets cannot be expected to have
ground-truth SKGs paired with each training example, we apply the AMR parsing tool described
in §2.2 on the training sentences to obtain silver-standard SKGs. We then follow the same training
procedure to continually pretrain the module into a customized imagination module for a specific
downstream dataset.

2.4 SCENE-AWARE VERBALIZATION

**Iterative Imagine-and-Verbalize At the inference time, we apply the trained imagination mod-**
ule iteratively to generate the most plausible SKG for each given concept set x[i], i.e., z[i][∗] =
arg maxzi P (z[i] **c[′], x[i]), where the context c[′]** includes both the given context c and the previously
_|_
generated sentences {y[j]} (j < i). The generated SKG is used by the scene-aware verbalization
module to model P (Y|C, X _, Z). The verbalization module generates the i-th sentence by sampling_
from P (y[i]|c[′], x[i], z[i][∗]). Multiple sentences are generated iteratively by alternating between the scene
imagination (to construct SKG) and verbalization (to produce the next sentence). See Figure 3 for
an illustration of this iterative inference process.


|Col1|Col2|
|---|---|
|zation Transformer2||
|||


{Context, previously generated sentences, concepts}


**Model Training Since both the linearized SKG**
(generated by the imagination module) and the target sentences are sequences by nature, we design
_P_ (Y|C, X _, Z) as a sequence-to-sequence gener-_
ative model and learn this verbalization module
by fine-tuning another pre-trained auto-regressive
LM, i.e., y[i] = Transformer(c[′], x[i], z[i]). To form
the input for generating the sentence y[i], we concatenate the context c[′], the concept set sequence
**x[i]** and z[i] into one sequence[3] as illustrated in Figure 3. We then train the model to maximize
_P_ (Y|C, X _, Z) by minimizing the negative log-_
likelihood:

_t=l_

|Col1|Col2|
|---|---|
|ation Transformer1||
|||


“A woman throws a frisbee and a dog catches it.”

**Verbalization** Transformer2

“Context: People are playing on the grass. Concepts:
dog <SEP> frisbee <SEP> catch <SEP> throw.
**Relations: throw <:ARG0> woman <SEP> throw**
<:ARG1> frisbee …”

Linearized SKG

**Imagination** Transformer1


log P (yt[i][|][y]<t[i] _[,][ c][′][,][ x][i][,][ z][i][)][.][ (3)]_
_t=1_

X


_Lverbalize = −_


_verbalize_ _t[|]_ _<t_
_L_ _−_ _t=1_ Figure 3: Our I&V method iteratively applies the

X imagination and the verbalization modules, by gen
For each training instance (y[i], c[′], x[i]), we con- erating one sentence in each iteration.
struct two types of SKG instances as the input z[i]:
(1) We perform AMR parsing on y[i] to obtain a silver-standard SKG; (2) We apply the trained imagination module to generate a SKG z[i][∗] = arg maxzi P (z[i] **c[′], x[i]), where c[′]** includes the given context
_|_
**c and the ground-truth prefix sentences {y[j]} (j < i). We find it beneficial to train the verbalization**
module over these two types of SKGs as evidenced by our ablation study (§A.7). During inference,
the SKG z[i] is generated by the imagination module, while c[′] includes the given context c and the
previous sentences {y[j]} (j < i) generated by the verbalization module.

3 EXPERIMENTAL SETUP

**Tasks & Datasets We consider two GCSR tasks: Concept2Sentence and Concept2Story. (1) Con-**
_cept2Sentence is a task of generating a single sentence for a given set of concepts and no context._

3Our ablation study in Appendix A.6 shows that including all these elements as input is helpful.


-----

We evaluate Concept2Sentence on the CommonGen (Lin et al., 2020) benchmark. Since the labels
of the official test set are not publicly available, we submit our method to the leaderboard to obtain
its performance. Notably, the concept sets in CommonGen’s test set are novel and do not appear in
the training set. We also create an in-house split of CommonGen to facilitate comparison between
different variants of our method and the baselines. (2) Concept2Story is a generalization of the concept2sentence task, where the goal is to generate a coherent story with K = 4 sentences given a set
of concepts and an initial verbal context. We construct two benchmarks based on the Visual Story
Telling (VIST) (Huang et al., 2016) and ROCStories (Mostafazadeh et al., 2016) datasets. Following CommonGen, we conduct part-of-speech tagging over the sentences and further lemmatize the
recognized verbs and nouns to obtain the concept sets.

**Baselines (1) Concept2Sentence: We consider several recent submissions to the leaderboard of**
CommonGen that leverage auxiliary information for GCSR. KFCNet (Li et al., 2021), Re-T5 (Wang
et al., 2021), and EKI-BART (Fan et al., 2020) are prototype-based models, which retrieve sentences
containing as many input concepts as possible from external captions and NLI datasets, and then use
these sentences as auxiliary inputs. VisCTG (Feng et al., 2021b) is an image-augmented model
which retrieves images from Google by using concepts as a query, followed by an image captioning model that generates captions as auxiliary inputs. KG-BART (Liu et al., 2020) is a knowledge
graph-augmented model which retrieves relations between concepts from ConceptNet as auxiliary
inputs. SAPPHIRE (Feng et al., 2021a) is a keyword-based model which extracts keywords from
sentences as auxiliary inputs only during training. We also compare to Node2Text, which fine-tunes
a pre-trained auto-regressive LM to take the concatenation of concepts as input and output the target
sentences. (2) Concept2Story: We augment Node2Text with the iterative generation pipeline as in
our method, which generates the next sentence given the provided context, previously generated sentences and the current concept set. In addition, we experiment with two representative methods from
the controlled text generation literature. Plan-and-write (Yao et al., 2019) first generates storyline
keywords, then uses the keywords to generate a story. We use the concept set and context to generate
storyline keywords. Action-Plan (Fan et al., 2019) uses predicate-argument pairs as storyline. We
adapt the KFCNet model to retrieve prototype sentences. All Concept2Story baselines are used in
an iterative generation pipeline, to enable fair comparison to our method.


**Evaluation Metric We evaluate systems**
against the K reference sentences provided by a dataset, by measuring the
similarities between the machine-generated
text and the gold references. Following CommonGen (Lin et al., 2020), we
adopt widely-used automatic metrics for
evaluating text generation, which focus
on (1) n-gram overlap: BLEU (Papineni
et al., 2002), ROUGE (Lin, 2004), and
METEOR (Banerjee & Lavie, 2005), and
(2) concept association: CIDEr (Vedantam et al., 2015) and SPICE (Anderson
et al., 2016). Lin et al. (2020) reports that
SPICE yields the best correlation with human judgments and thus we used it as the
main evaluation metric.

4 RESULTS AND ANALYSIS


Table 2: Performance comparison with the top-ranked,
published models on the official CommonGen test set.
_∗Note that KFCNet uses a much larger corpora (over 70M)_
to retrieve prototypes and on average less than one concept
in the concept sets is not covered (Li et al., 2021), while we
filter out any SKGs that contain concept sets that overlap
with CommonGen dataset.

**Model** **BLEU-4** **CIDEr** **SPICE**

KFCNet (Li et al., 2021)[∗] **43.62** **18.85** **33.91**
RE-T5 (Wang et al., 2021) 40.86 17.66 31.08
VisCTG (Feng et al., 2021b) 36.94 17.20 29.97
SAPPHIRE Feng et al. (2021a) 37.12 16.90 29.75
KG-BART Liu et al. (2020) 33.87 16.93 29.63
EKI-BART Fan et al. (2020) 35.95 17.00 29.58
T5-base (our implementation) 33.81 15.79 28.34
T5-large (our implementation) 32.85 15.76 28.38
T5-large (reported) 31.96 15.13 28.86

I&V (T5-base) 40.16 17.44 30.57
I&V (T5-large) 40.57 17.71 31.29


We design experiments to answer the following questions: (1) Does contextualized scene imagination improve the performance of GCSR models? (2) Does imagination allow GCSR models to
learn with less data? (3) How does each source of scene knowledge for pretraining affect the GCSR
performance? (4) Do generated SKGs make common sense and correspond to the generated text?

4.1 MAIN RESULTS

We compare our proposed approach with state-of-the-art text generation methods on two GCSR
tasks to understand whether scene imagination helps GCSR. Table 2 shows the performance of dif

-----

Table 3: Performance of the compared methods on the Concept2Story tasks. Best results are bold-faced. We
mark them with an asterisk if they exceed the second best with statistical significance (p-value < 0.05).

**Concept2Story-VIST** **Concept2Story-ROC**

T5-base BART-large T5-base BART-large

**Model** BLEU-4 CIDEr SPICE BLEU-4 CIDEr SPICE BLEU-4 CIDEr SPICE BLEU-4 CIDEr SPICE

Node2Text 20.64 25.41 58.55 18.52 22.91 55.48 23.31 29.32 57.66 20.60 26.09 53.80
Keyword 16.75 21.87 56.23 15.62 20.86 55.49 22.24 27.05 50.41 22.14 27.40 49.52
Action-Plan 17.84 22.77 57.11 16.20 21.10 54.77 21.15 27.32 56.14 20.45 26.29 54.32
Prototype 20.28 25.05 58.17 **22.81** **26.93** 58.84 23.59 29.48 57.68 26.76 31.60 58.35

I&V **21.05[∗]** **25.78[∗]** **59.21[∗]** 22.45 26.80 **59.11[∗]** **26.77[∗]** **32.33[∗]** **60.63[∗]** **28.30[∗]** **33.40[∗]** **60.39[∗]**

ferent models on CommonGen. We have the following observations. First, I&V drastically improves
the vanilla T5-large model (Node2Text), demonstrating the effectiveness of the imagination module
in GCSR. We also provide concrete examples in §A.4 which showcase how imagination fixes errors
made by Node2Text. All these errors can be attributed to the fact that Node2Text does not properly
capture the commonsense relations between concepts while our imagination module learns how concepts are related from indirect supervision. Second, our model outperforms other models using different auxiliary inputs, including prototypes (Re-T5 and EKI-BART), knowledge facts (KG-BART)
and images (VisCTG), showing the benefit of SKGs over these knowledge sources. Although our
model under-performs KFCNet, our analysis in their work reveals that 97.4% of the test cases have
perfectly matched prototypes, i.e., sentences containing all the queried concepts. It is thus unclear
whether KFCNet is conducting commonsense reasoning or merely rephrasing the prototypes. Note
that we filter out any collected SKGs that cover the concept sets from the downstream datasets. This
ensures that the imagination module is examined with its compositional generalization.

Table 3 shows the experimental results by I&V on the two Concept2Story datasets using T5-base and
BART-large as the backend respectively. Among most evaluation metrics, our method outperforms
Node2Text and baselines with other intermediate representations incorporated in the same backends.
This demonstrates that our imagination module can provide contextualized scene imagination that
are more helpful in guiding long narrative generation.

4.2 PERFORMANCE ANALYSIS

**How does the knowledge source affect GCSR?** We perform an ablation study in order to understand how effectively each source of SKGs contributes to the imagination. Specifically, we use each
of the following SKG sources to pre-train an imagination module using T5-large as the backend: the
silver-standard SKGs extracted from the training set from the downstream task (Task-AMR), and
the external SKGs: Caption-AMR, Story-AMR, and VG-SceneGraph (§2.2). For CommonGen, we
do not further fine-tune the imagination module in order to distinguish the contributions from each
knowledge source more clearly. For Concept2Story (ROCstories), we conduct further fine-tuning
using the task-AMR. Since this task provides the context as input, we find it helpful to adapt the
imagination module with the task dataset.

The results are shown in Table 4 and we have the following observations. For CommonGen, the
contribution comes mostly from the SKGs based on Caption-AMR while being less from VGSceneGraph. This may due to the fact that VG-SceneGraph is biased towards spatial relations and
attributes of objects. For Concept2Story, we find both Story-AMR and Caption-AMR to be helpful for continual pretraining. The former teaches the model to generate contextualized imagination
which is necessary for story generation in particular while the latter teaches the model about general
commonsense knowledge. For both datasets, the imagination modules that are pre-trained over all
the SKG instances yield significantly better results than the ones trained on the task-AMR datasets.
This validates our intuition that different sources of SKGs contain complementary commonsense
knowledge, and they should be used together for machine imagination.

**How does the backbone LM size affect the module’s performance?** We also ablate the LM
architecture of the imagination module and the verbalization module respectively to see how our
method work with different pre-trained LMs. For the imagination module, we use T5-base and
T5-large. This is to investigate how the capacity of LMs affects the learning of scene knowledge.


-----

Table 4: Performance of our method using different SKG sources to
train the imagination module, with T5-large as the backbone LM.

**CommonGen (in-house)** **Concept2Story-ROC**

**Knowledge Source** BLEU-4 CIDEr SPICE BLEU-4 CIDEr SPICE

Task-AMR 28.87 15.74 31.22 23.14 29.25 57.91
Caption-AMR 32.21 16.14 32.16 23.77 29.76 58.46
Story-AMR 23.73 13.51 27.53 24.17 30.10 58.59
VG-SceneGraph 21.00 13.36 29.07 22.84 25.33 53.96


Table 5: SPICE performance of our
method using different sizes of T5 as
backbone for the imagination module.

Dataset / Backbone LM **T5-base** **T5-large**

CommonGen (in-house) 32.00 33.49
Concept2Story-ROC 59.56 60.63

32.5

30.0

27.5

25.0

22.5


12.5

|Col1|Col2|Col3|Col4|
|---|---|---|---|
|||||
|||||
|||||
||||I&V (general) Node2Text Action-Plan KFCNet|


50 500 5000 full

I&V (general)
Node2Text
Action-Plan
KFCNet

# Training Examples (ComGen)

|All-SKG 33.27 16.95 33.49 26.77 32.33 60.63|Col2|Col3|Col4|Col5|Col6|
|---|---|---|---|---|---|
|All-SKG 33.27 16.95 33.49 26.77 32.33 60.63 62 60 60 Node2Text 60 I&V 50 50 58 SPICE 56 40 40 SPICE 54 30 52 30 I&V (fine-tuned) I&V (fine-tuned) I&V (general) I&V (general) 50 BART-base BART-large T5-base T5-large N Keo yd we2 oT rde sxt 20 N Keo yd we2 oT rde sxt 20 Action-Plan Action-Plan Prototype Prototype||||||
|||||||
|||||||
|||||I&V (fine-tun I&V (general)|ed)|
|||||Node2Text Keywords Action-Plan Prototype||

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||||I&V (fine-tun I&V (general)|ed)|
||||Node2Text Keywords Action-Plan Prototype||


50 500 5000 full

# Training Examples (VIST)


50 500 5000 full

# Training Examples (ROC)


Figure 4: Ablation study on backbone LM sizes of our verbalization
module and Node2Text using the
Concept2Story-ROC dataset.


Figure 5: Results (SPICE) of the low-resource experiment on the three
benchmark datasets with different number of training examples.


The results are shown in Table 5. Compared to T5-large, we observe a slight performance drop for
T5-base, which indicates that larger LMs are able to encode our rich set of SKG instances in a more
expressive manner. For the verbalization module, we use BART-base/large and T5-base/large. The
results are shown in Figure 4. We observe that compare to baseline, our method consistently yields
a better performance regardless of what LM architecture is used.

**Does imagination allow models to learn (faster) with less data?** Next, we study how the indirect
supervision provided to the imagination module help the system effectively learn with limited taskspecific training data. Accordingly, we conduct a low-resource experiment where we randomly
sample {50, 500, 5000} training and development examples from each dataset. For each data size,
we use 5 random seeds to obtain 5 different training and development splits. On each split, we train
and test with random initialization of 3 seeds, and we report the average on the total 15 ways of
results. In this study, the imagination module is fixed untrainable after continual pretraining and is
not fine-tuned over the sampled task datasets.

Figure 5 shows that our model consistently outperforms the baselines, and the performance gain is
larger when less training data are used. This indicates that rich sources of SKGs provide practical
forms of indirect supervision to complement limited task-specific training data. The robustness of
our model in low-resource settings also justifies the need for including contextualized SKGs as an
intermediate representation, which further enhances the verbalization module to generate plausible
sentences even with little training data.

**Is context helpful for imagination?** To validate that the textual context, including the provided
context as well as the previously generated sentences, is helpful for imagination in the Concept2Story task, we conduct an ablation study where we learn an uncontextualized imagination
module which only takes concepts as input. The final results on VIST and ROC datasets are 47.32
and 45.18 (SPICE) respectively, which are much lower than the results from contextualized I&V
(59.21 and 60.63). This demonstrates that the context is critical in generating SKGs which are more
relevant to the story line and thus lead to better text generation.

4.3 HUMAN EVALUATION ON GENERATED SKGS

We conduct human evaluation on the SKGs generated by our imagination module to examine their
quality. Annotators are presented with the input concepts, the generated SKGs, the predicted sen

-----

tences resulting from the corresponding SKGs and the ground-truth sentences for reference. For
each dataset, 100 instances are randomly chosen for evaluation. Annotators are students majoring
in computer science and not all of them know about AMR language prior to the human evaluation.
To facilitate annotators’ understanding of the evaluation task and AMR, we provide the detailed
instruction and the examples of AMR relations. The annotators are asked to judge for: 1) Complete_ness, whether the SKG includes all the concepts (both given and implicit) to constitute a coherent_
scene; 2) CommonSense, whether the SKG organizes the concepts in a way that follows common
sense; 3) Alignment, whether the generated sentence aligns with the SKG and 4) Similarity, whether
the predicted sentence is similar to any referenced sentence in semantic. Annotation is based on a
3-point scale: a) 0 – “I do not agree”, b) 0.5 – “I partially agree” and c) 1.0 – “I fully agree”.

Table 6 shows the evaluation results where we get a fair level Table 6: Human evaluation on the
of agreement measured by Fleiss Kappa (κ = 0.21). We generated SKGs regarding Completeness
observe that the generated SKGs are complete and follow (COM), CommonSense (CS) and Aligncommon sense in a high degree across three datasets, which _ment (AL) and Similarity (SIM)._
demonstrates the effectiveness of learning useful common- **COM** **CS** **AL** **SIM**
sense knowledge with vast indirect supervision from differ- CommonGen 97.30 90.15 89.90 88.30
ent resources. Moreover, the SKGs are well-aligned with the VIST 93.80 89.70 91.40 76.20
generated text, which indicates that the verbalization module ROC 95.70 86.60 87.80 75.68
consistently follows the guidance of the imagination module
when generating sentences. The moderate similarity scores validate that the generated text is generally similar to the natural language sentences annotated by humans.

5 RELATED WORK

**Knowledge-Enhanced GCSR** Recent works (Liu et al., 2020; Li et al., 2021) on GCSR propose
to retrieve external knowledge to enhance the text generation. Prototype-based models, including
EKI-BART (Fan et al., 2020), Re-T5 (Wang et al., 2021), and KFCNet (Li et al., 2021) retrieve
massive prototype sentences from external corpora like visual captions and Wikipedia as auxiliary
input to the LM. Though the retrieved prototype sentences provide high coverage on the concepts,
their model is supervised to compose sentences that are very similar to those existing prototypes. It
is thus unclear whether their models are conducting commonsense reasoning or only mimicking the
prototypes. KG-BART (Liu et al., 2020) incorporates the embedding of relational facts about the
concepts from ConceptNet into both the encoders and decoders of the BART architecture (Lewis
et al., 2020). As there could be multiple relations between two concepts, it is unclear how to select
the relation that fits a given context (Fadnis et al., 2019). Our imagination module infers the relations
between concepts by taking all the concepts into consideration and organizes them in a coherent way.

**Content Planning** Our method is also related to prior works (Goldfarb-Tarrant et al., 2020) that
propose intermediate representations as a way to “plan ahead” before generating long narratives.
Plan-and-write (Yao et al., 2019) generates chains of keywords as a storyline, but do not consider
relations between keywords (concepts) as we do. Action-plan (Fan et al., 2019) takes a step further
by using predicate-argument with semantic role labeling, but still does not involve all the concepts
in a sentence. Moreover, these methods are limited to obtaining supervision from task-specific
datasets, while we gather effective indirect supervision signals from rich multi-source, multi-modal
SKG representations without the need for additional annotations.

6 CONCLUSIONS

This paper proposed to enhance neural architectures for GCSR with an intermediate imagination
layer. We divided the GCSR process into two steps: imagination, which generated a plausible scene
knowledge graph for a given set of concepts, and verbalization, which transformed this scene graph
into a fluent sentence that corresponds to human common sense. The method was trained with
diverse scene knowledge graphs derived from both text and vision modalities. Our experiments
demonstrated the ability of the proposed method to perform GCSR effectively, by describing plausible scenes, and efficiently, by requiring less training data. The image caption graphs proved most
beneficial to learn from. Future work should investigate the impact of imagination on interactive
commonsense tasks, like dialogue generation, and include scene graphs from the audio modality.


-----

ACKNOWLEDGMENTS

We thank the anonymous reviewers and all the collaborators in USC INK research lab for their
valuable feedback. This material is based upon work sponsored by the DARPA MCS program under
Contract No. N660011924033 with the United States Office Of Naval Research.

REFERENCES

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pp. 382–398.
Springer, 2016.

Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin
Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. Abstract meaning representation
for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability
_with discourse, pp. 178–186, 2013._

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic
_evaluation measures for machine translation and/or summarization, pp. 65–72, 2005._

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin
Choi. COMET: Commonsense transformers for automatic knowledge graph construction. In
_Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp._
4762–4779, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/
[P19-1470. URL https://www.aclweb.org/anthology/P19-1470.](https://www.aclweb.org/anthology/P19-1470)

Kshitij Fadnis, Kartik Talamadupula, Pavan Kapanipathi, Haque Ishfaq, Salim Roukos, and
Achille Fokoue. Heuristics for interpretable knowledge graph contextualization. arXiv preprint
_arXiv:1911.02085, 2019._

Angela Fan, Mike Lewis, and Yann Dauphin. Strategies for structuring story generation. arXiv
_preprint arXiv:1902.01109, 2019._

Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuanjing Huang,
Nan Duan, and Ruofei Zhang. An enhanced knowledge injection model for commonsense generation. arXiv preprint arXiv:2012.00366, 2020.

Steven Feng, Jessica Huynh, Chaitanya Prasad Narisetty, Eduard Hovy, and Varun Gangal. SAPPHIRE: Approaches for enhanced concept-to-text generation. In Proceedings of the 14th Inter_national Conference on Natural Language Generation, pp. 212–225, Aberdeen, Scotland, UK,_
[August 2021a. Association for Computational Linguistics. URL https://aclanthology.](https://aclanthology.org/2021.inlg-1.21)
[org/2021.inlg-1.21.](https://aclanthology.org/2021.inlg-1.21)

Steven Y Feng, Kevin Lu, Zhuofu Tao, Malihe Alikhani, Teruko Mitamura, Eduard Hovy, and
Varun Gangal. Retrieve, caption, generate: Visual grounding for enhancing commonsense in text
generation models. arXiv preprint arXiv:2109.03892, 2021b.

Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. Content
planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020
_Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4319–4338,_
Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.
[emnlp-main.351. URL https://aclanthology.org/2020.emnlp-main.351.](https://aclanthology.org/2020.emnlp-main.351)

Michael Wayne Goodman. Penman: An open-source library and tool for AMR graphs. In Pro_ceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Sys-_
_tem Demonstrations, pp. 312–319, Online, July 2020. Association for Computational Linguis-_
[tics. doi: 10.18653/v1/2020.acl-demos.35. URL https://aclanthology.org/2020.](https://aclanthology.org/2020.acl-demos.35)
[acl-demos.35.](https://aclanthology.org/2020.acl-demos.35)

H Paul Grice. Logic and conversation, syntax and semantics. Speech Acts, 3:41–58, 1975.


-----

Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob
Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling.
In Proceedings of the 2016 Conference of the North American Chapter of the Association for
_Computational Linguistics: Human Language Technologies, pp. 1233–1239, 2016._

Daniel Keysers, Nathanael Sch¨arli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin,
Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference
_on Learning Representations, 2019._

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie
Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual
genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
[URL https://arxiv.org/abs/1602.07332.](https://arxiv.org/abs/1602.07332)

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the
_58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, 2020._

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, and Nan Duan. Kfcnet: Knowledge filtering and contrastive learning network for generative commonsense reasoning. arXiv
_preprint arXiv:2109.06704, 2021._

Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra, Har Simrat Singh, Jonathan Francis, Kaixin Ma,
Eric Nyberg, and Alessandro Oltramari. Lexically-constrained text generation through commonsense knowledge extraction and injection. arXiv preprint arXiv:2012.10813, 2020.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and
Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense
reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
[1823–1840, Online, November 2020. Association for Computational Linguistics. URL https:](https://www.aclweb.org/anthology/2020.findings-emnlp.165)
[//www.aclweb.org/anthology/2020.findings-emnlp.165.](https://www.aclweb.org/anthology/2020.findings-emnlp.165)

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization
_branches out, pp. 74–81, 2004._

Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S Yu. Kg-bart: Knowledge graph-augmented
bart for generative commonsense reasoning. arXiv preprint arXiv:2009.12677, 2020.

Manuel Mager, Ram´on Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee,
Radu Florian, and Salim Roukos. Gpt-too: A language-model-first approach for amr-to-text
generation. arXiv preprint arXiv:2005.09123, 2020.

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American
_Chapter of the Association for Computational Linguistics: Human Language Technologies, pp._
839–849, San Diego, California, June 2016. Association for Computational Linguistics. doi:
[10.18653/v1/N16-1098. URL https://aclanthology.org/N16-1098.](https://aclanthology.org/N16-1098)

Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence to backward and forward
sequences: A content-introducing approach to generative short-text conversation. arXiv preprint
_arXiv:1607.00970, 2016._

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association
_for Computational Linguistics, pp. 311–318, 2002._

Fabio Petroni, Tim Rockt¨aschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and
Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Con_ference on Empirical Methods in Natural Language Processing and the 9th International Joint_
_Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473, Hong Kong,_


-----

China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1250. URL
[https://www.aclweb.org/anthology/D19-1250.](https://www.aclweb.org/anthology/D19-1250)

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language
models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text
transformer. arXiv preprint arXiv:1910.10683, 2019.

Lei Sha. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the
_2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8692–_
8703, 2020.

Ruth Tincoff and Peter W Jusczyk. Some beginnings of word comprehension in 6-month-olds.
_Psychological science, 10(2):172–175, 1999._

Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image
description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern
_Recognition (CVPR), June 2015._

Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng.
Retrieval enhanced model for commonsense generation. arXiv preprint arXiv:2105.11174, 2021.

Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro Szekely, and Xiang Ren. Connecting the
dots: A knowledgeable path generator for commonsense question answering. _arXiv preprint_
_arXiv:2005.00691, 2020._

Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-andwrite: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial
_Intelligence, volume 33, pp. 7378–7385, 2019._

Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. In International conference on machine
_learning, pp. 5708–5717. PMLR, 2018._

Yan Zhang, Jonathon Hare, and Adam Pr¨ugel-Bennett. Fspool: Learning set representations with
featurewise sort pooling. In International Conference on Learning Representations, 2019.

Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren.
Rica: Evaluating robust inference capabilities based on commonsense axioms. arXiv preprint
_arXiv:2005.00782, 2020._

A APPENDIX

A.1 RULES FOR MAPPING VISUAL SCENE GRAPHS TO SKG

There are 3858 relation types in our processed VisualGenome dataset due to the noisy annotation.
We map these relations into 8 relations. For relations that are annotated as verbs by VisualGenome,
we break the relationship (subject, relation, object) into (relation, :ARG0, subject) and (relation,
:ARG1, oject). For other popular relations, we conduct the following mapping:(subject, be, object)→(subject, domain, object), (subject, displace, object)→(subject, possible, object),(subject,
have/of, object)→(subject, part, object),(subject, with, object)→(subject, poss, object),(subject,
on/behind/at/under/along/in/..., object)→(subject, location, object). The remaining relations that do
not follow the above mapping criteria are mapped to an ”other” relation. Note that the 7 non-”other”
relations make up 97.73% of the triplets in VisualGenome.


-----

Table 7: The most common relation types in SKG instances and their example triplets.

Relation types Examples

ARG1 (play, ARG1, guitar)
ARG0 (play, ARG0, man)
ARG2 (ask, ARG2, girl)
Location (play, Location, stage)
Time (play, Time, sing)
Op1 (down, Op1, stair)
Part (dog, Part, ear)

A.2 IMPLEMENTATION DETAILS

For the main experiments, we develop the imagination module by continually pre-train a T5-large
model over the caption, story, and vision SKGs. We then further adapt the imagination module
over each task dataset annotated with the silver-standard SKGs by further fine-tuning. To train the
verbalization module, we fine-tune T5-base and BART-large as two backend LMs. During training,
we use both silver-standard SKGs and generated SKGs, while averaging the training loss associated
with each of them. We use the Adam optimizer with weight decay 1e − 2. We search the optimal
hyper-parameters based on the perplexity over the development set, where the learning rate is chosen
from {2e − 6, 1e − 5, 3e − 5, 1e − 4}, the batch size is chosen from {8, 16, 32, 64, 128}.

A.3 STATISTICAL ANALYSIS

Table 8: Statistical analysis (p-values) for the ablation study on the backend LM used by the verbalization module and the low-resource experiment. < 0.01 indicates a significant improvement
and < 0.05 indicates a fairly significant improvement, and NA indicates that our method does not
outperform the best baseline.

BART-base BART-large T5-base T5-base

_< 0.01_ _< 0.01_ _< 0.01_ _< 0.01_

|# Training examples|CommonGen (in-house) VIST ROC|
|---|---|


|50 500 5000 All|< 0.01 < 0.01 < 0.01 < 0.01 < 0.01 < 0.01 < 0.01 NA < 0.01 NA < 0.05 < 0.01|
|---|---|



We conduct statistical significance analysis on the experiments involving baselines, which include
the ablation study on the backend LM used by the verbalization module (Figure 4) and the lowresource experiment (Figure 5). The p-values are shown in Table 8, where < 0.01 indicates a
significant improvement and < 0.05 indicates a fairly significant improvement, and NA indicates
that our method does not outperform the best baseline.

A.4 QUALITATIVE ANALYSIS ON HOW IMAGINATION HELPS

We show how imagination can help generating sentences that follow common sense via qualitative
analysis in Table 9-10. As comparison, we also show the results from the Node2Text baseline which
does not imagines. We organize the results based on 5 (not necessarily exclusive) types of errors
made by Node2Text, which include incorrect role attribution to 1) agents, 2) actions or 3) objects,
4) failing to infer the implicit concepts and 5) misunderstanding the relations between events.

A.5 QUALITY EVALUATION ON THE GENERATED SKGS

Since there are no grountruth SKGs annotated in downstream datasets, we use the silver-standard
SKGs as reference to give a rough estimation of the quality of the generated SKGs. We focus on re

-----

Table 9: Qualitative analysis on errors made without imagination and how imagination can help fix
the errors (Part 1). The left arrow ←− indicates the key relations that fix the errors.

|Error 1 (Incorrect Agent)|Example 1|Example 2|
|---|---|---|

|Input concepts|{owner, chase, dog, ball, throw}|{ski, rope, hold, boat, pull}|
|---|---|---|

|Text w/o imagination|The dog is chasing the ball and throwing it at the owner.|A woman skis downhill as she pulls a boat holding a rope.|
|---|---|---|



|Text w/ imagination|A dog chases a ball being thrown by its owner.|A boat pulls a skier who is holding a rope.|
|---|---|---|


Generated SKG (chase, ARG0, dog), (pull, ARG0, boat), ←−
(chase, ARG1, ball), (pull, ARG1, person),
(throw, ARG1, ball), (ski, ARG0, person),
(throw, ARG0, owner) ←− (hold, ARG0, person),

(hold, ARG1, rope)


|Error 2 (Incorrect Action)|Example 1|Example 2|
|---|---|---|

|Input concepts|{butter, pot, crack, egg, add}|{stand, tongue, stick}|
|---|---|---|

|Text w/o imagination|She adds eggs, crackers, and butter to a pot.|A boy stands next to a stick of his tongue.|
|---|---|---|



|Text w/ imagination|You crack an egg and add butter to a pot.|A man stands with his tongue stick- ing out.|
|---|---|---|


Generated SKG (crack, ARG0, you), (stand, ARG1, man),
(crack, ARG1, egg), ←− (man, part, tongue),
(add, ARG0, you), (stick, ARG0, man),
(add, ARG1, butter), (stick, ARG1, tongue), ←−
(add, ARG2, pot) (stick, ARG2, out)


|Error 3 (Incorrect Object)|Example 1|Example 2|
|---|---|---|

|Input concepts|{hit, bottle, shoe, open, wall}|{wear, talk, phone}|
|---|---|---|

|Text w/o imagination|Someone opens his shoe and hits a bottle on the wall.|A woman is wearing a cell phone and talking to the camera.|
|---|---|---|



|Text w/ imagination|A man opens a bottle and hits his shoe against a wall.|A man wearing glasses is talking on the phone.|
|---|---|---|

|Generated SKG|(open, ARG0, man), (open, ARG1, bottle), ←− (hit, ARG0, man), (hit, ARG1, shoe), ←− (shoe, poss, man), (hit, ARG2, against), (against, op1, wall)|(talk, ARG0, man), (wear, ARG0, man), (wear, ARG1, glasses), ←− (talk, medium, phone)|
|---|---|---|


call since the silver-standard SKGs may not cover all the plausible scenes. Our evaluation considers
the following three metrics. 1) Average recall of the given concepts (Explicit Concepts) to examine
whether an SKG contains all the given concepts. 2) Average recall of the implicit concepts (Implicit
Concepts) to examine whether an SKG also contains implicit concepts. The implicit concepts for
reference are the nodes from the silver-standard SKGs excluding the given concepts. 3) Average recall of the relations (Relation) to examine the proportion of the referenced relations that are covered
by the generated SKGs. Here, a relation is considered as correct only if the head concept, relation
and the tail concept all match the reference.

The results shown in Table 11 indicate a fairly good quality of the generated SKGs, which connect
all the given concepts for over 99% of the cases and have a large overlap (over 68%) with the silverstandard SKGs. Note that the particular low recall on implicit concepts is due to the fact that there
can be many different implicit concepts to constitute a complete SKG.


-----

Table 10: Qualitative analysis on errors made without imagination and how imagination can help fix
the errors (Part 2). The left arrow ←− indicates the key relations that fix the errors.

|Error 4 (Implicit Concepts)|Example 1|Example 2|
|---|---|---|

|Input concepts|{fill, liquid, machine, bottle}|{lasso, catch, horse, animal, ride}|
|---|---|---|

|Text w/o imagination|A machine holding a bottle filled with liquid.|Animals ride a horse that caught a lasso.|
|---|---|---|



|Text w/ imagination|A man holds a bottle filled with liq- uid in a machine.|A man riding a horse to catch an an- imal with a lasso.|
|---|---|---|


Generated SKG (hold, ARG0, man), ←− (ride, ARG0, man), ←−
(hold, ARG1, bottle), (ride, ARG1, horse),
(fill, ARG1, bottle, (ride, purpose, catch),
(fill, ARG2, liquid), (catch, ARG0, man),
(hold, location, machine) (catch, ARG1, animal),

(catch, instrument, lasso)


|Error 5 (Event Relations)|Example 1|Example 2|
|---|---|---|

|Input concepts|{trick, perform, begin, stunt, dance}|{stir, pour, pot, ingredient, begin}|
|---|---|---|

|Text w/o imagination|A group of people begin performing a stunt while performing a trick.|She begins stirring the ingredients in the pot and begins pouring them into the water.|
|---|---|---|



|Text w/ imagination|A man performs stunts and tricks as he begins to dance.|He pours the ingredients into the pot and begins to stir them.|
|---|---|---|

|Generated SKG|(perform, ARG0, man), (perform, ARG1, stunt), (perform, ARG1, trick), (perform, time, begin), ←− (begin, ARG0, man), (begin, ARG1, dance), (dance, ARG0, man)|(pour, ARG0, he, (pour, ARG1, ingredient), (pour, ARG2, pot), (begin, ARG0, he), ←− (begin, ARG1, stir), (stir, ARG0, he), (stir, ARG1, ingredient)|
|---|---|---|


Table 11: Quality evaluation (recall) on the generated SKGs with silver-standard SKGs as reference.

|Dataset|Explicit Concepts Implicit Concepts Relation|
|---|---|


|CommonGen (in-house) VIST ROC|99.81 17.07 72.10 99.96 35.66 68.19 99.95 61.86 74.04|
|---|---|



A.6 ABLATION STUDY ON INPUT TO I&V

For imagination, the inclusion of context helps the module to generate contextualized SKG which
is more relevant to the current storyline. To justify this design choice, we conduct an ablation
study where we learn an uncontextualized imagination module which only takes concepts as input.
The resulting SPICE scores are 47.32 and 45.18 on VIST and ROC datasets respectively, which
are much lower than the results from contextualized I&V (59.21 and 60.63 in SPICE respectively).
This demonstrates that the context is critical in generating relevant SKGs which lead to better text
generation.

For verbalization, the textual context is important for narrative generation in keeping the storyline
consistent. The concept input helps indicate what are the nodes while the SKG input is about the
edges. We conduct an ablation study on what input to include for verbalization. The results in
Table 12 show that adding concepts as input generally helps improve the performance of our method
while adding context is critical for story generation.


-----

Table 12: Ablation study on what input is fed to the verbalization module.

|Input|CommonGen (in-house) VIST ROC|
|---|---|


|SKG-only Concept + SKG Context + SKG Context + Concept + SKG|33.39 17.13 18.90 33.49 27.01 36.42 NA 57.99 58.06 NA 59.21 60.63|
|---|---|



Table 13: Ablation study on using 1) silver-standard SKGs, 2) generated SKGs or 3) both to train
the verbalization module.

|Input SKGs|CommonGen (in-house) VIST ROC|
|---|---|


|Silver-standard Generated Silver. + Generated|33.19 53.26 60.55 32.56 58.34 59.82 33.49 59.21 60.63|
|---|---|



Table 14: Ablation study on the design of the generation process.

|Generation Process|VIST ROC|
|---|---|


|All-at-once Independent Iterative (I&V)|57.16 54.83 27.01 36.42 59.21 60.63|
|---|---|



A.7 ABLATION STUDY ON WHAT SKGS TO USE WHEN LEARNING VERBALIZATION

We conduct an ablation study where we use 1) silver-standard SKGs only, 2) generated SKGs only
and 3) both types of SKGs during training the verbalization module. The results in Table 13 validates
that using both types of SKGs yield to the best performance of our method.

A.8 ABLATION STUDY ON CONCEPT-DROPOUT

We conduct an ablation study where we do not drop any concepts when we train the imagination
module. We then apply the imagination module on CommonGen and conduct the experiments. The
final performance is 28.28 in SPICE while the system with the imagination module trained with
concept-dropout achieves 33.49. This validates that dropping concepts is necessary since in the
downstream tasks not all the concepts are provided and the model needs to infer the implicit ones.

A.9 ABLATION STUDY ON THE GENERATION PROCESS

We conduct an ablation study to verify that the iterative generation process is more effective than 1)
generating all the sentences at once, and 2) generating each sentence independently. For baseline 1),
we learn an uncontextualized imagination module which only takes concepts as input and does not
need the previous generated context. We apply the uncontextualized imagination module to generate
all the SKGs at once and then the verbalization module generates all the target sentences at once by
taking the provided context, all the concept sets and all the SKGs as input. For baseline 2), we still
use the contextualized imagination module to generate an SKG at a time. But we learn a verbalization module which does not take the previously generated sentences as input and thus generate
each target sentence independently. We conduct the ablation study on the two datasets of the concept2story task (we do not consider the CommonGen benchmark here since there is only one target
sentence to be generated in CommonGen). The results of the average SPICE scores from 3 runs
are shown in Table 14. The two baselines are both outperformed by our iterative approach, which
verifies that the previously generated sentences are important for both imagination and verbalization
and thus the iterative generation process is necessary.


-----

Table 15: Evaluation results (SPICE) with I&V using silver-standard SKGs during inference.

|SKGs (inference)|CommonGen (in-house) VIST ROC|
|---|---|


|Silver-standard Generated|41.85 69.34 67.70 33.49 59.21 60.63|
|---|---|



A.10 EXPERIMENTS WITH SILVER-STANDARD SKGS DURING INFERENCE

We report the “oracle” performance of our system using silver-standard SKGs during inference to
estimate the upper-bound of our method. The result is shown in Table 15.


-----

