# VALUE FUNCTION SPACES: SKILL-CENTRIC STATE
 ABSTRACTIONS FOR LONG-HORIZON REASONING

**Dhruv Shah[γβ], Peng Xu[γ], Yao Lu[γ], Ted Xiao[γ]**

**Alexander Toshev[γ], Sergey Levine[γβ], Brian Ichter[γ]**

_γGoogle Research, Robotics @ Google_
_βBerkeley AI Research, UC Berkeley_

ABSTRACT

Reinforcement learning can train policies that effectively perform complex tasks.
However for long-horizon tasks, the performance of these methods degrades with
horizon, often necessitating reasoning over and chaining lower-level skills. Hierarchical reinforcement learning aims to enable this by providing a bank of lowlevel skills as action abstractions. Hierarchies can further improve on this by abstracting the space states as well. We posit that a suitable state abstraction should
depend on the capabilities of the available lower-level policies. We propose Value
Function Spaces: a simple approach that produces such a representation by using
the value functions corresponding to each lower-level skill. These value functions
capture the affordances of the scene, thus forming a representation that compactly
abstracts task relevant information and robustly ignores distractors. Empirical
evaluations for maze-solving and robotic manipulation tasks demonstrate that our
approach improves long-horizon performance and enables better zero-shot generalization than alternative model-free and model-based methods.

1 INTRODUCTION

For an agent to perform complex tasks in realistic environments, it must be able to effectively reason
over long horizons, and parse high-dimensional observations to infer the contents of a scene and
its affordances. It can do so by constructing a compact representation that is robust to distractors
and suitable for planning and control. Consider, for instance, a robot rearranging objects on a desk.
To succcessfully solve the task, the robot must learn to sequence a series of simple skills, such as
picking and placing objects and opening drawers, and interpret its observations to determine which
skills are most appropriate. This requires the ability to understand the capabilities of these simpler
skills, as well as the ability to plan to execute them in the correct order.

Hierarchical reinforcement learning (HRL) aims to enable this by leveraging abstraction, which
simplifies the higher-level control or planning problem. Typically, this is taken to mean abstraction
of actions in the form of primitive skills (e.g., options (Sutton et al., 1999)). However, significantly
simplifying the problem for the higher level requires abstraction of both states and actions. This is
particularly important with rich sensory observations, where standard options frameworks provide a
greatly abstracted state space, but do not simplify the perception or estimation problem. The nature
of the ideal state abstraction in HRL is closely tied to the action abstraction, as the most suitable abstraction of state should depend on the kinds of decisions that the higher-level policy needs to make,
which in turn depends on the actions (skills) available to it. This presents a challenge in designing
HRL methods, because it is difficult to devise a state abstraction that is both highly abstracted (and
therefore removes many distractors) and still sufficient to make decisions for long-horizon tasks.
This challenge differs markedly from representation learning problems in other domains, like computer vision and unsupervised learning (Schroff et al., 2015; van den Oord et al., 2018; Chen et al.,
2020), since it is intimately tied to the capabilities exposed to the agent via its skills.

We therefore posit that a suitable representation for a higher-level policy in HRL should depend on
the capabilities of the skills available to it. If this representation is sufficient to determine the abilities
and outcomes of these skills, then the high-level policy can select the skill most likely to perform
the desired task. This concept is closely tied to the notion of affordances, which has long been
studied in cognitive science and psychology as an action-centric representation of state (Gibson,


-----

^ĨÕĉѕ"ī±ŏÕī wñËĀѕăĻÕ wă±ËÕѕăĻÕѕñĉѕ"ī±ŏÕī ăďįÕѕ"ī±ŏÕī

ĶΩ ĶΪ ĶΫ Ķά Ķέ


**(a) A trajectory through the skill value function space for the task “Place blue block in drawer”. VFS, visualized**
on top of corresponding scene, captures positional information about the contents of the scene, preconditions
for interactions, and the effects of executing a feasible skill, making it suitable for high-level planning.

**(b) VFS ignores task-irrelevant factors like arm pose, color of the desk, or distractors. All configurations shown**
above are functionally equivalent and hence, map to the same VFS representation.

**Figure 1: Visualizing VFS embeddings in an example desk rearrangement task. VFS can capture the affor-**
dances of the low-level skills while ignoring exogenous distractors.

1977), and has inspired techniques in robotics and RL (Zech et al., 2017; Xu et al., 2021; Mandikal
& Grauman, 2021). Given a set of skills that span the possible interactions in an environment, we
propose that the value functions corresponding to these policies can provide a representation that
suitably captures the capabilities of the skills in the current state and thus can be used to form a
compact embedding space for high-level planning. We call this the Value Function Space (VFS).
Note that we intend to use these skills as high-level actions without modifications—the important
problems of skill discovery and test-time adaptation are beyond the scope of this paper.

Figure 1a illustrates the state abstraction constructed by VFS for the desk rearrangement example
discussed above: VFS captures the affordances of the skills and represents the state of the environment, along with preconditions for the low-level skills, forming a functional representation to
plan over. Since VFS constructs a skill-centric representation of states using the value functions
of the low-level skills, it captures functional equivalence of states in terms of their affordances: in
Figure 1b, states with varying object or arm positions, background textures, and distractor objects
are functionally equivalent for planning, and map to the same VFS embedding. This simplifies the
high-level planning problem, allowing the higher level policy to generalize to novel environments.

_Statement of Contributions. The primary contribution of this work is VFS, a novel state representa-_
tion derived from the value functions of low-level skills available to the agent. We show that VFS
leverages the properties of value functions, notably their ability to model possible skills and completed skills, to form an effective representation for high-level planning, and is compatible with both
model-free and model-based high-level policies. Empirical evaluations in maze-solving and robotic
manipulation demonstrate that the skill-centric representation constructed by VFS outperforms representations learned using contrastive and information-theoretic objectives in long-horizon tasks.
We also show that VFS can generalize to novel environments in a zero-shot manner.

2 RELATED WORK

Hierarchical RL has been studied extensively in the literature, commonly interpreted as a temporal
_abstraction of the original MDP. Early works have interpreted the hierarchy introduced in this setting_
as an abstraction of state and action spaces (Sutton et al., 1999; Dietterich, 2000; Thomas & Barto,
2012). The popular options framework Sutton et al. (1999); Precup (2000) provides a natural way of
incorporating temporally extended actions into RL systems. An agent that possesses the transition
model and reward model for such a SMDP (known as an option model) is capable of sample-based
planning in discrete (Kocsis & Szepesv´ari, 2006) and continuous state spaces (Konidaris et al., 2014;

Gopalan et al., 2017). However, doing so in environments with high-dimensional observations (such
as images) is challenging. In this work, we explore the efficacy of learned skills operating on highdimensional observations for long-horizon control in realistic environments.


-----

To improve the quality of lower-level policies, recent work in HRL has studied various facets of
the problem, including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa
et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016;
Tessler et al., 2017) and integrating goal-conditioned behaviors (Ghosh et al., 2019; Nachum et al.,
2019). In this work, we assume that the low-level skills are given, and do not focus on discovering
them. Instead, we focus on how skills can simplify the higher-level control problem by providing a
representation that is suitable for either model-based or model-free control.

The problem of learning meaningful long-horizon behavior by combining a set of lower-level skills,
or options, has been studied extensively in prior work. Popular approaches to composing skills
involve combining them in the space of value functions (Todorov, 2009; Hunt et al., 2019; da Silva
et al., 2009; Ziebart, 2010; Haarnoja et al., 2018), and more recently, using pseudo-rewards (Barreto
et al., 2019). While these methods have been very effective at composing low-level skills, they do not
address the question of learning a meaningful state abstraction using these skills. We take inspiration
from these works and use the value function space to construct a skill-centric state abstraction, along
with a high-level policy that composes these skills for temporally extended tasks.

Our method can hence be interpreted as a representation learning approach. Representation learning
techniques have been employed extensively in model-free RL by augmenting auxiliary tasks based
on reconstruction losses (Lange et al., 2012; Higgins et al., 2017; Yarats et al., 2019) and predicting
the future conditioned on past observations (Schmidhuber, 1990; Jaderberg et al., 2017; van den
Oord et al., 2018; Shelhamer et al., 2016). Contrastive learning has also been used in recent works
to discover a meaningful latent space and extract reward signals for RL (Sermanet et al., 2017;
Warde-Farley et al., 2018; Dwibedi et al., 2018; Laskin et al., 2020). Another popular approach to
learning useful representation is to optimize for effective transfer of the lower-level skills and the
learned state abstraction (Gupta et al., 2017; Konidaris & Barto; Goyal et al., 2019). Unlike these
prior works, our aim is specifically to learn a representation that is grounded in the capabilities of the
low-level skills, which gives us a skill-centric abstraction of high-dimensional observations. While
we do not explicitly optimize for transfer like Gupta et al. (2017), our experiments show that VFS
can generalize to novel environments while being robust to functionally-irrelevant distractors.

Alongside developments in model-free RL, prior work has also sought to learn predictive models
of the environment for sampling and planning. This has been demonstrated by learning dynamics
using future predictions (Watter et al., 2015; Oh et al., 2017; Ebert et al., 2017; Banijamali et al.,
2018; Ha & Schmidhuber; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang
et al., 2019), learning belief representations (Gregor et al., 2019; Lee et al., 2019) and representing
state similarity using the bisimulation metric (Castro, 2020; Zhang et al., 2021a; Agarwal et al.,
2021). The combination of learned model-free policies with structures like graphs (Savinov et al.,
2018; Eysenbach et al., 2019) and trees (Ichter et al., 2021) to plan over extended horizons has also
been demonstrated to improve generalization and exploration. While our primary objective is not
to develop better model-based RL algorithms, we show that our proposed state abstraction can be
utilized by model-based controllers to plan over temporally extended skills.

3 PRELIMINARIES

We assume that an agent has access to a finite set of temporally extended options O, or skills, which
it can sequence to solve long-horizon tasks. These skills can be trained for a wide range of tasks by
using manual reward specification (Huber & Grupen, 1998; Stulp & Schaal, 2011), using relabeling
techniques (Andrychowicz et al., 2017), or via unsupervised skill discovery (Daniel et al., 2016; Fox
et al., 2017; Warde-Farley et al., 2018; Sharma et al., 2020). Since many prior works have focused
on skill discovery, we do not explicitly address how these skills are produced.

We assume that each skill has a maximum rollout length of ⌧ time steps, after which it is terminated.
We also assume that each skillexpected cumulative skill reward executing skill oi 2 O is accompanied by a critic, or value function, oi from current state st. This is generally available Voi denoting the
for policies trained with RL (Huber & Grupen, 1998; Andrychowicz et al., 2017; Kalashnikov et al.,
2021) or learned via options discovery (Bacon et al., 2017; Klissarov et al., 2017; Tiwari & Thomas,
2019; Riemer et al., 2018b). While this may not be true for skills discovered by other methods, such
as segmentation, graph partioning or bottleneck discovery (McGovern & Barto, 2001; Menache
et al., 2002; Krishnamurthy et al., 2016; S¸ims¸ek & Barto, 2004; S¸ims¸ek et al., 2005; Riemer et al.,


-----

2018a), we can use policy evaluation to automatically obtain a value function—this can be done via
regression to empirical returns or temporal difference learning (Dann et al., 2014).

Our setting is closely related to the options framework of Sutton et al. (1999). Options are skills that
consist of three components: a policy ⇡ : S ⇥A ! [0, 1], a termination condition β : S [+] _! [0, 1],_
and an initiation set I ✓S, where S, A are the low-level state and action spaces in a fully observable
decision process. An optionwe have an initiation set, and show that the value functions provide this information. We assume hI, ⇡, βi is available at state st if and only if st 2 I. We do not assume
that the policies come with a termination condition, or alternatively, have a fixed horizon ⌧ . Next,
we describe a framework to formulate a decision problem that uses these options for planning.

We assume that the low-level observation and action space of the agent can be described as a fully
observable semi-Markov decision process (SMDP) M described by a tuple (S, O, R, P, ⌧, γ), where
_S ✓_ R[n] is the n-dimensional continuous state space; O is a finite set of temporally extended skills,
or options, with a maximum horizon of ⌧ time steps; R(s[0] _s, oi) is the task reward received after_
_|_
executing the skill oi _O at state s_ _S; P_ (s[0] _s, oi) is a PDF describing the probability of arriving_
in state s[0] _S after executing skill 2_ _o 2i_ _O at state|_ _s_ _S; γ_ (0, 1] is the discount factor. Given a
finite set of options, our objective is to obtain a high-level decision-policy that selects2 _2_ _2_ _2_ _among them._

Note that generally, many image-based problems are partially observed so that the entire history of
observation-action pairs may be required to describe the state. Explicitly addressing partial observability is outside the scope of our work, so we follow the convention in prior work by assuming full
observability and refer to images as states (Lillicrap et al., 2016; Nair et al., 2018).

4 SKILL VALUE FUNCTIONS AS STATE SPACES

The notion of value in RL is closely related to affordances, in that the value function predicts the capabilities of the skill being learned. The supervised learning problem of affordance prediction (Ugur
et al., 2009) can in fact be cast as a special case of value prediction (Sutton & Barto, 2018; Graves
et al., 2020). In this section, we construct a skill-centric representation of state derived from skill
value functions that captures the affordances of the low-level skills, and empirically show that this
representation is effective for high-level planning.

Given an SMDP (S, O, R, P, ⌧, γ) with a finite set of k skills oi _O trained with sparse out-_
come rewards and their corresponding value functions M _Voi_, we construct an embedding space 2 _Z by_
stacking these skill value functions. This gives us an abstract representation that maps a state st
to aFunction Space, or VFS for short. This representation captures k-dimensional representation Z(st) := [Vo1 (st), Vo2 (st), ..., V functional informationok (st)], which we call the Value about the
exhaustive set of interactions that the agent can have with the environment by means of executing
the skills, and is thus a suitable state abstraction for downstream tasks.

Figure 1(a) illustrates the proposed state abstraction for a conceptual desk rearrangement task with
eight low-level skills. A raw observation, such as an image of the robotic arm and desk, is abstracted
by a 8-dimensional tuple of skill value functions. This representation captures positional information
about the scene (e.g., both blocks are on the counter and drawer is closed since the corresponding
values are 1 at t0), preconditions for interactions (e.g., both blocks can be lifted since the “Pick”
values are high at t0), and the effects of executing a feasible skill (e.g., the value corresponding to
the “Open Drawer” skill increases on executing it at t1), making it suitable for high-level planning.

We hypothesize that since VFS learns a skill-centric representation of the scene, it is robust to
exogenous factors of variation, such as background distractors and appearance of task-irrelevant
components of the scene. This also enables VFS to generalize to novel environments with the same
set of low-level skill, which we demonstrate empirically. Figure 1(b) illustrates this for the desk
rearrangement task. All configurations shown are functionally equivalent and can be interacted with
identically. Intuitively, VFS would map these configurations to the same abstract state by ignoring
factors like arm pose, color of the desk, or additional objects, which do not affect the low-level skills.

5 MODEL-FREE RL WITH VALUE FUNCTION SPACES

In this section, we instantiate a hierarchical model-free RL algorithm that uses VFS as the state
abstraction and the skills as the low-level actions. We compare the long-horizon performance of VFS
to alternate representations for HRL trained with constrastive and information theoretic objectives


-----

for the task of maze-solving and find that VFS outperforms the next best baseline by up to 54% on
the most challenging tasks. Lastly, we compare the zero-shot generalization performance of these
representations and empirically demonstrate that the skill-centric representation constructed by our
method can successfully generalize to novel environments with the same set of low-level skills.

5.1 AN ALGORITHM FOR HIERARCHICAL RL

We instantiate a hierarchical RL algorithm that learns a Q-function Q(Z, o) at the high-level using
VFS as the “state” Z and the skills oi _O as the temporally extended actions. Given such a Q-_
function, a greedy high-level policy can be obtained by 2 _⇡Q(Z) = arg maxoi_ _O Q(Z, oi). We use_
_2_
DQN (Mnih et al., 2015), which uses mini-batches sampled from a replay buffer of transition tuples
(Zt, ot, rt, Zt+1) to train the learned Q-function to satisfy the Bellman equation. This is done using
gradient descent on the loss L = E (Q(Zt, ot) − _yt)[2], where yt = rt + γ maxot0_ _2O Q(Zt+1, ot0_ ).

In order to make the optimization problem more stable, the targets yt are computed using a separate
target network which is update at a slower pace than the main network. Q-learning also suffers
from an overestimation bias, due to the maximization step above, and harms learning. Hasselt et al.
(2016) address this overestimation by decoupling the selection of action from its evaluation. We use
this variant, called DDQN, for subsequent evaluations in this work. Note that we are using the skills
as given, without updating or finetuning them.

While there have been several improvements to DDQN (Hessel et al., 2018) and alternative algorithms for model-free RL in discrete action spaces (Schaul et al., 2015; Bellemare et al., 2017;
Christodoulou, 2019), and these improvements will certainly improve the overall performance of
our algorithm, our goal is to evaluate the efficacy of VFS as a state representation and we study it in
the context of a simple DDQN pipeline.

5.2 EVALUATING LONG-HORIZON PERFORMANCE IN MAZE-SOLVING

To evaluate the long-horizon performance of VFS against commonly used representation learning
methods, we use the versatile MiniGrid environment (Chevalier-Boisvert et al., 2018) in a fully
observable setting, where the agent receives a top-down view of the environment. We consider
two tasks: (i) MultiRoom, where the agent is tasked with reaching the goal by solving a variablesized maze spanning up to 10 rooms. The agent must cross each room and open the door to access the following rooms; (ii) KeyCorridor, where the agent is tasked with reaching a goal up to
7 rooms away and may face locked doors. The agent must find the key corresponding to a colorcoded door and open it to access subsequent rooms. Both these tasks have a sparse reward that
is only provided for successfully reaching the goal. This presents a challenging domain for longhorizon sequential reasoning, where tasks may require over 200 time steps to succeed, making a
great testbed for evaluating the ability of the state abstractions to capture relevant information for
sequencing multiple skills. The agents have access to the following temporally extended skills—
GoToObject, PickupObject, DropObject and UnlockDoor—where the first three skills
are text-conditioned and Object may refer to a door, key, box or circle of any color. Since MiniGrid is easily reconfigurable, we also generate a set of holdout MultiRoom mazes with different
grid layouts to evaluate the zero-shot generalization performance of the policies trained with these
representations. Note that the grid layouts, as well as object positions, for these tasks are randomly
generated for every experiment and are not static. Example grid layouts are shown in Figure 2. We
provide further implementation details in Appendix A.1.

**Baselines:** We compare VFS extensively against a variety of competitive baselines for representa
tion learning in RL using contrastive and information-theoretic objectives. We consider representations learned both offline and online (in loop with RL); note that VFS is constructed entirely from
the values of the available skills and is not learned. Further, all baselines have access to these skills.

1. Raw Observations: We train a high-level policy operating on raw input observations.
2. Autoencoder (AE): We use an autoencoder to extract a compact latent space using a recon
struction loss on an offline dataset of trajectories, similar to Lange et al. (2012).

3. Contrastive Predicting Coding (CPC): We learn a representation by optimizing the In
foNCE loss over an offline dataset of trajectories (van den Oord et al., 2018).


-----

**Representation** **MultiRoom** **KeyCorridor**

**2** **4** **6** **10** **3** **7**


Raw Observations 0.64 0.46 0.42 0.29 0.47 0.32

AE (Lange et al., 2012) 0.70 0.64 0.51 0.34 0.59 0.33

CPC (van den Oord et al., 2018) 0.77 0.69 0.55 0.37 0.63 0.35

VAE[†] (Yarats et al., 2019) 0.79 0.74 0.58 0.49 0.79 0.50

CURL[†] (Laskin et al., 2020) 0.82 0.76 0.63 0.43 **0.83** 0.54

VFS (Ours) **0.98** **0.92** **0.83** **0.77** 0.82 **0.68**

**Table 1: Success rates of different representations for model-free RL across varying levels of difficulty and**
time horizons (second row denotes complexity in terms of number of rooms). VFS explicitly captures the
capabilities of the low-level skills, and outperforms all baselines. Online methods (denoted by _[†]) learned jointly_
with the RL objective outperform their offline counterparts (AE and CPC), but their performance degrades for
longer-horizon tasks. Refer to Appendix B for additional performance metrics and ablations.

**Figure 2: Successful rollouts of HRL using VFS to solve long-horizon tasks in MultiRoom-6 (top) and**
_KeyCorridor-7 (bottom) by sequentially executing multiple low-level skills (labeled under the arrows)._


4. Online Variational Autoencoder (VAE): We learn a VAE representation jointly (online) with

the high-level policy operating on this representation (Yarats et al., 2019).

5. Contrastive Unsupervised Representations for RL (CURL): We learn a representation by

optimizing a contrastive loss jointly (online) with the RL objective (Laskin et al., 2020).

**Evaluation:** We run experiments for the two tasks with varying number of rooms, to study the

performance of the algorithms with increasing time horizon, and report the success rates in Table 1.
HRL from raw observations demonstrates a success rate of 64% in the two-room environment, which
can be attributed to the powerful set of skills available to the high-level policy, but this quickly drops
to 29% in the largest environment. The offline baselines (AE and CPC) construct a compact state
abstraction, which makes the learning problem easier for DDQN, and show significant improvements in smaller environments (MultiRoom-2,4 and KeyCorridor-3). However, they are unable to
improve the performance in larger environments. We hypothesize that this is due to the inability of
the representations to capture information necessary for the high-level policy, since they are learned
independent of the controller. The performance of representations learned online (VAE and CURL),
which are implemented analogous to their offline counterparts (AE and CPC, respectively) support
this hypothesis and improves the performance across all tasks by learning representations jointly
with the controller, and scoring up to 54% in the most challenging environment. We hypothesize
that the limited performance of these methods is due to the lack of direct influence of the downstream
task on the representation. Despite being constructed offline, VFS explicitly captures the capabilities
of the low-level skills and provides an action-centric representation for the high-level policy, greatly
simplifying the control problem. This is reflected in the performance of VFS across all tasks—it
outperforms all baselines, scoring 98% on the simplest task and up to 68% on the most challenging
task, beating the next best method by over 25%. Figure 2 shows sample rollouts of the high-level
policy using VFS as the state representation in the two tasks discussed. It is important to note that
while VFS outperforms the baselines in these tasks, our method is largely orthogonal to these representation learning algorithms and can be combined constructively to improve task performance—we
show this, along with further ablations in Appendix B.


-----

**Representation** **MR4** **MR10**

Raw 0.15 0.03

AE 0.41 0.23

CPC 0.47 0.20

VAE[†] 0.25 0.03

CURL[†] 0.27 0.07

VFS (Ours) **0.87** **0.67**

HRL-Target 0.92 0.77

**Table 2: Success rates for zero-shot**
generalization. VFS can generalize

zero-shot to novel environments.


D E


**Figure 3: We study the zero-shot generalization by using policies**
trained in KeyCorridor (a) and deploying in MultiRoom (b).

5.3 ZERO-SHOT GENERALIZATION


We evaluate the generalization abilities of these representations by training them for the KeyCorridor
task (as above) and evaluating on MultiRoom. Since the skills available in MultiRoom are a subset of
those in KeyCorridor, we ignore any invalid actions executed by the agent. Note that the high-level
agent has not been trained in MultiRoom and the policies are not updated in the target environments.

Table 2 shows the success rates of the representations in the MultiRoom tasks with 4 (MR4) and 10
rooms (MR10). Unsurprisingly, HRL with raw observations fails to generalize, because the maze
layout differs significantly (e.g. see Figure 3). AE and CPC learn a compact representation from
high-dimensional observations and allow the high-level policy to generalize to simpler tasks like
_MR4 and achieve up to 47% success rate. Interestingly, their online counterparts (VAE and CURL)_
also fail to generalize and perform poorly, likely because representations learned jointly with the
RL policy can overfit to the source environment. VFS learns a skill-centric representation that can
generalize zero-shot to tasks that use the same skills, and thus outperforms the next best baseline by
up to 180%, closely matching the performance of an HRL policy trained from scratch in the target
environment with online interaction (HRL-Target).

6 MODEL-BASED PLANNING WITH VALUE FUNCTION SPACES

In this section, we introduce a simple model-based RL algorithm that uses VFS as the “state” for
planning, which we term VFS-MB. We study the performance of VFS in the context of a robotic
manipulation task, and compare it to alternate representations for model-based RL. We find that the
performance of VFS as an abstract representation for raw image observations outperforms all baselines and closely matches that of a pipeline with access to oracular state of the simulator, showing
the efficacy of our method in long-horizon planning.

6.1 A SIMPLE ALGORITHM FOR MODEL-BASED RL

We use a simple model-based planner that learns a one-step predictive model, using VFS as the
“state.” Specifically, this model learns the transition dynamicssupervised learning using a dataset of prior interactions in the environment. Note that the predictive Zt+1 = f[ˆ](Zt, ot) 8 ot 2 O via

model (and the subscript of Z) is over high-level policy steps, which may be up to ⌧ steps of the lowlevel skills. This dataset can be collected simply by executing the available skills in the environment
for ⌧ steps, where ⌧ is the maximum horizon of the SMDP M, or until termination.

In order to use the learned model _f[ˆ](Zt, ot), a goal latent state Zg, and a scoring function ✏_ (e.g.

mean squared error) for the high-level task, we need to solve the following optimization problem for
the optimal sequence of skills (ot, . . ., ot+H−1) to reach the goal:


_✏( Z[ˆ]t+H_ _, Zg) : Z[ˆ]t = Zt,_ _Z[ˆ]t0+1 = f[ˆ]( Z[ˆ]t0_ _, ot0_ ) (1)


(ot, . . ., ot+H−1) = _otarg min,...,ot+H−1_


We use a sampling-based method to find solutions to the above equation. We use random shooting
(Rao, 2009) to randomly generate K candidate option sequences, predict the corresponding Z sequences using the learned model _f[ˆ], compute the rewards for all sequences, and pick the candidate_

action sequence leading to a latent state closest to the goal, according to Equation 1. We execute the


-----

policy using model-predictive control: the policy executes only the first skill ot, receives the updated
_Zt+1 from the environment, and recalculates the optimal action sequence iteratively. Figure 4 shows_
an overview of the algorithm. We provide further implementation details in Appendix A.2

Note that our goal is to study the behavior of VFS as a state representation in existing RL pipelines,
rather than developing a better model-based algorithm. Using sophisticated methods that adjust the
sampling distribution, as in the cross-entropy method (Finn & Levine, 2017; Hafner et al., 2018;
Chua et al., 2018) or path integral optimal control (Williams et al., 2015; Lowrey et al., 2019), as
well as more sophisticated models and planning or control methods would likely improve overall
performance further, but is outside the scope of this work.

ᬄүв᫾ĶЕѕᶢүӫүᶈгѕ҇ѕ᫾ĶӟΪ ᬄүв᫾ĶӟΪЕѕᶢүӫүᶈгѕ҇ѕ᫾ĶӟΫ « ᫾[Ω]Ķӟ> рwă±ËÕѕÊďĶîѕñĉѕÑī±ŏÕīсѕ᫾é

ᬄүвѕѕѕѕѕѕѕѕѕЕѕᶢăďįÕѕ"ī±ŏÕīгѕ҇ѕ ᬄүвѕѕѕѕѕѕѕѕѕЕѕᶢwñËĀѕăĻÕгѕ҇ѕ « ᫾[Ϊ]Ķӟ>

«

_RW_ ᫾W _RW ^9ʌL_ _RW`_ ᬄүвѕѕѕѕѕѕѕѕѕЕѕᶢwñËĀѕăĻÕгѕ҇ѕ ᬄүвѕѕѕѕѕѕѕѕѕЕѕᶢwă±ËÕѕДДДгѕ҇ѕ « ᫾[X]Ķӟ> DUJPLQL䌜1^__᫾[L]W7[][᫾]J[__`]


**Figure 4: Overview of model-based planning with VFS. We learn a one-step predictive model for the embed-**
ding Zt and use random shooting with a scoring function ✏ to find the best action to the encoded goal Zg.

6.2 APPLICATION: ROBOTIC MANIPULATION

We evaluate the performance of VFS-MB in a complex image-based task using a simulated manipulation environment with an 8-DoF robotic arm, similar to the setup used by Jang et al. (2021).
The robot only has access to high-dimensional egocentric visual observations. We consider the
task of semantic rearrangement, which requires rearranging objects into semantically meaningful
positions—e.g., grouped into (multiple) clusters, or moved to a corner (Figure 5). Efficiently solving this task requires the robot to plan over a series of manipulation maneuvers, interacting with up
to 10 distinct objects. This requires long-horizon planning, and presents a major challenge for RL
methods. Each of the methods we compare have access to a library of MoveANearB skills, trained
as a multi-task policy with MT-Opt (Kalashnikov et al., 2021). There are 10 objects, and 9 possible
destinations near which they can be moved, resulting in a total of 90 skills, which then comprise the
action space for planning. The skills control the robot at a frequency of 0.5 Hz, taking an average of
14 time steps to execute with a success rate of 94%. We provide further details about these skills in
Appendix A.2.

We consider two versions of the task, which arrange either 5 or 10 objects to semantic positions. The
_O5 environment uses a random subset of the 10 objects in every experiment; lesser objects allow_
a smaller planning horizon, making planning problem simpler than in the O10 environment, which
has all 10 objects. We randomize the object positions on the table and command the algorithms to
reach the same goal with a planning horizon H = 7 steps for the smaller environment and H = 15
steps for the larger environment, reporting the task success rate averaged over 20 experimental runs.

**Baselines:** To evaluate the efficacy of the proposed representation for high-level model-based

planning, we compare it against four alternative representations used in conjunction with the algorithm described in Section 6.1. All methods have access to the skills and use them as the low-level
action space. Raw Image learns a policy on raw visual observations from the robot’s onboard camera. VAE uses a variational autoencoder to project the observations to a 100−dimensional learned
embedding space, similar to Corneil et al. (2018). We train this VAE offline from trajectories collected by rolling out the models. CPC uses contrastive predictive coding (van den Oord et al., 2018)
to learn a similar representation from offline trajectories. We also compare against an Oracle State
baseline that has access to privileged information—the simulator state—and learns the model on
this. This gives us an upper bound for the performance of the baselines.

**Evaluation:** Table 3 shows the success rates on the two versions of the task for each prior method.

The model trained using image observations fares poorly and fails at all but the simplest starting
configurations. It succeeds in 2/20 experiments in O10, largely due to poor model predictions. The
VAE and CPC representations, which learn a more compact embedding space to plan over, succeed
in up to 65% of the tasks in the simpler O5 environment. However, their performance falls sharply in


-----

**Representation** **O5** **O10**

Raw Image 0.25 0.1

VAE 0.6 0.3

CPC 0.65 0.4

VFS (Ours) **1** **0.8**

Oracle State 1 0.85

**Table 3: Success rates for robotic**
manipulation. VFS outperforms all
baselines and closely matches oracular performance.


ZĲűøĲŲĦ ZĲűøĲǊĦø ZĲűø<ĦÔŒŒ ZĲűøŋŋĦø

[øÔŎŋĲĬČø [øÔŎĲŲĦ [øÔŎŋĲĬČø [øÔŎĲŲĦ


**Figure 5: Example rollout of model-based RL with VFS as the state**
representation for robotic manipulation. The robot plans over multiple
low-level skills to achieve the semantic goal “move all to top-right corner”. Red arrows specify the next skill planned by the model, and is
overlaid for visualization purposes only.


the O10 environment, where the longer horizons read to greater error accumulation. VFS constructs
an effective representation that captures the state of the scene as well as the affordances of the skills,
and succeeds in all tasks in the O5 environment, matching the oracle performance. It also succeeds
in 80% of the tasks in O10, closely matching the oracle’s performance of 85%.

To understand the factors captured by VFS, we sample encoded observations from a large number
of independent trajectories and visualize their 2D t-SNE embeddings (van der Maaten & Hinton,
2008). Figure 6 shows that VFS can successfully capture information about objects in the scene
and affordances (e.g. which object is in the robot’s arm and can be manipulated), while ignoring
distractors like the poses of the objects on the table and the arm.

**Figure 6: A t-SNE diagram of observations encoded by VFS, showing functionally equivalent observations**
mapped to the same representation. VFS discovers clusters with (top) the arm grasping the bowl with apple
and chocolate on the table, (right) bottle in arm with glass and chocolate on table; (left) sponge in arm with
chocolate on the table. Note that these observations occur across independent trajectories and are unlabeled.

7 DISCUSSION

We proposed Value Function Spaces as state abstractions: a novel skill-centric representation that
captures the affordances of the low-level skills. States are encoded into representations that are
_invariant to exogenous factors that do not affect values of these skills. We show that this rep-_
resentation is compatible with both model-free and model-based policies for hierarchical control,
and demonstrate significantly improved performance both in terms of successfully performing longhorizon tasks and in terms of zero-shot generalization to novel environments, which leverages the
invariances that are baked into our representation.

The focus of our work is entirely on utilizing a pre-specified set of skills, and we do not address
how such skills are learned. Improving the low-level skills jointly with the high-level policy could
lead to even better performance on particularly complex tasks, and would be an exciting direction
for future work. More broadly, since our method connects skills directly to state representations, it
could be used to turn unsupervised skill discovery methods directly into unsupervised representation
learning methods, which could be an exciting path toward more general approaches that retain the
invariances and generalization benefits of our method.


-----

ACKNOWLEDGMENTS

This research was partially supported by ARL DCIST CRA W911NF-17-2-0181. The authors would
like to thank Matthew Benice for help with setting up and debugging the robotic manipulation environment, and Ryan Julian for training the skills used in our experiments. The authors would also
like to thank Sumeet Singh, Katie Kang and Kyle Hsu for feedback on an earlier draft of this paper.

REPRODUCIBILITY STATEMENT

The primary contribution of our work, VFS, is a skill-centric representation designed to work with
existing model-based and model-free pipelines. The implementation simply involves concatenating
the value functions corresponding to the available skills into an embedding vector that can be used
for HRL or planning. We provide all necessary information for setting up VFS to work with a
reader’s RL algorithm of choice in Sections 5.1, 6.1 and Appendix A. Our model-free experiments
are conducted on an open-source gym environment and we provide configuration details for setting
up our quantitative experiments in Appendix A.1. We hope that this encourages the community to
utilize and build upon the ideas presented in the paper. We plan to release more information about
the proprietary environments and tasks used in a public release[1] of this article at a later date.

REFERENCES

Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare. Contrastive

behavioral similarity embeddings for generalization in reinforcement learning. In International
_Conference on Learning Representations, 2021. URL https://openreview.net/forum?_
id=qda7-sVg84.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter

Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.

Hindsight experience replay. In _Advances_ _in_ _Neural_ _Information_ _Processing_ _Sys-_

_tems,_ 2017. URL https://proceedings.neurips.cc/paper/2017/file/

453fadbd8a1a3af50a9df4df899537b5-Paper.pdf.

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of

_the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, 2017._

Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Hai Bui, and Ali Ghodsi. Robust

locally-linear controllable embedding. In AISTATS, 2018.

Andre Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Ayg¨un, Philippe Hamel, Daniel

Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard:
Combining skills in reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer,

F. d Alch´e-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys
_tems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips._
cc/paper/2019/file/251c5ffd6b62cc21c446c963c76cf214-Paper.pdf.

Marc G. Bellemare, Will Dabney, and R´emi Munos. A distributional perspective on reinforcement

learning. In Proceedings of the 34th International Conference on Machine Learning, 2017. URL
https://proceedings.mlr.press/v70/bellemare17a.html.

Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov

decision processes. In AAAI, 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework

for contrastive learning of visual representations. CoRR, 2020. URL https://arxiv.org/
abs/2002.05709.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment

for openai gym, 2018. URL https://github.com/maximecb/gym-minigrid.

1cs.berkeley.edu/⇠shah/VFS


-----

Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, 2019. URL http:

//arxiv.org/abs/1910.07207.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement

learning in a handful of trials using probabilistic dynamics models. In Proceedings of the 32nd
_International Conference on Neural Information Processing Systems, 2018._

Dane Corneil, Wulfram Gerstner, and Johanni Brea. Efficient model-based deep reinforcement

learning with variational state tabulation. In Proceedings of the 35th International Confer
_ence on Machine Learning, 2018._ URL https://proceedings.mlr.press/v80/

corneil18a.html.

Ozg¨¨ ur S¸ims¸ek and Andrew G. Barto. Using relative novelty to identify useful temporal abstrac
tions in reinforcement learning. In Proceedings of the Twenty-First International Conference on
_Machine Learning, 2004. URL https://doi.org/10.1145/1015330.1015353._

Ozg¨¨ ur S¸ims¸ek, Alicia P. Wolfe, and Andrew G. Barto. Identifying useful subgoals in reinforcement

learning by local graph partitioning. In Proceedings of the 22nd International Conference on
_Machine Learning, ICML ’05, 2005._ ISBN 1595931805. URL https://doi.org/10.

1145/1102351.1102454.

Marco da Silva, Fr´edo Durand, and Jovan Popovi´c. Linear bellman combination for control

of character animation. ACM Trans. Graph., 2009. URL https://doi.org/10.1145/
1531326.1531388.

Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative en
tropy policy search. Journal of Machine Learning Research, 2016. URL http://jmlr.org/
papers/v17/15-188.html.

Christoph Dann, Gerhard Neumann, and Jan Peters. Policy evaluation with temporal differences:

A survey and comparison. Journal of Machine Learning Research, 15(24):809–883, 2014. URL
http://jmlr.org/papers/v15/dann14a.html.

Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decompo
sition. J. Artif. Int. Res., 2000.

Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, and Pierre Sermanet. Learning actionable rep
resentations from visual observations. In 2018 IEEE/RSJ International Conference on Intelligent
_Robots and Systems (IROS), 2018._

Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with

temporal skip connections. In Annual Conference on Robot Learning, Proceedings of Machine
Learning Research, 2017.

Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer:

Bridging planning and reinforcement learning. In Advances in Neural Information Process
_ing Systems, 2019. URL https://proceedings.neurips.cc/paper/2019/file/_
5c48ff18e0a47baaf81d8b8ea51eec92-Paper.pdf.

Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017.

Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical re
inforcement learning. In International Conference on Learning Representations, 2017. URL
https://openreview.net/forum?id=B1oK8aoxe.

Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options.

_CoRR, 2017. URL http://arxiv.org/abs/1703.08294._

Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with

goal conditioned policies. In ICLR, 2019. URL https://openreview.net/forum?id=
Hye9lnCct7.

James J. Gibson. The theory of affordances. In Perceiving, acting, and knowing: toward an ecologi
_cal psychology. 1977. URL https://hal.archives-ouvertes.fr/hal-00692033._


-----

Nakul Gopalan, Marie desJardins, Michael L. Littman, James MacGlashan, S. Squire, Stefanie

Tellex, John Winder, and Lawson L. S. Wong. Planning with abstract markov decision processes.
In ICAPS, 2017.

Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick,

Sergey Levine, and Yoshua Bengio. Transfer and exploration via the information bottleneck. In
_International Conference on Learning Representations, 2019. URL https://openreview._
net/forum?id=rJg8yhAqKm.

Daniel Graves, Johannes G¨unther, and Jun Luo. Affordance as general value function: A computa
tional model. CoRR, 2020. URL https://arxiv.org/abs/2010.14289.

Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and A¨aron van den

Oord. Shaping Belief States with Generative Environment Models for RL. 2019.

Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant

feature spaces to transfer skills with reinforcement learning. In 5th International Conference on
_Learning Representations, ICLR 2017, 2017. URL https://openreview.net/forum?_
id=Hyq4yhile.

David Ha and J¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances

_in Neural Information Processing Systems._ URL https://papers.nips.cc/paper/

7512-recurrent-world-models-facilitate-policy-evolution.

Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine.

Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE International
_Conference on Robotics and Automation (ICRA), pp. 6244–6251, 2018. doi: 10.1109/ICRA.2018._
8460756.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James

Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551,
2018.

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning

behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL https://openreview.net/forum?id=S1lOTC4tDS.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q
learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney,

Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. _Proceedings of the AAAI Conference on Artificial In-_

_telligence, 32(1), 2018. URL https://ojs.aaai.org/index.php/AAAI/article/_
view/11796.

Irina Higgins, Arka Pal, Andrei A. Rusu, Lo¨ıc Matthey, Christopher P. Burgess, Alexander Pritzel,

Matthew M. Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot
transfer in reinforcement learning. ArXiv, abs/1707.08475, 2017.

Manfred Huber and Roderic A. Grupen. Learning robot control - using control policies as abstract

actions. In In NIPS’98 Workshop: Abstraction and Hierarchy in Reinforcement Learning, 1998.

Jonathan J Hunt, Andre Barreto, Timothy P Lillicrap, and Nicolas Heess. Composing entropic

policies using divergence correction, 2019. URL https://openreview.net/forum?id=
SJ4Z72Rctm.

Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and

_Automation Letters, 2019._

Brian Ichter, Pierre Sermanet, and Corey Lynch. Broadly-exploring, local-policy trees for long
horizon task planning. In Annual Conference on Robot Learning, 2021. URL https:

//openreview.net/forum?id=yhy25u-DrjR.


-----

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David

Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In
_International Conference on Learning Representations. OpenReview.net, 2017. URL https:_
//openreview.net/forum?id=SJ6yPD5xg.

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine,

and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th
_Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?_
id=8kbp23tSGYv.

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski,

Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. CoRR, 2021. URL https://arxiv.org/abs/2104.08212.

Martin Klissarov, Pierre-Luc Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end

for continuous action tasks. CoRR, abs/1712.00004, 2017. URL http://arxiv.org/abs/
1712.00004.

Levente Kocsis and Csaba Szepesv´ari. Bandit based monte-carlo planning. In Machine Learning:

_ECML 2006, 2006._

George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement

learning. In Proceedings of the 20th International Joint Conference on Artificial Intelligence.

George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains

using skill chaining. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (eds.),
_Advances in Neural Information Processing Systems, 2009._

George Konidaris, Leslie Kaelbling, and Tomas Lozano-Perez. Constructing symbolic represen
tations for high-level planning. Proceedings of the AAAI Conference on Artificial Intelligence,
2014. URL https://ojs.aaai.org/index.php/AAAI/article/view/9004.

Ramnandan Krishnamurthy, Aravind S. Lakshminarayanan, Peeyush Kumar, and Balaraman Ravin
dran. Hierarchical reinforcement learning using spatio-temporal abstractions and deep neural
networks. 2016. URL http://arxiv.org/abs/1605.05359.

Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi, and Joshua B. Tenenbaum. Hierar
chical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In
_Advances in Neural Information Processing Systems, 2016._

S. Lange, Martin A. Riedmiller, and Arne Voigtl¨ander. Autonomous reinforcement learning on raw

visual input data in a real world application. The 2012 International Joint Conference on Neural
_Networks (IJCNN), 2012._

Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa
tions for reinforcement learning. In International Conference on Machine Learning, 2020. URL
https://proceedings.mlr.press/v119/laskin20a.html.

Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:

Deep reinforcement learning with a latent variable model. In Advances in Neural Information
_Processing Systems, 2019. URL http://arxiv.org/abs/1907.00953._

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez,

Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2016.

Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan

online, learn offline: Efficient learning and exploration via model-based control. In International
_Conference on Learning Representations, 2019. URL https://openreview.net/forum?_
id=Byey7n05FQ.

Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual

affordances, 2021.


-----

Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning

using diverse density. In Proceedings of the Eighteenth International Conference on Machine
_Learning, ICML ’01, 2001._

Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut - dynamic discovery of sub-goals in rein
forcement learning. In Machine Learning: ECML 2002, 13th European Conference on Machine
_Learning, volume 2430 of LectureNotes in Computer Science, pp. 295–306, 2002._

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle
mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen,
Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning.
_Nature, 2015. URL http://dx.doi.org/10.1038/nature14236._

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim

Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement
learning. In ICML, 2016. URL https://proceedings.mlr.press/v48/mniha16.
html.

Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does

hierarchy (sometimes) work so well in reinforcement learning?, 2019.

Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual

reinforcement learning with imagined goals. In Proceedings of the 32nd International Conference
_on Neural Information Processing Systems, 2018._

Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In I. Guyon, U. V.

Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in
_Neural Information Processing Systems, 2017. URL https://proceedings.neurips._
cc/paper/2017/file/ffbd6cbb019a1413183c8d08f2929307-Paper.pdf.

Doina Precup. Temporal abstraction in reinforcement learning, 2000.

Anil V. Rao. A survey of numerical methods for optimal control. 2009.

Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In Advances in Neu
_ral Information Processing Systems, 2018a. URL https://proceedings.neurips.cc/_
paper/2018/file/cdf28f8b7d14ab02d12a2329d71e4079-Paper.pdf.

Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract op
tions. In _Advances_ _in_ _Neural_ _Information_ _Processing_ _Systems,_ volume 31,

2018b. URL https://proceedings.neurips.cc/paper/2018/file/

cdf28f8b7d14ab02d12a2329d71e4079-Paper.pdf.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo
bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on
_Computer Vision and Pattern Recognition (CVPR), June 2018._

Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory

for navigation. In International Conference on Learning Representations, 2018. URL https:
//openreview.net/forum?id=SygwwGbRW.

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima
tors. In Proceedings of the 32nd International Conference on Machine Learning, 2015. URL
https://proceedings.mlr.press/v37/schaul15.html.

J¨urgen Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent

neural networks for dynamic reinforcement learning and planning in non-stationary environments.
Technical report, 1990.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face

recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition
_(CVPR), 2015. URL http://dx.doi.org/10.1109/CVPR.2015.7298682._


-----

Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self
supervised learning from multi-view observation. In 2017 IEEE Conference on Computer Vision
_and Pattern Recognition Workshops (CVPRW), 2017._

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware

unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL https://openreview.net/forum?id=HJgLZR4KvH.

Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward:

Self-supervision for reinforcement learning. CoRR, 2016. URL http://arxiv.org/abs/
1612.07307.

Freek Stulp and Stefan Schaal. Hierarchical reinforcement learning with movement primitives. In

_2011 11th IEEE-RAS International Conference on Humanoid Robots, 2011._

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2018. URL

http://incompleteideas.net/book/the-book-2nd.html.

Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs:

A framework for temporal abstraction in reinforcement learning. _Artificial Intelli-_

_gence, 1999._ URL https://www.sciencedirect.com/science/article/pii/

S0004370299000521.

Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hi
erarchical approach to lifelong learning in minecraft. In Proceedings of the Thirty-First AAAI
_Conference on Artificial Intelligence, 2017._

Philip S. Thomas and Andrew G. Barto. Motor primitive discovery. In 2012 IEEE International

_Conference on Development and Learning and Epigenetic Robotics (ICDL), 2012._

Saket Tiwari and Philip S. Thomas. Natural option critic. Proceedings of the AAAI Conference on

_Artificial Intelligence, pp. 5175–5182, Jul. 2019. URL https://ojs.aaai.org/index._
php/AAAI/article/view/4452.

Emanuel Todorov. Compositionality of optimal control laws. In Advances in Neural Information

_Processing Systems, 2009. URL https://proceedings.neurips.cc/paper/2009/_
file/3eb71f6293a2a31f3569e10af6552658-Paper.pdf.

Emre Ugur, Erol S¸ahin, and Ethan Oztop. Predicting future object states using learned affordances.

2009. URL https://hdl.handle.net/11511/55960.

A¨aron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic
tive coding. CoRR, 2018. URL http://arxiv.org/abs/1807.03748.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. _Journal_

_of Machine Learning Research, 2008._ URL http://www.jmlr.org/papers/v9/

vandermaaten08a.html.

David Warde-Farley, Tom Van de Wiele, Tejas D. Kulkarni, Catalin Ionescu, Steven Hansen, and

Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. CoRR,
2018. URL http://arxiv.org/abs/1811.11359.

Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to

control: A locally linear latent dynamics model for control from raw images. In Advances in
_Neural Information Processing Systems, 2015._

Grady Williams, Andrew Aldrich, and Evangelos A. Theodorou. Model predictive path integral

control using covariance variable importance sampling. CoRR, 2015. URL http://arxiv.
org/abs/1509.01149.

Danfei Xu, Ajay Mandlekar, Roberto Mart´ın-Mart´ın, Yuke Zhu, Silvio Savarese, and Li Fei-Fei.

Deep affordance foresight: Planning through what can be done in the future, 2021.


-----

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improv
ing sample efficiency in model-free reinforcement learning from images. CoRR, abs/1910.01741,
2019. URL http://arxiv.org/abs/1910.01741.

Philipp Zech, Simon Haller, Safoura Rezapour Lakani, Barry Ridge, Emre Ugur, and Justus Pi
ater. Computational models of affordance in robotics: a taxonomy and systematic classification.
_Adaptive Behavior, 2017. URL https://doi.org/10.1177/1059712317726357._

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning

invariant representations for reinforcement learning without reconstruction. In International Con_ference on Learning Representations, 2021a. URL https://openreview.net/forum?_
id=-2FCwDKRREu.

Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic

options. In International Conference on Learning Representations, 2021b. URL https://
openreview.net/forum?id=r-gPPHEjpmw.

Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine.

SOLAR: Deep structured representations for model-based reinforcement learning. In ICML,

2019. URL https://proceedings.mlr.press/v97/zhang19m.html.

Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal

_Entropy. PhD thesis, 2010._


-----

