# LEARNING DIVERSE OPTIONS VIA INFOMAX TERMINATION CRITIC

**Anonymous authors**
Paper under double-blind review

ABSTRACT

We consider the problem of autonomously learning reusable temporally extended
actions, or options, in reinforcement learning. While options can speed up transfer
learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of
mutual information (MI) based skill learning, we hypothesize that more diverse
options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding
state transitions. We derive a scalable approximation of this MI maximization via
gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our
experiments demonstrate that IMTC significantly improves the diversity of learned
options without extrinsic rewards, combined with intrinsic rewards. Moreover, we
test the reusability of learned options by transferring options into various tasks,
confirming that IMTC helps quick adaptation, especially in complex domains
where an agent needs to manipulate objects.

1 INTRODUCTION

Behavior learning from environmental interaction is a fundamental problem in artificial intelligence
and robotics. Recently, combined with deep neural networks (DNN), reinforcement learning (RL)
has successfully learned complex behaviors guided by human-designed reward functions (Heess
et al., 2017; OpenAI et al., 2019). To reuse trained agents across multiple reward functions, or shortly
_tasks, abstracting a course of action as a higher-level building block (Barto and Mahadevan, 2003;_
Barto et al., 2013) would be useful. A representative formulation of action abstraction in RL is the
options framework (Sutton et al., 1999), where each option consists of a sub policy and its termination
condition. Serving as reusable behavioral blocks, options have the potential of accelerating learning
in new tasks (Brunskill and Li, 2014). For example, if a wheeled robot already had learned to lean
and turn left and right, it can learn to drive on unfamiliar roads faster than learning from scratch.
However, discovering reusable options remains challenging due to the difficulty of defining and
measuring reusability a priori. Although we can measure the reusability of options when we know the
task distribution (e.g., in a Bayesian sense (Solway et al., 2014)), it is difficult to measure it without
prior knowledge. Hence, in such cases, we need a reasonable heuristic assumption about future tasks.

In this paper, we follow a common assumption that diversifying options improves the reusability of
options (e.g., Barto et al. (2004)). That is, we assume that tasks are uniformly distributed over the
state space. Diverse options are expected to work to some extent for any task, thus we argue that this
is a reasonable heuristics. Under this assumption, the next problem is to measure the diversity of
options. The most intuitive strategy would be visiting all states as equally as possible. However, this
can be computationally too expensive when the environment has continuous or large state spaces. As
an alternative, in the context of learning skills without termination conditions, mutual information
(MI) maximization has been widely applied for learning diverse policies (Eysenbach et al., 2019;
Baumli et al., 2021; Choi et al., 2021). An advantage of MI maximization is the existence of tractable
approximations with DNN, scaling it up to more complex domains.

For learning diverse options, we maximize MI between options and their terminating states conditioned by a starting state. This MI maximization diversifies the destinations of an agent when
choosing different options at a state while keeping the transition as deterministic as possible, leading
to diverse and meaningful options. Specifically, we propose to maximize this MI by optimizing


-----

IMTC + VIC

Option 0

OC + VIC

Option 0


Expected value of Policy

0.8

0.6

0.4β(x)

0.2

|Col1|Col2|Col3|
|---|---|---|



Option 3 0.0

Expected value of Policy

0.8

0.6

0.4β(x)

0.2

|Col1|Col2|Col3|
|---|---|---|



Option 3 0.0


Option 1 Option 2

Option 1 Option 2


Figure 1: Left: PointBilliard environment. Right: Learned options by IMTC (upper row) and
OC (lower row). Arrows show policies, and heatmaps show termination conditions of options.

the termination conditions of options. We derive an unbiased gradient estimator of this MI w.r.t.
termination conditions using the termination gradient theorem (Harutyunyan et al., 2019). As a
key contribution, we reformulate the estimated gradient using Bayes’ rule and derive a tractable
approximation with a leaned classifier, yielding the InfoMax Termination Critic (IMTC) algorithm.
The easiness of approximation is advantageous compared to the Termination Critic (TC, Harutyunyan
et al. (2019)). The same objective is used by Variational Intrinsic Control (VIC, Gregor et al. (2016))
for diversifying sub-policies, but the original VIC used a constant termination probability for all
states and options. However, since VIC depends on relating starting and terminating states to an
option, we argue that diversifying termination conditions is also important. We show that our method
helps VIC learn clearer options.

In this paper, we first introduce backgrounds and notations, followed by the derivation of IMTC. We
implement IMTC on the Option Critic Architecture (OC, Bacon et al. (2017)) and Proximal Policy
Optimization (PPO, Schulman et al. (2017)). As a minor contribution, our implementation includes a
newly proposed optimistic advantage estimation that speeds up option learning for a single task. In
experiments, we qualitatively demonstrate that IMTC successfully learns diverse and meaningful
options in a reward-free RL (Jin et al., 2020) setting. Figure 1 shows the learned options by IMTC
and OC combined with VIC rewards in PointBilliard domain. We can see that options learned
by IMTC are clearly separated and different. Then we test the reusability of learned options in task
adaptation experiments: we first pre-train options without rewards and then transfer them on various
tasks. We show that IMTC helps quick adaptation to specific tasks especially in complex tasks where
object manipulation is required. Since IMTC is only for learning terminating conditions of options,
we can combine it with other methods for learning diverse sub-policies. Our experiments observed
that IMTC improves the performance of VIC and RVIC (Baumli et al., 2021).

2 BACKGROUND AND NOTATION

We assume the standard RL setting in the Markov decision process (MDP), following Sutton and
Barto (2018). MDP M consists of a tuple (X _, A, p, r), where X is the finite set of states, A_
is the finite set of actions, p : X × A × X → [0, 1] is the state transition function, r : X ×
_A →_ [rmin, rmax] is the reward function, where rmin, rmax are minimum and maximum. We use
_Xt and At for denoting random variables of the state and action experienced at time t. We define_
def
_G_ = _t=0_ _[γ][t][R][t][ as discounted cumulative return, where][ R][t][ =][ r][(][X][t][, A][t][)][ is the reward received at]_
time t and 0 < γ < 1 is the discount factor. We consider maximizing expected discounted return
E [G _π]. As an agent, we consider a memoryless policy π :_ [0, 1]. π induces two value
_|_ [P][∞] def _X × A →_
functions: action-value functionV _[π](x)_ =def _a_ _[π][(][a][|][x][)][Q][π][(][x, a][)][. We use] Q[π](x, a[ d])_ _[π]=[ for denoting an agent’s average occupancy measure] Eπ [G|X0 = x, A0 = a, π] and state-value function_

_d[π](x) = limn_ E[Pnt=0n[I][X]t [=][x][]], where I is the indicator function.
_→∞[P]_

Assuming that π is differentiable by the policy parameters θπ, the policy gradient (PG)
method (Williams, 1992) maximizes G[π] by updating θπ via gradient ascent. A common formulation
of PG estimates the gradient by ∇θπ _G[π]_ = Ex,a,π _∇θπ log π(a|x) A[ˆ](x, a)_, where _A[ˆ](x, a) is an_

def
estimation of the advantage function A[π](x, a) = Q[π]h(x, a) − _V_ _[π](x). Among many variants of PGi_


-----

methods, we implemented our method on PPO (Schulman et al., 2017). At each gradient step, PPO
updates θπ to maximize clip( _π[π]old[(][a](a[|][x]|x[)])_ _A,[ˆ]_ _−ϵ, ϵ), where clip(x, −ϵ, ϵ) = max(−ϵ, min(ϵ, x)) and πold_

is the policy before being updated. This clipping heuristics prevents π from updating too rapidly.

**Options Framework** Options (Sutton et al., 1999) provide a framework for formulating temporally
abstracted actions in RL. An option o ∈O consists of a tuple (I[o], β[o], π[o]), where I[o] _⊆X is the_
initiation β[o] : X → [0, 1] is a termination function, and π[o] is an intra-option policy. Following
related studies (Bacon et al., 2017; Harutyunyan et al., 2019), we assume that I[o] = X focuses on
learning β[o] and π[o]. We let µ : X × O → [0, 1] denote a policy over options. A typical RL agent
sample At from π(·|Xt). Analogously, at time t, an RL agent with options first samples a termination
_Tt from β[O][t]_ (Xt). If Tt = 1, Ot+1 is sampled from µ(·|Xt) and if not, the current option remains
the same. The next action At is sampled from π[O][t][+1] (·|Xt). For option learning methods, we use π
to denote the resulting policy induced by µ, ∪o∈Oπ[o], and ∪o∈Oβ[o].

**Option value functions** With options, we have three option-value functions Q, V, and U .
_O_ _O_ _O_
_Q_ is the option-value function denoting the value of selecting an option o at state x defined
byO Q (x, o) =def E [G _X0 = x, O0 = o]. Similar to the relationship between Q[π]_ and V _[π], we let_
_O_ _|_ def def
_VO denote the marginalized option-value function VO(x)_ = _o_ _[µ][(][o][|][x][)][Q][O][(][x, o][)][.][ U][O][(][x, o][)]_ =

(1 _β[o](x))Q_ (x, o) + β[o](x)V (x) is called the option-value function upon arrival (Sutton et al.,
_−_ _O_ _O_
1999) and denotes the value of reaching a state x with o and not having selected the new option. We

[P]

use these notations in Section 4.2.

**Termination gradient theorem** Analogously to p, we let P _[o]_ : X × O × X → [0, 1] denote the
state transition probability induced by options. When an agent is at xs and having an option o, the
probability that o ends at xf is given by:


_p[π][o]_ (x _xs)P_ _[o](xf_ _x),_ (1)
_|_ _|_


_P_ _[o](xf_ _|xs) = β[o](xf_ )Ixf =xs + (1 − _β[o](xs))_


def
where p[π][o] is the policy-induced transition function p[π][o] (x[′]|x) = _a∈A_ _[π][o][(][a][|][x][)][P]_ [(][x][′][|][x, a][)][. Here]

we assume that all options eventually terminate, such that P _[o]_ is a valid probability distribution over
_xf_ . Interestingly, P _[o]_ is differentiable with respect to the parameter of β[o]. Harutyunyan et al. (2019)

[P]

introduced the termination gradient theorem:

**Theorem 1. Let β[o]** _be parameterized by θβ, and let ℓβo denote the logit of β[o], i.e., ℓβo =_
log( 1 _β[o]β([o]x()x)_ [)][. We have]

_−_


_θβ_ _P_ _[o](xf_ _xs) =_
_∇_ _|_


_P_ _[o](x_ _xs)_ _θβ_ _ℓβo_ (x)(Ixf =x _P_ _[o](xf_ _x))._ (2)
_|_ _∇_ _−_ _|_


We use this theorem to derive an unbiased estimator of our target gradient.

3 INFOMAX TERMINATION CRITIC

We now present the InfoMax Termination Critic (IMTC) algorithm. To learn diverse options, we
propose to maximize the following MI at each state xs:

_I(Xf_ ; O _xs) = H(Xf_ _xs)_ _H(Xf_ _xs, O) = H(O_ _xs)_ _H(O_ _Xf_ _, xs),_ (3)
_|_ _|_ _−_ _|_ _|_ _−_ _|_

where I denotes the MI I(A; B|c) = H(A|c) − _H(A|B, c), H denotes entropy, and O is a random_
variable denoting options experienced by an agent. Precisely, we let η denote the probability of having
an option o when leaving a state x: η(o|x) = β[o](x)µ(o|x) + (1 − _β[o](x))_ _x[′]∈X_ _[p][π][o]_ [(][x][|][x][′][)][η][(][o][|][x][′][)]

and let η[π] denote η marginalized over dπ: η[π](o) = _x_ _[d][π][(][x][)][η][(][o][|][x][)][. Then, we define][ O][ as a ran-]_

_∈X_
dom variable with η[π]. By decomposing this MI as I(Xf ; O _xs) = H(X[P]f_ _xs)_ _H(Xf_ _xs, O),_
_|_ _|_ _−_ _|_
we can interprete this MI maximization as maximization of H(Xf _xs) and minimization of_

[P]

_|_
_H(Xf_ _xs, O). Maximizing H(Xf_ _xs) diversifies possible destinations of an agent. Thus, the_
_|_ _|_
resulting options are expected to be diverse in the sense that they likely lead to different destinations.
On the other hand, minimizing H(Xf _xs, O) makes option state transitions more deterministic,_
_|_
leading to more meaningful options.


-----

Options with high I(Xf;O|xs) (IMTC)


Options with high -H(Xf|o) (TC)


0.9

0.5


0.9

0.5


(x)


(x)


Option 0 Option 1 Option 2 0.1


Option 0 Option 1 Option 2 0.1


Figure 2: IMTC options (left) and TC options (right).

To build intuition, we searched options that maximize the MI (3) by a simple brute-force search
from a limited set of π[o] and β[o] in a 3 × 3 Gridworld environment. Specifically, we considered
only deterministic intra-option polciies and 0.1 or 0.8 as the value of β[o](x). The number of options
is fixed to |O| = 3. In this environment, an agent can take four actions: Up, Down, Left, and
Right. The taken action fails with probability 0.1, and it takes a uniformly random action. Figure 2
shows the result. Arrows represent intra-option policies and colors of the room represent termination
probabilities. Here, we can see the tendency that IMTC prefers options that have almost separated
termination regions and different policies per option. We also visualized TC (Harutyunyan et al.,
2019) options that maximize _H(Xf_ _o). For TC options, we can see a similar tendency as IMTC_
_−_ _|_
options, but options 0 and 1 of TC share the intra-option policies on the left side of the room. Thanks
to the diversity term H(Xf _xs), IMTC successfully avoids such a policy overlap. In addition, we can_
_|_
see that IMTC has smaller termination regions than TC, which can enhance long-range abstraction.

We propose to maximize the MI (3) by updating β[o] via gradient ascent w.r.t. θβ. For this purpose, we now derive an unbiased estimator of the gradient. First, we write the gradient of the
MI using the option transition model P _[o]_ and marginalized option-transition model P (xf _xs) =_
_|_

_o_ _[η][π][(][o][)][P][ o][(][x][f]_ _[|][x][s][)][.]_

**Proposition 1. Let β[o]** _be parameterized using a sigmoid function._ _Given a trajectory τ =_

P

_xs, . . ., x, . . ., xf sampled by π[o]_ _and β[o], we can obtain unbiased estimations of ∇θβ_ _H(Xf_ _|xs)_
_and_ _θβ_ _H(Xf_ _xs, O) by_
_∇_ _|_

_∇θβ_ _H(Xf_ _|xs) = Edπ,η_ _−∇θβ_ _ℓβo_ (x)β[o](x) log P (x|xs) − log P (xf _|xs)_ (4)
h  i

_∇θβ_ _H(Xf_ _|xs, O) = Edπ,η_ _−∇θβ_ _ℓβo_ (x)β[o](x) log P _[o](x|xs) −_ log P _[o](xf_ _|xs)_ _,_ (5)
h  i

_where ℓβo_ (x) denotes the logit of β[o](x).

Note that the additional term β[o] is necessary because x is not actually a terminating state. The proof
is based on Section 4 in Harutyunyan et al. (2019) and is given in Appendix A.1.

Now the targetting gradient can be written as:

_θβ_ _I(Xf_ ; O _xs) =_ _θβ_ _H(Xf_ _xs)_ _θβ_ _H(Xf_ _xs, O)_
_∇_ _|_ _∇_ _|_ _−∇_ _|_

= Edπ,η _−∇θβ_ _ℓβo_ (x)β[o](x) (log P (x|xs) − log P (xf _|xs) −_ (log P _[o](x|xs) −_ log P _[o](xf_ _|xs)))_ _,_
(6)
 

which means that we can approximate MI maximization by estimating P _[o]_ and P . However, in
RL, learning a density model over the state space in RL is often difficult, especially with large or
continuous state spaces. For example, it has been tackled with data compression methods (Bellemare
et al., 2016) and generative models (Ostrovski et al., 2017). Hence, we reformulate the gradient using
Bayes’ rule to avoid estimating P _[o]_ and P . The resulting term consists of the inverse option transition
function pO(o|xs, xf ), which denotes the probability of having an o given a state transition xs, xf .

**Lemma 1. We now have**

_θβ_ _I(Xf_ ; O _xs) =_ _θβ_ _H(Xf_ _xs)_ _θβ_ _H(Xf_ _xs, O)_
_∇_ _|_ _∇_ _|_ _−∇_ _|_

= Edπ,η _θβ_ _ℓβo_ (x)β[o](x) log p (o _xs, x)_ log p (o _xs, xf_ ) _._ (7)
_∇_ _O_ _|_ _−_ _O_ _|_
h  i

The proof is provided in Appendix A.2. Equation (7) requires estimating of p per updating π[o] and
_O_
_β[o], which is computational quite expensive. Thus, we approximate approximate the gradient (7) by_
regressing a classification model over options ˆpO(o|xs, xf ) on sampled option transition.


-----

4 IMPLEMENTATION

Since IMTC can be combined with any on-policy RL methods, we choose PPO (Schulman et al., 2017)
as a base algorithm because of its stability and ease of implementation. As notable implementation
details, this section explains the estimation of p and advantage estimation. We provide further
_O_
details and the full description of the algorithm in Appendix B.

4.1 ESTIMATING p
_O_

To estimate pO, we employ a classification model over options ˆpO(o|xs, xf ) and regress it on
sampled option transitions, as per Gregor et al. (2016). However, our preliminary experiments
observed that this online regression could be unstable because the supply of transition data depends
on the termination probability and can drastically increase or decrease during training. To address
this problem, we maintain a replay buffer BO, which stores option state transitions {(o, xs, xf )} to
stabilize the regression of ˆp . Note that using older option state transitions can introduce bias to
_O_
_pˆ_ because it depends on the current policy. However, we found that this is not harmful when the
_O_
capacity of the replay buffer is reasonably small.

4.2 ADVANTAGE ESTIMATION

The original PPO implementation employs GAE (Schulman et al., 2015b) for estimating the advantage,
which is important for learning performance (Andrychowicz et al., 2021). Therefore, we employed
two variants of GAE in experiments for option learning according to the experimental setup. In
the following paragraphs, we let N denote the rollout length used for advantage estimation and let
_t + k denote the time step at which the current option ot terminates. Thus, we need to consider_
the effect of option-switching in advantage estimation When k < N . Furthermore, we use two
variants of the option-specific TD errors δ(ot) = Rt + γQ (xt+1, ot) _Q_ (xt, ot) and δU(ot) =
_O_ _−_ _O_
_Rt + γU_ (xt+1, ot) _Q_ (xt, ot).
_O_ _−_ _O_

**Independent GAE for Reward-Free RL** For no-reward experiments with VIC, we used the
following variant of GAE:

min(k,N )


_Aˆ[o]ind_ [=][ −][Q][O][(][x][t][, o][t][) +]


(γλ)[i]δ(ot+i) (8)
_i=0_

X


Here, we ignore the future rewards produced by other options after the current option ot terminates.
This formulation enhances learning diverse intra-option policies per option.

**Upgoing GAE for task adaptation** For single-task learning, increasing the rollout step N often
speeds up learning (Sutton and Barto, 2018). However, future rewards after option termination heavily
depend on the selected option and have high variance, especially when learning diverse options. This
high variance of future rewards slows advantage learning and causes underestimation of _A[ˆ][o]. Thus,_
to prevent underestimation, we introduce an upgoing GAE (UGAE) for estimating advantage with
options:


_N_

_ki=0[(][γλ][)][i][δ]t[o]+i_ [+ max] (γλ)[i]δ(ot+i), 0 (k < N )

_Aˆ[o]upg_ [=][ −][Q][O][(][x][t][, o][t][) +] P upgoing estimationi=Xk+1 ! (9)

_Ni=0−1[(][γλ][)][i][δ]t[o]+i_ [+ (]| _[γλ][)][N]_ _[δ][U][(][o][t][+]{z[N]_ [)] } (otherwise).

Like the upgoing policy update (P Vinyals et al., 2019), the idea is optimistic regarding future rewards
after option termination by taking the maximum over 0. We use _A[ˆ][o]upg_ [for task adaptation experiments]
in Section 5, and confirmed its effectivity in the ablation study in Appendix C.6.

5 EXPERIMENTS

As presented in this section, we conducted two series of experiments to analyze the property of IMTC.
First, we qualitatively evaluated the diversity of options learned by IMTC with intrinsic rewards,
without any extrinsic rewards. Second, we quantitatively test the reusability of learned options by


-----

OC + VIC

Option 0

|Col1|Col2|Col3|Col4|
|---|---|---|---|


VIC (β=0.1)

Option 0

|Col1|Col2|Col3|Col4|
|---|---|---|---|


RVIC (β=0.1)

Option 0


Expected value of Policy


1.0

0.8

0.6

0.4β(x)

0.2


Option 1 Option 2

|Col1|Col2|Col3|
|---|---|---|



Option 1 Option 2

|Col1|Col2|Col3|
|---|---|---|



Option 1 Option 2


Option 3 0.0


Expected value of Policy 1.0

0.8

0.6

0.4β(x)

0.2

Option 3 0.0

Expected value of Policy 1.0

0.8

0.6

0.4β(x)

0.2


Option 3 0.0

|Col1|Col2|Col3|Col4|
|---|---|---|---|

|Col1|Col2|Col3|
|---|---|---|


IMTC + VIC Expected value of Policy

Option 0 Option 1 Option 2 Option 3

IMTC + RVIC Expected value of Policy

Option 0 Option 1 Option 2 Option 3

IMTC (R=0.01) Expected value of Policy

Option 0 Option 1 Option 2 Option 3


Figure 3: Learned intra-option policies (π[o]) and termination probabilities (β[o]) for each option in
PointMaze after training 4 × 10[6] steps. Arrows show the expected value of action, and heatmaps show
probabilities of each β[o]. Left: Options learning by IMTC with three different intrinsic rewards. We
can see that intra-option policies have various directions, and termination regions are clearly separated
across options. To see the difference between intrinsic rewards, RVIC and constant rewards have
an overlapped pair of options (option 1 and 3). Also, we can see that the magnitude of intra-option
policies tends larger with constant rewards. Right: Options learned by other methods. OC produces
a dead option 3 that terminates everywhere and never-ending options 0 and 1. About intra-option
policies, all methods successfully avoided learning the same policy, but they only have two directions.

task adaptation on a specific task. As a baseline of termination learning method, we compared
our method with OC (Bacon et al., 2017). OC is trained with VIC (Gregor et al., 2016) rewards
during pre-training. We did not compare IMTC with TC (Harutyunyan et al., 2019) because our TC
implementation failed to learn options with relatively small termination regions as reported in the
paper, and there is no official public code for TC[1]. During pre-training without extrinsic rewards,
IMTC receives intrinsic rewards when the current option terminates. We compare three IMTC
variants with different intrinsic rewards: (i) VIC (Gregor et al., 2016), (ii) RVIC (Baumli et al., 2021),
and (iii) constant value (RIMTC = 0.01). Note that RIMTC = 0.01 is chosen from [0.1, 0.05, 0.01]
based on the task adaptation results. We also compare IMTC with vanilla VIC and RVIC with fixed
termination probabilities. We used _xβ[o](x) = 0.1 since it performed the best in task adaptation_
_∀_
experiments, while 0.05 was used in Gregor et al. (2016). Note that RVIC’s objective I(Xs; O _xf_ ) is
_|_
different from ours, while IMTC and VIC share almost the same objective. Thus, the use of VIC is
more natural, and the combination with RVIC is tested to show the applicability of IMTC. Further
details of our VIC and RVIC implementation are found in Appendix B. In order to check only the
effect of the different methods for learning beta β[o], the rest of the implementation is the same for all
these methods. That is, OC, vanilla VIC, and vanilla RVIC are also based on PPO and advantage
estimation methods in Section 4.2. In this section, we fix the number of options as |O| = 4 for all
option-learning methods. We further investigated the effect of the number of options Appendix C,
where we confirmed that |O| = 4 is sufficient for most domains. All environments that we used for
experiments are implemented on the MuJoCo (Todorov et al., 2012) physics simulator. We further
describe the details in Appendix C.

**Option Learning From Intrinsic Rewards** We now qualitatively compare the options learned
by IMTC with options of other methods. Learned options depend on the reward structure in the
environment, which enables manually designing good reward functions for learning diverse options.
Thus, we employed a reward-free RL setting where no reward is given to agents. Instead, each
compared method uses some intrinsic rewards, as explained. We fix µ as µ(o|x) = _|O|1_ [in this]

experiment, since we assume that the future tasks are uniformly distributed. Intra-option policies are
trained by PPO (Schulman et al., 2017) and independent GAE (8). We show network architectures
and hyperparameters in Appendix C. We set the episode length to 1 × 10[4], i.e., an agent is reset
to its starting position after 1 × 10[4] steps. For all visualizations, we chose the best one from five
independent runs with different random seeds.

1Note that we also could not find any unofficial open source implementation.


-----

(a) PointReach (b) SwimmerReach (c) AntReach (d) PointBilliard

(e) AntBilliard (f) PointPush (g) AntPush

Figure 4: Domains used in task adaptation experiments.

We visualized learned options in PointReach environment shown in Figure 4a. In this environment,
an agent controls the ball initially placed at the center of the room. The state space consists of positions
(x, y) and velocities (∆x, ∆y) of an agent, and the action space consists of acceralations ( [∆]∆[x]t _[,][ ∆]∆[y]t_ [)][.]

Figure 3 shows the options learned in this environment after 4 × 10[6] steps. Each arrow represents the
mean value of intra-option policies, and the heatmaps represent β[o]. In this experiment, we observed
the effect of IMTC clearly, for both termination regions and intra-option policies. Interestingly, we
don’t see clear differences between options learned with VIC and RVIC rewards, while constant
rewards tend to make options peaker. OC failed to learn meaningful termination regions: option 0
and 1 never terminate, and option 3 terminates almost everywhere. This result confirms that IMTC
can certainly diversify options. Moreover, compared to vanilla VIC and RVIC, intra-option policies
learned by IMTC with VIC or RVIC rewards are clearer, in terms of both the magnitude and directions
of policies. We believe that this is because diversifying termination regions gives more biased samples
to the option classifiers employed by VIC and RVIC. Figure 1 also show options learned by IMTC
and OC in PointBilliard domain, where we can see the same tendency.

**Transferring skills via task adaptation** Now we quantitatively test the reusability of learned
options by task adaptation with specific reward functions. Specifically, we first trained agents with
intrinsic rewards as per the previous section. Then we transferred agents to an environment with the
same state and action space but with external rewards. We prepared multiple reward functions, which
we call tasks, for each domain and evaluated the averaged performance over tasks. We compare IMTC
with OC, vanilla VIC, vanilla RVIC, and PPO without pre-training. Also, we compare three variants
of IMTC with different intrinsic rewards during pre-training. For a fair comparison, UGAE (9) and
PPO are used for all options learning methods. Note that we found UGAE is very effective in this
experiments, as we show the ablation study in Appendix C.6. For vanilla VIC and vanilla RVIC,
termination probability is fixed to 0.1 through pre-training and task adaptation. ϵ-greedy based on
_Q_ with ϵ = 0.1 is used as the option selection policy µ. We hypothesize that diverse options learned
_O_
by IMTC can help quickly adapt to given tasks, supposing the diversity of tasks.

Figure 4 shows all domains used for task adaptation experiments. For simplicity, all tasks have goalbased sparse reward functions. I.e., an agent receives Rt = 1.0 when it satisfies a goal condition, and
otherwise the control cost −0.0001 is given. Red circles show possible goal locations for each task.
When the agent fails to reach the goal after 1000 steps, it is reset to a starting position. PointReach,
SwimmerReach, and AntReach are simple navigation tasks where an agent aim to just navigate
itself to the goal. We also prepared tasks with object manipulation: in PointBilliard and
AntBilliard an agent aims to kick the blue ball to the goal position, and in PointPush and
AntPush, it has to push the block out of the way to the goal. We pre-traine options learning
agents for 4 × 10[6] environmental steps and additionally trained them for 1 × 10[6] steps for each
task. Figure 5 shows learning curves and scatter plots drawn from five independent runs with


-----

PointReach SwimmerReach AntReach PointBilliard

1.0 0.8 0.7 1.0

0.8 0.6 0.6 0.8

0.6 0.5

Return 0.40.2 Return0.40.2 Return 0.40.30.2 Return0.60.4

0.0

0.1

0.2 0.0 0.0 0.2

0.4 0.1

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6

0.35 AntBilliard PointPush AntPush

1.0 0.5

0.30

0.25 0.8 0.4

0.20 0.6 0.3

Return0.150.100.05 Return 0.40.20.0 Return 0.20.10.0 IMTC + VICIMTC + RVICIMTC (R=0.01)OC + VICMethod

0.00 0.2 0.4 0.6 0.8 1.0 0.2 0.2 0.4 0.6 0.8 1.0 0.1 0.2 0.4 0.6 0.8 1.0 VIC ( =0.1)RVIC ( =0.1)

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 PPO


Figure 5: Learning curves for transfer learning experiments.

different random seeds per domain. [2] Here, we observed that IMTC with VIC or RVIC rewards
performed the best or was compatible with baselines. IMTC with VIC performed better than OC
with VIC except for AntRearch, which backs up the effectiveness of diversifying termination
regions for learning reusable options. Also, IMTC with VIC and IMTC with RCIC respectively
performed better in most of the tasks than VIC and RVIC with fixed termination probabilities.
This result suggests that IMTC can boost the performance of option learning methods based on
option classifiers, even when the objective is different as with RVIC. On the other hand, IMTC
with constant rewards (RIMTC = 0.01) performed worse than IMTC with VIC or RVIC rewards,
although it also learned diverse options as we show in Figure 3, suggesting the importance of adjusting
rewards. We further analyzed the evolution of intrinsic rewards of VIC and RVIC in Appendix C.5.
In addition, we can observe that IMTC’s performance is especially better than other methods in
relatively complex PointBilliard, AntBilliard, and AntPush, where object manipulation
is required. Considering that manipulated balls and boxes move faster than agents in these domains, a
choice of options can lead to larger differences in the future state. IMTC is suitable to these domains
since it maximizes the diversity of the resulting states, while PPO struggles to learn. Contrary,
IMTC’s performance is close to other methods in Reach tasks, where the goal states are relatively
close to the starting states in terms of euclidian distances.

**Gridworld experiments and limitation of the method** Although IMTC successfully learned
diverse options in MuJoCo experiments, our analysis in Figure 2 shows the possibility of learning
options that are not interesting but have large MI. We further investigated this possibility by visualizing
options in a classical four rooms gridworld in Appendix C.8. Interestingly, we observed that IMTC
could fall into diverse but unmeaningful options in that environment. We believe that IMTC is often
sufficient in a large environment where a randomly-initialized agent rarely gets the same trajectory.
However, when the number of possible trajectories is small, diversifying the destinations could be
insufficient. In such cases, it can be necessary to extend IMTC to diversify trajectories as done in
Sharma et al. (2020).


6 RELATED WORK

**Discovering diverse options** Since Sutton et al. (1999) first formulated the options, discovering
good options has been challenging. A classic concept for the goodness of options is a bottleneck
region (McGovern and Barto, 2001), which refers to important states for reaching diverse areas,
such as a passage between two rooms. Once an important region is discovered, we can construct an
option that guides an agent to that area. Various approaches have been proposed to define bottleneck
regions concisely and compute bottleneck options efficiently, including betweenness (Simsek and
Barto, 2008), Eigen options (Machado et al., 2017), successor options (Ramesh et al., 2019), and
covering options (Jinnai et al., 2019). Barto et al. (2013) discussed the importance of diverse options
for exploration, referring to an option construction method with a graph-based decomposition of
an MDP Vigorito and Barto (2010). We share the same motivation with these methods but have a


2We used seaborn (Waskom, 2021)’s lmplot with order=3 is used to draw learning curves.


-----

different approach. While these methods construct point options that bridge two states, we capture a
set of states by learning β[o] directly, making it easy to scale up with function approximation. To our
knowledge, (Jinnai et al., 2020) only succeeded in scaling up point options to continuous options
using approximated computation of the Laplacian.

**End-to-end learning of options** As described in the previous paragraph, many studies have attempted to construct options and then train intra-option policies. However, motivated by the recent
success of RL with DNN, Bacon et al. (2017) proposed OC to train intra-option policies and termination functions in parallel using a neural network as a function approximator. OC updates β[o] by
gradient ascent, so that maximize Q . Thus, the resulting terminating regions heavily depend on
_O_
the reward function. OC also proposed a PG-style method for learning intra-option policies. We
used a similar method to OC for learning policies and values, but we employed a different method
that does not depend on rewards for learning termination functions. Also, we proposed UGAE (9)
for enhancing OC-style policy learning. While OC maximizes the option-value function directly,
many heuristic objectives have been proposed with similar architectures, including deliberation cost
(Harb et al., 2018), interest (Khetarpal et al., 2020), and safety (Jain et al., 2018). Kamat and Precup
(2020) extended OC so that resulting options are diverse by maximizing the divergence between
intra-option policies, while IMTC considers the diversity of destinations. Notably, Harutyunyan
et al. (2019) proposed the termination critic (TC) that maximizes an information-theoretic objective
referred to as predictability _H(Xf_ _o). Our method is inspired by TC and maximizes a similar_
_−_ _|_
information-theoretic objective for diversity rather than predictability. In addition, TC requires
estimating P _[o](xf_ _xs) and a marginal distribution of P_ _[o], which can be quite difficult in environments_
_|_
with large or continuous state spaces. We avoid such difficult approximations using Bayes’ rule,
making IMTC more scalable.

**Mutual Information, Empowerment, and Skill Acquisition** MI also appears in the literature
regarding intrinsically motivated RL (Singh et al., 2004), as a driver of goal-directed behavior.
A well-known example is the empowerment (Klyubin et al., 2005; Salge et al., 2014), obtained
by maximizing MI between sequential k actions and the resulting state I(at, ..., at+k; xt+k _xt) =_
_|_
_H(xt+k_ _xt)_ _H(xt+k_ _at, ..., at+k, xt). Empowerment represents both large degree of freedom_
_|_ _−_ _|_
and good preparedness: i.e., larger H(xt+k _xt) implies that there can be more diverse future states,_
_|_
while smaller H(xt+k _at, ..., at+k, xt) indicates that an agent can realize its intention with greater_
_|_
certainty. In RL literature, empowerment is often implemented by maximizing the variational lower
bounds as intrinsic rewards (Mohamed and Rezende, 2015; Zhao et al., 2020). We can interpret our
objective I(Xf ; O _xs) as a variant of empowerment where a fixed number of options represents action_
_|_
sequences. Gregor et al. (2016) also employed this interpretation and introduced VIC for training
intra-option policies with fixed termination probabilities in a no reward RL setting. Our experiments
observed that IMTC helps VIC to learn meaningful intra-option policies. As a variant of VIC, Baumli
et al. (2021) proposed to use the reversed MI I(Xs; O _xf_ ). MI has been used for discovering diverse
_|_
skills without termination functions. Eysenbach et al. (2019) proposed maximizing MI between skills
and states I(O; X), and Sharma et al. (2020) extended the objective to a conditional MI I(O; X _[′]|x)._
These methods also maximize variational lower bounds of MI as rewards.

7 CONCLUSION

In this paper, we considered the problem of learning diverse options in RL. To learn diverse termination regions of options in a scalable way, we proposed to maximize MI between options and
terminating states per starting state. We derived an unbiased gradient estimator to approximately
maximize this MI, yielding the InfoMax Termination Critic (IMTC) algorithm. Also, we proposed a
practical implementation of IMTC with enhanced advantage estimation in Section 4.2. In reward-free
experiments, we visualized that IMTC helped learn diverse and clear options combined with intrinsic
rewards. We also showed that options learned by IMTC help an agent to quickly adapt to a specific
reward function by transferring learned options. Although our experiments observed that IMTC can
learn clear and meaningful options, a potential problem is that learning of a classification model ˆp
_O_
heavily depends on exploration. For example, an agent cannot explore a room well, it would not be
able to learn sufficiently diverse options. The relationship between diversity and exploration would be
interesting. Also, our analysis in Figure 2 and Gridworld experiments in Appendix C.8 suggest that
IMTC options can fall into small loops, forming uninteresting options. To prevent this, considering
distances between states, e.g., by using bisimulation metric (Castro and Precup, 2010), is a plausible
research direction.


-----

REPRODUCIBILITY STATEMENT

We publish anonymized source code used for all our experiments on [https://](https://anonymous.4open.science/r/imtc-anonymized-code-E5D1/)
[anonymous.4open.science/r/imtc-anonymized-code-E5D1/.](https://anonymous.4open.science/r/imtc-anonymized-code-E5D1/)

REFERENCES

M. Andrychowicz, A. Raichuk, P. Stanczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist,
O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actorcritic methods? a large-scale study. In 9th International Conference on Learning Representa_[tions, ICLR 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=](https://openreview.net/forum?id=nIAxjsniDzg&)_
[nIAxjsniDzg&.](https://openreview.net/forum?id=nIAxjsniDzg&)

P. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the Thirty_First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California,_
_[USA, pages 1726–1734, 2017. URL http://aaai.org/ocs/index.php/AAAI/AAAI17/](http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14858)_
[paper/view/14858.](http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14858)

A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discret. Event
_[Dyn. Syst., 13(1-2):41–77, 2003. doi: 10.1023/A:1022140919877. URL https://doi.org/](https://doi.org/10.1023/A:1022140919877)_
[10.1023/A:1022140919877.](https://doi.org/10.1023/A:1022140919877)

A. G. Barto, S. Singh, N. Chentanez, et al. Intrinsically motivated learning of hierarchical collections
of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages
112–19. Piscataway, NJ, 2004.

A. G. Barto, G. D. Konidaris, and C. M. Vigorito. Behavioral hierarchy: Exploration and representation. In G. Baldassarre and M. Mirolli, editors, Computational and Robotic Models of the Hierarchi_[cal Organization of Behavior, pages 13–46. Springer, 2013. doi: 10.1007/978-3-642-39875-9\_2.](https://doi.org/10.1007/978-3-642-39875-9_2)_
[URL https://doi.org/10.1007/978-3-642-39875-9_2.](https://doi.org/10.1007/978-3-642-39875-9_2)

K. Baumli, D. Warde-Farley, S. Hansen, and p Volodymyr Mnih. Relative variational intrinsic control.
In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, February 2-9, 2021,
2021.

M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying countbased exploration and intrinsic motivation. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon,
and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference
_on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages_
1471–1479, 2016. [URL http://papers.nips.cc/paper/6383-unifying-count-](http://papers.nips.cc/paper/6383-unifying-count-based-exploration-and-intrinsic-motivation)
[based-exploration-and-intrinsic-motivation.](http://papers.nips.cc/paper/6383-unifying-count-based-exploration-and-intrinsic-motivation)

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai
[gym. CoRR, abs/1606.01540, 2016. URL http://arxiv.org/abs/1606.01540.](http://arxiv.org/abs/1606.01540)

E. Brunskill and L. Li. Pac-inspired option discovery in lifelong reinforcement learning. In Pro_ceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China,_
_21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 316–324._
[JMLR.org, 2014. URL http://proceedings.mlr.press/v32/brunskill14.html.](http://proceedings.mlr.press/v32/brunskill14.html)

P. S. Castro and D. Precup. Using bisimulation for policy transfer in mdps. In M. Fox and D. Poole,
editors, Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010,
_[Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press, 2010. URL http://www.aaai.org/](http://www.aaai.org/ocs/index.php/AAAI/AAAI10/paper/view/1907)_
[ocs/index.php/AAAI/AAAI10/paper/view/1907.](http://www.aaai.org/ocs/index.php/AAAI/AAAI10/paper/view/1907)

J. Choi, A. Sharma, H. Lee, S. Levine, and S. S. Gu. Variational empowerment as representation
learning for goal-conditioned reinforcement learning. In M. Meila and T. Zhang, editors, Pro_ceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July_
_2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 1953–1963._
[PMLR, 2021. URL http://proceedings.mlr.press/v139/choi21b.html.](http://proceedings.mlr.press/v139/choi21b.html)


-----

R. Coulom. Reinforcement Learning Using Neural Networks, with Applications to Motor Control.
_(Apprentissage par renforcement utilisant des réseaux de neurones, avec des applications au_
_[contrôle moteur). PhD thesis, Grenoble Institute of Technology, France, 2002. URL https:](https://tel.archives-ouvertes.fr/tel-00003985)_
[//tel.archives-ouvertes.fr/tel-00003985.](https://tel.archives-ouvertes.fr/tel-00003985)

Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement
learning for continuous control. In M. Balcan and K. Q. Weinberger, editors, Proceedings of
_the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA,_
_June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1329–1338._
[JMLR.org, 2016. URL http://proceedings.mlr.press/v48/duan16.html.](http://proceedings.mlr.press/v48/duan16.html)

B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a
reward function. In 7th International Conference on Learning Representations, ICLR 2019, New
_[Orleans, LA, USA, May 6-9, 2019, 2019. URL https://openreview.net/forum?id=](https://openreview.net/forum?id=SJx63jRqFm)_
[SJx63jRqFm.](https://openreview.net/forum?id=SJx63jRqFm)

A. Geramifard, C. Dann, R. H. Klein, W. Dabney, and J. P. How. Rlpy: a value-function-based
reinforcement learning framework for education and research. J. Mach. Learn. Res., 16:1573–1578,
[2015. URL http://dl.acm.org/citation.cfm?id=2886799.](http://dl.acm.org/citation.cfm?id=2886799)

K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. CoRR, abs/1611.07507,
[2016. URL http://arxiv.org/abs/1611.07507.](http://arxiv.org/abs/1611.07507)

J. Harb, P. Bacon, M. Klissarov, and D. Precup. When waiting is not an option: Learning options
with a deliberation cost. In Proceedings of the Thirty-Second AAAI Conference on Artificial
_Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18),_
_and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18),_
_[New Orleans, Louisiana, USA, February 2-7, 2018, pages 3165–3172, 2018. URL https:](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17421)_
[//www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17421.](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17421)

A. Harutyunyan, W. Dabney, D. Borsa, N. Heess, R. Munos, and D. Precup. The termination
critic. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS
_2019, 16-18 April 2019, Naha, Okinawa, Japan, pages 2231–2240, 2019._ [URL http://](http://proceedings.mlr.press/v89/harutyunyan19a.html)
[proceedings.mlr.press/v89/harutyunyan19a.html.](http://proceedings.mlr.press/v89/harutyunyan19a.html)

N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A.
Eslami, M. A. Riedmiller, and D. Silver. Emergence of locomotion behaviours in rich environments.
_[CoRR, abs/1707.02286, 2017. URL http://arxiv.org/abs/1707.02286.](http://arxiv.org/abs/1707.02286)_

A. Jain, K. Khetarpal, and D. Precup. Safe option-critic: Learning safety in the option-critic
[architecture. CoRR, abs/1807.08060, 2018. URL http://arxiv.org/abs/1807.08060.](http://arxiv.org/abs/1807.08060)

C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu. Reward-free exploration for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18
_July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4870–_
[4879. PMLR, 2020. URL http://proceedings.mlr.press/v119/jin20d.html.](http://proceedings.mlr.press/v119/jin20d.html)

Y. Jinnai, J. W. Park, D. Abel, and G. D. Konidaris. Discovering options for exploration by minimizing
cover time. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International
_Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA,_
volume 97 of Proceedings of Machine Learning Research, pages 3130–3139. PMLR, 2019. URL
[http://proceedings.mlr.press/v97/jinnai19b.html.](http://proceedings.mlr.press/v97/jinnai19b.html)

Y. Jinnai, J. W. Park, M. C. Machado, and G. D. Konidaris. Exploration in reinforcement learning
with deep covering options. In 8th International Conference on Learning Representations, ICLR
_[2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://](https://openreview.net/forum?id=SkeIyaVtwB)_
[openreview.net/forum?id=SkeIyaVtwB.](https://openreview.net/forum?id=SkeIyaVtwB)

A. Kamat and D. Precup. Diversity-enriched option-critic. CoRR, abs/2011.02565, 2020. URL

[https://arxiv.org/abs/2011.02565.](https://arxiv.org/abs/2011.02565)


-----

K. Khetarpal, M. Klissarov, M. Chevalier-Boisvert, P. Bacon, and D. Precup. Options of interest:
Temporal abstraction with interest functions. In The Thirty-Fourth AAAI Conference on Artificial
_Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Con-_
_ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence,_
_EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 4444–4451. AAAI Press, 2020. URL_
[https://aaai.org/ojs/index.php/AAAI/article/view/5871.](https://aaai.org/ojs/index.php/AAAI/article/view/5871)

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun,
editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA,
_[USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/](http://arxiv.org/abs/1412.6980)_
[1412.6980.](http://arxiv.org/abs/1412.6980)

A. S. Klyubin, D. Polani, and C. L. Nehaniv. All else being equal be empowered. In M. S.
Capcarrère, A. A. Freitas, P. J. Bentley, C. G. Johnson, and J. Timmis, editors, Advances in
_Artificial Life, 8th European Conference, ECAL 2005, Canterbury, UK, September 5-9, 2005,_
_Proceedings, volume 3630 of Lecture Notes in Computer Science, pages 744–753. Springer, 2005._
[doi: 10.1007/11553090\_75. URL https://doi.org/10.1007/11553090_75.](https://doi.org/10.1007/11553090_75)

M. C. Machado, M. G. Bellemare, and M. H. Bowling. A laplacian framework for option discovery
in reinforcement learning. In Proceedings of the 34th International Conference on Machine
_Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2295–2304, 2017. URL_
[http://proceedings.mlr.press/v70/machado17a.html.](http://proceedings.mlr.press/v70/machado17a.html)

A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement learning using
diverse density. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the Eighteenth
_International Conference on Machine Learning (ICML 2001), Williams College, Williamstown,_
_MA, USA, June 28 - July 1, 2001, pages 361–368. Morgan Kaufmann, 2001._

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu.
Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd Interna_tional Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24,_
_[2016, pages 1928–1937, 2016. URL http://jmlr.org/proceedings/papers/v48/](http://jmlr.org/proceedings/papers/v48/mniha16.html)_
[mniha16.html.](http://jmlr.org/proceedings/papers/v48/mniha16.html)

S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors,
_Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information_
_Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2125–2133,_
[2015. URL http://papers.nips.cc/paper/5668-variational-information-](http://papers.nips.cc/paper/5668-variational-information-maximisation-for-intrinsically-motivated-reinforcement-learning)
[maximisation-for-intrinsically-motivated-reinforcement-learning.](http://papers.nips.cc/paper/5668-variational-information-maximisation-for-intrinsically-motivated-reinforcement-learning)

O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning.
In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on
_Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal,_
_[Canada, pages 3307–3317, 2018. URL https://proceedings.neurips.cc/paper/](https://proceedings.neurips.cc/paper/2018/hash/e6384711491713d29bc63fc5eeb5ba4f-Abstract.html)_
[2018/hash/e6384711491713d29bc63fc5eeb5ba4f-Abstract.html.](https://proceedings.neurips.cc/paper/2018/hash/e6384711491713d29bc63fc5eeb5ba4f-Abstract.html)

OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino,
M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan,
W. Zaremba, and L. Zhang. Solving rubik’s cube with a robot hand. CoRR, abs/1910.07113, 2019.
[URL http://arxiv.org/abs/1910.07113.](http://arxiv.org/abs/1910.07113)

G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with
neural density models. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International
_Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol-_
ume 70 of Proceedings of Machine Learning Research, pages 2721–2730. PMLR, 2017. URL
[http://proceedings.mlr.press/v70/ostrovski17a.html.](http://proceedings.mlr.press/v70/ostrovski17a.html)

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style,


-----

high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Informa_tion Processing Systems 32:_ _Annual Conference on Neural Information Processing Sys-_
_tems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 8024–8035,_
2019. [URL http://papers.nips.cc/paper/9015-pytorch-an-imperative-](http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library)
[style-high-performance-deep-learning-library.](http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library)

R. Ramesh, M. Tomar, and B. Ravindran. Successor options: An option discovery framework for
reinforcement learning. In S. Kraus, editor, Proceedings of the Twenty-Eighth International Joint
_Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3304–_
[3310. ijcai.org, 2019. doi: 10.24963/ijcai.2019/458. URL https://doi.org/10.24963/](https://doi.org/10.24963/ijcai.2019/458)
[ijcai.2019/458.](https://doi.org/10.24963/ijcai.2019/458)

C. Salge, C. Glackin, and D. Polani. Empowerment–An Introduction, pages 67–114. Springer Berlin
Heidelberg, Berlin, Heidelberg, 2014. ISBN 978-3-642-53734-9. doi: 10.1007/978-3-642-53734[9_4. URL https://doi.org/10.1007/978-3-642-53734-9_4.](https://doi.org/10.1007/978-3-642-53734-9_4)

A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning
in deep linear neural networks. In Y. Bengio and Y. LeCun, editors, 2nd International Conference
_on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track_
_[Proceedings, 2014. URL http://arxiv.org/abs/1312.6120.](http://arxiv.org/abs/1312.6120)_

J. Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In
F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine
_Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference_
_[Proceedings, pages 1889–1897. JMLR.org, 2015a. URL http://proceedings.mlr.press/](http://proceedings.mlr.press/v37/schulman15.html)_
[v37/schulman15.html.](http://proceedings.mlr.press/v37/schulman15.html)

J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous
[control using generalized advantage estimation. CoRR, abs/1506.02438, 2015b. URL http:](http://arxiv.org/abs/1506.02438)
[//arxiv.org/abs/1506.02438.](http://arxiv.org/abs/1506.02438)

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
[algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347.](http://arxiv.org/abs/1707.06347)

A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery
of skills. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa,
_Ethiopia, April 26-30, 2020. OpenReview.net, 2020._ [URL https://openreview.net/](https://openreview.net/forum?id=HJgLZR4KvH)
[forum?id=HJgLZR4KvH.](https://openreview.net/forum?id=HJgLZR4KvH)

Ö. Simsek and A. G. Barto. Skill characterization based on betweenness. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Sys_tems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Process-_
_ing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pages 1497–1504._
Curran Associates, Inc., 2008. [URL http://papers.nips.cc/paper/3411-skill-](http://papers.nips.cc/paper/3411-skill-characterization-based-on-betweenness)
[characterization-based-on-betweenness.](http://papers.nips.cc/paper/3411-skill-characterization-based-on-betweenness)

S. P. Singh, A. G. Barto, and N. Chentanez. Intrinsically motivated reinforcement learning. In
_Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems,_
_NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada], pages 1281–1288,_
[2004. URL http://papers.nips.cc/paper/2552-intrinsically-motivated-](http://papers.nips.cc/paper/2552-intrinsically-motivated-reinforcement-learning)
[reinforcement-learning.](http://papers.nips.cc/paper/2552-intrinsically-motivated-reinforcement-learning)

A. Solway, C. Diuk, N. Córdova, D. Yee, A. G. Barto, Y. Niv, and M. Botvinick. Optimal behavioral
hierarchy. PLoS Computational Biology, 10(8), 2014. doi: 10.1371/journal.pcbi.1003779. URL
[https://doi.org/10.1371/journal.pcbi.1003779.](https://doi.org/10.1371/journal.pcbi.1003779)

R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.

R. S. Sutton, D. Precup, and S. P. Singh. Between mdps and semi-mdps: A framework for temporal
abstraction in reinforcement learning. Artif. Intell., 112(1-2):181–211, 1999. doi: 10.1016/S0004[3702(99)00052-1. URL https://doi.org/10.1016/S0004-3702(99)00052-1.](https://doi.org/10.1016/S0004-3702(99)00052-1)


-----

E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In
_2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vil-_
_amoura, Algarve, Portugal, October 7-12, 2012, pages 5026–5033. IEEE, 2012. doi: 10.1109/_
[IROS.2012.6386109. URL https://doi.org/10.1109/IROS.2012.6386109.](https://doi.org/10.1109/IROS.2012.6386109)

C. M. Vigorito and A. G. Barto. Intrinsically motivated hierarchical skill learning in structured environments. IEEE Trans. Auton. Ment. Dev., 2(2):132–143, 2010. doi: 10.1109/TAMD.2010.2050205.
[URL https://doi.org/10.1109/TAMD.2010.2050205.](https://doi.org/10.1109/TAMD.2010.2050205)

O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell,
T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai,
J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden,
Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama,
D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis,
C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement
[learning. Nature, 575(7782):350–354, 2019. doi: 10.1038/s41586-019-1724-z. URL https:](https://doi.org/10.1038/s41586-019-1724-z)
[//doi.org/10.1038/s41586-019-1724-z.](https://doi.org/10.1038/s41586-019-1724-z)

M. L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021,
[2021. doi: 10.21105/joss.03021. URL https://doi.org/10.21105/joss.03021.](https://doi.org/10.21105/joss.03021)

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement
[learning. Mach. Learn., 8:229–256, 1992. doi: 10.1007/BF00992696. URL https://doi.org/](https://doi.org/10.1007/BF00992696)
[10.1007/BF00992696.](https://doi.org/10.1007/BF00992696)

Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba. Scalable trust-region method for deep
reinforcement learning using kronecker-factored approximation. In I. Guyon, U. von Luxburg,
S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances
_in Neural Information Processing Systems 30: Annual Conference on Neural Information_
_Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5279–5288,_
2017. [URL http://papers.nips.cc/paper/7112-scalable-trust-region-](http://papers.nips.cc/paper/7112-scalable-trust-region-method-for-deep-reinforcement-learning-using-kronecker-factored-approximation)
[method-for-deep-reinforcement-learning-using-kronecker-factored-](http://papers.nips.cc/paper/7112-scalable-trust-region-method-for-deep-reinforcement-learning-using-kronecker-factored-approximation)
[approximation.](http://papers.nips.cc/paper/7112-scalable-trust-region-method-for-deep-reinforcement-learning-using-kronecker-factored-approximation)

R. Zhao, P. Abbeel, and S. Tiomkin. Efficient online estimation of empowerment for reinforcement
[learning. CoRR, abs/2007.07356, 2020. URL https://arxiv.org/abs/2007.07356.](https://arxiv.org/abs/2007.07356)


-----

A OMITTED PROOFS

A.1 PROOF OF PROPOSITION 1

First, we make an assumption on the dependence of η from β.
**Assumption 1. The empirical distribution of options η(o|xs) is independent of termination conditions**
_∪o∈Oβ[o]._

This assumption does not strictly hold, and we can consider using, e.g., two-time scale optimization
to suppress the distribution shift of η by updating β[o]. We employed a replay buffer to mitigate this
issue in Section 4.1.
**Lemma 2. The following equations hold.**


_P_ _[o](x_ _xs)_ _θβ_ _ℓβo_ (x)
_|_ _∇_

log P (x _xs) + 1_
_|_ _−_
h

_P_ _[o](x_ _xs)_ _θβ_ _ℓβo_ (x)
_|_ _∇_

log P _[o](x_ _xs) + 1_
_|_ _−_
h


_θβ_ _H(Xf_ _xs) =_
_∇_ _|_ _−_

_θβ_ _H(Xf_ _xs, O) =_
_∇_ _|_ _−_


_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_


_P_ _[o](xf_ _x)_ log P (xf _xs) + 1_
_|_ _|_
_xf_

X  i

(10)

_P_ _[o](xf_ _x)_ log P _[o](xf_ _xs) + 1_
_|_ _|_
_xf_

X  i

(11)


Then, sampling x, xf from d[π] and o from η,

_∇θβ_ _H(Xf_ _|xs) = Edπ,η_ _−∇θβ_ _ℓβo_ (x)β[o](x) log P (x|xs) − log P (xf _|xs)_ (12)
h  i

_∇θβ_ _H(Xf_ _|xs, O) = Edπ,η_ _−∇θβ_ _ℓβo_ (x)β[o](x) log P _[o](x|xs) −_ log P _[o](xf_ _|xs)_ _._ (13)
h  i

**Proof of Lemma 2**

_Proof. First, we prove Equation (10). We have:_


_θβ_ _H(Xf_ _xs) =_ _θβ_
_∇_ _|_ _−∇_


_P_ (xf _xs) log P_ (xf _xs)_
_|_ _|_
_xf_

X


_∇θβ_ _P_ (xf _|xs) log P_ (xf _|xs) + P_ (xf _|xs)_ _[∇][θ]P[β]([P]x[(]f[x]x[f]_ _s[|][x])[s][)]_

_|_


= −

= −

= −

= −

= −

= −


_xf_


_θβ_ _P_ (xf _xs)_ log P (xf _xs) + 1_
_∇_ _|_ _|_
_xf_

X 


_η(o_ _xs)_ _θβ_ _P_ _[o](xf_ _xs)_ log P (xf _xs) + 1_
_|_ _∇_ _|_ _|_
Apply eq. (2)  

_η(o_ _xs)_ | _P_ _[o]{z(x_ _xs)_ } _θβ_ _ℓβo_ (x)(Ixf =x _P_ _[o](xf_ _x))_ log P (xf _xs) + 1_
_|_ _x_ _|_ _∇_ _−_ _|_ _|_
X 


_xf_

_xf_

X


_P_ _[o](x_ _xs)_ _θβ_ _ℓβo_ (x)
_|_ _∇_


_θβ_ _ℓβo_ (x) (Ixf =x _P_ _[o](xf_ _x))_ log P (xf _xs) + 1_
_∇_ _xf_ _−_ _|_ _|_
X  

_θβ_ _ℓβo_ (x) log P (x _xs) + 1_ _P_ _[o](xf_ _x)_ log P (xf _xs) + 1_
_∇_ _×_ _|_ _−_ _|_ _|_

_xf_

h X  i

sample
| {z }


_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_

sample
{z }


_P_ _[o](x_ _xs)_
_|_

sample
{z }


-----

Sampling x, xf _, o, we get (12). Then we prove Equation (11)._


_P_ _[o](xf_ _xs) log P_ _[o](xf_ _xs)_
_|_ _|_
_xf_

X


_θβ_ _H(Xf_ _xs, O) =_ _θβ_
_∇_ _|_ _−∇_


_η(o_ _xs)_
_|_


_∇θβ_ _P_ _[o](xf_ _|xs) log P_ _[o](xf_ _|xs) + P_ _[o](xf_ _|xs)_ _[∇][θ]P[β][o][P]([ o]x[(]f[x]x[f]_ _s[|][x])[s][)]_

_|_


= −

= −

= −

= −

= −


_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_

_η(o_ _xs)_
_|_

sample
{z }


_xf_


_θβ_ _P_ _[o](xf_ _xs)_ log P _[o](xf_ _xs) + 1_
_∇_ _|_ _|_
_xf_

X Apply eq. (2)  

| _P_ _[o]{z(x_ _xs)_ } _θβ_ _ℓβo_ (x)(Ixf =x _P_ _[o](xf_ _x))_ log P _[o](xf_ _xs) + 1_

_xf_ _x_ _|_ _∇_ _−_ _|_ _|_

X X 


_P_ _[o](x_ _xs)_ _θβ_ _ℓβ[o]_ (x)
_|_ _∇_


_θβ_ _ℓβ[o]_ (x) (Ixf =x _P_ _[o](xf_ _x))_ log P _[o](xf_ _xs) + 1_
_∇_ _xf_ _−_ _|_ _|_
X  

_θβ_ _ℓβo_ (x) log P _[o](x_ _xs) + 1_ _P_ _[o](xf_ _x)_ log P _[o](xf_ _xs) + 1_
_∇_ _×_ _|_ _−_ _|_ _|_

_xf_

h X  i

sample
| {z }


_P_ _[o](x_ _xs)_
_|_

sample
{z }


Sampling x, xf _, o, we get eq. (13)._

A.2 PROOF OF LEMMA 1

_Proof. First, using Bayes’ rule, we have:_


_P_ _[o](xf_ _xs) = [Pr(][o][|][x][f]_ _[, x][s][) Pr(][x][f]_ _[|][x][s][)]_ = _[p][O][(][o][|][x][f]_ _[, x][s][)][P]_ [(][x][f] _[|][x][s][)]_
_|_ Pr(o _xs)_ _η(o_ _xs)_

_|_ _|_


Then, we have:


log P _[o](xf_ _xs)_ log P (xf _xs) = log_ _[P][ o][(][x][f]_ _[|][x][s][)]_
_|_ _−_ _|_ _P_ (xf _xs)_

_|_

= log _pO(o|xsη,x(of| )xPs) (xf |xs)_

_P_ (xf _xs)_
_|_

= log _[p][O][(][o][|][x][s][, x][f]_ [)]

_η(o_ _xs)_
_|_

Applying this equation to the eq. (6), we have:

_θβ_ _I(Xf_ ; O _xs) =_ _θβ_ _H(Xf_ _xs)_ _θβ_ _H(Xf_ _xs, O)_
_∇_ _|_ _∇_ _|_ _−∇_ _|_

= Ed[π],η _−∇θβ_ _ℓβ[o]_ (x)β[o](x) log P (x|xs) − log P (xf _|xs) −_ log P _[o](x|xs) + log P_ _[o](xf_ _|xs)_
h  i

= Ed[π],η _∇θβ_ _ℓβo_ (x) log P _[o](x|xs) −_ log P (x|xs) _−_ log P _[o](xf_ _|xs) −_ log P (xf _|xs)_
h    i

= Edπ,η _θβ_ _ℓβo_ (x) log _[p][O][(][o][|][x][s][, x][)]_ log _[p][O][(][o][|][x][s][, x][f]_ [)]

_∇_ _η(o_ _xs)_ _−_ _η(o_ _xs)_
  _|_ _|_ []

= Edπ,η _θβ_ _ℓβo_ (x) log p (o _xs, x)_ log p (o _xs, xf_ )
_∇_ _O_ _|_ _−_ _O_ _|_
h  i


B IMPLEMENTATION DETAILS

**Clipped β loss** Common PPO implementation updates πθ multiple times. However, our preliminary
experiments observed that performing multiple updates for β[o] led to destructively large updates and


-----

**Algorithm 1 InfoMax Termination Critic with VIC, PPO Style**

1: Given: Initial option-value Q, option-policy π[o], and termination function β[o].
_O_
2: Let B be a replay buffer for storing option-transitions.
_O_
3: for k = 1, ... do
4: **for i = 1, 2, ..., N do** _▷_ Collect experiences from environment

5: Sample termination variable bi from β[o][i] (xi)

6: **if bi = 1 then**

7: Store (xs, xf _, oi), (xs+1, xf_ _, oi), ..., (xs+h, xf_ _, oi) to the replay buffer BO_

8: **end if**

9: Choose next option oi+1 by ϵ-Greedy

11:10: **end forReceive reward Ri and state xi+1, taking ai ∼** _πoi+1_ (xi)

12: **for k = 1, 2, ..., Num. of PPO epochs do** _▷_ Optimize π[o], Q, and β[o]
_O_

13: **for all xi in the trajectory do**

14: Compute RVIC from the target network

15: Compute _A[ˆ][o]ind_ [by (][8][) using][ R][VIC]

16: Update π[o](ai _xi) via PPO using_ _A[ˆ][o]_
_|_

17: Update QO(xi, o) to regress to _A[ˆ][o]_

18: **if oi has already terminated then**

19: Update β[o](xi) via (7)

20: **end if**

21: **end for**

22: **end for**

23: Train ˆp and ˆµ by option-transitions sampled from B
_O_

24: **if k mod KVIC then**

25: Update the target network for VIC

26: **end if**

27: end for


resulted in the saturation of β[o] to zero or one. Hence, to perform PPO-style multiple updates, we
introduce a clipped objective of eq. (7):


_L[CLIP](θβ) = clip(ℓβo_ (x) − _ℓβo_ old (x), −ϵβ, ϵβ)β[o]old[(][x][)] log pO(o|xs, x) − log pO(o|xs, xf )



(14)


where ϵβ is a small coefficient, and β[o]old [is an old][ β][o][ before updating. We also add maximization of]
the entropy of β[o] for preventing the termination probability saturating on zero or one. To this end,
we maximize L[CLIP](θβ) + cHβ _H(β[o](x)) via gradient ascent w.r.t. θβ, where cHβ is a weight of the_
entropy bonus.

**Full description of the algorithm** Algorithm 1 shows a full description of our implementation
of IMTC on PPO when combined with VIC rewards. As of the original PPO, it is built on the
A2C-style (Mnih et al., 2016; Wu et al., 2017) architecture with multiple synchronous actors and
a single learner. First, we collect N -step experiences interacting with environments. At line 7, we
append tuples corresponding to option transitions (xs, xf _, oi), ..., (xs+h, xf_ _, oi) to BO. Here, we do_
not use all transitions and store first hmax options to prevent memory shortage. We used hmax = 10 or
_hmax = 20. Then we update π[o], Q_, and β[o]. Line 13 is done via minibatch sampling in the actual
_O_
implementation. We also update ˆp for estimating the gradient (7), sampling from the replay buffer
_O_
_B_ . We empirically found that rapidly changing RVIC leads to unstable learning, especially when
_O_
IMTC is used in parallel. Thus, we employ a target network to compute RVIC and periodically update
it at line 26. We used KVIC = 20 in the experiments.

**Our implementation of VIC and RVIC** In experiments, we show that IMTC helps an agent
learn diverse options without reward signals. For this purpose, we employ VIC (Gregor et al.,
2016) and RVIC (Baumli et al., 2021) for providing intrinsic rewards. Here we explain our
VIC and RVIC implementation. VIC updates intra-option policies to maximize the lower
bound of MI (3) H(O|xs) − _H(O|xs, Xf_ ) ≥ _H(O|xs) + Eo,xf [log q(o|xs, xf_ )] as rewards,


-----

Figure 6: Neural Network architecture used for the Gridworld experiments (top) and continuous
control tasks (bottom).

where q can be any distribution and called an option inference model. We use ˆp as q. We
_O_
also learn ˆη(o _xs) from sampled option transitions and approximate H(O_ _xs) by H(O_ _xs) =_
_|_ _|_ _|_
_−_ [P]o∈O _[η][(][o][|][x][s][) log][ η][(][o][|][x][s][) =][ −][E][ [log][ η][(][o][|][x][s][)]][ ≈]_ [log ˆ]η(o|xs). To this end, we giving an agent

_RVIC = cVIC (log ˆp_ (o _xs, xf_ ) log ˆη(o _xs)) as an immediate reward when an option o terminates,_
_O_ _|_ _−_ _|_
where cVIC is a scaling coefficient. This is different from the original VIC implementation where
_H(O_ _xs) is treated as a constant. However, we empirically found this formulation helps diversity_
_|_
options in our preliminary experiments. In case of RVIC, we add a another classifier ˆq(o _xf_ ), and
_|_
constructed rewards by RRVIC = cVIC (log ˆp (o _xs, xf_ ) log ˆq(o _xf_ )) ˆq is learned in the same
_O_ _|_ _−_ _|_
fashion as ˆp and ˆη using the replay buffer B .
_O_ _O_

C EXPERIMENTAL DETAILS

Our anonymized code used for all our experiments is on [https://](https://anonymous.4open.science/r/imtc-anonymized-code-E5D1/)
[anonymous.4open.science/r/imtc-anonymized-code-E5D1/.](https://anonymous.4open.science/r/imtc-anonymized-code-E5D1/)

C.1 NETWORK ARCHITECTURE

Figure 6 illustrates the neural network architecture used in our experiments. We used a shared
convolutional encoder for Gridworld experiments, and two separated fully connected layers with
64 units for continuous control experiments. π[o] is parameterized as a Gaussian distribution with
separated networks for standard derivations per option, similar to Schulman et al. (2015a). We
used ReLU as an activator for all hidden layers and initialized networks by the orthogonal (Saxe
et al., 2014) initialization in all experiments. Note that the last layer for intra-option policy π[o] was
initialized with small values, following the standard practice (Andrychowicz et al., 2021). We used
the Adam (Kingma and Ba, 2015) optimizer in all experiments. Unless otherwise noted, we used the
default parameters in PyTorch (Paszke et al., 2019) 1.8.1.


-----

|Description|Value|
|---|---|
|γ|0.99|
|Adam Learning Rate|3 10−4 ×|
|Adam ϵ|1 10−4 ×|
|Clip parameter for ℓ (ϵ ) βo β|0.05|
|Num. timesteps per rollout|256|
|Num. actors|16|
|GAE λ|0.95|
|Num. epochs for PPO|10|
|Minibatch size for PPO|1024|
|Weight of H(πo) (c ) H|0.001|
|Weight of H(βo) (c ) Hβ|0.01|
|Gradient clipping|0.5|
|Capacity of B O|8192|
|Max num. option transitions to store (h ) max|20|
|Num. epochs for training pˆ and µˆ|4|
|Minibatch size for training pˆ and µˆ|2048|
|Scaling of R (c VIC) VIC|0.005|
|Synchronizing interval of the target network for VIC (K ) VIC|20|


Table 1: Used hyperparameters

C.2 HYPERPARAMETERS

Table 1 shows all hyperparameters used in IMTC + VIC experiments on MuJoCo continuous control
tasks.

C.3 ENVIRONMENT IMPLEMENTATION

Gridworld is based on RLPy (Geramifard et al. (2015), BSD3 License). We constructed continuous
control environments on the MuJoCo (commercial license, Todorov et al. (2012)), using OpenAI
Gym (MIT license, Brockman et al. (2016)). Especially, point environments are implemented based
on “PointMaze” in rllab (MIT license, Duan et al. (2016)) with some modifications, mainly around
collision detection. We also refered to the modified PointMaze code[3] (Apache 2.0 license) relased by
Nachum et al. (2018). The swimmer robot is originally used in Coulom (2002).

C.4 COMPUTATIONAL RESOURCES

All experiments are conducted on a private cluster with NVIDIA P100 GPUs. On the cluster, training
IMTC with VIC on MuJoCo PointMaze domain for 4 × 10[6] steps takes about 27 minutes.

C.5 COMPARISON OF VIC AND RVIC REWARDS

Figure 7 shows the

C.6 EFFECTIVITY OF UGAE

To investigate the effect of UGAE (9), we compared IMTC and OC with and without UGAE in
task adaptation experiments. IMTC (no UGAE) and OC (no UGAE) estimate the adavantage
ignoring the future rewards after switching options, similar to advantage estimation proposed by
Bacon et al. (2017). Figure 8 shows the result. We can see that UGAE improves the performance in
all domains for both IMTC and OC.


-----

Comparison of Intrinsic Rewards Comparison of log p(o|xs, xf)

1

|Col1|Col2|
|---|---|
|||
|||
|||
|||
|Method||
|IMTC + VIC IMTC + RVIC||
|VIC ( =0.1)||
|RVIC ( =0.1)||


Method

IMTC + VIC
IMTC + RVIC
VIC ( =0.1)
RVIC ( =0.1)


0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6

Comparison of Intrinsic Rewards

0.000

0.005

0.010

Method

0.015 IMTC + VIC

IMTC + RVIC

0.020 VIC ( =0.1)

RVIC ( =0.1)

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Environmental Steps (10[6]) 1e6

Figure 7: **Left:** Comparison of intrinsic rewards during training. **Right:** Comparison of
log ˆpO(o|xs, x) during training.


PointReach SwimmerReach AntReach PointBilliard

1.0 0.5 0.7 0.9

0.8 0.4 0.6 0.8

0.5

0.6 0.3 0.4 0.7

Return Return 0.2 Return 0.3 Return0.6

0.4

0.1 0.2

0.5

0.2 0.0 0.1

0.0 0.4

0.0 0.1 0.1

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6

AntBilliard PointPush AntPush

0.30 1.0 0.5

0.25

0.8 0.4

0.20

0.15 0.6 0.3

Return 0.10 Return Return 0.2

0.05 0.4 0.1 IMTCMethod

0.00 0.2 0.0 IMTC (no UGAE)

0.05 OC

0.0 0.1 OC (no UGAE)

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6


Figure 8: Learning curves for transfer learning experiments with

C.7 NUMBER OF OPTIONS


Figure 9 shows learning curves of IMTC agents with variable number of options (2, 4, 6, 8, 10, 12) in
task adaptation experiments. Increasing the number of options sometimes improves the performance
(e.g., in SwimmerReach), but the relationship is not obvious. However, we can confirm that
increasing the number of options does not have a bad effect on the performance, while it reduces
training steps per each intra-option policy, thanks to network sharing Appendix C.1.

C.8 OPTIONS LEARNED IN FOUR ROOMS


Figure 10 show the options learned in the classifical Four Rooms Gridworld (Sutton et al., 1999).

[3https://github.com/tensorflow/models/tree/v2.3.0/research/efficient-hrl](https://github.com/tensorflow/models/tree/v2.3.0/research/efficient-hrl)


-----

PointReach SwimmerReach AntReach PointBilliard

1.0 0.8 0.9

0.6

0.8 0.5 0.6 0.8

0.4 0.4 0.7

Return0.6 Return0.3 Return Return

0.6

0.2 0.2

0.4

0.1 0.5

0.0

0.2 0.0 0.4

0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6 Environmental Steps (10[6]) 1e6

AntBilliard 1.00 PointPush AntPush

0.4 0.5

0.95

0.4

0.3 0.90

0.3

Return0.2 Return0.85 Return 0.2 Num. Options

0.80 0.1 2

0.1 4

0.75 0.0 6

0.0 0.70 0.1 812

0.2Environmental Steps (100.4 0.6 [6]0.8) 1e61.0 0.2Environmental Steps (100.4 0.6 0.8[6]) 1e61.0 0.2Environmental Steps (100.4 0.6 0.8[6]) 1e61.0 16


Figure 9: Learning curves for transfer learning experiments.


IMTC + VIC 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

IMTC (R=0.01) 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

IMTC + RVIC 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

OC + VIC 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

VIC (β=0.1) 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

RVIC (β=0.1) 1.0

0.8

0.6

β

0.4

0.2

Option 0 Option 1 Option 2 Option 3 0.0

Figure 10: Options learned in four rooms environment


-----

