# DEEP INVERSE REINFORCEMENT LEARNING VIA AD## VERSARIAL ONE-CLASS CLASSIFICATION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Traditional inverse reinforcement learning (IRL) methods require a loop to find
the optimal policy for each reward update (called an inner loop), resulting in very
time-consuming reward estimation. In contrast, classification-based IRL methods, which have been studied recently, do not require an inner loop and estimate
rewards quickly, although it is difficult to prepare an appropriate baseline corresponding to the expert trajectory. In this study, we introduced adversarial oneclass classification into the classification-based IRL framework, and consequently
developed a novel IRL method that requires only expert trajectories. We experimentally verified that the developed method can achieve the same performance as
existing methods.

1 INTRODUCTION

Inverse reinforcement learning (IRL) (Russell, 1998) refers to the problem of estimating rewards for
reinforcement learning (RL) agents to acquire policies that can reproduce expert behavior. An RL
algorithm learns a policy that maximizes the cumulative discounted reward under a given reward
function. An IRL algorithm does the opposite; it estimates the reward from the given policies or
trajectories to satisfy the condition under the assumption that the expert is maximizing the reward.

IRL has been applied in two main areas (Ramachandran & Amir, 2007). The first is apprenticeship
learning, which enables the learning of complex policies for which it is difficult to design a reward
function. Compared to behavioral cloning, IRL is robust to the covariate shift problem (Ross et al.,
2011) and achieves superior performance even when the amount of data is small. The second is
reward learning, where IRL is used to estimate rewards from the trajectory data of human and animal
action sequences and to analyze the intention of the subject. In previous studies, IRL methods have
been used to analyze human walking paths (Kitani et al., 2012) and the behavior of nematodes
(Yamaguchi et al., 2018).

In traditional IRL methods, the IRL loop has an inner loop that computes the optimal policy for
the reward being estimated until convergence. This inner loop presents a difficulty in applying
IRL to tasks with a large state-action space because it is computation-intensive. As a solution to
this, classification-based IRL methods transform the IRL problem into a problem of classifying the
expert’s trajectory and the trajectory to be compared. Notable methods include AIRL (Fu et al.,
2017), LogReg-IRL (Uchibe, 2018), and T-REX (Brown et al., 2019).

These methods differ in the ways they are formulated, but they result in similar learning methods.
Online methods, such as AIRL, collect the trajectories to be compared from the environment. Contrastingly, offline methods, such as LogReg-IRL and T-REX, collect the trajectories to be compared
in advance, which enables them to further speed up and stabilize learning by not requiring access
to the environment during training. However, the learning performance of current offline methods
depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories,
which is difficult to collect.

In this study, we exploited the fact that the learning process of LogReg-IRL by binary classification
is equivalent to that of a discriminator in adversarial learning, such as with generative adversarial
networks (GANs) (Goodfellow et al., 2014). Specifically, we developed an innovative deep IRL
method, called state-only learned one-class classifier for IRL (SOLO-IRL), in which binary classification is replaced with adversarial one-class classification. Figure 1 compares the traditional


-----

Figure 1: Comparison of traditional IRL and the proposed SOLO-IRL.

and proposed IRL methods. The proposed method does not require an inner loop and is an offline
method; thus, it can be trained extremely fast. In addition, it does not require that trajectories be
compared. With these advantages, the proposed method greatly advances the application of IRL
methods to real-world problems.

2 PRELIMINARIES

2.1 MARKOV DECISION PROCESS (MDP)

RL is a learning problem based on the Markov decision process (MDP). The MDP consists of a
tuple M = ⟨S, A, P, R, γ⟩, where S is the state space, A is the action space, P is the state-transition
probability, R is the reward function, and γ is the discount factor indicating the degree of importance
for future rewards. In the MDP, the state-value function for state st at time t is represented by the
Bellman equation, as follows:


_p(s[′]_ _st, a)γV (s[′])_
_|_
_s[′]_

X


(1)


_V (st) = max_


_R(st, a) +_


where R(st, at) is the reward for taking action at in state st and p(st+1|st, at) is the probability of
transitioning to the next state st+1 when taking action at in state st.

2.2 LINEARLY SOLVABLE MDP (LMDP)

The linearly solvable MDP (LMDP) is an extension of the MDP in which the agent directly determines the transition probability u(st+1|st) from the current state st to the next state st+1 as the
control, instead of the action at in the MDP. Then, the Bellman equation is linearized under two
assumptions. First, the state-transition probability p(st+1 _st, u) is assumed to be expressed as the_
_|_
product of the uncontrolled transition probability ¯p(st+1 _st) and u as follows:_
_|_

_p (st+1_ _st, u(st+1_ _st)) = ¯p(st+1_ _st) exp_ _u(st+1_ _st)_ (2)
_|_ _|_ _|_ _{_ _|_ _}_

The uncontrolled transition probability ¯p(st+1 _st) indicates a transitional relationship between the_
_|_
states in the environment. When a transition is impossible, i.e., ¯p = 0, then p = 0.

The second assumption is that the reward R(st, u) is composed of a state-dependent reward r(st)
and penalty term DKL (p||p¯) for state-transition probability p over the divergence from the uncontrolled transition probability ¯p. This assumption can be formulated as follows:

_R (st, u(st+1|st)) = r(st) −_ _DKL (p(st+1|st, u(st+1|st))||p¯(st+1|st))_ (3)


-----

where DKL(Px||Py) represents the Kullback–Leibler (KL) divergence of Px and Py. By rearranging
Eq. (3) according to the definition of the KL divergence, the following equation is obtained:


_p(s[′]_ _st, u(s[′]_ _st))u(s[′]_ _st)_ (4)
_|_ _|_ _|_
_s[′]_

X


_R (st, u(st+1_ _st)) = r(st)_
_|_ _−_


Substituting Eq. (4) into the Bellman equation in Eq. (1) gives the following:

_V (st) = r(st) + max_ _p(s[′]_ _st, u(s[′]_ _st))_ _u(s[′]_ _st) + γV (s[′])_ (5)
_u_ _|_ _|_ _−_ _|_

_s[′]_

(X h i[)]

Eq. (2) is then substituted into Eq. (5) and the Lagrange multiplier applied with _s[′][ p][(][s][′][|][s][t][, u][) = 1]_

as a constraint. Finally, the max operator is removed, resulting in the linear Bellman equation as
follows:

[P]

exp _V (st)_ = exp _r(st)_ _p¯(s[′]_ _st) exp_ _γV (s[′])_ (6)
_{_ _}_ _{_ _}_ _|_ _{_ _}_

_s[′]_

X


The optimal control u[∗] in the LMDP is given by

_p(st+1_ _st) exp_ _γV (st+1)_
_u[∗](st+1_ _st) = [¯]_ _|_ _{_ _}_ (7)
_|_ _s[′][ ¯]p(s[′]|st) exp{γV (s[′])}_
P

2.3 LOGISTIC REGRESSION-BASED IRL (LOGREG-IRL)

LogReg-IRL (Uchibe, 2018) is a deep IRL method in the LMDP. The following is an overview of
the IRL framework in LogReg-IRL. By rearranging the linear Bellman equation in Eq. (6), the
following is obtained:


_p¯(s[′]_ _st) exp_ _γV (s[′])_ (8)
_|_ _{_ _}_
_s[′]_

X


exp _V (st)_ _r(st)_ =
_{_ _−_ _}_


Then, substituting Eq. (8) into Eq. (7) and rearranging the result gives

_p¯(st+1_ _st) exp_ _γV (st+1)_
_u[∗](st+1_ _st)_ = _|_ _{_ _}_
_|_ exp _V (st)_ _r(st)_

_{_ _−_ _}_
_u[∗](st+1_ _st)_
_|_ = exp _r(st) + γV (st+1)_ _V (st)_

_p¯(st+1_ _st)_ _{_ _−_ _}_
_|_

log _[u][∗][(][s][t][+1][|][s][t][)]_ = _r(st) + γV (st+1)_ _V (st)_ (9)

_p¯(st+1_ _st)_ _−_
_|_

Applying Bayes’ theorem to Eq. (9) we obtain


log _[u][∗][(][s][t][, s][t][+1][)]_ (10)

_p¯(st, st+1) [= log][ u]p¯[∗]([(]s[s]t[t]) [)]_ [+][ r][(][s][t][) +][ γV][ (][s][t][+1][)][ −] _[V][ (][s][t][)]_

The left-hand side and the first term on the right-hand side of Eq. (10) are the density-ratios. The
density-ratio pa/pb can be estimated by assigning the label η = 1 to the samples from the probability distribution pa, assigning η = 1 to the samples from pb, and training a classifier using
_−_
logistic regression (Qin, 1998; Cheng et al., 2004; Bickel et al., 2007). First, by Bayes’ theorem, the
following is obtained:

_pa(x)_ _p(η = 1_ _x)_ _p(η =_ 1)
= _|_ _−_
_pb(x)_ _p(η =_ 1 _x)_ _p(η = 1)_

_−_ _|_

log _[p][a][(][x][)]_ = log _[p][(][η][ = 1][|][x][)]_ (11)

_pb(x)_ _p(η =_ 1 _x) [+ log][ p]p[(][η](η[ =] = 1)[ −][1)]_

_−_ _|_

Next, the first discriminator D1(x) is defined by the sigmoid function σ(x) = 1/ 1 + exp( _x)_
_{_ _−_ _}_
and a neural network f (x):
_D1(x) = p(η = 1_ _x) = σ(f_ (x)) (12)
_|_


-----

where the second term on the right-hand side of Eq. (11) can be approximated by calculating the
sample number ratio Npa _/Npb and taking its logarithm. For the first term, the following equation_
can be obtained from the definition of the discriminator in Eq. (12):

_D1(x)_

log _[p][(][η][ = 1][|][x][)]_ = log

_p(η =_ 1 _x)_ 1 _D1(x)_
_−_ _|_ _−_

= log [1 + exp][{][f] [(][x][)][}]

1 + exp{−f (x)}
= log exp{f (x)}
= _f_ (x) (13)


From Eq. (13), when Npa = Npb, the following holds:

log _[p][a][(][x][)]_ (14)

_pb(x) [=][ f]_ [(][x][)]


Therefore, the density-ratio of the first term in Eq. (10) can be estimated by sampling the states
_sbaseline trajectory[∗]t_ _[∼]_ _[τ][ ∗]_ [and][ ¯]st ∼ ¯τ according to the uncontrolled transition probabilityτ¯ from the expert trajectory τ _[∗]_ according to the optimal control ¯p, followed by training u[∗] and the
with the following cross-entropy loss:
_L1(D1) =_ Es¯t _τ¯[[log(1][ −]_ _[D]1[(¯]st))]_ Es[∗]t _t_ [))]] (15)
_−_ _∼_ _−_ _[∼][τ][ ∗]_ [[log(][D][1][(][s][∗]

The density-ratio on the left-hand side of Eq. (10) is defined as follows using the trained f (x),
reward-estimating neural network ˜r(x), and state-value-estimating neural network _V[˜] (x):_

log _[u][∗][(][s][t][, s][t][+1][)]_ _r(st) + γV[˜] (st+1)_ _V (st)_ (16)

_p¯(st, st+1) [=][ f]_ [(][s][t][) + ˜] _−_ [˜]

The second discriminator D2 for the state-transition pair is defined as

_D2(x, y) = σ(f_ (x) + ˜r(x) + γV[˜] (y) _V (x))_ (17)
_−_ [˜]
As with D1, the discriminator D2 is trained by cross-entropy loss L2, given as
_L2(D2) = −E(¯st,s¯t+1)∼τ¯[[log(1][ −]_ _[D]2[(¯]st, ¯st+1))] −_ E(s[∗]t _[,s]t[∗]+1[)][∼][τ][ ∗]_ [[log(][D][2][(][s]t[∗][, s]t[∗]+1[))]] (18)

In the original LogReg-IRL, an L2 regularization term is added to the loss function. Following the
process described above, LogReg-IRL estimates the reward and state-value by classifying the expert
and baseline trajectories. Unlike traditional IRL methods, LogReg-IRL does not require RL in the
reward estimation process and, thus, it can be trained very quickly.

2.4 DIFFICULTY COLLECTING BASELINE TRAJECTORIES

LogReg-IRL showed that IRL in LMDP can be formulated by learning two discriminators. However, LogReg-IRL has a problem in that its learning performance is greatly affected by the baseline
trajectory. For the baseline trajectory, it is desirable to collect data that follow uncontrolled transition
probability ¯p, such as trajectories obtained under a random policy, for a wide range of states.

However, for some tasks, the number of states that can be transitioned by a random policy may
be limited. For example, in a game task, such as an Atari game, the game does not progress,
and in a driving simulator task such as TORCS (Wymann et al., 2000), the car crashes into a wall
immediately. In such cases, data according to a random policy cannot cover a wide range of states.

Several methods have been proposed to collect the baseline trajectory in LogReg-IRL. For Atari
games, a method using state-transition pairs from random policy in any state in the expert trajectory
was proposed (Uchibe, 2018). For TORCS, a method using the trajectory recorded by driving with
noise added to the action output of the expert agent was proposed (Kishikawa & Arai, 2021). However, those proposed methods are task-specific, and there is no well-established generalized method
for collecting baseline trajectories.

An inappropriate baseline trajectory leads to inappropriate reward estimation. The density-ratios
diverge with respect to the states that the agent can reach and those for which there is no baseline
trajectory, and high rewards are estimated where there are no experts. Therefore, an agent that learns
according to the estimated reward may acquire a different action from the expert as the optimal
policy.


-----

**Generator**


Fake

expert

data


**Discriminator**


Noise

True

expert

data


_True_

_or_

_Fake_

|Col1|Col2|Col3|Col4|Col5|
|---|---|---|---|---|
||||||
||Encoder||Decoder||
||||||
||||||


Density-ratio

Reward

State-value


Figure 2: Proposed SOLO-IRL.

3 STATE-ONLY LEARNED ONE-CLASS CLASSIFIER FOR IRL (SOLO-IRL)

We propose the novel IRL method SOLO-IRL, which estimates the reward given only the expert.
SOLO-IRL is a combination of an IRL framework based on LMDP, a transition generator based on
adversarial one-class classification, and least-squares loss. Each of these is explained below.

3.1 SOLUTION BY ADVERSARIAL ONE-CLASS CLASSIFICATION

Classification-based IRL methods are equivalent to learning a discriminator in adversarial learning
frameworks, such as GANs (Goodfellow et al., 2014), which are binary classification problems. This
means that the IRL problem can be solved by learning a discriminator that classifies the trajectory
as expert or not.

Broadening our perspective to other fields, we find that the anomaly detection problem also requires
a binary classifier to distinguish between normal and abnormal samples. However, in real problems, we often encounter situations in which we can obtain many normal samples but few abnormal
samples. Therefore, one-class classification is a method for obtaining a binary classifier using only
normal samples.

Recently, the adversarially learned one-class classifier (ALOCC) (Sabokrou et al., 2018) was proposed as a one-class classification method. In the ALOCC, the discriminator is adversarially trained
with a denoising autoencoder that generates fake normal samples. Consequently, the discriminator is trained as a binary classifier that identifies normal and abnormal samples using only normal
samples.

SOLO-IRL combines a classification-based IRL method with the ALOCC. The structure of SOLOIRL is illustrated in Figure 2, and its algorithm is given as Algorithm 1. In the following, we describe
the learning process and features of SOLO-IRL as well as the objective function used for learning
SOLO-IRL.

3.2 LEARNING PROCESS OF SOLO-IRL

The training of SOLO-IRL consists of two stages. In the first stage, we learn a discriminator D1
that classifies whether a state is sampled from expert data or not. Generator G1 is composed of an
encoder–decoder network R1 and generates a fake current state ¯st from the true current state s[∗]t [plus]
noise ν1, as shown in Eq. (19). Then, the generator learns such that the generated ¯st and s[∗]t [are]
close, and D1 judges ¯st to be the true current state.

_s¯t ←R1(s[∗]t_ [+][ ν][1][)] (19)

Meanwhile, discriminator D1 is defined by Eq. (12) and outputs the probability that a given state
is sampled from the expert. D1 learns to distinguish between states sampled from a true expert and
a fake expert from a generator. Here, the density-ratio representing the expertness of each state is
obtained in D1.


-----

In the second stage, generator G2 learns to generate a state-transition pair that is close to the sampled
expert data and that discriminator D2 judges as expert. Generator G2 uses two encoder–decoder
networks R2 and R3 to generate a fake state-transition pair (¯st, ¯st+1) from the expert’s current state
_s[∗]t_ [plus noise][ ν][2][, as given by Eqs. (20) and (21). Then, the generator learns such that the generated]
(¯st, ¯st+1) and (s[∗]t _[, s]t[∗]+1[)][ are close to each other, and discriminator][ D][2]_ [judges the generated data to]
be a true state-transition pair.

_s¯t_ _←_ _R2(s[∗]t_ [+][ ν][2][)] (20)

_s¯t+1_ _←_ _R3(s[∗]t_ [+][ ν][2][)] (21)

The training of D2 is the same as that of D1 in the first stage. Here, discriminator D2 contains a
reward network ˜r and state-value network _V[˜], as shown in Eq. (17). Finally, it works as an IRL_
algorithm to estimate the reward and state-value.

By introducing a generator and training the discriminator in an adversarial manner, the decision
boundary around the expert is refined, and the appropriate reward is estimated. This makes preparation of the baseline trajectory by trial and error unnecessary. In addition, because SOLO-IRL is
an offline method, it learns quickly without executing RL or interacting with the environment. To
the best of our knowledge, SOLO-IRL is the only method that estimates the reward and state-value
exclusively from expert trajectories.

3.3 LEAST-SQUARES LOSS AS ADVERSARIAL OBJECTIVE

To train the generator and discriminator, we propose using least-squares loss as an adversarial objective. The least-squares loss is represented by the following equations:

1
_Ladv(D)_ = _x_ _d[[(][D][(˜]x))[2]]_ (22)
2 [E][x][∼][d][∗] [[(][D][(][x][)][ −] [1)][2][] + 1]2 [E][˜]∼ [¯]

1
_Ladv(G)_ = _x_ _d[[(][D][(˜]x)_ 1)[2]] (23)
2 [E][˜]∼ [¯] _−_

where d[∗] and _d[¯] denote the true and fake states or state-transition pairs, respectively. The least-_
squares loss was proposed alongside LSGAN (Mao et al., 2017), which solved the learning instability and mode collapse problems of previous GANs. In SOLO-IRL, this least-squares loss is used
instead of the cross-entropy loss to address the problems of learning stability and mode collapse.

3.4 RECONSTRUCTION OBJECTIVE

Meanwhile, the generators G1 and G2 must be trained to be close to the true expert data in addition
to training by the adversarial objective. The reconstruction objective is trained by the least-squares
loss given by the following equations, as in the original ALOCC:

_Lrec(G1)_ = _s[∗]t_ _t_ [+][ ν][1][)][||][2] (24)
_||_ _[−R][1][(][s][∗]_

_Lrec(G2)_ = _s[∗]t_ _t_ [+][ ν][2][)][||][2] [+][ ||][s][∗]t+1 _t_ [+][ ν][2][)][||][2] (25)
_||_ _[−R][2][(][s][∗]_ _[−R][3][(][s][∗]_

Finally, the objective of discriminators D1 and D2 becomes Ladv(D), and the objective of generators G1 and G2 becomes Ladv(G) + Lrec(G). Using these objectives, SOLO-IRL estimates the
reward and state-value by training each neural network using the gradient descent method.

4 EXPERIMENTAL RESULTS AND DISCUSSION

We validated the performance of the proposed method in the OpenAI Gym environment (Brockman
et al., 2016). For the expert trajectory, we used a trajectory generated by an agent that had learned
the optimal action for the true reward as an expert. As RL algorithms, we used PPO (Schulman
et al., 2017) for the tasks with a discrete action space and TD3 (Fujimoto et al., 2018) for the tasks
with a continuous action space.

In SOLO-IRL, a three-layer multilayer perceptron was used as both the encoder and decoder in the
generator, and a three-layer multilayer perceptron was used as the discriminator. Adam (Kingma &
Ba, 2014) was used to optimize the neural network.


-----

**Algorithm 1 SOLO-IRL: State-Only Learned One-class Classifier for IRL**

**Require: Expert trajectories τ** _[∗], discount factor γ, noise ν1 and ν2, numbers of iterations n1 and n2_
**Ensure: Reward network ˜r(x), state-value network** _V[˜] (x)_

1: Initialize neural network f (x), 1(x)
_R_
2: Define discriminator D1(st) = σ(f (st))
3: for i = 0, · · ·, n1 do
4: Sample expert state s[∗]t [from][ τ][ ∗]

6:5: _sTrain¯t ←R D11 according to the loss(s[∗]t_ [+][ ν][1][)] _Ladv(D) in Eq. (22)_

7: Train R1 with loss Ladv(G) + Lrec(G1) in Eqs. (23) and (24)

8: end for
9: Initialize neural network ˜r(x), _V[˜] (x),_ 2(x), 3(x)
_R_ _R_

10: Define discriminator D2(st, st+1) = σ(f (st) + ˜r(st) + γV[˜] (st+1) _V (st))_
_−_ [˜]

11: Disable f updates
12: for i = 0, · · ·, n2 do
13: Sample expert current state s[∗]t [and next state][ s]t[∗]+1 [from][ τ][ ∗]

15:16:14: _ssTrain¯¯tt ←R+1 ←R D22 according to the loss(s3[∗]t(s[+][∗]t_ _[ ν][+][2][ ν][)]_ [2][)] _Ladv(D) in Eq. (22)_

17: Train R2 and R3 according to the loss Ladv(G) + Lrec(G2) in Eqs. (23) and (25)

18: end for


Table 1: Validation results for the OpenAI Gym tasks. These scores are cumulative true rewards
(averaged over 1000 trials); the higher the better.

CartPole BipedalWalker Hopper Walker2d

Expert 499.96 321.69 3682.28 5272.42
Random 22.50 _−99.61_ 17.87 1.87

LogReg-IRL (Uchibe, 2018) 498.42 _−118.85_ 5.30 376.44
SOLO-IRL (ours) 449.11 226.16 1084.46 774.87

Behavioral cloning 500.00 135.13 3677.99 4920.64
RED (Wang et al., 2019) 3633.72 3868.98

We compared SOLO-IRL to LogReg-IRL (Uchibe, 2018). The baseline trajectory in LogReg-IRL
was collected by running a random policy based on a uniform distribution in the environment. The
LogReg-IRL implementation was created based on the SOLO-IRL implementation with the following changes: removal of the generator, changing of the cross-entropy loss, and addition of a weight
decay term (coefficient: 1e-3) to the loss.

We used the following four tasks for validation.

**CartPole. CartPole (Gym, b) is a basic RL task in which the agent must keep a pendulum connected**
to a cart upright by controlling the cart. A survival reward is given to the pendulum as long as it
remains upright.

**BipedalWalker. BipedalWalker (Gym, a) is a task in which a robot with four joints learns a bipedal**
walking task. The agent is required to output the velocities of its two hips and knees as actions,
taking the state given by the virtual sensor as the input. The true reward comprises a bonus based on
the speed and a penalty based on the magnitude of the action or fall.

**MuJoCo. MuJoCo (Todorov et al., 2012) is an environment that collects tasks for the development**
of robotics. We selected Hopper and Walker2d as examples. Hopper is a task in which a one-legged
robot moves forward, and Walker2d is one in which a two-legged robot walks. These two tasks
require more complex continuous control than BipedalWalker.


-----

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75


0.120

0.125

0.130

0.135

0.140

0.145


1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

|0 5 0 5 0 5 0 5 0|Col2|
|---|---|


1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00


Velocity in X-axis direction


0.4

0.6

0.8

1.0

1.2

1.4

1.6


|0 5 0 5 0 5 0 5 0|Col2|
|---|---|


1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00

Velocity in X-axis direction

Figure 4: Visualization of the
reward estimated by LogRegIRL.


Figure 3: Visualization of the
distribution of expert trajectories (blue) and baseline trajectories (black).


Figure 5: Visualization of the
reward estimated by SOLOIRL.


**Analysis of results. First, in each environment, we evaluated the policy of the expert agent, the**
random policy, the policy of the agent trained according to the reward estimated by LogReg-IRL
and SOLO-IRL, and the policy obtained via behavioral cloning. We also tested the random expert
distillation (RED) (Wang et al., 2019), which is a recent method of imitation learning in the MuJoCo
environment[1].

There are two metrics for evaluating the performance of IRL: expected value difference (EVD)
(Levine et al., 2011) and expected cumulative reward. To calculate the EVD, we used the discount
rate. However, since the appropriate discount rate varies depending on the environment, the value
we use will also vary accordingly. Therefore, we decided to evaluate the performance using the
expected cumulative reward, which is a more common of the two.

The results are presented in Table 1. The results for CartPole show that SOLO-IRL was able to
achieve comparable with that of LogReg-IRL. Furthermore, with regard to the results for BipedalWalker, we saw that LogReg-IRL failed to learn, whereas SOLO-IRL obtained a good score.

The aforementioned results can be attributed to the fact that BipedalWalker has a vast state space,
whereas CartPole has a relatively small state space. Consequently, the random policy could only
collect data near the starting point and could not provide an appropriate baseline trajectory for the
expert trajectory that progressed to the goal.

Behavioral cloning and RED generally perform better than the other methods that were used in the
comparison. However, in BipedalWalker, the proposed method showed better performance than
behavioral cloning due to the changing environment. In addition, RED requires more information
than the proposed method because of the availability of RL. Therefore, our proposed method can be
said to be superior than the traditional methods in that it can learn purely using expert trajectories.

**Relationship between trajectory distribution and estimated reward in the BipedalWalker task.**
Among the 24 dimensions in the BipedalWalker state, Figure 3 visualizes the distribution of the
expert and baseline trajectories for “velocity in the X-axis direction” and “angle of hip 1,” and
Figures 4 and 5 visualize the estimated reward for these two dimensions at the starting point. As can
be seen, the baseline trajectory does not cover a part of the the expert trajectory. Owing to the lack
of a baseline trajectory, LogReg-IRL’s estimation failed, yielding a meaningless reward. In contrast,
SOLO-IRL’s estimation, which was trained via generating samples near the expert, resulted in an
appropriate reward that drove the agent forward.

**Behavior of adversarial and reconstruction objectives. Figures 6 and 7 illustrate the behavior**
of adversarial loss and reconstruction loss during training in SOLO-IRL. With adversarial learning,
the probabilities of truthfulness of the true and fake samples converge to an equilibrium near 0.5.
The discriminator narrows the decision boundary toward the expert neighborhood through learning,
and the generator eventually succeeds in producing samples that are close to the true. The learning
success of the generator is also evident from the convergence of the reconstruction loss.

**Summary of the experimental results. There is room for improving the performance of LogReg-**
IRL using complex rule-based policies or by combining multiple policies to collect baseline tra
1Since the author’s implementation of RED only supported MuJoCo, the experiments were also conducted
only on the MuJoCo task.


-----

Discriminator

0.53 Generator

0.52

Probability 0.51

0.50

0.49

0.48

0 200 400 600 800 1000

Training epoch


10 1

10 2

Reconstruction loss

0 200 400 600 800 1000

Training epoch


Figure 6: Changes in the probability
that the true sample (Discriminator) and
fake sample (Generator) are true, respectively, during training in the first
stage.


Figure 7: Reconstruction loss during
training in the first stage.


jectories; however, this is expected to be significantly more difficult than adjusting the noise in
SOLO-IRL. Therefore, it can be said that similar or better performance than LogReg-IRL can be
achieved more easily using SOLO-IRL.

5 RELATED WORKS

**IRL. A method similar to the proposed method is the AIRL, which is an extension of maximum**
entropy IRL (Ziebart et al., 2008), guided cost learning (Finn et al., 2016), and generative adversarial
imitation learning (GAIL) (Ho & Ermon, 2016). The theoretical background of the aforementioned
methods is related to the proposed method in terms of entropy regularization (Chow et al., 2018).
AIRL requires access to the environment, while the proposed method does not. In addition, DREX (Brown et al., 2020) is similar to the proposed method in that it creates the trajectory of the
comparison target by adding noise. Since the proposed method uses adversarial learning, it can
generate more appropriate samples for comparison and does not need to learn behavioral cloning.

**Imitation learning. There are several methods that have been recently proposed for imitating expert**
demonstrations, such as RED, disagreement-regularized imitation learning (Brantley et al., 2019),
and O-NAIL (Arenz & Neumann, 2020). All these methods are extensions of GAIL and require
information regarding the expert’s state-action pairs. The proposed method can be used to estimate
the reward from the trajectory comprising only states.

**Behavioral cloning. The simplest offline learning method to imitate expert trajectory data is behav-**
ioral cloning, and (Torabi et al., 2018) is such a method that can be applied to state-only trajectories.
However, compared to IRL methods, it is difficult to deal with the issues of policy transferability
and covariate shift using such a method.

6 CONCLUSION

In this study, we exploited the fact that the classification-based IRL framework is equivalent to
training a discriminator in adversarial learning and developed SOLO-IRL, in which a generator is
incorporated to generate fake expert data. SOLO-IRL can be trained quickly without the need for an
inner loop and easily estimates rewards with high performance exclusively from expert trajectories,
with no need for baseline trajectories.

However, although it is simpler than collecting baseline trajectories, the noise used for reconstruction
by the generator still requires adjustment. In future work, we will develop a method that does not
require noise adjustment. We will also consider application to more advanced image-based control
tasks and the incorporation of recent advances in GANs.


-----

REPRODUCIBILITY STATEMENT

In Appendices, we have described the details of the hyperparameters and noise needed to reproduce
our experiments. We also attached the source code used in our experiments.

REFERENCES

Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. arXiv preprint arXiv:2008.03525, 2020.

Steffen Bickel, Michael Br¨uckner, and Tobias Scheffer. Discriminative learning for differing training
and test distributions. In Proceedings of the 24th international conference on Machine learning,
pp. 81–88, 2007.

Kiant´e Brantley, Wen Sun, and Mikael Henaff. Disagreement-regularized imitation learning. In
_International Conference on Learning Representations, 2019._

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International
_Conference on Machine Learning, pp. 783–792. PMLR, 2019._

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning
via automatically-ranked demonstrations. In Conference on robot learning, pp. 330–359. PMLR,
2020.

Kuang Fu Cheng, Chih-Kang Chu, et al. Semiparametric density estimation under a two-sample
density ratio model. Bernoulli, 10(4):583–604, 2004.

Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in tsallis entropy regularized mdps. In International Conference on Machine Learning, pp. 979–988. PMLR,
2018.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control
via policy optimization. In International conference on machine learning, pp. 49–58. PMLR,
2016.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International Conference on Machine Learning, pp. 1587–1596. PMLR, 2018.

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint
_arXiv:1406.2661, 2014._

OpenAI Gym. BipedalWalker-v2. [https://gym.openai.com/envs/](https://gym.openai.com/envs/BipedalWalker-v2/)
[BipedalWalker-v2/, a.](https://gym.openai.com/envs/BipedalWalker-v2/)

[OpenAI Gym. CartPole-v1. https://gym.openai.com/envs/CartPole-v1/, b.](https://gym.openai.com/envs/CartPole-v1/)

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
_information processing systems, 29:4565–4573, 2016._

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
_arXiv:1412.6980, 2014._

Daiko Kishikawa and Sachiyo Arai. Estimation of personal driving style via deep inverse reinforcement learning. Artificial Life and Robotics, pp. 1–9, 2021.


-----

Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting.
In European Conference on Computer Vision, pp. 201–214. Springer, 2012.

Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with
gaussian processes. Advances in neural information processing systems, 24:19–27, 2011.

Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.
Least squares generative adversarial networks. In Proceedings of the IEEE international confer_ence on computer vision, pp. 2794–2802, 2017._

Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models.
_Biometrika, 85(3):619–630, 1998._

Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp. 2586–2591, 2007.

St´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international con_ference on artificial intelligence and statistics, pp. 627–635. JMLR Workshop and Conference_
Proceedings, 2011.

Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual
_conference on Computational learning theory, pp. 101–103, 1998._

Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially
learned one-class classifier for novelty detection. In Proceedings of the IEEE Conference on
_Computer Vision and Pattern Recognition, pp. 3379–3388, 2018._

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033.
IEEE, 2012.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint
_arXiv:1805.01954, 2018._

Eiji Uchibe. Model-free deep inverse reinforcement learning by logistic regression. Neural Process_ing Letters, 47(3):891–905, 2018._

Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In International Conference on
_Machine Learning, pp. 6536–6544. PMLR, 2019._

Bernhard Wymann, Eric Espi´e, Christophe Guionneau, Christos Dimitrakakis, R´emi Coulom, and
Andrew Sumner. Torcs, the open racing car simulator. Software available at http://torcs. source_forge. net, 4(6):2, 2000._

Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Ikue Mori, and
Shin Ishii. Identification of animal behavioral strategies by inverse reinforcement learning. PLoS
_computational biology, 14(5):e1006122, 2018._

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse
reinforcement learning. In Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.


-----

A HYPERPARAMETERS OF RL

This section describes the main hyperparameters used in the experiments. For RL, we used the same
settings for both the training of the expert agent and the training based on IRL results.

A.1 HYPERPARAMETERS OF TD3 AGENT

[The implementation of TD3 is based on https://github.com/chainer/chainerrl/](https://github.com/chainer/chainerrl/blob/master/examples/mujoco/reproduction/td3/train_td3.py)
[blob/master/examples/mujoco/reproduction/td3/train_td3.py,](https://github.com/chainer/chainerrl/blob/master/examples/mujoco/reproduction/td3/train_td3.py) which is
identical except for the environment name and the values listed in Table 2.

Table 2: Some of the hyperparameters employed in training the TD3 agent.

Hyperparameter Value

Training steps 10[6]

Initial exploration sample size 10[4]

Replay buffer size 10[6]

Batch size 10[3]

Discount rate 0.99

A.2 HYPERPARAMETERS OF PPO AGENT

[The implementation of PPO is based on https://github.com/chainer/chainerrl/](https://github.com/chainer/chainerrl/blob/master/examples/mujoco/train_ppo_gym.py)
[blob/master/examples/mujoco/train_ppo_gym.py, which is identical except for the](https://github.com/chainer/chainerrl/blob/master/examples/mujoco/train_ppo_gym.py)
environment name and the values listed in Table 3.

Table 3: Some of the hyperparameters employed in training the PPO agent.

Hyperparameter Value

Training steps 10[6]

Update interval 2048
Batch size 64
Entropy coefficient 0.001
Discount rate 0.99


-----

B IRL HYPERPARAMETERS

The hyperparameters employed in training SOLO-IRL and LogReg-IRL are shown in Table 4.

Table 4: Hyperparameters employed in training IRL.

Hyperparameter Value

Learning rate for network f 0.00004
Learning rate for network ˜r 0.00004
Learning rate for network _V[˜]_ 0.00004
Learning rate for network 1 0.0001
_R_
Learning rate for network 2 0.0001
_R_
Learning rate for network 3 0.0001
_R_
Number of inputs and outputs (|S|, 1024), (1024, 1024),
for each network layer (1024, 1024), (1024, 1)
Probability of dropout
0.0, 0.7, 0.7, 0.0
in each layer of discriminator

Probability of dropout
0.0, 0.0, 0.0, 0.0
in each layer of generator

leaky ReLU, leaky ReLU,
Activation function in each layer
leaky ReLU, None
Number of training epochs
1000
in the first stage

Number of training epochs
1000
in the second stage

Number of steps in one epoch
100
in the first stage

Number of steps in one epoch
100
in the second stage

Batch size 1024
Discount rate 0.99


-----

B.1 ADDITIVE NOISE

For CartPole, we added noise that follows a normal distribution N (0, 0.001) to the true state. For
the MuJoCo task, we added a normal distribution N (0, 0.00001) noise to Hopper and a normal
distribution N (0, 0.1) noise to Walker2d.

For each dimension of BipedalWalker, we added either noise based on the normal distribution or
noise for the labels. The tuning results for the experiment are shown in Table 5. The “Types of
noise” column in the table lists the noise addition operations described below, and the “Parameters”
column lists the noise parameters.

-  “Normal” _sadd_ _s[∗]_ + ν, ν _N_ (µ, σ[2])
_· · ·_ _←_ _∼_

-  “Label” _sadd_ min(max(s[∗] + ν, 0), 1), ν is randomly sampled from [ 1, 0, 1] with
probability · · · [pa, p ←b, pc] _−_

Table 5: Additive noise types and parameters for each dimension.

Dimension
Types of noise Parameters
in state

1 Normal _N_ (0, 0.01)
2 Normal _N_ (0, 0.01)
3 Normal _N_ (0, 0.01)
4 Normal _N_ (0, 0.01)
5 Normal _N_ (0, 0.0001)
6 Normal _N_ (0, 0.0001)
7 Normal _N_ (0, 0.01)
8 Normal _N_ (0, 0.01)
9 Label [0.1, 0.8, 0.1]
10 Normal _N_ (0, 0.0001)
11 Normal _N_ (0, 0.0001)
12 Normal _N_ (0, 0.01)
13 Normal _N_ (0, 0.01)
14 Label [0.1, 0.8, 0.1]
15 Normal _N_ (0, 0.01)
16 Normal _N_ (0, 0.01)
17 Normal _N_ (0, 0.01)
18 Normal _N_ (0, 0.01)
19 Normal _N_ (0, 0.01)
20 Normal _N_ (0, 0.01)
21 Normal _N_ (0, 0.01)
22 Normal _N_ (0, 0.01)
23 Normal _N_ (0, 0.01)
24 Normal _N_ (0, 0.01)


-----

B.2 SENSITIVITY TO NOISE SCALE

To analyze the sensitivity of the learning results to the noise scale, we conducted experiments in
which we changed the standard deviation σ of the normal distribution N (0, σ[2]) used to generate the
noise. Walker2d in the MuJoCo environment was used for the experiments.

The results are shown in Figure 8. It can be seen that the noise should have a certain level of
magnitude and should not be too small or too large. The appropriate noise level is considered to be
task-dependent.

1000

500

0

Score

500

1000

10 6 10 4 10 2 10[0]

Standard deviation of noise


Figure 8: Relationship between noise scale and score in the Walker2d task. The X-axis represents
the standard deviation of the normal distribution (logarithmic scale), and the Y-axis represents the
RL score (sum of true rewards) based on IRL results. The bars in the graph indicate the standard
deviation of the scores.

B.3 DETAILS OF BEHAVIORAL CLONING

There are various implementation methods for behavioral cloning. We used supervised learning
as the simplest behavioral cloning method; specifically, recording the expert’s state and action sequences and using the state as input and the action as output.

More specifically, we trained a multilayer perceptron f (x; θ) for tasks with a discrete action space
using the softmax cross-entropy function shown in Eq. (26) as the loss. The number of inputs in the
network was |S|, the number of outputs was equal to that of action options. The other settings of the
network were the same as those of IRL.

_L(θ) =_ E(st,at) [at log softmax (f (st; θ))] (26)
_−_ _∼D_

In the evaluation, the argmax policy in Eq. (27) was used:

_at = argmaxaf_ (st; θ) (27)

For tasks with a continuous action space, the multilayer perceptron was trained by using the squared
loss shown in Eq. (28). The number of inputs in the network was |S|, and the number of outputs
was |A|.

_L(θ) = E(st,at)_ (at _f_ (st; θ))[2][i] (28)
_∼D_ _−_

In the evaluation, the output of the network f (sth; θ) was used. Note that the output values were
clipped in the defined range of the action in the environment.

B.4 DETAILS OF RED

[To measure the RED scores, we used the author’s implementation https://github.com/](https://github.com/RuohanW/RED)
[RuohanW/RED. We used the same hyperparameters but changed the number of trajectories to](https://github.com/RuohanW/RED)
1000.


-----

