# A RISK-SENSITIVE POLICY GRADIENT METHOD

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Standard deep reinforcement learning (DRL) agents aim to maximize expected
reward, considering collected experiences equally in formulating a policy. This
differs from human decision-making, where gains and losses are valued differently
and outlying outcomes are given increased consideration. It also wastes an opportunity for the agent to modulate behavior based on distributional context. Several
approaches to distributional DRL have been investigated, with one popular strategy
being to evaluate the projected distribution of returns for possible actions. We
propose a more direct approach, whereby the distribution of full-episode outcomes
is optimized to maximize a chosen function of its cumulative distribution function (CDF). This technique allows for outcomes to be weighed based on relative
quality, does not require modification of the reward function to modulate agent
behavior, and may be used for both continuous and discrete action spaces. We show
how to achieve an asymptotically consistent estimate of the policy gradient for
a broad class of CDF-based objectives via sampling, subsequently incorporating
variance reduction measures to facilitate effective on-policy learning. We use the
resulting algorithm to train agents with different “risk profiles” in penalty-based
formulations of six OpenAI Safety Gym environments, observing that moderate
emphasis on improvement in training scenarios where the agent performs poorly
both increases the accumulation of positive rewards and decreases the frequency
of incurred penalties. We found that, in all environments tested, the same risk
profile can be used to produce both stronger overall performance than standard
Proximal Policy Optimization (PPO) and higher levels of positive reward than PPO
constrained by Lagrangians to maintain the same cost levels.

1 INTRODUCTION

While deep reinforcement learning (DRL) has been used to master an impressive array of simulated
tasks in controlled settings, it has not yet been widely adopted for high-stakes, real-world applications.
One reason for this gap is the lack of distributional perspective in standard artificial agents. Endowing
agents with such perspective could make their decision-making more robust, potentially leading to
increased safety, increased trust from humans, and more widespread real-world adoption.

In reinforcement learning (RL), risk arises due to uncertainty around the possible outcomes of an
agent’s future actions. It is a result of randomness in the operating environment, mismatch between
training and test conditions, and the inherent randomness of a stochastic policy. Risk-sensitive policies,
or those that consider more than a mean over the distribution of possible outcomes, offer the potential
for added robustness under uncertain and dynamic conditions. There is an evolving landscape of
algorithmic paradigms for handling risk in RL, from constraint-based approaches adapted from
optimal control (Achiam et al., 2017; Chow et al., 2019; Ray et al., 2019; Zhong et al., 2020) to
adversarial approaches emerging from AI Safety (Garc´ıa & Fernandez, 2015; Amodei et al., 2016).´
Within this landscape, learning approaches that optimize distributional measures offer the ability to
express design preferences over the full distribution of potential outcomes, through the specification
of a risk-sensitivity criterion.

Distributional RL has been studied for value-based methods, with a popular strategy being to use the
distributional Bellman equation to estimate the distribution of Q-values for each member of a discrete
set of potential actions (Bellemare et al., 2017; Dabney et al., 2018a;b). However, distributional RL
has not been widely explored for policy gradient methods, which could permit direct optimization of
risk-sensitive measures and naturally accommodate both discrete and continuous action spaces.


-----

In the following, we introduce a novel framework for risk-sensitive learning using policy gradients.
Our approach allows agents to be trained with different risk profiles through design-time specification
of both utility and weight functions, with the latter being defined over the estimated distribution of
full-episode rewards. This framework enables agent-based learning that captures aspects of human
decision-making, such as overemphasis of rare occurrences and diminishing marginal utility relative
to a reference outcome (Kahneman & Tversky, 1979). It also allows implementation of another
key strategy of human learning: emphasizing improvement on tasks where one is deficient. We
demonstrate the ability of our algorithm to use this strategy to improve performance relative to both
unconstrained and constrained methods in six OpenAI Safety Gym environments (Ray et al., 2019).

2 RELATED WORK

Constrained RL offers a set of approaches to safe exploration (Garc´ıa & Fernandez (2015)) that aim´
to enforce explicit constraints throughout the learning process via methods including Lagrangian
constraints (Ray et al. (2019)) and constraint coefficients (Achiam et al. (2017)). Differing from the
safe exploration scenario, we here consider problems with distinct training and test phases where
agent performance is to be evaluated. Our experiments indicate that risk-sensitive learning can offer
performance improvements over constrained learning in scenarios where safety constraints need not
be enforced during training.

Distributional approaches to risk-sensitive RL have primarily been explored in the value-based
setting. Therein, the value distribution has been explicitly modeled through categorical techniques
(Bellemare et al., 2017) or quantile regression (Dabney et al., 2018a) and used to improve both value
predictions and overall performance. Recent works utilize distributional modeling in the actor-critic
setting to enable application to continuous action spaces, again demonstrating improved performance
over baseline approaches (Ma et al., 2020; Zhang et al., 2021; Duan et al., 2021). In value-based
approaches, risk-sensitivity criteria are applied at run time as a nonlinear warping of the estimated
value distribution.

Policy gradient approaches offer additional promise for risk-sensitive RL, but require direct optimization of a parameterized policy with respect to a distributional objective. Some existing methods are
limited to a specific class of learning objective, such as the set of concave risk measures that allow
a globally-optimal solution (Tamar et al., 2015; Zhong et al., 2020). Others allow a broader class
of measures but are more restrictive in the class of policies that can be represented (Prashanth et al.,
2016). We aim for a risk-sensitive policy gradient approach that both offers significant flexibility in
the choice of learning objective and can learn policies parameterized by a deep neural network.

Various measures have been considered in the context of risk-sensitive RL, including exponential
utility (Pratt, 1964), percentile performance criteria (Wu & Lin, 1999), value-at-risk (Leavens, 1945),
conditional value-at-risk (Rockafellar & Uryasev, 2000), and prospect theory (Kahneman & Tversky,
1979). In this work, we consider a class of risk-sensitivity measures motivated by Cumulative
Prospect Theory (CPT) (Tversky & Kahneman, 1992). CPT uniquely models two key aspects of
human decision-making: (1) a utility function u, computed relative to a reference point that induces
more risk-averse behavior in the presence of gains than losses and (2) a weight function w that
prioritizes outlying events. Specific forms of u and w are given in Tversky & Kahneman (1992) (and
in Appendix A.4), but the general form of CPT admits a wide variety of risk-sensitive objectives.

Here we show how to train agents to optimize this class of objectives through sampling-based
estimation of their policy gradients and requisite variance reduction. The final algorithm resembles
well-known on-policy approaches such as Proximal Policy Optimization (Schulman et al., 2017b) and
is similarly widely applicable. Although we do not explore it here, the incorporation of an appropriate
risk-sensitivity criterion could additionally enable risk-aware exploration and adversarial training for
increased robustness (Pinto et al., 2017; Parisi et al., 2019; Zhang et al., 2020).

3 RISK-SENSITIVE POLICY OPTIMIZATION

In this section we formalize the class of distributional objectives to be considered, derive a samplingbased approximation of its policy gradient, enact variance reduction on this estimate, and use the
result to produce a practical learning algorithm.


-----

3.1 PRELIMINARIES: PROBLEM AND NOTATION

Standard deep reinforcement learning seeks to maximize the expected reward of an agent over
encountered trajectories; that is, it maximizes the objective

_J(θ) = Eτ_ _∼pθ(τ_ ) _r(st, at)_ _._ (1)

 Xt 

Here pθ(τ ) is the distribution over trajectories τ ≡ **s1, a1, . . ., sT, aT induced by a policy parameter-**
ized by θ; st, at, and r(st, at) denote the state, action, and reward at time t, respectively. To enable
the incorporation of distributional context, we instead consider the objective


+∞
_J(θ) =_ _u(r(τ_ ))
Z−∞


_w(Pθ(r(τ_ )) _dr(τ_ ), (2)



_dr(τ_ )


where u(r(τ )) is the utility associated with full-trajectory reward r(τ ) ≡ [P]t _[r][(][s][t][,][ a][t][)][ and][ w]_

is a piecewise differentiable weighting function of the CDF of trajectory reward Pθ(r(τ )) =
_r(τ_ )
_−∞_ _[p][θ][(][r][′][)][dr][′][.]_

Equation 2 is inspired by CPT (Tversky & Kahneman, 1992), which includes a pair of integrals ofR
this form. It was chosen for its generality; by using different utility functions u and weight functions
_w one may represent all of the risk measures mentioned in Section 2 and all of the risk measures_
evaluated by Dabney et al. (2018a). The form (2) reduces to (1) when u and w are both the identity
mapping. While designed for the episodic setting, the objective (2) may be considered for infinite
horizons through the choice of appropriately long windows.

3.2 RISK-SENSITIVE POLICY GRADIENT

To optimize the objective (2), we first derive an approximation to its gradient with respect to the policy
parameters θ. Working toward a representation that can be sampled, we assert the independence of
the reward on θ and use the chain rule to write


_∞_ _d_
_θJ(θ) =_ _u(r(τ_ ))
_∇_ _dr(τ_ )
Z−∞


_w[′](Pθ(r(τ_ ))) _θPθ(r(τ_ )) _dr(τ_ ), (3)
_∇_



where w[′] is the derivative of w with respect to Pθ(r(τ )). The gradient of the CDF may be written as
follows:


_r(τ_ )

_θPθ(r(τ_ ))= _θ_ _pθ(r[′])dr[′]_ = _θ_
_∇_ _∇_ Z−∞ _∇_ Zτ _[′][ H][(][r][(][τ]_ [)][ −] _[r][(][τ][ ′][))][p][θ][(][τ][ ′][)][dτ][ ′]_ (4)

=
Zτ _[′][H][(][r][(][τ]_ [)][−][r][(][τ][ ′][))][∇][θ][p][θ][(][τ][ ′][)][dτ][ ′][ =] Zτ _[′][H][(][r][(][τ]_ [)][−][r][(][τ][ ′][))][p][θ][(][τ][ ′][)][∇][θ][ log][ p][θ][(][τ][ ′][)][dτ][ ′][.]


Here the first equality is the integral representation of Pθ(r(τ )), the second uses the Heaviside step
function to select all trajectories with total reward ≤ _r(τ_ ), the third follows from the independence of
reward on θ, and the fourth follows from the expression for the derivative of the natural logarithm. In
the following, we also use the complementary expression


_∞_
1 _pθ(r[′])dr[′]_ =
_−_ _r(τ_ ) _−_
Z 


_τ_ _[′][ H][(][r][(][τ][ ′][)][ −]_ _[r][(][τ]_ [))][p][θ][(][τ][ ′][)][∇][θ][ log][ p][θ][(][τ][ ′][)][dτ][ ′][.][ (5)]


_θPθ(r(τ_ )) = _θ_
_∇_ _∇_


Either form, or a combination of the two, may be substituted into (3) and the result sampled over N
trajectories by first ordering trajectories i = 1 . . . N by increasing reward r(τ ). Then


_N_

_i_

_θJ(θ)_ _u(r(τi))_ _w[′]_
_∇_ _≈_ _N_

_i=1_  

X


_i_ 1
_θPθ(r(τi))_ _w[′]_ _−_
_∇_ _−_ _N_



_∇θPθ(r(τi−1))_ _,_ (6)



-----

where the term w[′](0) _θPθ(r(τ0))_ 0. This ordering scheme produces an asymptotically consistent
_∇_ _≡_
estimate, as shown in the context of CPT value estimation by Prashanth et al. (2016). _θPθ(r(τi))_
_∇_
may be sampled in one of two ways, based on either (4) or (5):


_Tj_

_θ log πθ(aj,t_ **sj,t)**
_∇_ _|_ _≈−_ _N[1]_
_t=1_

X


_Tj_

_∇θ log πθ(aj,t|sj,t)._ (7)
_t=1_

X


_θPθ(r(τi))_
_∇_ _≈_ _N[1]_


_j=1_


_j=i+1_


The expression (6) may be used to train a policy that optimizes the distributional objective (2) in a
manner similar to REINFORCE (Williams, 1992).

3.3 VARIANCE REDUCTION

Reducing the variance of sample-based gradient estimates enables faster learning. Here we take
several steps to reduce the variance of (6), as has been done with the policy gradient estimate of
REINFORCE (Williams, 1992).

First, note that cross-trajectory terms of the form f (τi, aj,t, sj,t) = u(r(τi))∇θ log πθ(aj,t|sj,t),
while nonzero, do not contribute to the gradient estimate in expectation when i ̸= j. A proof of this
assertion is given in Appendix A.1. Using (4) for the first term of (6) and (5) for the second allows us
to write _N_ _i_ _Tj_

_i_ 1

_θJ(θ)_ _u(r(τi))_ _w[′]_ _θ log πθ(aj,t_ **sj,t)**
_∇_ _≈_ _N_ _N_ _∇_ _|_

_i=1_    _j=1_ _t=1_

X X X (8)

_N_ _Tj_

_i_ 1 1
+w[′] _−_ _θ log πθ(aj,t_ **sj,t)** _._

_N_ _N_ _∇_ _|_

  _j=i_ _t=1_ 

X X


Removing cross-trajectory terms gives

_N_

_i_

_θJ(θ)_ _u(r(τi))_ _w[′]_
_∇_ _≈_ _N[1]_ _N_

_i=1_  

X


_i_ 1
+ w[′] _−_

_N_




_Ti_

_∇θ log πθ(ai,t|si,t)._ (9)

 _t=1_
X


Note that the weight coefficients (w[′]( _N[i]_ [) +][ w][′][(][ i][−]N[1] [))][ should be normalized over each batch. The]

expression (9) is equivalent to (6) in expectation, but with reduced variance (see Appendix A.1 for
justification). It has a clear intuition – trajectories are assigned utilities based on their rewards and
their contributions to the gradient are scaled by the derivative of the weight function, just as they are
in Cumulative Prospect Theory (Tversky & Kahneman, 1992).

Standard variance reduction techniques may be applied to this simplified form. Without further
assumption or introduction of additional bias, a static baseline b can be employed:


_i_
_u(r(τi))_ _b_ _w[′]_
_−_ _N_
 


_i_ 1
+ w[′] _−_

_N_




_Ti_

_∇θ log πθ(ai,t|si,t)_ (10)

 _t=1_
X


_θJ(θ)_
_∇_ _≈_ _N[1]_


_i=1_


Justification for this assertion is given in Appendix A.2. Learning may be further improved if we
additionally assume that utility may be allocated on a per-step basis. In this case, per-step utilities
are computed as the difference between what the full-episode utility would be if the episode were
to end at a given time step and what it would have been had the episode ended at the previous time
step. While not applicable in cases where episode utility is adjusted based on final outcome, this
assumption has the significant benefit of modeling the temporal allocation of rewards and aligns
with the standard formulation of RL. With it, the variance of (9) may be further reduced through the
incorporation of utility-to-go and a state-dependent baseline Vφ(si,t):


_i_
_w[′]_

_N_




_i_ 1
+w[′] _−_

_N_




_Ti_ _Ti_

_θ log πθ(ai,t_ **si,t)** _u(si,t′_ _, ai,t′_ ) _Vφ(si,t)_ (11)
_∇_ _|_ _−_

t=1 t[′]=t 
X X


_θJ(θ)_
_∇_ _≈_ _N[1]_


_i=1_


Here u(si,t′ _, ai,t′_ ) is the per-step utility. The value function Vφ(si,t) is parameterized by φ and
trained via regression to minimize _Ti_ 2

(φ) = _Vφ(si,t)_ _u(si,t′_ _, ai,t′_ ) _._ (12)
_L_ _−_

_i,t_  _t[′]=t_ 

X X


-----

A standard argument, similar to the approach taken in (Achiam, 2018), can be used to show that
the incorporation of utility-to-go does not change the expected value of (9). The addition of a
state-dependent baseline also does not introduce additional bias, as justified in Appendix A.2.

Finally, discount factors, bootstrapping, and trust regions may be used to provide additional variance
reduction, just as they are in conventional on-policy learning (Appendix A.2). These measures
may introduce additional bias to the policy gradient estimate, but typically lead to more sampleefficient learning. In our experiments, we evaluate the use of generalized advantage estimation (GAE;
(Schulman et al., 2016)) based on the utility-to-go as well as the clipping-based trust regions of
Proximal Policy Optimization (PPO; (Schulman et al., 2017b)). Incorporating these in the policy
gradient yields


_i_
_w[′]_

_N_




_i_ 1
+w[′] _−_

_N_




_Ti_

_∇θLclip_

 _t=1_
X


_θJ(θ)_
_∇_ _≈_ _N[1]_


log πθ(ai,t|si,t), A[π]u[(][s][i,t][,][ a][i,t][)] _, (13)_



_i=1_


where A[π]u[(][s][i,t][,][ a][i,t][)][ is the standard GAE except with per-step utilities in place of rewards. Trust]
regions are implemented similarly to PPO, pessimistically clipping policy updates to be within a
multiplicative factor of 1 ± ϵ of the existing policy:

_Lclip = min_ log πθ(ai,t|si,t)A[π]u[(][s][i,t][,][ a][i,t][)][,]
 _πθ(ai,t_ **si,t)** (14)

log clip _|_ _πθold_ (ai,t **si,t)** _A[π]u[(][s][i,t][,][ a][i,t][)]_ _._

_πθold(ai,t_ **si,t)** _[,][ 1][ ±][ ϵ]_ _|_

  _|_   

They are used to perform multiple policy updates using the same batch of data, providing learning
that is no longer strictly on-policy but that can be significantly more sample efficient. When following
this route, we apply the same early stopping mechanism, based on the Kullback-Leibler divergence
(DKL) between old and new policies, as was used by Ray et al. (2019).

3.4 LEARNING ALGORITHM

The above sample-based estimate of the policy gradient may be used to train agents to maximize
distributional objectives of the form (2). The resulting method, Cumulative Prospect Proximal Policy
Optimization (C3PO), is given in Algorithm 1 and mirrors standard on-policy learning.

**Algorithm 1 Cumulative Prospect Proximal Policy Optimization (C3PO)**

**Require: Policy: initial parameters θ0, learning rate αθ, updates per batch Mθ**
**Require: Value: initial parameters φ0, learning rate αφ, updates per batch Mφ**
**Require: Early stopping threshold DKL, stop, discount factor γ**

**for k = 1, 2, . . . do**

Collect set of episodes Dk = {τi} by running policy π(θk) in the environment
Compute per-step utilities u(si,t, ai,t)
Fit value function by regression:
**for m = 1, . . . Mφ do**

2

_φ ←_ _φ + αφ∇φ_ _i1[T][i]_ _i,t_ _Vφ(si,t) −_ [P]t[T][′][i]=t _[γ][t][′][−][t][u][(][s][i,t][′]_ _[,][ a][i,t][′]_ [)]

 

**end for** P P
Update utility-based advantage estimates A[π]u[(][s][,][ a][)][, using new][ V][φ][(][s][)]

Compute weight coefficients based on ordered episode outcomes and normalize
Update policy, using KL-based early stopping:
**for m = 1, . . . Mθ do**

**elseif Dθ ←KL(θπ +θ|| απθθoldN[1]) < DNi=1KL, stop[(][w][′][(][ i]N then[) +][ w][′][(][ i][−]N[1]** [))][ P]t[T]=1[i] _[∇][θ][L][clip][(log][ π][θ][(][a][i,t][|][s][i,t][)][, A]u[π][(][s][i,t][,][ a][i,t][))]_

break P

**end if**

**end for**

**end for**


-----

Figure 1: Example weight functions and their resulting coefficients in the policy gradient estimate
(9). In these plots, outcomes increase in quality from left to right. As in CPT (Tversky & Kahneman,
1992), weight coefficients are proportional to the derivative of the weight function.

Beyond the utility and weight components, Algorithm 1 differs from conventional methods in the
requirement to collect full episodes of data in each batch. This requirement can be removed if
outcomes can be defined over partial rather than full episodes, an assumption that is often viable and
matches human decision-making. For instance, while out of scope for this work, our approach could
be applied to the Atari suite (Bellemare et al., 2013) by considering the outcomes of fixed-length
windows and restructuring Algorithm 1 to mimic minibatch PPO (Schulman et al., 2017b).

4 EXPERIMENTS

To evaluate our approach, we sought to both establish that it can effectively optimize different
distributional objectives and explore the impact of using different objectives on agent outcomes. We
found the OpenAI Safety Gym (Ray et al., 2019) to be suitable for these purposes. Safety Gym is a
configurable suite of continuous, multidimensional control tasks wherein different types of robots
must navigate through obstacles with different dynamics to perform different tasks. By including
both positive and negative events in each training scenario, it allowed us to evaluate how our various
agents handled risk. Safety Gym is also highly stochastic: the locations of the goals and obstacles are
randomized, leading to outcome variability and forcing the agent to learn a generalized navigation
strategy.

Safety Gym logs adverse events but does not incorporate them into the reward function. As our
method relies solely on the training signal from the reward, we assigned each logged adverse event
a fixed, negative reward contribution in experiments using it or other unconstrained agents. Our
initial experiments were conducted with a reward contribution of −0.025, which was found to
allow agents to prioritize reaching goals but deter them from collisions with obstacles. To further
emphasize obstacle avoidance, we doubled this contribution to −0.05 in our experiments using
cautious weightings. These choices and the role they play are further discussed in Section 5.

To highlight distributional differences, we focused on the publicly available, obstacle-rich level 2
environments.[1] Avoiding the longer compute time of the “Doggo” robot, we evaluated the “Point”
and “Car” robots on each task (“Goal”, “Button”, and “Push”). Further details on these environments
and our rationale for choosing them are given in Appendix A.3.

In all experiments, we evaluated five random seeds and matched the hyperparameters used in the
baselines accompanying Safety Gym (Ray et al., 2019) as closely as possible. The neural networks
used to model both policy and value were multilayer perceptrons (MLPs), with two hidden layers of
256 units each and tanh activations. As in Ray et al. (2019), the policy network outputs the mean
values of a multivariate gaussian with diagonal covariance. The control variances are optimized but
independent of state. The full complement of variance reduction measures were used throughout; see
Appendix A.4 for experimental justification of this choice.

4.1 DIFFERING OBJECTIVES

Agent performance was explored under four different distributional objectives. In addition to expected
reward and CPT (configured to match the original form of Tversky & Kahneman (1992) and as given
in Appendix A.4), we optimized for cautious (η = 0.75) and aggressive (η = −0.75) versions of the

1In Safety Gym, the default environments have three levels (0, 1, 2); obstacle density increases with level.


-----

Figure 2: Impact of different distributional objectives in one environment (CarButton2). The shading
in the first 2 plots (and subsequent learning curves) reflects the standard deviation associated with
running over 5 random seeds. Left: Net reward (positive reward minus penalty) throughout learning.
Middle: Average number of cost events per episode during training (lower is better). Right: Agent
outcome distribution in testing (with sampling turned off). The cautious (Wang (η = 0.75)) weighting
shows higher reward and lower cost once trained.

.

distortion risk measure proposed in Wang (2000). This measure is defined as w(p) = Φ(Φ[−][1](p)+ _η),_
where Φ and Φ[−][1] are the standard normal cumulative distribution function and its inverse. While
we found this form to be convenient, the “Pow” metric in Dabney et al. (2018a) or any other set of
similarly shaped w curves should achieve a similar effect. In experiments using the objective from
Tversky & Kahneman (1992), the reference point was taken to be the mean episode reward of the
current batch, matching the tendency of humans to change their standards over time. The four weight
functions and their resulting coefficients in (9) are shown in Figure 1.

Plots of the total rewards (including penalties) in training, average cost events per episode in training,
and outcome distributions in testing are shown for one environment in Figure 2 and for two additional
environments in Appendix A.5. The trends were fairly consistent over the three environments
evaluated in this manner. While no explicit effort was made to handle cost (agents were given only
the sum of positive rewards and penalties), the cautious and aggressive weightings consistently
accumulated relatively low and high costs, respectively. The cautious (Wang(η = 0.75)) agent
typically also generated the highest positive and total rewards after 10 millions steps of training.

To generate the histograms in Figures 2, 5, 8, 9, and 13 as well as the numbers in Table 1, the trained
agents were deployed on a set of 5000 test episodes – 1000 for each of the 5 networks learned using
different random seeds in training. The resulting distributions therefore include contributions from
both aleatoric and epistemic uncertainty. Sampling was turned off, allowing the agents to choose
their perceived optimal action at each time step. In this context the benefit of emphasizing the lower
part of the outcome distribution (i.e., cautious weighting) became more pronounced, in part because
the methods that emphasize poor outcomes tended to maintain higher policy entropy (Appendix A.5).

4.2 CAUTIOUS WEIGHTINGS
To further explore the apparent benefits of cautious weightings in Safety Gym, we trained a series of
variably cautious agents by tuning η in the risk-averse weight function proposed by Wang (2000).
Histograms of their episode rewards in testing are given in Appendix A.5 and summarized in Table
1. In these environments, agent performance – both in terms of improving the lower end of the
reward distribution and on average – was seen to generally improve with increasing η until the range
_η ∈_ [0.75, 1.25], subsequently degrading. Additional comparisons were made with PPO (Schulman
et al., 2017b), which unsurprisingly was found to closely track performance of the “Uniform” agent.
We found that naively incorporating cautious weightings into PPO improved its performance (row
PPO + Wang(0.75) in Table 1), though not to the level of the full C3PO method with η = 0.75.

We then pursued a set of longer runs to compare C3PO with the cautious objective from Wang (2000)
to both unconstrained and constrained benchmarks. Here we did not tune η, keeping it fixed at 0.75
for all experiments. Comparisons with unconstrained methods for three environments are given in
Figures 3, 4, and 5 and for the remaining three environments in Appendix A.6. In addition to PPO,
we compared performance with Trust Region Policy Optimization (TRPO; Schulman et al. (2017a))
as configured in Ray et al. (2019). Since this TRPO configuration generally outperformed the PPO
configuration in Ray et al. (2019) from which we derived the hyperparameters for C3PO, we would
expect C3PO to be at a disadvantage compared to TRPO. However, we found C3PO had the highest


-----

|Col1|PointButton2|CarGoal2|CarButton2|
|---|---|---|---|
||Mean Std Q=0.5 Q=.05|Mean Std Q=0.5 Q=.05|Mean Std Q=0.5 Q=.05|
|Uniform CPT Value Wang (-0.75) Wang (0.5) Wang (0.75) Wang (1.0) Wang (1.25) Wang (1.50) Wang (1.75) PPO PPO + Wang(0.75)|22.5 7.1 22.8 10.7 16.5 7.3 16.3 4.9 17.5 6.1 17.6 7.6 23.3 6.3 23.3 13.1 24.2 6.7 24.4 13.7 24.7 6.0 24.9 15.1 25.4 6.1 25.4 15.9 23.6 5.9 23.7 14.1 23.4 6.2 23.5 13.5 19.0 6.4 18.9 9.1 20.4 8.4 21.7 2.6|18.1 6.6 18.0 7.4 13.9 7.7 14.6 -0.5 13.5 6.9 14.0 0.9 18.5 6.0 18.8 8.2 19.0 6.4 19.3 8.2 20.3 6.9 21.3 7.2 17.3 6.7 17.8 5.6 16.6 7.5 17.2 3.1 12.5 8.0 13.0 -0.6 15.8 6.0 15.8 5.8 17.7 6.7 18.0 6.3|12.4 9.1 14.0 -6.1 9.5 10.7 11.2 -11.0 6.4 10.7 8.9 -15.5 12.9 9.0 14.4 -5.6 14.3 9.5 15.8 -4.4 11.4 11.0 13.8 -12.2 12.7 10.4 14.8 -8.3 10.1 12.0 13.3 -17.9 7.3 13.0 11.1 -22.8 8.3 9.5 9.5 -9.8 12.1 9.5 13.6 -6.8|


**PointButton2** **CarGoal2** **CarButton2**

**Mean** **Std** **Q=0.5** **Q=.05** **Mean** **Std** **Q=0.5** **Q=.05** **Mean** **Std** **Q=0.5** **Q=.05**

**Uniform** 22.5 7.1 22.8 10.7 18.1 6.6 18.0 7.4 12.4 9.1 14.0 -6.1

**CPT Value** 16.5 7.3 16.3 4.9 13.9 7.7 14.6 -0.5 9.5 10.7 11.2 -11.0

**Wang (-0.75)** 17.5 6.1 17.6 7.6 13.5 6.9 14.0 0.9 6.4 10.7 8.9 -15.5

**Wang (0.5)** 23.3 6.3 23.3 13.1 18.5 6.0 18.8 **8.2** 12.9 9.0 14.4 -5.6

**Wang (0.75)** 24.2 6.7 24.4 13.7 19.0 6.4 19.3 8.2 **14.3** 9.5 **15.8** **-4.4**

**Wang (1.0)** 24.7 6.0 24.9 15.1 **20.3** 6.9 **21.3** 7.2 11.4 11.0 13.8 -12.2

**Wang (1.25)** **25.4** 6.1 **25.4** **15.9** 17.3 6.7 17.8 5.6 12.7 10.4 14.8 -8.3

**Wang (1.50)** 23.6 5.9 23.7 14.1 16.6 7.5 17.2 3.1 10.1 12.0 13.3 -17.9

**Wang (1.75)** 23.4 6.2 23.5 13.5 12.5 8.0 13.0 -0.6 7.3 13.0 11.1 -22.8

**PPO** 19.0 6.4 18.9 9.1 15.8 6.0 15.8 5.8 8.3 9.5 9.5 -9.8

**PPO + Wang(0.75)** 20.4 8.4 21.7 2.6 17.7 6.7 18.0 6.3 12.1 9.5 13.6 -6.8

Table 1: Testing statistics for episode rewards achieved by agents trained over 10 million steps with
different distributional objectives. Q = 0.5 is the median and Q = 0.05 refers to the location of the
0.05 quantile. Blue bold-face represents the best performance for a given environment; in all cases
these occur for moderately cautious weightings (η ∈ [0.75, 1.25]).

Figure 3: Average episode reward (including penalty) over training for different learning approaches
in three different environments. C3PO with Wang (η = 0.75) weighting outperforms others.

average reward (including penalty) in five of the six environments and lowest average cost in five
of the six environments. In addition, agents that used the cautious weightings tended to have more
stable and repeatable training, as evidenced by the tight distribution of their learning curves. This
tightness was found to reflect a lack of negative outlier episodes and potentially lower epistemic
uncertainty throughout training. Finally, note that the use of a nonzero penalty for cost events resulted
in significantly lower incurred costs than were observed with unconstrained agents trained without a
penalty (Ray et al., 2019). PPO and TRPO were seen to reach similar cost levels without a penalty;
these levels are indicated by red dashed lines in the cost figures.

Comparisons with versions of PPO and TRPO that use Lagrangian constraints (PPO-Lagrangian
and TRPO-Lagrangian; Ray et al. (2019)) to match the cost level of C3PO are shown in Figure 6
and Appendix A.7 . We see that, given the same level of cost incurred per episode, agents trained
using C3PO consistently achieve higher levels of reward than those trained with PPO-Lagrangian
and TRPO-Lagrangian. As above, training is seen to be more stable and repeatable using our risksensitive method. Additional comparisons were generated with Constrained Policy Optimization
(CPO; Achiam et al. (2017)), but are not shown in Figure 6 because they failed to maintain the cost
levels of the other methods. For completeness, they are given in Appendix A.7.

Figure 4: Average number of penalty events per episode (lower is better) over training for different
learning approaches in three different environments. The horizontal lines reflect the cost levels
reached by both PPO and TRPO training with zero penalty in Ray et al. (2019).


-----

Figure 5: Testing reward distributions (including penalty; sampling turned off) for long training runs
of three Safety Gym environments.

Figure 6: Comparison of positive contributions to episode reward during training for our approach
(yellow) and Lagrangian methods configured to have the same cost level.

5 DISCUSSION

The analysis above allows for sample-based policy gradient estimates of a broad class of distributional
objectives. Variance reduction measures were shown to enable efficient optimization based on these
estimates (Appendix A.4). However, it was not seen to be the case that a given distributional objective
could be most effectively optimized directly. Instead, the best results were generally obtained through
moderate emphasis on improving negative training outcomes (Table 1).

To understand this behavior, consider the interplay of optimization and exploration in the training
of cautious and aggressive agents. Cautious weightings continually emphasize the lower part of the
outcome distribution, pushing that part of the distribution upward and adjusting behavior the most
where it is most necessary. Once a part of the state space where the agent is deficient is adequately
addressed, a different part of the state space takes its place. Policy entropy remains high because of
the emphasis on problematic situations, ensuring adequate exploration. This trend continues with
increasing η, until the point where the agent begins to ignore high quality training outcomes too
much. Conversely, aggressive weightings continually emphasize the best outcomes in the distribution.
When an already strong outcome is given increased attention, it is likely to stay at the top. Hence
agents trained with aggressive weightings tend to become myopic, obsessing over a fraction of the
state space while neglecting the rest of it. They tend to explore inadequately and ironically fail to
attain better top-end performance than more cautious weightings.

Given the consistent performance gains observed using our method, we propose that it represents
a useful option for improving the performance and stability of on-policy learners. This should be
particularly true in the presence of a meaningful trade-off between positive and negative reward terms
and when there is significant stochasticity. While our approach does add an additional hyperparameter
– the shaping constant η – one value for that hyperparameter was seen to provide gains across all
environments tested. While our approach does not provide for a direct choice of cost limit as
constrained methods do, it is simpler to implement and was consistently seen to be more performant
for the cost level it reached. It is also likely possible to use cautious weightings in conjunction with
constrained RL, though this has not yet been investigated.

6 CONCLUSIONS

In this work, we proposed a risk-sensitive learning algorithm based on a policy gradient estimate
for a broad class of distributional objectives. When configured to emphasize improvement in
scenarios where the agent performs poorly, we found our method to compare favorably with existing
unconstrained and constrained on-policy learners.


-----

REFERENCES

[Josh Achiam. Openai spinning up: Proof for don’t let the past distract you. http://spinningup.](http://spinningup.openai.com/en/latest/spinningup/extra_pg_proof1.html)
[openai.com/en/latest/spinningup/extra_pg_proof1.html, 2018.](http://spinningup.openai.com/en/latest/spinningup/extra_pg_proof1.html)

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In
_Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 22–31, 2017._

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mane.´
Concrete problems in AI safety. arXiv:1606.06565, 2016.

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An
evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279,
jun 2013.

Marc G. Bellemare, Will Dabney, and Remi Munos. A distributional perspective on reinforcement´
learning. In Proceedings of the 34th International Conference on Machine Learning (ICML),
volume 70, pp. 449–458, 2017.

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar A. Due´nez-˜
Guzman. Lyapunov-based safe policy optimization for continuous control.´ _CoRR, abs/1901.10031,_
[2019. URL http://arxiv.org/abs/1901.10031.](http://arxiv.org/abs/1901.10031)

Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for´
distributional reinforcement learning. In Proceedings of the 35th International Conference on
_Machine Learning (ICML), volume 80, pp. 1096–1105, 2018a._

Will Dabney, Mark Rowland, Marc G. Bellemare, and Remi Munos. Distributional reinforcement´
learning with quantile regression. In Proceedings of the 32nd AAAI Conference on Artificial
_Intelligence (AAAI), pp. 2892–2901, 2018b._

Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional
soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE
_Transactions on Neural Networks and Learning Systems, pp. 1–15, 2021._

Javier Garc´ıa and Fernando Fernandez. A comprehensive survey on safe reinforcement learning.´
_Journal of Machine Learning Research, 16(42):1437–1480, 2015._

Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econo_metrica, 47(2):263–291, 1979._

Dickson H. Leavens. Diversification of investments. Trusts and Estates, 80:469–473, 1945.

Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. DSAC: Distributional soft
actor critic for risk-sensitive reinforcement learning. arXiv:2004.14547, June 2020.

German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual
lifelong learning with neural networks: A review. Neural Networks, 113:54–71, May 2019.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML),
2017.

L.A. Prashanth, Cheng Jie, Michael C. Fu, Steven I. Marcus, and Csaba Szepesvari. Cumulative´
prospect theory meets reinforcement learning: Prediction and control. In Proceedings of the 33nd
_International Conference on Machine Learning (ICML), pp. 1406–1415, 2016._

John W. Pratt. Risk aversion in the small and in the large. Econometrica, 32:122–136, 1964.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement
[learning. https://cdn.openai.com/safexp-short.pdf, 2019.](https://cdn.openai.com/safexp-short.pdf)

R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of conditional value-at-risk. Journal of
_Risk, 2:21–41, 2000._


-----

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional
continuous control using generalized advantage estimation. In Proceedings of the International
_Conference on Learning Representations (ICLR), 2016._

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region
policy optimization. In Proceedings of the 32nd International Conference on Machine Learning
_(ICML), 2017a._

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv:1707.06347, 2017b.

Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for
coherent risk measures. In Proceedings of the 28th International Conference on Neural Information
_Processing Systems (NeurIPS), pp. 1468–1476, 2015._

Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of
uncertainty. Journal of Risk and Uncertainty, 5(4):297–323, 1992.

Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal
_of Risk and Insurance, 67(1):15, March 2000._

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement
learning. Machine Learning, 8(3):229–256, May 1992.

Congbin Wu and Yuanlie Lin. Minimizing risk models in Markov decision processes with policies
depending on target values. Journal of Mathematical Analysis and Applications, 231:47–67, 1999.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh.
Robust deep reinforcement learning against adversarial perturbations on state observations. In
H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Proceedings of the 34th
_International Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp._
21024–21037, 2020.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning
via distributional risk in the dual domain. IEEE Journal on Selected Areas in Information Theory,
2(2):611–626, 2021.

Han Zhong, Ethan X. Fang, Zhuoran Yang, and Zhaoran Wang. Risk-sensitive deep RL: Varianceconstrained actor-critic provably finds globally optimal policy. arXiv:2012.14098, 2020.

A APPENDIX

A.1 EVALUATION OF CROSS-TRAJECTORY TERMS IN POLICY GRADIENT

In this section, we first show that the cross-trajectory terms in our policy gradient estimate (6) have
an expectation value of 0. We then argue that their removal leads to a policy gradient estimate with
reduced variance.

**Lemma 1. Cross-trajectory terms of the form f** (τi, aj,t, sj,t) = u(r(τi))∇θ log πθ(aj,t|sj,t), where
_i ̸= j, do not contribute to the gradient estimate (6) in expectation._

_Proof. First, note that_


_Eτi∼pθ(τ_ ),τj _∼pθ(τ_ )f (τi, aj,t, sj,t) = Eτi∼pθ(τ ),τj _∼pθ(τ_ )u(r(τi))∇θ log πθ(aj,t|sj,t)


_|_

_∇θ log πθ(aj,t|sj,t)_

[#]

_[τ][i]_


= Eτi∼pθ(τ )


_u(r(τi))Eτj_ _∼pθ(τ_ )


-----

Then consider the innermost expectation:


_Eτj_ _∼pθ(τ_ )


_∇θ log πθ(aj,t|sj,t)_ _[τ][i]_


**sj,t,aj,tp(sj,t, aj,t|πθ, τi)∇θ log πθ(aj,t|sj,t)daj,tdsj,t**


_p(sj,t_ _πθ, τi)_
**sj,t** _|_

_p(sj,t_ _πθ, τi)_
**sj,t** _|_


**aj,t** _πθ(aj,t|sj,t)∇θ log πθ(aj,t|sj,t)daj,tdsj,t_

_θπθ(aj,t_ **sj,t)daj,tdsj,t**
**aj,t** _∇_ _|_


_p(sj,t_ _πθ, τi)_ _θ_
**sj,t** _|_ _∇_


_πθ(aj,t_ **sj,t)daj,tdsj,t**
**aj,t** _|_


= **sj,tp(sj,t|πθ, τi)(∇θ1)dsj,t = 0.**
Z

To see why removal of cross-trajectory terms leads to reduced variance, consider that the
full expression (6) may be written as the sum of terms of the form f (τi, aj,t, sj,t) =
_u(r(τi))∇θ log πθ(aj,t, sj,t). Its variance is the sum of the total variance from terms where i = j,_
the total variance from terms where i ̸= j, and a term proportional to the covariance of these two
totals. However, because each term in the covariance contains at least one trajectory that differs from
the rest, the above reasoning may be applied to argue that the covariance is 0. Hence, the removal of
the cross-trajectory terms lowers the variance of the policy gradient estimate by the variance of the
cross-trajectory terms.

A.2 INTRODUCTION OF STATIC AND STATE-DEPENDENT BASELINES

**Lemma 2. A static baseline of the utility may be added to the policy gradient estimate (9) without**
_introduction of bias._

_Proof. The additional term is 0 in expectation as_

_i_ _i_ 1

_Eτi∼pθ(τ_ ) _b_ _w[′]_ _n_ + w[′] _−n_ _∇θ log πθ(ai,t|si,t)_

      

_i_ _i_ 1
=b _w[′]_ + w[′] _−_ _p(si,t, ai,t_ _πθ)_ _θ log πθ(ai,t_ **si,t)** _dai,tdsi,t_

_n_ _n_ **si,t,ai,t** _|_ _∇_ _|_

     Z  

_i_ _i_ 1
=b _w[′]_ + w[′] _−_ _p(si,t_ _πθ)_ _πθ(ai,t_ **si,t)** _θ log πθ(ai,t_ **si,t)dai,tdsi,t**

_n_ _n_ **si,t** _|_ **ai,t** _|_ _∇_ _|_

     Z Z

_i_ _i_ 1
=b _w[′]_ + w[′] _−_ _p(si,t_ _πθ)_ _θ_ _πθ(ai,t_ **si,t)dai,tdsi,t**

_n_ _n_ **si,t** _|_ _∇_ **ai,t** _|_

     Z Z

_i_ _i_ 1
=b _w[′]_ + w[′] _−_ _p(si,t_ _πθ)(_ _θ1)dsi,t = 0._

_n_ _n_ **si,t** _|_ _∇_

     Z


The contribution of the weight terms (w[′]( _n[i]_ [) +][ w][′][(][ i][−]n[1] [))][ may be pulled out of the integral between]

the first and second line because of its independence on both state and action. This term is fixed for a
given trajectory by the rank of its reward amongst the rewards accumulated on all trajectories in the
current batch.

In our variance reduction experiments (Appendix A.4), the “Base” agent uses b equal to the mean of
full-episode utility in the current batch.

As described in Section 3.3, we may further adjust the policy gradient estimate through introduction
of per-step utilities. In this case, we may justify the use of a state-dependent baseline through the
following.


-----

**Lemma 3. A state-dependent baseline Vφ(si,t) may be added to the policy gradient estimate (9)**
_without introduction of bias, if per-step utilities are assumed._

_Proof. The additional term is 0 in expectation as_


_i_

_Eτi∼pθ(τ_ ) _w[′]_ _n_

 

_i_
= _w[′]_

_n_

 

_i_
= _w[′]_

_n_

 

_i_
= _w[′]_

_n_

 

_i_
= _w[′]_

_n_

 


_i_ _i_ 1

_Eτi∼pθ(τ_ ) _w[′]_ _n_ + w[′] _−n_ _∇θ log πθ(ai,t|si,t)Vφ(si,t)_

     

_i_ _i_ 1
= _w[′]_ + w[′] _−_ _p(si,t, ai,t_ _πθ)Vφ(si,t)_ _θ log πθ(ai,t_ **si,t)dai,tdsi,t**

_n_ _n_ **si,t,ai,t** _|_ _∇_ _|_

     Z

_i_ _i_ 1
= _w[′]_ + w[′] _−_ _p(si,t_ _πθ)Vφ(si,t)_ _πθ(ai,t_ **si,t)** _θ log πθ(ai,t_ **si,t)dai,tdsi,t**

_n_ _n_ **si,t** _|_ **ai,t** _|_ _∇_ _|_

     Z Z

_i_ _i_ 1
= _w[′]_ + w[′] _−_ _p(si,t_ _πθ)Vφ(si,t)_ _θ_ _πθ(ai,t_ **si,t)dai,tdsi,t**

_n_ _n_ **si,t** _|_ _∇_ **ai,t** _|_

     Z Z

_i_ _i_ 1
= _w[′]_ + w[′] _−_ _p(si,t_ _πθ)Vφ(si,t)(_ _θ1)dsi,t = 0._

_n_ _n_ **si,t** _|_ _∇_

     Z

The rationale for pulling the w[′] terms out of the integral is the same as in Lemma 2.


_i_ 1
+ w[′] _−_

_n_



_i_ 1
+ w[′] _−_

_n_



_i_ 1
+ w[′] _−_

_n_



_i_ 1
+ w[′] _−_

_n_



_i_ 1
+ w[′] _−_

_n_




Finally, we note that the ability to pull the contribution of the weight terms (w[′]( _n[i]_ [) +][ w][′][(][ i][−]n[1] [))][ to the]

front of Equation 11 allows us to formulate advantage estimates based on per-step utility. Bootstrap
estimates of the value function Vφ(si,t) and Generalized Advantage Estimation as in Schulman et al.
(2016) can be conducted exactly as they are in standard on-policy learning, if rewards are replaced by
per-step utilities.

A.3 ADDITIONAL INFORMATION ON SAFETY GYM

As mentioned in Section 4, we chose to evaluate our approach using the OpenAI Safety Gym (Ray
et al., 2019). The choice was governed by our desire to test in conditions with clear cost-benefit
trade-offs, significant stochasticity, adequate complexity, and available benchmarks. While our
methods are not limited to particular task types or observation/action spaces, we found Safety Gym
to be suitable for exploring their potential.

The six environments chosen were the most obstacle-rich of the publicly available environments that
used the “Point” and “Car” robots. The Point robot is constrained to the 2D plane and has two control
dimensions: one for moving forward/backward and one for turning. The Car robot also has two
control dimensions, corresponding to independently actuated parallel wheels. It has a freely rotating
wheel and, while it is not constrained to the 2D plane, typically remains in it. While we expect our
results to extend to the remaining default robot, “Doggo”, we did not experiment with it because of
the order of magnitude longer training times it exhibited in Ray et al. (2019).

Several types of obstacles and tasks were present in the environments we evaluated. In all cases, the
robot is given a fixed amount of time (1000 steps) to complete the prescribed task as many times as
possible and is motivated by both sparse and dense reward contributions. In the “Goal” environments,
the robot must navigate to a series of randomly-assigned goal positions, with a new target being
assigned as soon as a goal is reached. In the “Button” environments, the robot must reach and press
a sequence of goal buttons while avoiding other buttons. In the “Push” task, the robot must push a
box to a series of goal positions. The set of obstacles are different for each task; among the three
environments there are a total of five different constraint elements (hazards, vases, incorrect buttons,
pillars, and gremlins), each with different dynamics. See Ray et al. (2019) for further details.

A.4 EMPIRICAL PERFORMANCE OF VARIANCE REDUCTION MEASURES

To gauge the impact of the variance reduction techniques outlined in Section 3.3, we evaluated their
performance in maximizing the value function of Cumulative Prospect Theory (Tversky & Kahneman,
1992). As mentioned in Section 3.1, this function has two integrals of the form (2):


-----

Figure 7: Impact of variance reduction measures on optimization of the CPT value function. Here
“Base” refers to the risk-sensitive policy gradient estimate (10), “UTG” adds utility-to-go and a neural
network baseline (11), “GAE” incorporates generalized advantage estimation, and “TR” implements
trust regions via clipping. Shading represents the variation over five random seeds.


_∞_
_J(θ) = −_ _u[−](r(τ_ ))
Z−∞

_∞_
+ _u[+](r(τ_ ))
Z−∞


_w[−](Pθ(r(τ_ ))) _dr(τ_ )
 (15)

_w[+](1_ _Pθ(r(τ_ ))) _dr(τ_ )
_−_ _−_



_dr(τ_ )

_d_

_dr(τ_ )


In Tversky & Kahneman (1992), the utility functions are computed relative to a reference point and
reflect the tendency of humans to be more risk-averse in the presence of gains than in the presence
of losses. The weight functions {w[+], w[−]} model our inclination to emphasize the best and worst
possible outcomes in our decision-making.

More specifically, in these experiments we used the piecewise utility functions u[+](r) = H(r −
_r0)(r −_ _r0)[σ]_ and u[−] = λH(r0 − _r)(p[η]r0 −_ _r)[σ]_ with static reference r0 = 10, σ = 0.88, and λ = 2.25.
The weight function w(p) = 1

(p[η]+(1−p)[η]) _η_ [was used, where][ η][ = 0][.][61][ for][ r < r][0][ and][ η][ = 0][.][69][ for]

_r_ _r0. Four methods were evaluated, incorporating progressive amounts of variance reduction:_
_≥_

-  Base: Risk-sensitive policy gradient with a static baseline (10)

-  UTG: Base with utility-to-go and a neural network baseline (11)

-  GAE: UTG with generalized advantage estimation ((13) without clipping)

-  TR: GAE with trust regions ((13) with clipping (14))

As shown in Figure 7, the incorporation of these techniques increased the sample efficiency of the
CPT value optimization significantly. Consequently, we used the full complement (TR) in all other
experiments.

A.5 DIFFERING OBJECTIVES: ADDITIONAL RESULTS

Below we include results for all environments for the experiments described in Section 4.1.


-----

Figure 8: Impact of different distributional objectives in remaining two environments of initial trials.
The shading reflects the standard deviation associated with running over 5 random seeds. Left: Net
reward (positive reward minus penalty) throughout learning. Middle: Average number of cost events
per episode during training (lower is better). Right: Agent outcome distribution in testing (i.e., with
sampling turned off).

Figure 9: Agent outcome distributions across trials run over increasingly cautious (η increasing)
objectives. Distributions correspond to results shown in Table 1.

In addition, we note the trend of policy entropies with different distributional objectives. In general,
more cautious weightings maintain higher entropy for longer than more aggressive weightings. Note
that these plots represent an upper bound because they do not account for action clipping by the
environment; see Ray et al. (2019) for details.

Figure 10: Policy entropy progression during training for three environments. Shading reflects the
observed variation over 5 random seeds.


-----

A.6 ADDITIONAL COMPARISONS WITH UNCONSTRAINED METHODS

Below are plots of average episode reward and average number of episode cost events throughout
training for the remainder of the environments on which we conducted long runs (Section 4.2). Also
included are histograms of testing performance for those runs.

Figure 11: Average episode reward (including penalty) over training for different unconstrained
learning approaches in remaining three environments.

Figure 12: Average number of cost events per episode (lower is better) over training for different
unconstrained learning approaches in remaining three environments. As above, the “zero-penalty”
line refers to the level reached by PPO and TRPO trained with no penalty in the reward (Ray et al.,
2019).

Figure 13: Testing reward distributions (including penalty; sampling turned off) for long training
runs in the remaining three Safety Gym environments. In five of the six environments, C3PO with
_η = 0.75 provides tangible benefit. A smaller η is likely required to improve performance on_
CarPush2.

A.7 ADDITIONAL COMPARISONS WITH CONSTRAINED METHODS

As mentioned in Section 4.2, we compared the performance of C3PO with constrained methods by
setting the cost limit of the constrained methods to match the cost level attained by C3PO. Here we
provide

-  the positive reward plots for the remaining three Safety Gym environments studied,

-  the cost plots for each of the six environments, and


-----

-  all plots for Constrained Policy Optimization (CPO; Achiam et al. (2017)).

The intent of the cost plots of Figure 15 is to show rough consistency between the cost levels of
our approach and Lagrangian methods configured to have the same cost limit. This is verified, but
other trends should be noted. First, while the Lagrangian-based methods typically follow the cost
constraint well, they cannot satisfy it in each batch. Second, our approach tends to have comparable
or lower cost rates throughout training. This safe exploration metric, defined in Ray et al. (2019),
refers to the average cost per episode over all of training up to a given point.

Results related to Constrained Policy Optimization (CPO; Achiam et al. (2017)) are included here but
not in the main text because, consistent with (Ray et al., 2019), we were not able to configure CPO to
respect the cost levels of the other constrained methods. Here we show the cost levels reached by
CPO compared with C3PO (Figure 16) as well as a comparison of the average episode rewards of
the two (Figure 17). For the latter, we employed the penalty scaling used by C3PO to enable a fair
comparison.

Figure 14: Comparison of positive contributions to episode reward during training for our approach
(yellow) and Lagrangian methods configured to have the same cost level. The plots for the other three
environments are shown in Figure 6.

Figure 15: Comparison of cost incurred (lower is better) during training for our approach and
Lagrangian methods configured to have the same cost level. As intended, cost levels are consistently
matched between the methods. As above, the “zero-penalty” line refers to the level reached by PPO
and TRPO trained with no penalty in the reward (Ray et al., 2019).


-----

Figure 16: Comparison of cost incurred (lower is better) during training for our method and Constrained Policy Optimization (Achiam et al., 2017) configured to have a matching cost limit. Results
are consistent with Ray et al. (2019). As above, the “zero-penalty” line refers to the level reached by
unconstrained PPO and TRPO trained with no penalty in the reward (Ray et al., 2019).

Figure 17: Comparison of average episode reward (including penalty) for C3PO and CPO.

A.8 SUPPLEMENTARY MATERIALS

The code used to produce these results is included in our Supplementary Materials.


-----

