# DEEP REINFORCEMENT LEARNING FOR DYNAMIC EX## PECTILE RISK MEASURES: AN APPLICATION TO

# EQUAL RISK OPTION PRICING AND HEDGING

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Motivated by the application of equal-risk pricing and hedging of a financial
derivative, where two operationally meaningful hedging portfolio policies needs
to be found that minimizes coherent risk measures, we propose in this paper a
novel deep reinforcement learning algorithm for solving risk-averse dynamic decision making problems. Prior to our work, such hedging problems can either only
be solved based on static risk measures, leading to time-inconsistent policies, or
based on dynamic programming solution schemes that are impracticable in realistic settings. Our work extends for the first time the deep deterministic policy
gradient algorithm, an off-policy actor-critic reinforcement learning (ACRL) algorithm, to solving dynamic problems formulated based on time-consistent dynamic
expectile risk measure. Our numerical experiments confirm that the new ACRL
algorithm produces high quality solutions to equal-risk pricing and hedging problems and that its hedging strategy outperforms the strategy produced using a static
risk measure when the risk is evaluated at later points of time.

1 INTRODUCTION

This paper considers solving risk-averse dynamic decision making problems arising from applications where risk needs to be evaluated according to risk measures that are coherent. In particular, we
draw our motivation from the financial application of equal-risk pricing (ERP) and hedging (Guo &
Zhu (2017)), where two dynamic hedging problems need to be solved, one for the buyer and one for
the seller of a financial derivative (a.k.a option), for determining a fair transaction price that would
expose both parties to the same amount of hedging risk. The need to meaningfully model each
party’s best hedging decision in a financial market, namely that no arbitrage is allowed, and to have
a meaningful comparison between the two parties’ risk exposures, namely that the risks should be
measured in the same units, has led to the use of coherent risk measures for capturing both parties’
hedging risks in this application (see Marzban et al. (2020)).

To this date, most solution methods proposed for solving risk-averse dynamic decision making problems under a coherent risk measure have either relied on traditional dynamic programming (DP),
which suffers from the curse of dimensionality and assumes the knowledge of a stochastic model
that precisely captures the dynamics of the decision environment, or on the use of a static risk measure, i.e., that disregards the temporal structure of the random variable (e.g. Marzban et al. (2020),
Carbonneau & Godin (2020), and Carbonneau & Godin (2021) in the case of the ERP application).
The latter raises the serious issue that the resulting policy could be time inconsistent, i.e. that the actions prescribed by the policy may be considered significantly sub-optimal once the state is visited.
In an application such as ERP, this issue implies that policies obtained based on static risk measures
will not be implemented in practice, raising the need to consider dynamic risk measures.

Focusing on deep reinforcement learning (DRL) methods, while there has been a large number of
approaches proposed to address risk averse Markov decision processes (MDPs) using coherent risk
measures, to the best of our knowledge, all of them, except for two exceptions, consider a static
risk measure (see Prashanth & Ghavamzadeh (2013); Chow & Ghavamzadeh (2014); Castro et al.
(2019); Singh et al. (2020); Urp´ı et al. (2021)) and therefore suffer from time-inconsistency. The


-----

two exceptions consist of Tamar et al. (2015) and Huang et al. (2021) who propose actor-critic reinforcement learning (ACRL) algorithms to deal with a general dynamic law-invariant coherent risk
measures. Unfortunately, the two algorithms respectively either assume that it is possible to generate
samples from a perturbed version of the dynamics, or rely on training three neural networks (namely
a state distribution reweighting network, a transition perturbation network, and a Lagrangean penalisation network) concurrently with the actor and critic networks. Furthermore, only Huang et al.
(2021) actually implemented their method. This was done on a toy tabular problem involving 12
states and 4 actions where it produced questionable performances[1].

In this paper, we develop a new model-free ACRL algorithm for solving a time-consistent risk
averse MDP under a dynamic expectile risk measure.[2] Overall, we may summarize the contribution
as follows:

-  Our ACRL algorithm is the first to naturally extend the popular model-free deep deter
ministic policy gradient algorithm (DDPG) (see Lillicrap et al. (2015)) to a risk averse
setting where a time consistent coherent risk measure is used. Unlike the ACRL proposed
in Huang et al. (2021), which employs five neural networks, our algorithm will only require an actor and a critic network. While our policy network will be trained following
a stochastic gradient procedure similar to Silver et al. (2014), we are the first to leverage
the elicitability property of expectile risk measures to propose a procedure for training the
“risk-to-go” deep Q-network that is also based on stochastic gradient descent.

-  Our ACRL is the first model-free DRL-based algorithm capable of identifying optimal risk

averse option hedging strategies that are time-consistent with respect to a dynamic coherent
risk measure, and of computing their associated equal risk prices. A side benefit of timeconsistency will be that after training for an option with a given maturity, one obtains equal
risk prices and hedging strategies for any other shorter maturities. While our algorithm
certainly has a broader set of applications, we believe that ERP constitutes an original and
fertile application in which to develop and test new risk averse DRL methods.



-  We evaluate the training efficiency and the quality of solution, in terms of quality of option

hedging strategies and of estimated equal risk prices, obtained using our ACRL algorithm
on a synthetic multi-asset geometric Brownian motion market model. These experiments
constitute the first real application of a risk averse DRL algorithm that employs a dynamic
coherent risk measure.

The rest of this paper is organized as follows. Section 2 introduces equal risk pricing and its associated DP equations. Section 3 proposes the new ACRL algorithm for general finite horizon risk
averse MDP with dynamic expectile measures. Finally, Section 4 presents and discusses our numerical experiments. We note that a reader only interested in the ACRL algorithm can skip right ahead
to Section 3.

2 APPLICATION: EQUAL RISK PRICING AND HEDGING UNDER DYNAMIC

EXPECTILE RISK MEASURES

As described in Marzban et al. (2020), the problem of ERP can be formalized as follows. Consider
a frictionless market, i.e. no transaction cost, tax, etc, that contains m risky assets, and a risk-free
bank account with zero interest rate. Let St : ⌦ _! R[m]_ denote the values of the risky assets adapted
to a filtered probability space (⌦, F, F := {Ft}t[T]=0[,][ P][)][, i.e. each][ S][t] [is][ F][t] [measurable. It is assumed]

that St is a locally bounded real-valued semi-martingale process and that the set of equivalent local
martingale measures is non-empty (i.e. no arbitrage opportunity). The set of all admissible selffinancing hedging strategies with the initial capital p0 2 R is shown by X (p0):

_t−1_

_X_ (p0) = (X : ⌦ _! R[T]_ " _t=0_ _[,]_ _Xt = p0 +_ _t[0]=0_ **_⇠t[>][0][ ∆][S][t][0][+1][,]_** _8t = 1, . . ., T_ ) _,_

" X

1 "
At the time of writing this paper, the risk averse implementation of this algorithm reported in Huang et al."

"[9{][⇠][t][}][T][ −][1]

(2021) is unable to recommend an optimal policy in a deterministic setting, while the risk neutral implementation produces policies that are outperformed by risk averse ones in a stochastic setting.

2Our ACRL algorithm exploits the elicitabilty property of expectile risk measures, which is the only elic
itable coherent risk measure.


-----

whereto the filtration ∆St+1 := F and captures the number of shares of each risky asset held in the portfolio during St+1 − **St, the hedging strategy ⇠t 2 R[m]** is a vector of random variables adapted
the period [t, t + 1], and Xt is the accumulated wealth.

Let F ({St}t[T]=1[)][ denote the payoff of a derivative. Throughout this paper, we assume][ F] [(][{][S][t][}][T]t=1[)]

admits the formulation of F (ST, YT ) where Yt is an auxiliary fixed-dimensional stochastic process
that is _t-measurable. This class of payoff functions is common in the literature, (see for example_
_F_
Bertsimas et al. (2001) and Marzban et al. (2020)). The problem of ERP is defined based on the
following two hedging problems that seek to minimize the risk of hedging strategies, one is for the
writer and the other is for the buyer of the derivative:

(Writer) _%[w](p0) =_ inf (1)

_X_ (p0) _[⇢][w][(][F]_ [(][S][T][,][ Y][T][ )][ −] _[X][T][ )]_
_2X_

(Buyer) _%[b](p0) =_ inf (2)

_X_ ( _p0)_ _[⇢][b][(][−][F]_ [(][S][T][,][ Y][T][ )][ −] _[X][T][ )][,]_
_2X_ _−_

where ⇢[w] and ⇢[b] are two risk measures that capture respectively the writer and the buyer’s risk
aversion. In words, equation (1) describes a writer that is receiving p0 as the initial payment and
implements an optimal hedging strategy for the liability F (ST, YT ). On the other hand, in (2) the
buyer is assumed to borrow p0 in order to pay for the option and then to manage a portfolio that
will minimize the risks associated to his final wealth. With equations (1) and (2), ERP defines a
fair price p[⇤]0 [as the value of an initial capital that leads to the same risk exposure to both parties, i.e.]

_%[w](p[⇤]0[) =][ %][b][(][p][⇤]0[)][.]_

In particular, based on Proposition 3.1 and the examples presented in section 3.3 of Marzban et al.
(2020), together with the fact that both ⇢[w] and ⇢[b] are dynamic recursive law invariant risk measures,
a Markovian assumption allows us to conclude that the ERP can be calculated using two sets of
dynamic programming equations.
**Assumption 2.1. [Markov property] There exists a sufficient statistic process** _t adapted to_
F such that {(St, Yt, t)}t[T]=0 _[is a Markov process relative to the filtration][ F][.]_ _Namely,_

P((St+s, Yt+s, t+s) 2 A|Ft) = P((St+s, Yt+s, t+s) 2 A|St, Yt, t) for all t, for all s ≥ 0,
_and all sets A._

Specifically, on the writer side, we can define VT[w][(][S][T][,][ Y][T][, ][T][ ) :=][ F] [(][S][T][,][ Y][T][ )][, and recursively]


_Vt[w][(][S][t][,][ Y][t][, ][t][) := inf]_ _⇢(_ **_⇠t[>][∆][S][t][+1]_** [+][ V][ w]t+1[(][S][t] [+ ∆][S][t][+1][,][ Y][t] [+ ∆][Y][t][+1][, ][t][+1][)][|][S][t][,][ Y][t][, ][t][)][,]

**_⇠t_** [¯] _−_

where ¯⇢(·|St, Yt, t) is a law invariant risk measure that uses P(·|St, Yt, t). This leads to

considering %[w](0) = V0[w][(][S][0][,][ Y][0][, ][0][)][.] On the other hand, for the buyer we similarly define:

_VT[b][(][S][T][,][ Y][T][, ][T][ ) :=][ −][F]_ [(][S][T][,][ Y][T][ )][ and]

_Vt[b][(][S][t][,][ Y][t][, ][t][) := inf]_ _⇢(_ **_⇠t[>][∆][S][t][+1]_** [+][ V][ b]t+1[(][S][t] [+ ∆][S][t][+1][,][ Y][t] [+ ∆][Y][t][+1][, ][t][+1][)][|][S][t][,][ Y][t][, ][t][)][,]

**_⇠t_** [¯] _−_

with %[b](0) = V0[b][(][S][0][,][ Y][0][, ][0][)][. The following lemma summarizes how DP can be used to compute]

the ERP.
**Lemma 2.1 (Marzban et al. (2020)). Under Assumption 2.1, the ERP that employs dynamic expec-**
_tile risk measure can be computed as: p[⇤]0_ [= (][V][ w]0 [(][S][0][,][ Y][0][, ][0][)][ −] _[V][ b]0_ [(][S][0][,][ Y][0][, ][0][))][/][2][.]

In what follows, we will further assume that the risk measure is a dynamic expectile risk measure.
**Definition 2.1. A dynamic expectile risk measure takes the form: ⇢(X) := ¯⇢0(¯⇢1(. . . ¯⇢T** 1(X)))
_−_

_where each ¯⇢(_ ) is an expectile risk measure that employs the conditional distribution based on _t._

_·_ _F_

_Namely, ¯⇢t(Xt+1) := arg minq ⌧_ E (q − _Xt+1)[2]+[|F][t]_ + (1 − _⌧_ )E (q − _Xt+1)[2]−[|F][t]_ _where Xt+1_

_is a random liability measureable on_ _t+1._

⇥ F ⇤ ⇥ ⇤

Like conditional value at risk, the expectile measure (see Bellini & Bignozzi (2015)) covers the
range of risk attitudes from risk neutrality, when ⌧ = 1/2, to worst-case risk, when ⌧ _! 1._

3 A NOVEL ACTOR-CRITIC ALGORITHM FOR RISK AVERSE MDP UNDER A
DYNAMIC EXPECTILE RISK MEASURE

With the dynamic programming equations in hand, it now becomes apparent that each option hedging problem in ERP can be formulated as a finite horizon Markov Decision Process (MDP) described


-----

with (S, A, r, P ). In this regard, the agent (i.e. the writer or buyer) interacts with a stochastic environment by taking an action at **_⇠t_** [ 1, 1][m] after observing the state st, which includes St, Yt, and _t. Note that to simplify exposition, in this section we drop the reference to ⌘_ _2_ _−_ _2 S_
the specific identity (i.e. w or b) of the agent in our notation. The action taken at each time t results in the immediate stochastic reward that takes the shape of the immediate hedging portfolio
return, i.e. rt(st, at, st+1) := ⇠t[>][∆][S][t][+1] [when][ t < T][ and otherwise of the option liability/payout]

_rT (sT, aT, sT +1) := F_ (ST, YT )(1 − 2 · 1{agent=writer}), which is insensitive to sT +1. Finally, the Markovian exogeneous dynamics described in Assumption 2.1 are modeled using P as
_P_ (st+1|st, at) = P(St+1, Yt+1, t+1|St, Yt, t). Overall, each of the two dynamic derivative
hedging problems presented in Section 2 reduce to a version of the following general risk averse
reinforcement learning problem:

_%(0) = V0(S0, Y0, 0) = min_ 0 [(¯]s0, ⇡0(¯s0)),

_⇡_ _[Q][⇡]_


where ¯s0 := (S0, Y0, 0) is the initial state in which the option is priced while Q[⇡]t [(][s][t][, a][t][) :=]

_⇢¯(−rt(st, at, st+1) + Q[⇡]t+1[(][s][t][+1][, ⇡][t][+1][(][s][t][+1][))][|][s][t][)][,][ Q][⇡]T_ [(][s][T][, a][T][ ) :=][ −][r][T][ (][s][T][, a][T][, s][T][ )][, and where][ ¯]⇢

is an expectile risk measure, i.e. ¯⇢(X) := arg minq E _⌧_ (q _X)[2]+_ [+ (1][ −] _[⌧]_ [)(][q][ −] _[X][)][2]_ . Equipped

_−_ _−_

with these definitions, we can now motivate our proposed extension of the model-free off-policy

⇥ ⇤

deterministic ACRL algorithm to the general finite horizon risk-averse MDP setting. In doing so,
we start with a proposition (see Appendix A.1 for a proof) that will provide the motivation for
a stochastic gradient scheme to optimize the actor network, while the optimization of the critic
network will follow from the elicitability property of the expectile risk measure.

**Proposition 3.1. Let ¯⇡** _be an arbitrary reference policy and µ an arbitrary distribution over the_

_initial state, such that there is a strictly positive probability of reaching all of S for all t ≥_ 1.[3] _For_
_any ⇡[⇤]_ _that satisfies_

_⇡[⇤]_ _2 arg min⇡_ [E][ ˜]stt+1⇠{⇠0P,...,T (·|s }t,,s⇡¯0t(⇠sµt))[Q[⇡]t˜ [(][s]t[˜][, ⇡]t[˜][(][s]t[˜][))]] (3)


_where_ _t[˜] is uniformly drawn, we necessarily have that ⇡[⇤]_ _2 arg min Q[⇡]0_ [(¯]s0, ⇡0(¯s0)) hence %(0) =

_Q[⇡]0_ _[⇤]_ [(¯]s0, ⇡0[⇤][(¯]s0)).

In the context of a deep reinforcement learning approach, we can employ a procedure based on offpolicy deterministic policy gradient (Silver et al., 2014) to optimize (3). Specifically, given a policy
network ⇡[✓], we wish to optimize:



[Q[⇡]t˜ _[✓]_ [(][s]t[˜][, ⇡]t˜[✓][(][s]t[˜][))]][,]


min


_t˜⇠{0,...,T −1}_

_st+1⇠P (·|st,⇡¯t(st))_


using a stochastic gradient algorithm. In doing so, we rely on the fact that:

_r✓Est+1t˜⇠{⇠P0 (,...,T·|st, −⇡¯t1(}st))[Q[⇡]t˜_ _[✓]_ [(][s]t[˜][, ⇡][✓][(][s]t[˜][))]]


_✓Q[⇡]t˜_ _[✓]_ [(][s]t[˜][, a][)]
_r_


_a=⇡t˜[✓][(][s]t[˜][)][ +][ r][a][Q]t[⇡]˜_ _[✓]_ [(][s]t[˜][, a][)][r][✓][⇡]t˜[✓][(][s]t[˜][)]


= E _t˜⇠{0,...,T −1}_

_st+1⇠P (·|st,⇡¯t(st))_

_⇡_ Est+1t˜⇠{⇠P0 (,...,T·|st, −⇡¯t1(}st))


_a=⇡t˜[✓][(][s]t[˜][)]_


_aQ[⇡]t˜_ _[✓]_ [(][s]t[˜][, a][)][r][✓][⇡]t˜[✓][(][s]t[˜][)]
_r_


_a=⇡t˜[✓][(][s]t[˜][)]_


Note that in the above equation, we have dropped the term that depends on _✓Q[⇡]t˜_ _[✓]_ as is commonly
_r_

done in off-policy deterministic gradient methods and usually motivated by a result of Degris et al.
(2012), who argue that this approximation preserves the set of local optima in a risk neutral setting,
i.e. ¯⇢(·) := E[·]. While we do consider as an important subject of future research to extend this

motivation to more general risk measures, our numerical experiments (see Section 4.3) will confirm
empirically that the quality of this approximation permits the identification of nearly optimal hedging
policies.

3In our option hedging problem, given that st is entirely exogenous, the distribution of st+1 is unaffected

by ¯⇡, which can therefore be chosen arbitrarily. Moreover, µ can put all the mass on ¯s0.


-----

Given that we do not have access to an exact expression for Q[⇡]t˜ _[✓]_ [(][s]t[˜][, a][)][, this operator needs to be]

estimated directly from the training data. Exploiting the fact that ⇢ is a utility-based shortfall risk
measure, we get that:


_Q[⇡]t_ [(][s][t][, a][t][)][ 2][ arg min]q


Est+1 _P (_ _st,at)[`(q + rt(st, at, st+1)_ _Q[⇡]t+1[(][s][t][+1][, ⇡][t][+1][(][s][t][+1][)))]]_
_⇠_ _·|_ _−_


where `(y) := (⌧ **1{y > 0}−** (1 _−_ _⌧_ )1{y  0})y[2] is the score function associated to the ⌧ -expectile
risk measure. As explained in Theorem 3.2 of Shen et al. (2014), in a tabular MDP environment one
can apply the following stochastic gradient step:

_Qˆt(st, at)_ _Qt(st, at)_ _↵@`( Q[ˆ]t(st, at) + rt(st, at, st+1)_ _Qt+1(st+1, ⇡t+1(st+1)),_

[ˆ] _−_ _−_ [ˆ]

where @`(y) := 2(⌧ max(0, y) − (1 − _⌧_ ) max(0, −y)) is the derivative of `(y), within a properly
designed Q-learning algorithm and have the guarantee that _Q[ˆ]t(st, at) will almost surely converge to_

_Q[⇡]t_ [(][s][t][, a][t][)][ for all][ t][,][ s][t][, and][ a][t][.]

In the non-tabular setting, we replace _Q[ˆ][⇡]t_ [(][s][t][, a][t][)][ with two estimators: i.e.] the “main” net
work Q[⇡]t [(][s][t][, a][t][|][✓][Q][)][ for the immediate conditional risk and the “target” network][ Q]t[⇡][(][s][t][, a][t][|][✓][Q][0] [)]

for the next period’s conditional risk. The procedure consists in iterating between a step that

attempts to make the main network Q[⇡]t [(][s][t][, a][t][|][✓][Q][)][ a good estimator of][ ⇢][(][−][r][(][s][t][, a][t][, s][t][+1][) +]

_Q[⇡]t+1[(][s][t][+1][, a][t][+1][|][✓][Q][0]_ [))][ and a step that replaces the target network][ Q]t[⇡][(][s][t][, a][t][|][✓][Q][0] [)][ with a network]

more similar to the main one Q[⇡]t [(][s][t][, a][t][|][✓][Q][)][. The former is achieved, similarly as with the policy]

network, by searching for the optimal ✓[Q] according to:



[`(Q[⇡]t˜ [(][s]t[˜][,][ ¯]⇡t˜[(][s]t[˜][)][|][✓][Q][)+][r][t][(][s]t[˜][,][ ¯]⇡t˜[(][s]t[˜][)][, s]t[˜]+1[)][−][Q][⇡]t˜+1[(][s]t[˜]+1[, ⇡]t[˜]+1[(][s]t[˜]+1[)][|][✓][Q][0] [))]][,]


min _t˜_ 0,...,T 1

_✓[Q][ E]_ _⇠{_ _−_ _}_

_st+1⇠P (·|st,⇡¯t(st))_


which suggests a stochastic gradient update of the form ✓[Q] _✓[Q]_ _−_ _↵∆, where ∆_ is

_@`(Q[⇡]t˜_ [(][s]t[˜][,][ ¯]⇡t˜[(][s]t[˜][)][|][✓][Q][)+][r][t][(][s]t[˜][,][ ¯]⇡t˜[(][s]t[˜][)][, s]t[˜]+1[)][−][Q][⇡]t˜+1[(][s]t[˜]+1[, ⇡]t[˜]+1[(][s]t[˜]+1[)][|][✓][Q][0] [))][r]✓[Q] _[Q][⇡]t˜_ [(][s]t[˜][,][ ¯]⇡t˜[(][s]t[˜][)][|][✓][Q][)][ .]

These two types of updates are integrated in our proposed expectile-based actor-critic deep RL (a.k.a.
ACRL) algorithm. A first version, Algorithm 1, is designed for a simulation-based environment.
One may note that in each episode, the reference policy ¯⇡t is updated to be a perturbed version of the

main policy network in order to focus the accuracy of the main critic network’s value and derivatives
on actions that are more likely to be produced by the main policy network. We also choose to update
the target networks using convex combinations operations as is done in Lillicrap et al. (2015) in
order to improve stability of learning. A second more general version of ACRL, which mimics the
original DDPG, by generating minibatches using a replay buffer can also be found in Appendix A.2.

We finally note that in our problem, _P_ (st+1|st, at) = _P_ (st+1|st, a[0]t[)] =

P(St+1, Yt+1, t+1|St, Yt, t), meaning that the action is not affecting the distribution of
state in the next period. This is a direct consequence of using a translation invariant risk measure,
which eliminates the need to keep track of the accumulated wealth in the set of state variables as
explained in Marzban et al. (2020) and allows the reward function to provide an immediate signal
regarding the quality of implemented actions. In the context of our deep reinforcement learning
approach, we observed that convergence speed is significantly improved in training due to this
property (see Figure 4 in Appendix).

4 EXPERIMENTAL RESULTS

In this section we provide two different sets of experiments that are run over one vanilla and one
basket option. We will compare both algorithmic efficiency and quality, in terms of pricing and
hedging strategies, of the dynamic risk model (DRM), which employs a dynamic expectile risk
measure and is solved using our new ACRL algorithm, and the static risk model (SRM), which
employs a static expectile measure and is solved using an AORL algorithm similar to Carbonneau &
Godin (2021). All experiments are done using simulated price processes of five risky assets: AAPL,
AMZN, FB, JPM, and GOOGL. The price paths are simulated using correlated Brownian motions
considering the empirical mean, variance, and the correlation matrix of five reference stocks (AAPL,
AMZN, FB, KPM, and GOOGL) over the period that spans from January 2019 to January 2021. In


-----

**Algorithm 1: The actor-critic RL algorithm for the dynamic recursive expectile option hedging**
problem (ACRL)

Randomly initialize the main actor and critic networks’ parameters ✓[⇡] and ✓[Q];
Initialize the target actor, ✓[⇡][0] _✓[⇡], and critic, ✓[Q][0]_ _✓[Q], networks;_
**for j = 1 : #Episodes do**

Randomly select t 2 {0, 1, ..., T − 1};
Sample a minibatch of N triplets {(s[i]t[, a][i]t[, s][i]t+1[)][}][N]i=1 [from][ P] [(][·|][s][t][,][ ¯]⇡t(st)), where

_⇡¯t(st) := ⇡t(st_ _✓[⇡]) +_ (0, σ);
_|_ _N_

Set the realized losses yt[i] [:=][ −][r][t][(][s]t[i][, a][i]t[, s][i]t+1[) +][ Q][t][+1][(][s][i]t+1[, ⇡][t][+1][(][s][i]t+1[|][✓][⇡][0] [)][|][✓][Q][0] [)][;]

Update the main critic network:


**_✓[Q]_** **_✓[Q]_** **_↵_** **[1]**
**_−_** **_N_**


**_@`(Qt(s[i]t[, a][i]t[|][✓][Q][)][ −]_** **_[y]t[j][)][r]✓[Q]_** **_[Q][t][(][s]t[i][, a][i]t[|][✓][Q][) ;]_**


**_i=1_**


Update the main actor network:


_✓[⇡]_ _✓[⇡]_ _↵_ [1]
_−_ _N_

Update the target networks:


_raQt(s[i]t[, a][|][✓][Q][)][|]a=⇡t(s[i]t[|][✓][⇡][)][r][✓][⇡]_ _[⇡][t][(][s]t[i][|][✓][⇡][) ;]_


_i=1_


_✓[Q][0]_ _↵✓[Q]_ + (1 − _↵)✓[Q][0]_ _,_ _✓[⇡][0]_ _↵✓[⇡]_ + (1 − _↵)✓[⇡][0]_ ; (4)

**end**

both experiments, the maturity of the option will be one year and the hedging portfolios will be
rebalanced on a monthly basis. Table 3 in the appendix provides the descriptive statistics of our
underlying hidden stochastic process.

In what follows, we first explain the architectures of our ACRL model. Then, the training procedure of the networks under the dynamic risk measurement is elaborated. Finally, the main numerical results of the paper are presented for pricing and hedging a vanilla, where the precision
of our approach will be empirically demonstrated, and a basket option. All codes are available at
[https://anonymous.4open.science/r/ERP-Dynamic-Expectile-RM-4BEA.](https://anonymous.4open.science/r/ERP-Dynamic-Expectile-RM-4BEA/README.md)

4.1 ACTOR AND CRITIC NETWORK ARCHITECTURE

Our implementation of the ACRL algorithm involves two simple networks presented in Figure 1.
Since the underlying assets follow a Brownian motion, the actor and critic networks can define
the input state as the logarithm of the cumulative returns of each asset and the time remaining to
maturity (i.e. dimension = m + 1). The actor network is composed of three fully connected layers
where the number of neurons are considered to be k = 32 in the first two layers and then maps back
to the number of assets in the last layer to generate the investment policy accordingly for each asset.
The activation functions in our networks are considered to be tanh functions. In the last layer, this
implies that the actions will lie in [−1, 1][m]. The critic network only concatenates the m dimensional
action information vector after its third layer. In the case of SRM, only the actor network is used.

4.2 TRAINING PROCEDURE AND LEARNING CURVES

Recall that in an SRM setting, overfitting of any DRL algorithm can be controlled by measuring the
performance of the trained policy on a validation data set using an empirical estimate of the riskaverse objective as validation score. Unfortunately, this is no longer possible in the case of DRMs
since the risk measure relies on conditional risk measurements of the trajectories produced by our
policy. In theory, estimates of such conditional measurements could be obtained by training a new
critic network using the validation set (while maintaining the policy fixed to the trained one). In
practice, this is highly computationally demanding to perform in the training stage and raises a new


-----

Figure 1: The architecture of the actor and critic networks in ACRL algorithm.

issue of how to control overfitting of the validation score estimate. Our solution for this problem is
to rely on using static risk measures as validation score, namely a set of static expectiles at risk levels
that are larger or equal to the risk level of the DRM. Figures 2 and 3 in the appendix show examples
of learning curves for the validation performance of DRM and SRM approaches on vanilla and
basket options at a risk level of ⌧ = 90%, with a maturity T = 12. SRM appeared to have a faster
rate of convergence than DRM, due to its simpler architecture. Being a time-inconsistent model,
SRM must however be retrained whenever the maturity of the option is modified. When comparing
convergence rates between vanilla and basket options, we observed similar behavior, which indicates
that the training time might not be very sensitive to the number of assets, thus suffering less from
the curse of dimensionality. We finally note that both our training and validation sets included 1000
trajectories from the underlying geometric Brownian motion process, implying that the procedure
can be applied in settings where only historical data is available.

4.3 VANILLA OPTION HEDGING AND PRICING

In our first set of experiments, we consider pricing and hedging an at-the-money vanilla call option
on AAPL. In this setting, it is possible to obtain (approximately) optimal solutions by dynamic
programming via discretization of the state space. The initial price of AAPL is set to 78.81 and
options with time to maturity ranging from one month to one year are considered. Both DRM and
SRM are trained using a one year maturity/horizon.

With the trained DRM and SRM policy networks, we can evaluate the writer and the buyer’s (outof-sample) risk exposure over a pre-specified time horizon so as to calculate the corresponding ERP.
We consider the following three metrics for measuring the realized risk under different hedging
policy and explain the methods used for calculating the metrics:

-  Out-of-sample static expectile risk: Given a trained policy, use the test data to calculate the

static expectile risk. This is the metric that should be minimized by the SRM.

-  RL based out-of-sample dynamic expectile risk estimation: Given the trained policy, use the

test data to only train a critic network using ACRL to produce an estimate of out-of-sample
dynamic expectile risk. This is an estimate of the metric minimized by the DRM.



-  DP based out-of-sample dynamic expectile risk estimation: Given a trained policy, evaluate

the “true” dynamic expectile risk by solving the dynamic programming equations using a
high precision discretization of the states, actions, and transitions.[4] This serves as the true
metric minimized by the DRM.

We note that our RL based estimate of out-of-sample dynamic risk is a novel approach, which tackles
the important challenge of policy evaluation in RL with dynamic risk measures.

Table 1 summarizes the evaluations of out-of-sample dynamic risk for DRM policies trained for 1
year maturity at risk level ⌧ = 90% then applied to options of different maturities ranging from 12

4Note that this metric is available neither for the case of basket option nor in a data-driven environment.


-----

Table 1: The out-of-sample dynamic and static 90%-expectile risk imposed to the two sides of
vanilla at-the-money call options over AAPL.

|Col1|Col2|Time to maturity|
|---|---|---|
|Policy|Est.†|12 11 10 9 8 · · · 4 3 2 1|
|Dynamic 90%-expectile risk|||
|Writer’s DRM|RL DP|0.77 0.73 0.69 0.65 0.62 · · · 0.45 0.38 0.29 0.23 0.75 0.71 0.68 0.65 0.61 · · · 0.43 0.38 0.31 0.23|
|Buyer’s DRM|RL DP|-0.22 -0.21 -0.20 -0.19 -0.18 · · · -0.11 -0.09 -0.07 -0.05 -0.23 -0.22 -0.21 -0.20 -0.18 · · · -0.12 -0.11 -0.08 -0.06|
|Static 90%-expectile risk|||
|Writer’s SRM Writer’s DRM|ED ED|0.55 0.54 0.54 0.53 0.53 · · · 0.48 0.46 0.41 0.31 0.56 0.54 0.52 0.50 0.47 · · · 0.36 0.33 0.29 0.24|
|Buyer’s SRM Buyer’s SRM|ED ED|-0.35 -0.33 -0.30 -0.27 -0.23 · · · -0.09 -0.07 -0.07 -0.06 -0.36 -0.34 -0.32 -0.30 -0.28 · · · -0.18 -0.14 -0.11 -0.06|
|Equal risk prices with DRM|||
|True ERP||0.49 0.47 0.45 0.42 0.40 · · · 0.28 0.24 0.19 0.14|
|DRM SRM|RL RL|0.50 0.47 0.45 0.42 0.40 · · · 0.28 0.24 0.18 0.14 0.49 0.46 0.44 0.43 0.40 · · · 0.30 0.27 0.24 0.22|



_† Estimation (Est.) is either made based on reinforcement learning (RL), discretized dynamic programming_
(DP), or the empirical distribution (ED).

months to 1 months. One can observe that the risk of the writer decreases monotonically for options
of shorter maturities, whereas the risk of the buyer increases monotonically. This is consistent
with the fact that there is less uncertainty for a shorter hedging horizon, which favors the writer’s
risk exposure more than the buyer’s when considering an at-the-money option. This also provides
the evidence that the DRM policies, albeit only trained based on the longest time to maturity, i.e.
one year, can be well applied to hedge options with shorter time to maturity and be used to draw
consistent conclusion. Another important observation one can make is that the RL based out-ofsample dynamic risk estimate is generally very close to the DP based estimate across all conditions.

Table 1 also reports the out-of-sample static risk for both SRM policies and DRM policies. The
results are interesting and perhaps surprising. First, the DRM policies outperform SRM policies in
terms of static risk exposure for short maturities, even though they were trained using a different
risk measure. Second, unlike with DRM, we observed at other risk levels (see Figure 6(e) and (f)
in Appendix) that the static risk of SRM policies for the seller (resp. buyer) can increase (resp.
decrease) when hedging an option with shorter maturity. The possibility that a seller’s policy may
actually increase risk when applied to an option with shorter maturity is clearly problematic here as
it is inconsistent with the fact that there is less uncertainty (and lower expected value) regarding the
payout of such options. Both observed phenomenon are consequences of the fact that SRM violates
the time consistency property. We suspect that the possibility that SRM policies may not account
properly for risk aversion at some future time point or for other range of option maturities should
seriously hinder their use in practice.

Finally, Table 1 reports the equal risk prices calculated based on RL based out-of-sample dynamic
risk estimate and based on the discretized DP (referred as True ERP).[5] One first confirms that the RL
based estimate is of high quality, with a maximum approximation error of 0.01 over all maturities.
Moreover, we can see that the prices for the SRM polices are generally higher than the prices for
the DRM polices, perhaps due to the fact that it is the writer that benefits most from the improved
DRM policy than the buyer, as he is more exposed to tail risks in this transaction. We further refer
the reader to section B.2 of the appendix for additional results regarding the performance of SRM
and DRM in this vanilla option setting.

4.4 BASKET OPTION HEDGING AND PRICING

In our second set of experiments, we extend the application of ERP pricing framework to the case
of basket options where traditionnal DP solution schemes are not computationally tractable. In

5Note that in a real data-driven setting, the ERP could either be estimated using the in-sample trained critic

network, or by calculating our RL based estimate using some freshly reserved data to reduce statistical biases.


-----

particular, we consider an at-the-money basket option with the strike price of 753$ on five underlying
assets: AAPL, AMZN, FB, JPM, and GOOGL, where the option payoff is determined by the average
price of the underlyings. In this section, dynamic risk is only estimated using the RL based estimator
defined in Section 4.3, given that exact DP resolution has become intractable.

Table 2: The out-of-sample dynamic and static 90%-expectile risk imposed to the two sides of basket
at-the-money call options. Associated ERPs under the DRM are also compared.

|Col1|Col2|Time to maturity|
|---|---|---|
|Policy|Est.†|12 11 10 9 8 · · · 4 3 2 1|
|Dynamic 90%-expectile risk|||
|Writer’s DRM|RL|3.92 3.62 3.38 3.15 2.95 · · · 2.00 1.70 1.39 1.10|
|Buyer’s DRM|RL|-0.48 -0.49 -0.51 -0.52 -0.50 · · · -0.47 -0.37 -0.33 -0.29|
|Static 90%-expectile risk|||
|Writer’s SRM Writer’s DRM|ED ED|2.43 2.36 2.28 2.16 2.08 · · · 1.61 1.45 1.26 0.94 2.38 2.28 2.18 2.06 1.96 · · · 1.51 1.39 1.20 0.92|
|Buyer’s SRM Buyer’s SRM|ED ED|-1.31 -1.24 -1.15 -1.01 -0.94 · · · -0.56 -0.48 -0.36 -0.22 -1.39 -1.32 -1.24 -1.13 -1.07 · · · -0.66 -0.56 -0.40 -0.23|
|Equal risk prices with DRM|||
|DRM SRM|RL RL|2.20 2.06 1.95 1.84 1.73 · · · 1.24 1.04 0.86 0.70 2.23 2.10 2.01 1.91 1.79 · · · 1.21 1.03 0.92 0.82|



_† Estimation (Est.) is either made based on reinforcement learning (RL), or the empirical distribution (ED)._

Table 2 presents the dynamic risk obtained from training the DRM policy for a one year maturity
option and applying it on the test data for maturity ranging from 1 to 12 months. Similar to the
vanilla option case, the dynamic risk of the writer is monotonically decreasing as we get closer
to the maturity of the option, while for the writer the monotonic behavior seems to be slightly
perturbed by estimation error. The table also compares the static risk under DRM and SRM. One
can first recognize the same monotone convergence to zero of the two sides of the options. However,
contrary to the case of the vanilla option, the difference between the static risk performance of
DRM and SRM policies are rather similar for all maturity times. It therefore appears that in these
experiments with a basket option, both SRM and DRM produce more similar polices. One possible
reason could be that the range of “optimal” risk averse investment plans, whether using DRM or
SRM, is more limited. Indeed, while for the vanilla option, we observed that the optimal policies
generated investments in the range [0, 1] and [-1, 0] for the writer and the buyer respectively, for
the basket option we observed wealth allocations that are more concentrated around 0.20 (i.e. the
uniform portfolio known for its risk hedging properties) and -0.20 for each of the 5 assets asset
respectively. Finally, Table 2 presents the equal risk prices computed based on our RL based out-ofsample dynamic risk estimator. Once again, the higher ERP price for the SRM policy are notable,
which again can be attributed to the better performing (in terms of dynamic risk) hedging policy
produced by ACRL for the DRM, compared to the policy produced by AORL for the SRM. Further
details are presented in section B.3 of the Appendix.

5 CONCLUSION

Motivated by the application of ERP, in this paper we considered solving risk averse MDP problems
formulated based on dynamic expectile risk measures, and proposed a novel ACRL algorithm that
extends the model-free off-policy deterministic ACRL algorithm to a general finite horizon riskaverse MDP setting. In comparison to existing model-free deep RL methods for solving risk-averse
MDP formulated based on dynamic risk measures, our method is more amenable to practical implementation, allowing for tackling real applications such as the ERP problem. Indeed, as a natural
risk-averse extension of the popular model-free DDPG, our method can easily accommodate any
finite horizon MDP applications solved by DDPG. More in-depth studies of these other applications
are left for future work. The extension of our method to an infinite horizon MDP setting is also
worth investigating further. Finally, the exploration of our method to accommodate other utilitybased shortfall risk measures should also be of great interest for future study.


-----

REFERENCES

Fabio Bellini and Valeria Bignozzi. On elicitable risk measures. Quantitative Finance, 15(5):725–

733, 2015.

Dimitris Bertsimas, Leonid Kogan, and Andrew W Lo. Hedging derivative securities and incomplete

markets: an ✏-arbitrage approach. Operations research, 49(3):372–397, 2001.

Alexandre Carbonneau and Fr´ed´eric Godin. Equal risk pricing of derivatives with deep hedging.

_Quantitative Finance, pp. 1–16, 2020._

Alexandre Carbonneau and Fr´ed´eric Godin. Deep equal risk pricing of financial derivatives with

multiple hedging instruments. arXiv preprint arXiv:2102.12694, 2021.

Dotan Di Castro, J. Oren, and Shie Mannor. Practical risk measures in reinforcement learning.

_ArXiv, abs/1908.08379, 2019._

Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization in MDPs. Ad
_vances in neural infor- mation processing systems, abs/1406.3339:3509–3517, 2014._

Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the

_29th International Coference on International Conference on Machine Learning, ICML’12, pp._
179–186, Madison, WI, USA, 2012. Omnipress.

Ivan Guo and Song-Ping Zhu. Equal risk pricing under convex trading constraints. Journal of

_Economic Dynamics and Control, 76:136–151, 2017._

Audrey Huang, Liu Leqi, Zachary C. Lipton, and Kamyar Azizzadenesheli. On the convergence and

optimality of policy gradient for markov coherent risk, 2021.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,

David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv
_preprint arXiv:1509.02971, 2015._

Saeed Marzban, Erick Delage, and Jonathan Yumeng Li. Equal risk pricing and hedging of financial

derivatives with convex risk measures. arXiv preprint arXiv:2002.02876, 2020.

L.A. Prashanth and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive MDPs.

_Advances in neural infor- mation processing systems, abs/1406.3339:252–260, 2013._

Alexander Shapiro. Interchangeability principle and dynamic equations in risk averse stochastic

programming. Operations Research Letters, 45(4):377–381, 2017.

Yun Shen, Michael J. Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement

learning. Neural Computation, 26(7):1298–1328, 2014.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.

Deterministic policy gradient algorithms. In International conference on machine learning, pp.
387–395. PMLR, 2014.

Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distribu
tional reinforcement learning. In Alexandre M. Bayen, Ali Jadbabaie, George Pappas, Pablo A.
Parrilo, Benjamin Recht, Claire Tomlin, and Melanie Zeilinger (eds.), Proceedings of the 2nd
_Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine_
_Learning Research, pp. 958–968, 2020._

Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for co
herent risk measures. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.),
_Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015._

N´uria Armengol Urp´ı, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learn
ing. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.


-----

