# EVALUATING ROBUSTNESS OF COOPERATIVE MARL

**Anonymous authors**
Paper under double-blind review

ABSTRACT

In recent years, a proliferation of methods were developed for multi-agent reinforcement learning (MARL). In this paper, we focus on evaluating the robustness
of MARL agents in continuous control tasks. In particular, we propose the first
model-based approach to perform adversarial attacks for continuous MARL. The
attack aims at adversarially perturbing the states of agent(s) to misguide them to
take damaging actions that lower team rewards. A deep neural network is trained
to represent the dynamics of the environment. We then solve an optimization problem with the learned dynamics model to yield small perturbations. In addition, we
discuss several strategies to optimally select adversary agents for the attack. Numerical experiments on multi-agent Mujoco tasks verify the effectiveness of our
proposed approach.

1 INTRODUCTION

Deep neural networks are known to be vulnerable to adversarial examples, where a small and often
imperceptible adversarial perturbation can easily fool the state-of-the-art deep neural network classifiers (Szegedy et al., 2013; Nguyen et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016).
Since then, a wide variety of deep learning tasks have been shown to also be vulnerable to adversarial attacks, ranging from various computer vision tasks to natural language processing tasks (Jia &
Liang, 2017; Zhang et al., 2020b; Jin et al., 2020; Alzantot et al., 2018).

Perhaps unsurprisingly, deep reinforcement learning (DRL) agents are also vulnerable to adversarial
attacks, as first shown in Huang et al. (2017) for atari games DRL agents. In Huang et al. (2017), the
authors study the effectiveness of adversarial examples on a policy network trained on Atari games
under the situation where the attacker has access to the neural network of the target policy. In Lin
et al. (2017), the authors further investigate a strategically-time attack by attacking trained agents on
Atari games at a subset of the time step. Meanwhile, Kos & Song (2017) use the fast gradient sign
method (FGSM) to generate adversarial perturbation on the A3C algorithm (Mnih et al., 2016) and
explore training with random noise and FGSM perturbation to improve resilience against adversarial
examples. While the above research endeavors focused on actions that take discrete values, another
line of research tackles a more challenging problem on DRL with continuous action spaces (Weng
et al., 2019; Gleave et al., 2019). Specifically, Weng et al. (2019) consider a two-step algorithm
which determines adversarial perturbation to be closer to a target state using a learnt dynamics
model, and Gleave et al. (2019) propose a physically realistic thread model and demonstrates the
existence of adversarial policies in zero-sum simulated robotics games.

While most of the existing DRL attack algorithms focus on the single DRL agent setting[1], in this
work we propose to study the vulnerability of multi-agent DRL, which has been widely applied in
many safety-critical real-world applications including swarm robotics (Dudek et al., 1993), electricity distribution, and traffic control (OroojlooyJadid & Hajinezhad, 2019). In particular, we focus on
the collaborative multi-agent reinforcement learning (c-MARL) setting, where a group of agents is
trained to generate joint actions to maximize certain (team) reward function. We note that c-MARL
is a more challenging yet interesting setting than the single DRL agent setting, as the interactions
between agents commands consideration of additional layers of complications.

**Contribution.** Our contribution can be summarized as follows:

1We are aware of only one recent work (Lin et al., 2020) on attacking c-MARL agents with discrete action
space, rather than continuous action space considered in our work.


-----

-  We propose a new adversarial attack framework on MARL with continuous action space,
where we name it cMBA (model-based attack on c-MARL). The attack comprises two
steps: learning a representation of the environment dynamics and then determining proper
observation perturbations based on the learned dynamics model.

-  We formulate the process of selecting an victim agent as a mixed-integer programming
problem and propose an approximate formulation that can be efficiently solved by a firstorder method.

-  We evaluate our attack framework with two baselines and four multi-agent MuJoCo tasks
(Peng et al., 2020), with number of agents ranging from 2 to 6, to verify the effectiveness
of our approach. Our model-based attack consistently outperforms the two baselines in all
tested environments.

**Paper outline. Section 2 discusses related works in adversarial attacks for MARL along with the**
problem settings. We describe the main attack framework cMBA in Section 3, including the training
of the dynamics model and solving the sub-optimization problem. In Section 3.3, we introduce a
mixed-integer program to find the optimal set of victim agents and propose an approximate formulation that can be efficiently solved using a first-order optimization method. Section 4 presents the
evaluation of our approach on several multi-agent MuJoCo environments. Therein, we also study
the performance of different variants of our model-based attack as discussed in Section 3.3.

2 RELATED WORK

**Adversarial attacks for DRL agents.** In the c-MARL setting, although there exists a significant
literature focusing on the adversarial training of MARL agents (Phan et al., 2020; 2021; Zhang et al.,
2020a), yet a very few of them addresses the scenario of adversarial attacks in c-MARL. Lin et al.
(2020) propose a two-step attack on a single agent in a c-MARL environment, where they extend
existing methods to generate adversarial examples guided by a learned adversarial policy. The attack
in Lin et al. (2020) is evaluated under the StarCraft Multi-Agent Challenge (SMAC) environment
(Samvelyan et al., 2019) where the action spaces are discrete. To the best of our knowledge, there
has not been work considering adversarial attacks on the c-MARL setting with continuous action
spaces. The closest approach to ours is the enchanting attack in Lin et al. (2017) and the modelbased attack in Weng et al. (2019); however, these two works focused on single DRL agent setting.
The enchanting attack in Lin et al. (2017) uses a video prediction model to predict the future state
and finds perturbation that minimizes a distance function between the predicted future state and a
target state. Similarly, Weng et al. (2019) consider two formulations for observation and action
perturbation that incorporates a learned dynamic model for future state prediction. In addition to the
model-based approach, in this paper, we consider the selection of victim agents, which is unique to
the multi-agent DRL setting.

**Training c-MARL agents.** One simple approach to train c-MARL agents is to let each agent learn
their own policy independently as in Independent Q-Learning (IQL) (Tan, 1993; Tampuu et al.,
2017). However, this training strategy does not capture the interaction between agents. Alternatively, one can follow a common paradigm called centralized training with decentralized execution
(CTDE) (Oliehoek et al., 2008; Oliehoek & Amato, 2016; Kraemer & Banerjee, 2016). Gupta et al.
(2017) extend DQN (Mnih et al., 2015), TRPO (Schulman et al., 2015), and DDPG (Lillicrap et al.,
2015) to c-MARL, where a parameter sharing neural network policy is trained in a centralized fashion but still allows decentralized behavior across agents. Similar to this, COMA (Foerster et al.,
2018) is an actor-critic method that uses a centralized critic to estimate counterfactual advantage
for decentralized policies. While COMA uses a single critic for all agents, MADDPG (Lowe et al.,
2017) learns a centralized critic for each agent and evaluates on continuous control. Instead of having fully centralized critic, DOP (Wang et al., 2020b) and FACMAC (Peng et al., 2020) apply the
value decomposition idea (Sunehag et al., 2017; Rashid et al., 2018; 2020; Son et al., 2019; Wang
et al., 2020a) into multi-agent actor-critic by having a centralized but factorized critic.


-----

3 CMBA: MODEL-BASED ATTACK FOR C-MARL

3.1 PROBLEM SETTING

We consider multi-agent tasks with continuous action spaces modeled as a Decentralized Partially
Observable Markov Decision Process (Dec-POMDP) (Oliehoek & Amato, 2016). A Dec-POMDP
has a finite set of agents N ≡{1, · · ·, n} associated with a set of states S describing global states,
a set of continuous actions Ai, and a set of individual partial observation Oi for each agent i ∈N .
Given the current observationπRPs[t]θi[+1]ii : : : S × A S × A O∈Si →A. In addition, Dec-POMDP is associated with an initial state distributionii →S →i. The next state agentR and observes a local observation. After that, agent o[i]t _[∈O][i] i[, the action] i receives a reward moves to is determined by the state transition function[ a]t[i] o[∈A][i]t+1_ _r[i][∈O][is selected by a parametrized policy]t[i]_ [calculated from a reward function][i] [correlated with the global state] P0 and a discount
factor γ. In this setting, each agent tries to find the policy that maximizes its own total expected
return Ri = _t=0_ _[γ][t][r]t[i]_ [where][ T][ is the sampling horizon.]

3.2 PROBLEM FORMULATION[P][T]

Our goal is to generate adversarial perturbations imposed to the victim agents’ input (state) in order
to deteriorate the total team reward. The added noise encourages the victim agents’ observation to
be close to a desired adversarial state corresponding to low reward. To avoid sampling from the
environment, we use a pre-trained model that learns the dynamics of the environment to predict
the next state from the perturbed observation and current action, then find the suitable noise that
minimizes the distance between the predicted next state and a predefined target state. This process
can be formulated as an optimization problem as follows.

Formally, we consider a multi-agent setting with |N| = n agents, each agent i ∈N observes state
_s[i]t_ [locally and takes action][ a]t[i][. Let][ s][t] [= (][s][1]t _[,][ · · ·][, s][n]t_ [)][ ∈S][ be the global state at time step][ t][ which]
is concatenated from local states s[i]t [for each agent][ i][ = 1][,][ · · ·][, n][. We also denote the joint action]
_aat time stept = (a[1]t_ _[,][ · · ·] t[, a], i.e. the set of agents that can be attacked. Let[n]t_ [)][ concatenated from each agent’s action][ a]t[i][. Let] fd[ L] : S × A →S[t] _[∈N][ be the set of victim agents] be a parametrized_
function that approximates the dynamics of the environment, where A is the set of concatenated
actions, one from each _i. We assume that a pre-trained MARL policy π(_ ) is given. Let starget
_A_ _·_
be the target state which can lead to poor performance to the agent. We denote ϵ as an upper bound
on budget constraint w.r.t some p-norm _p. The state perturbation ∆s = (∆s[1],_ _, ∆s[n]) (we_
_∥·∥_ _· · ·_
suppress the dependence on t of ∆s to avoid overloading the notation) to st is the solution to the
following problem
min _d(ˆst+1, starget)_
∆s=(∆s[1],···,∆s[n])

s.t. _sˆt+1 = fd(st, at)_
_a[i]t_ [=][ π][(][s]t[i] [+ ∆][s][i][)] (1)
∆s[i] = 0, ∀ _i /∈Lt_
_ℓ_ _st + ∆s_ _u_
_S∆ ≤s_ _p_ _ϵ_ _≤_ _S_
_∥_ _∥_ _≤_

where 0 is a zero vector, and the observation space is constrained within the (vectorized) intervals
between ℓ and u .
_S_ _S_

Let us first provide some insights for the formulation (1). For each agent, using the trained policy π,
we can compute the corresponding action a[i]t [given its (possibly perturbed) local state][ s]t[i] [or][ s]t[i] [+∆][s][i][.]
From the concatenated state-action pair (st, at), we can predict the next state ˆst+1 via the learned
dynamics model fd. Then by minimizing the distance between the predicted next state and the target
state under the budget constraint, we are forcing the agent to move closer to a damaging state in the
next transition. We also note that the last equality constraint indicates that we only allow perturbing
agents within the victim set _t._
_L_

Problem (1) can be efficiently solved by proximal-gradient-based methods. Firstly, by substituting
the definition of at into ˆst+1, (1) is equivalent to


min _d(fd(st, π(st + x)), starget)_
_x_ (2)
s.t. _x_ _t_
_∈C[p]_


-----

where Ct[p] [:=][ {][x][ = (][x][1][,][ · · ·][, x][n][) :][ ∥][x][∥]p _[≤]_ _[ϵ, ℓ][S]_ _[−]_ _[s][t]_ _[≤]_ _[x][ ≤]_ _[u][S]_ _[−]_ _[s][t]_ [and][ x][i] [=][ 0][ for][ i /]∈Lt}. If
we choose the distance function as d(a, b) = ∥a − _b∥[2], we can use the projected gradient descent_
(PGD) algorithm (Nesterov, 2003) to solve (2). The PGD iteration to update yk at iteration k can be
described as
_yk+1 = projCt[p]_ [[][y][k][ −] _[η][∇][y][d][(][f][d][(][s][t][, π][(][s][t][ +][ y][))][, s][target][)]]_ (3)

where projCt[p] [(][·][)][ is the projection to the convex set][ C]t[p] [and][ η][ is the learning rate. The projection is]
simple to calculate since Ct[p] [is the intersection of a unit ball in the][ p][-norm and a box.]

The whole attack process can be summarized in Algorithm 1.

**Algorithm 1 cMBA algorithm at timestep t**

1: Initialization:

Given st, starget, π, fd, _t; initialize ∆s0; choose learning rate η > 0_
_L_

2: For k := 0, · · ·, K − 1 do
3: Compute at = (a[1]t _[,][ · · ·][, a]t[n][)][ as]_

_π(sit_ [+ ∆][s][i][)] if i _t_
_a[i]t_ [=] _∈L_
_π(s[i]t[)]_ otherwise.



4: Compute ˆst+1 = fd(st, at)

5: Update ∆s as
∆sk+1 = proj _t [∆sk_ _η_ ∆sd(ˆst+1, starget)]
_C_ _−_ _∇_

6: End For

**Learning dynamics model. The most important factor that affects the solution of (2) is the quality**
of the learned dynamics model fd. If the dynamics is known, we can solve (2) exactly so that
the solution is indeed the optimal perturbation. However, it is hardly the case in practice because
we often approximate the dynamics using function approximators such as neural networks. The
parameter w for fd is the solution of the following optimization problem

min _fd(s[i]cur[, w][)][ −]_ _[s]next[i]_ (4)
_w_

_iX∈D_

[2]

where D is a collection of state transition {(s[i]cur[, s][i]next[)][}][|D|]i=1 [where][ s][next][ is the actual state that]
the environment transitions to after taking action at determined by a given policy. In particular, we
separately collect transitions using the pre-trained policy π and a random policy to obtain _train_
_D_
and _random. Then the dataset_ is built as = _train_ _random. Problem (4) is a standard_
supervised learning problem and can be solved using gradient-based learning. We describe the full D _D_ _D_ _D_ _∪D_
process of training the dynamics model in Algorithm 2 where the GradientBasedUpdate step
is any gradient-descent-type update.


**Algorithm 2 Training dynamics model**

1: Initialization:

Given pre-trained policy πtr and a random policy πrd; initialize w0

2: Formpolicy D π :=tr and D πtrainrd, respectively. ∪Drandom by collecting a set of transitions Dtrain and Drandom using

3: For k := 0, 1, · · · do

_wk+1 = GradientBasedUpdate(D, wk)_

4: End For


3.3 VICTIM AGENT SELECTION

In this subsection, we discuss different strategies to select victim agents when performing adversarial
attacks in the MARL setting. Suppose that we would like to perform an attack on any agent but the
budget on the victim agents is na _n. In other words, an attacker can only choose a subset of_
_≤_


-----

agents to attack during each time step, and the goal of an attacker is to choose the most vulnerable
agents so that the total reward is utmost affected. We note that this scenario is unique in the setting
of multi-agent DRL setting, as in the single DRL agent setting we can only attack the same agent all
the time. To start with, we first formulate a mixed-integer program to perform the attack on a set of
optimally victim agents as below:
min∆s,w _d(ˆst+1, starget)_
s.t. _sˆt+1 = fd(st, at)_
_at = π(st + ⟨∆s, w⟩)_
_ℓ_ _st + ∆s_ _u_ (5)
_S∆ ≤s_ _p_ _ϵ_ _≤_ _S_
_w∥_ _i_ _∥_ _≤0, 1_
_∈{_ _}_

_i_ _[w][i][ =][ n][a]_

where we introduce a new binary variable w to select the appropriate agents’ input to perturb and

P

_na is the total number of victim agents we can attack._

Due to the existence of the new binary variable, problem (5) is much harder to solve than before.
We instead solve a proxy of (5) as follows
min∆s,θ _d(ˆst+1, starget)_
s.t. _sˆt+1 = fd(st, at)_
_at = π(st +_ ∆s, W (st, θ) )
_⟨_ _⟩_ (6)
_ℓ_ _st + ∆s_ _u_
_S∆ ≤s_ _p_ _ϵ_ _≤_ _S_
0∥ _∥W ≤(st, θ)_ 1
_≤_ _≤_

where W (·, θ) is a function parametrized by θ that takes current state as input and outputs the
weight to distribute the noise to each agent. Suppose we represent W (·, θ) by a neural network, we
can rewrite the formulation (6) as
min∆s,θ _d(fd(st, π(st +_ ∆s, W (st, θ) )), starget)
_⟨_ _⟩_ (7)
s.t. ∆s _t_
_∈C[p]_

because the last constraint in (6) can be enforced by using a softmax activation in the neural network
_W_ (·, θ). As a result, (7) can be efficiently solved by using PGD. We present the pseudo-code of
the attack in Algorithm 3. After K steps of PGD update, we define the index in _j as the j-th_
_−_
largest value within W (st, θK) R[n], i.e. we have Wi(n) (st, θK) _Wi(n_ 1) (st, θK)
_∈_ _≥_ _−_ _≥· · · ≥_
_Wi(1)(st, θK). Let Ij be the index set of top-j largest outputs of the W_ (st, θK) network. The

final perturbation returned by our victim agent selection strategy will be ∆s = ((∆s)[1], · · ·, (∆s)[n])
where (∆s)[i] = 0 if i /∈Ina and (∆s)[i] = (∆sK)[i] if i ∈Ina .

[c] [c] [c]

**Algorithm 3 cMBA with victim agent selection at timestep t**

[c] [c]

1: Initialization:

Given st, starget, π, fd, na; initialize ∆s0; choose learning rate η > 0 λ > 0.

2: For k := 0, · · ·, K − 1 do
3: Compute at = π(st + ⟨∆s, W (st, θ)⟩)

4: Compute ˆst+1 = fd(st, at)

5: Update ∆s: ∆sk+1 = projCt[p] [[∆][s][k][ −] _[η][∇][∆][s][d][(ˆ]st+1, starget)]_

7:6: End ForUpdate θ: θk+1 = θk − _λ∇θd(ˆst+1, starget)_
8: Compute _na =_ _i(n),_ _, i(n_ _na)_ such that
_I_ _{_ _· · ·_ _−_ _}_

_Wi(n)_ (st, θK) ≥ _Wi(n−1)_ (st, θK) ≥· · · ≥ _Wi(1)(st, θK)_


9: Return ∆s = ((∆s)[1], · · ·, (∆s)[n]) such that

[c] [c] [c] (∆s)[i] = (∆sK)i if i ∈Ina

**0** otherwise


[c]

**Remark 3.1 For this attack, we assume each agent i has access to the other agent’s observation to**
_form the concatenated state st._


-----

EXPERIMENTS


We perform the attack on various multi-agent MuJoCo (MA-MuJoCo) environments including
Walker2d (2x3), HalfCheetah (2x3), HalfCheetah (6x1), and Ant (4x2). The pair environment
name (config) indicates the name of MuJoCo environment along with the agent partition,
where a configuration of 2x3 means there are in total 2 agents and each agent has 3 actions. Note
that we cannot directly apply methods such as the fast gradient sign method (FGSM) or JSMA (an
attack using saliency map) as in Lin et al. (2020), since those methods require a “target” action
which is not available in continuous action space. Therefore, we consider two baselines: Uniform
and Gaussian baselines where the perturbation follows either Uniform distribution U (−ϵ, ϵ) or
Normal distribution N (0, ϵ).

**Variants of model-based attack. We consider the following variants of our model-based attack:**

-  Model-based attack on fixed agents: perform Algorithm 1 using a fixed set of victim agents
_L._

-  Best model-based attack on fixed agents: among the model-based attack on fixed agents,
pick the subset that achieves the best performance.



-  Model-based attack on random agents: perform Algorithm 1 with _t, which is sampled_
_L_
uniformly from N such that |Lt| = na (na is the total victim agents).

-  Model-based attack with learned victim selection: perform Algorithm 3 to select vulnerable
agents and perform the attack on them.

-  Greedy victim selection: perform a sweep over subsets of agents with size na. For each
subset, perform Algorithm 1 to obtain proper perturbation and retrieve the corresponding
objective value (distance between predicted state and target state). Select the subset corresponding to the lowest objective value as the victim agents.

**Experiment setup. We first use MADDPG (Lowe et al., 2017) to train MARL agents for the**
four MA-MuJoCo environments listed above. Using the trained agents, we collect datasets containing one million transitions to train the dynamics model for each environment. The dynamics
model is a fully connected neural network with three hidden layers of 1000 neurons. We also use
a fully-connected neural network for W (st, θ) in (6) with two hidden layers of 200 neurons. We
use AdamW (Loshchilov & Hutter, 2017) as the optimizer and select the best learning rate from
_{1, 5} × {10[−][5], 10[−][4], 10[−][3]} (the best learning rate is the one achieving lowest prediction error on_
a test set of 80, 000 samples). For our model-based attack, we run PGD for K = 30 steps to solve
(2) and (5). We perform each attack over 16 episodes then average the rewards. We also illustrate
the standard deviation of rewards using the shaded area in the plots.

**Model-free baselines vs model-based attack. In this experiment, we run the two baseline attacks**
along with our model-based attack on the four MA-MuJoCo environments, in which there is only
one victim agent (na = 1). Figure 1 illustrates the performance when we perform these attacks
on specific agents under multiple budget levels using ℓ -norm. For a fixed agent, our model-based
_∞_
attack consistently outperforms the two baselines. In particular, our model-based attack yields much
lower rewards under low budget constraints (small ϵ) compared to the two baselines.

To better visualize the performance difference, 2 illustrates the environment with and without attacks
captured at different time-steps. From Figure 2, our model-based attack is able to make the MuJoCo
agent flip, which terminates the episode at the 409-th timestep. The episode length and total rewards
for each variant are: No attack(1000, 2644.60), Uniform(1000, 1758.97), Gaussian(891, 1399.51),
**Ours(409, 287.06).**

We also investigate how the state values change during these attacks. Figure 3 presents different
recordings of state values under adversarial attacks compared to no attack. Consider state index 8,
which represents the horizontal velocity of the agent. For the HalfCheetah environment, as the
goal is to make the agent move forward as fast as possible, we expect the reward to be proportional
to this state value. From Figure 3, all three attacks have fairly sparse red fractions across timesteps,
which result in a much lower reward compared to the no-attack setting. Among the three attacks,
our model-based ones appear to have the most sparse red fractions leading to the lowest rewards. In
addition, the model-based attack appears to show its advantage in environments with more agents.


-----

2500

2000

1500

1000

500

0

0.0 0.1 0.2 0.3 0.4 0.5

Mean rewards - Walker2d (2x3) 5000 Mean rewards - HalfCheetah (2x3)

4000

3000

2000

Mean Reward

1000

0

Noise Level, || s||


0.0 0.1 0.2 0.3 0.4 0.5

Noise Level, || s||






Uniform noise - Agent 0 Gaussian noise - Agent 1
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Gaussian noise - Agent 1 cMBA (ours) - Agent 1


Uniform noise - Agent 0 Gaussian noise - Agent 1
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Gaussian noise - Agent 0 cMBA (ours) - Agent 1


Mean rewards - HalfCheetah (6x1)


Mean rewards - HalfCheetah (6x1)

0.0 0.1 0.2 0.3 0.4 0.5

3000

2500

2000

1500

Mean Reward

1000

500

0

Noise Level, || s||


3000

2500

2000

1500


1000

500

0

0.0 0.1 0.2 0.3 0.4 0.5

Noise Level, || s||






Uniform noise - Agent 0 Gaussian noise - Agent 2
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Uniform noise - Agent 2 cMBA (ours) - Agent 1
Gaussian noise - Agent 0 cMBA (ours) - Agent 2
Gaussian noise - Agent 1


Uniform noise - Agent 3 Gaussian noise - Agent 5
Uniform noise - Agent 4 cMBA (ours) - Agent 3
Uniform noise - Agent 5 cMBA (ours) - Agent 4
Gaussian noise - Agent 3 cMBA (ours) - Agent 5
Gaussian noise - Agent 4


Mean rewards - Ant (4x2)


Mean rewards - Ant (4x2)


3000

2000


1000

0

1000


2000



3000

2000

Mean Reward 10000

1000

2000

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200

Noise Level, || s|| Noise Level, || s||





Uniform noise - Agent 0 Gaussian noise - Agent 1
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Gaussian noise - Agent 0 cMBA (ours) - Agent 1


Uniform noise - Agent 2 Gaussian noise - Agent 3
Uniform noise - Agent 3 cMBA (ours) - Agent 2
Gaussian noise - Agent 2 cMBA (ours) - Agent 3


Figure 1: Model based attack vs baseline attacks on fixed agents.


-----

Step 0 Step 80 Step 160 Step 240 Step 320 Step 400 Step 409

No attack

Uniform

Gaussian

Ours

Figure 2: Various attacks on Agent 0 in Ant (4x2) environment with ∥∆s∥∞ _≤_ 0.1. It can be seen
that the agent flip at the end of episode under our model-based attack, demonstrating the effectiveness of our algorithm.


In particular, our approach results in lower rewards under a smaller budget as seen in HalfCheetah
(6x1) and Ant (4x2) environments.


HalfCheetah_2x3 - No attack (Agent 0), Reward: 4536.39

200 400 600 800

Timesteps


16

14

12

10

8

6

4

2

0

16

14

12

10


10

20


20

10

0



10

20


200 400 600 800

HalfCheetah_2x3 - Uniform noise, || s|| 0.2 (Agent 0), Reward: 2209.57

Timesteps


20

10

0


16

14

12

10

8

6

4

2

0

16

14

12

10


10

20


200 400 600 800

HalfCheetah_2x3 - Gaussian noise, || s|| 0.2 (Agent 0), Reward: 1782.00

Timesteps


20

10

0


10

20


0 200 400 600 800

20

10

0

HalfCheetah_2x3 - cMBA (ours), || s|| 0.2 (Agent 0), Reward: 1112.37

Timesteps

Figure 3: Recordings of state values in an episode under different attack on Agent 0 in HalfCheetah(
2x3) environment.


-----

**Effectiveness of learned victim selection. We further demonstrate the performance of our model-**
based attack with and without learned victim selection on one victim agent. Figure 4 illustrates
the performance of four model-based variants: the best performing model-based attack on the fixed
agent (Best Fixed Agent), model-based attack on the random agent (Random Agent), model-based
with learned victim selection as in Algorithm 3 (Learned victim Selection), and model-based with
greedy victim selection (Greedy). Surprisingly, even though the greedy approach employs a bruteforce strategy to select the agent with the closest distance, its performance reveals that selecting
victim agents greedily might not be a good strategy. The learned victim selection seems to be
better than the random or greedy strategy and is comparable with the random one in HalfCheetah
environment. The best-fixed agent variant has good performance under low budget constraints in
HalfCheetah (6x1) and Ant (4x2) environments. We want to emphasize that although the learned
victim selection strategy in Section 3.3 is sub-optimal, it works well in these four environments.


Our cMBA Variants - Walker2d (2x3)


Our cMBA Variants - HalfCheetah (2x3)


3000

2500

2000

1500

1000

500


5000


0.0 0.1 0.2 0.3 0.4 0.5

4000

3000

2000

Mean Reward

1000

0

Noise Level, || s||

Best Fixed Agent Learned Victim Selection
Random Agent Greedy


Our cMBA Variants - HalfCheetah (6x1)


0.0 0.1 0.2 0.3 0.4 0.5


Noise Level, || s||

Best Fixed Agent Learned Victim Selection
Random Agent Greedy


Our cMBA Variants - Ant (4x2)



3000

2500

2000

1500

1000

500


3000

2000

1000

Mean Reward 0

1000

20000.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200

0.0 0.1 0.2 0.3 0.4 0.5 Noise Level, || s||

Noise Level, || s||

Best Fixed Agent Learned Victim Selection

Best Fixed Agent Learned Victim Selection Random Agent Greedy
Random Agent Greedy

Figure 4: Performance of different variants of our proposed model-based attack (c-MBA) with different choices on victim agents.


5 CONCLUSIONS

In this paper, we propose a new attack algorithm named cMBA for evaluating the robustness of
c-MARL environment with continuous action space. Our cMBA algorithm is the first to consider
the c-MARL and the adversarial perturbation is computed by solving an optimization problem with
learned dynamics models. The cMBA approach outperforms the other baselines attack by a large
margin under 4 multi-agent MuJoCo environments especially ones with larger number of agents.
Unique to multi-agent setting, we also study different strategies to select victim agents. Extensive
experiment results on standard c-MARL benchmarks show that our proposed model-based attack
and victim-selection strategy can successfully degrade the performance of well-trained c-MARL
agents while outperforming other baselines by a large margin. A future direction is to consider a
timed attack strategy where the perturbation is added at certain timesteps. We can also consider
longer planning step, i.e. trying to reach a target state within the next T steps instead of 1.


-----

ETHICS STATEMENT

This paper does not contain ethics concerns.

REFERENCES

Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei
Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998,
2018.

Gregory Dudek, Michael Jenkin, Evangelos Milios, and David Wilkes. A taxonomy for swarm
robots. In Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and
_Systems (IROS’93), volume 1, pp. 441–447. IEEE, 1993._

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.
Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial
_Intelligence, volume 32, 2018._

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615, 2019.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
examples. arXiv preprint arXiv:1412.6572, 2014.

Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using
deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent
_Systems, pp. 66–83. Springer, 2017._

Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks
on neural network policies. arXiv preprint arXiv:1702.02284, 2017.

Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems.
_arXiv preprint arXiv:1707.07328, 2017._

Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline
for natural language attack on text classification and entailment. In Proceedings of the AAAI
_conference on artificial intelligence, volume 34, pp. 8018–8025, 2020._

Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. _arXiv preprint_
_arXiv:1705.06452, 2017._

Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for
decentralized planning. Neurocomputing, 190:82–94, 2016.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv
_preprint arXiv:1509.02971, 2015._

Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On
the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and
_Privacy Workshops (SPW), pp. 62–68. IEEE, 2020._

Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748,
2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. _arXiv preprint_
_arXiv:1711.05101, 2017._

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actorcritic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275, 2017.


-----

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level
control through deep reinforcement learning. nature, 518(7540):529–533, 2015.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim
Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement
learning. In International conference on machine learning, pp. 1928–1937. PMLR, 2016.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer
Science & Business Media, 2003.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer
_vision and pattern recognition, pp. 427–436, 2015._

Frans A Oliehoek and Christopher Amato. _A concise introduction to decentralized POMDPs._
Springer, 2016.

Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289–353, 2008.

Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. arXiv preprint arXiv:1908.03963, 2019.

Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram
Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European sympo_sium on security and privacy (EuroS&P), pp. 372–387. IEEE, 2016._

Bei Peng, Tabish Rashid, Christian A Schroeder de Witt, Pierre-Alexandre Kamienny, Philip HS
Torr, Wendelin B¨ohmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy
gradients. arXiv preprint arXiv:2003.06709, 2020.

Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein,
Horst Sauer, Reiner Schmid, Jan Wieghardt, Marc Zeller, et al. Learning and testing resilience
in cooperative multi-agent systems. In Proceedings of the 19th International Conference on Au_tonomous Agents and MultiAgent Systems, pp. 1055–1063, 2020._

Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia LinnhoffPopien. Resilient multi-agent reinforcement learning with adversarial value decomposition. In
_Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11308–11316,_
2021.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and
Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4295–4304. PMLR, 2018.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob N Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. J. Mach. Learn. Res., 21:178–1, 2020.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas
Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon
Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference on machine learning, pp. 1889–1897. PMLR,
2015.

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning
to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna_tional Conference on Machine Learning, pp. 5887–5896. PMLR, 2019._


-----

Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max
Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition
networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow,
and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.

Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan
Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning.
_PloS one, 12(4):e0172395, 2017._

Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings
_of the tenth international conference on machine learning, pp. 330–337, 1993._

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033.
IEEE, 2012.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020a.

Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Off-policy multiagent decomposed policy gradients. arXiv preprint arXiv:2007.12322, 2020b.

Tsui-Wei Weng, Krishnamurthy Dj Dvijotham, Jonathan Uesato, Kai Xiao, Sven Gowal, Robert
Stanforth, and Pushmeet Kohli. Toward evaluating robustness of deep reinforcement learning
with continuous control. In International Conference on Learning Representations, 2019.

Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sunil Mallya, and Tamer Basar. Robust multiagent reinforcement learning with model uncertainty. In NeurIPS, 2020a.

Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on
deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent
_Systems and Technology (TIST), 11(3):1–41, 2020b._


-----

A MODE DETAILS ON EXPERIMENT SETUP IN SECTION 4

**Specifying target state for each environment. To perform our model based attack, we need to**
specify a target state that potentially worsens the total reward. In multi-agent MuJoCo environments,
each agent has access to its own observation of the agent consisting the position-related and velocityrelated information. The position-related information includes part of x, y, z coordinates and the
quarternion that represents the orientation of the agent. The velocity-related information contains
global linear velocities and angular velocities for each joint in a MuJoCo agent. We refer the reader
to Todorov et al. (2012) for more information about each MuJoCo environment. Now we describe
the design of this target state for each environment as follows:

-  Walker (2x3) environment: Since the episode ends whenever the agent falls, i.e. the z
coordinate falls below certain threshold. In this environment, the target state has a value of
0 for the index that corresponds to the z coordinate of the MuJoCo agent (index 0).

-  HalfCheetah (2x3) and HalfCheetah (6x1) environments: As the goal is to make agent
moves as fast as possible, we set the index corresponding to the linear velocities to 0 (index
8).

-  Ant (4x2) environment: As the agent can move freely in a 2D-plan, we set the index
corresponding to the x, y linear velocities to 0 (indices 13 and 14).

B ADDITIONAL EXPERIMENTS

In this section, we present experimental results in addition to ones presented in Section 4.

Figure 5 illustrates the environment with and without attacks captured at different timesteps. From
Figure 5, our model based attack is able to make the MuJoCo agent fall down which terminates
the episode at the 65-th timestep. The episode length and total rewards for each variant are: No
attack(478, 1736.44), Uniform(382, 1037.73), Gaussian(90, 32.24), Ours(63, −34.26). Figure 6
illustrates how the state values change during an episode. These state values correspond to the agent
shown in Figure 5. If we look at how the noise values change as in Figure 7, the noise generated by
our approach appears to maximize the permissible noise budget.

In addition to the ℓ∞-norm budget constraint, we also evaluate adversarial attacks using the ℓ1norm constraint. Note that using ℓ1-norm for budget constraint is more challenging as the attack
needs to distribute the noise across all states while in the ℓ -norm the computation of perturba_∞_
tion for individual state is independent. Figure 8 illustrates an episode of a Walker agent with and
without attack. Our cMBA approach is able to make the agent fall at the 255-th timestep with the
episode length and rewards for each setting as: No attack(661, 2513.57), Uniform(403, 1144.77),
Gaussian(455, 1637.16), Ours(253, 759.85). From the noise values in Figure 7, our cMBA
method appears to put noise on a few selective states at each timestep, similar to the Gausian noise
setting.

**Additional experiments using the approach in Lin et al. (2020) for continuous action spaces:**
In this experiment, we follow the approach in Lin et al. (2020) in the Ant (4x2) environment where
we train an adversarial policy for one agent trying to minimize the total team reward while the
remaining agents use the trained MARL policy. The adversarial policy is trained for 1 million
timesteps. The results using this approach compared with our approach and the two baselines are
presented in Figure 11.

From Figure 11, our cMBA approach outperforms Lin et al. (2020)’s approach as it is able to achieve
lower team rewards when attacking the same agent. We also note that Lin et al. (2020)’s approach
is better than the two baselines using Uniform and Gaussian noise.

**Effect of using ∥·∥1 budget constraint:** In this experiment, we replace the ∥·∥∞ by the ∥·∥1 for
budget constraint. Using ∥·∥1 constraint will produce a significant different adversarial noise pattern


-----

Step 0 Step 15 Step 30 Step 45 Step 60 Step 65

Figure 5: Various attacks on Agent 0 in Walker (2x3) environment with ∥∆s∥∞ _≤_ 0.2.


Walker_2x3 - No attack (Agent 0), Reward: 1736.44

10 20 30 40 50 60

Timesteps


10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0


16

14

12

10

8

6

4

2

0

16

14

12

10



10 20 30 40 50 60

Walker_2x3 - Uniform noise, || s|| 0.2 (Agent 0), Reward: 1037.73

Timesteps


10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0


16

14

12

10

8

6

4

2

0

16

14

12

10


10 20 30 40 50 60

Walker_2x3 - Gaussian noise, || s|| 0.2 (Agent 0), Reward: 32.24

Timesteps

10 20 30 40 50 60

Walker_2x3 - cMBA (ours), || s|| 0.2 (Agent 0), Reward: -34.26

Timesteps


Figure 6: Record of state values in an episode under different attack on Agent 0 in Walker (2x3)
environment with ∥∆s∥∞ _≤_ 0.2.


-----

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.20

0.15

0.10

0.05

0.00

0.05

0.10

0.15

0.20


16

14

12

10

8

6

4

2

0

16

14

12

10


10 20 30 40 50 60

Walker_2x3 - Uniform noise, || s|| 0.2 (Agent 0), Reward: 1037.73

Timesteps


10 20 30 40 50 60

Walker_2x3 - Gaussian noise, || s|| 0.2 (Agent 0), Reward: 32.24

Timesteps


16

14

12

10


10 20 30 40 50 60

Walker_2x3 - cMBA (ours), || s|| 0.2 (Agent 0), Reward: -34.26

Timesteps


Figure 7: Record of noise values in an episode under different attack on Agent 0 in Walker (2x3)
environment with with ∥∆s∥∞ _≤_ 0.2.

Step 0 Step 50 Step 100 Step 150 Step 200 Step 250 Step 255


Figure 8: Various attacks on Agent 0 in Walker (2x3) environment with ∆s 1 0.5.
_∥_ _∥_ _≤_


-----

Walker_2x3 - No attack (Agent 0), Reward: 2513.57

50 100 150 200 250

Timesteps


5

10


16

14

12

10

8

6

4

2

0

16

14

12

10

|Col1|10|
|---|---|
||1 5 0|
|||


Walker_2x3 - Uniform noise, || s||1 0.5 (Agent 0), Reward: 1144.77

50 100 150 200 250

Timesteps


5

10

|Col1|10|
|---|---|
||1 5 0|
|||


Walker_2x3 - Gaussian noise, || s||1 0.5 (Agent 0), Reward: 1637.16

50 100 150 200 250

Timesteps


5

10


16

14

12

10

8

6

4

2

0

16

14

12

10

|Col1|10|
|---|---|
||1 5 0|
|||


Walker_2x3 - cMBA (ours), || s||1 0.5 (Agent 0), Reward: 759.85

50 100 150 200 250

Timesteps


5

10

|Col1|10|
|---|---|
||1 5 0|
|||


Figure 9: Record of state values in an episode under different attack on Agent 0 in Walker (2x3)
environment with with ∆s 1 0.5.
_∥_ _∥_ _≤_


-----

Walker_2x3 - Uniform noise, || s||1 0.5 (Agent 0), Reward: 1144.77


16

14

12

10

8

6

4

2

0

16

14

12

10


0.4

0.2

0.0

0.2

0.4


50 100 150 200 250

Timesteps

Walker_2x3 - Gaussian noise, || s||1 0.5 (Agent 0), Reward: 1637.16

|Col1|0 0 0|
|---|---|
|||
|||


0.4

0.2

0.0

0.2

0.4


50 100 150 200 250

Timesteps

Walker_2x3 - cMBA (ours), || s||1 0.5 (Agent 0), Reward: 759.85

|Col1|0 0 0|
|---|---|
|||
|||


16

14

12

10


0.4

0.2

0.0

0.2

0.4

|Col1|0 0 0|
|---|---|
|||
|||


0 50 100 150 200 250

Timesteps

Figure 10: Record of noise values in an episode under different attack on Agent 0 in Walker (2x3)
environment with with ∆s 1 0.5.
_∥_ _∥_ _≤_


Mean rewards - Ant (4x2)


Mean rewards - Ant (4x2)


3000

2000


1000

0

1000


2000



3000

2000

Mean Reward 10000

1000

2000

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200

Noise Level, || s|| Noise Level, || s||





Uniform noise - Agent 0 Lin et al (2020) - Agent 0
Uniform noise - Agent 1 Lin et al (2020) - Agent 1
Gaussian noise - Agent 0 cMBA (ours) - Agent 0
Gaussian noise - Agent 1 cMBA (ours) - Agent 1


Uniform noise - Agent 2 Lin et al (2020) - Agent 2
Uniform noise - Agent 3 Lin et al (2020) - Agent 3
Gaussian noise - Agent 2 cMBA (ours) - Agent 2
Gaussian noise - Agent 3 cMBA (ours) - Agent 3


Figure 11: Adversarial attacks using ∥·∥∞ budget constraint in Ant (4x2) environment.


-----

compared to ∥·∥∞ as producing adversarial noise when using ∥·∥∞ is independent for each state
while it is not the case for 1. We run different attacks on two environments, HalfCheetah (6x1)
_∥·∥_
and Ant (4x2), and the results are presented in Figure 12 and Figure 13. From these figures, cMBA
outperforms other baselines. In particular, it is able to achieve much lower rewards under smaller
budget constraint which shows the advantage of our approach.


Mean rewards - HalfCheetah (6x1)


Mean rewards - HalfCheetah (6x1)

2 4 6 8

Noise Level, || s||1


3000

2500

2000

1500

1000

500


3000

2500

2000

1500

1000

500



10


0 2 4 6 8 10

Noise Level, || s||1






Uniform noise - Agent 0 Gaussian noise - Agent 2
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Uniform noise - Agent 2 cMBA (ours) - Agent 1
Gaussian noise - Agent 0 cMBA (ours) - Agent 2
Gaussian noise - Agent 1


Uniform noise - Agent 3 Gaussian noise - Agent 5
Uniform noise - Agent 4 cMBA (ours) - Agent 3
Uniform noise - Agent 5 cMBA (ours) - Agent 4
Gaussian noise - Agent 3 cMBA (ours) - Agent 5
Gaussian noise - Agent 4


Figure 12: Adversarial attacks using ∥·∥1 budget constraint in HalfCheetah (6x1) environment.


Mean rewards - Ant (4x2)


Mean rewards - Ant (4x2)


3000

2500

2000

1500

1000

500

0

500

1000


3000

2500

2000

1500

1000

500

0

500

1000




0 2 4 6 8 10

Noise Level, || s||1


0 2 4 6 8 10

Noise Level, || s||1






Uniform noise - Agent 0 Gaussian noise - Agent 1
Uniform noise - Agent 1 cMBA (ours) - Agent 0
Gaussian noise - Agent 0 cMBA (ours) - Agent 1


Uniform noise - Agent 2 Gaussian noise - Agent 3
Uniform noise - Agent 3 cMBA (ours) - Agent 2
Gaussian noise - Agent 2 cMBA (ours) - Agent 3


Figure 13: Adversarial attacks using ∥·∥1 budget constraint in Ant (4x2) environment.


**Adversarial attacks using dynamics model with various accuracy:** In this test, we compare the
performance of our attack when using trained dynamics model with different mean-squared error
(MSE). We use the Ant (4x2) environment and consider the following 3 dynamics models:

-  The first model is trained using 1 million samples for 100 epochs and select the model with
the best performance. We denote this model ”1M - Best epoch”.

-  The second model is trained using 1 million samples for only 1 epoch. We denote this
model ”1M - 1st epoch”.

-  We further reduce the number of samples used for training the dynamic model to only
200,000 and train for only 1 epoch. We denote this model ”200K - 1st epoch”.


-----

These models are evaluated on a predefined test set consisting of 100,000 samples. The test MSE of
these models are 0.33, 0.69, 0.79, respectively, with the initial test MSE of 1.241.


Effect of dynamics model accuracy - Ant (4x2)

3000

2500

2000

1500

1000

500


Effect of dynamics model accuracy - Ant (4x2)


500

1000


Figure 14: Adversarial attacks using 3 dynamics models in Ant (4x2) environment.

3000

2500

2000

1500

1000

Mean Reward

500

0 0

500

1000

0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.200 0 2 4 6 8 10

Noise Level, || s|| Noise Level, || s||1

1M - Best epoch 1M - 1st epoch 200K - 1st epoch 1M - Best epoch 1M - 1st epoch 200K - 1st epoch


Figure 14 depicts the attacks using these three models on the same agent in Ant (4x2) environment
using ∥·∥∞ budget constraint. Interestingly, the dynamics model trained with only 200,000 samples for 1 epoch can achieve comparable performance with the other two models using much more
samples.


**Comparison between attacking all agents vs attacking 1 agent:** In this experiment, we compare
performance of different approaches when attacking one victim agent or simply perturbing all agents
under the same budget constraint. We compare different approaches in the Ant (4x2) environment.
In particular, we use the two baselines and our cMBA approach to attack each agent out of 4 agent,
denoted as (agent i) for i = 0, 1, 2, 3. We also use these approach in which we simultaneously
perturb the inputs to all agents, denoted as (4 agents). In addition, we also illustrate the performance
of our victim selection scheme when we only want to attack either 1 or two agents at the same
time. We report the mean and standard deviation of rewards using these approach with several
values of budget level ε in Table 1 where we use for the budget constraint. From Table 1,
_∥· ∥∞_
the performance is getting better as the attacker is stronger, i.e. more agents are attacked at the
same time. This makes sense as for, the adversarial noise for each agent can be computed
_∥· ∥∞_
independently.

Now, we consider the ∥· ∥1 budget constraint. This is a more interesting setting as the total noise
budget is fixed the adversarial noise of one agent will affect one from the other. The results are
shown using several values of budget level ε in Table 2. We can observe that in this case attacking
more agents is not always effective.


-----

Table 1: Comparison between adversarial attacks on a single agent and all agents in Ant (4x2)

|environment under ∥· ∥∞≤ε budget c|constraint.|Col3|Col4|Col5|
|---|---|---|---|---|
||Rewards: Mean (standard deviation)||||
|Methods|ε = 0.025|ε = 0.05|ε = 0.075|ε = 0.1|
|Uniform noise (agent 0)|2332 (594)|2296 (88)|1773 (941)|1542 (554)|
|Uniform noise (agent 1)|2233 922)|2033 (565)|1579 (856)|1293 (385)|
|Uniform noise (agent 2)|2028 (695)|1713 (649)|988 (650)|322 (652)|
|Uniform noise (agent 3)|2579 (66)|2484 (73)|2159 (623)|2208 (279)|
|Uniform noise (4 agents)|2089 (450)|1093 (753)|299 (781)|62 (921)|
|Gaussian noise (agent 0)|2450 (134)|2030 (405)|1660 (663)|1067 (536)|
|Gaussian noise (agent 1)|2204 (709)|1992 (337)|1447 (406)|1141 (381)|
|Gaussian noise (agent 2)|2256 (297)|1313 (977)|287 (923)|154 (570)|
|Gaussian noise (agent 3)|2550 (83)|2331 (567)|1927 (1250)|2115 (312)|
|Gaussian noise (4 agents)|2022 (303)|761 (1083)|86 (688)|-133 (809)|
|cMBA (agent 0)|923 (340)|244 (737)|516 (737)|-8 (306)|
|cMBA (agent 1)|1417 (48)|553 (269)|108 (303)|-60 (309)|
|cMBA (agent 2)|227 (955)|-652 (1097)|-241 (660)|-631 (911)|
|cMBA (agent 3)|2024 (87)|861 (1298)|567 (1012)|-172 (725)|
|cMBA (4 agents)|1116 (379)|314 (734)|72 (407)|-239 (281)|
|Leaned Victim Selection (1 agent)|-165 (805)|624 (103)|256 (201)|-2 (123)|
|Learned Victim Selection (2 agent)|405 (308)|-176 (853)|-588 (1026)|-325 (730)|



Table 2: Comparison between adversarial attacks on a single agent and all agents in Ant (4x2)

|environment under ∥· ∥1 ≤ε budget co|onstraint.|Col3|Col4|Col5|
|---|---|---|---|---|
||Rewards: Mean (standard deviation)||||
|Methods|ϵ = 1.5|ϵ = 2.0|ϵ = 2.5|ϵ = 3.0|
|Uniform noise (agent 0)|2189 (151)|2093 (148)|1615 (524)|1421 (780)|
|Uniform noise (agent 1)|2039 (139)|1715 (282)|1387 (586)|1243 (303)|
|Uniform noise (agent 2)|1542 (826)|1073 (755)|581 (920)|436 (317)|
|Uniform noise (agent 3)|2468 (75)|2332 (282)|2306 (131)|2136 (319)|
|Uniform noise (4 agents)|2224 (353)|2037 (412)|1734 (541)|1473 (558)|
|Gaussian noise (agent 0)|1752 (877)|1676 (650)|1431 (774)|1493 (554)|
|Gaussian noise (agent 1)|1775 (344)|1442 (619)|1257 (487)|1036 (554)|
|Gaussian noise (agent 2)|1326 (766)|1049 (666)|776 (763)|516. (842)|
|Gaussian noise (agent 3)|2103 (559)|2319 (104)|2086 (480)|1896 (603)|
|Gaussian noise (4 agents)|1614 (776)|1553 (793)|1544 (568)|1470 (476)|
|cMBA (agent 0)|2068 (145)|1699 (169)|1432 (136)|1282 (136)|
|cMBA (agent 1)|2005 (355)|1682 (346)|1505 (222)|1189 (585)|
|cMBA (agent 2)|1554 (781)|1080 (792)|433 (1202)|513 (987)|
|cMBA (agent 3)|2302 (80)|2032 (528)|2099 (92)|1604 (1069)|
|cMBA (4 agents)|2212 (225)|1909 (332)|1671 (212)|1504 (201)|
|Leaned Victim Selection (1 agent)|706 (1118)|355 (635)|-326 (931)|-139 (608)|
|Learned Victim Selection (2 agent)|1274 (779)|720 (939)|152 (787)|156 (638)|


Rewards: Mean (standard deviation)

Methods _ϵ = 1.5_ _ϵ = 2.0_ _ϵ = 2.5_ _ϵ = 3.0_

Uniform noise (agent 0) 2189 (151) 2093 (148) 1615 (524) 1421 (780)

Uniform noise (agent 1) 2039 (139) 1715 (282) 1387 (586) 1243 (303)

Uniform noise (agent 2) 1542 (826) 1073 (755) 581 (920) 436 (317)

Uniform noise (agent 3) 2468 (75) 2332 (282) 2306 (131) 2136 (319)

Uniform noise (4 agents) 2224 (353) 2037 (412) 1734 (541) 1473 (558)

Gaussian noise (agent 0) 1752 (877) 1676 (650) 1431 (774) 1493 (554)

Gaussian noise (agent 1) 1775 (344) 1442 (619) 1257 (487) 1036 (554)

Gaussian noise (agent 2) 1326 (766) 1049 (666) 776 (763) 516. (842)

Gaussian noise (agent 3) 2103 (559) 2319 (104) 2086 (480) 1896 (603)

Gaussian noise (4 agents) 1614 (776) 1553 (793) 1544 (568) 1470 (476)

cMBA (agent 0) 2068 (145) 1699 (169) 1432 (136) 1282 (136)

cMBA (agent 1) 2005 (355) 1682 (346) 1505 (222) 1189 (585)

cMBA (agent 2) 1554 (781) 1080 (792) 433 (1202) 513 (987)

cMBA (agent 3) 2302 (80) 2032 (528) 2099 (92) 1604 (1069)

cMBA (4 agents) 2212 (225) 1909 (332) 1671 (212) 1504 (201)

Leaned Victim Selection (1 agent) 706 (1118) 355 (635) -326 (931) -139 (608)

Learned Victim Selection (2 agent) 1274 (779) 720 (939) 152 (787) 156 (638)


-----

