# EVADING ADVERSARIAL EXAMPLE DETECTION DEFENSES
## WITH ORTHOGONAL PROJECTED GRADIENT DESCENT

**Oliver Bryniarski[∗]** **Nabeel Hingun[∗]** **Pedro Pachuca[∗]** **Vincent Wang[∗]**
UC Berkeley UC Berkeley UC Berkeley UC Berkeley

**Nicholas Carlini**
Google

ABSTRACT

Evading adversarial example detection defenses requires finding adversarial examples that must simultaneously (a) be misclassified by the model and (b) be
detected as non-adversarial. We find that existing attacks that attempt to satisfy
multiple simultaneous constraints often over-optimize against one constraint at the
cost of satisfying another. We introduce Selective Projected Gradient Descent and
_Orthogonal Projected Gradient Descent, improved attack techniques to generate_
adversarial examples that avoid this problem by orthogonalizing the gradients when
running standard gradient-based attacks. We use our technique to evade four stateof-the-art detection defenses, reducing their accuracy to 0% while maintaining a
0% detection rate.

1 INTRODUCTION

Generating adversarial examples (SZS[+]14; BCM[+]13), inputs designed by an adversary to cause a
neural network to behave incorrectly, is straightforward. By performing input-space gradient descent
(CW17b; MMS[+]17), it is possible to maximize the loss of arbitrary test examples. This process is
both efficient and highly effective. But despite much effort by the community, attempts at designing
defenses against adversarial examples have been largely unsuccessful and gradient-descent attacks
continue to circumvent new defenses (ACW18; TCBM20).

As a result, many defenses aim to make generating adversarial examples more difficult by requiring
additional constraints on inputs for them to be considered successful. Defenses that rely on detection,
for example, will reject inputs if a secondary detector model determines the input is adversarial
(MGFB17; XEQ17). Turning a benign input x into an adversarial example x[′] thus now requires
fooling both the original classifier, f, and the detector, g, simultaneously.

Traditionally, this is done by constructing a single loss function L that jointly penalizes the loss on f
and the loss on g (CW17a), e.g., by defining L(x[′]) = L(f ) + λL(g) and then minimizing L(x[′]) with
gradient descent. Unfortunately, many evaluations using this strategy have had limited success—not
only must λ be tuned appropriately, but the gradients of f and g must also be well behaved.

**Our contributions. We develop a new attack technique designed to construct adversarial examples**
that simultaneously satisfy multiple constraints. Our attack approach is a modification of standard
gradient descent (MMS[+]17) and requires changing just a few lines of code. Given two objective
functions f and g, instead of taking gradient descent steps that optimize the joint loss function f + _λg,_
we selectively take gradient descent steps on either f or g. This makes our attack both simpler and
easier to analyze than prior attack approaches.

We use our technique to evade four state-of-the-art and previously-unbroken defenses to adversarial examples: the Honeypot defense (CCS’20) (SWW[+]20), Dense Layer Analysis (IEEE Euro
S&P’20) (SKCB19), Sensitivity Inconsistency Detector (AAAI’21) (TZLD21), and the SPAM detector presented in Detection by Steganalysis (CVPR’19) (LZZ[+]19). In all cases, we successfully
reduce the accuracy of the protected classifier to 0% while maintaining a detection AUC of less than
0.5—meaning the detector performs worse than random guessing.

_∗Equal contributions. Authored alphabetically._


-----

2 BACKGROUND

2.1 NOTATION

We consider classification neural networks f : R[d] _→_ R[n] that receive a d-dimensional input vector
(in this paper, images) x ∈ R[d] and output an n-dimensional prediction vector f (x) ∈ R[n]. We let
_g : R[d]_ _→_ R denote some other function which also must be considered, where g(x) ≤ 0 when the
constraint is satisfied and g(x) > 0 if it is violated. Without loss of generality, in a detection defense
this function g is the detector and higher values corresponding to higher likelihood of the input being
an adversarial example. To denote the true label of x is given by y we write c(x) = y. In an abuse of
notation, write y = f (x) to denote the arg-max most likely label under the model f .

2.2 ADVERSARIAL EXAMPLES

Adversarial examples (SZS[+]14; BCM[+]13) have been demonstrated in nearly every domain in which
neural networks are used. (ASE[+]18; CW18; HPG[+]17) Given an input x corresponding to label c(x)
and classifier f, an adversarial example is a perturbation x[′] of the input such that d(x, x[′]) < ϵ and
_f_ (x[′]) ̸= c(x) for some metric d. Additionally, an adversarial example can be targeted if, given a
target label t ̸= c(x) we have f (x[′]) = t with d(x, x[′]) < ϵ. The metric d is most often that induced
by a p-norm, typically either 2 or . With small enough perturbations under these metrics,
_|| · ||_ _|| · ||∞_
the adversarial example x[′] is not perceptibly different from the original input x.

**Datasets.** We attack each defense on the dataset that it performs best on. All of our defenses
operate on images. For three of these defenses, this is the CIFAR-10 dataset (KH09), and for one,
it is the ImageNet dataset (DDS[+]09). For each defense we attack, we constrain our adversarial
examples to the threat model originally considered to perform a fair re-evaluation, but also generate
adversarial examples with standard norms used extensively in prior work in order to make crossdefense evaluations meaningful. We perform all evaluations on a single GPU. Our attacks on
CIFAR-10 require just a few minutes, and for ImageNet a few hours (primarily due to the defense
having a throughput of one image per second).

2.3 DETECTION DEFENSES

We focus our study on detection defenses. Rather than directly improve the robustness of the model
(MMS[+]17; RSL18; LAG[+]19; CRK19), detection defenses classify inputs as adversarial or benign
(MGFB17; XEQ17) so they can be rejected. While there have been several different strategies
attempted to detect adversarial examples in the past (GSS15; MGFB17; FCSG17; XEQ17; MC17;
MLW[+]18; RKH19), many of these approaches were broken with adaptive attacks that designed new
loss functions tailored to each defense (CW17a; TCBM20).

2.4 GENERATING ADVERSARIAL EXAMPLES WITH PROJECTED GRADIENT DESCENT

Projected Gradient Descent (MMS[+]17) is a powerful first-order method for finding such adversarial
examples. Given a loss L(f, x, t) that takes a classifier, input, and desired target label, we optimize
over the constraint set Sϵ = {z : d(x, z) < ϵ} and solve

_x[′]_ = arg minz∈Sϵ _L(f, z, t)_ (1)

by taking stepsis the step size, and xi+1 x = Π0 is randomly initialized (Sϵ (xi − _α∇xi_ _L(f, xiMMS, t)). Here[+]17). This paper adapts PGD in order to solve ΠSϵ denotes projection onto the set Sϵ, α_
optimization problems which involve minimizing multiple objective functions simultaneously. For
notational simplicity, in the remainder of this paper we will omit the projection operator ΠSϵ .

**Attacks using PGD.** Recent work has shown that it is possible to attack models with adaptive
_attacks that target specific aspects of defenses. For detection defenses this process is often ad hoc,_
involving alterations specific to each given defense (TCBM20). An independent line of work develops
automated attack techniques that are reliable indicators of robustness (CH20); however, in general,
these attack approaches are difficult to apply to detection defenses.


-----

|g(x) < 0|f(x) < 0|
|---|---|
|||

|g(x) < 0|f(x) < 0|
|---|---|
|||

|g(x) < 0|f(x) < 0|
|---|---|
|||


Figure 1: A visualization showing how a standard Lagrangian attack fails when ours succeeds
over a non-convex loss landscape. Given two circular regions corresponding to when f (x) < 0
(above) and g(x) < 0 (below), we would like to find the central region where both are satisfied.
**(left) With Lagrangian PGD, the attack falls in a local minimum and fails to satisfy both constraints**
simultaneously regardless of the value λ selected. (middle) Our S-PGD attack first moves towards
the upper region by minimizing f (x). Once this constraint is satisfied (and f (x) < 0), we begin
to minimize g(x); however this overshoots to a point where f (x) > 0. A final step recovers a
valid solution to both. (right) Our O-PGD attack follows the same trajectory for the first two steps
optimizing f (x). However after this it takes steps orthogonal to f (x) maintaining the constraint
_f_ (x) < 0 while simultaneously minimizing g(x), giving a valid solution more quickly.

3 RETHINKING ADVERSARIAL EXAMPLE DETECTION

Before we develop our improved attack technique to break adversarial example detectors, it will be
useful to understand why evaluating adversarial example detectors is more difficult than evaluating
standard classifiers.

Early work on adversarial examples often set up the problem slightly differently than we do above
in Equation 1. The initial formulation of an adversarial example (SZS[+]14; CW17b) asks for the
smallest perturbation δ such that f (x + δ) is misclassified. In the targeted case, this means solving
arg min ∥δ∥2 such that f (x + δ) = t
for a given target t that is not equal to the original correct label for x. Solving this problem as stated
is intractable. It requires searching over a nonlinear constraint set, which is not feasible for standard
gradient descent. As a result, detection papers typically (SWW[+]20; SKCB19) reformulate the search
with a Lagrangian relaxation
arg min ∥δ∥2 + λL(f, x + δ, t) (2)
This formulation is simpler, but still (a) requires tuning λ to work well, and (b) is only guaranteed to
be correct for convex functions L—that it works for non-convex models like deep neural networks is
not theoretically justified. It additionally requires carefully constructing loss functions L (CW17b).

Equation 1 simplifies the setup considerably by exchanging the constraint and objective. Whereas in
Equation 2 we search for the smallest perturbation that results in misclassification, Equation 1 instead
finds an input x + δ with δ < ϵ that minimizes the classifier’s loss. This is a simpler formulation
because now the constraint is convex, and so we can run standard gradient descent optimization.

**Evading detection defenses is difficult because there are now two non-linear constraints. Not**
only must the input be constrained by a distortion bound and be misclassified by the base classifier,
but we must also have that they are not detected, i.e., with g(x) < 0. This new requirement is
nonlinear, and now it becomes impossible to side-step the problem by merely swapping the objective
and the constraint as we did before: there will always be at least one constraint that is a non-linear
function, and so standard gradient descent techniques can not directly apply.

In order to resolve this difficulty, the existing literature applies the same Lagrangian relaxation as
was previously applied to constructing minimum-distortion adversarial examples. That is, breaking a
detection scheme involves solving
arg minx∈Sϵ _L(f, x, t) + λg(x)_ (3)

where λ is a hyperparameter that controls the relative importance of fooling the classifier versus fooling the detector. However, this formulation again brings back all of the reasons why the community
moved past minimum-distortion adversarial examples.


-----

3.1 A MOTIVATING EXAMPLE

Let f (⃗x) = exp(−1) − exp(−∥⃗x − _⃗e∥2[2][)][ −]_ _[ε][ and][ g][(][⃗]x) = exp(−1) −_ exp(−∥⃗x + ⃗e∥2[2][)][ −] _[ε][ where]_
_⃗e ∈_ R[N] and ∥⃗e∥ = 1, as visualized in Figure 1. By setting ε to a small constant, the only solution
that satisfies both f (⃗x) < 0 and g(⃗x) < 0 can be made arbitrarily close to the origin ⃗x = _[⃗]0._

However, no standard Lagrangian formulation will be able to find this solution. Consider the sum
_h(⃗x; λ) = f_ (⃗x) + λg(⃗x); then we can show that for all λ we will have arg min⃗x[h][(][⃗]x; λ) ̸= 0. To
see this, observe that while it is possible for the gradient ∇h([⃗]0; λ) = 0 (one of the conditions for a
value to be a local minima), the loss surface is always “concave down” at the origin. It will always
be possible to move slightly closer to ⃗e or −⃗e and decrease the loss. Therefore, minimizing h(x)
will never be able to find a valid stable solution to this extremely simple problem, as it will always
collapse to finding a solution of either ⃗e or −⃗e, which only satisfies one of the two equations.

4 OUR ATTACK APPROACHES

We now present our attack strategy designed to generate adversarial examples that satisfy two
constraints. As we have been doing, each of our attack strategies defined below generates a targeted
adversarial example x[′] so that f (x[′]) = t but g(x[′]) < 0. Constructing an untargeted attack is nearly
identical except for the substitution of maximization instead of minimization.

4.1 SELECTIVE GRADIENT DESCENT

Instead of minimizing the weighted sum of f and g, our first attack never optimizes against a
constraint once it becomes satisfied. That is, we write our attack as

_A(x, t) =_ _x[′]:arg min∥x−x[′]∥<ϵ_ _L(f, x[′], t) · 1[f_ (x)L ̸=update t] +(x,t g) (x[′]) · 1[f (x) = t] _._ (4)

The idea here is that instead of minimizing a convex combination of the two loss functions, we| {z }
selectively optimize either f or g depending on if f (x) = t, ensuring that updates are always helping
to improve either the loss on f or the loss on g.

Another benefit of this style is that it decomposes the gradient step into two updates, which prevents
_imbalanced gradients, where the gradients for two loss functions are not of the same magnitude and_
result in unstable optimization (JMW[+]20). In fact, our loss function can be viewed directly in this
lens as following the margin decomposition proposal (JMW[+]20) by observing that

(f, x, t) if f (x) = t
update(x, t) = _∇L_ _̸_ (5)
_∇L_ _g(x)_ if f (x) = t.
∇

That is, each iteration either take gradients on f or on g depending on whether f (x) = t or not.

Recalling the motivating example from Figure 1, this selective optimization formulation would
be able to find a valid solution. No matter where we initialized our adversarial example search,
minimizing with respect to f (x) will eventually give a valid solution near e. Once this happens, we
then switch to optimizing against g(x) (because f (x) is satisfied). From here we will eventually
converge on the solution x ≈ _[⃗]0._

4.2 ORTHOGONAL GRADIENT DESCENT

The prior attack, while mathematically correct, might encounter numerical stability difficulties. Often,
the gradients of f and g point in opposite directions, that is, ∇f ≈−∇g. As a result, every step
spent optimizing f causes backwards progress on optimizing against g. This results in the optimizer
constantly “undoing” its own progress after each step that is taken. To address this problem, we
would like to "remove" the portion of one gradient optimization step that "undoes" the progress of a
previous optimization step.

To do this, we call on vector projections. Note that ∇g(x)[⊥] = ∇L(f, x, t) − proj∇g(x)∇L(f, x, t)
is orthogonal to the gradient ∇g(x), and similarly ∇L(f, x, t)[⊥] is orthogonal to ∇L(f, x, t). We


-----

integrate this fact with Equation 5 to give a slightly different update rule that again solves Equation 4,
however this time by optimizing:


_∇L(f, x, t) −_ proj∇g(x)∇L(f, x, t) if f (x) ̸= t (6)
_∇g(x) −_ proj∇L(f,x,t)∇g(x) if f (x) = t.


_Lupdate(x, t) =_


The purpose of this update is to take gradient descent steps with respect to one of f or g in such a
way that we do not significantly disturb the loss of the function not chosen. In this way, we prevent
our attack from taking steps that undo work done in previous iterations of the attack.

It is also important to note that, in the high-dimensional space in which a typical neural network
operates, the gradients of f and g are practically never exactly opposite, that is a situation where
_∇f = −∇g. In this case, the projection of ∇f onto ∇g and ∇g onto ∇f would be 0 and we could_
not make any meaningful optimizations towards satisfying either constraint with OPGD.

Again recalling Figure 1, by taking steps that are orthogonal to f (x) we can ensure that once we reach
the acceptable region for f, we never leave it, and much more quickly converge on an adversarial
example that evades detection.

5 CASE STUDIES

We validate the efficacy of our attack by using it to circumvent four previously unbroken, state-of-theart defenses accepted at top computer security or machine learning venues. Three of the case studies
utilizes models and code obtained directly from their respective authors. In the final case the original
authors provided us with matlab source code that was not easily used, which we re-implemented.

**Attack Success Rate Definition. We evaluate the success of our attack by a standard metric called**
_attack success rate at N (SR@N for short) (SKCB19). We use SR@N to ensure comparability across_
different case studies but more importantly between our results and our case studies’ original results.
SR@N is defined as the fraction of targeted attacks that succeed when the defense’s false positive
rate is set to N %. (To adjust a defense’s false positive rate it suffices to adjust the detection threshold
_φ so that inputs are rejected as adversarial when g(x) > φ.) For example, a 94% SR@5 could either_
be achieved through 94% of inputs being misclassified as the target class and 0% being detected as
adversarial, or by 100% of inputs being misclassified as the target class and 6% being detected as
adversarial, or some combination thereof. We report SR@5 and SR@50 for our main results [1], and
for completeness also give the full ROC curve of the detection rate for a more complete analysis.

**Attack Hyperparameters. We use the same hyperparmaeter setting for all attacks shown below. We**
set the distortion bound ε to 0.01 and .031; several of these papers exclusively make claims using the
value of 0.01 (SWW[+]20; SKC[+]20; TZLD21), but the value 0.031 = 8/255 is more typical in the
literature (TCBM20). We run our attack for N = 1000 iterations of gradient descent with a step size
_α =_ 10ε [(that is, the step size changes as a function of][ ε][ which follows standard advice (][MMS][+][17][)).]

5.1 HONEYPOT DEFENSE

The first paper we consider is the Honeypot Defense (SWW[+]20). Instead of preventing attackers
from directly constructing adversarial examples, the authors propose to lure attackers into producing
specific perturbations that are easy to find and hard to ignore. These perturbations are called
“honeypots” or trapdoors and can be easily identified by a detector. For their evaluation on the MNIST
and CIFAR-10 dataset, the authors use 5 sets of randomly selected 3 × 3 squares per label.

Formally, consider an input x to the classifier, f . During training, f is injected with a honeypot,
∆. The signature of a particular honeypot, S∆, is the expectation of the neuron activations of f
over multiple sample inputs containing ∆. During inference, the internal neuron activation pattern
_e(x) is compared to S∆_ using cosine similarity. Specifically, for a predefined threshold φ, if
cos(e(x), S∆) > φ, then x is flagged as adversarial. One additional modification the authors make is
to use neuron randomization. When creating the signatures, a random sample of neuron activations

1The value 5% is used in many prior defenses in the literature (MLW+18; XEQ17), and 50% is an extreme
upper bound and would reduce the model’s accuracy by half.


-----

|Attack|eps=0.01 eps=0.031 SR@5 SR@50 SR@5 SR@50|
|---|---|

|(SWW+20) Reproduction|0.02 - 0.13∗ -|- - - -|
|---|---|---|


reproduce the authors attack, we reach slightly a higher attack

1.0

Attack eps=0.01 eps=0.031 Orthogonal

Selective

SR@5 SR@50 SR@5 SR@50 0.8

(SWW[+]20) 0.02 - - - 0.6
Reproduction 0.13[∗] - - 
0.4

Orthogonal **1.0** **0.93** **1.0** **0.92** True Positive Rate 0.2
Selective 0.998 0.505 0.996 0.628

0.0

0.0 0.2 0.4 0.6 0.8 1.0

(a) Attack success rate for our two proposed attacks. _[∗]When we_ False Positive Rate

success rate of 13%.


(b) Defense ROC, ε = 0.01, AUC of 0.30


Figure 2: Honeypot attack evaluation. Compared to the originally reported 2% success rate, our
attack reaches a 100% attack success rate under the same distortion bound. While the ROC curve
does cross over the x = y line, this only occurs after a FPR of 70% which is completely unusable.

is ignored and the same set of neurons is ignored when calculating the cosine similarity during
adversarial attack detection.

We test our method of generating adversarial images on CIFAR-10 with an l∞ bound of ϵ = 2558 [.]

The original model has a baseline accuracy of .87. To test the defense, we generate targeted attacks
choosing the target from among the incorrect labels. As a simple baseline attack, we confirm that
FGSM achieves an accuracy of .06 on the classifier but the ROC AUC of the detector is .994.

To evaluate the efficacy of our attack, we use the authors pre-trained code, and convert their models
from TensorFlow to PyTorch to run our attacks. We find that our attack is extremely effective and has
an attack success rate above 99.9%, even at a 50% false positive rate, and an AUC of 0.30. In Table
2 we give the attack results for several configurations of our attack. We shared our results with the
original defense authors who confirmed our adversarial examples successfully evaded their detection
(an important step, given that we had converted the model to PyTorch).

5.2 DENSE LAYER ANALYSIS

Many recent defenses analyze the hidden activations of neural networks while processing benign
and adversarial inputs (e.g., following (MGFB17)). These defenses aim to capitalize on differences
in activation patterns among benign and adversarial inputs to train a separate classifier capable of
detecting inputs as adversarial.

The most recent paper in this direction Sperl et al. extract dense layer activation patterns among
benign and adversarial inputs and train a secondary binary classifier that detects adversarial examples
(SKCB19). The authors do this by first performing a forward pass through a target neural network
with both adversarial and benign inputs to create a mixed-feature dataset of activation-label pairs.
Then, using the mixed-feature dataset, they train a secondary binary classifier capable of discerning
between adversarial and benign inputs. When evaluating their models, the authors pass an input
through the target model to obtain the activation feature vectors for a particular input as well as a
potential classification. They then pass this feature vector through the secondary classifier. If the
secondary classifier alerts that the input was adversarial, the classification is thrown away. Otherwise,
classification proceeds as normal.

Sperl et al. evaluate this defense with 5 leading adversarial attacks on the MNIST and CIFAR-10
datasets using several models and report high accuracies for benign inputs and high detection rates
for adversarial inputs. The authors report a worst-case individual attack accuracy of 0.739.

In accordance with our framework, we assign the cross entropy loss of the classifier to our primary
function and binary cross entropy loss of the detector as our secondary function.

We obtain source code and pre-trained defense models from the authors in order to ensure that our
attack matches the defense as closely as possible. We now detail the results of our attack at ϵ = .01


-----

|Attack eps=0.01 eps=0.031 SR@5 SR@50 SR@5 SR@50|Col2|Col3|
|---|---|---|
|(SKC+20) Reproduction|≤0.13∗ - 0.20+ -|- - - -|

|Orthogonal Selective|0.374 0.163 0.83 0.441|1.0 0.718 1.0 0.865|
|---|---|---|


for what could be achieved at 5% FPR. [+]When we reproduce the

Attack eps=0.01 eps=0.031

1.0

SR@5 SR@50 SR@5 SR@50 Orthogonal

Selective

0.8

(SKC[+]20) _≤0.13[∗]_ - - Reproduction 0.20[+] - - - 0.6

Orthogonal 0.374 0.163 **1.0** 0.718 0.4
Selective **0.83** **0.441** **1.0** **0.865** True Positive Rate 0.2

0.0

(a) Attack success rate for our two proposed attacks. _[∗]The original_ 0.0 0.2 0.4 0.6 0.8 1.0
paper reported only at a 20% FPR, we take this as an upper bound False Positive Rate

[+]

authors attack, we reach slightly higher success rate of 20%.


(b) Defense ROC, ε = 0.01, AUC of 0.38


Figure 3: DLA attack evaluation. Our attack succeeds with 83% probability compared to the original
evaluation of 13% (with ε = 0.01), and 100% of the time under the more typical 8/255 constraint.

and at ϵ = .03 at false positive rates of 5% and 50% in Figure 3. We find that our attack is extremely
effective, resulting in an accuracy of 0 at a detection rate of 0 with a false positive rate of 5% under
_ϵ = .03 bounds and an AUC of 0.38. Finally, to validate that our attack succeeded, we again shared_
the resulting adversarial examples with the authors who confirmed our attack results.

5.3 SENSITIVITY INCONSISTENCY OF SPATIAL-TRANSFORM DOMAIN

We next evaluated our attack on the Sensitivity Inconsistency Detector (SID) proposed by Tian et
_al. (TZLD21). This defense relies on the observations of Fawzi et al. (FMDFS18) that adversarial_
examples are movements, in the form of perturbations, of benign inputs in a decision space along
an adversarial direction. Tian et al. then conjecture that, because adversarial examples are likely
to lie near highly-curved decision boundaries, and benign inputs lie away from such boundaries,
fluctuations in said boundaries will often result in a change in classification of adversarial examples
but not in classification of benign inputs.

To measure sensitivity against decision boundary transformations, Tian et al. design a dual classifier
which is the composition of a weighted additive wavelet transform layer and a DNN classifier with
the same structure as the original classifier. When doing a forward pass of the system, the authors
run an input through both the primal and the dual model, then pass both results to the detector that
discriminates among adversarial and benign classes. With these models, the authors then define their
so-called feature of sensitivity inconsistency S(x0).

_S(x0) =_ _fi(x0)_ _gi(x0)_ _i=1_
_{_ _−_ _}[K]_
where fi(x0) and gi(x0) are the predictions of the primal and the dual respectively. Input x0 is
classified as adversarial if S(x0) is greater than a threshold φ. SID achieves improved adversarial
example detection performance, especially in cases with small perturbations in inputs. The authors
report a worst-case, individual attack detection AUC % of 0.95.

Now, we want to create adversarial examples that are misclassified by the original model and not
flagged as adversarial by the Sensitivity Inconsistency Detector. We assign the loss of our target
model to our primary function and the loss of the Sensitivity Inconsistency Detector as our secondary
function. The initial target model had an accuracy of .94 and deemed .06 of all inputs adversarial.

We again obtain source code from the authors along with pre-trained models to ensure evaluation
correctness. We describe our attack’s results at ϵ = .01 and at ϵ = .03 at false positive rates of 5%
and 50% in Figure 4. Our attack works well in this case and induces an accuracy of 0 at a detection
rate of 0 with a false positive rate of 5% under ϵ = .03 bounds with an AUC of 0.25.

5.4 DETECTION THROUGH STEGANALYSIS

Since adversarial perturbations alter the dependence between pixels in an image, Liu et al. (LZZ[+]19)
propose a defense which uses a steganalysis-inspired approach to detect “hidden features” within an


-----

|Attack|eps=0.01 eps=0.031 SR@5 SR@50 SR@5 SR@50|
|---|---|


|(TZLD21)|≤0.09∗ -|- -|
|---|---|---|


|Orthogonal Selective|0.931 0.766 0.911 0.491|1.0 0.984 1.0 0.886|
|---|---|---|


(a) Attack success rate for our two proposed attacks. _[∗]The orig-_
inal paper only reports AUC values and does not report true
positive/false positive rates. The value of 9% was obtained by
running PGD on the author’s defense implementation.


1.0

Orthogonal
Selective

0.8

0.6

0.4

True Positive Rate 0.2

0.0

0.0 0.2 0.4 0.6 0.8 1.0

False Positive Rate


(b) Defense ROC, ε = 0.01, AUC of 0.25


Figure 4: SID attack evaluation. Our attack succeeds with 93% probability compared to the original
evaluation of 9% under a ε = 0.01-norm constraint, and 100% under a ε = 0.031.

image. These features are then used to train binary classifiers to detect the perturbations. Unlike the
prior defenses, this paper evaluates on ImageNet, reasoning that small images such as those from
CIFAR-10 and MNIST do not provide enough inter-pixel dependency samples to construct efficient
features for adversarial detection, so we attack this defense on ImageNet.

As a baseline, the authors use two feature extraction methods: SPAM and Spatial Rich Model. For
each pixel Xi,j of an image X, SPAM takes the difference between adjacent pixels along 8 directions.
For the rightward direction, a difference matrix A[→] is computed so that A[→]i,j [=][ X][i,j][ −] _[X][i,j][+1][. A]_
transition probability matrix M _[→]_ between pairs of differences can then be computed with

_Mx,y[→]_ [=][ Pr][(][A]i,j[→] +1 [=][ x][|][A]i,j[→] [=][ y][)]

where x, y ∈{−T, ..., T _}, with T being a parameter used to control the dimensionality of the final_
feature set F . We use T = 3 in accordance with that used by the authors. The features themselves
are calculated by concatenating the average of the non-diagonal matrices with the average of the
diagonal matrices:


_F1,...,k =_ _[M][ →]_ [+][ M][ ←] [+][ M][ ↑] [+][ M][ ↓]


_Fk+1,...,2k =_ _[M][ ↗]_ [+][ M][ ↖] [+][ M][ ↘] [+][ M][ ↙]


In order to use the same attack implementation across all defenses, we reimplemented this defense
in PyTorch (the authors implementation was in matlab). Instead of re-implementing the full Fisher
Linear Discriminant (FLD) ensemble (KFH12) used by the authors, we train a 3-layer fully connected
neural network on SPAM features and use this as the detector. This allows us to directly investigate
the claim that SPAM features can be reliably used to detect adversarial examples, as FLD is a highly
non-differentiable operation and is not a fundamental component of the defense proposal.

The paper also proposes a second feature extraction method named “Spatial Rich Model” (SRM)
that we do not evaluate against. This scheme follows the same fundamental principle as SPAM in
modeling inter-pixel dependencies—there is only a marginal benefit from using these more complex
models, and so we analyze the simplest variant of the scheme.

Notice that SPAM requires the difference matrices A to be discretized in order for the dimensionality
of the transition probability matrices M to be finite. To make this discretization step differentiable
and compatible with our attacks, we define a count matrix X where, for example, Xx,y[→] [counts, for]
every pair i, j, the number of occurrences of y in A[→]i,j [and][ x][ in][ A]i,j[→] +1[.][ M][ →]x,y [is then defined by:]

_Xx,y[→]_
_Mx,y[→]_ [=][ P] [(][A]i,j[→] +1 [=][ x][|][A]i,j[→] [=][ y][) =]

_x[′][ X]x[→][′],y_

P

To construct a differentiable approximation, consider without loss of generality the rightward difference matrix A[→]1 [for an image. We construct a shifted copy of it][ A]2[→] [so that][ A]2[→]i,j [=][ A]1[→]i,j+1[. We]
then define a mask K so that

_Ki,j = 1[x ≤_ _A[→]2i,j_ _[< x][ + 1][ ∩]_ _[y][ ≤]_ _[A]1[→]i,j_ _[< y][ + 1]]_


-----

|Attack +|eps=0.01 SR@5 SR@50|eps=0.031 SR@5 SR@50|
|---|---|---|
|(LZZ+19)|0.03 -|.03 -|
|Orthogonal|0.988 0.54|1.0 0.62|


(a) Attack success rate for our proposed attack. For computational
efficiency, we only run our Orthogonal attack as the detection
model has a throughput of one image per second.


1.0

Orthogonal

0.8

0.6

0.4

True Positive Rate 0.2

0.0

0.0 0.2 0.4 0.6 0.8 1.0

False Positive Rate


(b) Defense ROC, ε = 0.01, AUC of 0.44


Figure 5: Steganalysis attack evaluation. We find it difficult to decrease the detection score lower
than the original score on the non-adversarial input, thus the AUC is almost exactly 0.5.

Each element of the intermediate matrix Xx,y[→] [counts the number of pairs in][ A]1[→] [and][ A]2[→] [which]
would be rounded to x and y respectively after discretization:


_i,j_ [(][K][ ◦] _[A]2[→][)]i,j_


_Xx,y[→]_ [=]


where ◦ is the Hadamard product. If we normalize X _[→]_ so that the sum of elements in each column
is equal to 1, we get the probability of difference values x ∈ _A[→]2_ [conditioned on column][ y][ ∈] _[A]1[→][.]_
Thus, for any pair of indices i, j,

_Mx,y[→]_ [=][ P] [(][A]2[→]i,j [=][ x][|][A]1[→]i,j [=][ y][) =] _Xx,y[→]_

_x[′][ X]x[→][′],y_
Using this differentiable formulation of SPAM feature extraction, we train an auxillary detector asP
described above and use its gradients to apply our attack on the original, non-differentiable detector.

The authors evaluate their defense on 4 adversarial attacks and report high accuracy for benign inputs
and high detection rates for adversarial inputs. The best attack they develop still has a success rate
less than 3%. In contrast, our attack on SPAM using the differentiable approximation has a success
rate of 98.8% when considering a 5% false positive rate, with an AUC of 0.44, again less than the
random guessing threshold of 0.5.

6 CONCLUSION

Generating adversarial examples that satisfy multiple constraints simultaneously (e.g., requiring
that an input is both misclassified and deemed non-adversarial) requires more care than generating
adversarial examples that satisfy only one constraint (e.g., requiring only that an input is misclassified).
We find that prior attacks unnecessarily over-optimizes one constraint when another constraint has
not yet been satisfied. Our new attack methogology is designed to avoid this weakness, and as a
result can reduce the accuracy of four previously-unbroken detection methods to 0% accuracy while
maintaining a 0% detection rate at 5% false positive rates.

We believe our attack approach is generally useful, but it is not a substitute for trying other attack
techniques. We do not envision this attack as a complete replacement for standard Lagrangian-based
attacks, but rather a complement; defenses must carefully consider their robustness to both prior
attacks as well as this new one. Notice, for example, that for one of the four defenses we study
Selective PGD performs better than Orthogonal PGD—indicating these attacks are complementary to
each other. Additionally, automated attack tools (CH20) would benefit from adding our optimization
trick to their collection of known techniques that could (optionally) compose with other attacks. We
discourage future work from blindly applying this attack without properly understanding its design
criteria. While this attack is effective for the defenses we consider, it is not the only way to do so, and
may not be the correct way to do so in future defense evaluations. Evaluating adversarial example
defenses will necessarily require adapting any attack strategies to the defense’s design.


-----

ACKNOWLEDGEMENTS

We thank the authors of the papers we use in the case studies, who helped us answer questions
specific to their respective defenses and agreed to share their code with us. We are also grateful to
Alex Kurakin and the anonymous reviewers for comments on drafts of this paper.

ETHICS STATEMENT

All work that improves adversarial attacks has potential negative societal impacts. Yet, we believe
that it is better for those vulnerabilities to be known rather than relying on security through obscurity.
We have attacked no deployed system, and so cause no direct harm; and by describing how our attack
works, future defenses will be stronger. We have communicated the results of our attack to the authors
of the papers we break as a form of responsible disclosure, and also to ensure the correctness of our
results.

REPRODUCIBILITY STATEMENT

All of the code we used to generate our results will be made open source in a GitHub repository. The
datasets we use (MNIST, CIFAR10, ImageNet) are available online and widely studied. We obtained
original copies of the code associated with 3 of the 4 case studies. We used code either directly from
the authors or code released publicly alongside an academic paper. In the case of steganalysis, we
reimplemented the paper to the best of our ability. We also provide a Python class constructor so that
future work can test or improve our results. Again, we relayed results to the authors of each paper
and received confirmation that our adversarial examples were indeed adversarial and not detected by
the author’s original implementations.

REFERENCES

[ACW18] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false
sense of security: Circumventing defenses to adversarial examples. In International
_Conference on Machine Learning, 2018._

[ASE[+]18] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. CoRR,
abs/1804.07998, 2018.

[BCM[+]13] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndi´c, Pave[˜]l
Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at
test time. In Joint European conference on machine learning and knowledge discovery
_in databases, pages 387–402. Springer, 2013._

[CH20] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with
an ensemble of diverse parameter-free attacks. In Proceedings of the 37th International
_Conference on Machine Learning, pages 2206–2216. PMLR, 2020._

[CRK19] Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness
via randomized smoothing. arXiv preprint arXiv:1902.02918, 2019.

[CW17a] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected:
Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on
_Artificial Intelligence and Security, pages 3–14, 2017._

[CW17b] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural
networks. In 2017 IEEE symposium on security and privacy, pages 39–57. IEEE, 2017.

[CW18] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on
speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1–7, 2018.


-----

[DDS[+]09] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A
large-scale hierarchical image database. In 2009 IEEE conference on computer vision
_and pattern recognition, pages 248–255. Ieee, 2009._

[FCSG17] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting
adversarial samples from artifacts. arXiv preprint arXiv:1703.00410, 2017.

[FMDFS18] Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Stefano
Soatto. Empirical study of the topology and geometry of deep networks. In Proceedings
_of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June_
2018.

[GSS15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing
adversarial examples. International Conference on Learning Representations, 2015.

[HPG[+]17] Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel.
Adversarial attacks on neural network policies. CoRR, abs/1702.02284, 2017.

[JMW[+]20] Linxi Jiang, Xingjun Ma, Zejia Weng, James Bailey, and Yu-Gang Jiang. Imbalanced gradients: A new cause of overestimated adversarial robustness. arXiv preprint
_arXiv:2006.13726, 2020._

[KFH12] Jan Kodovsky, Jessica Fridrich, and Vojtˇech Holub. Ensemble classifiers for steganalysis
of digital media. In IEEE Transactions on Information Forensics and Security, pages
432–444, 2012.

[KH09] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images.
_Master’s thesis, Department of Computer Science, University of Toronto, 2009._

[LAG[+]19] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana.
Certified robustness to adversarial examples with differential privacy. In 2019 IEEE
_Symposium on Security and Privacy (SP), pages 656–672. IEEE, 2019._

[LZZ[+]19] Jiayang Liu, Weiming Zhang, Yiwei Zhang, Dongdong Hou, Yujia Liu, Hongyue
Zha, and Nenghai Yu. Detection based defense against adversarial examples from the
steganalysis point of view. In Proceedings of the IEEE/CVF Conference on Computer
_Vision and Pattern Recognition, pages 4825–4834, 2019._

[MC17] Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial
examples. In Proceedings of the 2017 ACM SIGSAC conference on computer and
_communications security, pages 135–147, 2017._

[MGFB17] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting
adversarial perturbations. arXiv preprint arXiv:1702.04267, 2017.

[MLW[+]18] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant
Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv:1801.02613,
2018.

[MMS[+]17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian
Vladu. Towards deep learning models resistant to adversarial attacks. International
_Conference on Learning Representations, 2017._

[RKH19] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test
for detecting adversarial examples. In International Conference on Machine Learning,
pages 5498–5507. PMLR, 2019.

[RSL18] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against
adversarial examples. arXiv preprint arXiv:1801.09344, 2018.

[SKC[+]20] Philip Sperl, Ching-Yu Kao, Peng Chen, Xiao Lei, and Konstantin Böttinger. Dla:
Dense-layer-analysis for adversarial example detection. In 2020 IEEE European
_Symposium on Security and Privacy (EuroS&P), pages 198–215. IEEE, 2020._


-----

[SKCB19] Philip Sperl, Ching-yu Kao, Peng Chen, and Konstantin Böttinger. DLA: dense-layeranalysis for adversarial example detection. CoRR, abs/1911.01921, 2019.

[SWW[+]20] Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y Zhao.
Gotta catch’em all: Using honeypots to catch adversarial attacks on neural networks. In
_Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications_
_Security, pages 67–83, 2020._

[SZS[+]14] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
and Rob Goodfellow, Ian an d Fergus. Intriguing properties of neural networks. In
_International Conference on Learning Representations (ICLR), 2014._

[TCBM20] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive
attacks to adversarial example defenses. CoRR, abs/2002.08347, 2020.

[TZLD21] Jinyu Tian, Jiantao Zhou, Yuanman Li, and Jia Duan. Detecting adversarial examples from sensitivity inconsistency of spatial-transform domain. arXiv preprint
_arXiv:2103.04302, 2021._

[XEQ17] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial
examples in deep neural networks. arXiv preprint arXiv:1704.01155, 2017.


-----

