# GEOMETRIC AND PHYSICAL QUANTITIES IMPROVE E(3) EQUIVARIANT MESSAGE PASSING


**Johannes Brandstetter[∗]**
University of Amsterdam
Johannes Kepler University Linz
brandstetter@ml.jku.at


**Rob Hesselink[∗]**
University of Amsterdam
r.d.hesselink@uva.nl


**Elise van der Pol**
UvA-Bosch DeltaLab
University of Amsterdam
e.e.vanderpol@uva.nl


**Erik J Bekkers**
University of Amsterdam
e.j.bekkers@uva.nl

ABSTRACT


**Max Welling**
UvA-Bosch DeltaLab
University of Amsterdam
m.welling@uva.nl


Including covariant information, such as position, force, velocity or spin is important in many tasks in computational physics and chemistry. We introduce Steerable
E(3) Equivariant Graph Neural Networks (SEGNNs) that generalise equivariant
graph networks, such that node and edge attributes are not restricted to invariant
scalars, but can contain covariant information, such as vectors or tensors. This
model, composed of steerable MLPs, is able to incorporate geometric and physical
information in both the message and update functions. Through the definition of
steerable node attributes, the MLPs provide a new class of activation functions
for general use with steerable feature fields. We discuss ours and related work
through the lens of equivariant non-linear convolutions, which further allows us to
pin-point the successful components of SEGNNs: non-linear message aggregation
improves upon classic linear (steerable) point convolutions; steerable messages
improve upon recent equivariant graph networks that send invariant messages. We
demonstrate the effectiveness of our method on several tasks in computational
physics and chemistry and provide extensive ablation studies.

1 INTRODUCTION

The success of Convolutional Neural Networks (CNNs) (LeCun et al., 1998; 2015; Schmidhuber,
2015; Krizhevsky et al., 2012) is a key factor for the rise of deep learning, attributed to their capability
of exploiting translation symmetries, hereby introducing a strong inductive bias. Recent work has
shown that designing CNNs to exploit additional symmetries via group convolutions has even further
increased their performance (Cohen & Welling, 2016; 2017; Worrall et al., 2017; Cohen et al.,
2018; Kondor & Trivedi, 2018; Weiler et al., 2018; Bekkers et al., 2018; Bekkers, 2019; Weiler &
Cesa, 2019). Graph neural networks (GNNs) and CNNs are closely related to each other via their
aggregation of local information. More precisely, CNNs can be formulated as message passing
layers (Gilmer et al., 2017) based on a sum aggregation of messages that are obtained by relative
position-dependent linear transformations of neighbouring node features. The power of message
passing layers is, however, that node features are transformed and propagated in a highly non-linear
manner. Equivariant GNNs have been proposed before as either PointConv-type (Wu et al., 2019;
Kristof et al., 2017) implementations of steerable (Thomas et al., 2018; Anderson et al., 2019; Fuchs
et al., 2020) or regular group convolutions (Finzi et al., 2020). The most important component in
these methods are the convolution layers. Although powerful, such layers only (pseudo[1]) linearly
transform the graphs and non-linearity is only obtained via point-wise activations.

1Methods such as SE(3)-transformers (Fuchs et al., 2020) and Cormorant (Anderson et al., 2019) include an
input-dependent attention component that augments the convolutions.


-----

In this paper, we propose non-linear E(3) equivariant message passing layers using the same principles
that underlie steerable group convolutions, and view them as non-linear group convolutions. Central
to our method is the use of steerable vectors and their equivariant transformations to represent and
process node features; we present the underlying mathematics of both in Sec. 2 and illustrate it
in Fig. 1 on a molecular graph. As a consequence, information at nodes and edges can now be
rotationally invariant (scalar) or covariant (vector, tensor). In steerable message passing frameworks,
the Clebsch-Gordan (CG) tensor product is used to steer the update and message functions by
geometric information such as relative orientation (pose). Through a notion of steerable node
attributes we provide a new class of equivariant activation functions for general use with steerable
feature fields (Weiler et al., 2018; Thomas et al., 2018). Node attributes can include information such
as node velocity, force, or atomic spin. Currently, especially in molecular modelling, most datasets
are build up merely of atomic number and position information. In this paper, we demonstrate the
potential of enriching node attributes with more geometric and physical quantities. We demonstrate
the effectiveness of SEGNNs by setting a new state of the art on n-body toy datasets, in which our
method leverages the abundance of geometric and physical quantities available. We further test our
model on the molecular datasets QM9 and OC20. Although here only (relative) positional information
is available as geometric quantity, SEGNNs achieve state of the art on the IS2RE dataset of OC20,
and competitive performance on QM9. For all experiments we provide extensive ablation studies.

The main contributions of this paper are: (i) A generalisation of equivariant GNNs such that node and
edge attributes are not restricted to scalars. (ii) A new class of equivariant activation functions for
steerable vector fields, based on the introduction of steerable node attributes and steerable multi-layer
perceptrons, which permit the injection of geometric and physical quantities into node updates. (iii)
A unifying view on various equivariant GNNs through the definition of non-linear convolutions.
(iv) Extensive experimental ablation studies that shows the benefit of steerable over non-steerable
(invariant) message passing, and the benefit of non-linear over linear convolutions.

Figure 1: Commutation diagram for an equivariant operator φ applied to a 3D molecular graph with
steerable node features (visualised as spherical functions); As the molecule rotates, so do the node
features. The use of steerable vectors allows neural networks to exploit, embed, or learn geometric
cues such as force and velocity vectors.

2 GENERALISED E(3) EQUIVARIANT STEERABLE MESSAGE PASSING

with feature vectorsMessage passing networks.edge. Graph neural networks (GNNs) (Scarselli et al., 2009; Kipf & Welling, 2017; Defferrard et al., fi ∈ R[c][n] attached to each node, and edge attributesConsider a graph G = (V, E), with nodes a viji ∈V ∈ R and edges[c][e] attached to each eij ∈E,
2016; Battaglia et al., 2018) are designed to learn from graph-structured data and are by construction
permutation equivariant with respect to the input. A specific type of GNNs are message passing
networks (Gilmer et al., 2017), where a layer updates node features via the following steps:
compute message mij from node vj to vi: **mij = φm (fi, fj, aij),** (1)


aggregate messages and update node features vi: **fi[′]** [=][ φ][f] fi, **mij** _,_ (2)

_j∈NX (i)_

 

where N (i) represents the set of neighbours of node vi, and φm and φf are commonly parameterised
by multilayer perceptrons (MLPs).


-----

**Equivariant message passing networks. Our objective is to build graph neural networks that are**
robust to rotations, reflections, translations and permutations. This is a desirable property since some
prediction tasks, such as molecular energy prediction, require E(3) invariance, whereas others, like
force prediction, require equivariance. From a technical point of view, equivariance of a function φ
to certain transformations means that for any transformation parameter g and all inputs x we have
_Tg[′][[][φ][(][x][)] =][ φ][(][T][g][[][x][])][,][ where][ T][g]_ [and][ T][ ′]g [denote transformations on the input and output domain of]
_φ, respectively. Equivariant operators applied to atomic graphs allow us to preserve the geometric_
structure of the system as well as enriching it with increasingly abstract directional information. We
build E(3) equivariant GNNs by constraining the functions φm and φf of Eqs. (1-2) to be equivariant,
which in return guarantees equivariance of the entire network. In the following, we introduce the core
components behind our method; full mathematical details and background can be found in App. A.

**Steerable features. In this work, we achieve equivariant graph neural networks by working with**
_steerable feature vectors, which we denote with a tilde, e.g. a vector_ **h[˜] is steerable. Steerability**
of a vector means that for a certain transformation group with transformation parameters g, the
vector transforms via matrix-vector multiplication D(g)h[˜]. For example, a Euclidean vector in R[3]
is steerable for rotations g = R ∈ SO(3) by multiplying the vector with a rotation matrix, thus
**D(g) = R. We are however not restricted to only work with 3D vectors; via the construction of**
steerable vector spaces, we can generalise the notion of 3D rotations to arbitrarily large vectors.
Central to our approach is the use of Wigner-D matrices D[(][l][)](g) [2]. These are (2l + 1 × 2l + 1)dimensional matrix representations that act on (2l+1)-dimensional vector spaces. These vector spaces
that are transformed by l-th degree Wigner-D matrices will be referred to as type-l steerable vector
_spaces and denoted with Vl. We note that we can combine two independent steerable vector spaces_
_Vspace then transforms by the direct sum of Wigner D-matrices, i.e., vial1 and Vl2 of type l1 and l2 by the direct sum, denoted by V = Vl1 ⊕ DV(l2g. Such a combined vector) = D[(][l][1][)](g) ⊕_ **D[(][l][2][)](g),**
which is a block-diagonal matrix with the Wigner-D matrices along the diagonal. We denote the direct
sum of type-the same vector space withl vector spaces up to degree nV := V ⊕ _V l ⊕ = L. . . by ⊕ VVL. Regular MLPs are based on transformations := V0 ⊕_ _V1 ⊕· · · ⊕_ _VL, and n copies of the_
_n times_

between d-dimensional type-0 vector spaces i.e., R[d] = dV0, and are a special case of our steerable

| {z }

MLPs that act on steerable vector spaces of arbitrary type.

**Steerable MLPs. Like regular MLPs, steerable MLPs are constructed by interleaving linear map-**
pings (matrix-vector multiplications) with non-linearities. Now however, the linear maps transform
between steerable vector spaces at layer i − 1 to layer i via **h[˜][i]** = Wa[i]˜h[˜][i][−][1] _. Steerable MLPs thus_
have the same functional form as regular MLPs, although, in our case, the linear transformation
matrices Wa[i]˜[, defined below, are conditioned on geometric information (e.g. relative atom positions)]
which is encoded in the steerable vector ˜a. Both vectors ˜a and **h[˜] are steerable vectors. In this work**
we will however use the vector ˜a to have geometric and structural information encoded and the steer
the information flow of **h[˜] through the network. In order to guarantee that Wa[i]˜** [maps between steerable]
vector spaces, the matrices are defined via the Clebsch-Gordan tensor product. By construction the
resultant MLPs are equivariant for every transformation parameter g via

MLP(] **D(g)h[˜]0) = D[′](g)MLP([]]** **h[˜]0),** (3)

provided that the steerable vectors ˜a that condition the MLPs are also obtained equivariantly.

**Spherical harmonic embedding of vectors.through the evaluation of spherical harmonics We can convert any vector Ym[(][l][)]** [:][ S][2][ →] [R][ at] _∥xx∥_ [. For any] x ∈ R[ x][3][ ∈]into a type-[R][3] _l vector_

_T_

**a˜[(][l][)]** = _Ym[(][l][)]_ **xx** (4)

_∥_ _∥_ _m=_ _l,_ _l+1,...,l_

_−_ _−_

  

is a type-l steerable vector. The spherical harmonic functions Ym[(][l][)] [are functions on the sphere][ S][2]
and we visualise them as such in Figure 2. We will use spherical harmonic embeddings to include
geometric and physical information into steerable MLPs.

**Mapping between steerable vector spaces.** The Clebsch-Gordan (CG) tensor product ⊗cg :
_Vl1 × Vl2 →_ _Vl is a bilinear operator that combines two O(3) steerable input vectors of types l1 and_

2In order to be O(3)—not just SO(3)—equivariant, we include reflections. See App. A for more detail.


-----

Figure 2: Left: representation of an O(harmonic embedding of vector x, e.g. relative orientation, velocity or force. Right: for each subspace3) steerable vector **h[˜] ∈** _VL = V0 ⊕_ _V1 ⊕_ _V2, a spherical_
the embedding with the basis functions Ym[(][l][)] [is shown (a). The transformation of][ ˜]h via D[(][l][)](g) acts
on each subspace of Vl separately (b).

_l2 and returns another steerable vector of type l. Let_ **h[˜][(][l][)]** _∈_ _Vl = R[2][l][+1]_ denote a steerable vector of
type l and h[(]m[l][)] [its components with][ m][ =][ −][l,][ −][l][ + 1][, . . ., l][. The CG tensor product is given by]

_l1_ _l2_

(h[˜][(][l][1][)] _⊗cg[w]_ **h[˜][(][l][2][)])[(]m[l][)]** [=][ w] _C([(]l[l,m]1,m[)]1)(l2,m2)[h][(]m[l][1]1[)][h]m[(][l][2]2[)]_ _[,]_ (5)

_m1X=−l1_ _m2X=−l2_

in which w is a learnable parameter that scales the product, and C([(]l[l,m]1,m[)]1)(l2,m2) [are the Clebsch-]
Gordan coefficients that ensure that the resulting vector is type-l steerable. The CG tensor product is
a sparse tensor product, as generally many coefficients are zero. Most notably, C([(]l[l,m]1,m[)]1)(l2,m2) [= 0]
whenever l < |l1 − _l2| or l > l1 + l2. While Eq. (5) only describes the product between steerable_
vectors of a single type, e.g. **h[˜][(][l][1][)]** _∈_ _Vl1 and_ **h[˜][(][l][2][)]** _∈_ _Vl2, it is directly extendable to mixed type_
steerable vectors that may have multiple channels/multiplicities within a type. In this case, every
input to output sub-vector pair gets its own index in a similar way as the weights in a standard linear
layer are indexed with input-output indices. We then denote the CG product with ⊗cg[W] [with boldfaced]
**W to indicate that it is parametrised by a collection of weights. In order to stay close to standard**
notation used with MLPs, we treat the CG product with a fixed vector ˜a in one of its inputs as a
_steerable linear layer conditioned on ˜a, denoted with_

**Wa˜h[˜] := h[˜] ⊗cg[W]** **a[˜],** or **Wa˜[(][d][)˜]h := h[˜] ⊗[W]cg** [(][d][)] **a˜,** (6)
where the latter indicates that the CG weights depend on some quantity d, e.g. relative distances.

**Steerable activation functions.** The common recipe for deep neural networks is to alternate
linear layers with element-wise non-linear activation functions. In the steerable setting, careful
consideration is required to ensure that the activation functions are equivariant; currently available
classes of activations include Fourier-based (Cohen et al., 2018), norm-altering (Thomas et al., 2018),
or gated non-linearities (Weiler et al., 2018). We use gated non-linearities in all our architectures
and shortly discuss their working principle in Sec. C in the appendix. The resulting steerable MLPs
themselves in turn provide a new class of steerable activation functions, that is the first in its kind in
directly leveraging local geometric cues. Namely, through steerable node attributes ˜a, either derived
from the physical setup (forces, velocities) or from predictions (similar to gating). The MLPs can be
applied node-wise and be generally used in steerable feature fields as non-linear activations.

2.1 STEERABLE E(3) EQUIVARIANT GRAPH NEURAL NETWORKS

We extend the message passing equations (1)-(2) and define a message passing layer that updates
steerable node features [˜]fi ∈ _VL at node vi via the following steps:_

**m˜** _ij = φm_ ˜fi, [˜]fj, ∥xj − **xi∥[2], ˜aij** _,_ (7)
 


˜fi[′] [=][ φ][f]


˜fi,






**m˜** _ij, ˜ai_ _._ (8)



_j∈NX (i)_




-----

Here,steerable MLPs,exist, such as pair-wise distance ∥xj − **xi∥ ˜a[2]** _ijis the squared relative distance between two nodes ∈_ _VL and ˜ai ∈xVjL are steerable edge and node attributes. If additional attributesxi_, they can either be concatenated to the attributes that vi and vj, φm and φf are O(3)
condition our steerable MLPs, or as is more commonly done (Sec. 3) add as inputs to ∥ _−_ _∥_ _φm and φf_ .
We do the latter, and stack all inputs to a single steerable vector by which the steerable MLP layer
is given: **h[˜][1]** = σ(Wa˜ij **h[˜][0]i** [)][,][ with][ ˜]h[0]i [=] ˜fi, [˜]fj, ∥xj − **xi∥[2][]** _∈_ _Vf ⊕_ _Vf ⊕_ _V0, where Vf is the user_
specified steerable vector space of node representations. The message network _φm is steered via edge_
attributes ˜aij, and the node update network φf is similarly steered via node attributes ˜ai.

**Injecting geometric and physical quantities.** In order to make SEGNNs more expressive, we
include geometric and physical information in the edge and node updates. For that purpose, the edge
attributes are obtained via the spherical harmonic embedding (Eq. (4)) of relative positions, in most
cases, but possibly also relative force or relative momentum. The node attributes could e.g. be the
average edge embedding of relative positions over neighbours of a node, i.e., ˜ai = 1 (i) _j_ (i) **a[˜]ij,**

_|N_ _|_ _∈N_

and could additionally include node force, spin or velocities, as we do in the N-body experiment. The

P

use of steerable node attributes in the steerable MLPs that define φf allows us to not just integrate
geometric cues into the message functions, but also leverage it in the node updates. We observe that
the more geometric and physical quantities are injected the better SEGNNs perform.

3 MESSAGE PASSING AS CONVOLUTION, RELATED WORK

Recent literature shows a trend towards building architectures that improve performance by means of
maximally preserving equivariance through groups convolutions (either in regular or tensor-product
form, see App. B). Convolutions, however, are ”just” linear operators and non-linearities are only
introduced through point-wise activation functions. This is in contrast to architectures that are built
without explicit use of group convolutions, but instead rely on the highly non-linear framework of
message passing. In the following we show that many related works are connected through a notion
of non-linear convolution, a term that we coin based on the following. Any linear operator which
is equivariant is a group convolution (Kondor & Trivedi, 2018; Bekkers, 2019) and their discrete
implementations can be written in message passing form. We then call any non-linear operator that is
equivariant, and which can be written in simple message passing form, a non-linear convolution. This
framing allows us to place related work in a unifying context and to identify two important aspects of
successful architectures: (i) equivariant layers improve upon invariant ones and (ii) non-linear layers
improve upon linear ones. Both come together in SEGNNs via steerable non-linear convolutions.

**Point convolutions as equivariant linear message passing. Consider a feature map f : R[d]** _→_ R[c][l] .
A convolution layer (defined via cross-correlation) with a point-wise non-linearity σ is given by


**f** _[′](x) = σ_ _,_ (9)
 ZR[d][ W][(][x][′][ −] **[x][)][f]** [(][x][′][)d][x][′]

with W : R[d] _→_ R[c][l][+1][×][c][l] a convolution kernel that provides for every relative position a matrix
that linearly transforms features from R[c][l] to R[c][l][+1]. Point convolutions, generally referred to as
PointConvs (Wu et al., 2019), and SchNet (Kristof et al., 2017) implement Eq. (9) on point clouds.
For a sparse input feature map consisting of location-feature pairs (xi, fi), the point convolution is
given by fi[′] [=][ P]j∈N (i) **[W][(][x][j]** _[−][x][i][)][f][i][,][ which describes a][ linear message passing layer][ of Eqs. (1)-(2)]_

in which the messages are mij = W(xj − **xi)fj and the message update fi[′]** [=][ P]j **[m][ij][. In the above]**

convolutions, the transformation matrices W are conditioned on relative positions xj **xi, which is**
typically done in one of the following three approaches. (i) Classically, feature maps are processed on −
dense regular grids with shared neighbourhoods and thus the transformations Wij can be stored for a
finite set of relative positionsnon-uniform grids such as point clouds. Continuous kernel methods parametrise the transformations xj − **xi in a single tensor. This method however does not generalise to**
either by (ii) expanding W into a continuous basis or (iii) directly parametrising them with MLPs via
**W(xj** **xi) = MLP(xj** **xi). Steerable kernel methods[3]** are of type (ii) and rely on a steerable
_−_ _−_


3See (Lang & Weiler, 2020) for a general theory for G-steerable kernel constraints for compact groups.


-----

_basis, such as 3D spherical harmonics Ym[(][l][)][, via]_


**Wm[(][l][)][(][∥][x][j]** _m_ [(][x][j] (10)
_mX=−l_ _[−]_ **[x][i][∥][)][Y][ (][l][)]** _[−]_ **[x][i][)][,]**


**W(xj** **xi) =**
_−_


with basis coefficients Wm[(][l][)] [typically depending on pair-wise distances. We next show how such]
kernels connect to steerable group convolutions and provide a detailed background in App. B.

**Steerable (group) convolutions.** In context of the steerable framework of Sec. 2, linear feature
transformations W are equivalent to steerable linear transformations (Eq. (6)) conditioned on the
scalar “1”. I.e, with hi ∈ R[d] in the usual and **h[˜] ∈** _dV0 the steerable setting, messages are obtained by_

**mij = W(xj −** **xi)hi** _⇔_ **m˜** _ij = W1(xj −_ **xi)h[˜]i .** (11)

When the transformations are parametrised in a steerable basis (10) we can make the identification[4]

**mij = W(xj −** **xi)hi** _⇔_ **m˜** _ij = Wa˜ij_ [(][∥][x]j _[−]_ **[x]i[∥][)˜]hi,** (12)

weights that parametrise the CG tensor product depend on pair-wise distances. We can thus performin which ˜aij = Ym[(][l][)][(][x]j _[−]_ **[x]i[)][ are spherical harmonic embeddings (Eq.][ (4)][) of][ x]j** _[−]_ **[x]i[, and the]**
convolutions either in the regular or in the steerable setting, where the latter has the benefit of allowing
us to directly derive what the convolution result would be if the kernel were to be rotated via

**mij = W(R[−][1](xj −** **xi))hi** _⇔_ **m˜** _ij = D(R)Wa˜ij_ [(][∥][x]j _[−]_ **[x]i[∥][)˜]hi .** (13)

Steerable vectors obtained via convolutions with steerable kernels thus generate signals on O(3) via
the Wigner-D matrices. This relation is in fact established by the inverse Fourier transform on O(3)
(cf. App. A), by which we can treat steerable feature vectors [˜]fi at each location xi as a function on the
group O(3). Steerable convolutions thus produce feature maps on the full group E(3) that for every
possible translation/position x and rotation R provide a feature response f (x, R). It is precisely
this mechanism of transforming convolution kernels via the group action (via W(g[−][1] **xj) =**

_·_
**W(R[−][1](xj** **x)) that underlies group convolutions (Cohen & Welling, 2016). Message passing via**
explicit kernel rotations (l.h.s. of (13)) corresponds to − _regular group convolutions, and via steerable_
transformations (r.h.s. of (13)) to steerable group convolutions.

The equivariant steerable methods (Thomas et al., 2018; Anderson et al., 2019; Miller et al., 2020;
Fuchs et al., 2020) that we compare against in our experiments can all be written in convolution form


**Wa˜ij** [(][∥][x]j _[−]_ **[x]i[∥][)˜]fj,** or ˜fi[′] [=]
_j∈NX (i)_


**Wa˜ij** [(˜]fi, [˜]fj, ∥xj − **xi∥)[˜]fj,** (14)
_j∈NX (i)_


˜fi[′] [=]


where, in the latter case, the linear transformations additionally depend on an input dependent attention mechanism as in (Anderson et al., 2019; Fuchs et al., 2020), and can be seen as a steerable
PointConv version of attentive group convolutions (Romero et al., 2020). In these attention-based
cases, convolutions are augmented with input dependent weights α via Wa˜ij [(˜]fi, [˜]fj, ∥xj − **xi∥) =**
_αinput features still happens linearly and thus describes what one may call a pseudo-linear transforma-([˜]fi,_ [˜]fj)Wa˜ij [(][∥][x]j _[−]_ **[x]i[∥][)][. This makes the convolution non-linear, however, the transformation of]**
tion. Finally, the recently proposed LieConv (Finzi et al., 2020) and NequIP (Batzner et al., 2021)
also fall in the convolutional message passing class. LieConv is a PointConv-type variation of regular
_group convolutions on Lie groups (Bekkers, 2019). NeuqIP follows the approach of Tensor Field_
Networks (Thomas et al., 2018), and weighs interactions using an MLP with radial basis functions
as input. These functions are obtained as solution of the Schrodinger equation. Finally, steerable¨
methods fall into a more general class of coordinate independent convolutions (Weiler et al., 2021)
which allow to ensure equivariance locally, even when global symmetries can not be defined.

**Equivariant message passing as non-linear convolution.** EGNNs (Satorras et al., 2021) are
equivariant to transformations in E(n) and outperform most aforementioned steerable methods. This
is somewhat surprising as it sends invariant messages, which are obtained via MLPs of the form

**mij = MLP(fi, fj, ∥xj −** **xi∥[2]) = σ(W[(][k][)](. . . (σ(W[(1)]hi)))),** (15)

4Exact correspondence is obtained by a sum reduction over the steerable vector components (App. B).


-----

whereconvolutions due to their dependency on relative positions. There are, however, two important hi = (fi, fj, ∥xj − **xi∥[2]). These messages resemble the convolutional messages of point**
differences: (i) the messages are non-linear transformations of the neighbouring feature values fj
via an MLP and (ii) the messages are only conditioned on the distance between point pairs, and are
therefore E(n) invariant. As such, we regard EGNN layers as non-linear convolutions with isotropic
_message functions (the non-linear counterpart of rotationally invariant kernels). In our work, we lift_
the isotropy constraint and generalise to non-linear steerable convolutions via messages of the form

**m˜** _ij =_ MLP[]] **a˜ij** [(˜]fi, [˜]fj, ∥xj − **xi∥[2]) = σ(Wa[(]˜[n]ij[)][(][. . .][ (][σ][(][W]a[(1)]˜ij** **h[˜]i)))),** (16)

withcould e.g. be a spherical harmonic embedding ofh[˜]i = ([˜]fi, [˜]fj, ∥xj − **xi∥[2]) ∈** _Vf ⊕_ _Vf ⊕_ _V0. The MLP is then conditioned on attribute xj_ **xi. This allows for the creation of messages ˜aij, which**
more general than those found in convolution, while carrying covariant geometrical information. −

**Related equivariant message passing methods.** A different but also fully message passing
based approach can be found in Geometric Vector Perceptrons (GVP) (Jing et al., 2020), Vector
Neurons (Deng et al., 2021), and PaiNN (Schutt et al., 2021). Compared to SEGNNs which treat¨
equivariant information as fully steerable features, these architectures update scalar-valued attributes
using the norm of vector-valued attributes, and therefore with O(3) invariant information. These
methods restrict the flow of information between attributes of different types, whereas the ClebschGordan tensor product in SEGNNs allows for interaction between spherical harmonics of all orders
throughout the network. Methods such as Dimenet++ (Klicpera et al., 2020), SphereNet (Liu et al.,
2021), and GemNet (Klicpera et al., 2021) incorporate relative orientation in a second-order message
passing scheme that considers angles between the central point and neighbours-of-neighbours. In
contrast, our method directly leverages angular information in a first-order message passing scheme
using steerable vectors. We attribute the success of such methods to the fact that they equivariantly
process point clouds of local orientations (xi, rij) ∈ R[3] _× S[2], defined by relative positions between_
atoms, by sending messages between edges (local orientations) rather than nodes. As such, they can
be thought of as non-linear regular group convolutions on the homogeneous space of positions and
orientations (R[3] _× S[2]) with isotropic (zonal) message functions, where the symmetry constraint is_
induced by the quotient R[3] _× S[2]_ _≡_ SE(3)/SO(2) (Bekkers, 2019, Thm. 1).

4 EXPERIMENTS

**Implementation details. The implementation of SEGNN’s O(3) steerable MLPs is based on the**
e3nn library (Geiger et al., 2021a). We either define the steerable vector spaces as V = nVL=lmax
(N-body, QM9 experiments), i.e., n copies of steerable vector spaces up to order lmax, or by dividing
an n-dimensional vector space V into L approximately equally large type-l sub-vector spaces (OC20
experiments) as is done in Finzi et al. (2021). Furthermore, for a fair comparison between experiments
with different lmax, we choose n such that the total number of weights in the CG products corresponds
to that of a regular (type-0) linear layer. Further implementation details are in App. C.

**SEGNN architectures and ablations. We consider several variations of SEGNNs. On all tasks we**
have at least one fully steerable (lf > 0, la > 0) SEGNN tuned for the specific task at hand. We
perform ablation experiments to investigate two main principles that sets SEGNNs apart from the
literature. A1 The case of non-steerable vs steerable EGNNs is obtained by applying the same SEGNN
network with different specifications of maximal spherical harmonic order in the feature vectors
(lf ) and the attributes (la). EGNN (Satorras et al., 2021) arises as a special case with lf = la = 0.
These models will be labelled SEGNN. A2 In this ablation, we use steerable equivariant point conv
methods (Thomas et al., 2018) with messages as in Eq. (14) and regular gated non-linearities as
activation/update function, labelled SElinear, and compare it to the same network but with messages
obtained in a non-linear manner via 2-layer steerable MLPs as in Eq. (16), labelled as SEnon-linear.

**N-body system.** The charged N-body particle system experiment (Kipf et al., 2018) consists
of 5 particles that carry a positive or negative charge, having initial position and velocity in a 3dimensional space. The task is to estimate all particle positions after 1.000 timesteps. We build upon
the experimental setting introduced in (Satorras et al., 2021). Steerable architectures are designed
such that the parameter budget at lf = 1 and la = 1 matches that of the EGNN implementation. We
input the relative position to the center of the system and the velocity as vectors of type l = 1 with
odd parity. We further input the norm of the velocity as scalar, which altogether results in an input


-----

Table 1: Mean Squared Error (MSE) in the N-body system experiment, and forward time in seconds
for a batch size of 100 samples running on a GeForce RTX 2080 Ti GPU. Results except for EGNN
and SEGNN are taken from (Satorras et al., 2021) and verified. Runtimes are re-measured.

Method MSE Time [s]

SE(3)-Tr. (Fuchs et al., 2020) .0244 .0742
TFN (Thomas et al., 2018) .0155 .0182
NMP (Gilmer et al., 2017) .0107 .0017
Radial Field (K¨ohler et al., 2019) .0104 .0019
EGNN (Satorras et al., 2021) .0070± .00022 .0029

SElinear (lf = 2, la = 2) .0116± .00021 .0640
SEnon-linear (lf = 1, la = 1) .0060± .00019 .0310
SEGNNG (lf = 1, la = 1) .0056± .00025 .0250
SEGNNG+P (lf = 1, la = 1) .0043± .00015 .0260

vector **h[˜]** _V0_ _V1_ _V1. The output is embedded as difference vector to the initial position (odd_
parity), i.e. ∈ ˜o _⊕_ _V1. In doing so, we keep E( ⊕_ 3) equivariance for vector valued inputs and outputs.
_∈_
The edge attributes are obtained via the spherical harmonic embedding ofEq. 4. Messages additionally have the product of charges and absolute distance included. SEGNN xj − **xi as described in**
architectures are compared to steerable equivariant (SElinear) and steerable non-linear point conv
methods (SEnon-linear). Results and ablation studies are shown in Tab. 1. A full ablation is outlined
in App. C. Steerable architectures obtain the best results for lf = 1 and la = 1, and don’t benefit
from higher orders of lf and la. We consider a first SEGNN architecture where the node attributes
are the averaged edge embeddings, i.e. mean over relative orientation, labelled SEGNNG since
only geometric information is used. The second SEGNN architecture has the spherical harmonics
embedding of the velocity added to the node attributes. It can thus leverage geometric (orientation)
and physical (velocity) information, and is consequently labelled SEGNNG+P. Including physical
information in addition to geometric information in the node updates considerably boosts SEGNN
performance. We further test SEGNNs on a gravitational 100-body system (Sec. C.2 in the appendix).

**QM9. The QM9 dataset (Ramakrishnan et al., 2014; Ruddigkeit et al., 2012) consists of small**
molecules up to 29 atoms, where each atom is described with 3D position coordinates and one-hot
mode embedding of its atomic type (H, C, N, O, F). The aim is to regress various chemical properties
for each of the molecules, optimising on the mean absolute error (MAE) between predictions and
ground truth. We use the dataset partitions from Anderson et al. (2019). Table 2 shows SEGNN results
on the QM9 dataset. In Table 3, we show that by steering with the relative orientation between atoms
we observe that for higher (maximum) orders of steerable feature vectors, the performance increases,
especially when a small cutoff radius is chosen. While previous methods use relatively large cutoff
radii of 4.5-11A, we use a cutoff radius of 2[˚] A. Doing so results in a sharp reduction of the number[˚]
of messages per layer, as shown in App. C. Tables 2 and 3 together show that SEGNNs outperform
an architecturally comparable baseline EGNN, whilst stripping away attention modules from it and
reducing graph connectivity from fully connected to only 2A distant atoms. It is however apt to[˚]
note that runtime is still limited by the relatively expensive calculation of the Clebsch-Gordan tensor
products. We further note that SEGNNs produce results on par with the best performing methods
on the non-energy variables, however lag behind state of the art on the energy variables (G, H, U,
_U0). We conjecture that such targets could benefit from more involved (e.g. including attention or_
neighbour-neighbour interactions) or problem-tailored architectures, such as those compared against.

**OC20.** The Open Catalyst Project OC20 dataset (Zitnick et al., 2020; Chanussot et al., 2021),
consists of molecular adsorptions onto surfaces. We focus on the Initial Structure to Relaxed Energy
(IS2RE) task, which takes as input an initial structure and targets the prediction of the energy in the
final, relaxed state. The IS2RE training set consists of over 450,000 catalyst adsorbate combinations
with 70 atoms on average. Optimisation is done for MAE between the predicted and ground truth
energy. Additionally, performance is measured in the percentage of structures in which the predicted
energy is within a 0.02 eV threshold (EwT). The four test splits contain in-distribution (ID) catalysts
and adsorbates, out-of-domain adsorbates (OOD Ads), out-of-distribution catalysts (OOD Cat), and
out-of-distribution adsorbates and catalysts (OOD Both). Table 4 shows SEGNN results on the OC20
dataset and comparisons with existing methods. We compare to models which have obtained results
by training on the IS2RE training set such as SphereNet (Liu et al., 2021) and DimeNet++ (Klicpera


-----

et al., 2019; 2020). The best SEGNN performance is seen for lf = 1 and la = 1 (see App. C). A full
ablation study, comparing performance and runtime for different orders lf and la is found in App. C).

Table 2: Performance comparison on the QM9 dataset. Numbers are reported for Mean Absolute
Error (MAE) between model predictions and ground truth.

Task _α_ ∆ε _εHOMO_ _εLUMO_ _µ_ _Cν_ _G_ _H_ _R[2]_ _U_ _U0_ ZPVE
Units bohr[3] meV meV meV D cal/mol K meV meV bohr[3] meV meV meV

NMP .092 69 43 38 .030 .040 19 17 .180 20 20 1.50
SchNet * .235 63 41 34 .033 .033 14 14 .073 19 14 1.70
Cormorant .085 61 34 38 .038 .026 20 21 .961 21 22 2.02
L1Net .088 68 46 35 .043 .031 14 14 .354 14 13 1.56
LieConv .084 49 30 25 .032 .038 22 24 .800 19 19 2.28
TFN .223 58 40 38 .064 .101 -  -  -  -  -  - 
SE(3)-Tr. .142 53 35 33 .051 .054 -  -  -  -  -  - 
DimeNet++ * **.043** **32** 24 19 .029 .023 **7** **6** .331 6 6 1.21
SphereNet * .046 **32** **23** **18** .026 **.021** 8 **6** .292 7 6 **1.12**
PaiNN * .045 45 27 20 **.012** .024 **7** **6** **.066** **5** **5** 1.28
EGNN .071 48 29 25 .029 .031 12 12 .106 12 12 1.55

SEGNN (Ours) .060 42 24 21 .023 .031 15 16 .660 13 15 1.62

-  these methods use different train/val/test partitions.

Table 3: QM9 ablation study to compare SEGNN performances for different (maximum) orders of
steerable feature vectors (lf ) and attributes (la). Models with only trivial features are akin to the
invariant EGNN. The method with a fully connected graph uses soft edge estimation (Satorras et al.,
2021). Forward time is measured for a batch of 128 samples running on a GeForce RTX 3090 GPU.

Task _α_ ∆ε _εHOMO_ _εLUMO_ _µ_ _Cν_
Units Cutoff radius bohr[3] meV meV meV D cal/mol K Time [s]

(S)EGNN (lf = 0, la = 0) -  .091 53 34 28 .042 .043 0.016
(S)EGNN (lf = 0, la = 0) 5A[˚] .105 57 36 31 .047 .047 0.014
SEGNN (lf = 1, la = 1) 5A[˚] .080 49 29 25 .032 .034 0.058
(S)EGNN (lf = 0, la = 0) 2A[˚] .240 98 60 60 .340 .077 0.014
SEGNN (lf = 1, la = 2) 2A[˚] .074 48 27 25 .031 .035 0.046
SEGNN (lf = 2, la = 3) 2A[˚] .060 42 24 21 .023 .031 0.096

Table 4: Comparison on the OC20 IS2RE task in terms of Mean Absolute Error (MAE) between
model predictions and ground truth energy and % of predictions within ϵ = 0.02 eV of the ground
truth (EwT). Numbers are taken from the OC20 leaderboard. SEGNNs outperform all competitors.

|Energy MAE [eV] ↓|EwT ↑|
|---|---|


|Model ID OOD Ads OODCat OOD Both|ID OOD Ads OOD Cat OOD Both|
|---|---|


|Median baseline 1.7499 1.8793 1.7090 1.6636 CGCNN 0.6149 0.9155 0.6219 0.8511 SchNet 0.6387 0.7342 0.6616 0.7037 EdgeUpdateNet 0.5839 0.7252 0.6016 0.6862 EnergyNet 0.6366 0.717 0.6387 0.6626 DimeNet++ 0.5620 0.7252 0.5756 0.6613 SphereNet 0.5630 0.7030 0.5710 0.6380|0.71% 0.72% 0.89% 0.74% 3.40% 1.93% 3.10% 2.00% 2.96% 2.33% 2.94% 2.21% 3.48% 2.35% 3.30% 2.57% 3.30% 2.20% 3.07% 2.34% 4.25% 2.07% 4.10% 2.41% 4.47% 2.29% 4.09% 2.41%|
|---|---|


|SEGNN (Ours) 0.5327 0.6921 0.5369 0.6790|5.37% 2.46% 4.91% 2.63%|
|---|---|



5 CONCLUSION

We have introduced SEGNNs which generalise equivariant graph networks, such that node and edge
information is not restricted to be invariant (scalar), but can also be vector- or tensor-valued. SEGNNs
are the first networks which allow the steering of node updates by leveraging geometric and physical
cues, introducing a new class of equivariant activation functions. We demonstrate the potential of
SEGNNs by applying it to a wide range of different tasks. Extensive ablation studies have further
shown the benefit of steerable over non-steerable (invariant) message passing, and the benefit of
non-linear over linear convolutions. On the OC20 ISRE taks, SEGNNs outperform all competitors.


-----

6 REPRODUCIBILITY STATEMENT

We have included error bars, reproducibility checks and ablation studies wherever we found it
necessary and appropriate. For example, for the N-body experiment we have reproduced the results
of Satorras et al. (2021), we have ablated different (maximum) orders of steerable feature vectors,
and we have stated mean and standard deviation of the results which we have obtained by running the
same experiments eight times with different initial seeds. For the OC20 experiments, we have stated
official numbers from the Open Catalyst Project challenge in the paper. Thus, our comparisons to
other methods are obtained from their respective best entries to the challenge. We have further ablated
runtimes, different cutoff radii and different (maximum) orders of steearable feature vectors, see
Sec. C in the appendix. We have done this by running the experiments eight times with different initial
random seeds to be able to report mean and standard deviation of the results. For the reproducibility
of the QM9 experiments, we have uploaded our code and included a command which reproduces
results of the α variable.

We have described our architecture in Sec. 2.1 and provided further implementation details in
Appendix Section C. We have not introduced new mathematical results. However, we have used
concepts from different mathematical fields and therefore included a self-contained mathematical
exposition in our appendix. We have further included proof of equivariance and properties of Wigner
D matrices and spherical harmonics in the appendix. In Sec. 3, we have introduced a unifying view
on various equivariant graph neural networks through the definition of non-linear convolutions, which
allows us to draw comparisons with many other methods; we verify our findings in the N-body
experiments of Sec. 4.

7 ETHICAL STATEMENT

The societal impact of SEGNNs is difficult to predict. However, SEGNNs are well suited for physical
and chemical modeling and therefore potential shortcuts for computationally expensive simulations.
And if used as such, SEGNNs might potentially be directly or indirectly related to reducing the
carbon footprint. On the downside, relying on simulations always requires rigorous cross-checks and
monitoring, especially when simulations or simulated quantities are learned.

ACKNOWLEDGMENTS

Johannes Brandstetter thanks the Institute of Advanced Research in Artificial Intelligence (IARAI)
and the Federal State Upper Austria for the support. This work is part the research programme VENI
(grant number 17290), financed by the Dutch Research Council (NWO). The authors thank Markus
Holzleitner for helpful comments on this work.

REFERENCES

Sanchez-Gonzalez Alvaro, Godwin Jonathan, Pfaff Tobias, Ying Rex, Leskovec Jure, and Battaglia
Peter. Learning to simulate complex physics with graph networks. In Proceedings of the 37th
_International Conference on Machine Learning, volume 119, pp. 8459–8468, 2020._

Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural
networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett´
(eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.,
2019.

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi,
Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar
Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey
Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet
Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases,
deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.


-----

Simon Batzner, Tess E Smidt, Lixin Sun, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari,
and Boris Kozinsky. Se (3)-equivariant graph neural networks for data-efficient and accurate
interatomic potentials. arXiv preprint arXiv:2101.03164, 2021.

Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations,
2019.

Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco
Duits. Roto-translation covariant convolutional networks for medical image analysis. In Interna_tional conference on medical image computing and computer-assisted intervention, pp. 440–448._
Springer, 2018.

[Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.](https://www.wandb.com/)
[com/. Software available from wandb.com.](https://www.wandb.com/)

Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane
Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop Sriram,
Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi. Open
catalyst 2020 (oc20) dataset and community challenges. ACS Catalysis, 0(0):6059–6072, 2020. doi:
[10.1021/acscatal.0c04525. URL https://doi.org/10.1021/acscatal.0c04525.](https://doi.org/10.1021/acscatal.0c04525)

Lowik Chanussot, Abhishek Das, Siddharth Goyal, Thibaut Lavril, Muhammed Shuaibi, Morgane
Riviere, Kevin Tran, Javier Heras-Domingo, Caleb Ho, Weihua Hu, Aini Palizhati, Anuroop
Sriram, Brandon Wood, Junwoong Yoon, Devi Parikh, C. Lawrence Zitnick, and Zachary Ulissi.
The open catalyst 2020 (oc20) dataset and community challenges, 2021.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference
_on machine learning, pp. 2990–2999. PMLR, 2016._

Taco S. Cohen and Max Welling. Steerable cnns. In International Conference on Learning Represen_tations (ICLR), 2017._

Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical cnns. In International
_Conference on Learning Representations (ICLR), 2018._

Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks¨
on graphs with fast localized spectral filtering. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon,
and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran
Associates, Inc., 2016.

Walter Dehnen and Justin I Read. N-body simulations of gravitational dynamics. The European
_Physical Journal Plus, 126(5):1–28, 2011._

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas
Guibas. Vector neurons: A general framework for so(3)-equivariant networks. arXiv preprint
_arXiv:2104.12229, 2021._

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In
_ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019._

Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional
neural networks for equivariance to lie groups on arbitrary continuous data. In International
_Conference on Machine Learning, pp. 3165–3176. PMLR, 2020._

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. arXiv preprint arXiv:2104.09459, 2021.

William T Freeman, Edward H Adelson, et al. The design and use of steerable filters. IEEE
_Transactions on Pattern analysis and machine intelligence, 13(9):891–906, 1991._

Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d
roto-translation equivariant attention networks. arXiv preprint arXiv:2006.10503, 2020.


-----

Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi, Maurice Weiler, Michał Tyszkiewicz, Simon Batzner, Jes Frellsen, Nuri Jung,
Sophia Sanborn, Josh Rackers, and Michael Bailey. e3nn/e3nn: 2021-04-21, April 2021a. URL
[https://doi.org/10.5281/zenodo.4708275.](https://doi.org/10.5281/zenodo.4708275)

Mario Geiger, Tess Smidt, Alby M., Benjamin Kurt Miller, Wouter Boomsma, Bradley Dice, Kostiantyn Lapchevskyi, Maurice Weiler, Michał Tyszkiewicz, Simon Batzner, Jes Frellsen, Nuri Jung,
Sophia Sanborn, Josh Rackers, and Michael Bailey. e3nn/e3nn: 2021-05-10, May 2021b. URL
[https://doi.org/10.5281/zenodo.4745784.](https://doi.org/10.5281/zenodo.4745784)

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural
message passing for quantum chemistry. In International Conference on Machine Learning, pp.
1263–1272. PMLR, 2017.

Yacov. Hel-Or and Patrick Teo. Canonical decomposition of steerable functions. In Proceedings
_CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp._
809–816, 1996.

Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learning
from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
_arXiv:1412.6980, 2014._

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational
inference for interacting systems. In International Conference on Machine Learning, pp. 2688–
2697. PMLR, 2018.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
In International Conference on Learning Representations (ICLR), 2017.

Johannes Klicpera, Janek Groß, and Stephan Gunnemann. Directional message passing for molecular¨
graphs. In International Conference on Learning Representations, 2019.

Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Gunnemann.¨ Fast and
uncertainty-aware directional message passing for non-equilibrium molecules. arXiv preprint
_arXiv:2011.14115, 2020._

Johannes Klicpera, Florian Becker, and Stephan Gunnemann. Gemnet: Universal directional graph¨
neural networks for molecules. arXiv preprint arXiv:2106.08903, 2021.

Jonas Kohler, Leon Klein, and Frank No¨ e. Equivariant flows: sampling configurations for multi-body´
systems with symmetric energies. arXiv preprint arXiv:1910.00753, 2019.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural
networks to the action of compact groups. In International Conference on Machine Learning, pp.
2747–2755. PMLR, 2018.

Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch–gordan nets: a fully fourier space spherical
convolutional neural network. Advances in Neural Information Processing Systems, 31:10117–
10126, 2018.

Schutt Kristof, Kindermans Pieter-Jan, Sauceda Huziel, Chmiela Stefan, Tkatchenko Alexandre,¨
and Klaus-Robert Muller. Schnet: a continuous-filter convolutional neural network for modeling¨
quantum interactions. In Proceedings of the 31st International Conference on Neural Information
_Processing Systems, pp. 992–1002, 2017._

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097–1105,
2012.

Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kernels.
In International Conference on Learning Representations, 2020.


-----

Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to´
document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444,
2015.

Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message
passing for 3d graph networks. arXiv preprint arXiv:2102.05013, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. _arXiv preprint_
_arXiv:1711.05101, 2017._

Andreas Mayr, Sebastian Lehner, Arno Mayrhofer, Christoph Kloss, Sepp Hochreiter, and Johannes
Brandstetter. Boundary graph neural networks for 3d simulations. arXiv preprint arXiv:2106.11299,
2021.

Benjamin Kurt Miller, Mario Geiger, Tess E Smidt, and Frank Noe. Relevance of rotationally´
equivariant convolutions for predicting molecular properties. arXiv preprint arXiv:2008.08461,
2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style,
high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.

Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiv preprint
_arXiv:1710.05941, 2017._

Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld. Quantum
chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 08 2014.

David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivariant
convolutional networks. In Proceedings of the 37th International Conference on Machine Learning,
Proceedings of Machine Learning Research, pp. 8188–8199, 2020.

Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166
billion organic small molecules in the chemical universe database gdb-17. Journal of chemical
_information and modeling, 52(11):2864–2875, 2012._

Jun J. Sakurai and Jim Napolitano. Modern Quantum Mechanics. Cambridge University Press, 2
edition, 2017.

Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks.
_arXiv preprint arXiv:2102.09844, 2021._

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The
graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009.

Jurgen Schmidhuber. Deep learning in neural networks: An overview.¨ _Neural networks, 61:85–117,_
2015.

Kristof T Schutt, Oliver T Unke, and Michael Gastegger. Equivariant message passing for the¨
prediction of tensorial properties and molecular spectra. arXiv preprint arXiv:2102.03150, 2021.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.
Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.
_arXiv preprint arXiv:1802.08219, 2018._

Pfaff Tobias, Fortunato Meire, Sanchez-Gonzalez Alvaro, and Battaglia Peter W. Learning meshbased simulation with graph networks, 2020.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing
ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.

Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural
_Information Processing Systems, 32:14334–14345, 2019._


-----

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns:
Learning rotationally equivariant features in volumetric data. In Advances in Neural Information
_Processing Systems, volume 31. Curran Associates, Inc., 2018._

Maurice Weiler, Patrick Forre, Erik Verlinde, and Max Welling. Coordinate independent convolutional´
networks–isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint
_arXiv:2106.06020, 2021._

Marysia Winkels and Taco S Cohen. 3d g-cnns for pulmonary nodule detection. In International
_Conference on Medi- cal Imaging with Deep Learning (MIDL), 2018._

Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
_Proceedings of the European Conference on Computer Vision (ECCV), pp. 567–584, 2018._

Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic
networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on
_Computer Vision and Pattern Recognition, pp. 5028–5037, 2017._

Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point
clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 9621–9630, 2019.

Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and
interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018.

C. Lawrence Zitnick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras-Domingo,
Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, Muhammed Shuaibi,
Anuroop Sriram, Kevin Tran, Brandon Wood, Junwoong Yoon, Devi Parikh, and Zachary Ulissi.
An introduction to electrocatalyst design using machine learning for renewable energy storage,
2020.


-----

A MATHEMATICAL BACKGROUND

This appendix provides the mathematical background and intuition for steerable MLPs. We remark
that the reader may appreciate several related works, such as, (Thomas et al., 2018; Anderson
et al., 2019; Fuchs et al., 2020), as excellent alternative resources[5] to get acquainted with the
group/representation theory used in this paper. In this appendix we introduce the theory from our
own perspective which is tuned towards the idea of steerable MLPs and our viewpoint on group
convolutions. It provides complementary intuition to the aforementioned resources. The main
concepts explained in this appendix are:

1. Group definition and examples of groups (Section A.1). The entire framework builds upon
notions from group theory and as such a formal definition is in order. In this paper, we
model transformations such as translation, rotation and reflection as groups.

2. Invariance, equivariance and representations (Section A.2). A function is said to be invariant
to a transformation if its output is unaffected by a transformation of the input. A function
is said to be equivariant if its output transforms predictably under a transformation of the
input. In order to make the definition precise, we need a definition of representations; a
representation formalises the notion of transformations applied to vectors in the context of
group theory.

3. Steerable vectors, Wigner-D matrices and irreducible representations (Section A.3).
Whereas regular MLPs work with feature vectors whose elements are scalars, our steerable
MLPs work with feature vectors consisting of steerable feature vectors. Steerable feature
vectors are vectors that transform via so-called Wigner-D matrices, which are representations
of the orthogonal group O(3). Wigner-D matrices are the smallest possible group representations and can be used to define any representation (or conversely, any representation can
be reduced to a tensor product of Wigner-D matrices via a change of basis). As such, the
Wigner-D matrices are irreducible representations.

4. Spherical harmonics (Section A.4). Spherical harmonics are a class of functions on the
sphere S[2] and can be thought of as a Fourier basis on the sphere. We show that spherical
harmonics are steered by the Wigner-D matrices and interpret steerable vectors as functions
on S[2], which justifies the glyph visualisations used in this paper. Moreover, spherical
harmonics allow the embedding of three-dimensional displacement vectors into arbitrarily
large steerable vectors.

5. Clebsch-Gordan tensor product and steerable MLPs (Section A.5). In a regular MLP one
maps between input and output vector spaces linearly via matrix vector multiplication and
applies non-linearities afterwards. In steerable MLPs one maps between steerable input
and steerable output vector spaces via the Clebsch-Gordan tensor product. Akin to the
learnable weight matrix in regular MLPs, the learnable Glebsch-Gordan tensor product is
the workhorse of our steerable MLPs.

After these concepts are introduced we will in Section B revisit the convolution operator in the light
of the steerable, group theoretical viewpoint that we take in this paper. In particular, we show that
steerable group convolutions are equivalent to linear group convolutions with convolution kernels
expressed in a spherical harmonic basis. With this in mind, we argue that our approach via message
passing can be thought of as building neural networks via non-linear group convolutions.

A.1 GROUP DEFINITION AND THE GROUPS E(3) AND O(3)

**Group definition.** A group is an algebraic structure that consists of a set G and a binary operator ·,
the group product, that satisfies the following axioms: Closure: for all h, g ∈ _G we have h · g ∈_ _G;_
_Identity: there exists an identity element e ∈_ _G; Inverse: for each g ∈_ _G there exists an inverse_
element g[−][1] _∈_ _G such that g[−][1]_ _· g = g · g[−][1]_ = e; and Associativity: for each g, h, i ∈ _G we have_
(g · h) · i = g · (h · i).

**The Euclidean group E(3).** In this work, we are interested in the group of three-dimensional
translations, rotations, and reflections which is denoted with E(3), the 3D Euclidean group. Such

5Each of these works presents unique view points that greatly influenced the writing of this appendix.


-----

transformations are parametrised by pairs of translation vectors x ∈ R[3] and orthogonal transformation
matrices R ∈ O(3). The E(3) group product and inverse are defined by

_g · g[′]_ := (Rx[′] + x, RR[′]),

_g[−][1]_ := (R[−][1]x, R[−][1]),

with g = (x, R), g[′] = (x[′], R) ∈ E(3). One can readily see that with these definitions all four
group axioms are satisfied, and that it therefore defines a group. The group product can be seen as a
description for how two E(3) transformations parametrised by g and g[′] applied one after another are
described by single transformation parametrised by g · g[′]. The transformations themselves act on the
vector space of 3D positions via the group action, which we also denote with ·, via


_g · y := Ry + x,_


where g = (x, R) ∈ E(3) and y ∈ R[3].


**The orthogonal group O(3) and special orthogonal group SO(3).** The group E(3) = R[3] ⋊ O(3)
is a semi-direct product (denoted with ⋊) of the group of translations R[3] with the group of orthogonal
transformations O(3). This means that we can conveniently decompose E(3)-transformations in an
O(3)-transformation (rotation and/or reflection) followed by a translation. In this work we will mainly
focus on dealing with O(3) transformations as translations are trivially dealt with. When representing
the group elements of O(3) with matrices R, as we have done before, the group product and inverse
are simply given by the matrix-product and matrix-inverse. I.e., with g = R, g[′] = R[′] _∈_ _G = O(3)_
the group product and inverse are defined by

_g · g[′]_ := RR[′] _,_

_g[−][1]_ := R[−][1] _._


The group acts on R[3] by matrix-vector multiplication, i.e., g · y := Ry. The group elements of O(3)
are square matrices with determinant −1 or 1. Their action on R[3] defines a reflection and/or rotation.

The special orthogonal group SO(3) has the same group product and inverse, but excludes reflections.
The group thus consists of matrices with determinant 1.

**The sphere S[2]** **is a homogeneous space of SO(3).** The sphere is not a group as we cannot define a
group product on S[2] that satisfies the group axioms. It can be convenient to treat it as a homogeneous
space of the groups O(3) or SO(3). A space X is called a homogeneous space of a group G if for any
two points x, y ∈X there exists a group element g ∈ _G such that y = g · x._

The sphere S[2] is a homogeneous space of the rotation group SO(3) since any point on the sphere can
be reached via the rotation of some reference vector. Consider for example an XYX parametrisation
of SO(3) rotations in which three rotations are applied one after another via

**Rα,β,γ = Rα,nx** **Rβ,ny** **Rγ,nx,** (A.1)

with nx and ny denoting unit vectors along the x and y axis, and Rα,nx denotes a rotation of α
degrees around axis nx. We can model points on the sphere in a similar way via Euler angles via

**nα,β := Rα,β,0nx .** (A.2)

So, with two rotation angles, any point on the sphere can be reached. In the above we set γ = 0 in
the parametrisation of the rotation matrix (an element from SO(3)) that rotates the reference vector
**nx, but it should be clear that with any γ the same point nα,γ is reached. This means that there are**
many group elements in SO(3) that all map nx to the same point on the sphere.

A.2 INVARIANCE, EQUIVARIANCE AND REPRESENTATIONS

**Group representations.** We previously defined the group product, which tells us how elements of
a group interact. We also showed that the groups E(3) and O(3) can transform the three dimensional
vector space R[3] via the group action. We usually think of the groups E(3) and O(3) as groups that
describe transformations on R[3], but these groups are not restricted to transformations on R[3] and can
generally act on arbitrary vector spaces via representations. A representation is an invertible linear


-----

transformation ρ(g) : V → _V parametrised by group elements g ∈_ _G that acts on some vector space_
_V, and which follows the group structure (it is a group homomorphism) via_


_ρ(g)ρ(h)v = ρ(g · h)v,_

with v ∈ _V ._

A representation can act on infinite-dimensional vector spaces such as functions. E.g., the so-called
_left-regular representations Lg of E(3) on functions f : R[3]_ _→_ R on R[3] is given by

_g[f_ ](x) = f (g[−][1]x),
_L_

i.e., it transforms the function by letting g[−][1] act on the domain from the left. Here we used the
notation Lg[f ] to indicate that Lg transforms the function f first, which creates a new function
_L(g)[f_ ](x), which is then sampled at x.

When representations transform finite dimensional vectors v ∈ _V = R[d], they are d × d dimensional_
matrices. In this work, we denote such matrix representations with boldface D. A familiar example of
a matrix representation of O(3) on R[3] are the matrices g = R ∈ O(3) themselves, i.e., D(g)x = Rx.

Finally, any two representations, say D(g) and D[′](g), are equivalent if they relate via a similarity
transform via
**D[′](g) = Q[−][1]D(g)Q,**

i.e., such representations describe one and the same thing but in a different basis, and the change of
basis is carried out by Q. Now that representations have been introduced we can formally define
equivariance.

**Invariance and equivariance.** Equivariance is a property of an operator φ : X →Y that maps
between input and output vector spaces X and Y. Given a group G and its representations ρ[X] and ρ[Y]
which transform vectors in X and Y respectively, an operator φ : X →Y is said to be equivariant if
_it satisfies the following constraint_

_ρ[Y]_ (g)[φ(x)] = φ(ρ[X] (g)[x]), for all g ∈ _G, x ∈X ._ (A.3)

Thus, with an equivariant map, the output transforms predictably with transformations on the input.
One might say that no information gets lost when the input is transformed, merely re-structured.
One way to interpret Eq. (A.3) is therefore that the operators ρ[X] (g) : X →X and ρ[Y] (g) : Y →Y
describe the same transformation, but in different spaces.

Invariance is a special case of equivariance in which ρ[Y] = I _[Y]_ for all g ∈ _G. I.e., an operator_
_φ : X →Y is said to be invariant if it satisfies the following constraint_

_φ(x) = φ(ρ[X]_ (g)[x]), for all g ∈ _G, x ∈X ._ (A.4)

Thus, with an invariant operator, the output of φ is unaffected by transformations applied to the input.


A.3 STEERABLE VECTORS, WIGNER-D MATRICES AND IRREDUCIBLE REPRESENTATIONS

One strategy to build equivariant MLPs is to define input and output spaces of the MLPs and define
how these spaces transform under the action of a group. This then sets an equivariance constraint on
the operator that maps between these spaces. By only working with such equivariant operators we
can guarantee that the entire learning framework is equivariant.

In our work, the proposed graph neural networks are translation equivariant by construction as any
form of spatial information only enters the pipeline in the form of relative positions between nodes
(xj **xi). Then, any remaining operations are designed to be O(3) equivariant such that, together**
with the given translation equivariance, the complete framework is fully E( − 3) equivariant. Since
translations are trivially dealt with, we focus on SO(3) and O(3) and show how to build equivariant
MLPs through the use of the Clebsch-Gordan tensor product.

**Wigner-D matrices are irreducible representations.** For SO(3) there exists a collection of representations, indexed with their order l ≥ 0, which act on vector spaces of dimension 2l + 1. These
representations are called Wigner-D matrices and we denote them with D[(][l][)](g). The use of Wigner-D


-----

matrices is motivated by the fact that any matrix representation D(g) of SO(3) that acts on some
vector space V can be “reduced” to an equivalent block diagonal matrix representation with Wigner-D
matrices along the diagonal:

**D[(][l][1][)](g)**

**D(g) = Q[−][1](D[(][l][1][)](g)** **D[(][l][2][)](g)** _. . . )Q = Q[−][1]_ **D[(][l][2][)](g)** **Q,** (A.5)

 

_⊕_ _⊕_

...

 
 

with Q the change of basis that makes them equivalent. The individual Wigner-D matrices themselves
cannot be reduced and are hence irreducible representations of SO(3). Thus, since the block diagonal
representations are equivalent to D we may as well work with them instead. This is convenient since
each block, i.e., each Wigner-D matrix D[(][l][i][)], only acts on a sub-space Vl1 of V . As such we can
factorise V = Vl1 _Vl2_ _. . ., which motivates the use of steerable vector spaces and their direct_
sums as presented in Sec. 2. ⊕ _⊕_

The Wigner-D matrices are the irreducible representations of SO(3), but we can easily adapt these
representations to be suitable for O(3) by including the group of reflections as a direct product. We
will still refer to these representations as Wigner-D matrices in the entirety of this work, opting to
avoid the distinction in favour of clarity of exposition. We further remark that explicit forms of
the Wigner-D matrices can e.g. be found books such as in Sakurai & Napolitano (2017) and their
numerical implementations in code libraries such as the e3nn library (Geiger et al., 2021a).

**Steerable vector spaces.** The (2l + 1)-dimensional vector space on which a Wigner-D matrix of
order l acts will be called a type l steerable vector space and is denoted with Vl. E.g., a type-3 vector
**h ∈** _V3 is transformed by g ∈_ O(3) via h 7→ **D3(g)h. We remark that this definition is equivalent to**
the definition of steerable functions commonly used in computer vision (Freeman et al., 1991; Hel-Or
& Teo, 1996) via the viewpoint that steerable vectors can be regarded as the basis coefficients of a
function expanded in a spherical harmonic basis. We elicit this viewpoint in Sec. A.4 and B.

At this point we are already familiar with type-0 and type-1 steerable vector spaces. Namely, type-0
vectors h ∈ _V0 = R are scalars, which are invariant to transformations g ∈_ O(3), i.e., D0(g)h = h.
Type-1 features are vectors h ∈ R[3] which transform directly via the matrix representation of the
group, i.e., D1(g)h = Rh.

A.4 SPHERICAL HARMONICS

**Spherical harmonics.** Related to the Wigner-D matrices and their steerable vector spaces are
the spherical harmonics[6]. Spherical harmonics are a class of functions on the sphere S[2], akin to
the familiar circular harmonics that are best known as the 1D Fourier basis. As with a Fourier
basis, spherical harmonics form an orthonormal basis for functions on S[2]. In this work we use the
real-valued spherical harmonics and denote them with Ym[(][l][)] [:][ S][2][ →] [R][.]

**Spherical Harmonics are Wigner-D functions.** One can also think of spherical harmonics as
functions _Y[˜]m[(][l][)]_ [:][ SO][(3)][ →] [R][ on SO][(3)][ that are invariant to a sub-group of rotations via]

_Ym[(][l][)][(][n][α,β][) =][ Y][ (]m[l][)][(][R][α,β,γ][n][x][) =: ˜]Ym[(][l][)][(][R][α,β,γ][)][,]_

in which we used the parametrisation for S[2] and O(3) given in (A.2) and (A.1) respectively. Then,
by definition, _Y[˜]m[(][l][)]_ [is invariant with respect to rotation angle][ γ][, i.e..,][ ∀]γ [0,2π) [:] _Y˜m[(][l][)][(][R]α,β,γ[) =]_
_∈_
_Y˜m[(][l][)][(][R]α,β,0[)][. This viewpoint of regarding the spherical harmonics as][ γ][-invariant functions on O(][3][)]_
helps us to draw the connection to the Wigner-D functions Dmn[(][l][)] [that make up the][ 2][l][ + 1][ ×][ 2][l][ + 1]
elements of the Wigner-D matrices. Namely, the n = 0 column of Wigner-D functions are also
_γ-invariant and, in fact, correspond (up to a normalisation factor) to the spherical harmonics via_


1

_m0[(][R][α,β,γ][)][ .]_ (A.6)
2l + 1 _[D][(][l][)]_


_Ym[(][l][)][(][n][α,β][) =]_


6Solutions to Laplace’s equation are called harmonics. Solutions of Laplace’s equation on the sphere are
therefore called spherical harmonics.


-----

**The mapping from vectors into spherical harmonic coefficients is equivariant.** It then directly
follows that vectors of spherical harmonics are steerable by the Wigner-D matrices of the same degree.
Let a[(][l][)](n) := (Y−[(][l]l[)][(][n][)][, . . ., Y]l[ (][l][)](n))[T] be the embedding of a direction vector n ∈ _S[2]_ in spherical
harmonics. Then this vector embedding is equivariant as it satisfies

_∀R′∈SO(3) ∀n∈S2 :_ **a[(][l][)](R[′]n) = D[(][l][)](R[′])a[(][l][)](n) .** (A.7)
Using the S[2] and O(3) parametrization of (A.2) and (A.1) this is derived as follows

_Y_ [(][l]l[)][(][R][′][n][α,β][)] _D[(][l][)]l0[(][R][′][R][α,β,γ][)]_ _D[(][l][)]l0[(][R][α,β,γ][)]_
_−_ _−_ _−_
. . .

**a[(][l][)](R[′]nα,β) =**  ..  =  ..  = D[(][l][)](R[′])  .. 

Yl[(][l][)](R[′]nα,β)  _Dl[(]0[l][)][(][R][′][R][α,β,γ][)]_   _Dl[(]0[l][)][(][R][α,β,γ][)]_ 
     

_Y_ [(][l]l[)][(][n][α,β][)]
_−_
.

= D[(][l][)](R[′])  ..  = D[(][l][)](R[′])a[(][l][)](nα,β) .

Yl[(][l][)](nα,β)
 

**Steerable vectors represent steerable functions on S[2].** Just like the 1D Fourier basis forms a
complete orthonormal basis for 1D functions, the spherical harmonics Ym[(][l][)] [form an orthonormal]
basis for L2(S[2]), the space of square integrable functions on the sphere. Any function on the sphere
can thus be represented by a steerable vector ˜a _V0_ _V1_ _. . . when it is expressed in a spherical_
harmonic basis via _∈_ _⊕_ _⊕_

_l_

_f_ (n) = _a[(]m[l][)][Y][ (]m[l][)][(][n][)][ .]_ (A.8)
Xl≥0 _mX=−l_

Since spherical harmonics form an orthonormal basis, the coefficient vector a can directly be obtained
by taking L2(S[2])-inner products of the function f with the spherical harmonics, i.e.,


_a[(]m[l][)]_ [= (][f, Y]m[ (][l][)][)]L2(S[2]) [=]


_m_ [(][n][)d][n][ .] (A.9)
_S[2][ f]_ [(][n][)][Y][ (][l][)]


Equation (A.9) is sometimes referred to as the Fourier transform on S[2], and Eq. (A.8) as the inverse
spherical Fourier transform. Thus, one can identify steerable vectors a with functions on the sphere
_S[2]_ via the spherical Fourier transform.

In this paper, we visualise such functions on S[2] via glyph-visualisations which are obtained as surface
plots by taking n ∈ _S[2]_ and scaling it by |f (n)| = sign (f (n)) f (n):

**n |f** (n)| **n ∈** _S[2]_ _⇐⇒_


where each point on this surface is color-coded with the function value f (n). The visualisations are
thus color-coded spheres that are stretched in each direction n via ∥f (n)∥.

**Steerable vectors also represent steerable functions on O(3).** In order to draw a connection
between group equivariant message passing and group convolutions, as we did in Sec. 3 of the main
paper, it is important to understand that steerable vectors also represent functions on the group SO(3)
via an SO(3)-Fourier transform. The collection of (2l + 1)[2] Wigner-D functions Dmn[(][l][)] [form an]
orthonormal basis for L2(SO(3)). This orthonormal basis allows for a Fourier transform that maps
between the function space L2(SO(3)) and steerable vector space V ; the forward and inverse Fourier
transform on SO(3) are respectively given by


_a[(]mn[l][)]_ [=]

_f_ (g) =


_f_ (g)Dmn[(][l][)] [(][g][)d][g,] (A.10)
O(3)


_a[(]mn[l][)]_ _[D]mn[(][l][)]_ [(][g][)][,] (A.11)
_nX=−l_


_l≥0_


_m=−l_


-----

with dg the Haar measure of the group. Noteworthy, the forward Fourier transform generates a matrix
of Fourier coefficients, rather than a vector in spherical case. The coefficent matrix is steerable by
left-multiplication with the Wigner-D matrices of the same type D[(][l][)].

A.5 CLEBSCH-GORDAN PRODUCT AND STEERABLE MLPS

In a regular MLP one maps between input and output vector spaces linearly via matrix-vector
multiplication and applies non-linearities afterwards. In steerable MLPs, one maps between steerable
input and output vector spaces via the Clebsch-Gordan tensor product and applies non-linearities
afterwards. Akin to the learnable weight matrix in regular MLPs, the learnable Glebsch-Gordan
tensor product is the main workhorse of our steerable MLPs.

**Clebsch-Gordan tensor product.** The Clebsch-Gordan (CG) tensor product allows us to map
between steerable input and output spaces. While there is much to be said about tensors and tensor
products in general, we here intend to focus on intuition. In general, a tensor product involves the
multiplication between all components of two input vectors. E.g., with two vectorsh2 ∈ R[d][2], the tensor product is given by **h1 ∈** R[d][1] and

_h1h1_ _h1h2_ _. . ._

**h1 ⊗** **h2 = h1h[T]2** [=] h2...h1 _h2...h2_ _. . ...._ _,_

 
 

which we can flatten into a d1d2-dimensional vector via an operation which we denote with vec(h1
**h2). In our steerable setting we would like to work exclusively with steerable vectors and as such we ⊗**
would like for any two steerable vectorsagain steerable with a O(3) representationh[˜] D1 ∈(gV)l such that the following equivariance constraint is1 and **h[˜]2 ∈** _Vl2, that the tensor product’s output is_
satisfied:
**D(g)(h[˜]1** **h2) = (D[(][l][1][)](g)h[˜]1)** (D[(][l][2][)](g)h[˜]2) . (A.12)
_⊗_ [˜] _⊗_

Via the identity vec(AXB) = (B[T] _⊗_ **A) vec(X), we can show that the output is indeed steerable:**

vec (D[(][l][1][)](g)h[˜]1)(D[(][l][2][)](g)h[˜]2)[T][ ] = vec **D[(][l][1][)](g)h[˜]1h[˜][T]2** **[D][(][l][2][)][T][ (][g][)]**
  

= **D[(][l][2][)](g)** **D[(][l][1][)](g)** vec **h˜1h[˜][T]2** _._
_⊗_
   

The resulting vector vec( h[˜]1 **h2) is thus steered by a representation D(g) = D[(][l][2][)](g)** **D[(][l][1][)](g).**
Since any matrix representation of O( ⊗ [˜] 3) can be reduced to a direct sum of Wigner-D matrices (see ⊗
(A.5)), the resulting vector can be organised via a change of basis into parts that individually transform
via Wigner-D matrices of different type. I.e.steerable sub-vector spaces of type l. **h[˜] = vec(h[˜]1 ⊗** **h[˜]2) ∈** _V = V0 ⊕_ _V1 ⊕_ _. . ., with Vl the_

With the CG tensor product we directly obtain the vector components for the steerable sub-vectors of
type l as follows. Let **h[˜][(][l][)]** _∈_ _Vl = R[2][l][+1]_ denote a steerable vector of type l and h[(]m[l][)] [its components]
with m = −l, −l + 1, . . ., l. Then the m-th component of the type l sub-vector of the output of the
tensor product between two steerable vectors of type l1 and l2 is given by


_l1_

_m1X=−l1_


_l2_

_C([(]l[l,m]1,m[)]1)(l2,m2)[h]m[(][l][1]1[)][h][(]m[l][2]2[)]_ _[,]_ (A.13)
_m2X=−l2_


(h[˜][(][l][1][)] _⊗cg_ **h[˜][(][l][2][)])[(]m[l][)]** [=][ w]


in which w is a learnable parameter that scales the product and C([(]l[l,m]1,m[)]1)(l2,m2) [are the Clebsch-]
Gordan coefficients. The CG tensor product is a sparse tensor product, as generally many of the
_C([(]l[l,m]1,m[)]1)(l2,m2)_ [components are zero. Most notably,][ C]([(]l[l,m]1,m[)]1)(l2,m2) [= 0][ whenever][ l <][ |][l][1][ −] _[l][2][|][ or]_
_l > l1 + l2. E.g., a type-0 and a type-1 feature vector cannot create a type-2 feature vector. Well_
known examples of the CG product are the scalar product (l1 = 0, l2 = 1, l = 1), which takes as
input a scalar and a type-1 vector to generate a type-1 vector, the dot product (l1 = 1, l2 = 1, l = 0)
and cross product (l1 = 1, l2 = 1, l = 1).

Examples of instances of the CG product are:


-----

-  The product of two scalars is a CG product which takes as input two type-0 “vectors” to
generate another scalar (l1 = 0, l2 = 0, l = 0).

-  The scalar product, which takes as input a scalar and a type-1 vector to generate a type-1
vector (l1 = 0, l2 = 1, l = 1).

-  The dot product, which takes two type-1 vectors as input to produce a scalar (l1 = 1, l2 =
1, l = 0).

-  The cross product, which takes two type-1 vectors as input to produce another type-1 vector
(l1 = 1, l2 = 1, l = 1).

In a standard linear layer, an input vector is transformed via multiplication with the weight matrix,
where weights and elements of the input vector are simply multiplied. For l > 0 elements of the
Clebsch-Gordan tensor product of Eq. (A.13) more than one mathematical operation is needed for the
connection with one weight, as can easily be verified if h[(]m[l][1]1[)] [and][ h][(]m[l][2]2[)] [are thought of as e.g. type-1]
vectors. This makes calculation of the Clebsch-Gordan tensor product slow for higher order irreps, as
can be observed in all experiments.

**Steerable MLPs.** The CG tensor product thus allows us to map two steerable input vectors to a
new output vector and can furthermore be extended to define a tensor product between steerable
vectors of mixed types. The CG tensor product can be parametrised by weights where the product
is scaled with some weight w for each triplet of types (l1, l2, l) for which the CG coefficients are
non-zero[7]. We indicate such CG tensor products with ⊗cg[W][. While in principle the CG tensor product]
takes two steerable vectors as input, in this work we mainly use it with one of its input vectors “fixed”.
The CG tensor product can then be regarded as a linear layer that maps between two steerable vector
spaces and we denote this Wa˜h[˜] := h[˜] ⊗cg[W] **a[˜], with ˜a the steerable vector that is considered to be**
fixed. With this viewpoint we can design MLPs in the same way as we are used to with regular linear
layers, and establish clear analogies with convolution layers (Sec. 3 of the main paper).

While in general the CG tensor product between two steerable vectors of type l1 and l2 can contain
steerable vectors up to degree l = l1 + l2, one typically “band-limits” the output vector by only
considering the steerable vectors up to some degree lmax. In our experiments we band-limit the
hidden feature representations to lmax = lf .

Finally, the amount of interaction between the steerable sub-vectors in the hidden representations
vectorof degree ˜a on which the steerable linear layer is conditioned. Namely, the CG tensor product onlyh[˜] ∈ _VL=lf = V0 ⊕_ _V1 ⊕_ _. . . Vlf is determined by the maximum order la of the steerable_
produces typeto embed positions as steerable vectors up to order l steerable sub-vectors for |lf − _la| ≥ lal = 5 ≤| when the hidden representations are oflf + la|. For example, it is not sensible_
degree lf = 2; the lowest steerable vector type that can be produced with the CG product of a l = 5
sub-vector of ˜a with any sub-vector of the hidden representation vector will be l = 3 and since the
hidden representations are band-limited to a maximum type of lf = 2 higher order vectors will be
ignored.

B STEERABLE GROUP CONVOLUTIONS

**Steerable functions.** Through the equivalence of steerable vectors and spherical functions via the
spherical Fourier transform, it is clear that our definition of steerable vectors coincides with the more
common definition of steerable functions, as commonly used in computer vision (Freeman et al.,
1991). A function f is called steerable (Hel-Or & Teo, 1996) under a transformation group G if any
transformation g ∈ _G on f can be written as a linear combination of a fixed, finite set of n basis_
functions _φi_ :
_{_ _}_


_αi(g)φi = α[T]_ (g)Φ, (B.1)
_i=1_

X


_gf =_
_L_


in which Lg is the left regular representation of the group G that performs the transformation on
_f (see section A.2). In case of functions in a spherical harmonic basis, which we denote with_

7In terms of the e3nn library (Geiger et al., 2021a) one then says a path exists between these 3 types.


-----

_fa :_ _l_ 0 _lm=_ _l_ _[a]m[(][l][)][Y][ l]m[(][n][)][, it directly follows from Eq. (A.7) that they are steerable via]_

_≥_ _−_
P _gfa(n) = fD(g)a(n) ._ (B.2)

_L_

[P]

In terms of the steerable function definition in Eq. (B.1), this means that α(g) = D(g)a and
**Φ = (Y0[(0)], Y** [(1)]1 _[, Y][ (1)]0_ _, . . . )[T]_, i.e. the set of basis functions flattened into a vector.
_−_

**A translation-rotation template machting motivation for (steerable) group convolutions.** The
notion of steerability becomes particularly clear when viewed in the computer vision context (Freeman
et al., 1991), where one may be interested in detecting visual features under arbitrary rotations. Let
us consider the case of 3D cross-correlations of a kernel k : R[3] _→_ R with an input feature map
_f : R[3]_ _→_ R:

_f_ _[′](x) = (k ⋆f_ )(x) = (B.3)

R[3][ k][(][x][′][ −] **[x][)][f]** [(][x][′][)d][x][′][ .]

Z

We think of such a kernel k as describing a particular visual pattern. In many applications, we want
to detect such patterns under arbitrary rotations. For example, in 3D medical image data there is no
preferred orientation and features (such as blood vessels, lesions, ...) can appear under any angle,
and the same holds for particular atomic patterns in molecules. So ideally, one wants to apply the
convolution kernel under all such transformations and obtain feature maps via

_f_ _[′](x, R) =_ (B.4)

R[3][ k][(][R][−][1][(][x][′][ −] **[x][))][f]** [(][x][′][)d][x][′][ .]

Z

By repeating the convolutions with rotated kernels we are able to detect the presence of a certain
feature at all possible angles. In a group convolution context (Cohen & Welling, 2016), the above is
what one usually calls a lifting group convolution (Bekkers, 2019) (feature maps are lifted from R[3]
to the group SE(3)).

**Group convolutions.** In regular group convolutional neural networks one continues to work with
such higher dimensional feature maps in which the kernels are also functions on the group. The
lifting and subsequent group convolutions then all have the same form and are defined via the group
action on R[3] and group product respectively via


(B.5)
R[3][ k][(][g][−][1][ ·][ x][′][)][f] [(][x][′][)d][x][′][,]

_SE(3)_ _k(g[−][1]_ _· g[′])f_ (g[′])dg[′] _,_ (B.6)


_f_ _[′](g) =_

_f_ _[′](g) =_


where g ∈ SE(3), dg the Haar measure on the group and where · in (B.5) and (B.6) respectively
denote the group action on R[3] and group product of SE(3) (cf. Section A.1). Note that Eq. (B.5) is
exactly the same as Eq. (B.4) but in different notation.

The lifting group convolution thus generates a function on the joint space of positions R[3] and rotations
SO(3). In numerical implementations, this space needs to be discretised, i.e., for a particular finite
grid of rotations we want to store the results of the convolutions for each rotation R. This approach
then requires that the convolution kernel is continuous and can be sampled under all transformations.
Hence, such kernels can be expanded in a continuous basis such as spherical harmonics (Weiler
et al., 2018), B-splines (Bekkers, 2019) or they can be parametrised via MLPs (Finzi et al., 2020).
Alternatively, the kernels are only transformed via a sub-group of transformations in E(3) that leaves
the grid on which the kernel is defined intact, as in (Worrall & Brostow, 2018; Winkels & Cohen,
2018). An advantage of regular group convolution methods is that normal point-wise activation
functions can directly be applied to the feature maps; a down-side is that these methods are only
equivariant to the sub-group on which they are discretised. When expressing the convolution kernel
in terms of spherical harmonics, however, there is no need for such a discretisation at all and one can
obtain the response of the convolution at any rotation R after convolving with the basis functions.
This works as follows.

**Steerable template matching.** Suppose a 3D convolution kernel that is expanded in a spherical
harmonic basis up to degree L as follows


_c[(]m[l][)][(][∥][x][∥][)][Y][ (]m[l][)]_
_mX=−l_


_∥xx∥_ _,_ (B.7)



_k(x) = kc˜(_ **x** )[(][x][) :=]
_∥_ _∥_


-----

with ˜c = (c[(0)]0 _[, c][(1)]1[, c][(1)]0_ _. . . )[T]_ the vector of basis coefficients that can depend on **x** . The coef_−_ _∥_ _∥_
ficients can e.g. be parametrised with an MLP that takes as input ∥x∥ and returns the coefficient
vector. We note that such coefficients are then O(3) invariant, i.e., **R** O(3) : ˜c( **Rx** ) = ˜c( **x** ).
_∀_ _∈_ _∥_ _∥_ _∥_ _∥_
Furthermore, we labelled the vector with a “ ˜ ” to indicate it is a steerable vector as it represents the
coefficients relative to a spherical harmonic basis. It then follows (from Eq. (B.2)) that the kernel is
steerable via
_k(R[−][1]x) = kD(R)˜c(_ **x** )[(][x][)][,]
_∥_ _∥_
i.e., via a transformation of the coefficient vector ˜c by it O(3) representation D(R).

This steerability property, together with linearity of the convolutions and basis expansion, implies that
with such steerable convolution kernels we can obtain their convolutional response at any rotation
directly from convolutions with the basis functions. Instead of first expanding the kernel in the basis
by taking a weighted sum of basis functions with their corresponding coefficients, and only then doing
the convolutions, we can change this order and first do the convolution with the basis functions and
sum afterwards. In doing so we create a vector of responses [˜]f (x) = (f0[(0)][(][x][)][, f][ (1)]1 [(][x][)][, f]0[ (1)][(][x][)][, . . .][ )][T]
_−_
of which the elements are given by

_fm[(][l][)][(][x][) = ((][c]m[l]_ _[Y][ (]m[l][)][)][ ⋆f][ in][)(][x][)]_ (B.8)

= _m_ [(][∥][x][∥][)][Y][ (]m[l][)][(][x][′][ −] **[x][)][f]** [(][x][′][)d][x][′][ .]

R[3][ c][(][l][)]

Z

Then the result with the convolution kernel (Eq. (B.7)) is obtained simply by a sum over the vector
components which we denote with SumReduce as follows
_l,m_


_f_ _[′](x) = SumReducel,m_ ([˜]f _[′](x)) :=_ _fm[′][(][l][)][(][x][)][ .]_

Xl _mX=−l_

If one were to be interested in the rotated filter response at x one can first rotate the steerable vector
˜f _[′](x) via the matrix representation D(R) and only then do the reduction. I.e., once the convolutions_
of (B.8) are done the lifting group convolution result is directly obtained via

_f_ _[′](x, R) = SumReducel,m_ (D(R)[˜]f _[′](x)) ._ (B.9)

A convolution with a kernel expressed in a spherical harmonics basis thus generates a signal on O(3)
at each point x.

When only considering the m = 0 component for each order l in the kernel parametrisation, the
kernels have an axial symmetry in them due to which the convolution results in a point-wise spherical
signal. This is in fact the situation that we consider in our weighted CG products (Eq. 5) where
we do not index the weights with m. This choice leads to computational efficiency. It is however
not necessary to constrain the kernels as such and continue with steerable group convolutions on
the full space R[3] ⋊ SO(3), this would require working with CG products with weights indexed by
both m and m[′] (Weiler et al., 2018). Since in this work we choose to with steerable vector spaces
_Vcalled generalised steerable convolutions, of which the SO(3) convolutions are a special case with = m0V0 ⊕_ _m1V1 ⊕_ _. . . with arbitrary multiplicities ml for each type, we implement what is_
_ml = (2l + 1) (Kondor et al., 2018)._

**Steerable group convolutions.** When working with steerable convolution kernels one does not
have to work with a grid on O(3). There are reasons to avoid working with a grid on O(3) as one
cannot numerically obtain exact equivariance if the chosen grid is not a sub-group of O(3). When
limiting to a discrete subgroups one can guarantee exact equivariance to the sub-group, but ideally
one obtains equivariance to the entire group O(3). Steerable methods provide a way to build neural
networks entirely independent of any sampling on O(3), since, as we have seen, steerable convolutions
directly result in functions on the entire group O(3) via steerable vectors. That is, at each location x
_we have a steerable vector_ [˜]f (x) which represents a function on O(3) via the inverse Fourier transform
_given in Eq. (A.11). In fact, the sum reduction in Eq. (B.9) corresponds to a discrete inverse Fourier_
transform.

Finally, let us revisit steerable group convolutions one more time but now in the steerable vector
notation used throughout this paper. The steerable convolution, as described in (B.8), is an operator


-----

that maps between steerable vector fields of different types. In the above example the input feature
map was one that only provided a single scalar value per location x, i.e. a type-0 steerable vector,
and the output was a steerable vector field containing steerable vectors up to type L. The transition
from type-0 vector field to a type-L vector field happened via tensor products with steerable vectors
of spherical harmonics, and this tensor product was parametrised by the coefficients ˜c. Suppose a
convolution of a single type-l1 steerable input field with a convolution kernel that is expanded in a
spherical harmonic basis of only type l2 via k(x) = _m=_ _l2_ _[w][(][∥][x][∥][)][Y][ (]m[l][)]_ **xx** . The kernel can
_−_ _∥_ _∥_

then be represented as a steerable vector field in itself as **k[˜](x) = w(** **x** )˜a(x), with ˜a the type-l2
_∥_ _∥_
spherical harmonic embedding given in Eq. 5 of the main paper. Such a steerable convolution maps[P][l][2]
from input feature maps [˜]f : R[3] _→_ _Vl1 using a convolution kernel_ **k[˜] : R[3]** _→_ _Vl2 to an output_
˜f _[′]_ : R[3] _→_ _Vl via_
˜f _[′](x) =_ ˜f (x) ⊗cg[w][(][∥][x][∥][)] **a˜(x[′]** _−_ **x)dx[′]** _._

R[3]

Z

Here we assumed steerable feature vector fields of a single type, but in general such convolutions
can map between vector fields of mixed type analogous to the standard convolutional case in CNNs
where mappings occur between multi-channel type-0 vector fields.

**Steerable and regular group convolutions.** In conclusion, both steerable and regular group convolutions produce feature maps on the entire group E(3) and they are equivalent when the regular
convolution kernel is expanded in a steerable basis. In regular group convolutions, the response
is stored on a particular grid which is e.g. the Cartesian product of a regular 3D grid with a particular discretisation of O(3). In regular group convolutions, we directly index the responses with
(x, R) ∈ E(3) 7→ **f** (x, R). In steerable convolutions the filter responses x → [˜]f (x) are stored at
each spatial grid point x in steerable vectors [˜]f (x) from which functions on the full group O(3) can
be obtained via an inverse Fourier transform[8]. As such one recovers the regular representation via
(x, R) 7→F _[−][1][[˜]f_ (x)](R). Regular group convolutions can however not be perfectly equivariant to
all transformations in O(3) due to discretisaton artifacts, or they are only equivariant to a discrete
sub-group of O(3). Steerable group convolutions on the other hand are exactly equivariant to all
transformations in O(3) via steerable representations of O(3).

8This Fourier transform in fact enables working with classic point-wise activation functions that could be
applied point-wise to each location in a spherical or O(3) signal as in (Cohen et al., 2018). It is important to
note that one cannot readily apply activation functions such as ReLU directly to the steerable coefficients, but
one could sample the O(3) signal on a grid via the inverse Fourier transform, apply the activation function, and
transform back into the steerable basis. This is the approach taken in (Cohen et al., 2018). In this paper we work
entirely in the steerable domain (O(3) Fourier space) and work with gated non-linearities (Weiler et al., 2018).


-----

C EXPERIMENTS

C.1 PSEUDOCODE OF SEGNN AND ABLATED ARCHITECTURES

In this subsection, we provide pseudocode for our implementation SEGNN and steerable ablations.
Pseudocode for steerable equivariant point conv layers such as (Thomas et al., 2018) with messages as
in Eq. (14) and regular gated non-linearities as activation/update function, labelled SElinear, is outlined
in Alg. 1. Pseudocode for the same network but with messages obtained in a non-linear manner
via 2-layer steerable MLPs as in Eq. (16) is outlined in Alg. 2. This layer is labelled as SEnon-linear.
Finally, pseudocode for SEGNN implementation is provided in Alg. 3. SEGNN layers allow for the
inclusion of geometric and physical quantities via node attributes ˜ai. These new node update layers
can be regarded as new steerable activation function and allow a functionality which is not possible
in layers such as SElinear or SEnon-linear. A more detailed explanation of the Clebsch-Gordan tensor
product (CGTensorProduct), the spherical harmonic embedding (SphericalHarmonicEmbedding) and
the application of gated non-linearities can be found in the e3nn (Geiger et al., 2021a) library.

**Injecting geometric and physical quantities.** In the following algorithms the relative position
vector xij between node fi and node fj is given as an example for a geometric quantity used to steer
the message layers. To get more concrete, we look at Alg. 3 (SEGNN) and discuss edge attributes
and node attributes. Steerable attributes first of all have to be embedded via spherical harmonics.
Steerable edge attribute ˜aij in most cases consist of relative positions. Instead of or additional to
relative positions also relative forces or relative velocities could be used. And finally, also node
specific quantities such as force can be added together to serve as edge attributes. Steerable node
attributes are in most cases real physical quantities. In Alg. 3 they are sketched with vi[1] [and][ v]i[2][, and]
might comprise velocity, acceleration, spin, or force. However, also adding the spherical harmonics
embedding of relative positions at node fi results in a steerable node attribute. Lastly, it is to note
that if multiple steerable attributes exist they can be either added or concatenated. However, the
latter results in significantly more weights in the CG tensor product and does not result in better
performance in any of the conducted experiments.

**Gated non-linearities.** The gate activation is a direct sum of two sets of irreps. The first set is the
set of scalar irreps passed through activation functions. The second set is the set of higher order irreps,
which is multiplied by an additional set of scalar irreps that is introduced solely for the activation
layer and passed through activation functions. For example, in Alg. 1, the scalar irreps gij are the
additional scalar irreps, introduced to “gate” the non-scalar irreps of ˜mij.

**Algorithm 1 Code sketch of an SElinear message passing layer. The SElinear layer updates steerable**
node featuresintroduced in Weiler et al. (2018). A fully documented code for this layer will be applicable in our[˜]fi[′] _[←]_ [˜]fi. The additional scalar irreps gi are used to “gate” the non-zero order irreps as
repo.

**Require:** [˜]fi, xij _▷_ Steerable nodes [˜]fi, relative position vector xij between node [˜]fi and node [˜]fj

**function O3 TENSOR PRODUCT(input1, input2)**

output ← CGTensorProduct(input1, input2) _▷_ Apply CG tensor product following Eq. (6)

output ← output + bias _▷_ Add bias to zero order irreps

return output

**end function**
**ah˜˜ijij ←** SphericalHarmonicEmbedding(fi **fj** **xij** **xij)** _▷▷Spherical harmonic embedding ofConcatenate input for messages between xij (Eq. (4))[˜]fi,_ [˜]fj

_←_ [˜] _⊕_ [˜] _⊕∥_ _∥[2]_

**m˜** _ij ⊕_ **gij ←** O3 TENSOR PRODUCT(h[˜]ij, ˜aij) ▷ Calculate messages ˜mij and scalar irreps gij

**m˜** _i ⊕_ **gi ←** _j_ **m˜** _ij ⊕_ _j_ **gij** _▷_ Aggregate messages ˜mij and scalar irreps gij
X X

˜fi[′] _[←]_ [˜]fi + Gate(m ˜ _i, Swish(gi))_ _▷_ Transform nodes via gated non-linearities


-----

**Algorithm 2 Code sketch of an SEnon-linear message passing layer. The SEnon-linear layer updates**
steerable node featuresirreps as introduced in Weiler et al. (2018). A fully documented code for this layer will be applicable[˜]fi[′] _[←]_ [˜]fi. The additional scalar irreps gi are used to “gate” the non-zero order
in our repo.

**Require:** [˜]fi, xij _▷_ Steerable nodes [˜]fi, relative position vector xij between node [˜]fi and node [˜]fj

**function O3 TENSOR PRODUCT(input1, input2)**

output ← CGTensorProduct(input1, input2) _▷_ Apply CG tensor product following Eq. (6)

output ← output + bias _▷_ Add bias to zero order irreps

return output

**end function**
**function O3 TENSOR PRODUCT SWISH GATE(input1, input2)**

output **gi** O3 TENSOR PRODUCT(input1, input2) _▷_ Output plus scalar irreps gi

outputreturn outputgated ⊕ _← ←Gate(output, Swish(gi))_ _▷_ Transform output via gated non-linearities

**end function**

**ha˜˜ijij ←** SphericalHarmonicEmbedding(fi **fj** **xij** **xij)** _▷▷Spherical harmonic embedding ofConcatenate input for messages between xij (Eq. (4))[˜]fi,_ [˜]fj
_←_ [˜] _⊕_ [˜] _⊕∥_ _∥[2]_

**m˜** _ij_ O3 TENSOR PRODUCT SWISH GATE(h[˜]ij, ˜aij) _▷_ First non-linear message layer

**m˜** _ij ←gij_ O3 TENSOR PRODUCT(m ˜ _ij, ˜aij)_ _▷_ Second linear message layer
_⊕_ _←_

**m˜** _i ⊕_ **gi ←** _j_ **m˜** _ij ⊕_ _j_ **gij** _▷_ Aggregate messages ˜mij and scalar irreps gij
X X


˜fi[′] _[←]_ [˜]fi + Gate(m ˜ _i, Swish(gi))_ _▷_ Transform nodes via gated non-linearities

**Algorithm 3 Code sketch of an SEGNN message passing layer. The SEGNN layer updates steerable**
node featuresintroduced in Weiler et al. (2018). A fully documented code for this layer will be applicable in our[˜]fi[′] _[←]_ [˜]fi. The additional scalar irreps gi are used to “gate” the non-zero order irreps as
repo.

**Require:** [˜]fi, xij, vi[1][,][ v]i[2] _[▷]_ [Steerable nodes][ ˜]fi, relative position vector xij between node [˜]fi and node
˜fj, geometric or physical quantities vi[1][,][ v]i[2] [such as velocity, acceleration, spin, or force.]

**function O3 TENSOR PRODUCT(input1, input2)**

output ← CGTensorProduct(input1, input2) _▷_ Apply CG tensor product following Eq. (6)

output ← output + bias _▷_ Add bias to zero order irreps

return output

**end function**
**function O3 TENSOR PRODUCT SWISH GATE(input1, input2)**

output **gi** O3 TENSOR PRODUCT(input1, input2) _▷_ Output plus scalar irreps gi

outputreturn outputgated ⊕ _← ←Gate(output, Swish(gi))_ _▷_ Transform output via gated non-linearities

**end function**
**a˜ij ←** SphericalHarmonicEmbedding(xij) _▷_ Spherical harmonic embedding of xij (Eq. (4))

**vv˜˜ii[2][1]** _[←][←]_ [SphericalHarmonicEmbedding(][SphericalHarmonicEmbedding(][v][v]ii[1][2][)][)] _▷▷_ Spherical harmonic embedding ofSpherical harmonic embedding of v vii[1][2] [(Eq. (4))][(Eq. (4))]

**a˜i ←** _j_ **a˜ij + ˜vi[1]** [+ ˜]vi[2] _▷_ Node attributes
X

**h˜ij** **fi** **fj** **xij** _▷_ Concatenate input for messages between [˜]fi, [˜]fj
_←_ [˜] _⊕_ [˜] _⊕∥_ _∥[2]_

**m˜** _ij_ O3 TENSOR PRODUCT SWISH GATE(h[˜]ij, ˜aij) _▷_ First non-linear message layer

**m˜** _ij ←_ O3 TENSOR PRODUCT SWISH GATE(m ˜ _ij, ˜aij)_ _▷_ Second non-linear message layer
_←_

**m˜** _i_ **m˜** _ij_ _▷_ Aggregate messages ˜mij
_←_ _j_
X

˜˜ffii[′][′] _[←][←]_ [˜]f[O3 T]i+ O3 T[ENSOR]ENSOR[ P][RODUCT] PRODUCT[ S][WISH]([˜]fi[′][,][ ˜][ G]ai)[ATE][(][˜]fi ⊕ **m˜** _i, ˜ai)_ _▷_ _▷First non-linear node update layerSecond linear node update layer_


-----

C.2 EXPERIMENTAL DETAILS

**Compared methods.** We compared against methods discussed in Sec. 3 of the main paper, namely
NMP (Gilmer et al., 2017), SchNet (Kristof et al., 2017), Cormorant (Anderson et al., 2019),
L1Net (Miller et al., 2020), LieConv (Finzi et al., 2020), TFN (Thomas et al., 2018), SE(3)transformer (Fuchs et al., 2020), and EGNN (Satorras et al., 2021). We additionally compare
to CGCNN (Xie & Grossman, 2018), PaiNN (Schutt et al., 2021), Dimenet++ (Klicpera et al., 2020)¨
and SphereNet (Liu et al., 2021).

**Implementation details for N-body dataset.** SEGNN architectures, point conv methods (SElinear),
and steearable non-linear point conv methods (SEnon-linear) consist of three parts (sequentially applied):

1. Embedding network: Inputsteerable linear layer conditioned on node attributes →{CGa˜i _[→]_ [SwishGate] ˜a[ →]i and which are applied per node.[CG]a˜i _[}][, wherein][ CG]a˜i_ [denotes a]

2. Four message passing layers as described in Sec. 2.1 for SEGNN architectures and 7
convolution layers for SElinear and SEnon-linear ablations.

3. A prediction network: CG[0]a˜i
Linear}, in which CG[0]a˜ {i [denotes a steerable linear layer conditioned on node attributes][→] [Swish][ →] [Linear][ →] [MeanPool][ →] [Linear][ →] [Swish][ →][ ˜]ai
and which maps to a vector of 64 scalar (type-0) features. The remaining layers are regular
linear layers, as in Satorras et al. (2021). CG[0]a˜i [and Linear are applied per node.]
All steerable architectures are designed such that the parameter budget at lf = 1 and la = 1
matches that of the tested EGNN implementation. We optimise models using the Adam
optimiser (Kingma & Ba, 2014) with learning rate 1e-4, weight decay 1e-8 and batch size
100 for 10000 epochs and minimise the mean absolute error. .

The SwishGate refers to a gated non-linearity (Weiler et al., 2018) when l > 0, and a swish activation
(Ramachandran et al., 2017) otherwise. A message network→→ CGInstanceNorma˜ij _[→]_ [SwishGate]}, with an instance normalisation (Ulyanov et al., 2016) implementation that is[}][, while the update network][ φ]f [consists of] φm consists of[ {][CG]a˜i _[→] {CG[SwishGate]a˜ij_ _[→]_ [SwishGate][ →] [CG]a˜i
compatible with steerable vectors, based on the BatchNorm implementation of the e3nn library
(Geiger et al., 2021a). In the default setting, however, the instance normalisation is turned off. We use
skip-connections in the message passing layers, adapting Eq. (8) to yield [˜]fi[′] [= ˜]fi +φf ([˜]fi, _j_ **m[˜]** _ij, ˜ai)._

The convolution layers which are used instead of the message passing layers for SElinear and SEnon-linear
ablations consist of: [P]

-  SElinear: CGa˜ij _j_ (i)
_{_ _[→]_ [P] _∈N_ _[→]_ [SwishGate][ →] [InstanceNorm][ }]

-  SEnon-linear: CGa˜ij **a˜ij** _j_ (i)
_{_ _[→]_ [SwishGate][ →] [CG] _[→]_ [P] _∈N_ _[→]_ [SwishGate][ →] [InstanceNorm][}]


-----

Table C.1: A full ablation study on the performance and runtime of different (maximum) orders of
steerable features (lf ) and attributes (la). The ablation study includes steerable linear convolutions,
steerable non-linear convolutions, as well as SEGNN approaches. SEnon-linear++ uses the structural
information (velocity) in the two embedding and read-out layers. Performance is measure using the
Mean Squared Error (MSE).

Method MSE Time [s]

SElinear (lf = 0, la = 0) .0344 .003
SElinear (lf = 0, la = 1) .0352 .003
SElinear (lf = 1, la = 1) .0130 .031
SElinear (lf = 1, la = 2) .0121 .035
SElinear (lf = 2, la = 2) .0116 .065
SElinear (lf = 2, la = 3) .0121 .075

SEnon-linear (lf = 0, la = 0) .0382 .004
SEnon-linear (lf = 0, la = 1) .0388 .004
SEnon-linear (lf = 1, la = 1) .0061 .030
SEnon-linear (lf = 1, la = 2) .0060 .034
SEnon-linear (lf = 2, la = 2) .0061 .065
SEnon-linear (lf = 2, la = 3) .0060 .074

SEnon-linear++ (lf = 0, la = 0) .0392 .004
SEnon-linear++ (lf = 0, la = 1) .0390 .004
SEnon-linear++ (lf = 1, la = 1) .0057 .031
SEnon-linear++ (lf = 1, la = 2) .0057 .036
SEnon-linear++ (lf = 2, la = 2) .0058 .067
SEnon-linear++ (lf = 2, la = 3) .0062 .078

SEGNNG (lf = 0, la = 0) .0099 .004
SEGNNG (lf = 0, la = 1) .0098 .005
SEGNNG (lf = 1, la = 1) .0056 .025
SEGNNG (lf = 1, la = 2) .0058 .031
SEGNNG (lf = 2, la = 2) .0060 .061
SEGNNG (lf = 2, la = 3) .0062 .068

SEGNNG+P (lf = 0, la = 0) .0102 .004
SEGNNG+P (lf = 0, la = 1) .0096 .005
SEGNNG+P (lf = 1, la = 1) .0043 .026
SEGNNG+P (lf = 1, la = 2) .0044 .032
SEGNNG+P (lf = 2, la = 2) .0041 .063
SEGNNG+P (lf = 2, la = 3) .0041 .071

Table C.2: SEGNN and EGNN performance comparison on the N-body system experiment. Performance is compared for different number of available training samples, and measured using the Mean
Squared Error (MSE). SEGNNs are significantly more data efficient.

Method Training samples MSE

SEGNNG+P (lf = 1, la = 1) 1000 .0068± .00023
SEGNNG+P (lf = 1, la = 1) 3000 .0043± .00015
SEGNNG+P (lf = 1, la = 1) 10000 .0037± .00014

EGNN 1000 .0094± .00035
EGNN 3000 .0070± .00022
EGNN 10000 .0048± .00018


-----

**Gravitational N-body dataset.** The gravitational 100-body particle system experiment is a selfdesigned toy experiment, which is conceptually similar to the charged N-body experiment described
in the main paper. However, we extend the number of particles from 5 to 100, use gravitational
interaction, no boundary conditions. Particle positions are initialised from a unit Gaussian, particle
velocities are initialised with norm equal to one and random direction and particle mass is set to one.
Trajectories are generated with by integrating gravitational acceleration in natural units using the
leapfrog integrator with δt = 0.001 for 5000 steps. Force smoothing of 0.1 is applied (Dehnen &
Read, 2011).

We predict position or force at step t = 4 given the state at t = 3. Such problems are often
encountered in physical and astrophysical simulations (Alvaro et al., 2020; Tobias et al., 2020; Mayr
et al., 2021), where usually the force (or acceleration) is used to Euler-update the positions. Thus,
reliable force prediction is of great interest. An exemplary trajectory is shown in Fig. C.1. For the
following experiments, we use 10.000 simulation trajectories in the training set, and 2.000 trajectories
for the validation and test set, respectively. The implemented architectures are 4 layer graph neural
networks with roughly equal parameter budget, and we train each network for 250 epochs. The
SEGNN layers are exactly the same as described above for the charged N-body dataset, the EGNN
and MPNN layers are the same as described in Satorras et al. (2021).

Figure C.1: Trajectory between t = 4 and t = 5 of 100 particles under gravitational interaction.
Marker shows final position at end of simulation, with opacity decreasing over time.

Table C.3 shows results for different numbers of considered interactions (neighbours) for SEGNNs,
as well as for message passing networks (MPNNs) and EGNNs. For SEGNN implementation, we
input the relative position to the center of the system and the velocity as vectors of type l = 1 with
odd parity. We further input the norm of the velocity as scalar, which altogether results in an input
vector **h[˜]** _V0_ _V1_ _V1. The output is embedded as difference vector to the initial position (odd_
parity) or the directly predicted force after 1.000 timesteps, i.e. ∈ _⊕_ _⊕_ ˜o _V1. In doing so, we keep E(3)_
_∈_
equivariance for vector valued inputs and outputs. The edge attributes are obtained via the spherical
harmonic embedding ofthe edge attributes plus the spherical harmonic embedding of the velocity. The MPNNs take the xj − **xi as described in Eq. 4. The node attributes comprise the sum of**
position and the velocity as input. Due to the vector-valued inputs and outputs, equivariance is not
preserved. For EGNNs, equivariance is only preserved for the prediction of future positions, for the
force prediction it is not.

MPNNs and SEGNN(lf = 0, la = 0) are conceptually very similar. Interestingly, MPNNs, which
use PyTorch’s Linear operation, are approximately 5 times faster than our SEGNN implementation,
which uses Einsum operations. In general, the position and force prediction for the 100-body
problem is intrinsically very difficult due to the wide range of different dynamics, where the main
error contributions arise from a few outlier trajectories. However, aside from unavoidable errors,


-----

SEGNNs generalise much better than default MPNNs or EGNNs do. The results strongly suggest the
applicability of SEGNNs to large physical (simulation) datasets.

Table C.3: Mean Squared Error (MSE) for positional (pos) and force prediction in the gravitational
N-body system experiment, and forward time in seconds for a batch size of 20 samples running on an
NVIDIA GeForce RTX 3090 GPU.

|5 neighbours|20 neighbours|50 neighbours|
|---|---|---|

|5 neighbours|20 neighbours|50 neighbours|
|---|---|---|
|Method pos force Time[s]|pos force Time[s]|pos force Time[s]|
|MPNN .297 .299 .0012 EGNN .301 unstable .0024 SEGNN(l = 0, la = 0) .292 .296 .0085 f SEGNN(l = 1, la = 1) .265 .273 .0208 f|.277 .273 .0014 .256 unstable .0025 .266 .276 .0088 .237 .244 .0212|.262 .268 .0029 .239 unstable .0047 .251 .265 .0100 .212 .223 .0416|


-----

**Implementation details for QM9.** Our QM9 hyperparameters strongly resemble those found in
(Satorras et al., 2021), whose architecture we took as a starting point for our own. The network
consists of three parts (sequentially applied):

1. Embedding network: Inputsteerable linear layer conditioned on node attributes →{CGa˜i _[→]_ [SwishGate] ˜a[ →]i and which are applied per node.[CG]a˜i _[}][, wherein][ CG]a˜i_ [denotes a]

2. Seven message passing layers as described in Sec. 2.1.

3. A prediction network: CG[0]a˜i
Linear}, in which CG[0]a˜ {i [denotes a steerable linear layer conditioned on node attributes][→] [Swish][ →] [Linear][ →] [MeanPool][ →] [Linear][ →] [Swish][ →][ ˜]ai
and which maps to a vector of 128 scalar (type-0) features. The remaining layers are regular
linear layers, as in Satorras et al. (2021). CG[0]a˜i [and Linear are applied per node.]

Message network, update network and skip-connections are used as described for the N-body task.
In contrast to the other tasks, the hidden irreps correspond to N copies for every order, selected to
correspond to a 128-dimensional linear layer in terms of number of parameters.

We optimise models using the Adam optimiser (Kingma & Ba, 2014) with learning rate 5e-4, weight
decay 1e-8 and batch size 128 for 1000 epochs and minimise the mean absolute error. The learning
rate is reduced by a factor of ten at 80% and 90% of training. Targets are normalised by subtracting
the mean and dividing by the mean absolute deviation. We found that pre-processing the molecular
graphs for a specific cutoff radius on QM9 was beneficial.

Figure C.2 shows the number of messages as a function of cutoff radius for the train partition of QM9.
It shows that a network with cutoff radius of 2A sends[˚] _∼_ 6× fewer messages on average in every
layer compared to a cutoff radius of 5 A, with far smaller variance.[˚]

Figure C.2: Mean number of messages as a function of cutoff radius for the QM9 train partition.
Standard deviation is indicated by the shaded region. The dashed black line indicates the transition
from disconnected to connected graphs and may be thought of as a minimum cutoff radius.

**Implementation details for the Open Catalyst Project OC20 dataset.** Our OCP hyperparameters resemble those from the N-body task, but with an overall larger network architecture. The
network consists of three parts (sequentially applied):

1. Embedding network: Inputsteerable linear layer conditioned on node attributes →{CGa˜i _[→]_ [SwishGate] ˜a[ →]i and which are applied per node.[CG]a˜i _[}][, wherein][ CG]a˜i_ [denotes a]

2. Seven message passing layers as described in Sec. 2.1.

3. A prediction network: CG[0]a˜i
Linear}, in which CG[0]a˜ {i [denotes a steerable linear layer conditioned on node attributes][→] [Swish][ →] [Linear][ →] [MeanPool][ →] [Linear][ →] [Swish][ →][ ˜]ai
and which maps to a vector of 256 scalar (type-0) features. The remaining layers are regular
linear layers, as in (Satorras et al., 2021). CG[0]a˜i [and Linear are applied per node.]


-----

Message network, update network and skip-connections are used as described for the QM9 tasks.
The input is a 256-dimensional one-hot representation of atom types, all hidden vectors are 256dimensional or their steerable “weight balanced” equivalent (cf Sec. 4), where we divide into equally
large type-l sub-vector spaces.

We optimise models using the AdamW optimiser (Loshchilov & Hutter, 2017) with learning rate
1e-4, weight decay 1e-8 and batch size 8 for 20 epochs and minimise the mean absolute error. The
learning rate is reduced by a factor of ten at 50% and 75% of training.

**Ablation study for the Open Catalyst Project OC20 dataset, speed comparison.** Table C.1
shows SEGNN ablation of the performance when using different (maximum) orders of steerable
features lf and attributes la, ablation for different cutoff radii of 5A and 3[˚] A, as well as runtime[˚]
comparisons. All results are obtained on the validation set. The case la = 0 and lf = 0 corresponds
to our EGNN implementation. The SElinear corresponds to our implementation as outlined in Alg. 1
and can be seen as instance of e.g. TFNs Thomas et al. (2018). All methods have been optimised
using roughly the same parameter budget as SEGNNs.

Table C.4: Comparison on the validation sets of the OC20 IS2RE tasks. Model selection is done
on those datasets which are conceptually the nearest. We compare the performance for the usage of
different (maximum) orders of steerable features (lf ) and attributes (la), as well as the performance
for cutoff radii of 5A and 3[˚] A on the OC20 dataset. The case[˚] _la = 0 and lf = 0 corresponds to our_
EGNN implementation. The SElinear corresponds to our implementation as outlined in Alg. 1 and
can be seen as instance of e.g. TFNs Thomas et al. (2018). Runtime numbers are reported for one
forward pass and are measured for a batch of size 8 on a Nvidia Tesla V100 GPU. Higher orders of
_lf and la in general do not improve results. The Dimenet++ runtime is given as reference._

|Energy MAE [eV] ↓|EwT ↑|
|---|---|


|Energy MAE [eV] ↓|EwT ↑|Time [s]|
|---|---|---|
|Model ID OOD Ads OOD Cat OOD Both|ID OOD Ads OOD Cat OOD Both|-|
|EGNN(lf = 0, la = 0, 5 A˚) 0.5497 0.6851 0.5519 0.6102 EGNN(lf = 0, la = 0, 3 A˚) 0.5931 0.7411 0.6182 0.6564 SElinear(lf = 1, la = 1, 5 A˚) 0.5841 0.7655 0.6358 0.7001 SElinear(lf = 1, la = 1, 3 A˚) 0.6020 0.7991 0.6631 0.7104 SEGNN(lf = 1, la = 1, 5 A˚) 0.5310 0.6432 0.5341 0.5777 SEGNN(lf = 1, la = 1, 3 A˚) 0.5523 0.6761 0.5609 0.6127 SEGNN(lf = 1, la = 2, 5 A˚) 0.5337 0.6419 0.5389 0.5764 SEGNN(lf = 1, la = 2, 3 A˚) 0.5497 0.6662 0.5587 0.6060 SEGNN(lf = 2, la = 2, 3 A˚) 0.5369 0.6473 0.5335 0.5822 Dimenet++ - - - -|4.99% 2.50% 4.71% 2.88% 4.87% 2.48% 4.34% 2.63% 4.32% 2.51% 4.55% 2.66% 4.27% 2.29% 4.12% 2.56% 5.32% 2.80% 4.89% 3.09% 5.03% 2.59% 4.41% 2.70% 5.41% 2.78% 4.89% 3.09% 4.98% 2.51% 4.40% 2.60% 5.22% 2.82% 4.87% 2.98% - - - -|0.025 0.015 0.028 0.022 0.080 0.040 0.112 0.051 0.234 0.062|



D LICENSES

This work would not have been possible without the existence of free software. As we have built
extensively on existing assets, we mention them and their licenses here.

Our codebase uses Python under the PSF License Agreement, using NumPy under the BSD 3-Clause
”New” or ”Revised” License. Our models were implemented in PyTorch (Paszke et al., 2019) (BSD
license) using CUDA (proprietary license). We used PyTorch extensions such as PyTorch Geometric
(Fey & Lenssen, 2019) (MIT license) and e3nn (Geiger et al., 2021b) (MIT license). For tracking
runs we used Weights & Biases (Biewald, 2020) (MIT license).

The QM9 (Ramakrishnan et al., 2014; Ruddigkeit et al., 2012) dataset was used under CC0 1.0
Universal (CC0 1.0) Public Domain Dedication and OC20 (Chanussot et al., 2020) was used under a
Creative Commons Attribution 4.0 License. The QM9 dataloaders were adapted from (Anderson
et al., 2019) under an Educational and Not-for-Profit Research License.


-----

