# TOWARDS BUILDING A GROUP-BASED UNSUPERVISED REPRESENTATION DISENTANGLEMENT FRAMEWORK

**Yang Tao[1][∗], Xuanchi Ren[2]** **, Yuwang Wang[3][†], Wenjun Zeng[4]** **, Nanning Zheng[1]**

1Xi’an Jiaotong University, 2HKUST,, 3Microsoft Research Asia, 4EIT

ABSTRACT

Disentangled representation learning is one of the major goals of deep learning, and
is a key step for achieving explainable and generalizable models. A well-defined
theoretical guarantee still lacks for the VAE-based unsupervised methods, which
are a set of popular methods to achieve unsupervised disentanglement. The Group
Theory based definition of representation disentanglement mathematically connects
the data transformations to the representations using the formalism of group. In
this paper, built on the group-based definition and inspired by the n-th dihedral
_group, we first propose a theoretical framework towards achieving unsupervised_
representation disentanglement. We then propose a model, based on existing VAEbased methods, to tackle the unsupervised learning problem of the framework. In
the theoretical framework, we prove three sufficient conditions on model, group
structure, and data respectively in an effort to achieve, in an unsupervised way,
disentangled representation per group-based definition. With the first two of the
conditions satisfied and a necessary condition derived for the third one, we offer
additional constraints, from the perspective of the group-based definition, for the
existing VAE-based models. Experimentally, we train 1800 models covering the
most prominent VAE-based methods on five datasets to verify the effectiveness
of our theoretical framework. Compared to the original VAE-based methods,
these Groupified VAEs consistently achieve better mean performance with smaller
variances.

1 INTRODUCTION

Learning independent and semantic representations of which individual dimension has interpretable
meaning, usually referred to as disentangled representations learning, is critical for artificial intelligence research (Bengio et al., 2013). Such disentangled representations are useful for many tasks:
domain adaptation (Li et al., 2019; Zou et al., 2020), zero-shot learning (Lake et al., 2017), and
adversarial attacks (Alemi et al., 2016), etc. Intuitively, a disentangled representation should reflect
the factors of variations behind the observed data of the world, and one latent unit is only sensitive to
changes of an individual factor.

Due to the facts that obtaining the ground-truth labels requires significant human effort and humans
can learn those factors unsupervisedly, unsupervised representation disentanglement draws much
attention from researchers recently. A lot of methods are proposed base on some intuitions. Most of
the state-of-the-art methods (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al.,
2018; Kumar et al., 2017) are based on Variational Autoencoder (VAE) (Kingma & Welling, 2013).
These methods are fully unsupervised and can be applied to a variety of complex datasets (Lee et al.,
2020). However, these methods suffer from the unidentifiability problem (Locatello et al., 2019b)
due to a lack of theoretical guarantee. Another stream of works (Chen et al., 2016; Lin et al., 2020;
Khrulkov et al., 2021; Lee et al., 2020) leverage generative adversarial network (GAN) (Goodfellow
et al., 2014) to achieve disentanglement but are not interpretable. In general, a well-defined theoretical
guarantee is needed for those methods.

The research of symmetry in physics demonstrates that infinitesimal transformations that conform
to some symmetry groups on physical objects can reflect their nature (Anderson, 1972; Noether,

_∗Work done during internships at Microsoft Research Asia._
_†Corresponding author_


-----

1915). Recently, inspired by this research on symmetry, Higgins et al. (2018) proposed a group-based
definition of disentangled representation. They argue that the symmetries, i.e., the transformations that
change certain aspects of data and keep other aspects unchanged, ideally reflect the underlying data
structure. The group-based definition is a formal and rigorous mathematical definition of faithful and,
ideally, interpretable representation of the generative factors of data, which is widely accepted (Greff
et al., 2019; Mathieu et al., 2019; Khemakhem et al., 2020). Subsequently, due to the fact that the
definition is defined by the world state (i.e., Ground Truth) and based on the assumption (CasellesDupre et al., 2019) that this definition should be useful for downstream tasks such as a Reinforcement´
Learning, Caselles-Dupre et al. (2019), Quessard et al. (2020), Painter et al. (2020) propose´
environment-based (to provide world state) methods to learn such disentangled representations in
Reinforcement Learning settings. These inspire us to ask the following question: how would the
definition benefit unsupervised representation disentanglement, and how to learn such a disentangled
representation conforming to the definition in the setting of unsupervised representation learning?

In Group Theory[1], the n-th dihedral group (Judson, 2020) is a set of all permutations of polygons
vertices, forming a permutation group under the operation of composition (Miller, 1973). The
generators in an n-th dihedral group, i.e., flip and rotation, can be regarded as the disentangled factors
and also transformations. In this paper, inspired by the n-th dihedral group, we answer the above
questions and address the challenge by proposing a theoretical framework to make the definition
practically applicable for unsupervised representation disentanglement. We then propose a model to
tackle the learning problem of the framework and verify its effectiveness. We theoretically prove in
Section 3.2 the three sufficient conditions towards achieving disentangled representation per groupbased definition, which are referred to as model, group structure, and data constraint, respectively.
With these conditions, we offer additional constraints from the perspective of the definition. The
additional constraints encourage existing VAE-based models to satisfy the symmetry requirement
that comes from the nature of factors. Finally, we provide a learning model based on the existing
VAE-based methods in an effort to fulfill the three conditions (with the model and group structure
constraint and a necessary condition for the data constraint satisfied). As an intuitive understanding,
we introduce the additional constraints to reorganize the latent space to restrict its symmetry in an
unsupervised way. These additional constraints indeed narrow down the solution space of VAEbased models. Detailed discussion in Sec. 5.4. Our model consistently achieves statistically better
performance in prominent metrics (higher means and lower variances) than corresponding existing
VAE-based models on five datasets, demonstrating that the group-based definition together with our
proposed framework further encourages disentanglement.

Our main contributions are summarized as follows:

-  To our best knowledge, we are the first to provide a theoretical framework to make the
formal group-based mathematical definition of disentanglement practically applicable to
_unsupervised representation disentanglement._

-  Our theoretical framework provides additional constraints from the perspective of groupbased definition for the existing VAE-based methods.

-  We propose a learning model of the framework by deriving and integrating additional loss
into existing VAE-based models, in an effort to make the learned representation conform to
the group-based definition without relying on the environment (as done in Caselles-Dupre´
et al. (2019); Quessard et al. (2020); Painter et al. (2020)).

2 RELATED WORKS

Different definitions have been proposed for disentangled representation (Bengio et al., 2013; Higgins
et al., 2018; Suter et al., 2019). However, only the group-based definition proposed by Higgins et al.
(2018) focuses on the disentangled representation itself and is mathematically rigorous, which is well
accepted (Caselles-Dupre et al., 2019; Quessard et al., 2020; Painter et al., 2020; Diane Bouchacourt,´
2021). Nevertheless, Higgins et al. (2018) do not propose a specific learning method based on their
definition. Before this rigorous definition was proposed, there had been some success in identifying
generative factors in static datasets (without interaction with environment), e.g., β-VAE (Higgins et al.,

1We assume some basic familiarity with the fundamentals of Group Theory and Group Representation
Theory. Please refer to Appendix A for some basic concepts.


-----

2017), Anneal-VAE (Burgess et al., 2018), β-TCVAE (Chen et al., 2018), and FactorVAE (Kim &
Mnih, 2018). More recent works (Srivastava et al., 2020; Shao et al., 2020; Kim et al., 2019; Lezama,
2018; Rezende & Viola, 2018) also do not consider the group-based definition. Therefore, how
group-based definition will facilitate these methods is still an open question. Besides, all these works
suffer from the unidentifiability problem (Locatello et al., 2019b), which is a challenging problem
in this literature. From group-based definition, our framework points out that, the unidentifiability
problem could be solved once the data constraint is satisfied. However, in this work, we can only get
a necessary condition for data constraint, and we still can not solve this challenging problem.

As pointed out in Quessard et al. (2020), it is not straightforward to reconcile the probabilistic inference methods with the group-based definition framework. Caselles-Dupre et al. (2019), Quessard´
et al. (2020), Painter et al. (2020) leverage the interaction with the environment (assuming it is
available) as supervision instead of minimizing the total correlation as the VAE-based methods do.
Consequently, the effectiveness of these methods is limited to the datasets with the environment
available. Our framework learns a representation conforming to the group-based definition without
_relying on the environment. Pfau et al. (2020) propose a non-parametric method to unsupervisedly_
learn linear disentangled planes in data manifold under a metric. However, as pointed out by the authors, the method does not generalize to held-out data and performs poorly when trying to disentangle
directly from pixels.

To summarize, the existing probabilistic inference methods lack theoretical support, while the
application scope of existing methods based on the group-based mathematical definition Higgins
et al. (2018) is very limited. To the best of our knowledge, our work is the first to reconcile the
_probabilistic generative methods with the inherently deterministic group-based definition framework_
_of Higgins et al. (2018)._

3 THE GROUP-BASED FRAMEWORK FOR UNSUPERVISED REPRESENTATION
DISENTANGLEMENT

Our goal is to explore the benefit of the group-based definition for unsupervised representation
disentanglement and learn such a disentangled representation. The background of the group-based
definition is provided in Section 3.1. Section 3.2 presents the theoretical framework towards achieving
_unsupervised disentanglement, in which we derive three sufficient conditions on the model, group_
structure, and data, respectively. The conditions on the model and group structure provide additional
constraints for the existing VAE-based models.

3.1 GROUP-BASED DEFINITION

We briefly review the group-based definition of disentangled representation Higgins et al. (2018).
Considering a group G acting on world state space W (can be understood as ground-truth) of data
space O and representation space Z via group action ·W and group action ·Z respectively. For a
mapping f = b ◦ _h, where b and h denote the data generative process and encoding, we state: the_
mapping f is equivariant between the actions on W and Z if
_g · f_ (w) = f (g · w), ∀g ∈ _G, ∀w ∈_ _W._ (1)

**Definition 1with respect to Assume G if: ( Gi) group action can be decomposed as of G on Z G exits. = G (1ii ×) G the mapping2 × · · · × G fm is equivariant between the. The set Z is disentangled**
_actions onaffected only by the corresponding W and Z. (iii) There is a decomposition Gi._ _Z = Z1 × Z2 × · · · × Zm such that each Zi is_

It is challenging to apply the group-based definition to an unsupervised disentanglement setting in
practice because the definition refers to the world state space W, the group action of G on W, and
mapping b which are typically inaccessible in practice. We tackle the challenge by re-framing the
definition in a new framework in the following section.

3.2 PROPOSED THEORETICAL FRAMEWORK

Since when the representation is disentangled, one latent unit in the representation space is only
sensitive to changes of an individual generative factor, we make the following assumptions: G


-----

is a direct product of m cyclic groups (as suggested by Higgins et al. (2018) and for simplicity):
_G = (Z/nZ)[m]_ = Z/nZ × Z/nZ × · · · × Z/nZ, where n is the assumed total number of possible
values for a factor and m is the total number of factors; we further assume Z is a set with the same
elements in G. Therefore, the group actions of G on Z can be set to be element-wise addition, i.e.,
_g · z = g + z, ∀z ∈_ _Z, g ∈_ _G. For the generator of dimension i of G, gi = (0, . . ., 1, . . ., 0), gi_
only affects the i-th dimension of z by gi + z. In addition, the action of each generator gi on w only
affects a single dimension of w.

As we can seen from Equation 1 above, the group action is defined on w, which is often not accessible,
making it difficult to apply the definition in practice. Therefore, for the unsupervised setting, we
would like to use permutations on the data space O (which only provides data without labels) to
substitute the group actions on W . Specifically, inspired by the n-th dihedral group (Dummit &
Foote, 1991), we construct a permutation group Φ, serving the role of an “agent” of G. The actions
of G on W can be performed by φg Φ on O, which can be formulated as
_∈_

_f_ (g _w) = h(φg_ _b(w)) = h(φg_ _o),_ _w_ _W, g_ _G,_ (2)
_·_ _·_ _·_ _∀_ _∈_ _∈_

where o denotes the data (e.g., image) corresponding to the world state w through the mapping
function b. If the above equation holds, we state that the “agent” permutation group Φ exists. We first
give the conditions for the existence of this “agent” permutation group Φ, then derive the additional
condition to achieve such disentanglement. We accomplish these two objectives in Theorem 1 with
the proof provided in Appendix B. Theorem 1 states that a general permutation group Φ on O can
serve as an agent group (agent group exists) if and only if both (i) and (ii) are satisfied. If the agent
group exists, and its permutations (actions on O) can be defined by an autoencoder-like model as
shown in the equation in (iii), then Z is disentangled with respect to G.

**Theorem 1 For the group G = (Z/nZ)[m], a permutation group Φ on O, a representation space Z,**
_a World State space W_ _, and mapping b and h, Equation 2 holds if and only if (i) Φ is isomorphic to_
_G, and (ii) For each generator of dimension i of G, gi, there exists a φi_ Φ, i = 1, . . ., m, such
(thatiii φ) φi ·g b( bw(w) =) = b( hg[−]i ·[1] w(g), f ∀(ww ∈)) _Ww, andW, φ φi is a generator ofg_ Φ, then Z is disentangled with respect to Φ; Further, if Equation 2 holds and ∈ _G, where_
_φg is the corresponding element in ·_ _·_ _∀ Φ ∈ of g under the isomorphism. ∈_

In Theorem 1, (i) states that the relation between the elements (i.e., group structure) is preserved
between Φ and G, and we denote it as the group structure constraint; (ii) actually indicates a data
constraint that all variations in the data can be generated by compositions of some basic permutation
_generators_ _φi_ _i=1,...,m. We denote it as the data constraint; (iii) states that the permutations in_
_{_ _}_
the agent group Φ are defined by encoding, action, and decoding, which is referred to as the model
_constraint. Note that in Theorem 1, only the data constraint refers to the world state w._

Here is a sketch of the proof: data constraint is a special case of Equation 2 for a generator, and
_group structure constraint is a relation-preserving constraint on compositions of generators, and_
satisfying both constraints will thus result in that Equation 2 holds for any general element in Φ, and
vice versa. Moreover, we can derive Equation 1 for disentanglement when combining the model
_constraint and Equation 2._

The model constraint specifies the way to permute the data. When the data is permuted, its world
state changes. Therefore, how the world states transit between each other is modeled by the model
_constraint applied on the data. The isomorphism between Φ and G ensures that the world state space_
_W and data space O have the same symmetry. In this way, the model applied on the data learns the_
transition of the world states. Note that we aim to bring this group-based definition, which requires
ground truth by default, into the unsupervised setting. Now only the data constraint refers to the
world states, and it seems almost impossible to derive a sufficient condition for it without the labels.
We thus make a trade-off in which we use a necessary condition in the next section.

4 GROUPIFIED VAE: A LEARNING METHOD OF THE FRAMEWORK

Let’s look closer into the three constraints, respectively. Firstly, we consider the model constraint,
_φg_ _o = h[−][1](g_ _h(o))_ _o_ _O, φg_ Φ, which suggests that the action of Φ on O can be
implemented using an autoencoder-like network that performs encoding, action on its representation · _·_ _∀_ _∈_ _∈_
space, and decoding. Given an autoencoder-like network with an encoder h and a decoder d, since d


-----

Decoder

Encoder

Decoder

(a) (b)

Figure 1: Overview of the implementation (Groupified VAE). (a) Illustration of permutation group
ΦΦ = are permutations on {φg|g ∈ _G} defined on a VAE-based model, where O. Specifically, when optimized, G φ = ( andZ/n φZ) are horizontal and vertical[m]. The generators φi, φj ∈_

movements. φi is defined as the solid orange arrows illustrate: encode an image o to representation z,
perform η on z to get z, add gi to z, and decode back to the image. This process can be regarded as an
exchange of images in dataset (permutation), as the dashed orange arrow shows. These permutations
form a group Φ. (b) The Isomorphism Loss, which guarantees that Φ is isomorphic to G, includes
Abel Loss La constraining the commutativity, and Order Loss Lo constraining the cyclicity.

is approximately the inversion of h, the model constraint can be formulated as

_φg_ _o = h[−][1](g_ _h(o)) =∆_ _d(g_ _h(o)),_ _o_ _O, g_ _G,_ (3)
_·_ _·_ _·_ _∀_ _∈_ _∈_

together with further implementation of Φ as described in Section 4.1, the model constraint can be
fulfilled. Secondly, The data constraint requires that all variations in the data can be generated by
compositions of some basic permutations generators. Previous VAE-based works (Higgins et al.,
2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) aim to generate
the data with independent generative factors, which is in line with the data constraint. Intuitively, if
the VAE-based model can generate the data from statistical independent basic latent units and each
unit corresponds to the basic permutation generator, the data constraint may be satisfied. Based
on the intuition above, we prove that if the world state is independently sampled per dimension, the
minimization of total correlation is a necessary condition for the data constraint (see Appendix E).
Therefore, we can leverage existing VAE-based models to fulfill the data constraint to some extent for
the unsupervised setting. Lastly, to satisfy the group structure constraint, we derive a self-supervised
_Isomorphism Loss which can be incorporated into the VAE-based model as described in Section 4.2._

4.1 IMPLEMENTATION OF GROUP Φ

The key is to implement the group actions of G on Z into the VAE-based models, we need to map
the representation z to a group that is isomorphic to G (cyclic representation space). Therefore, we
construct a function η to achieve this mapping. Moreover, this mapping is required to be differentiable,
in order for back-propagation to be adopted for optimization. According to Group Theory, there
is an isomorphism between G and the n-th root unity group: {exp((2πiz)/n)|z ∈ Z[m]}, where
_n, m are the same as in G. Therefore, the representation z can be mapped to z by the function η as_
_z = η(z) = exp((2πiz)/n) (see Figure 1 (a)). However, z can not be mapped to directly as it has_
complex numbers, but we can use Euler’s formula: exp((2πiz)/n) = sin((2πz)/n)+i cos((2πz)/n)
to map z to its real and imaginary part, i.e., vector sin((2πz)/n) and cos((2πz)/n). The two vectors
are concatenated and fed to the decoder.

For ease of implementation, the permutation group Φ can be approximately generated by compositions
of generators, i.e., Φ =< φ1, φ2, . . ., φm >. Recall that the generator φi of group Φ is defined as
_φin Figure 1 (a). Fori · o = d(gi · h(o)) = φi, we implement d(h(o) + gi), ∀ goi ∈ hO(o, where) by adding gi is generator 1 (without loss of generality) to the of dimension i in G, as shown i-th_
_·_
dimension of h(o), then make it cyclic by function η. Similarly, for φ[−]i [1][, we add the value of][ n][ −] [1][.]


-----

4.2 IMPLEMENTATION OF THE ISOMORPHISM

In this section, to satisfy the group structure constraint (isomorphism), we derive two equivalent
constraints, which are then converted into an Isomorphism Loss _I_ . Many groups are uniquely
_L_
determined by the properties of the generators, e.g., group G =< a, b|a[2] = b[2] = e, ab = ba >. In
addition, since the group Φ is isomorphic to G, Φ is also expected to be commutative and cyclic. In
light of this, we derive two constraints on generators that are equivalent to the isomorphism condition,
as described in Theorem 2. Please refer to Appendix C for the proof.

**Theorem 2 The defined permutation group Φ =< φ1, φ2, . . ., φm > is isomorphic to G =**
((Zii/n) ∀Zφ)i ∈[m] _if and only if:Φ, 1 ≤_ _i ≤_ _m (, we havei) for ∀_ _generators φ[n]i_ [=][ e][, where] φi, φ[ e]j ∈[ is the identity element of group]Φ, 1 ≤ _i, j ≤_ _m, we have φi[ Φ]φj[.] = φjφi, and_

The first constraint requires the group Φ to be an abelian group (Judson, 2020). Therefore, we denote
it as Abel constraint and the loss derived from it as the Abel Loss _a. The second is a constraint on_
_L_
the order of elements. We thus denote it as the Order constraint and the loss derived from it as the
Order Loss _o. See Appendix F for a more detailed implementation._
_L_

**Abel Loss.minimize** For the Abel constraint:φi (φj _o)_ _φj_ (φi _o) ∀,φio, φj ∈O to meet the Abel constraint, as shown in Figure 1Φ, 1 ≤_ _i, j ≤_ _m, we have φiφj = φjφi. We_
(b). The Abel Loss is the sum of the losses of all combinations of two ∥ _·_ _·_ _−_ _·_ _·_ _∥_ _∀_ _∈_ _generators. Denote the set of_
combinations as C = {(i, j)|1 ≤ _i, j ≤_ _m}. The Abel Loss is defined as follows_


_φi_ (φj _o)_ _φj_ (φi _o)_ _._ (4)
_∥_ _·_ _·_ _−_ _·_ _·_ _∥_
(i,jX)∈C


_La =_


_o∈O_


**Order Loss.element in group For the Order constraint: Φ (identity mapping). Note that with ∀φi ∈** Φ, 1 ≤ _i n ≤ times composition ofm, we have φ[n]i_ [=][ e] φ[, where]i, it is difficult for the[ e][ is the][ identity]
gradient to back-propagate. We thus use an approximation that uses 2 times of composition instead.
When the autoencoder can do the reconstruction well, this approximation holds, see appendix E
for details. Similar to Abel Loss, we minimize _φi_ (φi[n][−][1] _o)_ _o_ _,_ _o_ _O to satisfy the Order_
constraint. The whole process is illustrated in Figure 1 (b). However, the equation is not symmetrical ∥ _·_ _·_ _−_ _∥_ _∀_ _∈_
and leads to bias. Therefore, we use the following symmetrical form instead:


_φi_ (φ[n]i _[−][1]_ _o)_ _o_ + _φi[n][−][1]_ (φi _o)_ _o_ _._ (5)
_∥_ _·_ _·_ _−_ _∥_ _∥_ _·_ _·_ _−_ _∥_



_Lo =_


_o∈O_


1≤i≤m


With the above two loss functions optimized, the isomorphism condition is satisfied, which can be
illustrated by Theorem 3. Please refer to Appendix D for the proof.

**Theorem 3φand the Order Loss function (Equation 5) are optimized.iφj = φjφ The following two conditions are equivalent:i and ∀φi ∈** Φ, 1 ≤ _i ≤_ _m, we have φ[n]i_ [=][ e][ (] ([ii]i)[)] ∀[ the Abel Loss function (Equation 4)]φi, φj ∈ Φ, 1 ≤ _i, j ≤_ _m, we have_

Since the Abel Loss and Order Loss are equally important for satisfying the isomorphism condition,
we assign equal weight to them. Thus, the Isomorphism Loss is LI = Lo + La. With the
implementation of group Φ, the model constraint is satisfied. We optimize the Isomorphism Loss to
satisfy the group structure constraint. To further satisfy the data constraint to some extent as described
in Section 4, we leverage VAE-based models and optimize their original loss (that minimizes the total
correlation), denoted as LV AE. Therefore, the Total Loss is L = LV AE + γI _LI_, where γI is the
weight of Isomorphism Loss. We denote the above VAE-based implementation as Groupified VAE.

5 EXPERIMENTS

We first verify the effectiveness of Groupified VAE quantitatively in learning disentangled representations on several datasets and several VAE-based models. Then, we show its effectiveness qualitatively
on two typical datasets. After that, we perform a case study on the dSprites dataset to analyze the
effectiveness, and conduct ablation studies on the losses and hyperparameters. For the performance
comparison of two downstream tasks (abstract reasoning Van Steenkiste et al. (2019) and fairness
evaluation Locatello et al. (2019a)), and more comprehensive results, please see Appendix I.


-----

5.1 DATASETS AND BASELINE METHODS

To evaluate our method, we consider several datasets: dSprites (Higgins et al., 2017), Shapes3D (Kim
& Mnih, 2018), Cars3D (Reed et al., 2015), and the variants of dSprites introduced by Locatello et
al. (Locatello et al., 2019b): Color-dSprites and Noisy-dSprites. Please refer to Appendix G for the
details of the datasets.

We choose the following four baseline methods as representatives of the existing VAE-based models,
which are denoted as Original VAEs. We verify the effectiveness of our implementation based on
those methods. β-VAE (Higgins et al., 2017) introduces a hyperparameter β in front of the KL
regularizer of the VAE loss. It constrains the VAE information capacity to learn the most efficient
representation. AnnealVAE (Burgess et al., 2018) progressively increases the bottleneck capacity
so that the encoder learns new factors of variation while retaining disentanglement in previously
learned factors. FactorVAE (Burgess et al., 2018) and β-TCVAE (Chen et al., 2018) both penalize
the total correlation (Watanabe, 1960), but estimate it with adversarial training (Nguyen et al., 2010;
Sugiyama et al., 2012) and Monte-Carlo estimator respectively.

DCI BetaVAE MIG FactorVAE
dSprits

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000.35...233828 ± ± ± ± 0 0 0 0.065...101010 **0000...463941.36 ± ± ± ± 0 0 0 0...085056074.11** 0000....86897584 ± ± ± ± 0 0 0 0....026083050040 **0000....878618689 ± ± ± ± 0 0 0 0...0067051020.038** 0000....14271723 ± ± ± ± 0 0 0 0....09709206710 **0000....37343124 ± ± ± ± 0 0 0 0....089061061093** **0000....70517468 ± ± ± ± 0 0 0 0....098068098094** **0000....63687570 ± ± ± ± 0 0 0 0....058089075098**

DCI BetaVAE MIG FactorVAE
Cars3d

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000....18222124 ± ± ± ± 0 0 0 0....059046054049 **0000....24252526 ± ± ± ± 0 0 0 0....041046040046** 000.99..99199. ±0 ± ± ± 1 1 4. 06eee −. −0 −443 **0.99111 ±...000 ± ± ± 1. 0 0 05e...000 −** **4** 0000....07107409810 ± ± ± ± 0 0 0 0....021032027016 **0000....10111111 ± ± ± ± 0 0 0 0....014032033033** 0000....81908882 ± ± ± ± 0 0 0 0....066039040062 **0000....93879393 ± ± ± ± 0 0 0 0....034028034034**


DCI BetaVAE MIG FactorVAE
Shapes3d

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000..52..444766 ± ± ± ± 0 0 0 0..051..1761010 **0000....49726056 ± ± ± ± 0 0 0 0....06506107810** **0000....86978291 ± ± ± ± 0 0 0 0....076072055039** 0000....90898096 ± ± ± ± 0 0 0 0....045075042086 0000....48283340 ± ± ± ± 0 0 0 0....047181318 **0000....47504243 ± ± ± ± 0 0 0 0....0900521511** **0000....81827589 ± ± ± ± 0 0 0 0....074064056098** **0000....82839079 ± ± ± ± 0 0 0 0....066043066046**

Table 1: Performance (mean ± std) on different datasets and different models with different metrics.
We evaluate β-VAE, AnnealVAE, FactorVAE, and β-TCVAE on dSprites, Cars3d, Shapes3d, NoisydSprites, and Color-dSprites for 1800 settings. These settings include different random seeds and
hyperparameters, refer to Appendix G for the details. We only show the first three datasets here. For
more results, please refer to Appendix I.

5.2 QUANTITATIVE EVALUATIONS

This section performs quantitative evaluations on the datasets and models introduced with different
random seeds and different hyperparameters. Then, we evaluate the performance of the Original
and Groupified VAEs in terms of several popular metrics: BetaVAE score (Higgins et al., 2017),
DCI disentanglement Eastwood & Williams (2018) (DCI in short), MIG (Chen et al., 2018), and
FactorVAE score (Kim & Mnih, 2018). We assign three or four hyperparameter settings for each
model on each dataset. We run it with ten random seeds for each hyperparameter setting to minimize
the influence of random seeds. Therefore, we totally run ((3 × 10 × 3 + 10 × 3 × 3) × 2) × 5 = 1800
models. We evaluate each metric’s mean and variance for each model on each dataset to demonstrate
the effectiveness of our method. As shown in Table 1, these Groupified VAEs have better performance
(numbers marked bold in Table 1) than the Original VAEs on almost all the cases.

On Shapes3d, the Groupified VAEs outperform the Original ones on all the metrics except for
BetaVAE scores, suggesting some disagreement between BetaVAE scores and other metrics. Similar
disagreement is also observed between the variances of MIG and other metrics on Cars3d. Note that
the qualitative evaluation in Appendix J shows that the disentanglement ability of Groupified VAEs is
better on Shapes3d and Cars3d.


-----

(a) Original (b) Groupified


Figure 3: Traversal results of two factors (floor
color, scale) of Original and Groupified βTCVAE. The traversal results of Groupified
VAEs are cyclic.


Figure 2: Visual traversal comparison between
Original and Groupified β-TCVAE. The traversal
results of Groupified VAEs are less entangled.

5.3 QUALITATIVE EVALUATIONS


We qualitatively show the Groupified VAEs achieve better disentanglement than the Original ones.
As shown in Figure 2, the traversal results of Groupified β-TCVAE on Shape3d and Car3d are less
entangled. For more qualitative evaluation, please refer to Appendix J. To verify that the Groupified
VAEs learn a cyclic representation space (where n = 10), we provide the traversal results of [0,18]
with a step of 2 for both the Groupified and Original β-TCVAE on Shape3d in Figure 3. We observe
that the traversal results of Groupified VAEs are of high quality with a period of 10 (equal to n).
However, the Original VAEs generate low-quality images without cyclicity. For the comparison of
the results on CelebA (real-world datasets), please see appendix J.

5.4 VISUALIZATION OF THE LEARNED REPRESENTATION SPACE


To understand how our theoretical framework helps the existing VAE-based models to improve the
disentanglement ability, we take dSprites as an example, visualize the learned representation space,
and show the typical score distributions of the metrics. First, we visualize the space spanned by the
three most dominant factors (x position, y position, and scale).

As shown in Figure 5 (for more results, please refers to Appendix L), the spaces learned by the Original
VAEs collapse, while the spaces of the Groupified VAEs only bend a little bit. The main reason
is that the Isomorphism Loss, serving as a self-supervision signal, suppresses the representation
space distortion and encourages the disentanglement of the learned factors. As Figure 4 shows,
the Groupified VAEs consistently achieve better mean performance with smaller variances. The
isomorphism reduces the search space of the network so that the Groupified VAEs converge to the
ideal disentanglement solution.

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score


Figure 4: Performance distribution of Original and Groupified AnnealVAE on dSprites (demonstrated
by the Violin Plot (Hintze & Nelson, 1998)). Variance is due to different hyperparameters and random
seeds. We observe that Groupified AnnealVAE improves the average performance with smaller
variance in terms of BetaVAE score (a), DCI disentanglement (b), and MIG (c), and has a comparable
mean performance with smaller variance in terms of FactorVAE score (d).

Groupified Factor Size n = 10
Original

_n = 5_ _n = 10_ _n = 15_ w/o Abel w/o Order Groupified


DCI 0.27 ± 0.10 0.34 ± 0.062 **0.38 ± 0.055** **0.38 ± 0.064** 0.28 ± 0.11 0.34 ± 0.056 **0.38 ± 0.055**

Table 2: Ablation study on the factor size n and Isomorphism Loss. DCI disentanglement is listed
(mean ± std).


-----

(a) C = 10, Groupified (b) C = 20, Groupified (c) C = 25, Groupified (d) C = 30, Groupified

(e) C = 10, Original (f) C = 20, Original (g) C = 25, Original (h) C = 30, Original

Figure 5: The representation space spanned by the learned factors by Original (bottom row) and
_Groupified AnnealVAE (top row). The position of each point is the disentangled representation of the_
corresponding image. An ideal result is all the points form a cube and color variation is continuous.
The increase of C (a hyperparameter of AnnealVAE) results in a collapse of representation space
of the Original VAE. The collapse is suppressed by the Isomorphism Loss, which leads to better
disentanglement.

5.5 ABLATION STUDY

We perform an ablation study on the assumed total number of possible values for a factor (factor size)
_n, Abel Loss_ _a, and Order Loss_ _o. We take the AnnealVAE trained on dSprites as an example._
_L_ _L_
We only consider the DCI disentanglement metric here. We investigate the influence of factor size n.
Besides, to evaluate the effectiveness of the two constraints, the models with the Abel Loss alone or
Order Loss alone added are also evaluated. In this setting, we fix n to 10. We compute the mean and
variance of the performance for 30 settings of hyperparameters and random seeds. Table 2 shows
that the isomorphism plays a role of cycle consistency in the representation space, leading to better
disentanglement. The performance is robust to the factor size n, as the models learn to adapt to
different n in the training process. The models with only the Abel Loss or Order Loss applied have
improved performance compared to the originals. The former (Abel Loss) performs better than the
latter, suggesting that commutativity plays a more important role. Note that the number of factors m
can be learned and is not a hyperparameter. See Appendix F for details. γI is empirically set to 1.

6 CONCLUSION

In this paper, we have opened the possibility of applying group-based definition to unsupervised
disentanglement by proposing a theoretical framework. The group structure and model constraint
in the framework are effective for existing VAE-based unsupervised disentanglement methods. In
addition, by establishing the feasibility of learning the representation conforming to the definition
in unsupervised settings, we have exhibited the consistently better mean performance with lower
variance attributed to the definition. We believe our work constitutes a promising step towards
_unsupervised disentanglement with theoretical guarantee. As to the limitation, we only provide_
a necessary condition for the data constraint, as a result, we can not address the unidentifiability
problem. Tackling the unidentifiability problem with the group-based definition is beyond the scope
of this work, we will leave it as future work. In addition, a natural extension of our framework is to
use lie group Hall (2015) (which is also a manifold) to extend our framework.


-----

REFERENCES

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information
bottleneck. arXiv preprint arXiv:1612.00410, 2016.

Philip W Anderson. More is different. Science, 177(4047):393–396, 1972.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new
perspectives. TPAMI, 35(8):1798–1828, 2013.

Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta -vae. arXiv preprint
_arXiv:1804.03599, 2018._

Hugo Caselles-Dupre, Michael Garcia Ortiz, and David Filliat. Symmetry-based disentangled´
representation learning requires interaction with environments. In NeurIPS, pp. 4606–4615, 2019.

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of
disentanglement in variational autoencoders. In NeurIPS, pp. 2610–2620, 2018.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan:
interpretable representation learning by information maximizing generative adversarial nets. In
_NeurPIS, 2016._

Stephane Deny Diane Bouchacourt, Mark Ibrahim. Addressing the topological defects of disentan[glement, 2021. URL https://openreview.net/forum?id=cbdp6RLk2r7.](https://openreview.net/forum?id=cbdp6RLk2r7)

David S Dummit and Richard M Foote. Abstract algebra, volume 1999. Prentice Hall Englewood
Cliffs, NJ, 1991.

Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of
disentangled representations. In ICLR, 2018.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406.2661,
[2014. URL http://arxiv.org/abs/1406.2661.](http://arxiv.org/abs/1406.2661)

Klaus Greff, Raphael Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran,¨
Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning
with iterative variational inference. arXiv preprint arXiv:1903.00450, 2019.

Brian Hall. Lie groups, Lie algebras, and representations: an elementary introduction, volume 222.
Springer, 2015.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a
constrained variational framework. ICLR, 2017.

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende,
and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint
_arXiv:1812.02230, 2018._

Jerry L Hintze and Ray D Nelson. Violin plots: a box plot-density trace synergism. The American
_Statistician, 52(2):181–184, 1998._

Thomas W Judson. Abstract algebra: theory and applications. Virginia Commonwealth University
Mathematics, 2020.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders
and nonlinear ica: A unifying framework. In AISTATS, pp. 2207–2217, 2020.

Valentin Khrulkov, Leyla Mirvakhabova, Ivan Oseledets, and Artem Babenko. On disentangled
[representations extracted from pretrained gans, 2021. URL https://openreview.net/](https://openreview.net/forum?id=VCAXR34cp59)
[forum?id=VCAXR34cp59.](https://openreview.net/forum?id=VCAXR34cp59)


-----

Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML, 2018.

Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic. Bayes-factor-vae: Hierarchical
bayesian deep auto-encoder models for factor disentanglement. In Proceedings of the IEEE/CVF
_International Conference on Computer Vision, pp. 2979–2987, 2019._

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. _arXiv preprint_
_arXiv:1312.6114, 2013._

Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled
latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848, 2017.

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building
machines that learn and think like people. Behavioral and brain sciences, 40, 2017.

Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with
disentangled representation. In ECCV, 2020.

Jose Lezama. Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision.´
In International Conference on Learning Representations, 2018.

Yu-Jhe Li, Ci-Siang Lin, Yan-Bo Lin, and Yu-Chiang Frank Wang. Cross-dataset person reidentification via unsupervised pose disentanglement and adaptation. In ICCV, 2019.

Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality:
Self-supervised model training and selection for disentangling gans. In ICML, 2020.

Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Scholkopf, and¨
Olivier Bachem. On the fairness of disentangled representations. In NeurIPS, pp. 14611–14624,
2019a.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Scholkopf,¨
and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, pp. 4114–4124, 2019b.

Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in
variational autoencoders. In ICML, pp. 4402–4412, 2019.

Willard Miller. Symmetry groups and their applications. Academic Press, 1973.

Alexander McFarlane Mood. Introduction to the Theory of Statistics. McGraw-hill, 1950.

XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals
and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory,
56(11):5847–5861, 2010.

Emmy Noether. Der endlichkeitssatz der invarianten endlicher gruppen. Mathematische Annalen, 77
(1):89–92, 1915.

Matthew Painter, Adam Prugel-Bennett, and Jonathon Hare. Linear disentangled representations and
unsupervised action estimation. NeurIPS, 33, 2020.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,
Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in
pytorch. NeurIPS 2017 workshop, 2017.

David Pfau, Irina Higgins, Alex Botev, and Sebastien Racani´ ere. Disentangling by subspace diffusion.`
_NeurIPS, 33, 2020._

Robin Quessard, Thomas D Barrett, and William R Clements. Learning group structure and disentangled representations of dynamical environments. arXiv preprint arXiv:2002.06991, 2020.

Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In NeurIPS,
2015.


-----

Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.00597, 2018.

Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang,
and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In International Confer_ence on Machine Learning, pp. 8655–8664. PMLR, 2020._

Akash Srivastava, Yamini Bansal, Yukun Ding, Cole Hurwitz, Kai Xu, Bernhard Egger, Prasanna
Sattigeri, Josh Tenenbaum, David D Cox, and Dan Gutfreund. Improving the reconstruction of
disentangled representation learners via multi-stage modelling. arXiv preprint arXiv:2010.13187,
2020.

Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman
divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical
_Mathematics, 64(5):1009–1044, 2012._

Raphael Suter, Djordje Miladinovic, Bernhard Scholkopf, and Stefan Bauer. Robustly disentangled¨
causal mechanisms: Validating deep representations for interventional robustness. In ICML, pp.
6056–6065. PMLR, 2019.

Sjoerd Van Steenkiste, Francesco Locatello, Jurgen Schmidhuber, and Olivier Bachem. Are dis-¨
entangled representations helpful for abstract visual reasoning? In NeurIPS, pp. 14245–14258,
2019.

Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research
_and development, 4(1):66–82, 1960._

Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast
decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint
_arXiv:1901.07017, 2019._

Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning.
_arXiv preprint arXiv:2106.03375, 2021._

Yang Zou, Xiaodong Yang, Zhiding Yu, BVK Kumar, and Jan Kautz. Joint disentangling and
adaptation for cross-domain person re-identification. In ECCV, 2020.


-----

A PRELIMINARIES

**_Group: A set G together with a binary operation ◦_** as: ◦ : G × G → _G satisfying the following_
properties:

-  Associativity: ∀ _a, b, c ∈_ _G, s.t.(a ◦_ _b) ◦_ _c = a ◦_ (b ◦ _c)._

-  Identity: ∃ _e ∈_ _G, s.t. ∀_ _a ∈_ _G, e ◦_ _a = a ◦_ _e = a._

-  Inverse: ∀ _a ∈_ _G, ∃_ _a[−][1]_ _∈_ _G : a ◦_ _a[−][1]_ = a[−][1] _◦_ _a = e._


It is customary to represent a group with a set G and the binary operation ◦ as a pair (G, ◦). When
the binary operation is clear, we represent group (G, ◦) as G and use multiplication to represent the
binary operation ◦, i.e., a ◦ _b = ab, ∀_ _a, b ∈_ _G._

**_Group Action: Let (G, ◦) be a group and P be a set. By the group actions of (G, ◦) on P_**, we mean a
mapping:

_·P : G × P →_ _P,_ (6)

such that

-  ∀a, b ∈ _G, p ∈_ _P, (a ◦_ _b) · p = a · (b · p)._

-  e · p = p, where e is the identity element of G.


**_Symmetry Group and Permutation Group: Let Σ be a nonempty set, the bijections from Σ to itself_**
are called Permutations. S(Σ) denotes the set containing all the permutations on Σ. S(Σ) forms a
group under the binary operation: composition of functions, which is called Symmetry Group. A
subgroup of S(Σ) is called Permutation Group.

**_Abelian Group: If the commutative law (∀_** _a, b ∈_ _G, a ◦_ _b = b ◦_ _a) holds in a group (G, ◦), such a_
group is called an abelian group.

**_Subgroup: If a subset H of a group G is itself a group under the operation of G, we say that H is a_**

|Col1|Col2|2|
|---|---|---|
|1|||
||||
||||


|Col1|Col2|2|
|---|---|---|
|4|||
||||
||||


|Col1|Col2|2|
|---|---|---|
|3|||
||||
||||


|Col1|Col2|2|
|---|---|---|
|2|||
||||
||||


|Col1|Col2|2|
|---|---|---|
|1|||
||||
||||


|Col1|Col2|2|
|---|---|---|
|3|||
||||
||||


subgroup of G.

1 2 1 2 1 2 1 2

2 1 4 3
1 4 3 2

4 3 4 3 4 3 4 3

4 3 3 2 2 1 1 4

𝑖𝑑= 𝑓[଴]𝑟[଴] (1432) = 𝑓[଴]𝑟[ଵ] (13)(24) = 𝑓[଴]𝑟[ଶ] (1234) = 𝑓[଴]𝑟[ଷ]

1 2 1 2 1 2 1 2

3 4 1 2
4 1 2 3

4 3 4 3 4 3 4 3

1 2 2 3 3 4 4 1

(14)(23) = 𝑓[ଵ]𝑟[଴] (24) = 𝑓[ଵ]𝑟[ଵ] (12)(34) = 𝑓[ଵ]𝑟[ଶ] (13) = 𝑓[ଵ]𝑟[ଷ]

Figure 6: 4-th Dihedral Group (D4): The groups of symmetries for square. Note that permutation
means 1 → 2, 2 → 3, 3 → 1.


**_Equivalence Relation: An equivalence relation on a set B is a subset U ⊂_** _B × B satisfying: (It is_
customary to represent (a, b) ∈ _U as a ∼_ _b)_

-  Reflexive: ∀ _a ∈_ _B, (a, a) ∈_ _U_ .

-  Symmetric: (a, b) ∈ _U ⇐⇒_ (b, a) ∈ _U_ .

-  Transitive: (a, b) ∈ _U and (b, c) ∈_ _U ⇒_ (a, c) ∈ _U_ .


-----

**_Equivalence Classequivalence class containing: Let ∼_** be an equivalence relation on a set a is the subset a =∆ _{b ∈_ _C|b ∼_ _a} ⊂ B. We take anB._ _a ∈_ _B, and then the_

**_Homomorphism Let (G, ·) and (H, ◦) be two groups. A homomorphism f_**, from G to H, is a
mapping f : G → _H, such that f_ (x · y) = f (x) ◦ _f_ (y), ∀x, y ∈ _G._

**_Isomorphism: A homomorphism f : G →_** _H which is bijective is called an isomorphism. Two_
groups are said to be isomorphic if there exists an isomorphism between them.

**Related Groups:**
(i) Additive group of integers modulo n: Let n denote a positive integer. We define an equivalence
relation on Z as a ∼ _b ⇔_ _a = b (mod n) (a and b have the same remainder modulo n). The relation_
divides Z into n equivalent classes: 0, 1, . . ., n − 1, where i represents the equivalent class containing
_i, i.e., i = {m ∈_ Z|m = i (mod n)}. Let Zn = {0, 1, . . ., n − 1}, and Zn forms a group under the
binary operation: a + b = a + b, denoted by Z/nZ.
2πia
(ii) Group of n-th root unity: Let n denote a positive integer and Cn = {e _n |0 ≤_ _a ≤_ _n −_ 1, a ∈

Z be n roots of x[n] = 1, and then Cn forms a group under complex multiplication. For the mapping:
_}_ 2πia
_f : Cn_ (Zn, +) defined by f (e _n ) = a, we have_
_→_

2πia 2πib 2πi(a+b) 2πia 2πib
_f_ _e_ _n_ _· e_ _n_ = f _e_ _n_ = a + b = a + b = f _e_ _n_ _· f_ _e_ _n_ (7)
       

Therefore, f is a homomorphism, and thus is an isomorphism.

**_Congruence Class: Let n denote a positive integer, and we define an equivalence relation on Z as_**
_a ∼_ _b ⇔_ _a = b (mod n). The relation divides Z into n equivalent classes: 0, 1, . . ., n −_ 1, which
are congruence classes and the elements of the additive group of integers modulo n.

**_Subgroup Generated by B: If B is a nonempty subset of the group (G, ◦), the set defined by_**
_< B >=_ _a_ _G_ _a = a1_ _a2_ _an with either ai_ _B or a[−]i_ [1] _B_ forms a subgroup of G,
which is called the subgroup generated by { _∈_ _|_ _◦_ _◦· · · ◦_ _B, e.g., G =< a, b ∈_ _|a[2]_ = e, ab∈ =} ba > in the toy example
of Section 1. Here a1, . . . an are called Generators, e.g., a, b in G.

**_Cyclic Group A group G is called cyclic if there is an element a ∈_** _G such that G = {a[n]|n ∈_ Z} =<
_a >, e.g., Z/nZ. Such an element a is called a generator of G._

**_Symmetry: A symmetry of a geometric figure is a rearrangement of the figure preserving the_**
arrangement of its sides and vertices as well as its distances and angles.

**_Dihedral Group: The permutation group formed by the symmetries of a regular n-sided polygon,_**
denoted by Dn.

_D4 as an example: The vertices of a square are numbered by {1, 2, 3, 4}, which is analogous to_
an image dataset, the symmetries are analogous to image transformations. We often abbreviate
permutation 1 → 2, 2 → 3, 3 → 4, 4 → 1 as (1234). The elements of group D4 are shown in Figure
6, from which we know that all of the transformations are compounded by two basic permutations:
horizontal flip f and rotate 90 degrees clockwise r. Please note that f and r are analogous to
disentangled factors. What’s more, the Group can be generated by these basic permutations f, r with
some properties, i.e., D4 =< f, r|f [2] = 1, r[4] = 1, fr = r[−][1]f >. The constraints f [2] = 1, r[4] =
1, fr = r[−][1]f are analogous to the group constraints in this paper.

B PROOF FOR THEOREM 1

In our setting, the group-based definition is equivalent to the equivariant condition g · _f_ (w) = f (g · _w)_
for the following reason. Since the group actions of G on Z is the element wise addition, (a) the
group action of G on Z exists and (b) group actions of Gi only affect Zi (refer to Section 3.2). These
two conditions of the group-based definition thus hold.

_Proof. In the following, we prove in Step 1 that the necessary and sufficient conditions of Φ playing_
the “agent” role, i.e., f (g · w) = h(φ · b(w)), are (i) Φ is isomorphic to G, and (ii) there exist
_φi_ Φ, s.t. φi _b(w) = b(gi_ _w), i = 1, 2, . . ., m. We then prove in Step 2 that if equation_
_f_ ( ∈g · w) = h(φ · · b(w)) and the definition of · _φ ∈_ Φ hold, then Z is disentangled per group-based
definition.


-----

**Step 1: In the following, we prove that the necessary and sufficient conditions of f** (g·w) = h(φ·b(w))
are (i) and (ii) above. Note that this holds even without a specific definition of Φ.

_⇒b(g) For a general permutation groupi_ _w), i = 1, 2, . . ., m. Assume that there exists an isomorphism Φ on O, we assume there exist τ φ :i such that G_ Φ, where group φi · b(w) =
_G = ( ·_ Z/nZ)[m]. From Group Theory, we know that there exists µ ∈ _Aut(Φ) such that the following →_
equation holds, where Aut(Φ) denotes the group of automorphisms of Φ, i.e., µ is an isomorphism
_µ : Φ →_ Φ. We denote the composition of µ, τ as σ, and have

_σ(gi) = µ(τ_ (gi)) = φi, i = 1, 2, . . ., m. (8)

Since we have φi _b(w) = b(gi_ _w), i = 1, 2, . . ., m, the following equation holds,_
_·_ _·_

_σ(gi)_ _b(w) = φi_ _b(w) = b(gi_ _w), i = 1, 2, . . ., m._ (9)
_·_ _·_ _·_

Since both µ, τ are isomorphism, σ = µ ◦ _τ is also an isomorphism, which indicates that φi is_
a generator of Φ. For ∀φ ∈ Φ, since σ is a bijection, there exists g ∈ _G = (Z/nZ)[m], such that_
_σ(g) = φ and g =_ _i_ _[k][i][g][i][, k][i][ ∈]_ [Z][, where][ g][i][, i][ = 1][,][ 2][, . . ., m][ denotes the generators in][ G][. In order]

to calculate φ · b(w), we consider a specific example σ(g1 + g2).

_σ(g1+g2)_ _b(w) =[P] σ(g1)_ (σ(g2) _b(w)) = σ(g1)_ _b(g2_ _w) = b(g1_ (g2 _w)) = b((g1+g2)_ _w). (10)_

_·_ _·_ _·_ _·_ _·_ _·_ _·_ _·_

Therefore, for the general element φ, we have


_kigi_ _w) = b(g_ _w)._ (11)
_·_ _·_


_φ · b(w) = σ(_


_kigi)_ _b(w) = b(_
_·_


In general, we have φ · b(w) = b(g · w) and thus we have h ◦ _b(g · w) = f_ (g · w) = h(φ · b(w))
when taking h on both sides.

_⇐b(g)i Here we prove thatw), i = 1, 2, . . ., m ( are two necessary conditions of equationi) G is isomorphic to Φ and (ii) there exists f_ (g _w φ) =i such that h(φ_ _b(w φ)) =i · b h((wφ) =x)._
We take · _f_ _[−][1]_ on both sides of the equation. For convenience, we rewrite · _g · w, φ ·· b(w) as g(w) and·_
_φ(b(w)), then we have_

_g(w) = f_ _[−][1]_ _◦_ _h ◦_ _φ ◦_ _b(w) = b[−][1]_ _◦_ _φ ◦_ _b(w)._ (12)

Note that the notation ◦ here denotes the composition of functions. We define the mapping τ between
Φ and G as follows:
_τ_ (φ) = g = b[−][1] _◦_ _φ ◦_ _b._ (13)

Note that b is a bijection, thus τ is a bijection. We take φi, φj Φ and we have
_∈_

_τ_ (φi _φj)_ = b[−][1] _φi_ _φj_ _b_
_◦_ = (b[−][1]◦ _φ ◦i_ _b) ◦_ (b[−][1] _φj_ _b)_ (14)
= τ (φi)◦ _τ ◦(φj) ◦._ _◦_ _◦_
_◦_

Therefore, τ is a homomorphism and thus is an isomorphism. i.e., Φ is isomorphic to G. Recall that
group G has the form of G = (Z/nZ)[m], where it is a direct product of m cyclic groups. For ∀φ ∈ Φ,
we have
_φ · b(w) = h[−][1]_ _◦_ _f_ (g(w)) = b(g(w)). (15)

For generatorsderived from the above equation: gi ∈ _G, i = 1, 2, . . ., m and the corresponding φi = τ_ _[−][1](gi), we have a specific one_

_φi_ _b(w) = b(gi(w)) = b(gi_ _w)._ (16)
_·_ _·_


Q.E.D.

**Step 2: As discussed in Section 1 and Section 3, we define φ as φ · x = φ · b(w) = h[−][1](σ[−][1](φ) ·**
_h(x)) = h[−][1](g · f_ (w)), where σ is the same as in Step 1. Since σ is a bijection, for ∀φ ∈ Φ, σ[−][1](φ)
uniquely exists, and thus φ is well-defined. We bring it into f (g · w) = h(φ · b(w)), and derive
_f_ (g _·_ _w) = g_ _·_ _f_ (w), i.e., the representation space Z is disentangled with respect to G. Specifically, for
the existing φi, i = 1, 2, . . ., m, from equation 8, we have σ[−][1](φi) = gi and φi · _x = h[−][1](gi ·_ _f_ (w)).

Q.E.D.


-----

C PROOF FOR THEOREM 2

In order to easily distinguish a general element and a generator, we use ϕ to stand for the general
element in Φ and φi to stand for generators in Φ. In addition, for mapping τ : G → Φ, notation
_ker(τ_ ) is generally the inverse image of e, i.e., ker(τ ) = {g ∈ _G|τ_ (g) = e}, where e is the identity
element in Φ.

_Proof. Note that Section 3.4 describes the implementation of the elements in Φ, but there is no guaran-_
tee that Φ is a group. Therefore, here we first prove in Step 1 that set Φ =< φ1, φ2, . . ., φm|φiφj =
_φjφi, φ[n]i_ [=][ e, i][ = 1][,][ 2][, . . ., m][ and][ j][ = 1][,][ 2][, . . ., m >][ forms a group under the composition of]
functions, where e satisfies that eφi = φi, then prove in Step 2 the necessary and sufficient condition
for the isomorphism is to meet both the Abel constraint and the Order constraint.

**Step 1: To prove a set forms a group under some operation, we only need to verify that the elements**
of the set satisfy the following three properties: 1. Associativity 2. Identity 3. Inverse.

1. We first verify the Associativity property: ∀ _ϕs, ϕt, φl ∈_ Φ, s.t. (ϕsϕt)ϕl = ϕs(ϕtϕl).

Recall that in Section 3.2, the representation space is a set with the same elements in G. The
group action of G on Z is the addition. We have g · z ∈ _Z, ∀g ∈_ _G, z ∈_ _Z, and then we have_
_dS((gO ·) z is the symmetry group on images set) ∈_ _O, ∀g ∈_ _G, z ∈_ _Z. Therefore, φ Oi is a permutation:, then for_ _φi, φ φj, φi :k O →φO1, φ, i.e.,2, . . . φ φi ∈mS, we have(O), where_
_∀_ _∈{_ _}_

(φiφj)φk = φi(φjφk), (17)

the generators of Φ thus satisfy the Associativity property. However, whether the general elements of
Φ satisfy the Associativity property is unknown. We take ∀ _ϕs, ϕt, ϕl ∈_ Φ, where ϕt = _i_ _[φ]i[t][i]_ _[, ϕ][s][ =]_

_i_ _[φ]i[s][i]_ _[, ϕ][l][ =][ Q]i_ _[φ]i[l][i]_ [and][ t][i][, s][i][, l][i][ ∈] [Z][, and we have]

[Q]

Q

(ϕtϕs)ϕl = _φ[t]i[i]_ _φ[s]i_ _[i]_ _φ[l]i[i]_ [=] _φ[t]i[i]_ _φ[s]i_ _[i]_ _φ[l]i[i]_ = ϕt(ϕsϕl). (18)

_i_ _i_ _i_ _i_ _i_ _i_ !

Y Y ! Y Y Y Y

Therefore, the set of mappings Φ satisfies the Associativity property under the composition of
functions.

2. We then verify the Identity property: ∃ _e ∈_ Φ, s.t. ∀ _ϕ ∈_ Φ, eϕ = ϕe = ϕ.

For generators φi, φj ∈{φ1, φ2, . . . φm}, we have φ[n]i _[φ][j][ =][ eφ][j][ =][ φ][j][. Therefore, for general]_
elements, we take ∀ _ϕ ∈_ Φ, ϕ = _i_ _[φ]i[k][i]_ [, and we have:]

_φ[n]i_ _[ϕ][ =][ e]_ _φ[k]i_ _[i]_ [= (][eφ][j][)][φ]j[k][j] _[−][1]_ _φ[k]i_ _[i]_ [=][ φ]j[k][j] _φ[k]i_ _[i]_ [=] _φ[k]i_ _[i]_ [=][ ϕ.] (19)

[Q]

Yi _i{Yi≠_ _j}_ _i{Yi≠_ _j}_ Yi

This states that the identity element of Φ is e = φ[n]i
the Identity property. _[∈]_ [Φ][. Therefore, the set of mappings][ Φ][ satisfies]

3. We finally verify the Inverse property: ∀ _ϕ ∈_ Φ, ∃ _ϕ[−][1]_ _∈_ _G : ϕϕ[−][1]_ = ϕ[−][1]ϕ = e.

For the generators φi ∈{φ1, φ2, . . . φm}, we have φ[k]i _[φ]i[n][−][k]_ = φ[n]i [=][ e,][ 1][ ≤] _[k][ ≤]_ _[n, k][ ∈]_ [Z][. For any]
general element ϕ ∈ Φ, ϕ = _i_ _[φ]i[k][i]_ [, we have:]

_ϕ_ _φ[n]i_ _[−][k][i]_ = _φ[k]i_ _[i]_ [(][φ]j[k][j] _[φ]j[n][−][k][j]_ ) _φ[n]i_ _[−][k][i]_ = = e, (20)

[Q] _· · ·_

Yi _i{Yi≠_ _j}_ _i{Yi≠_ _j}_

we denote _i_ _[φ][n]i_ _[−][k][i]_ as ϕ[−][1] and have ϕϕ[−][1] = e. Similarly, we also have ϕ[−][1]ϕ = e. These

two equations state that for any general element ϕ ∈ Φ, ϕ = _i_ _[φ]i[k][i][, we have an inverse element]_
_ϕ[−][1]_ = _i_ _[φ][Q][n]i_ _[−][k][i]_ in Φ. Therefore, the set of mappings Φ satisfies the Inverse property.

[Q]

To summarize, Φ is a group.

[Q]

Q.E.D.

**Step 2: In this step, we prove that the necessary and sufficient condition for the isomorphism**
(Z/nZ)[m] _∼< φ1, φ2, . . ., φm > is the satisfaction of φiφj = φjφi, i = 1, 2, . . ., m and j =_
1, 2, . . ., m, and φ[n]i [=][ e, i][ = 1][,][ 2][, . . ., m][.]


-----

_⇒) Assume the isomorphism is τ : G = (Z/nZ)[m]_ _→_ Φ =< φ1, φ2, . . ., φm >, for generators

_φφji, φ, wherej ∈{ gφi1, i, φ = 12, . . . φ, 2, . . ., mm}, there exist denotes the generators in gα =_ _i_ _[α][i][g][i][, g] G[β][ =], which is the m-dimensional one-hot vector[ P]i_ _[β][i][g][i][ ∈]_ _[G, s.t. τ]_ [(][g][α][) =][ φ][i][, τ] [(][g][β][) =]
(0, . . ., 1, . . ., 0) _G with 1 in position i and 0 elsewhere, and[P]_ _αi, βi_ Z. Since G is an Abelian
group, we have _∈_ _∈_

_φiφj = τ_ (gα)τ (gβ) = τ (gαgβ) = τ (gβgα) = τ (gβ)τ (gα) = φjφi. (21)

For generatorscomposition of itself is φi ∈{φ1, φ2, . . . φm}, we have gα = _i_ _[α][i][g][i][ ∈]_ _[G, s.t. τ]_ [(][g][α][) =][ φ][i][, the][ n][ times]

_φ[n]i_ [=][ τ] [(][g][α][)][n][ =][ τ] [(][ng][α][) =][ τ] [(] _αingi) =[P] τ_ ( _αieG) = τ_ (eG) = e, (22)

_i_ _i_

X X

where eG, e are identity elements of G and Φ respectively. In Equation 22, τ (eG) = e holds because
_τ is an isomorphism, and ker(τ_ ) = _eG_ . The sufficiency is proven.
_{_ _}_

_⇐) In the following, we prove that when two conditions are satisfied simultaneously, the mapping τ_
we define is an isomorphism. Considering the mapping τ : Φ → _G, defined as_
_τ : φi 7→_ _gi;_ (23)
_τ : φiφj_ _gi + gj._
 _7→_

For general elements ϕt, ϕs ∈ Φ, where ϕt = _i_ _[φ]i[t][i]_ _[, ϕ][s][ =][ Q]i_ _[φ]i[s][i]_ [and][ t][i][, s][i][ ∈] [Z][, we have]

_τ_ (ϕtϕs) = τ _φ[t]i[i]_ _φ[s]i_ _[i]_ [Q]= τ _φ[t]i[i][+][s][i]_ = (ti + si)gi. (24)

_i_ _i_ ! _i_ ! _i_

Y Y Y X

Note that these terms can be merged since φiφj = φjφi. The summation on the right side of
Equation 24 is partitioned into two parts as follows


_φ[t]i[i]_


_φ[s]i_ _[i]_


= τ (ϕt) + τ (ϕs). (25)


(ti + si)gi =


_tigi +_


_sigi = τ_


+ τ


Consequently, we have τ (ϕtϕs) = τ (ϕt) + τ (ϕs), which states that τ is a homomorphism. Then, we
only need to prove that mapping τ is bijective. First, we prove τ is injective, i.e., ker(τ ) = _eG_,
_{_ _}_
after that, we prove τ is surjective, i.e., one can find the inverse image of any element.

Since it is not hard to obtain τ (ϕ) = τ (e _ϕ) = τ_ (e) + τ (ϕ), we have τ (e) = (0, . . ., 0) = eG, i.e.,
_·_
_e ∈_ _τ_ _[−][1](eG). Assume there is ϕl =_ _i_ _[φ]i[l][i]_ _[, s.t. τ]_ [(][ϕ][l][) =][ e][G][, we have]

_τ_ (ϕl) = _i_ _ligi = eG ⇒_ _li[Q]|n (li are divisible by n) ⇒_ _ϕl =_ _i_ _φ[l]i[i]_ [=] _i_ _e = e._ (26)
X Y Y

The equation above states that τ (eG) = _e_ (τ : Φ Φ) and mapping τ is injective. We take

_[−][1]_ _{_ _}_ _→_
_∀_ _g ∈_ _G, g = (k1, k1, . . ., km) ∈_ (Z/nZ)[m], and have


_φ[k]i_ _[i]_


(27)


_g = (k1, . . ., 0) +_ + (0, . . ., km) =
_· · ·_


_kigi = τ_


Hence, we haveτ is bijective and homomorphism, τ _[−][1](g) =_ _i_ _[φ]i[k][i]_ _τ[∈] is an isomorphism.[Φ][, which indicates that mapping][ τ][ is surjective. Since mapping]_

Q.E.D. [Q]

D PROOF FOR THEOREM 3

_Proof. In the following, we prove that the necessary and sufficient condition for satisfying Abel and_
Order constraints is that both Abel and Order Loss are minimized.

_⇒image) For the condition: o ∈_ _O can be obtained by ∀_ _φi, φj ∈_ Φ, 0 ≤ _i, j ≤_ _m, we have φiφj = φjφi. The constraint on any_

_φi(φj(o)) = φj(φi(o))_ _φi(φj(o))_ _φj(φi(o)) = 0._ (28)
_⇒_ _−_


-----

For the set of combinations of factors C = {(i, j)|1 ≤ _i, j ≤_ _m} and the set containing images O,_
we have
_La =_ _oX∈O_ (i,jX)∈C _∥φi · (φj · o) −_ _φj · (φi · o)∥_ is minimized. (29)

We obtain the Abel loss. For the Order loss, we first consider n times composition of the same
mapping φi, we have
_φ[n]i_ [=][ e][ ⇒] _[φ]i[n][−][1]φi = φiφi[n][−][1]_ = e. (30)

Therefore, we then have φ[−]i [1] = φ[n]i _[−][1]. For a single image o_ _O, we have_
_∈_

_φi(φ[n]i_ _[−][1](o)) = φi(φ[−]i_ [1][(][o][)) =][ o][ ⇒] _[φ][i][(][φ]i[−][1][(][o][))][ −]_ _[o][ = 0][.]_ (31)

Thus, for the set of factors and the set containing all images O, we have


_φi_ (φ[−]i [1] _o)_ _o_ is minimized. (32)
0≤Xi≤m _∥_ _·_ _·_ _−_ _∥_


_o∈O_


To eliminate the bias of optimization, we optimize the symmetry form of the Order Loss, and we have


( _φi_ (φ[−]i [1] _o)_ _o_ + _φ[−]i_ [1] (φi _o)_ _o_ ) is optimized. (33)
0≤Xi≤m _∥_ _·_ _·_ _−_ _∥_ _∥_ _·_ _·_ _−_ _∥_


_Lo =_


_o∈O_


The Order Loss is obtained.

_⇐)When the Abel Loss La is optimized, for ∀_ _o ∈_ _O, we have_

_φi(φj(o))_ _φj(φi(o)) = 0_ _φi(φ(o)) = φj(φi(o))._ (34)
_−_ _⇒_

Therefore, forobtain the Abel constraint. When the Order Loss ∀ _φi, φj ∈_ Φ, i = 1, 2, . . ., m and L j = 1o is minimized, for, 2, . . ., m, we have ∀ _o ∈ φOi, we haveφj = φjφi, and we_

_φi(φ[−]i_ [1][(][o][))][ −] _[o][ = 0][ ⇒]_ _[φ][i][(][φ]i[−][1][(][o][)) =][ o.]_ (35)

This implies that
_φ[n]i_ [=][ φ][i] _i_ = φi _φ[−]i_ [1] = e. (36)

_[◦]_ _[φ][n][−][1]_ _◦_

Therefore, we obtain the Order constraint. The Group constraints are satisfied.


-----

E THE DATA CONSTRAINT

In this section, we prove that if each dimension of the world state is independently sampled (p(w) =
Πip(wi), where p denotes the probability mass function (for discrete random variable) or probability
density function (for continuous random variable)), then the minimization of total correlation (p(z) =
Πip(zi)) is a necessary condition to satisfy the data constraint (see Theorem 4 and Theorem 5 below).

Our goal is to learn a representation z conforming to the group-based definition of disentanglement
with an encoder h and a decoder d. Then the data constraint can be formulated as follows. For a
generatorcorresponding permutation of gi ∈ _G, we have b g(ig under the isomorphism betweeni · w) = φi · b(w) = h[−][1](gi · G z) and, where Φ. We reorganize the formula z = f_ (w) and φi is the
and have h _b(gi_ _w) = f_ (gi _w) = gi_ _z._
_◦_ _·_ _·_ _·_

**Theorem 4and the action of each generator Assume that the action of generator gi on w only affects a single dimension of gi on z is gi · z = z + g wi (element-wise addition), (see the assumptions in**
_Section 3.2). Then the equation f_ (gi _w) = gi_ _z,_ _w_ _W is equivalent to: for each i = 1, 2, . . ., n,_
_there exists a bijective function γi s.t. z ·_ _i = γi ·(wj ∀) for some ∈_ _j._

_Proof. ⇒) Without loss of generality, we assume gi only affects the j-th dimension of w. If the_
equation f (gi _w) = gi_ _z,_ _w_ _W holds, it’s obvious that, for each i, there exists a bijective_
function γi s.t. · zi = γi(w ·j). _∀_ _∈_

_⇐for) In the following, we use i = 1, . . ., n, the functions (g zi ·iw =)i γ ((i(gwi ·j)z hold for some)i) to denote the j, by using an index permutation i-th dimension of vector gi ·_ _w ( jg =i ·_ _z π). If(i)_
we can rewrite the function as zi = γi(wπ(i)). Therefore, z can be formulated as follows

_f_ (w1, . . ., wn) = z = (z1, . . ., zn) = (γ1(wπ(1)), . . ., γn(wπ(n))). (37)

Please note that how the world state transits on dimension i dose not affect how disentangled the
representation is. Therefore, we define action of generator gi on w as follows.

_gi_ _w = gi_ (w1, . . ., wn) = (w1, . . ., (gi _w)j, . . ., wn) = (w1, . . ., γi[−][1]((gi_ _z)i), . . ., wn), (38)_
_·_ _·_ _·_ _·_

Then, we take f on both sides of the above equation, and apply Equation 37, and have

_f_ (gi _w)_ = f (w1, . . ., γi[−][1]((gi _z)i), . . ., wn) = (γ1(wπ(1)), . . ., γi(γi[−][1]((gi_ _z)i)), . . ., γn(wπ(n)))_
_·_ _·_ = (z1, . . ., (gi _z)i, . . ., zn) = gi_ _z_ _·_
_·_ _·_ (39)
Q.E.D.

Note that it is obvious that the following theorem (Theorem 5) holds for discrete random variables
and bijective functions. However, since the world state space is dense and the encoder h and decoder
_d are differentiable in general, we also prove for the case where w and z are treated as continuous_
random variables. Theorem 5 states that the minimization of total correlation (p(z) = Πip(zi)) is a
necessary condition to make equations zi = γi(wπ(i)) satisfied, where i = 1, 2, . . ., n and π(i) is an
index permutation.


**Theorem 5 For the independent random variables w1, w2, . . ., wn, considering the functions zi =**
_δi(w1, w2, . . ., wn), where i = 1, 2, . . ., n and each δi is a bijective, differentiable function. For_
_i = 1, . . ., n, if there exists an index permutation π(i) s.t., we have_ _∂z∂wπ(ii)_ = 0̸ _and_ _∂z∂wπ(ki)_ [= 0][, where]

_k = 1, 2 . . ., n but k ̸= i, then the new random variables z1, z2, . . ., zn are independent._

_Proof. We treat the random variables w1, w2, . . ., wn as an n-dimensional random vector W =_
(w1, w2, . . ., wn). Similarly, we write Z = (zπ(1), zπ(2), . . ., zπ(n)), which is rearranged by index
permutation π(i). According to Change of Variable Theorem For Random Vectors (Mood, 1950), we
have

_p(Z)_ _J(W_ ) = p(W ) = Πip(wi) (40)
_|_ _|_

where J(W ) is the jacobian matrix of Z w.r.t. W, the (i, j)-th entry of it is _∂z∂wπ(ji)_ [. Since for each]

_i = 1, . . ., n, we have_ _∂z∂wπ(ii)_ = 0̸ and _∂z∂wπ(ki)_ [= 0][, where][ k][ = 1][,][ 2][ . . ., n][ but][ k][ ̸][=][ i][, then the jacobian]


-----

matrix can be formulated as follows


_∂zπ(i)_
_J(W_ ) = Πi (41)

_∂wi_

According to Change of Variable Theorem For Random Variable ( Mood (1950)), we have


_p(zπ(i))|_ _[∂z]∂w[π][(]i[i][)]_ _| = p(wi)_ (42)

Bring Equation 41 and Equation 42 into Equation 40, we have


Πip(zπ(i))Πi _∂z∂wπ(ii)_

_p(Z) = Π[Π]i[i][p]∂z∂w[(][w]π(i[i]i)[)]_ = Πi _∂z∂wπ(ii|)_ _[|]_ = Πip(zπ(i)) = Πip(zi) (43)

_|_ _[|]_ _|_ _[|]_


Q.E.D.


Please note that the inverse proposition of the theorem above does not hold. As a counterexample, for
zero mean independent gaussian random variables w1, w2 with common variance σ[2], new random
variables z1 = _w1[2]_ [+][ w]2[2] [(the norm of vector][ (][w][1][, w][2][)][),][ z][2][ = tan][−][1][(][w][2][/w][1][)][ (the angle between]

vector (w1, w2)p and (0, 1)) are independent (Mood, 1950) but _∂w∂zij_

addition, the sufficiency would be satisfied for some specific settings, e.g., there are only two[̸][= 0][ for all][ i, j][ ∈{][1][,][ 2][}][. In]
independent uniformly distributed random variables w1, w2, and the functions z1, z2 are linear
functions.

Combining Theorems 4 and 5, we have that the minimization of total correlation (p(z) = Πip(zi)) is
a necessary condition to satisfy the data constraint.

F DETAILS OF IMPLEMENTATION

F.1 ABEL LOSS DETAILS

As mentioned in the main paper, the Abel Loss of the VAE-based models is as follows:

_La =_ _oX∈O_ (i,jX)∈C _∥φi · (φj · o) −_ _φj · (φi · o)∥,_ (44)

where φi (φj _o) = φi(φj(o)) represents the top path of Figure 7 (a), and φj_ (φi _o) = φj(φi(o))_
represents the bottom path of Figure 7 (a). For better optimization, we constrain such consistency on · _·_ _·_ _·_
their representation (straight dotted double arrow in Figure 7 (a)) instead of the reconstructed images.
Besides, we constrain the consistency between the representations of intermediate images (curved
dotted double arrow in Figure 7 (a)).

Encoder Decoder Encoder Decoder Encoder Decoder Encoder Decoder

|Encoder|Decoder|
|---|---|



Encoder Decoder Encoder Decoder Encoder Decoder Encoder Decoder

(a) Abel Loss (b) Order Loss

Encoder Decoder Encoder Decoder

Encoder Decoder Encoder Decoder

(b) Order Loss

Encoder Decoder Encoder Decoder

Encoder Decoder Encoder Decoder

(a) Abel Loss

Figure 7: Overview of the Isomorphism Loss. The Abel Loss and Order Loss constrain the commutativity and cyclicity of permutation group Φ, respectively. The dotted lines in the figure represents
reconstruction loss.
F.2 ORDER LOSS DETAILS

As mentioned in the main paper, the Order Loss of VAE-based models is as follows:


_φi_ (φ[−]i [1] _o)_ _o_ + _φ[−]i_ [1] (φi _o)_ _o_ _,_ (45)
_∥_ _·_ _·_ _−_ _∥_ _∥_ _·_ _·_ _−_ _∥_



_Lo =_


_o∈O_


_i∈I_


-----

where φi (φ[−]i [1] _o) represents the lower path of Figure 7 (b), and φ[−]i_ [1] (φi _o) represents the_
upper path of Figure 7 (b). Similar to the Abel Loss, we do not constrain such consistency on the · _·_ _·_ _·_
reconstructed images for better optimization but on their representations instead (the long curved
dotted line in Figure 7 (b)). Besides, we constrain the consistency between the representations of
intermediate images (short curved dotted lines in Figure 7 (b)).

F.3 THE NUMBER OF FACTORS m

For the given VAE-based models, dimensional KL divergence (indicating the meaningful dimensions)
increases during the training process. Therefore, we use m dimensions of which the corresponding
dimensional KL divergence KLi _T_, where T is a hyperparameter, which is empirically set to 30
in our experiments. _≥_


-----

G DETAILS OF EXPERIMENTS

G.1 DATASET DETAILS

In all the experiments, we resize the images to 64 × 64 resolution. We introduce all the datasets used
in our paper in detail.

**dSprites Higgins et al. (2017): dSprites contains 737,280 binary 2D shapes (heart, oval and square)**
images with five ground truth factors: shape (3 values), scale (6 values), orientation (40 values),
x-position (32 values), y-position (32 values). Then we introduce the variants of dSprites (Color
dSprites and Noisy dSprites) created by Locatello et al. Locatello et al. (2019b).

**Color dSprites, the shapes of the images in dSprites are randomly colored.**

**Noisy dSprites, the background of the images in dSprites is noise.**

**Shapes3D Kim & Mnih (2018): Shapes3D contains 480,000 images of 3D shapes with 6 ground**
truth factors: shape (4 values), scale (8 values), orientation (15 values), floor color (10 values), wall
color (15 values), object color (10 values).

**Cars3D Reed et al. (2015): This dataset consists of 183 car CAD models, each rendered from 24**
azimuth directions and 4 elevations.

Encoder Decoder

Input: 64×64 × number of channels Input: 10 or 20
4 × 4 conv, 32 ReLU, stride 2 FC, 256 ReLU
4 × 4 conv, 32 ReLU, stride 2 FC, 256 ReLU
4 × 4 conv, 32 ReLU, stride 2 FC, 4 × 4 × 32 ReLU
4 × 4 conv, 32 ReLU, stride 2 4 × 4 deconv, 32 ReLU, stride 2
FC 256 ReLU 4 × 4 deconv, 32 ReLU, stride 2
FC 256 ReLU 4 × 4 deconv, 32 ReLU, stride 2
FC 2× 10 4 × 4 deconv, number of channels, stride 2

Table 3: Architecture of the encoder and decoder of VAEs. For Original VAE, the dimension of input
of the decoder is 10. For Groupified VAE, the dimension is 20. Note that the number of representation
dimensions of Groupified VAE is still 10, which is the same as Original VAE, the comparison with
Original VAE is fair Watters et al. (2019).

G.2 ARCHITECTURE FOR ENCODER AND DECODER

We follow Locatello et al. Locatello et al. (2019b) to use the same architecture of VAEs in all of the
experiments: the activation function used is ReLU except for the last layer of decoder, as shown in
Table 3. For the details of the Discriminator in FactorVAE, please refer to Table 5 (a) and (c).

Model Parameter Value

_β-VAE_ _β_ [10; 20; 30;]
AnnealedVAE C [10; 20; 30;]
start [3e4; 4e4; 5e4;]
end [2e4; 3e4; 4e4;]
FactorVAE _γ_ [5; 10; 15]
_β-TCVAE_ _β_ [6; 9; 12]
random seed [1; 2; 3; 4; 5; 6; 7; 8; 9;]
group [True; False]

Table 4: Hyperparameters and random seeds for every model.


-----

(c) Architecture of Discriminator

Discriminator

FC, 1000 LReLU
FC, 1000 LReLU
FC, 1000 LReLU
FC, 1000 LReLU
FC, 1000 LReLU
FC, 1000 LReLU
FC, 2


(a) Optimizer for Discriminator

Parameter Values

Batch size 64
Optimizer Adam
Adam: beta1 0.9
Adam: beta2 0.999
Adam: epsilon 1.0e-8
Adam: learning rate 0.0001


(b) General hyperparameters for VAE

Parameter Values

Batch size 64
Representation dimension 10
Optimizer Adam
Adam: beta1 0.9
Adam: beta2 0.999
Adam: epsilon 1.0e-8
Adam: learning rate 0.0001
Decoder type Bernoulli


Table 5: Shared hyperparameters in all experiments. LReLU stands for leaky ReLU.

G.3 EXPERIMENT SETTINGS

We run using different hyperparameters and random seeds for every VAE-based model implemented
by Pytorch Paszke et al. (2017). As shown in Table 4, for β-VAE, we assign 3 choices for β and 10
random seeds for both the Original and Groupified VAEs: 3 × 10 × 2 = 60 settings for each dataset.
Similarly, we also assign 60 settings for FactorVAE and β-TCVAE. For AnnealVAE, we assign three
choices for C and 3 choices for the start and end pair, also assign 10 random seeds. In summary,
for all 5 datasets, we run (((3 × 10 × 2) × 3) + 3 × 3 × 10 × 2) × 5 = 1800 models. For other
hyperparameters, please refer to Table 5 (b).

H RELATION TO SOME PREVIOUS WORKS

H.1 SYMMETRY-BASED DISENTANGLEMENT

Caselles-Dupre et al. (2019) argue that the symmetry-based disentanglement requires interaction´
with the environments. Specifically, for a given disentangled representation z w.r.t. a world with
some group action on W, there are multiple other worlds (same world states and symmetry) of
a static dataset that have different group actions on each dimension of W . For those worlds, the
representation z is not disentangled, per the group-based definition. However, in our work, we do
not assume that a static dataset has a world equipped with some specific fixed group action on W in
advance, instead we use permutation group Φ as an agent to learn a world with proper group actions.
For this learned world, there is only one representation that satisfies the definition. Therefore, we
do not need an environment to provide group actions on W to determine which world it is. Please
see our proof in Theorem 1. In the testing phase, since we input the images to the model directly
to derive their representation, the disentanglement of the representation does not rely on how the
world state transitions between each other (i.e., as a result of group actions on W ). Therefore, our
framework can learn such a disentangled representation without interaction with the environments.

H.2 LIE GROUP VAE

Zhu et al. (2021) argue that the representation space being a vector space is sub-optimal since it
requires the model to learn to discard different scales of variations. They propose to use lie group
as the representation space instead and use Hessian Penalty to encourage disentanglement. Our
framework is complimentary to their proposed method. The lie group representation can be applied
to extend our framework. Here we leave it for future work. In addition, our proposed representation
mapped by the sine and cosine function (exp((2πiz)/n) for a real vector z) is also a lie group.


-----

I MORE QUANTITATIVE RESULTS AND SCORE DISTRIBUTION

I.1 QUANTITATIVE RESULTS

The performance of the Original and Groupified VAEs on all five datasets is shown in Table 6. Our
method outperforms the original one on most of the datasets in terms of nearly all the metrics.

DCI BetaVAE MIG FactorVAE
dSprits

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000....35232838 ± ± ± ± 0 0 0 0....065101010 **0000...463941.36 ± ± ± ± 0 0 0 0...085056074.11** **0000....89758486 ± ± ± ± 0 0 0 0....083050026040** **0000....878618689 ± ± ± ± 0 0 0 0...0067051020.038** 0000....14271723 ± ± ± ± 0 0 0 0....09709206710 **0000....37343124 ± ± ± ± 0 0 0 0....089061061093** **0000....70517468 ± ± ± ± 0 0 0 0....098068098094** **0000....63687570 ± ± ± ± 0 0 0 0....058089075098**

DCI BetaVAE MIG FactorVAE
Cars3d

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000....18222124 ± ± ± ± 0 0 0 0....059046054049 **0000....24252526 ± ± ± ± 0 0 0 0....041046040046** 000.99..99199. ±0 ± ± ± 1 1 4. 06eee −. −0 −443 **0.99111 ±...000 ± ± ± 1. 0 0 05e...000 −** **4** 0000....07107409810 ± ± ± ± 0 0 0 0....021032027016 **0000....10111111 ± ± ± ± 0 0 0 0....014032033033** 0000....81908882 ± ± ± ± 0 0 0 0....066039040062 **0000....93879393 ± ± ± ± 0 0 0 0....034028034034**

Noisy DCI betaVAE MIG FactorVAE
dSprits Original Groupified Original Groupified Original Groupified Original Groupified

Anneal VAEFactorVAEβBetaVAE-TCVAE **0000....114056053081 ± ± ± ± 0 0 0 0....018013036062** **0000....099087060111 ± ± ± ± 0 0 0 0....057051022053** 0000....682624631605 ± ± ± ± 0 0 0 0....090036081053 **0000....647644684635 ± ± ± ± 0 0 0 0....055031070050** **0000....077030035040 ± ± ± ± 0 0 0 0....022027030046** **0000....066065047068 ± ± ± ± 0 0 0 0....046055032042** 0000....355434353437 ± ± ± ± 0 0 0 0....093080091098 **0000....407468481431 ± ± ± ± 0 0 0 0....071098087097**

DCI BetaVAE MIG FactorVAE
Shapes3d

Original Groupified Original Groupified Original Groupified Original Groupified

AnnealVAEFactorVAEβ-TCVAEβ-VAE 0000....44526647 ± ± ± ± 0 0 0 0....1760511010 **0000....49726056 ± ± ± ± 0 0 0 0....06506107810** **0000....86978291 ± ± ± ± 0 0 0 0....076072055039** 0000....90898096 ± ± ± ± 0 0 0 0....045075042086 0000....48283340 ± ± ± ± 0 0 0 0....047181318 **0000....47504243 ± ± ± ± 0 0 0 0....0900521511** **0000....81827589 ± ± ± ± 0 0 0 0....074064056098** **0000....82839079 ± ± ± ± 0 0 0 0....066043066046**


Color DCI betaVAE MIG FactorVAE
dSprits Original Groupified Original Groupified Original Groupified Original Groupified

Anneal VAEFactorVAEβBetaVAE-TCVAE 0000....174268294338 ± ± ± ± 0 0 0 0....097103101052 **0000....328337322395 ± ± ± ± 0 0 0 0....130114104082** 0000....798843861876 ± ± ± ± 0 0 0 0....094038038024 **0000....844856862881 ± ± ± ± 0 0 0 0....050031029031** 0000....103219203169 ± ± ± ± 0 0 0 0....058084080040 **0000....243252236269 ± ± ± ± 0 0 0 0....118104091090** **0000....718739591711 ± ± ± ± 0 0 0 0....148086065068** **0000....648786692730 ± ± ± ± 0 0 0 0....094080092050**

Table 6: Performance (mean ± std) on different datasets and different models with different metrics.
We evaluate β-VAE, AnnealVAE, FactorVAE, and β-TCVAE on dSprites, Cars3d, Shapes3d, NoisydSprites, and Color-dSprites for 1800 settings. These settings include different random seeds and
hyperparameters.

I.2 ABSTRACT REASONING & FAIRNESS
Abstract reasoning↑ Unfairness scores↓

As pointed out by Locatello et al. Locatello et al. Original 0.948 ± 0.031 0.023 ± 0.007
(2019b), the disentangled representation’s down- Groupified **0.954 ± 0.028** **0.018 ± 0.008**
stream tasks should also be verified. Therefore, we verify the effectiveness of the repre- Table 7: Downstream task performance on the
sentations learned by the Groupified VAEs on models trained on the representation learned by
Shapes3d in two downstream tasks: abstract rea- original and groupified FactorVAE.
soning Van Steenkiste et al. (2019) and fairness evaluation Locatello et al. (2019a). As Table 7
shows, the performance of the abstract reasoning models fine-tuned on the representation learned by
the Groupified FactorVAEs is better than the original ones. In terms of fairness evaluation, we can
observe that the unfairness scores of the representation learned by the Groupified FactorVAEs are
lower than the Original ones.


I.3 SCORE DISTRIBUTION

The detailed distribution of the performance is shown in this section (demonstrated by the Violin
Plot Hintze & Nelson (1998)). The performance distributions on dSprits, Car3d, Noisy dSprites,


-----

Color-dSprites, and Shapes3d are shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12,
respectively.

I.4 COMPARISON WITH METHOD INTERACTION WITH THE ENVIRONMENT

Our work considers an unsupervised setting, which is a more practical one. To understand the price
to pay, we compare our unsupervised Groupified models to the methods that use the interaction with
the environment as supervision. Here we provide the comparison between RGrVAE (Painter et al.,
2020) and our Groupified β-TCVAE on dSprites, Shapes3D, and Color dSprites as shown in Table 8.

DCI BetaVAE MIG FactorVAE
Datasets

RGrVAE Groupified RGrVAE Groupified RGrVAE Groupified RGrVAE Groupified

Color dSpritesShapes3DdSprites 000...528311 ± ± ± 0 0 0...058056072 000...367240 ± ± ± 0 0 0...110061082 010...970053 ± ± ± 0 0 0...039000294 000...869688 ± ± ± 0 0 0...038042031 000...082503 ± ± ± 0 0 0...042031028 000...244727 ± ± ± 0 0 0...093090090 000...869831 ± ± ± 0 0 0...073032309 000...709079 ± ± ± 0 0 0...098046050

Table 8: Performance (mean ± variance) on different datasets of RGrVAE and Groupified β-TCVAE
with different metrics. These settings include different random seeds and hyperparameters.

Since there are no results reported on Shapes3D and Color dSprites, we conduct experiments on
these two datasets with the official implementation[2] using the recommended hyper-parameters. In
addition, the results reported in Painter et al. (2020) are of models trained with 16 latent units and
3 random seeds. We also conduct experiments on dSprites with 10 latent units and 10 random
seeds, which is our setting. From Table 8, we observe that there is still a gap between RGrVAE and
_Groupified β-TCVAE, especially on Shapes3D. However, the latent learned by RGrVAE is not as_
pure as Groupified β-TCVAE (lower MIG). Additionally, RGrVAE performs poorly because a factor
(color) is not modeled in the environment of Color dSprites.

I.5 COMPARISON WITH CONTROLVAE

In this section, we provide a comparison between Original and Groupified ControlVAE (Shao
et al., 2020). For ControlVAE, we use the official implementation[3] and follow the default setting,
_Cmax = 25. We follow Locatello et al. (2019b) to set the hyperparameter interval to 10. For other_
parameters, we follow Shao et al. (2020). The results on dSprites, Shapes3D, and Color dSprites are
presented in Table 9.

DCI BetaVAE MIG FactorVAE
Datasets

Original Groupified Original Groupified Original Groupified Original Groupified

Color dSpritesShapes3DdSprites 000...315947 ± ± ± 0 0 0...093144111 000...468554 ± ± ± 0 0 0...115165055 000...838694 ± ± ± 0 0 0...084142045 000...929796 ± ± ± 0 0 0...061149024 000...162128 ± ± ± 0 0 0...062212095 000...277235 ± ± ± 0 0 0...123151055 000...625980 ± ± ± 0 0 0...087174070 000...748884 ± ± ± 0 0 0...110171022

Table 9: Performance (mean ± variance) on different datasets of ControlVAE and Groupified ControlVAE with different metrics. These settings include different random seeds and hyperparameters.

From Table 9, we observe that our method consistently improved the performance of ControlVAE
under the same hyper-parameters, especially on Shapes3D.

I.6 DETAILED RESULTS UNDER DIFFERENT HYPER-PARAMETERS

In order to provide a more convincing comparison, we compare our method and the original one at
different levels of regularization parameters. We take β-TCVAE as an example. As Table 10, 11, and
12 show, our method consistently improves the performance of the original methods on most of the
metrics, especially on MIG and DCI.

2
[https://github.com/MattPainter01/UnsupervisedActionEstimation](https://github.com/MattPainter01/UnsupervisedActionEstimation)
3
[https://github.com/shj1987/ControlVAE-ICML2020](https://github.com/shj1987/ControlVAE-ICML2020)


-----

Regulize DCI BetaVAE MIG FactorVAE
strength

Original Groupified Original Groupified Original Groupified Original Groupified

_βββ = 12 = 6 = 9_ 000...333735 ± ± ± 0 0 0...079051057 000...343638 ± ± ± 0 0 0...106103114 000...868587 ± ± ± 0 0 0...025028024 000...858786 ± ± ± 0 0 0...032026051 000...151720 ± ± ± 0 0 0...042053071 000...202430 ± ± ± 0 0 0...093097098 000...686371 ± ± ± 0 0 0...100092084 000...687071 ± ± ± 0 0 0...105072104


Table 10: Performance (mean ± variance) on dSprites of Original and Groupified β-TCVAE with
different metrics. The results under different regularize strength are reported.

Regulize DCI BetaVAE MIG FactorVAE
strength

Original Groupified Original Groupified Original Groupified Original Groupified

_βββ = 12 = 6 = 9_ 000...566775 ± ± ± 0 0 0...080089053 000...687673 ± ± ± 0 0 0...071048026 000...989796 ± ± ± 0 0 0...028037044 000...979693 ± ± ± 0 0 0...036048048 000...313851 ± ± ± 0 0 0...123196151 000...425151 ± ± ± 0 0 0...098084082 000...889092 ± ± ± 0 0 0...046078059 000...908990 ± ± ± 0 0 0...052045040


Table 11: Performance (mean ± variance) on Shapes3D of Original and Groupified β-TCVAE with
different metrics. The results under different regularize strength are reported.

Regulize DCI BetaVAE MIG FactorVAE
strength

Original Groupified Original Groupified Original Groupified Original Groupified

_βββ = 12 = 6 = 9_ 000...353631 ± ± ± 0 0 0...051019061 000...434035 ± ± ± 0 0 0...073077081 000...898986 ± ± ± 0 0 0...009013031 000...898887 ± ± ± 0 0 0...016019020 000...171716 ± ± ± 0 0 0...038037044 000...292724 ± ± ± 0 0 0...073104083 000...737466 ± ± ± 0 0 0...094077060 000...827975 ± ± ± 0 0 0...027048051


Table 12: Performance (mean ± variance) on Color dSprites of Original and Groupified β-TCVAE
with different metrics. The results under different regularize strength are reported.

-VAE
_β_

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score


(e) BetaVAE score (f) DCI disentanglement (g) MIG (h) FactorVAE score

(i) BetaVAE score (j) DCI disentanglement (k) MIG (l) FactorVAE score


(m) BetaVAE score (n) DCI disentanglement (o) MIG (p) FactorVAE score

Figure 8: Performance distribution on dSprites. Variance is due to different hyperparameters and
random seeds. We consider four metrics: BetaVAE score, DCI disentanglement, MIG, and FactorVAE
score. We observe that Groupified VAEs outperform the original ones.


-----

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score

(e) BetaVAE score (f) DCI disentanglement (g) MIG (h) FactorVAE score


(i) BetaVAE score (j) DCI disentanglement (k) MIG (l) FactorVAE score


(m) BetaVAE score (n) DCI disentanglement (o) MIG (p) FactorVAE score

Figure 9: Performance distribution on Cars3d. Variance is due to different hyperparameters and
random seeds. We observe that Groupified models outperform the Original ones.


-----

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score

(e) BetaVAE score (f) DCI disentanglement (g) MIG (h) FactorVAE score


(i) BetaVAE score (j) DCI disentanglement (k) MIG (l) FactorVAE score


(m) BetaVAE score (n) DCI disentanglement (o) MIG (p) FactorVAE score

Figure 10: Performance distribution on Noisy dSprites. Variance is due to different hyperparameters
and random seeds. We observe that Groupified VAEs outperform the Original ones.


-----

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score

(e) BetaVAE score (f) DCI disentanglement (g) MIG (h) FactorVAE score


(i) BetaVAE score (j) DCI disentanglement (k) MIG (l) FactorVAE score


(m) BetaVAE score (n) DCI disentanglement (o) MIG (p) FactorVAE score

Figure 11: Performance distribution on Color dSprites. Variance is due to different hyperparameters
and random seeds. We observe that Groupified VAEs outperform the Original ones.


-----

(a) BetaVAE score (b) DCI disentanglement (c) MIG (d) FactorVAE score

(e) BetaVAE score (f) DCI disentanglement (g) MIG (h) FactorVAE score


(i) BetaVAE score (j) DCI disentanglement (k) MIG (l) FactorVAE score


(m) BetaVAE score (n) DCI disentanglement (o) MIG (p) FactorVAE score

Figure 12: Performance distribution on Shapes3d. Variance is due to different hyperparameters and
random seeds. We observe that Groupified models outperform the Original ones.


-----

MORE QUALITATIVE RESULTS


We evaluate our methods qualitatively on two typical datasets: Cars3d and Shapes3d. We visualize
the traversal results of the Original and Groupified FactorVAE and β-TCVAE. For every factor, we
traverse five randomly sampled representations. As shown in Figure 13 and Figure 14, the traversal
results of the Groupified FactorVAE and β-TCVAE show that these models learn less entangled
representations on Shapes3D (e.g., Orientation of FactorVAE and Scale and Shape of β-TCVAE).

Similarly, as shown in Figure 15 and Figure 16, the Groupified FactorVAE and β-TCVAE achieve
better disentanglement ability on Car3d (e.g., Rotation of FactorVAE and Yaw of β-TCVAE).

For the real-world dataset, we show the qualitative comparison of the Groupified and original
AnnealVAE trained on CelebA. As Figure 17 shows, for most of the factors, our model can extract
cleaner semantics than the original model. For example, the hair color is entangled with the face
shape in the original model, but cleaner in the Groupified model.

Original

Groupified

(a) Orientation (b) Wall color (c) Object color (d) Floor color (e) Scale (f) Shape


Figure 13: Learned latent variables using Original and Groupified FactorVAE on Shapes3d dataset.
The traversal range is (-2, 2).

Original

Groupified

(a) Orientation (b) Wall color (c) Object color (d) Floor color (e) Scale (f) Shape


Figure 14: Learned latent variables using Original and Groupified β-TCVAE on Shapes3d dataset.
The traversal range is (-2, 2).


-----

(a) Yaw (b) Color (c) Whiten (d) Rotation

Figure 15: Learned latent variables using Original and Groupified FactorVAE on Car3d dataset. The
traversal range is (-2, 2).


(a) Azimuth (b) Yaw (c) Color (d) Whiten (e) Rotation


Figure 16: Learned latent variables using Original and Groupified β-TCVAE on Car3d dataset. The
traversal range is (-2, 2).

(a) Original (b) Groupified


hair length

gender


skin color

azimuth


hair style

skin color2

face shape


smile


glasses

Figure 17: Learned latent variables using Original and Groupified AnnealVAE on CeleBa dataset.
The traversal range is (-2, 2). The factors learned by Groupified model are less entangled.


-----

K MEANINGFUL DIMENSION VISUALIZATIONS

When we assign some dimensions to Isomorphism Loss for Groupified AnnealVAEs, e.g., the first-5
dimensions, we have an interesting observation that the assigned dimensions are meaningful in
_Groupified AnnealVAEs. The KL divergence increases continuously on these assigned dimensions_
after the Isomorphism Loss is applied to them. Note that the KL divergence loss in AnnealVAE
indicates the amount of information encoded. As Figure 18 shows, the KL divergence of assigned
dimensions increases at the beginning of training, which means Isomorphism Loss results in that
the assigned dimensions become meaningful. Finally, the assigned first-five dimensions learn to
encode the semantics of x position, y position, scale, and orientations. The possible reason is that
the latent factors are learned and disentangled in the assigned dimensions due to the punishment of
the Isomorphism Loss. To illustrate that controllable dimensions in Groupified AnnealVAE are not
an exception, we provide more visualizations. The results of covering all hyperparameters settings
with two random seeds are shown in Figure 18 to Figure 26, suggesting that their dimensions are
controllable.

(a) Dimensional KL in training (b) Dimensional KL in 3e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 3e5 steps (f) Images traversal of dimensions

Figure 18: Meaningful dimensions visualization for C = 10, end = 30000 (different random seeds).
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.


-----

(a) Dimensional KL in training (b) Dimensional KL in 4e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 4e5 steps (f) Images traversal of dimensions

Figure 19: Meaningful dimensions visualization for C = 10, end = 40000 (different random seeds).
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different amounts after training (b). As the image
traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So as (d), (e), and (f),
which are results of a run with a different random seed.

(a) Dimensional KL in training (b) Dimensional KL in 5e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 5e5 steps (f) Images traversal of dimensions

Figure 20: Meaningful dimensions visualization for C = 10, end = 50000 (different random seeds).
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.


-----

(a) Dimensional KL in training (b) Dimensional KL in 3e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 3e5 steps (f) Images traversal of dimensions

Figure 21: Meaningful dimensions visualization for C = 20, end = 30000 (different random seeds).
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different amounts after training (b). As the image
traversal results (c) show, the meaningful dimensions are learned in 0-4 dims. So as (d), (e), and (f),
which have a different random seed.

(a) Dimensional KL in training (b) Dimensional KL in 4e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 4e5 steps (f) Images traversal of dimensions

Figure 22: Meaningful dimensions visualization for C = 20, end = 40000 (different random seeds).
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.


-----

(a) Dimensional KL in training (b) Dimensional KL in 5e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 5e5 steps (f) Images traversal of dimensions

Figure 23: Meaningful dimensions visualization for C = 20, end = 50000 (different random seeds).
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.

(a) Dimensional KL in training (b) Dimensional KL in 3e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 3e5 steps (f) Images traversal of dimensions

Figure 24: Meaningful dimensions visualization for C = 30, end = 30000 (different random seeds).
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.


-----

(a) Dimensional KL in training (b) Dimensional KL in 4e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 4e5 steps (f) Images traversal of dimensions

Figure 25: Meaningful dimensions visualization for C = 30, end = 40000 (different random seeds).
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.

(a) Dimensional KL in training (b) Dimensional KL in 5e5 steps (c) Images traversal of dimensions

(d) Dimensional KL in training (e) Dimensional KL in 5e5 steps (f) Images traversal of dimensions

Figure 26: Meaningful dimensions visualization for C = 30, end = 50000 (different random seeds).
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b). As the image traversal
results (c) show, the meaningful dimensions are learned in 0-4 dims. So are (d), (e), and (f), which
are results of a run with a different random seed.


-----

L MORE REPRESENTATION SPACE VISUALIZATIONS

To illustrate that the Isomorphism Loss suppresses the representation space collapse in VAE-based
methods is not an exception, we provide more visualizations of the representation space visualization
of Groupified VAEs. The Original and Groupified β-VAEs, AnnealVAEs, FactorVAEs, and βTCVAEs are shown in Figure 27, Figure 28, Figure 29, and Figure 30, respectively, which implies
that the representation space of Groupified VAEs are organized better than the original ones.

(a) β = 10, Groupified (b) β = 10, Original (c) β = 10, Groupified (d) β = 10, Original

(e) β = 20, Groupified (f) β = 20, Original (g) β = 20, Groupified (h) β = 20, Original

(i) β = 30, Groupified (j) β = 30, Original (k) β = 30, Groupified (l) β = 30, Original

Figure 27: The representation space span by the Groupified and Original β-VAE. We train the
models with the same hyperparameter but different random seeds for different runs. The 3D location
of each point is the disentangled representation of the corresponding image. Moreover, an ideal result
is that all the points form a cube, and color variation is continuous. Higher hyper-parameter β results
in the collapse of representation space. The collapse is suppressed by the Isomorphism Loss, which
leads to better disentanglement.


-----

(a) C = 10, Groupified (b) C = 10, Original (c) C = 10, Groupified (d) C = 10, Original

(e) C = 20, Groupified (f) C = 20, Original (g) C = 20, Groupified (h) C = 20, Original

(i) C = 30, Groupified (j) C = 30, Original (k) C = 30, Groupified (l) C = 30, Original

Figure 28: The representation space span by the Groupified and Original AnnealVAE. We train the
models with the same hyperparameter but different random seeds for different runs. The 3D location
of each point is the disentangled representation of the corresponding image. Moreover, an ideal result
is that all the points form a cube, and color variation is continuous. Higher hyper-parameter C results
in the collapse of representation space. The collapse is suppressed by the Isomorphism Loss, which
leads to better disentanglement.


-----

(a) γ = 5, Groupified (b) γ = 5, Original (c) γ = 5, Groupified (d) γ = 5, Original

(e) γ = 10, Groupified (f) γ = 10, Original (g) γ = 10, Groupified (h) γ = 10, Original

(i) γ = 15, Groupified (j) γ = 15, Original (k) γ = 15, Groupified (l) γ = 15, Original

Figure 29: The representation space span by the Groupified and Original FactorVAE. We train the
models with the same hyperparameter but different random seeds for different runs. The 3D location
of each point is the disentangled representation of the corresponding image. Moreover, an ideal result
is that all the points form a cube, and color variation is continuous. The collapse is suppressed by the
Isomorphism Loss, which leads to better disentanglement.


-----

(a) β = 6, Groupified (b) β = 6, Original (c) β = 6, Groupified (d) β = 6, Original

(e) β = 9, Groupified (f) β = 9, Original (g) β = 9, Groupified (h) β = 9, Original

(i) β = 12, Groupified (j) β = 12, Original (k) β = 12, Groupified (l) β = 12, Original

Figure 30: The representation space span by the Groupified and Original β-TCVAE. We train the
models with the same hyperparameter but different random seeds for different runs. The 3D location
of each point is the disentangled representation of the corresponding image. Moreover, an ideal result
is that all the points form a cube, and color variation is continuous. Higher hyper-parameter C results
in the collapse of representation space. The collapse is suppressed by the Isomorphism Loss, which
leads to better disentanglement.


-----

