# LEARNING TO PERCEIVE OBJECTS BY PREDICTION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

The representation of objects is the building block of higher-level concepts. Infants develop the notion of objects without supervision. The prediction error of
future sensory input is likely the major teaching signal for infants. Inspired by this,
we propose a new framework to extract object-centric representation from single
2D images by learning to predict future scenes in the presence of moving objects.
We treat objects as latent causes whose function to an agent is to facilitate efficient
prediction of the coherent motion of their parts in visual input. Distinct from previous object-centric models, our model learns to explicitly infer objects’ locations
in 3D environment in addition to segmenting objects. Further, the network learns
a latent code space where objects with the same geometric shape and texture/color
frequently group together. The model requires no supervision or pre-training of
any part of the network. We provide a new synthetic dataset with more complex
textures on objects and background and found several previous models not based
on predictive learning overly rely on clustering colors and lose specificity in object segmentation. Our work demonstrates a new approach for learning symbolic
representation grounded in sensation and action.

1 INTRODUCTION

Visual scenes are composed of various objects in front of backgrounds. Discovering objects from
2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin et al.,
2018; Wang et al., 2019) and this can potentially provide better abstraction of the environment for
reinforcement learning (RL), e.g. Veerapaneni et al. (2020). The appearance and spatial arrangement
of objects, together with the lighting and the viewing angle, determine the 2D images formed on the
retina or a camera. Therefore, objects are latent causes of 2D images, and discovering object is a
process of inferring latent causes (Kersten et al., 2004). The predominant approach in computer
vision for identifying and localizing objects rely on supervised learning to infer bounding boxes
(Ren et al., 2015; Redmon et al., 2016) or pixel-level segmentation of objects (Chen et al., 2017).
However, the supervised approach requires expensive human labeling. It is also difficult to label
every possible category of objects. Therefore, an increasing interest has developed recently to build
unsupervised or self-supervised models to infer objects from images, such as MONet (Burgess et al.,
2019), IODINE (Greff et al., 2019) slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al.,
2019; 2021), C-SWM (Kipf et al., 2019) and mulMON (Nanbo et al., 2020). Our work also focuses
on the same unsupervised object-centric representation learning (OCRL) problem, but offers a new
learning objective and architecture to overcome the limitation of existing works in segmenting more
complex scenes and explicitly represents objects’ 3D locations.

The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects
of pure colors and background lacking complex textures. As recently pointed out, the success of
several recent models based on a variational auto-encoder (VAE) architecture (Kingma & Welling,
2013; Rezende et al., 2014) depends on a reconstruction bottleneck that needs to be intricately balanced (Engelcke et al., 2020). To evaluate how such models perform on scenes with more complex
surface textures, we created a new dataset of indoor scenes with diverse texture patterns on the objects and background. We found that several existing unsupervised OCRL models overly rely on
clustering pixels based on their colors. A challenge in our dataset and in real-world perception is
that sharp boundaries between different colors exist both at contours of objects and within the surface of the same object. A model essentially has to implicitly learn a prior knowledge of which types
of boundaries are more likely to be real object contours. The reconstruction loss in existing works
appears to be insufficient for learning this prior.


-----

To tackle this challenge, we draw our inspiration from development psychology and neuroscience.
Infants understand the concept of object as early as 8 months, before they can associate objects
with names (Piaget & Cook, 1952; Flavell, 1963). The fact that infants are surprised when object
are hidden indicates that they have already learned to segment discrete objects from the scene and
that their brains constantly make prediction and check for deviation from expected outcome. As
the brain lacks direct external supervision for object segmentation, the most likely learning signal
is from the error of this prediction. In the brain, a copy of the motor command (efference copy) is
sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate
the prediction of changes in sensory input due to self-generated motion (Feinberg, 1978). What
remains to be predicted are changes in visual input due to the motion of external objects. Therefore,
we believe that the functional purpose of grouping pixels into object is to allow the prediction of
the motion of the constituting pixels in a coherent way by tracking very few parameters (e.g., the
location, pose, and speed of an object). Driven by this hypothesis, our contribution in this paper is:
(1) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware objectcentric representation, which we call Object Perception by Predictive LEarning (OPPLE); (2) we
provide a new dataset[1] with complex surface texture and motion by both the camera and objects to
evaluate object-centric representation models; (3) we found several previous models overly rely on
clustering colors to segment objects; (4) although our model leverages image prediction as learning
objective, the architecture generalize the ability of object segmentation and spatial localization to
single-frame images.

2 METHOD

2.1 PROBLEM FORMULATION

We denote a scene as a set of distinct objects and a background S = {O1, O2, . . ., OK, B}, where K
is the number of objects in scene. At any moment t, we denote two state variables, the location and
pose of each object relative to the perspective of an observer (camera), as x[(]1:[t][)]K [and][ φ][(]1:[t][)]K[, where][ x][(]k[t][)]
is the 3-d coordinate of the k-th object and φ[(]k[t][)] is its yaw angle from a canonical pose, as viewed
from the reference frame of the camera (for simplicity, we do not consider pitch and roll here and
leave it for future work to extend to 3D pose). At time t, given the location of the camera o[(][t][)] _∈_ R[3]

and its facing direction α[(][t][)], S renders a 2D image on the camera as I [(][t][)] _∈_ R[w][×][h][×][3], where w × h
is the size of the image. Our goal is to train a neural network that infers properties of objects given
only a single image I [(][t][)] as the sole input without external supervision and with only the information
of the intrinsics and egomotion of the camera without explicit knowledge of its extrinsic matrix:

_{z1:[(][t]K[)]_ _[,][ π]1:[(][t]K[)]_ +1[,][ ˆ]x[(]1:[t][)]K[,][ p][(]φ[t]1:[)] _K_ _[}][ =][ f][obj][(][I]_ [(][t][)][)] (1)

Here, z1:[(][t]K[)] [is a set of view-invariant vectors representing the identity of each object][ k][. “View-]
invariant” is loosely defined as **_zk[(][t][)]_** **_zk[(][t][+∆][t][)]_** _<_ **_zk[(][t][)]_** **_zl[(][t][)]_**
cases, i.e., the vector codes are more similar for the same object across views than they are different | _−_ _|_ _|_ _−_ _[|][ for][ k][ ̸][=][ l][ and][ ∆][t >][ 0][ in most]_
across objects. π1:[(][t]K[)] +1
objects or the background ([∈] [R][P][(][K]k[+1)][π][kij][×][ = 1][w][×][h][ are the probabilities that each pixel belongs to any of the][ for any pixel at][ i, j][), which achieve object segmentation.]

To localize objects, ˆx[(]1:[t][)]K [are the estimated locations of each object relative to the observer and][ p]φ[(][t]1:[)] _K_
are estimated probability distribution of the poses of each object. Each p[(]φ[t]k[)]
distribution over b equally-spaced bins of yaw angles in (0, 2π). _[∈]_ [R][b][ is a probability]

2.2 OVERALL PRINCIPLE: LEARNING OBJECT REPRESENTATION BY PREDICTING THE
FUTURE

Our hypothesis is that the notion of object emerges to meet the need of efficiently predicting the
future fates of all parts of an object. With the (to be learned) ability to infer an object’s pose and
location from each frame, the object’s speed of translation and rotation can be estimated from consecutive frames. If depth is further inferred for each pixel belonging to an object, then the optical

1We will release upon publication of the paper


-----

flow of each pixel can be predicted based on the object’s speed and the position of each pixel relative
to the object’s center. The pixel-segmentation of an object essentially prescribes which pixels should
move together with the object. With the predicted optical flow, one can further predict part of the
next image by warping the current image. The parts of the next image unpredictable by warping
include surfaces of objects or the background that are currently occluded but will become visible,
and the region of a scene newly entering the view due to self- or object-motion. These portions can
only be predicted based on the learned statistics of the appearance of objects and background, which
we call ”imagination”. In this work, we will show that with the information of self-motion, knowledge of geometry (rule of rigid-body movement) and the assumption of smooth object movement,
the object representations captured by function fobj and depth perception can be learned without
supervision.

2.3 NETWORK ARCHITECTURE

To demonstrate the hypothesized principle above, we build our OPPLE networks as illustrated in
Figure 1, which process two consecutive images individually and make prediction for the next image
with the information extracted from them.

t-1 networks

...
3

Object info from t-1: 21

Depth perception

_h_ Object-based

t prediction by

warping

Object extraction

_f_ Final

prediction

Segmented

self-motion objects: ...321 Identity matching, motion estimationobjects Combination weight

object-based

Imagination imagination

_g_

t+1

Ground truth
Learning from prediction error


Figure 1: Architecture for the Object Perception by Predictive LEarning (OPPLE) network
Images at each time point are processed by the Object Extraction and Depth Perception Networks
independently. All networks use U-Net structure. The dotted boxes indicate the entire OPPLE
network acting at each time step (details omitted for t − 1). Motion information of each object is
estimated from the spatial information extracted for each object between t−1 and t. Objects between
frames are soft-matched by a score depending on the distance between their latent codes. Self- and
object motion information are used together with object segmentation and depth map to predict the
next image by warping the current image. The segmented object images and depth, together with
their motion information and the observer’s motion, are used by the imagination network to imagine
the next scene and fill the gap not predictable by warping. The error between the final combined
prediction and the ground truth of the next image provides teaching signals for all three networks.
**Object extraction network. We build an object extraction network fθobj by modificaiton of a U-Net**
(Ronneberger et al., 2015) to extract representation for each object, the pixels it occupies and its
spatial location and pose. A basic U-Net is composed of a convolutional encoder and a transposed
convolutional decoder, while each encoder layer sends a skip connection to the corresponding decoder layer, so that the decoder can combine both global and local information. Inside our fθobj,
an image I [(][t][+1)] first passes through the encoder. Additional Atrous spatial pyramid pooling layer
(Chen et al., 2017) is inserted between the middle two convolutional layers of the encoder to expand
the receptive field. The top layer of the encoder outputs a feature vector e[t] capturing the global information of the scene. A Long-Short Term Memory (LSTM) network further repeatedly reads in e[(][t][)]


-----

and sequentially outputs one code for an object at a time. Each object code is then mapped through
a one-layer fully connected network to predict object code zk[(][t][)][, object location][ ˆ]x[(]k[t][)] [and object pose]
probability pφ(kt) [,][ k][ = 1][,][ 2][,][ · · ·][, K][. The inferred location is restricted to the visible range of the]
camera with an upper limit of distance. The pose prediction is represented as log pφ(kt) [for numerical]

stability. Each object code vector zk[(][t][)] is then independently fed through the decoder with shared
skip connection from the encoder. The decoder outputs one channel for each pixel, representing an
un-normalized log likelihood that the pixel belongs to the object k. The unnormalized logit maps
for all objects are concatenated with a map of all zero for the background, and compete through a
softmax function to output the probabilistic segmentation map πk[(][t][)][.]

**Depth perception network. We use a standard U-Net for depth perception function hθdepth that**
processes images I [(][t][)] and output a single-channel depth map D[(][t][)].

**Object-based imagination network. We build the imagination network gθimag also with a modified**
U-Net. The input is concatenated image I [(][t][)] and log of depth log(D[(][t][)]) inferred by the depth
perception network, both multiplied element-wise by one probabilistic mask πk[(][t][)][. The output of the]
encoder network is concatenated with a vector composed of the observer’s moving velocity vobs[(][t][)] [and]
rotational speed ωobs[(][t][)][, and the estimated object location][ ˆ]x[(]k[t][)][, velocity][ ˆ]vk[(][t][)] and rotational speed ˆωk[(][t][)]
before entering the decoder. The decoder outputs five channels for each pixel: three for predicting
RGB colors, one for depth and one for the probability of the pixel belonging to any object k or
background and is used to weight the predicted color and depth for the final ”imagination”.

2.4 LEARNING OBJECT REPRESENTATION BY PREDICTION

Below, we explain details of the approach. The pseudocode of the algorithm is provided in the
appendix.

2.4.1 PREDICTION BY WARPING

We first describe the prediction of part of the next image by warping the current image. Here we
consider only rigid objects and the fates of all visible pixels belonging to an object. With depth
**_D[(][t][)]_** _∈_ R[w][×][h] = hθ(I [(][t][)]) of all pixels in a view inferred by the Depth Perception network based
on visual features in the image I [(][t][)], the 3D location of a pixel ˆm[(]([t]i,j[)] ) [at any coordinate][ (][i, j][)][ in]
the image, can be determined given the focal length d of the camera. We use the inferred current
and previous locations, ˆx[(]k[t][)] and ˆxk[(][t][−][1)] to estimate the instantaneous velocity of the K _[th]_ object
**_vˆk[(][t][)]_** = ˆx[(]k[t][)] **_x[(]k[t][−][1)]_** for all pixels (i, j) attributed to that K _[th]_ object. Similarly, with the inferred the

_[−]_ [ˆ]
current and previous pose probabilities of the object, p[(]φ[t]k[)] [and][ p][(]φ[t]k[−][1)], we can obtain the likelihood

of its angular velocity p(φ[(]k[t][)][, φ]k[(][t][−][1)] _| ωk[(][t][)]_ = ω). We obtain the posterior distribution of the object’s
next pose p(φ[(]k[t][+1)] _| φ[(]k[t][)][, φ]k[(][t][−][1)]) with a Von Mises prior. More details about warping and pose_
estimation are added in the Appendix.

Assuming a pixel (i, j) belongs to object k, we use the estimated motion information ˆvk[(][t][)] and
_p(ωk[(][t][)]_ _| φ[(]k[t][)][, φ]k[(][t][−][1)]) of the object, together with the current location and pose of the object and the_
current 3D location ˆm[(]([t]i,j[)] ) [of the pixel, to predict the 3D location][ m][′][(]k,[t][+1)](i,j) [of the pixel at the next]

moment as m[′][(]k,[t][+1)](i,j) [=][ M]−[ (][t]ω[)] obs[[][M][ (]ωˆ[t]k[)][( ˆ]m[(]([t]i,j[)] ) **_x[(]k[t][)][)+ ˆ]x[(]k[t][)]_** [+ ˆ]vk[(][t][)] obs[]][, where][ M][ (]−[t]ω[)] obs [and][ M]ω[ (]ˆ[t]k[)]

_[−]_ [ˆ] _[−]_ **_[v][(][t][)]_**
are rotational matrices due to the rotation of the observer and the object, respectively, and vobs[(][t][)] [is the]
velocity of the observer (relative to its own reference frame at t). In this way, assuming objects move
smoothly most of the time, if the self motion information is known, the 3D location of each visible
pixel can be predicted. If a pixel belongs to the background, ωK+1 = 0 and vK+1 = 0 (K +1 is the
background’s index). Given the predicted 3D location, the target coordinate (i[′], j[′])[(]k[t][+1)] of the pixel
on the image and its new depth Dk[′] [(][i, j][)][(][t][+1)][ can be calculated. This prediction of pixel movement]
allows predicting the image I _[′][(][t][+1)]_ and depth D[′][(][t][+1)] by weighting the colors and depth of pixels
predicted to land near each pixel at the discrete grid of the next frame.


-----

Considering that object attribution of each pixel has to be inferred, we can write object attribution as
a probability of belonging to each object, across all pixels, πk[(][t][)][,][ k][ = 1][,][ 2][,][ · · ·][, K][ +1][. The predicted]
motion of each pixel should be described as a probability distribution over K + 1 discrete target
locations p(x[′][(]([t]i,j[+1)]) [) =][ P]k[K]=1[+1] _[π]kij[(][t][)]_ _[·]_ _[δ][(][x][′][(]k,[t][+1)](i,j)[)][, i.e., pixel][ (][i, j][)][ has a probability of][ π]kij[(][t][)]_ [to move to]

location x[′][(]k,[t][+1)](i,j)[. Using probabilistic prediction of pixel movements, we partially predict the color]
of next image at the pixel grids where some original pixels from the current will land by weighting
their contribution I _[′][(]Warp[t][+1)][(][p, q][)][. This contribution term is used to calculate a weight for effect of a]_
pixel (i,j) on position (p,q) on the grid, and is written as wk(i, j, p, q). More details in the appendix.

2.4.2 IMAGINATION

For the regions not fully predictable by warping current image with appendix equation (7), i.e.,
for (p, q) where _k,i,j_ _[w][k][(][i, j, p, q][)][ <][ 1][, we learn a function][ g][ that “imagines” the appearance]_

**_I_** _[′][(]k[t]Imag[+1)]_ _kImag_
the next frame, and the predicted probabilities that each pixel in the next frame belongs to each[∈] [R][w][×][h][×][P][3][ and the pixel-wise depth][ D][′][(][t][+1)] _[∈]_ [R][w][×][h][ of the object or background][ k][ in]
object or the background π[′][(]k[t]Imag[+1)]

_[∈]_ [R][w][×][h][. The function takes as input portion of the current]

image corresponding to each object I [(][t][)] _⊙_ **_πk[(][t][)]_** and the inferred depth D[(][t][)] _⊙_ **_πk[(][t][)][, both extracted]_**
by element-wise multiplying with the probabilistic segmentation mask πk[(][t][)][, the information of the]
camera’s self motion, and the location and motion of that object:

_{I_ _[′][(]k[t]Imag[+1)][,][ D][′]k[(][t]Imag[+1)][,][ π][′]k[(][t]Imag[+1)][}][ =][ g][(][I]i[(][t][)]_ _⊙_ **_πk[(][t][)][,][ D][(][t][)][ ⊙]_** **_[π]k[(][t][)][,][ ˆ]x[(]k[t][)][,][ ˆ]vk[(][t][)][,][ ˆ]ωk[(][t][)][, ω]obs[(][t][)][)]_** (2)


The “imagination” specific for each object and the background can then be merged using the weights
prescribed by π[′][(]1:[t][+1)]KImag[:][ I] _[′]Imag[(][t][+1)]_ = _k_ **_[I]_** _[′][(]k[t]Imag[+1)]_ _[⊙]_ **_[π][′][(]k[t]Imag[+1)][, and][ D][′][(]Imag[t][+1)]_** = _k_ **_[D][′][(]k[t]Imag[+1)]_** _[⊙]_ **_[π][′]k[(][t]Imag[+1)][.]_**

2.4.3 COMBINING WARPING AND IMAGINATION[P] [P]

The final predicted image or depth map are weighted average of the prediction made by warping the
current image or predicted depth map and the corresponding predictions by imagination:

**_I_** _[′][(][t][+1)]_ = I _[′][(]Warp[t][+1)]_ [+][ I] _[′][(]Imag[t][+1)]_ (3)

_[⊙]_ **_[W][Warp]_** _[⊙]_ [(1][ −] **_[W][Warp][)]_**

Here, WWarp ∈ R[w][×][h] with each element WWarp(p, q) = max{[P]k,i,j _[w][(][i, j, p, q][)][,][ 1][}][. The intuition]_

is that imagination is only needed when there is not sufficient contribution for predicting a pixel by
warping. The same weighting applies for generating the final predicted depth D[′][(][t][+1)].

In addition to the image, the states of the objects can also be predicted. The location of object k at
_t + 1 can be predicted as x[′][(]k[t][+1)]_ = M [(][t]ω[)] obs[(ˆ]x[(]k[t][)] + ˆvk[(][t][)] **_vobs[(][t][)][)][. Its new pose probability can be]_**
_−_ _−_
predicted by p[′][(][t][+1)](φk + ωobs = γ2) = _p(ˆωk[(][t][)]_ = ω)p(φ[(]k[t][)] = γ1) for γ2 equal to

_ωγ1,ω,γ2π,ω,ω2−γ+21∈π_
_{_ _−_ P _}_

each fixed yaw angle bin. To obtain p[′][(][t][+1)](φk) instead of p[′][(][t][+1)](φk + _ωobs) at the same set of bins,_
the vector p[′][(][t][+1)](φk + _ωobs) can be shifted by multiplication with a pre-computed matrix composed_
of a bank of shifted Von Mises distributions to “move” the probability mass on the angle bins.

There is one important issue of object-centric representation when making prediction for future
images: in order to predict the spatial state of each object at t + 1 based on the views at t and
_t −_ 1, the network needs to match the representation of an object at t from the representation of the
same object at t − 1. As the dimensions of features (e.g., shape, surface texture, size, etc.) grows,
the number of possible objects grows exponentially. Therefore, we cannot simply match object
representations based on the order by which an LSTM extracts objects, as this requires learning a
consistent order over enormous amount of objects. Instead, we take a soft-matching approach: we
take a subset of the features in zk[(][t][)] extracted by f as an identity code for each object. For object
_k at time t, we calculate the distance between its identity code and those of all objects at t −_ 1,
and pass the distances through a radial basis function to serve as a matching score rkl indicating
how closely the object k matches each of the previous objects. The scores are used weight all the


-----

estimated translational and rotational speeds for object k each assuming a different object l were the
true object k at t − 1. We additionally introduce a fixed identity code zK+1 = 0 for the background
and set the predicted motion of background to zero.

2.4.4 LEARNING OBJECTIVE

Above, we have explained how the next image input I _[′][(][t][+1)], the depth map D[′][(][t][+1)]_ and the spatial
states of each object, x[′][(]k[t][+1)] and p[′][(]φ[t]k[+1)] can be predicted based on object-centric representation

extracted by a function f from the current and previous images Ii[(][t][)] and Ii[(][t][−][1)], the depth D[′][(][t][)]

extracted by a function h, combined with the prediction from object-based imagination function g
that are all to be learned. Among the three prediction targets, only the ground truth of visual input
**_I_** [(][t][+1)] is available, while the other can only be inferred by f and h from I [(][t][+1)]. Therefore, for
the prediction targets other than I [(][t][+1)], we use the self-consistent loss between the predicted value
based on t and t − 1 and the inferred value based on t + 1 as additional regularization terms to learn
the functions f and g.

To learn the functions f, g and h, we approximate them with deep neural networks with parameters
_θ and optimize θ to minimize the following loss function:_

_L = Limage + λdepthLdepth + λspatialLspatial + λmapLmap_ (4)

Here, _Limage_ = MSE(I _[′][(][t][+1)], I_ [(][t][+1]) is the image prediction error. _Ldepth_ =
MSE(log(D[′][(][t][+1)]), log( D[ˆ] [(][t][+1)])) is the error between the predicted and inferred depth. These provide major teaching signals. Lspatial = _k=1_ _[|][ P]l[K]=1[+1]_ _[r][kl][x][′][(]l[t][+1)]_ _−_ **_xˆ[(]k[t][+1)]|[2]_** _−_ [P]k[K]=1 [min][{|]x[ˆ][(]rand[t][+1)] _−_
**_xself-consistent loss on spatial information prediction. The first term is the error between inferredˆ[(]k[t][+1)]|, δ} +_** _k=1_ _[|]x[ˆ][(]k[t][+1)]_ _−_ [P]i,j **_m[ˆ][P][(]i,j[t][K][+1)]πkij|[2]_** + _k=1_ _[D][KL][(ˆ]p[(]φ[t]k[+1)]||_ _l=1_ _[r][kl][p][′][(]φ[t]l[+1)]) is the_

and predicted location of each object, while the calculation of the predicted location incorporates

[P][K] [P][K] [P][K][+1]

soft matching between objects in consecutive frames. The second term is the negative term of
contrastive loss, which we found empirically prevents the network from reaching a local minimum
where all objects are inferred at the same location relative to the camera (and covering minimal
regions of the picture). **_xˆrand is the inferred object location from a random sample within the_**
same batch. The third term penalizes the discrepancy between the inferred object location and the
average location of pixels in its segmentation mask. The last term is the KL-divergence between the
predicted and inferred pose for each object atavoids loss of gradient due to zero probability of object belonging and discourages overlap between t +1. Lmap = ReLu(10[−][4] _−_ **_π1:K)+_** **_πk ·_** **_πl, for k ̸= l_**
maps of different objects.

2.5 DATASET

We procedurally generated a dataset composed of 306445 triplets of images captured by a virtual
camera with field of view of 90 degrees in a square room. The camera translates horizontally and
pans with random small steps between consecutive frames to facilitate the learning of depth perception. 3 objects with random shape, size, surface color or textures are spawned at random locations in
the room and each move with a randomly selected constant velocity and panning speed. The translation and panning of the the camera is known to the networks. No other ground truth information
is provided. The first two frames serve as data and the last frame serve as the prediction target at
_t + 1. An important difference between this dataset and other commonly used synthetic datasets for_
OCRL is that more complex and diverse textures are used on both the objects and the background.
We further evaluated on a richer version of the Traffic dataset (Henderson & Lampert, 2020) in A.6.

2.6 COMPARISON WITH OTHER WORKS

To compare our work with the states-of-the-art models of unsupervised object-centric representation learning, we trained MONet (Burgess et al., 2019), slot-attention (Locatello et al., 2020) and
GENESIS[2] (Engelcke et al., 2021) on the same dataset. Although these models are trained on single
images, all images of each triplets are used for training.

2We failed to obtain reasonable result by training GENESIS V2 on our dataset, thus we adopted a GENESIS
network pre-trained on GQN dataset and retrained on our dataset with K=7.


-----

To address the concern that the original configurations of the models are not optimized for more difficult dataset, we trained variants of some of the models
with large network size. For MONet, we tested channel numbers of [32, 64, 128, 128] (MONet-128) and

[32, 64, 128, 256, 256] (MONet-128-bigger) for the
hidden layers of encoder of the component VAE instead of [32, 32, 64, 64] and adjusted decoder layers
sizes accordingly, and increased the base channel from
64 to 128 for the attention network. For slot attention,
we tested a variant which increased the number of features in the attention component from 64 to 128 (slotattention-128). Slot numbers were chosen as 4 except
for GENESIS.

3 RESULTS


|Model|ARI-fg IoU|
|---|---|
|MONet MONet-128 MONet-128-bigger slot-attention slot-attention-128 GENESIS our model (OPPLE)|0.31 0.08 0.33 0.22 0.33 0.15 0.41 0.31 0.39 0.54 0.17 0.03 0.46 0.35|


Model ARI-fg IoU

MONet 0.31 0.08

MONet-128 0.33 0.22

MONet-128-bigger 0.33 0.15

slot-attention 0.41 0.31

slot-attention-128 0.39 **0.54**

GENESIS 0.17 0.03

our model (OPPLE) **0.46** 0.35

Table 1: Performance of different models
on object segmentation.


After training the networks, we evaluate them on 4000 test images unused during training but generated randomly with the same procedure, thus coming from the same distribution. We compare
the performance of different models mainly on their segmentation performance. Additionally, we
demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth
of the scene. The performance of depth perception is illustrated in the appendix.

3.1 OBJECT SEGMENTATION

Following prior works (Greff et al., 2019; Engelcke et al., 2019; 2021), we evaluated segmentation
with the Adjusted Rand Index of foreground objects (ARI). In addition, for each image, we matched
ground-true objects and background with each of the segmented class by ranking their Intersection
over Union (IoU) and quantified the average IoU over all foreground objects[3]. The performance is
summarized in table 2.6 .

Our model outperforms all compared models on ARI and is
second to a slot-attention-128 Original images
in IoU. As shown in Figure 2,
MONet and GENESIS appear to Ground truth
heavily rely on color to group segmentation
pixels into the same masks.
Even though some of these mod- MONet-128
els almost fully designate pixels of an object to a mask, the Slot-attentionmasks lacks specificity in that 128
they often include pixels with

GENESIS*

similar colors from other objects
or background. Patterns on the

Our model

backgrounds are often treated as (OPPLE)
objects as well. The reason
of such drawbacks awaits fur- Figure 2: Example of object segmentation by different models
ther investigation but we postulate there may be fundamental limitation in the approach that learns purely from static discrete images. Patches in the background with coherent color offer room to compress information similarly
as objects with coherent colors do, and their shapes re-occur across images just as other objects.
Our model is able to learn object-specific masks because these masks are used to predict optical
flow specific to each object. A wrong segmentation would generate large prediction error even if
the motion of an object is estimated correctly. Such prediction error forces the masks to be concentrated on object surface. They emerge first at object boundaries where the prediction error is the
largest and gradually grow inwards during training. Figure 3A-D further compares the distribution

3Due to the artifact at the border of slot-attention model, IoU was calculated excluding those border pixels


-----

A Our model B MONet-128 C slot-attention-128 D GENESIS

(OPPLE)

E

F 0 IoU 1 G H

observation
null distribution


Figure 3: A-D: distribution of IoU. All models have IoU < 0.01 for about 1/4 of objects. Only
OPPLE shows a bi-modal distribution while other models’ IoU are more skewed towards 0. E-F:
object localization accuracy of OPPLE for object’s polar angle and distance relative to the camera.
Each dot is a valid object with color representing its segmentation IoU. Angle estimation is highly
accurate for well segmented objects (red dots). Distance is under-estimated for farther objects. G:
objects with failed segmentation (blue dots) are mostly far away and occupying few pixels. H The
numbers of objects sharing the same shape or texture with their nearest neighbour objects in latent
space are significantly above chance.

of IoU across models. Most models have IoU < 0.01 for about 1/4 of objects. Only OPPLE and
slot-attention-128 show bi-modal distributions while other models’ IoU are more skewed towards
0. Figure 3G plots each object’s distance and size on the picture with colors corresponding to their
IoUs in our model. Objects with poor segmentation (blue dots) are mostly far away from the camera
and occupy few pixels. This is reasonably because motion of farther objects causes less shift on the
images and thus provide weaker teaching signal for the network to assign their pixels as separate
from the background. For other models, blue dots are more spread even for near objects (not shown).

3.2 OBJECT LOCALIZATION

The Object Extraction Network infers object location relative to the camera. We convert the inferred
locations to angles and distance in polar coordinate relative to the camera. Figure 3E-F plot the
true and inferred angles and distance, color coded by objects’ IoUs. For objects well segmented
(red dots), their angles are estimated high accurately (concentrated on the diagonal in E). Distance
estimation is negatively biased for farther objects, potentially because the regularization term on the
distance between the predicted and inferred object location at frame t + 1 favors shorter distance
when estimation is noisy. Note that the ability to explicitly infer object’s location is not available in
other models compared.

3.3 MEANINGFUL LATENT CODE

Because a subset of the latent code (10 dimensions) was used to calculate object matching scores
between frames in order to soft-match objects, this should force the object embedding z to be similar
for the same objects. We explored the geometry of the latent code by examining whether the nearest
neighbours of each of the object in the test data with IoU > 0.5 are more likely to have the same
property as themselves. 772 out of 3244 objects’ nearest neighbour had the same shape (out of
11 shapes) and 660 objects’ nearest neighbour had the same color or texture (out of 15). These
numbers are 28 to 29 times the standard deviation away from the means of the distribution expected
if the nearest neighbour were random (Figure 3H). This suggests the latent code reflects meaningful
features of objects. However, texture and shape are not the only factors determining latent code, as
we found the variance of code of all objects with the same shape and texture to still be big.

4 RELATED WORK

Our work is on the same tracks as two recent trends in machine learning: object-centric representation (Locatello et al., 2020) and self-supervised learning (Chen et al., 2020). We take the same


-----

logic as self-supervised learning that learning to predict part of the data based on another part forces
a neural network to learn important structures in the data. However, most of the existing works in
self-supervised learning do not focus on object-based representation, but instead encode the entire
scene as one vector. Other works on object-centric representations overcome this by assigning one
representation to each object, as we do. Although works such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019), slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019)
and PSGNet (Bear et al., 2020) can also segment objects and some of them can ”imagine” complete objects based on codes extracted from occluded objects or draw objects in the correct order
consistent with occlusion, few works explicitly infer an object’s location in 3D space together with
segmentation purely by self-supervised learning, with the exception of a closely related work O3V
(Henderson & Lampert, 2020). Both our works learn from videos. One major distinction is that
O3V interleaves spatial and temporal convolution, thus it still require video as input at test time. In
contrast, our three major networks process each image independently. Therefore, once trained, our
network can generalize to single images. Another distinction from Henderson & Lampert (2020) and
many other works is that our model learns from prediction instead of reconstruction. Contrastivelearning of structured world model (Kipf et al., 2019) also learns object masks and predict their
future states by linking each object mask with a node in a Graphic Neural Network (GNN). The
order of mapping object slot to nodes of GNN is fixed through time, and the actions to objects are
coded with specific associated nodes. This arrangement may become infeasible with combinatorial
number of different possible objects as the order of assigning different objects to a limited number
of nodes may not be consistent across scenes. We solve this by a soft matching of object representation between different time points, which does not require the RNN in the Object Extraction
Network to learn a fixed order of extracting different types of objects. On the neuroscience side, our
work is highly motivated by recent works on predictive learning (O’Reilly et al., 2021) which also
yields view-invariance representation while self-motion signal is available. O’Reilly et al. (2021)
used biologically plausible but less efficient learning and applied their model to an easier dataset
with objects without background, and did not learn object localization. We should note that explicit
spatial localization and depth perception were not pursued in previous works on self-supervised
object-centric learning, and the images in our dataset have significantly richer texture information
than those demonstrated in previous works (Burgess et al., 2019; Kipf et al., 2019), making the task
more challenging. Although view synthesis is not our central goal, the principle illustrated here can
be combined with recent advancement in 3D-aware image synthesis (Wiles et al., 2020).

5 DISCUSSION

We provide a new approach to learn object-centric representation that includes explicit spatial localization of objects, object segmentation from image, automatic matching the same objects across
scene based on a learned latent code and depth perception as a by-product. All of the information
extracted by our networks are learned without supervision and no pre-training on other tasks is involved. The only additional information required is that of observer’s self-motion, which is available
in the brain as efference copy. This demonstrate the possibility of learning rich embodied information of object, one step toward linking neural networks with symbolic representation in general. We
expect future works to develop self-supervised learning model for natural categories beyond simple
object identity, building on our work.

The work demonstrates that the notion of object can emerge as a necessary common latent cause of
the pixels belonging to the object for the purpose of efficiently explaining away the pixels’ coherent
movement across frames. In our experiment, object spatial location is inferred more easily than
object pose (which we have not fully investigated), thus the predicted warping relies more on object
translation than rotation. As a limitation, almost all existing object-centric representation works,
including ours, focus on rigid bodies and simple environment. Future works need to explore how to
learn object representation for deformable objects, objects with more complex shapes and lighting
conditions, and more cluttered environment, towards more realistic application. There is important
implication for learning 3D-aware object-based representation, because for such representation to be
useful eventually for robotics and RL, agents need to understand an object’s spatial relation to itself
based on vision. Learning object-centric representation with explicit 3D information is an important
step toward embodied representation of the environment.


-----

REFERENCES

Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy
Schwartz, Li Fei-Fei, Jiajun Wu, Joshua B Tenenbaum, et al. Learning physical graph representations from visual scenes. arXiv preprint arXiv:2006.12373, 2020.

Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt
Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and
fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):
834–848, 2017.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. In Hal Daum´e III and Aarti Singh (eds.), Proceed_ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of_
_Machine Learning Research, pp. 1597–1607. PMLR, 13–18 Jul 2020._

Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representations for generalizable robot learning. In 2018 IEEE International Conference on Robotics and
_Automation (ICRA), pp. 7111–7118. IEEE, 2018._

Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. _arXiv preprint_
_arXiv:1907.13052, 2019._

Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Reconstruction bottlenecks in objectcentric generative models. arXiv preprint arXiv:2007.06245, 2020.

Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object
representations without iterative refinement. arXiv preprint arXiv:2104.09958, 2021.

Irwin Feinberg. Efference copy and corollary discharge: implications for thinking and its disorders.
_Schizophrenia bulletin, 4(4):636, 1978._

John H Flavell. The developmental psychology of jean piaget. 1963.

Klaus Greff, Rapha¨el Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel
Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation
learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov
(eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of
_Proceedings of Machine Learning Research, pp. 2424–2433. PMLR, 09–15 Jun 2019._

Paul Henderson and Christoph H Lampert. Unsupervised object-centric video generation and decomposition in 3d. arXiv preprint arXiv:2007.06705, 2020.

Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. Annu.
_Rev. Psychol., 55:271–304, 2004._

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
_arXiv:1412.6980, 2014._

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. _arXiv preprint_
_arXiv:1312.6114, 2013._

Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models.
_arXiv preprint arXiv:1911.12247, 2019._

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold,
Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055, 2020.


-----

Li Nanbo, Cian Eastwood, and Robert B Fisher. Learning object-centric representations of multiobject scenes from multiple views. In 34th Conference on Neural Information Processing Sys_tems, 2020._

Randall C O’Reilly, Jacob L Russin, Maryam Zolfaghar, and John Rohrlich. Deep predictive learning in neocortex and pulvinar. Journal of Cognitive Neuroscience, 33(6):1158–1196, 2021.

Jean Piaget and Margaret Trans Cook. The origins of intelligence in children. 1952.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified,
real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern
_recognition, pp. 779–788, 2016._

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object
detection with region proposal networks. Advances in neural information processing systems, 28:
91–99, 2015.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning,
pp. 1278–1286. PMLR, 2014.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer_assisted intervention, pp. 234–241. Springer, 2015._

Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu,
Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement
learning. In Conference on Robot Learning, pp. 1439–1456. PMLR, 2020.

Dequan Wang, Coline Devin, Qi-Zhi Cai, Fisher Yu, and Trevor Darrell. Deep object-centric policies
for autonomous driving. In 2019 International Conference on Robotics and Automation (ICRA),
pp. 8853–8859. IEEE, 2019.

Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view
synthesis from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision
_and Pattern Recognition, pp. 7467–7477, 2020._


-----

A APPENDIX

A.1 PSEUDO CODE OF THE OPPLE FRAMEWORK

**Algorithm 1 Developing object-centric representation by predicting future scene**
InitInitialize: Network parameters θ

**Input: images I** [(][t][−][1)], I [(][t][)] R[w][×][h][×][3], self-motion vobs[(][t][−][1)], ωobs[(][t][−][1)], vobs[(][t][)][, ω]obs[(][t][)]
_∈_

**Output: prediction I** _[′][(][t][+1)], segmentation π1:[(][t]K[−][1)]+1[,][ π]1:[(][t]K[)]_ +1[, objects’ codes][ z]1:[(][t]K[−][1)][,][ z]1:[(][t]K[)] [, objects’]

locations and poses ˆx[(]1:[t][−]K[1)][,][ p]φ[(][t]1:[−]K[1)][,][ ˆ]x[(]1:[t][)]K[,][ p][(]φ[t]1:[)] _K_

**for τ = {t −** 1, t} do

scene code e[(][τ] [)] U-NetEncoderfθ (I [(][τ] [)])
_←_
object code z1:[(][τ]K[)] _[,][ location][ ˆ]x[(]1:[τ]K[)]_ _[,][ pose][ p][(]φ[τ]1:[)]K_ _θ_ [(][e][(][τ] [)][)]

background code zK+1 = 0 _[←]_ [LSTM][f]
depth D[(][τ] [)] _hθ(I_ [(][τ] [)])
_←_

**end forsegmentation mask π1:[(][τ]K[)]** +1 _[←]_ [Softmax (U-NetDecoder]fθ [(][I] [(][τ] [)][,][ z]1:[(][τ]K[)] [)][,][ 0)]
object matching scoresfor k ← 1 to K do _rkl ←_ RBF(zk[(][t][)][,][ z]l[(][t][−][1)]), k, l ∈ 1 : K + 1

object motion ˆv1:K, ω1:K ← _rk,l, ˆx[(]k[t][)][,][ ˆ]x[(]l[t][−][1)], p[(]φ[t]k[)][,][ p][(]φ[t]l[−][1)], l = 1 : K + 1_

onject-specific optical flowk **_vˆ1:K, ω1:K, vobs, ωobs, D[(][t][)]_**

**end for** _←_
**_I_** _[′]warp[(][t][+1)]_ Warp(I [(][t][)], optical flow1:K+1)
_←_

**_I_** _[′]imagine[(][t][+1)]_ 1:K+1[, log][(][D][(][t][)][)][ ⊙] **_[π]1:[(][t]K[)]_** +1[,][ v][obs][, ω][obs][,][ ˆ]v1:K+1, ˆx1:K)

_[←]_ _[g][θ][(][I]_ [(][t][)][ ⊙] **_[π][(][t][)]_**

final image prediction: I _[′][(][t][+1)]_ _←_ **_I_** _[′][(]warp[t][+1)][,][ I]_ _[′][(]imagine[t][+1)]_ _[,][ warping weights]_

update parameters: θ _θ_ _γ_ _θ[_ **_I_** _[′][(][t][+1)]_ **_I_** [(][t][+1)] + regularization loss]
_←_ _−_ _∇_ _|_ _−_ _|[2]_


A.2 PERFORMANCE ON DEPTH PERCEPTION

We demonstrate a few example images and the inferred depth. Our network can capture the global
3D structure of the scene, although details on object surfaces are still missing. Because background
occurs in every training sample, the network appears to bias the depth estimation on objects towards
the depth of the walls behind, as is also shown in the scatter plot.

Images

Inferred depth

Ground truth depth Inferred depth

Ground truth depth


Figure 4: Comparison between ground truth depth and inferred depth


-----

A.3 NETWORK TRAINING AND DATASET

We trained the three networks jointly using ADAM optimization Kingma & Ba (2014) with a learning rate of 3e _−_ 4, ϵ = 1e _−_ 6 and other default setting in PyTorch, with a batch size of 24. 40 epochs
were trained on the dataset. We set λspatial = 1.0, λdepth = 0.1 and Lmap = 0.005 Images in datasets
are rendered in Unity environment and downsampled to 128 × 128 resolution for training. Images
are rendered in sequence of 7 steps each time in a room with newly selected texture and object.
Camera moves with random steps and random rotation between consecutive frames. All possible
triplets equally spaced by 1, 2, and 3 frames form training samples.

The model was implemented in PyTorch and trained on NVidia RTX 6000. We will release the code
and dataset upon publication of the manuscript.

A.4 METHOD

A.4.1 PREDICTION BY WARPING

We first describe the prediction of part of the next image by warping the current image. Here we
consider only rigid objects and the fates of all visible pixels belonging to an object. With depth
**_D[(][t][)]_** _∈_ R[w][×][h] = hθ(I [(][t][)]) of all pixels in a view inferred by the Depth Perception network based
on visual features in the image I [(][t][)], the 3D location of a pixel at any coordinate (i, j) in the
image, where |i| ≤ _[w][−]2_ [1] [,][|][j][| ≤] _[h][−]2_ [1] [, can be determined given the focal length][ d][ of the camera as]

**_mˆ_** [(]([t]i,j[)] ) [=] _√Di[2][(]+[t][)]j(i,j[2]+)d[2][ ·][ [][i, d, j][]][. Here, we take the coordinate of the center of an image as (0,0). On]_

the other hand, with the inferred ˆx[(]k[t][)] and ˆx[(]k[t][−][1)], the current and previous locations of the object k
that the pixel (i, j) belongs to, from I [(][t][)] and I [(][t][−][1)] respectively, we can estimate the instantaneous
velocity of the object ˆvk[(][t][)] = ˆx[(]k[t][)] **_xˆ[(]k[t][−][1)]. Similarly, with the inferred the current and previous_**
_−_
pose probabilities of the object, p[(]φ[t]k[)] [and][ p][(]φ[t]k[−][1)], we can obtain the likelihood of its angular velocity


_p(φ[(]k[t][)][, φ]k[(][t][−][1)]_ _| ωk[(][t][)]_ = ω) ∝


_p(φ[(]k[t][)]_ = γ1) _p(φ[(]k[t][−][1)]_ = γ2) (5)
_·_


_{ωγ1−,γ2π,ω,ω2,γ1−+2γ2∈π}_


By additionally imposing a prior distribution (we use Von Mises distribution) over ωk[(][t][)] that
favors slow rotation, we can obtain the posterior distribution of the object’s angular velocity
_p(ωk[(][t][)]_ _| φ[(]k[t][)][, φ]k[(][t][−][1)]), and eventually the posterior distribution of the object’s next pose p(φ[(]k[t][+1)]_ _|_
_φ[(]k[t][)][, φ]k[(][t][−][1)])._

Assuming a pixel (i, j) belongs to object k, using the estimated motion information ˆvk[(][t][)] and p(ωk[(][t][)]
_|_
_φ[(]k[t][)][, φ]k[(][t][−][1)]) of the object, together with the current location and pose of the object and the current_
3D location ˆm[(]([t]i,j[)] ) [of the pixel, we can predict the 3D location][ m][′][(]k,[t][+1)](i,j) [of the pixel at the next]
moment as

**_m[′][(]k,[t][+1)](i,j)_** [=][ M]−[ (][t]ω[)] obs[[][M][ (]ωˆ[t]k[)][( ˆ]m[(]([t]i,j[)] ) _[−]_ **_x[ˆ][(]k[t][)][) + ˆ]x[(]k[t][)]_** [+ ˆ]vk[(][t][)] _−_ **_vobs[(][t][)][]]_** (6)

where M−[(][t]ω[)] obs [and][ M]ω[ (]ˆ[t]k[)] [are rotational matrices due to the rotation of the observer and the object,]

respectively, and vobs[(][t][)] [is the velocity of the observer (relative to its own reference frame at][ t][). In this]
way, assuming objects move smoothly most of the time, if the self motion information is known, the
3D location of each visible pixel can be predicted. If a pixel belongs to the background, ωK+1 = 0
and vK+1 = 0 (K + 1 is the background’s index). Given the predicted 3D location, the target coordinate (i[′], j[′])[(]k[t][+1)] of the pixel on the image and its new depth Dk[′] [(][i, j][)][(][t][+1)][ can be calculated. This]
prediction of pixel movement allows predicting the image I _[′][(][t][+1)]_ and depth D[′][(][t][+1)] by weighting
the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame,
as explained in Sec 2.4.2.


-----

A.4.2 WARPING CONTRIBUTION WEIGHT

As the object attribution of each pixel is not known but is inferred by fobj(I [(][t][)]), it is represented for
every pixel as a probability of belonging to each object and the background πk[(][t][)][,][ k][ = 1][,][ 2][,][ · · ·][, K][ +]
1. Therefore, the predicted motion of each pixel should be described as a probability distribution
over K + 1 discrete target locations p(x[′][(]([t]i,j[+1)]) [) =][ P]k[K]=1[+1] _[π]kij[(][t][)]_ _[·][ δ][(][x][′][(]k,[t][+1)](i,j)[)][, i.e., pixel][ (][i, j][)][ has a]_

probability of πkij[(][t][)] [to move to location][ x][′][(]k,[t][+1)](i,j) [at the next time point, for][ k][ = 1][,][ 2][,][ · · ·][, K][ +1][. With]

such probabilistic prediction of pixel movement for all visible pixel (i, j)[(][t][)], we can partially predict
the colors of the next image at the pixel grids where some original pixels from the current view will
land nearby by weighting their contribution:

_k,i,j_ _[w][k][(][i,j,p,q][)][I]_ [(][t][)][(][i,j][)]

**_I_** _[′][(]Warp[t][+1)][(][p, q][) =]_ P _k,i,j_ _[w][k][(][i,j,p,q][)]_ _,_ if _k,i,j_ _[w][k][(][i, j, p, q][)][ >][ 0]_ (7)

(0, P otherwise

[P]

We define the weight of the contribution from any source pixel (i, j) to a target pixel (p, q) as

_wk(i, j, p, q) = πkij[(][t][)]_ _k_ (i,j) max 1 _i′[(]k[t][+1)]_ _p_ _, 0_ max 1 _j[′][(]k[t][+1)]_ _q_ _, 0_ (8)

_[·][ e][−][β][·][D][′]_ [(][t][+1)] _·_ _{_ _−|_ _−_ _|_ _} ·_ _{_ _−|_ _−_ _|_ _}_
The first term incorporates the uncertainty of which object a pixel belongs to. The second term
_e[−][β][·][D][′]_ _k[(][t][+1)](i,j) resolves the issue of occlusion when multiple pixels are predicted to move close_
to the same pixel grid by down-weighting the pixels predicted to land farther from the camera.
These last two terms mean that only the source pixels predicted to land within a square of of 2 × 2
pixels centered at any target location (p, q) will contribute to the color I _[′][(]Warp[t][+1)][(][p, q][)][. The depth]_

map D[′][(]Warp[t][+1)] can be predicted by the same weighting scheme after replacing I [(][t][)](i, j) with each

predicted depth D[′][(]k[t][+1)](i, j) assuming the pixel belongs to object k.

A.5 DEPENDENCY OF SEGMENTATION PERFORMANCE ON OBJECT SIZE AND DISTANCE
ACROSS MODELS

Our model (OPPLE) MONet-128

1

Slot-attention-128 Genesis *

0

IoU

A.6 ILLUSTRATION OF PREDICTION QUALITY

In the figure below, we display the first, second image, one of the masked objects, the predicted third
image, and the ground truth of the third image, both for our dataset, and for a richer version of the
Traffic dataset used by (Henderson & Lampert, 2020). We provide reference lines and some circles
to aid the comparison between images and evaluate the warping quality. Imagination quality can be
insprected usually at one side of the predicted images (the camera motion is typically larger in our
dataset).


-----

-----

