# SUCCESSIVE POI RECOMMENDATION VIA BRAIN## INSPIRED SPATIOTEMPORAL AWARE REPRESENTA- TION

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Existing approaches usually perform spatiotemporal representation in the spatial
and temporal dimensions, respectively, which isolates the spatial and temporal natures of the target and leads to sub-optimal embeddings. Neuroscience research
has shown that the mammalian brain entorhinal-hippocampal system provides efficient graph representations for general knowledge. Moreover, entorhinal grid
cells present concise spatial representations, while hippocampal place cells represent perception conjunctions effectively. Thus, the entorhinal-hippocampal system provides a novel angle for spatiotemporal representation, which inspires us to
propose the SpatioTemporal aware Embedding framework (STE) and apply it to
POIs (STEP). STEP considers two types of POI-specific representations: sequential representation and spatiotemporal conjunctive representation, learned using
sparse unlabeled data based on the proposed graph-building policies. Notably,
STEP jointly represents the spatiotemporal natures of POIs using both observations and contextual information from integrated spatiotemporal dimensions by
constructing a spatiotemporal context graph. Furthermore, we introduce a user privacy secure successive POI recommendation method using STEP, and it achieves
the state-of-the-art performance on two benchmarks. In addition, we demonstrate
the excellent performance of the STE representation approach in other spatiotemporal representation-centered tasks through a case study of traffic flow prediction
problem. Therefore, this work provides a novel solution to spatiotemporal aware
representation and paves a new way for spatiotemporal modeling-related tasks.

1 INTRODUCTION

With the rapid growth of location-based web services like Instagram and Yelp, there has been a
seismic shift in how people interact with locations around them. Through exploitation of Points-ofInterest (POIs) and their contexts, successive POI recommendation can benefit users and businesses
greatly. As a core of POI information utilization, encoding POIs into vector representation space
is of great significance for advanced POI analysis and downstream applications. Existing studies
attempt to represent POI from different perspectives and collaborate with user preference modeling
to achieve recommendation. Since consecutive check-ins are usually highly correlated, naturally,
sequence modeling approaches like the Markov chain model were used to capture the check-in sequential characteristics of POIs (Ye et al., 2011; Liu et al., 2013; Zhang, 2014; Feng et al., 2015).
Employing tensor factorization technique, the works (Yang et al., 2017; Wang et al., 2018) modeled
target users and POIs separately by interacted features for POI recommendation. More recently,
enlightened by neural networks’ success, recurrent neural nets were remolded to represent POIs
and user preferences implicitly (Liu et al., 2016a; Zhao et al., 2019; Zhu et al., 2017). Considering
the geographical attributes of POIs, researchers have used power-law distribution, Gaussian distribution, or hierarchical tiling methods to depict the geographical influence over POI distributional
features (Ye et al., 2011; Lian et al., 2014; Feng et al., 2017; Chang & Kim, 2020; Luo et al., 2020).
However, geographical modeling methods above only provide single-scale or coarse-grained manually designed representations of POI geographical influences, which is deficient in capturing the
POI-specific spatial features. Also, the arbitrary modeling might even lead to over-parameterization.
While temporal dimension offers indeterminate auxiliary information for POI modeling, to utilize


-----

the POI temporal information within the check-ins, some works using time interval, time state variables or temporal transition vectors to promote the POI representing (Zhao et al., 2019; 2016; 2017;
Li et al., 2018; Manotumruksa et al., 2018; Zhao et al., 2020). However, these methods focused
on utilizing general temporal patterns among all POIs and failed to exploit the POI-specific visiting
time patterns sufficiently. Still, the POI-specific spatiotemporal characteristics were not adequately
mined and utilized.

**Inspirations. The entorhinal-hippocampal system plays a central role in the mammal cognition**
architecture. The Nobel Prize-winning neuroscience research (O’keefe & Nadel, 1978) demonstrated that entorhinal grid cells provide an effective multi-scale periodic spatial representation (Yuan
et al., 2015; Banino et al., 2018; Mai et al., 2020; Dang et al., 2021). Moreover, the entorhinalhippocampal system is also critical for the non-spatial inference that relies on understanding the
associations between perceptions from various perspectives (Whittington et al., 2018; Stachenfeld
et al., 2018; Whittington et al., 2020). Some promising researches cast spatial and non-spatial problems as connected graphs and point out the cells inside entorhinal-hippocampal structure provide
efficient conjunctive representation for those graphs (Stachenfeld et al., 2018; Gustafson & Daw,
2011). As the representation mechanism in the entorhinal-hippocampal system was extensively
studied, it is widely accepted that conjunctions of representations from different aspects form the
hippocampal representation for relational memory (Whittington et al., 2018; 2020; Eichenbaum,
2017; MacDonald et al., 2011; Sargolini et al., 2006). For the general spatiotemporal aware embedding, various contexts can be constructed into affinity graphs for latent representation learning.
Furthermore, strategies like conjunctive-representing in entorhinal-hippocampal structure can be
translated to improve the quality of the representations (see Fig.1 left part).

In this paper, borrowing inspirations from the entorhinal-hippocampal system, we propose the SpatioTemporal aware Embedding framework, namely STE and apply to POIs (STEP) for successive
POI recommendation, the model architectures are shown in Figure 1. Firstly, we build context graphs
to enable unsupervised embedding learning on sparse check-ins. Secondly, we employ a sequential
model to represent POIs from the check-in sequence perspective. Most importantly, we elaborate a
spatiotemporal model consists of a grid-cell spatial encoder and a visiting time encoder to capture
the POI-specific spatiotemporal characteristics. The spatiotemporal model learns to get the POI spatiotemporal latent representations using the spatiotemporal context graph. Finally, we implement
successive POI recommendation systems based on the STEP and achieve high-performance using
simple recurrent neural networks as recommenders.

This work’s main contributions are summarized as follows:

(1) Motivated by the graph-representing strategy of structural knowledge in the entorhinalhippocampal system, we solve the spatiotemporal aware embedding learning in a graph-based unsupervised learning manner through specific context graph building policies, especially the spatiotemporal context graph, to fully exploit rich unlabeled data.

(2) Inspired by the conjunctive representation mechanism in the entorhinal-hippocampal complex,
we present a spatiotemporal model with a grid-cell spatial encoder and a time pattern encoder to
utilize the spatiotemporal information. The conjunctive representing approach based on a unique
spatiotemporal context graph addresses the problem of previous spatiotemporal modeling methods
in which spatial and temporal information are isolated and represented separately.

(3) We introduce a successive POI recommendation system by incorporating STEP and simple sequence predictors to show the feasibility of implementing specific applications based on the proposed STE framework. We perform experiments on large real-world datasets to demonstrate the
effectiveness of STEP, and our method outperforms baselines according to experimental results.

(4) Moreover, compared with classical recommendation systems, our POI-centered solution can
avoid the ethical risks of artificial intelligence, like personal data leakage, as it does not need access to private information such as user preferences. Furthermore, our framework can be applied
to more valuable applications like wildlife preservation and urban traffic scheduling as a general
spatiotemporal aware modeling method.

2 PRELIMINARIES


-----

E-H system:

ɠ _Graph-representing_


Grid cells within E-H:

ɡ Self-position representing


Place cells within E-H:

ɢ Conjunctive-representing

_Observations & Contexts_

_Spatial & Temporal_


**Entorhinal-Hippocampal**

**Cognitive System**



Contextual Info. ɠ Context Graph Building
Specific Info.

Context Graphs Visiting Locations

timestamps **_t_** (x, y)

Sequential Model

Time Pattern Encoder Grid-cell Encoder

ɡSpatial Model

Merge

ɢSpatiotemporal Model


**SpatioTemporal aware Embedding framework**


Figure 1: Representing mechanisms in entorhinal-hippocampal system (E-H system for short) and
the framework of spatiotemporal aware embedding model. The proposed STE framework consists of
three essential parts: the context graph building strategies to construct simplified affinity graphs, the
spatiotemporal model to extract rich item-specific spatiotemporal features, and the sequential model
to extract sequential feature embeddings. The uniqueness about our spatiotemporal information
usage is that we represent item from spatiotemporal perspective (not isolated) using observations
and contexts conjunctively.


Table 1: Notations in this paper.

Notation Definition

_t[i]j_ _j-th timestamp of pi_
**_t[i]_** Visiting time pattern matrix of pi
**_e[i]spa_** Spatial vector representation of pi
**_e[i]seq_** Sequential vector representation of pi
**_e[i]st_** Spatiotemporal conjunctive representation of pi
**_e[i]step_** STEP vector representation of pi
_⊕_ Tensor concatenation operation


Given a set of POIs with corresponding coordinates P = {pi}, pi = (xi, yi), a checkin sequence is one set of continuous checkins of one user in one day, denoted as Sj =
_{(p1, t[1]1[)][, ...,][ (][p][n][, t]m[n]_ [)][}][. Unlike previous works,]
we do not regard all check-in records of a user
as one sequence since check-ins with relatively
long intervals are not very relevant. Although
we assign notation to users for generality, the
user information is not used in the training
phase except to split sequences.


We define context graphs as graphs that encode context information as affinity among POIs. Various
contexts in the check-in records can be easily built into graphs Gp = {Vp, Ep}, where Vp is the
set of POIs and Ep is the set of edges between adjacent POIs. The edges in context graphs represent the correlation between neighboring POIs defined by geographical distance, relative position
in check-in sequences, or spatiotemporal adjacent criterions. We summarize notations in this paper
using Table 1.

**Data description. The Instagram Check-in dataset (Chang et al., 2018) was collected from Insta-**
gram in New York and the data was preprocessed in the same manner as previous works (Zhao et al.,
2016; 2017). The Instagram Check-in dataset has been pre-processed when it is made public, it includes 2,216,631 check-in records at 13,187 POIs of 78,233 users. Check-in sequences are sorted by
timestamps, the first 70% are used as a training set and the remaining 30% for validation and testing.
The Gowalla dataset is a globally-collected large-scale social media dataset (Cho et al., 2011). We
eliminate users with fewer than ten check-ins and POIs accessed by fewer than ten users. Then the
check-in records are sorted according to timestamps and first 70% check-ins are used for training
and the remaining latest records for testing. We perform vivid data analyses in the Appendix due to
the space constraints.

3 SPATIOTEMPORAL AWARE EMBEDDING MODEL OF POIS

We illustrate components of STEP: the sequential model in Sec. 3.1, the spatiotemporal model
in Sec. 3.2, and state the STEP-based successive POI recommendation method in Sec. 3.3. We
adopt a simple-minded (no-parameters) edge weighting policy for constructing all context graphs.
Weight Ai,j = 1 if vertices i and j is connected, this simplification avoids the necessity of choosing
edge-weighting parameters.


-----

3.1 SEQUENTIAL MODEL

The sequential sub-model represents POIs using context graph Gseq. Given one POI and its context
in the check-in sequence, entry Ai,j in adjacency matrix of Gseq is 1 if pi, pj are within the same
context window. This is a common way to mine the sequential correlations of tokens like words
(Mikolov et al., 2013) or POIs (Lim et al., 2020). Our sequential model aims to predict true contextual POIs, i.e., connected vertices in Gseq. Intuitively, minimizing the objective function over
all target-neighbor pairs guarantees that POIs sharing similar sequential context will have shorter
distances in embedding space (Hadsell et al., 2006). To avoid the intractable summation over the
whole context space, we follow the noise contrastive sampling approach (Gutmann & Hyv¨arinen,
2012; Mikolov et al., 2013) to get an approximated surrogate loss


I(γ = 1) log σ(e[i]seq _seq[) +][ I][(][γ][ =][ −][1) log][ σ][(][−][e][i]seq_ _seq[)]_ _,_ (1)

_[·][ e][j]_ _[·][ e][j]_



_seq(θseq) =_
_L_ _−_


_pi,pj_ _∈P_


where γ = 1 if (pi, pj) is a sequential neighboring pair and γ = −1 if not, indicator I outputs 1
when the argument condition is true and otherwise 0. This unsupervised loss can also be seen as
taking expectation with respect to the distribution P(pi, pj, γ) over P, which is conditioned on the
POI sequential context graph Gseq.

3.2 SPATIOTEMPORAL MODEL

In this section, we illustrate the POI spatiotemporal conjunctive embedding model in detail. The
proposed spatiotemporal model is composed of two key components: a POI spatial model and a POI
visiting time encoder, an intuitive illustration can be found in Fig. 1.

3.2.1 SPATIAL MODEL

The spatial sub-model takes location observations (xi, yi) and spatial context graph Gspa to produce
spatial representations.

**Grid-cell encoding. Inspired by the multi-scale periodic representation of grid cells in mammals,**
we formulate our POI spatial contextual encoder to use sinusoidal and cosinusoidal functions of
different scales to encode the raw locations of POIs in geographical space following previous works
Gao et al. (2019); Mai et al. (2020). Given a POI pi = (xi, yi) ∈ R[2], the grid-cell model based
encoder encodes the coordinates in 2-D Euclidean space into spatial latent representations in R[d][spa] .
We denote the grid cell encoder based spatial embedding of POI pi as

**_e[i]spa_** [=][ φ][(][ψ][(][x][i][, y][i][);][ θ][spa][)][,] (2)

where
_ψ(xi, yi) = ψ[1](xi, yi)_ _ψ[s](xi, yi)_ _ψ[S](xi, yi)_ (3)
_· · · ⊕_ _· · · ⊕_
is concatenated multi-scale representations of 6S dimensions, S denotes the number of grid scales
**_aand2= φ[− represents fully connected non-linear layers.1/2,_** _√3/2][T], a3=[−1/2, −√3/2][T]_ _∈_ R[2], at each frequency, position codesConsidering three unit vectors a1=[1, 0][T],

_ψ[s](xi, yi) = ψ1[s]_ 2 3 (4)

_[⊕]_ _[ψ][s]_ _[⊕]_ _[ψ][s]_

are computed via

_ψk[s][(][x][i][, y][i][) =]_ cos( [[][x][i][, y][i][]][ ·][ a][k] ), sin( [[][x][i][, y][i][]][ ·][ a][k] ) _, k_ 1, 2, 3 (5)

_ρλmin_ _ρλmin_ _∈{_ _}_

 


and ρ = (λmax/λmin)[s/][(][S][−][1)]. λmin and λmax are the minimum and maximum scale values, here
we use S = 64 following the previous work (Mai et al., 2020) and set λmax = 1km, λmin = 0.1km.

**Spatial-neighboring definition. We project the coordinates in the geographical coordinate system**
WGS84 to the projection coordinate system NAD27 to get locations of POIs in R[2]. For each entry Ai,j in adjacency matrix of spatial context graph Gspa, we assign Ai,j using the geographical
distances. Specifically, we computed the geographical distances between POIs and construct an
undirected spatial context graph Gspa with uniform edges among the top-ten closest POIs (nearest
neighbors policy). As the grid cell encoder can handle geographical distributions at different scales


-----

(Mai et al., 2020), we do not use a specific radius (ϵ-neighborhoods policy) to filter the neighboring
POIs to fully exploit the multi-scale representation capability. The spatial graph construction process is related to Lim et al. (2020), in which the edges are weighted according to average distances
to enable graph attention computations.

Given a target POI pi, neighboring contextual POI set Pspa[+] [and negative set][ P]spa[−] [sampled from]
_Gspa, the unsupervised embedding learning can simply be maximizing the log-likelihood of ob-_
serving the true context POIs. We can formulate this target with negative sampling via a general
objective function:

(ctx) = log σ(e[j]ctx _ctx[) + 1]_ log σ( **_e[k]ctx_** _ctx[)]_ _,_ (6)
_O_ _−_ _pXi∈P_ _pjX∈Pctx[+]_  _[·][ e][i]_ _K_ _pkX∈Pctx[−]_ _−_ _[·][ e][i]_ 

 

where ctx indicates the context graph type and ctx ∈{seq, spa, st} in this work, σ is the sigmoid
function and K denotes the number of samples in negative sample set Pctx[−] [. Following Eq.6, the]
loss function for the spatial context embedding model is:

_spa(θspa) =_ (spa). (7)
_L_ _O_


3.2.2 SPATIOTEMPORAL CONTEXT GRAPH CONSTRUCTION

For constructing spatiotemporal context graph Gst, we want to
mine the item-specific spatiotemporal conjunctions, so for each
entry Ai,j of the adjacency matrix of Gst, we assign Ai,j = 1
following the hierarchy of neighboring timestamps → _temporal_
_neighboring →_ _spatiotemporal neghboring:_

1. Neighboring timestamps. Given an arbitrary timestamp pair
(wkdt1, t(2t)2, time interval) where wkd ( ∆t) = 1t ≜ if|t t1 is weekend else 0. For one time − _t2| and ∆wkd ≜_ _|wkd_ (t1) − t,
_|_
its temporal-neighboring timestamps are those within the neighborhood window and satisfy ∆wkd = 0, as shown in Fig. 2. h is
a hyper-parameter indicates temporal context window width and
`h ∈` (0, 24) hours.

2. **POI temporal neighboring.** POI pi and pj with corresponding visiting timestamp sets _i =_ _t[i]1[, t][i]2[, . . .][ }][ and][ T][j]_ [=]
_T_ _{_
_{t[j]1[, t][j]2[, . . .][ }][. The number of neighboring timestamp pairs][ (][t][i][, t][j][)]_
_> m, where t[i]_ _i, t[j]_ _j, m is a threshold._
_∈T_ _∈T_

3. POI spatiotemporal neighboring. If pi, pj are spatial and
temporal neighboring, they are spatiotemporal neighboring.

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|(7)|
|---|---|---|---|---|---|---|---|---|
|00:00|||||||||
|02:00|||||t2-24-h|t1-h|0||
|04:00|||||t1-24|t 1|kd ≠||
|06:00|||||t2-24+h|t1+h|Δw||
|08:00|||||||||
|10:00|||||||||
|12:00|||||||||
|14:00|||||||||
|16:00||t|2-24-h|t2-h|t2+24-h||||
|18:00|||t2-24|t 2|t2+24||||
|20:00||t|2-24+h|t2+h|t2+24+h||||
|22:00|||||||||
||Sun. Mon. Tues W ed. Thur. Fri. Sat. Sun.||||||||


Inter-day Dimension

Figure 2: Temporal neighboring examples with h = 2.
For t1, some times are excluded from the temporal neighborhood window as ∆wkd ̸= 0.


It is redundant for individual POI-specific temporal modeling since solely relying on time information, we are not able to recommend reasonable candidate POIs (visits may take place contemporarily
all over the world). Thus, time is regarded as a supplementary dimension of basic spatial infor
temporal and geographical information.


mation, our spatiotemporal context graph provides an effective way to combine the POI-specific

Timestamps High

**04-01-19-04**

NCODING **06-14-17-3809-22-12-4511-20-02-21** Accumulation

**…**

217 on the scale of year but fickle on the hour and date scale. In our accessing time pattern encodingLow Access Frequency

218 scheme, visits of POIs are counted by hour and date, after this procedure, every POI has a visiting

on the scale of year but fickle on the hour and date scale. In our accessing time pattern encoding219220 time pattern represented by a raw matrixare normalized and applied Gaussian kernel for smoothing, this process also augment the accessing t) to provide observa- **t[0]** 242 R[24][⇥][366]366 . The raw accessing time pattern matrices

2016a; Zhao et al., 2019; Zhu et al., 2017; Zhao et al., 2016), we fo-cus on the POI-specific temporal patterns rather than general temporal218219220 scheme, visits of POIs are counted by hour and date, after this procedure, every POI has a visitingtime pattern represented by a raw matrixare normalized and applied Gaussian kernel for smoothing, this process also augment the accessing221222223 time data in a reasonable way. The final matrix representationof day-night, weekday-weekend, and seasonal alternatives for further temporal context processing.Some typical accessing time patterns matrices are visualized in Figure 4 using heatmaps. t[0] _2 R[24][⇥][366]. The raw accessing time pattern matricesNormalization [0, 0.93, 0.13, ..., 0.05][0, 133, 19, ..., 7]_ **tAugmentation can preserve valuable information t[0]**
characteristics among all timestamps. Compared with previously used221 time data in a reasonable way. The final matrix representation t can preserve valuable information

222 of day-night, weekday-weekend, and seasonal alternatives for further temporal context processing.224 **3.4.4** **Spatial-Temporal Embedding Learning**

time interval-based or hard-coded methods, our encoding scheme can223 Some typical accessing time patterns matrices are visualized in Figure 4 using heatmaps.

of day-night, weekday-weekend, and seasonal alternatives for further temporal context processing.. The raw accessing time pattern matrices[0, 0.93, 0.13, ..., 0.05][0, 133, 19, ..., 7]

Some typical accessing time patterns matrices are visualized in Figure 4 using heatmaps.

**t can preserve valuable information**

tensorise the item-specific temporal information more precisely and able224 **3.4.4** **Spatial-Temporal Embedding Learning225226** To conduct the POIs’ spatial-temporal conjunctive representation learning, given POIsponding visiting time pattern matrix t[i], the spatial-temporal conjunctive representation is defined asFigure 3: Schematic of _pi and corre-_
to provide a reliable decision basis for spatiotemporal modeling.227

225 To conduct the POIs’ spatial-temporal conjunctive representation learning, given POIe[i]st [=][ φ][✓]stthe visiting time encod-[(][φ][✓]time [(][t][i][)][,][ e][i]spa[)][ ⊕] **[e][i]spa pi[,] and corre-** (9)

**Visiting time encoding. Since the visiting time patterns are relatively226227** sponding visiting time pattern matrix228229 wherebe well preserved while the visiting time patterns can be combined effectively. For a target POI φ indicates artificial neural network layers ande[i] **tφ[i], the spatial-temporal conjunctive representation is defined as(φ** (t[i]) e[i] ) ⊕ing process.e[i] **est 2 R[d][st]** . The basic spatial representation can(9) _pi, we_


-----

stable on the scale of year but fickle on the hour and the date scale, in our encoding scheme, visits
are counted by the check-in timestamps to raw matrices t[′] _∈_ R[24][×][366]. Then, the raw matrices t[′] are
normalized and applied Gaussian kernel for smoothing, this process also reasonably augment the
POI accessing time data. The visiting time encoding process is shown schematically in Fig. 3. The
final matrix representation t = smooth(norm(t[′])) retains the fine-grained check-in time patterns as
well as rough item-specific visiting time features.

3.2.4 SPATIOTEMPORAL CONJUNCTIVE EMBEDDING LEARNING

We formulate the spatiotemporal conjunctive representation as:

**_e[i]st_** [=][ φ][θ]st [(][φ][θ]time[(][t][i][)][,][ e][i]spa[)][ ⊕] **_[e][i]spa[,]_** (8)

where φ indicates fully-connected layers. Follow the formulation in Eq.6, we implement the spatiotemporal conjunctive representation learning by minimizing:

(θtime, θst) = (st). (9)
_L_ _O_

We sample Pst[+] [and][ P]st[−] [from][ G][st][, where][ P]st[+] [is the set of spatiotemporal-neighboring POIs whereas]
_Pst[−]_ [is the set of negative POIs.]

During the optimization procedure, the spatial model is jointly optimized as a sub-model of spatiotemporal model, the full objective of the spatiotemporal model is

_Lst = L(θtime, θst) + λspaLspa(θspa),_ (10)

_λspa is a weighting factor for preserving the spatial context information during the spatiotemporal_
modeling. We first sample a batch of spatial context Gspa to optimize the spatial context loss Lspa
to preserve geographical context. Next we sample a batch of spatiotemporal context Gst to optimize
the spatiotemporal loss Lst to preserve the spatiotemporal context. We repeat above procedures for
_I0 and I1 = I0/λspa iterations respectively to approximate the balancing factor λspa. We update_
all parameters _θspa, θtime, θst_ of spatiotemporal model in iterations until the overall loss _st_
_{_ _}_ _L_
converges.

3.3 SUCCESSIVE POI RECOMMENDATION WITH STEP

We present the STEP (STE of POIs)-based recommendation method in detail in this section. Taking
the spatiotemporal data as input, we construct context graphs G and feed the observations (locations and time patterns) into the STEP model to perform embedding learning. The embeddings
are smoothed according to corresponding context graphs to preserve contextual information. Then,
POI vector representations (embeddings) eseq, est are merged as spatiotemporal aware embedding
**_estep and fed into the recommender to generate an estimated embedding ˆe. Specifically, we use_**
concatenation
**_e[i]step_** [=][ e]seq[i] _[⊕]_ **_[e]st[i]_** (11)

to merge the two sequential and spatiotemporal embeddings. This merging policy can preserve
information from different spaces without extra-parameters and not requires the embeddings to be
in the same dimension (e.g. dst = dseq ) thus provides more flexibility. We adopt two-layer recurrent
networks as the recommender model.

**Embedding model optimizing. Parameters in the STEP embedding model are optimized according**
to corresponding loss function L(θ∗) = L∗ + α||θ∗||2, where L∗ _∈{Lseq, Lst}, θ∗_ _∈{θseq, Θst},_
Θst = {θspa, θtime, θst}. α is the weighting factor of the 2-norm regularizer.

**Recommender model optimizing. The predictor is then optimized with the pre-trained STEP em-**
bedding model for the next POI recommendation task. During the training phase, given a n-length
check-in sequence Sj, we can get corresponding STEP embedding series of POIs {e[(1)]step[, . . .,][ e][(]step[gt][)] _[}][,]_
the last POI is regarded as the recommendation target. The target of the recommender is to predict
latent representation ˆestep similar to the embedding of true successive POI e[(]step[gt][)] [, formally described]
as:
arg max `sim` **_eˆstep, e[(]step[gt][)]_** _._ (12)
_θpred_

_j_
_SX∈S_  


-----

Table 2: Comparisons with baselines on two datasets, we mark best values with bold fonts and
underline the suboptimal ones. CAPE-based methods are not applicable on Gowalla (no tweets were
provided) and we do not report results of some methods on Instagram Check-in as they cannot be
reproduced faithfully. †: with user preference consideration, ‡: using semantic content information.

DATASET Instagram Check-in Gowalla

METHOD\METRIC HIT@1 HIT@5 HIT@10 MRR HIT@1 HIT@5 HIT@10 MRR

Random+GRU 0.1197 0.2207 0.2726 0.1792 0.0715 0.0725 0.0732 0.0727
Random+LSTM 0.1207 0.2225 0.2751 0.1805 0.0722 0.0736 0.0749 0.0737
Skip-Gram+GRU 0.1356 0.2419 0.3040 0.1919 0.1090 0.2111 0.2617 0.1612
Skip-Gram+LSTM 0.1318 0.2344 0.2984 0.1875 0.1085 0.2101 0.2585 0.1594
CAPE+GRU‡ 0.1390 0.2433 0.3079 0.1953 _N/A_ _N/A_ _N/A_ _N/A_
CAPE+LSTM‡ 0.1381 0.2412 0.3054 0.1939 _N/A_ _N/A_ _N/A_ _N/A_
Geo+GRU 0.1619 0.2616 0.3248 0.2093 0.1267 0.2309 0.2834 0.1684
Geo+LSTM 0.1622 0.2594 0.3128 0.1875 0.1233 0.2296 0.2811 0.1701

ST-RNN† 0.1054 0.2019 0.2426 0.1681 0.0519 0.0953 0.1304 0.2187
STGN†STGCN†LSTPM† 0.1261−− 0.2134−− 0.3121−− 0.1957−− 0.02560.04240.1468 0.25060.07840.1134 0.29830.11440.1625 0.05900.08420.1998
STP-DGAT†STP-UDGAT† _−−_ _−−_ _−−_ _−−_ 0.13440.1475 0.24140.2911 0.26530.3285 0.18560.2130


STEP+RNN 0.2458 0.3170 0.3502 0.2822 0.1495 0.2878 0.3634 0.2222
STEP+GRU **0.2467** 0.3057 0.3336 0.2781 0.1490 0.2912 0.3636 0.2233

STEP+LSTM 0.2454 **0.3204** **0.3556** **0.2835** **0.1539** **0.2968** **0.3728** **0.2282**

The objective function of recommender is:

_pred(θpred) =_ log σ[′][ ]sim(e[(]step[gt][)] _[,][ ˆ]estep)_ log( _σ[′](sim(e[i]step[,][ ˆ]estep)))_ _,_ (13)
_L_ _−_  _−_ 

_j_ _pi_
_SX∈S_  X∈P

where σ[′] = exp(LeakyReLU ( )) and sim( _,_ ) = **aa·bb** 

_·_ _·_ _·_ _||_ _||·||_ _||_ [. During the testing phase, we compute]

the cosine similarity scores to rank the candidate POIs to generate recommendation lists.

Table 3: Context graphs statistics
4 EXPERIMENTS
on two POI datasets.



We perform the successive POI recommendation task on the
Instagram Check-in dataset, Gowalla dataset, and the traffic
flow forecasting task on TaxiBJ15 and TaxiBJ.

4.1 SUCCESSIVE POI RECOMMENDATION TASK

|AVERAGE VALUE|Ins.|Gow.|
|---|---|---|
|Records per POI E per POI seq E per POI spa E per POI st|168.1 66.2 10.0 0.997|34.2 35.5 10.0 0.596|


**Metrics. During the system inferencing phase, the recommendation system recommends a POI list**
according to the estimated scores of candidate POIs for every trial sequence. We apply widely-used
metrics HIT@K (if the ground truth is within the top-k of the list, a score of 1 is awarded, else
0), k = 1, 5, 10 and MRR (Mean Reciprocal Rank) for evaluation. These metrics reflect different
aspects of the recommendation lists, HIT@K measures the rate of valid recommendation among all
trials, whereas MRR scores the quality of the entire recommendation list.

**Hyper-parameter settings. We set the hyper-parameters of our proposed method to the following**
default values. We set context window size in the POI sequential model to 2 and adopt h = 2,
```
m = 11 for building Gst. We utilize Adam optimizer with batch size 512, β1 = 0.9, β2 = 0.999 and

```
set the initial learning rate to 0.001 followed by a reduce-on-plateau decay policy, the decay factor
is 0.1 during the training. Weighting factors α, λspa are set to 1 × 10[−][4], 0.2 and the embedding
dimensions _dseq, dspa, dst_ are set to 32, 64, 96 .
_{_ _}_ _{_ _}_

We compare the STEP-based successive POI recommendation method with representative methods:
**Embedding-recommender methods. We choose six methods consisting of three embedding mod-**
els and two recommenders. For the embedding model, we use the following embedding models.
(1) Random, (2) Skip-Gram (Liu et al., 2016b), (3) CAPE (Chang et al., 2018) and (4) Geo (Mai


-----

24.7

24.6

24.5

24.4

24.3

24.2

0.29

0.28

0.27

0.26


15.5

15.4

15.3

15.2

15.1

15.0

Instagram Gowalla

0.24

0.23

0.22

0.21


24.7

24.6

24.5

24.4

24.3

24.2

0.29

0.28

0.27

0.26


15.5

15.4

15.3

15.2

15.1

15.0

0.24

0.23

0.22

0.21


24.7

24.6

24.5

24.4

24.3

24.2

0.29

0.28

0.27

0.26


15.5

15.4

15.3

15.2

15.1

15.0

0.24

0.23

0.22

0.21

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Instagra|Col9|Col10|m|Col12|Col13|Col14|Gowa|Col16|lla|Col18|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|||||||||||||||||||
|||||||||||||||||||
|||||||||||||||||||
|||||||||||||||||||
|||||||||||||||||||
|||||||||||||||||||


Figure 4: Effect of hyper-parameters h, m, λspa on Instagram Check-in and Gowalla datasets. The
means and standard deviations are computed over five runs using different random seeds.

et al., 2020). For the recommender model, two-layer networks based on (1) GRU unit (Merri &
Fellow, 2014) and (2) LSTM unit (Hochreiter & Schmidhuber, 1997) are used. One-stage meth**ods. We choose representative one-stage methods as baselines. (1) ST-RNN (Liu et al., 2016a). (2)**
_STGN (Zhao et al., 2019), a LSTM variant which models visit preferences with time and distance_
considerations, and the improved variant STGCN. (3) LSTPM (Sun et al., 2020) is a LSTM-based
method. (4) STP-DGAT and STP-UDGAT (Lim et al., 2020) are spatial-temporal-preference user
dimensional graph attention networks.

**Comparison results. According to results in Table 5, our method outperforms the baselines by**
significant margins on both datasets, and the gains in recommendation accuracy are especially substantial on the Instagram dataset with rich temporal information (according to Table 3, POIs in
Gowalla have less visiting timestamps). The one-stage recurrent network-based methods, such as
_ST-RNN, LSTPM surpassing basic embedding-based methods by more sufficient exploitation of user-_
preference spatiotemporal properties. However, these methods remain noncompetitive with STEPbased ones, although the STEP stands without user preference consideration. The advantageous performance of our method over the competitors can be attributed to its efficient use of the item-specific
spatiotemporal nature. We observe significant performance improvements of the STEP-based methods in terms of MRR, indicating the STEP-based methods provide better candidates lists on both
datasets, benefit from the efficiency of the proposed spatiotemporal aware embedding model. The
usage of LSTM units in recommender slightly improves the performance compared with basic RNN
cells because of their advantages in gate functions of recurrent connections. Moreover, the basic
RNN recurrent net equipped with STEP embedding can also outperform one-stage SOTAs. This
also proves the effectiveness of our brain-inspired spatiotemporal aware embedding model.

**Effect of hyper-parameters. We study the effect of newly-introduced hyper-parameters h, m, λspa,**
and report the results using HIT@1 and MRR in Fig.4. h and m control the sparsity of the spatiotemporal context graph and λspa regulates the importance of the spatial context smoothness term in the
spatiotemporal model objective function. We alter h to build spatiotemporal context graphs with decreasing sparsity as larger h corresponds to coarse temporal-neighboring condition. The increasing
```
h within a certain range results in performance improvements but appears detrimental to the precise

```
top-1 recommendation. The best performance (determined by MRR) is obtained when h = 2. The
choice of h is task-related, according to our results on two dataset, h = 2 can be a good initial value
for POI recommendation according to our results. This value can be further adjusted for different
application scenarios or datasets to build spatiotemporal context graphs with desired sparsity. We set
```
m from 3 to 15, larger m leads to a sparser spatiotemporal context graph. We observe the performance

```
slightly changes after increasing the threshold m, when m = 11, the best performance is obtained.
We also investigate the effect of the balancing factor λspa for spatiotemporal model training, the
recommendation system achieves the best performance when λspa = 0.2 and further increases only
bring minor improvements, thus we select 0.2 as a default value in this work, this also helps reduce
unnecessary iterations during the model training.

**Ablation study. We study the effectiveness of STEP modules by performing successive POI rec-**
ommendation task with LSTM recommender, the method using standard STEP embedding model is
referred as FULL.
After the removal of (A) POI-specific time information processing module, the spatiotemporal
model degenerates to a spatial model. According to Table 4, the method performs worse without (A) on both datasets. According to Table 3, POIs in Gowalla have sparser specific observation ts and contextual information from Gst. This results in more significant performance


-----

**Query**

Chrysler Building


**Top 5 retrieved POIs**


Grand Central Oyster Bar

**midtown, famous seafood spot**

Metropolitan Oval

**bronx, park**


Grand Central Terminal

**midtown, landmarks & historical**
**buildings, train stations**

Bea

**midtown, cocktail bars**


Grand Hyatt New York

**midtown, hotels, venues &**
**event spaces**

Ann Loftus Playground

**upper manhattan, park**

The Campbell

**midtown, cocktail bars,**
**lounges**

Sushi Yasuda

**midtown, japanese food**


MTA 42nd St. Grand Central

**midtown, public transportation**

Roosevelt Island Sports Park

**roosevelt island, park**


Num Pang Kitchen

**midtown, sandwiches,**
**cambodian**

Unix Gallery

**upper east side, art gallery**


**midtown, landmarks &**
**historical buildings**


Grand Hyatt New York

**midtown, hotels, venues &**
**event spaces**

Luke’s Lobster

**midtown, seafood**


La Biblioteca

**midtown, night club, lounges**

Huntington Harbor

**huntington, harbor**


Tudor City Greens

**midtown, parks**

InterContinental N.Y. Barclay

**midtown, hotels**


Momosan Ramen&Sake

**midtown, ramen, tapas, small**
**plates**

Grand Central Oyster Bar

**midtown, famous seafood spot**


Figure 5: Retrieval results on Instagram dataset Top 5 candidates are placed from left to right.
Green and red flag marks represent positive and negative POIs determined by both opening hours
and geographical position. Under names of those POIs, we list their region and representative tags
collected from websites with bold fonts.

improvements on the Instagram set than on Table 4: Effectiveness of using (A) tempoGowalla. We replace POI visiting time pattern ma- ral information, (B) visiting time encoder,
trices t (applying Sec.3.2.3) with random-initialized (C) grid-cell encoder, (D) spatial embedding
matrices in R[24][×][366]. We note that the use of (B) preserving and (E) the spatiotemporal model.
POI visiting time encoder improves the recommen- We mark the best ones with bold fonts and
dation performance on both datasets, and the im- underline the suboptimal ones.
provement are positively correlated with the temporal information abundance of the dataset. We use a
one-layer neural network location encoder ψ[′](x, y) FULL 0.2454 0.3204 **0.3556 0.2835**
to replace (C) grid-cell encoder in the STEP to W/O (A) 0.2433 0.2544 0.2607 0.2504
demonstrate its effectiveness. Results in Table 4 W/O (B) 0.2452 0.3151 0.3442 0.2798
demonstrate that the grid-cell encoder improves the
quality of STEP representation and leads to better
successive POI recommendation performances on
both datasets. We observe noticeable performance FULL **0.1539 0.2968** **0.3728 0.2282**
drops after the removal of (D) spatial embedding W/O (A) 0.1461 0.2825 0.3540 0.2174
preserving in Table 4 since the spatiotemporal conjunctive representation integrate spatial and tempo- Gowalla
ral attributes at the cost of spatial information loss
(due to the dimensional reduction). The use of (D)

|underline the|e suboptimal ones. HIT@1 HIT@5 HIT@10 MRR|
|---|---|
||HIT@1 HIT@5 HIT@10 MRR|
|FULL W/O (A) Instagram W/O (B) W/O (C) W/O (D) only (E)|0.2454 0.3204 0.3556 0.2835 0.2433 0.2544 0.2607 0.2504 0.2452 0.3151 0.3442 0.2798 0.2399 0.3037 0.3305 0.2727 0.2298 0.2531 0.2674 0.2451 0.2466 0.2840 0.3021 0.2664|
|FULL W/O (A) Gowalla W/O (B) W/O (C) W/O (D) only (E)|0.1539 0.2968 0.3728 0.2282 0.1461 0.2825 0.3540 0.2174 0.1509 0.2921 0.3664 0.2248 0.1006 0.2169 0.2922 0.1657 0.0973 0.2117 0.2866 0.1610 0.1252 0.2142 0.2519 0.1683|

spatial embedding information preserving alleviate this problem to a certain extent.
A simple LSTM recommender can achieve competitive recommendation performance without
check-in sequential information consideration (Table 4 only(E)), demonstrating the effectiveness
of the spatiotemporal model in STEP. As visiting sequential information provides relatively coarsegrained POI depictions, the removal of the sequential model even leads to HIT@1 improvement on
the Instagram dataset. Also, the metric fallen on Gowalla (-18.6%, -27.8%, -32.4%, -26.2%) is more
significant than those on Instagram set (+3.1%, -11.4%, -15.0%, -6.0%), exactly opposed to their
temporal information abundance, quantitatively evaluated by average timestamps per POI.

4.2 PERFORMING TRAFFIC FLOW FORECASTING TASK WITH STE


To demonstrate the generalizability of STE, we perform traffic flow forecasting task with the
proposed spatiotemporal embedding methods. The relevant content is presented in detail in Appendix.A.

5 CONCLUSION AND DISCUSSION


In this paper, we propose the spatiotemporal aware embedding framework STE and apply it to POIs
(STEP). To the best of our knowledge, this is the first work that translating entorhinal-hippocampal
representing mechanisms to the spatiotemporal embedding. Inspired by the graph-representing policy in brain entorhinal-hippocampal system, STEP captures sequential and spatiotemporal representation from unlabeled sparse data through context graph building and graph-based embedding
learning. Moreover, STEP provides a highly efficient spatiotemporal model motivated by grid cells’
multi-scale spatial representation and place cells’ conjunctive representation, which overcomes the
problem caused by frequently used separate-representing. STEP-based successive POI recommendation method outperforms baselines and SOTAs on two real datasets without user preference invasion. Furthermore, this work presents a practical framework for effective spatiotemporal aware
modeling of general items, enabling more valuable spatiotemporal-related tasks such as wildlife
preservation and urban traffic management.


-----

REFERENCES

Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski,
Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert
Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlie
Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis
Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):429–433, 2018.

Buru Chang and Seoyoon Kim. Learning Graph-Based Geographical Latent Representation for
Point-of-Interest Recommendation. In Proceedings of the ACM International Conference on In_formation & Knowledge Management, pp. 135–144, 2020._

Buru Chang, Yonggyu Park, Donghyeon Park, Seongsoon Kim, and Jaewoo Kang. Content-aware
hierarchical point-of-interest embedding model for successive POI recommendation. In Proceed_ings of the International Joint Conference on Artificial Intelligence, pp. 3301–3307, 2018._

Eunjoon Cho, Seth A Myers, and Jure Leskovec. Friendship and mobility: user movement in
location-based social networks. In Proceedings of the ACM SIGKDD International Conference
_on Knowledge Discovery and Data Mining, pp. 1082–1090, 2011._

Suogui Dang, Yining Wu, Rui Yan, and Huajin Tang. Why grid cells function as a metric for space.
_Neural Networks, 142:128–137, 2021._

Howard Eichenbaum. On the integration of space, time, and memory. Neuron, 95(5):1007–1018,
2017.

Shanshan Feng, Xutao Li, Yifeng Zeng, Gao Cong, Yeow Meng, and Chee Quan. Personalized
Ranking Metric Embedding for Next New POI Recommendation. In Proceedings of the Interna_tional Joint Conference on Artificial Intelligence, 2015._

Shanshan Feng, Gao Cong, Bo An, and Yeow Meng Chee. POI2Vec : Geographical Latent Representation for Predicting Future Visitors. In Proceedings of the AAAI Conference on Artificial
_Intelligence, pp. 102–108, 2017._

Ruiqi Gao, Jianwen Xie, and Ying, Nian Zhu, Songchunand Wu. Learning Grid Cells as Vector Representation of Self-Position Coupled with Matrix Representation of Self-Motion. In Proceedings
_of the International Conference on Learning Representations, 2019._

Nicholas J Gustafson and Nathaniel D Daw. Grid Cells, Place Cells, and Geodesic Generalization
for Spatial Reinforcement Learning. PLoS Computational Biology, 7(10):1–14, 2011.

Michael U Gutmann and Aapo Hyv¨arinen. Noise-contrastive estimation of unnormalized statistical
models, with applications to natural image statistics. Journal of Machine Learning Research, 13
(2), 2012.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant
mapping. In Proceedings of the IEEE Computer Society Conference on Computer Vision and
_Pattern Recognition (CVPR), volume 2, pp. 1735–1742, 2006._

Minh X Hoang, Yu Zheng, and Ambuj K Singh. Fccf: forecasting citywide crowd flows based on
big data. In ACM SIGSPATIAL, 2016.

Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735–1780, 1997.

Ranzhen Li, Yanyan Shen, and Yanmin Zhu. Next Point-of-Interest Recommendation with Temporal
and Multi-level Context Attention. Proceedings of the IEEE International Conference on Data
_Mining, pp. 1110–1115, 2018._

Defu Lian, Cong Zhao, Xing Xie, Guangzhong Sun, Enhong Chen, and Yong Rui. GeoMF : Joint
Geographical Modeling and Matrix Factorization for Point-of-Interest Recommendation. In Pro_ceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Min-_
_ing, pp. 831–840, 2014._


-----

Nicholas Lim, Bryan Hooi, and Xueou Wang. STP-UDGAT : Spatial-Temporal-Preference User
Dimensional Graph Attention Network for Next POI Recommendation. In Proceedings of the
_ACM International Conference on Information & Knowledge Management, pp. 845–854, 2020._

Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. Predicting the next location: A recurrent model
with spatial and temporal contexts. In Proceedings of the AAAI Conference on Artificial Intelli_gence, pp. 194–200, 2016a._

Xin Liu, Yong Liu, Karl Aberer, and Chunyan Miao. Personalized Point-of-Interest Recommendation by Mining Users ’ Preference Transition. In Proceedings of the ACM International Confer_ence on Information & Knowledge Management, pp. 733–738, 2013._

Xin Liu, Yong Liu, and Xiaoli Li. Exploring the Context of Locations for Personalized Location
Recommendations. In Proceedings of the International Joint Conference on Artificial Intelli_gence, pp. 1188–1194, 2016b._

Hui Luo, Jingbo Zhou, Zhifeng Bao, and Shuangli Li. Spatial Object Recommendation with Hints
: When Spatial Granularity Matters. In Proceedings of the International ACM SIGIR Conference
_on Research and Development in Information Retrieval, pp. 781–790, 2020._

Christopher J MacDonald, Kyle Q Lepage, Uri T Eden, and Howard Eichenbaum. Hippocampal
?time cells? bridge the gap in memory for discontiguous events. Neuron, 71(4):737–749, 2011.

Gengchen Mai, Janowicz Krzysztof, Yan Bo, Zhu Rui, Cai Ling, and Ni Lao. Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells. In Proceedings of the
_International Conference on Learning Representations, 2020._

Jarana Manotumruksa, Craig Macdonald, and Iadh Ounis. A Contextual Attention Recurrent Architecture for Context-Aware Venue Recommendation. In Proceedings of the International ACM
_SIGIR Conference on Research and Development in Information Retrieval, pp. 555–564, 2018._

Bart Van Merri and Cifar Senior Fellow. Learning Phrase Representations using RNN Encoder
– Decoder for Statistical Machine Translation. In Proceedings of the Conference on Empirical
_Methods in Natural Language Processing, pp. 1724–1734, 2014._

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words
and Phrases and their Compositionality. In Proceedings of the International Conference on Neural
_Information Processing Systems, 2013._

John O’keefe and Lynn Nadel. The hippocampus as a cognitive map. Oxford: Clarendon Press,
1978.

Francesca Sargolini, Marianne Fyhn, Torkel Hafting, Bruce L McNaughton, Menno P Witter, MayBritt Moser, and Edvard I Moser. Conjunctive representation of position, direction, and velocity
in entorhinal cortex. Science, 312(5774):758–762, 2006.

Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus as a
predictive map. Nature Neuroscience, 20(11):1643–1653, 2018.

Ke Sun, Tieyun Qian, Tong Chen, Yile Liang, Quoc Viet Hung Nguyen, and Hongzhi Yin. Where
to Go Next: Modeling Long- and Short-Term User Preferences for Point-of-Interest Recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 214–221, 2020.

Yongxin Tong, Yuqiang Chen, Zimu Zhou, Lei Chen, Jie Wang, Qiang Yang, Jieping Ye, and
Weifeng Lv. The simpler the better: a unified approach to predicting original taxi demands based
on large-scale online platforms. In Proceedings of the 23rd ACM SIGKDD international confer_ence on knowledge discovery and data mining, pp. 1653–1662, 2017._

Hao Wang, Huawei Shen, Wentao Ouyang, and Xueqi Cheng. Exploiting POI-specific geographical influence for point-of-interest recommendation. In Proceedings of the International Joint
_Conference on Artificial Intelligence, pp. 3877–3883, 2018._


-----

James C.R. Whittington, Timothy H. Muller, Caswell Barry, Shirley Mark, and Timothy E.J.
Behrens. Generalisation of structural knowledge in the hippocampal-entorhinal system. In Pro_ceedings of the International Conference on Neural Information Processing Systems, 2018._

James C.R. Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil
Burgess, and Timothy E.J. Behrens. The Tolman-Eichenbaum Machine: Unifying Space and
Relational Memory through Generalization in the Hippocampal Formation. Cell, 183(5):1249–
1263.e23, 2020.

SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.
Convolutional lstm network: A machine learning approach for precipitation nowcasting. In Ad_vances in neural information processing systems, pp. 802–810, 2015._

Carl Yang, Lanxiao Bai, Chao Zhang, Quan Yuan, and Jiawei Han. Bridging collaborative filtering
and semi-supervised learning: A neural approach for POI recommendation. In Proceedings of the
_ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1245–_
1254, 2017.

Mao Ye, Peifeng Yin, Wang-chien Lee, and Dik-lun Lee. Exploiting Geographical Influence for
Collaborative Point-of-Interest Recommendation. In Proceedings of the International ACM SIGIR
_Conference on Research and Development in Information Retrieval, pp. 325–334, 2011._

Miaolong Yuan, Bo Tian, Vui Ann Shim, Huajin Tang, and Haizhou Li. An entorhinal-hippocampal
model for simultaneous cognitive map building. In Proceedings of the AAAI Conference on Arti_ficial Intelligence, volume 29, pp. 582–596, 2015._

Jia-dong Zhang. LORE : Exploiting Sequential Influence for Location Recommendations. In Pro_ceedings of the ACM SIGSPATIAL International Conference on Advances in Geographic Infor-_
_mation Systems, pp. 103–112, 2014._

Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, and Xiuwen Yi. Dnn-based prediction model for
spatio-temporal data. In ACM SIGSPATIAL, 2016.

Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide
crowd flows prediction. In AAAI, 2017.

Pengpeng Zhao, Haifeng Zhu, Yanchi Liu, Jiajie Xu, Zhixu Li, Fuzhen Zhuang, Victor S Sheng, and
Xiaofang Zhou. Where to Go Next : A Spatio-Temporal Gated Network for Next POI Recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.

Pengpeng Zhao, Haifeng Zhu, Yanchi Liu, Zhixu Li, Jiajie Xu, and Victor S. Sheng. Where to Go
Next: A Spatio-Temporal Gated Network for Next POI Recommendation. IEEE Transactions on
_Knowledge and Data Engineering, 4347(c):1–13, 2020._

Shenglin Zhao, Tong Zhao, Haiqin Yang, Michael R Lyu, and Irwin King. STELLAR : SpatialTemporal Latent Ranking for Successive Point-of-Interest Recommendation. In Proceedings of
_the AAAI Conference on Artificial Intelligence, pp. 315–321, 2016._

Shenglin Zhao, Tong Zhao, Irwin King, and Michael R. Lyu. Geo-Temporal Sequential Embedding
Rank for Point-of-interest Recommendation. In Proceedings of the International World Wide Web
_Conference, pp. 153–162, 2017._

Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. What to do
next: Modeling user behaviors by time-lstm. In Proceedings of the International Joint Conference
_on Artificial Intelligence, pp. 3602–3608, 2017._


-----

A CASE STUDY: TRAFFIC FLOW FORECASTING WITH STE

As an essential basis of traffic scheduling, accurate traffic flow forecasting is of great significance.
In this part, we perform traffic flow forecasting with the proposed STE, which is abbreviated as
STE-TG (STE of Traffic Grid).

A.1 PROBLEM FORMULATION

Figure 6: An illustration of (a) grid map partition and (b) flow of a grid.

Following previous works (Hoang et al., 2016; Zhang et al., 2016), we partition a city into an M ×N
_traffic grid map based on the longitude and latitude where a traffic grid represents a geographical_
region. For a grid (m, n) that lies at the m[th] row and the n[th] column, two types of traffic flows at
_k[th]_ timestamp are considered, namely inflow and outflow, defined as


_x[in,m,n]k_

_x[out,m,n]k_


_i > 1_ _gi_ 1 / (m, n) _gi_ (m, n) (14)
_T rXk∈P_ _|{_ _|_ _−_ _∈_ _∧_ _∈_ _}|_


_x[out,m,n]k_ = _i_ 1 _gi_ (m, n) _gi+1 /_ (m, n) (15)

_T rXk∈P_ _|{_ _≥_ _|_ _∈_ _∧_ _∈_ _}|_

where Trk means the trajectory at the k[th] timestamps, gi indicates the geographical coordinate,
_gi ∈_ (m, n) means point gi lies within grid (m, n) and | · | means the cardinality of a set. At
timestamp k, the traffic flow of grid (m, n) is represented as Xk[m,n] = (x[in,m,n]k _, x[out,m,n]k_ ). The
goad of traffic flow forecasting is: given Xk, k 0, 1, ..., t 1, predict Xt.
_∈{_ _−_ _}_

A.2 METHODOLOGY

As illustrated in Fig.1 in the main text, the input of STE-TG is composed of observations (coordinates and visiting time patterns) and contexts (context graphs). Unlike STEP, as there is no sequential
traffic information, the sequential model is not applicable.

**STE usage** Following Sec.3.2.1 in the main body of the article, we use spatial observation and
build a spatial context graph in the same way. Specifically, same hyper-parameter settings of
_S, λmin/max is adopted in grid-cell encoder, and top-10 policy is used for building Gspa. Also,_
the embedding objective of the spatial sub-model is still the term in Eq.7. Different from POIs, the
timestamps of the traffic grid flow data are collected using a standard time interval. Thus both the
encoding of visiting time patterns and the construction of spatiotemporal context graph could be
simplified. Notably, we generate the visiting time pattern matrix as the temporal observation input
for the spatiotemporal model in the same way we did for POIs, and since the timestamp is standard,
we skipped the accumulation step in Fig.3. For constructing Gst, we define the grid temporal adjacency using a specified distance function d(·, ·) which is calculated by the normalized traffic flow
difference value of two traffic grids.


_. (16)_
i


_d((m, n), (m[′], n[′]))_ = △ [1]


`abs` _x[in,m,n]k_ _x[in,m]k_ _[′][,n][′]_
_−_



+ abs _x[out,m,n]k_ _x[out,m]k_ _[′][,n][′]_
_−_



_k=1_


-----

Figure 7: Visualization of predicted results and ground truth traffic flow on TaxiBJ dataset. Figures
in the top row are ground truth flows and the bottom ones are the predicted values. STE-TG can
achieve accurate traffic flow prediction only using the traffic information of the previous time step
and the spatiotemporal representations estetg of the center grid and the adjacent grids.

We use the top-10 strategy to get the temporal neighboring relations of grids with low distances. Together with the spatial context information Gspa, Gst is built to help the embedding model preserve
the spatiotemporal conjunctions between the traffic grids. The objective function used to optimize
_the spatiotemporal model remains the same as in the main text._

**Predictor Design** The flow of one grid is significantly related to the nature of the grids around it
because an outflow from one region can lead to an inflow to neighboring regions. In this work, the
spatiotemporal natures of grids and correlations between grids are encoded in the embedding vector
_estetg. Thus, Given a query Xt[m,n]1_ [, spatiotemporal embeddings of four geographical closest grids]
_−_
and grid (m, n) itself will also be fed into predictor fθ to produce the estimation

_Xˆt[m,n]_ = (ˆx[in,m,n]t _, ˆx[out,m,n]t_ ) = fθ(Xt[m,n]1 _[,][ e][m,n][)]_ (17)
_−_

where e[m,n] = (e[m,n]stetg[, e][neigh]stetg [)][ and][ e]stetg[neigh] [indicates embeddings of four spatial neighbors of grid]
(m, n). We use a simple three-layer MLP fθ with ReLU activations for implementation, the dimension of hidden state is 256. The loss function used is mean squared error ||X[ˆ] − _X||2[2][.]_

A.3 EXPERIMENTS

Following the same pipeline we used in POI recommendation task, we first train the spatiotemporal
embedding model STE-TG and then optimize the recommender model on the training set. All
models are optimized using Adam optimizer with default βs, the learning rate is set to 1 × 10[−][3].
The embedding dimensions _dspa, dst_ are set to 64, 96 .
_{_ _}_ _{_ _}_


-----

Table 5: Comparisons with baselines on TaxiBJ15 dataset.

|METHOD|RMSE|
|---|---|
|ARIMA SARIMA VAR CNN DeepST-CPTM STE-TG (Ours)|25.58 29.11 25.59 26.08 22.59 21.92|



Table 6: Comparisons with baselines on TaxiBJ dataset.

|METHOD|RMSE|
|---|---|
|HA ARIMA SARIMA LinUOTD ConvLSTM DeepST-CPTM STE-TG (Ours)|57.69 22.78 26.88 21.23 19.54 18.18 18.03|



**Metrics** We measure the traffic flow forecasting performance by Root Mean Square Error (RMSE)
as


RMSE


(xi _xˆi)[2]_ (18)
_−_


where ˆx and x are the predicted value and ground truth, respectively and t is the number of all
queries.

**Datasets** We perform experiments on two datasets:

-  TaxiBJ15 (Zhang et al., 2016) consists of calculated inflow/outflow based on taxicab GPS
data collected in Beijing city in 2015.

-  TaxiBJ (Zhang et al., 2017) is an extended version including data from 2013 to 2016.

For TaxiBJ15, data from the last week is used for testing. For TaxiBJ, we choose the last four weeks
as the testing data.

**Baselines** We choose representative baselines for comparison:

-  HA: a predict method by the average value of historical values.

-  ARIMA (Auto-Regressive Integrated Moving Average)

-  SARIMA: seasonal ARIMA

-  VAR (Vector Auto-Regressive)

-  ST-ANN: It first extracts spatial (nearby 8 regions’ values) and temporal (8 previous time
intervals) features, then fed into an ANN.

-  DeepST (Zhang et al., 2016): a deep neural network (DNN)-based prediction model for
spatiotemporal data, which shows state-of-the-art results on crowd flows prediction, we
only consider the most potent variant, i.e. DeepST-CPTM.

-  ST-ResNet (Zhang et al., 2017) ST-ResNet models citywide traffic flow at different times
into 2D images to perform prediction.

-  LinUOTD (Tong et al., 2017): a linear regression method with a spatio-temporal regularization.

-  ConvLSTM (Xingjian et al., 2015): a LSTM variant with convolution modules.


-----

**Results** We listed the comparison results in Table.5 and Table.6. In STE-TG, only the grid map
and historical traffic flow are used to perform flow prediction, while existing works usually take
meta-data like weather into account. However, we observe that the hastily constructed traffic flow
prediction methods STE-TG achieve competitive performance, which also demonstrates the effectiveness of the proposed STE framework.


-----

