# JOINTLY LEARNING TOPIC SPECIFIC WORD AND DOC## UMENT EMBEDDING

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Document embedding generally ignores underlying topics, which fails to capture
polysemous terms that can mislead to improper thematic representation. Moreover, embedding a new document during the test process needs a complex and
expensive inference method. Some models first learn word embeddings and later
learn underlying topics using a clustering algorithm for document representation;
those methods miss the mutual interaction between the two paradigms. To this
point, we propose a novel document-embedding method by weighted averaging
of jointly learning topic-specific word embeddings called TDE: Topical Document Embedding, which efficiently captures syntactic and semantic properties by
utilizing three levels of knowledge -i.e., word, topic, and document. TDE obtains
document vectors on the fly simultaneously during the jointly learning process
of the topical word embeddings. Experiments demonstrate better topical word
embeddings using document vector as a global context and better document classification results on the obtained document embeddings by the proposed method
over the recent related models.

1 INTRODUCTION

In recent years the neural network-based methods have shown great success in various NLP-related
tasks (Zhao et al., 2021). Tow popular complementary natural language processing methods are
word2vec (Mikolov et al., 2013a;b) and Latent Dirichlet Allocation (LDA)(Blei et al., 2003); but
executing a model in two-step lacks mutual interaction between the two paradigms and fails to capture mutual reinforcement between themselves. Doc2vec (Le & Mikolov, 2014) generalizes the
idea of vector representation for documents as a stand-alone method. Though it is expensive to infer
new document vectors during the test process, and regardless of assigning PV to each word in the
document as a single topic, it outperforms BOW (Zhang et al., 2010) and LDA (Fan et al., 2008)
in various text understanding tasks. Encouraged by word2vec and Doc2vec, Niu et al. (2015) introduced Topic2Vec. It incorporates the LDA topic model into NPLM (Bengio et al., 2003). However,
Topic2Vec cannot capture the polysomic characteristics of terms as LDA models topic occurrence
almost independently without capturing topic correlations (He et al., 2017). The purpose of conducting this research is to introduce an efficient multi topical document representation with flowing
objectives:

_• Firstly, learn probabilistic topical word embeddings using document vector as a global_
context where each word can be part of different underlying topics of the document instead
of just a single paragraph vector. Thus, avail to capture polysemous terms.

_• Secondly, embedding documents by simply averaging the topical word embeddings instead_
of imposing an expensive inference method to learn the document embedding separately.

Unlike clustering-based document modeling with the ability to capture multi-sense of words (Gupta
et al., 2019), here the challenge is to learn polysemous words in a jointly learning framework for
topic-specific word and document embedding. Capturing polysemous terms is vital because the
same word around different context words can represent different themes (Liu et al., 2020) and
without noticing them can mislead to an unexpected thematic representation of a document. The
Jointly Learning Word Embeddings and Latent Topics (Shi et al., 2017) comes to the rescue for
addressing the polysemous word problem, which learns different topical word embeddings using


-----

Skip-gram Topical word Embedding (STE) framework. STE gets the term distributions of the topics
and the topic distributions of the document using PLSA(Hofmann, 2013) like mixture weighting
for each data point. However, STE doesn’t deal with document embedding in its jointly learning
process.
In this research, we propose a novel document embedding framework called TDE: Topical Document Embedding. TDE learns topical word and document embeddings jointly. It incorporates the
STE-diff framework (Shi et al., 2017) and follows the simple averaging of word embedding with a
corruption mechanism for document embedding introduced by Chen (2017).

2 MOTIVATION AND RELATED WORK

Huang et al. (2012) proved that combining local and global context performs better for predicting target words in the neural network-based language modeling, but this model doesn’t support
topic-specific word embeddings. Liu et al. (2015) proposed topic-specific word embeddings where
document embeddings get by aggregating over all topical-word embeddings of each word in the
document. Arora et al. (2017) extend embedding to the sentence level by smooth inverse frequency
(SIF) weighted averaging of the words in a sentence. However, this model is lack of addressing multiple sense of words. Moody (2016) introduced a hybrid model called lda2vec, which suffers from
uncorrelated topics due to the incorporation of LDA. For word sense disambiguation with correlated
topic modeling, Chaplot & Salakhutdinov (2018) replaced the LDA topic proportions by synset proportions for a document. Other than LDA topic modeling, Arora et al. (2018) have introduced a
Gaussian random walk model for word sense disambiguation. This work can address polysemous
word representation but doesn’t cover document-level embeddings. Mekala et al. (2016) introduced
the Gaussian mixture model-based soft clustering method called SCDV, which incorporates both the
topic and word embeddings. However, SCDV is not suitable for multisentence documents. Hence
Luo et al. (2019) proposed P-SIF, which is fit for large documents with multiple sentences while
addressing the underlying topics as well. Though P-SIF overcomes SCDV’s shortcomings, it is still
a two-step process.
From the motivation of the above advantages and drawbacks, we introduce a novel document modeling capable of learning document embedding simultaneously during the jointly learning process of
topic-specific word embeddings. This model creates a pivot by concatenating a document vector and
center word vector to predict the context word vector. Here the document vector works as a memory backup for missing information. The output document embeddings produced by this model is
more expressive as it represents each document into a higher dimensional embedding space similar
to P-SIF.

3 MODEL DESCRIPTION

Inspired by the Skip-gram Topical word Embedding (STE) (Shi et al., 2017) and the simplicity of
the Document Vector through Corruption(Chen, 2017), we design a novel document embedding
model called TDE: Topical Document Embedding. In contrast to Doc2VecC, TDE constructs global
context (document vector) as a weighted averaging of the randomly selected topic-specific word
embeddings. Unlike STE, TDE uses a generating function, which predicts context words by given
the topics and pivot. Pivot constructs by concatenating global context (document vector) and center
word (Figure1). The probability of predicting a context word depends on the inner product of the
topic-specific embedding of that word (context word) and the pivot, which ensures the participation
of underlying word, topic, and document levels knowledge.

3.1 TOPICAL DOCUMENT EMBEDDING

A matrix D represents the documents in the corpus, D = d1, d2, . . ., dn of size n. Each document d
contains a variable-length sequence of words W = w1, w2, . . ., wTd . Each word w in the vocabulary
_V of size v of the corpus is generally associated with an input projection matrix Uϵ R[h][×][v]_ of size
_h and an output projection matrix V_ _[T]_ _ϵ R[v][×][h]_ of size h. The input projection matrix represents the
input to the hidden space, whereas the output projection matrix represents the hidden to output space.
Matrix V _[T]_ has the same dimension as matrix U but a transpose matrix. The column uw of matrix
_U is the input vector of word w, and the column vw of matrix V_ _[T]_ is the output vector of word w.


-----

Figure 1: Predicting context words by giving pivots
and underlying topics of the document.


Figure 2: Topics assignment of polysemous words.


TDE follows an unbiased mask-out/drop-out corruption mechanism in each update of its stochastic
process for randomly select topic-specific word embeddings to create the global context. Under
corruption mechanism, each dimension of the original document d overwrites with the probability q
and sets the uncorrupted dimensions (1 1 _q)_ [times of their original values to make the corruption]

_−_
unbiased. The corruption mechanism reduces word-level noise in the document by forcing the
embeddings of common and non-discriminative ones close to zero. This corruption mechanism
leads TDE to speed up the training process by significantly pruning the number of parameters for
the backpropagation mechanism, document vectors represented by only embeddings of remaining
words. The resultant global context is thus as below:

0, _with probability q_


(1)

_x˜d_ (2)


_x˜d =_

 1−d _q_ _[,]_ _otherwise_

_Document vector (global context) ˜x = [1]_


where T is the length of the document (number of words) and the document vector ˜x is the weighted
average of the words in a document, weighted by the corruption mechanism mentioned above.
The probability of observing the context word wt+i given the center word wt as well as the global
context ˜x and the topics zt+i and zt for context and center word respectively as

_exp_ _Vw[T]t+i, zt+i_ _[.][ (][U][w]t[, z]t_ [+] _T1_ _[U]x[˜]d_ [)]
_p(wt+i_ _wt, zt+i, zt, ˜x) =_ (3)
_|_ _w[′]_ _ϵ Λ[exp]_ _Vw[T][′]_ _, zt+i_ _[.][ (][U][w][t][, z][t][ +]_ _T1_ _[U]x[˜]d_ [)]

 

P


where Λ is the vocabulary of the whole corpus and input projection matrix constructs as U =
_Uwt, zt +_ _T1_ _[U]x[˜]d_ [.]

The denominator in equation (3) is generally intractable due to the term _w[′]_ _ϵ Λ[. As such, the]_

computational cost is proportional to the size of Λ. To deal with this intractability requires an efficient approximation method. TDE employs the approximation with the negative sampling (Mikolov

[P]

et al., 2013a; Chen, 2017). In TDE, the probability of predicting a context word wt+i in document
_d depends on the pivot wtx˜ and the topic distributions of the document. The pivot constructed by_
concatenating the center word wt and the global context ˜x as
_Pivot = wtx˜ = wt + ˜x_ (4)

So, for every skip-gram ⟨wt, wt+i⟩, where the topic assignment zt+i of wt+i is independent of the
topic assignment zt of wt we have :


_p(wt+i_ _wtx, d˜_ ) =
_|_


_p(wt+i_ _wtx, z˜_ _t+i, zt) p(zt+i, zt_ _d)_
_zt+i=1_ _|_ _|_

X

_k_

_p(wt+i_ _wtx, z˜_ _t+i, zt) p(zt_ _d)p(zt+i_ _d)_
_zt+i=1_ _|_ _|_ _|_

X


_zt=1_

_k_

_zt=1_

X


(5)


-----

Following a similar analogy as STE-diff, equation (5) assumes that the context word and the center
word may come from the same topic or may come from two different topics as well without imposing any hard constraint (Figure 2). Here, multiple meanings of a word depend on their contexts
instead of directly k topics disjoint subspace partitioning as commonly does in clustering-based
topic modeling(Gupta et al., 2016). In TDE, word embeddings from different topics co-exist in a
common shared space. Instead of the free flow of scattered words in the embedding space, different
topical word embeddings stay close together document-wise rather than topic-wise gathering. Creating pivots by concatenating the center word with the document vector provides extra guidance as
a global context with a complete thematic sense of the document. Thus, TDE represents more consistent topical word embeddings over STE-diff. In TDE, the compositionality of words encodes in
the learned embeddings; as a result, the requirement of heuristic weighting for test documents is no
more required; each output document vector is represented simply as an average of the topic-specific
word embeddings. Given the learned projection matrix Uw,z, TDE forms document embedding dvec
as below:


_dvec =_ _z=1_
_⊕[k]_


_Uw,z_ (6)
_w ϵ D_

X


where T is the length (number of words) of the document and k represents the underlying topics of
the document.
In equation (6), the term ⊕z[k]=1 [represents topic-wise concatenation. In this case, words in the same]
topic are just simple average as they represent the same thematic meaning. However, words in
different topics carry different themes. Thus, concatenate over topics increases the expressive power
of a document. This strategy represents documents into a higher k × s dimension, where k is the
underlying topics, and s is the dimension of the word embeddings.

3.1.1 ALGORITHM DESIGN

The inference framework of TDE (Algorithm 1) is similar to STE-diff, based on the EM algorithm
with negative sampling (Shi et al., 2017). In each iteration, it updates the word embeddings first, then
based on updated word embeddings, updates the document embedding (global context) ˜x. Finally,
evaluate the topic distribution p(z | d) of each document. To update word embeddings, iterates over
each skip-gram _wt, wt+i_ using negative sampling and evaluate the posterior topic distribution.
_⟨_ _⟩_
So, the estimation of the probability of predicting the context word wt+i given pivot wtx˜ and topic
assignment z denoted by p(wt+i _wtx, z˜_ ) as follows:
_|_


log p(wt+i _wtx, z˜_ ) log σ _Vw[T]t+i, z_ _[.][ (][U][w]t[, z]_ [+ 1] _xd_ [)]
_|_ _∝_ _T [U][˜]_



(7)



log σ _Vw[T]i, z_ _[.][ (][U][w]t[, z]_ [+ 1] _xd_
_−_ _T [U][˜]_



_Ewi∼p_


where σ(x) = 1+exp1 ( _x)_ [and][ w][i][ is a negative instance, which is sampled from the unigram distri-]

_−_
bution p(.) over terms in the vocabulary and has a value similar to Gupta et al. (2019)’s findings.
The overall training objective is how well the model can predict surrounding words, which is for
each document d, given a sequence of words w1, w1, . . . ., wTd, then the log-likelihood Ld is defining as

_Td_

_Ld =_ Xt=1 _−c≤Xi≤c; i=0̸_ log p(wt+i | wtx, d˜ ) = 0 (8)

where c is the size of the training windows.
The overall log-likelihood L is then

= _d_ (9)
_L_ _L_

_d_

X

The gradients of the objective function to evaluate log p(wt+i _wtx, d˜_ ) for each term concerning
the input and the output projection matrix U and V _[T]_ respectively can formulate as follows: |


_Ld =_


_t=1_


_∂L_ = _ξw[′]_ _wt_ _σ(Vw[T][′]_ _,z_ _[U][w][t]x, z[˜]_ [)] _. Vw[T][′]_ _,z[. p][(][z][ |][ d, w][t]x, w[˜]_ _′_ ) (10)

_Uwt ˜x, z_ _−_ _−_
 


-----

_V∂[T]L_ = − _ξw[′]_ _wt_ _σ(Vw[T][′]_ _,z_ _[U][w][t]x, z[˜]_ [)] _. Uwt ˜x,z[. p][(][z][ |][ d, w]tx, w[˜]_ _′_ ) (11)
_w[′]_ _,z_ _−_
 


where


_′_
1, _if w_ _in the context window of wt_

0, _otherwise_


(12)


_ξw[′]_ _wt =_


In TDE, the update rule of U and _V_ _[T]_ maximizes p(wt+i _wtx, z˜_ ), thus log p(wt+i _wtx, z˜_ ) for each
_|_ _|_
skip-gram _wt, wt+i_ instead of maximizing log p(wt+i _wt, z) as STE does. This observation_
shows both of the models flow the same procedure. STE has proved capable of discovering good ⟨ _⟩_ _|_
topics and identifying high-quality word embedding vectors. Thus, TDE inherits all the prominences; moreover, it keeps Doc2VecC’s simplicity for document embeddings and extends reliability
by filling up essential information gaps using global document context. When concerning topics
for the embedding process, the generally used hierarchical Softmax or Negative sampling methods
are incapable of obtaining the embeddings efficiently. The EM-negative sampling Algorithm thus
comes to the rescue.
In the E-step, the posterior topic distribution for each skip-gram _wt, wt+i_ in document d evaluate
_⟨_ _⟩_
using the Bayes rule:

_′_ _′_
_p(zk′_ _x, wt+i) =_ _[p][(][w][t][+][i][ |][z]k[, w][t]x[˜])p(zk_ _[|][ d][)]_ (13)

_[|][ d, w][t]_ [˜] _z_ _[p][(][w][t][+][i][ |][z, w][i]x[˜])p(z | d)_

In the M-step, given the posterior topic distribution, it is now time to update the topic-specific word

P

embeddings and the topic distribution of the document by maximizing the following Q function:


_Td_

_t=1_

X


_Q =_


_p(z_ _d, wtx, w˜_ _t+i) log(p(z_ _d)p(wt+i_ _z, wtx˜))_
_|_ _|_ _|_


_−c≤i≤c; i=0̸_


_n(d, wt, wt+i)_
_{wt, wXt+i} ϵ Pd_


_p(z_ _d, wtx, w˜_ _t+i) [log(z_ _d) + log(p(wt+i_ _z, wtx˜))]_
_|_ _|_ _|_

(14)


Where Pd is the set of skip-gram in document d and n(d, wt, wt+i) denotes the number of skipgram _wt, wt+i_ in document d.
_⟨_ _⟩_
For each document d, the prior topic distribution p(z | d) can be updated using the Lagrange multiplier while keeping the normalization constraints i.e. _z_ _[p][(][z][ |][ d][) = 1][.]_

_wt,wt+i_ _ϵ Pd_ _[n][(][d, w][t][, w][t][+][i][)][p][(][z][ |][ d, w][t]x, w[˜]_ _t+i)_

_p(z_ _d) =_ _{_ _}_ [P] (15)
_|_ P _wt,wt+i_ _ϵ Pd_ _[n][(][d, w][t][, w][t][+][i][)]_

_{_ _}_

The EM negative sampling algorithm in TDE follows a similar workflow as in STE. TDE can eval-P
uate the likelihood estimation p(wt+i _z, wtx˜) for each skip-gram_ _wt, wt+i_ . As a result, each
topic represents by the ranked list of the bi-grams, similar to STE. As the document vector is learn- | _⟨_ _⟩_
ing simultaneously with word embeddings by averaging the word vectors, the time complexity of
_p(wt+i_ _z, wtx˜) is thus_ Λ, where Λ is the size of vocabulary in the document and K is the
number of topics, which is the same as STE. | _|_ _|[2][×][K]_ _|_ _|_
_′_
For each word w in a given new document d, the posterior topic distribution infers by considering
_′_
the topic distribution of the context words wc and the document d, while keeping the values of the
learned input and output projection matrices U and V _[T]_ respectively unchanged. As instead of the
center word w itself, we use pivot wx˜. Therefore, using the Bayes rule:

_′_ _′_ _′_
_p(z_ _wx, w˜_ _c, d_ ) log(z _d_ ) + log p(wc _z, wx, d˜_ ) (16)
_|_ _∝_ _|_ _|_

where wc is the set of the context words of word w, and wx˜ is the pivot created by concatenating
_′_
global context ˜x with the center word vector w. The log-likelihood term log p(wc _z, wx, d˜_ ) can
_′_ _|_
define as the sum of log p(wc _wx, z˜_ ), and log(z _d_ ) is the corresponding prior probability. The
probability log p(wc _wx, z˜_ ) can be computed using Eq.7 | _|_
The output document vector creates directly using Eq.6 without imposing an inference method for |
the document separately.


-----

**Algorithm 1 TDE using EM negative sampling**

1: Initialize Input and Output projection matrix U and V _[T]_ respectively and the topic distribution
of the document p(z | d).

2: for out iter = 1 to Max Out iter do
3: **for each document d in D do**

4: **for each word w in U do**

5: Create document vector (global context) ˜x using Eq.1 and Eq.2

6: **end for**

7: **for each skip-gram ⟨wt, wt+i⟩** in d do

8: Sample negative instances from the distribution P.

9: Construct pivot wtx˜ using Eq.4

10: Update likelihood function p(wt+i _wtx, z˜_ ) using Eq.7 and posterior topic distribution
_′_ _|_
_p(zk_ _x, wt+i) using Eq.13_

11: **for in iter[|][ d, w][t] = 1[˜]** to Max in iter do

12: Update projection matrix U and V _[T]_ using gradient decent method with Eq.10 and
Eq.11 respectively.

13: **end for**

14: **end for**

15: Update topic distribution p(z | d) using Eq.15

16: **end for**

17: end for
/* Creating document vectors using test dataset */

_′_ _′_
18: for each test document d in D **do**

19: **for each word w in updated Uw,z do**

20: Construct the output document vector dvec using Eq.6

21: **end for**

22: end for

Table 1: Document classification using linear SVM (libliner-2.30) classifier

|Models|Number of Train docs|Number of Test docs|Accuracy (%)|Error rate (%)|
|---|---|---|---|---|
|TDE|50,000 100,000 (all)|50,000 100,000 (all)|75.00 87.50|25.00 12.50|
|Doc2VecC|50,000 100,000 (all)|50,000 100,000 (all)|72.29 87.32|27.71 12.68|


Models Number of Train docs Number of Test docs Accuracy (%) Error rate (%)

50,000 50,000 **75.00** **25.00**
TDE

100,000 (all) 100,000 (all) **87.50** **12.50**

50,000 50,000 72.29 27.71
Doc2VecC

100,000 (all) 100,000 (all) 87.32 12.68


4 EXPERIMENTS AND ANALYSIS

The purpose of this research is to implement a simple but effective document embedding method.
This section is dedicated to demonstrate a detailed analysis of the experiments and obtained results.
To analyze document embeddings obtained by the proposed model (TDE), we use a linear SVM
classifier to check the performance of the document classification problem. We also extend our
analysis to topical word embeddings efficiency check as we proposed to utilize document vector as
a global context in the process of jointly learning word-topic embeddings.

4.1 DOCUMENT CLASSIFICATION FOR DOCUMENT EMBEDDING EVALUATION

Doc2VecC doesn’t consider topic-specific words in its document embeddings; it is thus in the
same embedding space as the word. On the other hand, TDE represents documents into higher
(number of clusters × number of word embeddings features) dimensional space.
We use a subset of the Wikipedia dump (Chen, 2017), which contains over 300,000 Wikipedia pages
in 100 categories. In our experiments both TDE and Doc2VecC learn 100,000 unique documents
representation. For both models, we set embedding size to 400, window size to 10, negative sampling to 8, subsampling frequent words to 10e-4, number of threads to 10, sentence sample to 0.1,
and choose liblinear-2.30 as a linear SVM classifier (Fan et al., 2008), for TDE we set number of
topics to 10, inner and outer iteration to 15. Table 1 shows both TDE and Doc2VecC (base model)
perform better classification on seen data than unseen data. For TDE and Doc2VecC, we gave the


-----

first 50,000-document vectors as a train set; and the last 50,000 document vectors as a test set,
where TDE gives 75% accuracy with a 25% error rate and Doc2VecC exhibits 72.29% accuracy
with a 27.71% error rate. When we provide all 100,000 document vectors for both train and test,
TDE gives 87.5% accuracy with a 12.5% error rate, and Doc2VecC shows 87.32% accuracy with a
12.68% error rate. In Table 1, highlighted cells represent better scores obtained by TDE.

4.2 WORD EMBEDDING EVALUATION

For word embedding evaluation, we use Stanford Contextual Word Similarity (SCWS) data set
(Huang et al., 2012). SCWS dataset contains 2003 word pairs; most of them are monosemous
words, only 241-word pairs containing the same term within a word pair among them only 50 have
a similarity score less than 5.0. That indicates a small number of challenging polysemous words in
the SCWS dataset. To obtain word embeddings for both STE-diff and TDE, we use the 20Newsgrup
dataset to train the models. For both models, we set embedding size to 400, number of topics to 20,
window size to 10, negative sampling to 8, subsampling frequent words to 10e-4, number of threads
to 10, both inner and outer iteration to 15, and minimum word count to 5. We take 1693 word
pairs under consideration from the SCWS dataset (commonly found in the 20Newsgrup and SCWS
datasets) for Spearman’s rank correlation coefficient measures between models scores (achieved by
TDE and STE-diff) and the human judgments of similarity scores in the SCWS dataset. For each
pair of words, pairwise cosine similarity uses for the word similarity evaluation of the base (STEdiff) and the proposed(TDE) model; for both models, we use two variants of similarity scoringAgvSimC and MaxSimC (Shi et al., 2017). Table 2 exhibits obtained scores by the base and the



Table 3: PMI scores as topic coherence

|Model|T=5|T=10|T=15|T=20|
|---|---|---|---|---|
|TDE|-69.36|-331.74|-729.65|-1349.46|
|STE-diff|-85.23|-374.84|-875.77|-1599.42|

|Table 2: S|Spearman correlation (|(ρ × 100)|
|---|---|---|
|Model|Similarity Metrics|ρ 100 ×|
|TDE|MaxSimC AvgSimC|39.22 39.04|
|STE-diff|MaxSimC AvgSimC|31.56 33.28|


Model T=5 T=10 T=15 T=20

TDE **-69.36** **-331.74** **-729.65** **-1349.46**

STE-diff -85.23 -374.84 -875.77 -1599.42


Model Similarity Metrics _ρ × 100_

MaxSimC **39.22**
TDE

AvgSimC **39.04**

MaxSimC 31.56
STE-diff

AvgSimC 33.28


proposed model under the conditions mentioned above. According to Table 2, TDE obtains almost
similar scores for MaxSimC (39.22) and the AvgSimC (39.04) whereas, in STE-diff, MaxSimC gets
31.56 and AvgSimC gets 33.28. Highlighted cells indicate better scores in the table obtained by
TDE.

4.3 WORD EMBEDDING VISUALIZATION

To visualize the interaction between words and latent topics in the embedding space, we use the
Wikipedia dump from April 2010 (Shaoul, 2010), similar to STE. For both base (STE-diff) and proposed (TDE) models, we set embedding size to 400, number of topics to 10, window size to 10,
negative sampling to 8, subsampling frequent words to 10e-4, number of threads to 10, the inner
and the outer iterations to 15. For visualization in Fig 3 and Fig 4, we use the t-SNE algorithm
(Van der Maaten & Hinton, 2008), where each node represents a topic-specific word vector. For
both models, we set perplexity to 40, n iteration to 2500, and random state to 23. We consider
the word party with two different meanings political party and social gathering as a polysemous
word for visualization. In Fig 3, party#9 is close to government#9 indicating political party but
party#7 is far from government#9. our extensive analysis found party#0, party#1, party#2, party#6,
and party#9 contain government#9 and government#0 within topmost 100 similar words, whereas
party#3, party#4, party#5, party#7, party#8 are close to word summers#5, indicating a gathering and
celebration. These words contain summer#5 in the topmost 50 similar words.
In Fig 4, It seems party and government exist little scattered in the embedding space for STE-diff
than TDE; it indicates learning better embeddings by TDE over STE-diff. Our extensive experiment found, only party#6 contains government#6 within the topmost 100 similar words, whereas
summer#4 exists in the most 150 similar words of party#4 only.


-----

Figure 3: TDE polysemous word embedding visualization.


Figure 4: STE-diff polysemous word
embedding visualization.


Table 4: TDE topics generating bigrams for 20Newsgrups dataset

|god heaven jesus died believe god jesus god jesus christ father son father jehovah god father jehovah elohim christ god|image files open file file format file server file manager image processing file size file stream output file postscript file|mr president national security national institute president clinton clinton administration senior administration national center press conference bill clinton press release|
|---|---|---|
|alt.atheism soc.religion.christian talk.religion.misc|comp.graphics comp.os.ms-windows.misc comp.sys.ibm.pc.hardware comp.sys.mac.hardware comp.windows.x|talk.politics.misc talk.politics.guns talk.politics.midest|



4.4 TOPIC COHERENCE

In TDE, pointwise mutual information (PMI) scores are using to evaluate topic coherence as PMI
scores strongly correlate with human annotations of topic coherence (Newman et al., 2010; Lau
et al., 2014). For this experiment, we use the 20Newsgrup dataset. The parameter setting for both
models base (STE-diff) and proposed (TDE) are the same as mentioned in section 4.2. Table 3
exhibits overall PMI scores for different numbers of top words (T) under 20 topics. The highlighted
cells in the table indicate higher coherence, as we obtained negative coherence scores so, the smaller
is better.
Topics in TDE can be interpreted by bigrams words as shown in the publication (Shi et al., 2017)
for STE-same. In our experiments, we found for STE-diff, each topic represents bigrams randomly


-----

Table 5: Multi-class document classification for word embedding evaluation.

|Model|Accuracy|Precision|Recall|F-measure|
|---|---|---|---|---|
|PV|75.4|74.9|74.3|74.3|
|STE-same|80.4|80.3|80.4|80.2|
|TDE|80.5|80.5|80.5|80.3|
|STE-diff|82.9|82.5|82.3|82.5|


Model Accuracy Precision Recall F-measure

PV 75.4 74.9 74.3 74.3

STE-same 80.4 80.3 80.4 80.2

TDE 80.5 80.5 80.5 80.3

STE-diff 82.9 82.5 82.3 82.5


from various categories without thematic grouping of bigrams, whereas TDE maintains thematic
gathering of bigrams. Table 4 represents three topical representations of bigram words with their
source categories for TDE. The first column represents a religion-related topic, the second is related
to the computer-related topic, and the third is related to politics.

4.5 DOCUMENT CLASSIFICATION FOR WORD EMBEDDING EVALUATION

In this experiment, we create document vectors by TF-IDF weighted averaging of each word embeddings under every document, using equation (17). For both models, the base (STE-diff) and the
proposed (TDE), we use the 20Newsgroup dataset with the same parameter settings mentioned in
section 4.2.


_p(z_ _wx, c˜_ ).Uw, z (17)
_|_
_z=1_

X


_dw = tfidfw ×_


where Uw,z is the learned input projection matrix, c represents context words, wx˜ represents pivots
(concatenating center words w with the global document context ˜x), and tfidfw is the TF-IDF score
of w in the document.
For document classification in this section, we use several standard evaluation metrics for the multiclass document classification tasks, e.g., Accuracy, Precision, Recall, and F-measure for LinearSVM
classifier. We compare scores obtained by TDE with the obtained scores by PV, STE-diff, and STEsame models published by Shi et al. (2017). Table 5 represents obtained scores using evaluation
metrics in the classification task, where TDE beats STE-same and PV, but STE-diff performs better
in this case. TDE achieves better topic coherence than STE-diff, but STE-diff achieves better document classification, which we see a similar analogy between STE-same and STE-diff in Shi et al.
(2017)’s work.

5 CONCLUSION

TDE: Topical Document Embedding is a jointly learning word-topic and document embeddings
method, which learns document embedding on the fly simultaneously with the jointly learning process of the topic-specific word embeddings by taking document vector as a global context. As a
result, TDE captures better syntactic and semantic properties by utilizing word, topic, and documentlevel information. It poses the simplest form of document embedding without imposing complex
inference methods for document vectors. As such, each document vector is created by a simple
weighted averaging of the topic-specific word embeddings, which is effective regardless of document size. Moreover, using the mask-out/drop-out corruption mechanism enhances learning speed
and ensures quality embeddings as well. Our extensive experimental findings reveal that besides
the document embeddings, the proposed joint learning method also captures better topical word
embeddings over the base model.

6 REPRODUCIBILITY STATEMENT

Model source code written in C programming language, to run the code need to type ./go.sh in
the terminal (optional: for creating make file type make CFLAG=”-lm -pthread -O3 -Wall -funroll_loops”) . Model parameters can be reset in this file (e.g., training dataset). All datasets used in_
this research are available in preprocessed form (use ready for the model) in the data folder of the
package (Model sourceCode Datasets). The model output should be reproduced effortlessly in a
Linux machine with GCC C++ compiler installed.


-----

For model evaluaiton, we have added evaluation source code files ste d2vc scws similarity.py and
_visualize.ipynb. The evaluation procedure requires python-2.7, numpy-1.16.1, scikit-learn-0.17._
For convenience, we also included obtained evaluation results and detail analysis for embedding
visualization and pearson´s correlation in the obtained results subfolder for both base and proposed
models.

REFERENCES

Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence
embeddings. ICLR, 2017.

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Compu_tational Linguistics, 6:483–495, 2018._

Yoshua Bengio, R´ejean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic
language model. The journal of machine learning research, 3:1137–1155, 2003.

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of
_machine Learning research, 3:993–1022, 2003._

DevendrKnowledge-based word sense disambiguation using topic modelsa Singh Chaplot and Ruslan Salakhutdinov. Knowledge-based word sense disambiguation using topic model. In Thirty_Second AAAI Conference on Artificial Intelligence, 2018._

Minmin Chen. Efficient vector representation for documents through corruption. ICLR, 2017.

Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A
library for large linear classification. the Journal of machine Learning research, 9:1871–1874,
2008.

Vivek Gupta, Harish Karnick, Ashendra Bansal, and Pradhuman Jhala. Product classification in
e-commerce using distributional semantics. arXiv:1606.06083, 2016.

Vivek Gupta, Ankit Saw, Pegah Nokhiz, Harshit Gupta, and Partha Talukdar. Improving document
classification with multi-sense embeddings. arXiv:1911.07918, 2019.

Junxian He, Zhiting Hu, Taylor Berg-Kirkpatrick, Ying Huang, and Eric P Xing. Efficient correlated
topic modeling with topic embedding. In Proceedings of the 23rd ACM SIGKDD International
_Conference on Knowledge Discovery and Data Mining, pp. 225–233, 2017._

Thomas Hofmann. Probabilistic latent semantic analysis. arXiv:1301.6705, 2013.

Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual
_Meeting of the Association for Computational Linguistics, volume 1, pp. 873–882, 2012._

Jey Han Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically
evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the
_European Chapter of the Association for Computational Linguistics, pp. 530–539, 2014._

Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Interna_tional conference on machine learning, pp. 1188–1196. PMLR, 2014._

Qi Liu, Matt J Kusner, and Phil Blunsom. A survey on contextual embeddings. arXiv:2003.07278,
2020.

Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Topical word embeddings. In Twenty_ninth AAAI conference on artificial intelligence, pp. 2418–2424, 2015._

Junyu Luo, Min Yang, Ying Shen, Qiang Qu, and Haixia Chai. Learning document embeddings with
crossword prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume
33(01), pp. 9993–9994, 2019.


-----

Dheeraj Mekala, Vivek Gupta, Bhargavi Paranjape, and Harish Karnick. Scdv: Sparse composite
document vectors using soft clustering over distributional representations. arXiv:1612.06778,
2016.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv:1301.3781, 2013a.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information pro_cessing systems, pp. 3111–3119, 2013b._

Christopher E Moody. Mixing dirichlet topic models and word embeddings to make lda2vec.
_arXiv:1605.02019, 2016._

David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic
coherence. In Human language technologies: The 2010 annual conference of the North American
_chapter of the association for computational linguistics, pp. 100–108, 2010._

Liqiang Niu, Xinyu Dai, Jianbing Zhang, and Jiajun Chen. Topic2vec: learning distributed representations of topics. In 2015 International conference on asian language processing (IALP), pp.
193–196. IEEE, 2015.

Cyrus Shaoul. The westbury lab wikipedia corpus. Edmonton, AB: University of Alberta, 131, 2010.

Bei Shi, Wai Lam, Shoaib Jameel, Steven Schockaert, and Kwun Ping Lai. Jointly learning word
embeddings and latent topics. In Proceedings of the 40th International ACM SIGIR Conference
_on Research and Development in Information Retrieval, pp. 375–384, 2017._

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
_learning research, 9(11), 2008._

Yin Zhang, Rong Jin, and Zhi-Hua Zhou. Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 1(1-4):43–52, 2010.

He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal
transport. ICLR, 2021.


-----

