All Articles

My Understanding of Word2Vec

word2vec

The appearance of word2vec has greatly promoted the development of natural language processing. It is a classic model for generating word embeddings. In this article, I will explain my understanding of word2vec.

Objective

The objective of word2vec is to learn word embeddings from a large corpus of text. Word embeddings are dense vectors of real numbers, one for each word in the vocabulary. In the context of neural networks, they are used to represent words in such a way that words that have similar meanings will have similar representation. There are two main methods to train word embeddings: CBOW (Continuous Bag-of-Words Model) and Skip Gram (Continuous Skip-gram Model).

  1. CBOW: Predict the current word based on the context
  2. Skip Gram: Predict the context based on the current word

In both training methods, the input is a one-hot vector representing the current word, and the output is a probability distribution over the vocabulary. The last hidden layer in the middle layer is the word embedding.

The most attractive feature of word2vec is that it can capture both the semantic and syntactic information of words. The semantic information refers to the meaning of words, such as the relationship between king and queue. The syntactic information refers to the relationship between words, such as the relationship between small and smallest.

king⃗−queen⃗≈man⃗−woman⃗\vec{king} - \vec{queen} \approx \vec{man} - \vec{woman}

smallest⃗−small⃗≈biggest⃗−big⃗\vec{smallest} - \vec{small} \approx \vec{biggest} - \vec{big}

The process of training word embeddings is similar to the process of baby language learning. The meaning of each word is inferred from the context. Although it does not solve the actual meaning of each word, it is theoretically applicable to various languages - analytic language (such as Chinese) / synthetic language (fusional language, such as English; agglutinative language, such as Japanese).

Important Functions

Annotations

  • JJ : loss function
  • PP : probability
  • DD : training corpus
  • VV : vocabulary
  • cc : context window size
  • ww : word
  • w⃗\vec{w} : word vector

Loss Function of CBOW

JCBOW=−∑wt∈Dlog⁡P(wt∣wt−c,⋯ ,wt−1,wt+1,⋯ ,wt+c)\mathbf{J_{CBOW}} = - \sum_{w_{t} \in D} \log P(w_{t} \mid w_{t-c}, \cdots, w_{t-1}, w_{t+1}, \cdots, w_{t+c})

P(wt∣wt−c,⋯ ,wt−1,wt+1,⋯ ,wt+c)=ewt⃗T∑−c≤j≤c,j≠0wt+j⃗∑w∈Vew⃗T∑−c≤j≤c,j≠0wt+j⃗\mathbf{P(w_{t} \mid w_{t-c}, \cdots, w_{t-1}, w_{t+1}, \cdots, w_{t+c})} = \frac{e^ {\vec{w_{t}}^{T} \sum_{-c \leq j \leq c, j \neq 0} \vec{w_{t+j}}}}{\sum_{w \in V} e^ {\vec{w}^{T} \sum_{-c \leq j \leq c, j \neq 0} \vec{w_{t+j}}}}

Loss Function of Skip Gram

JSkipGram=−∑wt∈D∑−c≤j≤c,j≠0log⁡P(wt+j∣wt)\mathbf{J_{SkipGram}} = - \sum_{w_{t} \in D} \sum_{-c \leq j \leq c, j \neq 0} \log P(w_{t+j} \mid w_{t})

P(wt+j∣wt)=ewt⃗Twt+j⃗∑w∈Vew⃗Twt+j⃗\mathbf{P(w_{t+j} \mid w_{t})} = \frac{e^{ \vec{w_{t}}^{T} \vec{w_{t+j}}}}{\sum_{w \in V} e^{ \vec{w}^{T} \vec{w_{t+j}}}}

Published Sep 18, 2014

Flying code monkey