**This is an old revision of the document!**


Word2Vec

Intro

Word2Vec can guess a word’s association with other words, or cluster documents and define them by topic. It makes qualities into quantities, and similar things and ideas are shown to be “close” in its 500-dimension vectorspace.

Word2Vec is not classified as “deep learning” because it is only a 2-layer neural net.

Notation

Algebraic notation

knee - leg = elbow - arm

English logic

knee is to leg as elbow is to arm

Logical analogy notation

knee : leg :: elbow : arm

Input → text corpus Output → set of vectors (neural word embeddings)

More research: Cosine similarity, dot product equation

Models

Continuous bag of words (CBOW) model

Using context to predict a target word. Faster.

Skip-gram model

Using a word to predict a target context. Produces more accurate results on large datasets.

“Just as Van Gogh’s painting of sunflowers is a two-dimensional mixture of oil on canvas that represents vegetable matter in a three-dimensional space in Paris in the late 1880s, so 500 numbers arranged in a vector can represent a word or group of words.”

Each word is a point in a 500-dimensional vectorspace.

More than three layers in a neural network (including input and output) qualifies as “deep” learning. Deep means more than one hidden layer.

Implementation

Word2Vec can be implemented in DL4J, TensorFlow

Examples

Rome - Italy = Beijing - China, so Rome - Italy + China = Beijing

king : queen :: man : woman

house : roof :: castle : [dome, bell_tower, spire, crenellations, turrets]

China : Taiwan :: Russia : [Ukraine, Moscow, Moldova, Armenia]

mind/word2vec.1474842842.txt.gz · Last modified: 2016/09/25 22:34 by bayb2
Back to top
CC Attribution-Share Alike 4.0 International
chimeric.de = chi`s home Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0