SAL Report

Distributed Representations of Words and Phrases and their Compositionality

T Mikolov, I Sutskever, K Chen, GS Corrado… - Advances in Neural …, 2013

Efficient Estimation of Word Representations in Vector Space

T Mikolov, K Chen, G Corrado, J Dean - arXiv preprint arXiv:1301.3781, 2013

Raw_data

Skip-gram

Cloth&Accessories

The training objective of the Skip-gram model is to find word representations that are useful for

predicting the surrounding words in a sentence or a document. More formally, given a sequence of

training words w1,w2,w3, . . . ,wT , the objective of the Skip-grammodel is to maximize the average

log probability

Book

Text : T

word : w (represeted by a d dimenstional vector)

context : w(-k)...w(-1)w(1)...w(k)

word in context : w(c)

Theta : model parameter

p(w(c)|w) : probability of w(c) when w occur

C(w) : in Text T, the context of w, w(c) in C(w)

V : vocabulary

D : all w and their C(w)

Electronic

Music

Assumption:

Similar Word Similar Context

Sports&Outdoors

For computationally efficiency:

Hierarchical Softmax,

Negative Sampling

Results

Electronic

Book

Cloth

Category: elec result...

total:2000

true positive:787 false positive:300

false negative:213 true negative700

positive precision:0.7240110395584176

negative precision:0.7667031763417306

positive recall:0.787

negative recall:0.7

Accuracy:0.7435

Category: elec result...

total:2000

true positive:734 false positive:248

false negative:266 true negative752

positive precision:0.7474541751527495

negative precision:0.7387033398821218

positive recall:0.734

negative recall:0.752

Accuracy:0.743

Category: elec result...

total:2000

true positive:741 false positive:270

false negative:259 true negative730

positive precision:0.7329376854599406

negative precision:0.7381193124368048

positive recall:0.741

negative recall:0.73

Accuracy:0.7355

Category: book result...

total:2000

true positive:663 false positive:223

false negative:337 true negative777

positive precision:0.7483069977426636

negative precision:0.6974865350089766

positive recall:0.663

negative recall:0.777

Accuracy:0.72

Category: cloth result...

total:2000

true positive:870 false positive:256

false negative:130 true negative744

positive precision:0.7726465364120781

negative precision:0.851258581235698

positive recall:0.87

negative recall:0.744

Accuracy:0.807

Category: book result...

total:2000

true positive:657 false positive:196

false negative:343 true negative804

positive precision:0.7702227432590856

negative precision:0.7009590235396687

positive recall:0.657

negative recall:0.804

Accuracy:0.7305

Category: book result...

total:2000

true positive:625 false positive:230

false negative:375 true negative770

positive precision:0.7309941520467836

negative precision:0.6724890829694323

positive recall:0.625

negative recall:0.77

Accuracy:0.6975

Category: cloth result...

total:2000

true positive:839 false positive:224

false negative:161 true negative776

positive precision:0.7892756349952963

negative precision:0.8281750266808965

positive recall:0.839

negative recall:0.776

Accuracy:0.8075

Category: cloth result...

total:2000

true positive:809 false positive:404

false negative:191 true negative596

positive precision:0.6669414674361088

negative precision:0.7573062261753494

positive recall:0.809

negative recall:0.596

Accuracy:0.7025

Music

Sports

Category: sports result...

total:2000

true positive:767 false positive:279

false negative:233 true negative721

positive precision:0.7332695984703633

negative precision:0.7557651991614256

positive recall:0.767

negative recall:0.721

Accuracy:0.744

Category: sports result...

total:2000

true positive:756 false positive:357

false negative:244 true negative643

positive precision:0.6792452830188679

negative precision:0.7249154453213078

positive recall:0.756

negative recall:0.643

Accuracy:0.6995

Category: music result...

total:2000

true positive:641 false positive:248

false negative:359 true negative752

positive precision:0.7210348706411699

negative precision:0.6768676867686768

positive recall:0.641

negative recall:0.752

Accuracy:0.6965

Category: music result...

total:2000

true positive:691 false positive:264

false negative:309 true negative736

positive precision:0.7235602094240837

negative precision:0.7043062200956938

positive recall:0.691

negative recall:0.736

Accuracy:0.7135

Category: sports result...

total:2000

true positive:743 false positive:308

false negative:257 true negative692

positive precision:0.7069457659372027

negative precision:0.7291886195995785

positive recall:0.743

negative recall:0.692

Accuracy:0.7175

Category: music result...

total:2000

true positive:714 false positive:286

false negative:286 true negative714

positive precision:0.714

negative precision:0.714

positive recall:0.714

negative recall:0.714

Accuracy:0.714