1. NCCU Academic Hub
  2. Publications
  3. College of Commerce
  4. Theses
  5. Item

Publications-Theses

Article View/Open

Publication Export

Google ScholarTM

NCCU Library

Citation Infomation

Related Publications in TAIR

題名 詞彙向量的理論與評估基於矩陣分解與神經網絡
Theory and evaluation of word embedding based on matrix factorization and neural network
作者 張文嘉
Jhang, Wun Jia
貢獻者 翁久幸<br>馬偉雲
Weng, Chiu Hsing<br>Ma, Wei Yun
張文嘉
Jhang, Wun Jia
關鍵詞 矩陣分解
初始值
自然語言處理
神經網絡
Matrix factorization
Initalization
Natural language processing
Neural network
日期 2017
上傳時間 2-Mar-2017 11:10:02 (UTC+8)
摘要 隨著機器學習在越來越多任務中有突破性的發展，特別是在自然語言處理問題上，得到越來越多的關注，近年來，詞向量是自然語言處理研究中最令人興奮的部分之一。在這篇論文中，我們討論了兩種主要的詞向量學習方法。一種是傳統的矩陣分解，如奇異值分解，另一種是基於神經網絡模型（具有負採樣的Skip-gram模型（Mikolov等人提出，2013），它是一種迭代演算法。我們提出一種方法來挑選初始值，透過使用奇異值分解得到的詞向量當作是Skip-gram模型的初始直，結果發現替換較佳的初始值，在某些自然語言處理的任務中得到明顯的提升。
Recently, word embedding is one of the most exciting part of research in natural language processing. In this thesis, we discuss the two major learning approaches for word embedding. One is traditional matrix factorization like singular value decomposition, the other is based on neural network model (e.g. the Skip-gram model with negative sampling (Mikolov et al., 2013b)) which is an iterative algorithm. It is known that an iterative process is sensitive to initial starting values. We present an approach for implementing the Skip-gram model with negative sampling from a given initial value that is using singular value decomposition. Furthermore, we show that refined initial starting points improve the analogy task and succeed in capturing fine-gained semantic and syntactic regularities using vector arithmetic.
參考文獻 Marco Baroni, Georgiana Dinu, and German Kruszewski. Don`t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1), pages 238-247, 2014.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1-127, 2009.
Christophe Biernacki, Gilles Celeux, and Gerard Govaert. Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models. Computational Statistics & Data Analysis, 41(3):561-575, 2003.
Paul S Bradley and Usama M Fayyad. Refining initial points for k-means clustering. In ICML, volume 98, pages 91-99. Citeseer, 1998.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics, 2012.
John Caron. Experiments with lsa scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391, 1990.
Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.
Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.`s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL (2), pages 302-308, 2014a.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization.In Advances in Neural Information Processing Systems, pages 2177-2185, 2014b.
Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171-180, 2014.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225, 2015.
Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104-113, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. Evaluation in information retrieval. Introduction to information retrieval, pages 151-175, 2008.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ecient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pages 746-751, 2013c.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346. ACM, 2011.
Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014.
Gerard Salton and Michael J McGill. Introduction to modern information retrieval. 1986.
Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.
Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In ACL (1), pages 455-465, 2013a.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer, 2013b.
Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 41-47. ACM, 2003.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.
Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013.
描述 碩士
國立政治大學
統計學系
103354027
資料來源 http://thesis.lib.nccu.edu.tw/record/#G0103354027
資料類型 thesis
dc.contributor.advisor 翁久幸<br>馬偉雲zh_TW
dc.contributor.advisor Weng, Chiu Hsing<br>Ma, Wei Yunen_US
dc.contributor.author (Authors) 張文嘉zh_TW
dc.contributor.author (Authors) Jhang, Wun Jiaen_US
dc.creator (作者) 張文嘉zh_TW
dc.creator (作者) Jhang, Wun Jiaen_US
dc.date (日期) 2017en_US
dc.date.accessioned 2-Mar-2017 11:10:02 (UTC+8)-
dc.date.available 2-Mar-2017 11:10:02 (UTC+8)-
dc.date.issued (上傳時間) 2-Mar-2017 11:10:02 (UTC+8)-
dc.identifier (Other Identifiers) G0103354027en_US
dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/107015-
dc.description (描述) 碩士zh_TW
dc.description (描述) 國立政治大學zh_TW
dc.description (描述) 統計學系zh_TW
dc.description (描述) 103354027zh_TW
dc.description.abstract (摘要) 隨著機器學習在越來越多任務中有突破性的發展，特別是在自然語言處理問題上，得到越來越多的關注，近年來，詞向量是自然語言處理研究中最令人興奮的部分之一。在這篇論文中，我們討論了兩種主要的詞向量學習方法。一種是傳統的矩陣分解，如奇異值分解，另一種是基於神經網絡模型（具有負採樣的Skip-gram模型（Mikolov等人提出，2013），它是一種迭代演算法。我們提出一種方法來挑選初始值，透過使用奇異值分解得到的詞向量當作是Skip-gram模型的初始直，結果發現替換較佳的初始值，在某些自然語言處理的任務中得到明顯的提升。zh_TW
dc.description.abstract (摘要) Recently, word embedding is one of the most exciting part of research in natural language processing. In this thesis, we discuss the two major learning approaches for word embedding. One is traditional matrix factorization like singular value decomposition, the other is based on neural network model (e.g. the Skip-gram model with negative sampling (Mikolov et al., 2013b)) which is an iterative algorithm. It is known that an iterative process is sensitive to initial starting values. We present an approach for implementing the Skip-gram model with negative sampling from a given initial value that is using singular value decomposition. Furthermore, we show that refined initial starting points improve the analogy task and succeed in capturing fine-gained semantic and syntactic regularities using vector arithmetic.en_US
dc.description.tableofcontents List of Figures 4
List of Tables 5
1 Introduction 6
2 Background Theory 8
2.1 Co-occurrence Matrix 9
2.2 Singular Value Decomposition 10
2.3 Skip-gram Model with Negative Sampling 10
3 Combination of SVD and SGNS 12
4 Experimental Setup 14
4.1 Hyperparameters 14
4.2 Training Details 16
4.3 Test Data Set 17
5 Results and Discussion 18
5.1 Main Results 18
5.2 Error Analysis 25
6 Conclusion 27
Bibliography 28		zh_TW
dc.format.extent 876806 bytes-
dc.format.mimetype application/pdf-
dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0103354027en_US
dc.subject (關鍵詞) 矩陣分解zh_TW
dc.subject (關鍵詞) 初始值zh_TW
dc.subject (關鍵詞) 自然語言處理zh_TW
dc.subject (關鍵詞) 神經網絡zh_TW
dc.subject (關鍵詞) Matrix factorizationen_US
dc.subject (關鍵詞) Initalizationen_US
dc.subject (關鍵詞) Natural language processingen_US
dc.subject (關鍵詞) Neural networken_US
dc.title (題名) 詞彙向量的理論與評估基於矩陣分解與神經網絡zh_TW
dc.title (題名) Theory and evaluation of word embedding based on matrix factorization and neural networken_US
dc.type (資料類型) thesisen_US
dc.relation.reference (參考文獻) Marco Baroni, Georgiana Dinu, and German Kruszewski. Don`t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1), pages 238-247, 2014.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1-127, 2009.
Christophe Biernacki, Gilles Celeux, and Gerard Govaert. Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models. Computational Statistics & Data Analysis, 41(3):561-575, 2003.
Paul S Bradley and Usama M Fayyad. Refining initial points for k-means clustering. In ICML, volume 98, pages 91-99. Citeseer, 1998.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics, 2012.
John Caron. Experiments with lsa scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391, 1990.
Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.
Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.`s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL (2), pages 302-308, 2014a.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization.In Advances in Neural Information Processing Systems, pages 2177-2185, 2014b.
Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171-180, 2014.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225, 2015.
Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104-113, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. Evaluation in information retrieval. Introduction to information retrieval, pages 151-175, 2008.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ecient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pages 746-751, 2013c.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346. ACM, 2011.
Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014.
Gerard Salton and Michael J McGill. Introduction to modern information retrieval. 1986.
Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.
Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In ACL (1), pages 455-465, 2013a.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer, 2013b.
Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 41-47. ACM, 2003.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.
Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013.		zh_TW