學術產出-Theses
Article View/Open
Publication Export
-
題名 詞彙向量的理論與評估基於矩陣分解與神經網絡
Theory and evaluation of word embedding based on matrix factorization and neural network作者 張文嘉
Jhang, Wun Jia貢獻者 翁久幸<br>馬偉雲
Weng, Chiu Hsing<br>Ma, Wei Yun
張文嘉
Jhang, Wun Jia關鍵詞 矩陣分解
初始值
自然語言處理
神經網絡
Matrix factorization
Initalization
Natural language processing
Neural network日期 2017 上傳時間 2-Mar-2017 11:10:02 (UTC+8) 摘要 隨著機器學習在越來越多任務中有突破性的發展，特別是在自然語言處理問題上，得到越來越多的關注，近年來，詞向量是自然語言處理研究中最令人興奮的部分之一。在這篇論文中，我們討論了兩種主要的詞向量學習方法。一種是傳統的矩陣分解，如奇異值分解，另一種是基於神經網絡模型（具有負採樣的Skip-gram模型（Mikolov等人提出，2013），它是一種迭代演算法。我們提出一種方法來挑選初始值，透過使用奇異值分解得到的詞向量當作是Skip-gram模型的初始直，結果發現替換較佳的初始值，在某些自然語言處理的任務中得到明顯的提升。
Recently, word embedding is one of the most exciting part of research in natural language processing. In this thesis, we discuss the two major learning approaches for word embedding. One is traditional matrix factorization like singular value decomposition, the other is based on neural network model (e.g. the Skip-gram model with negative sampling (Mikolov et al., 2013b)) which is an iterative algorithm. It is known that an iterative process is sensitive to initial starting values. We present an approach for implementing the Skip-gram model with negative sampling from a given initial value that is using singular value decomposition. Furthermore, we show that refined initial starting points improve the analogy task and succeed in capturing fine-gained semantic and syntactic regularities using vector arithmetic.參考文獻 Marco Baroni, Georgiana Dinu, and German Kruszewski. Don`t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1), pages 238-247, 2014.Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1-127, 2009.Christophe Biernacki, Gilles Celeux, and Gerard Govaert. Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models. Computational Statistics & Data Analysis, 41(3):561-575, 2003.Paul S Bradley and Usama M Fayyad. Refining initial points for k-means clustering. In ICML, volume 98, pages 91-99. Citeseer, 1998.Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics, 2012.John Caron. Experiments with lsa scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391, 1990.Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.`s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL (2), pages 302-308, 2014a.Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization.In Advances in Neural Information Processing Systems, pages 2177-2185, 2014b.Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171-180, 2014.Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225, 2015.Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104-113, 2013.Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. Evaluation in information retrieval. Introduction to information retrieval, pages 151-175, 2008.Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ecient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013b.Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pages 746-751, 2013c.Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346. ACM, 2011.Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014.Gerard Salton and Michael J McGill. Introduction to modern information retrieval. 1986.Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In ACL (1), pages 455-465, 2013a.Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer, 2013b.Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 41-47. ACM, 2003.Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013. 描述 碩士
國立政治大學
統計學系
103354027資料來源 http://thesis.lib.nccu.edu.tw/record/#G0103354027 資料類型 thesis dc.contributor.advisor 翁久幸<br>馬偉雲 zh_TW dc.contributor.advisor Weng, Chiu Hsing<br>Ma, Wei Yun en_US dc.contributor.author (Authors) 張文嘉 zh_TW dc.contributor.author (Authors) Jhang, Wun Jia en_US dc.creator (作者) 張文嘉 zh_TW dc.creator (作者) Jhang, Wun Jia en_US dc.date (日期) 2017 en_US dc.date.accessioned 2-Mar-2017 11:10:02 (UTC+8) - dc.date.available 2-Mar-2017 11:10:02 (UTC+8) - dc.date.issued (上傳時間) 2-Mar-2017 11:10:02 (UTC+8) - dc.identifier (Other Identifiers) G0103354027 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/107015 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 統計學系 zh_TW dc.description (描述) 103354027 zh_TW dc.description.abstract (摘要) 隨著機器學習在越來越多任務中有突破性的發展，特別是在自然語言處理問題上，得到越來越多的關注，近年來，詞向量是自然語言處理研究中最令人興奮的部分之一。在這篇論文中，我們討論了兩種主要的詞向量學習方法。一種是傳統的矩陣分解，如奇異值分解，另一種是基於神經網絡模型（具有負採樣的Skip-gram模型（Mikolov等人提出，2013），它是一種迭代演算法。我們提出一種方法來挑選初始值，透過使用奇異值分解得到的詞向量當作是Skip-gram模型的初始直，結果發現替換較佳的初始值，在某些自然語言處理的任務中得到明顯的提升。 zh_TW dc.description.abstract (摘要) Recently, word embedding is one of the most exciting part of research in natural language processing. In this thesis, we discuss the two major learning approaches for word embedding. One is traditional matrix factorization like singular value decomposition, the other is based on neural network model (e.g. the Skip-gram model with negative sampling (Mikolov et al., 2013b)) which is an iterative algorithm. It is known that an iterative process is sensitive to initial starting values. We present an approach for implementing the Skip-gram model with negative sampling from a given initial value that is using singular value decomposition. Furthermore, we show that refined initial starting points improve the analogy task and succeed in capturing fine-gained semantic and syntactic regularities using vector arithmetic. en_US dc.description.tableofcontents List of Figures 4List of Tables 51 Introduction 62 Background Theory 82.1 Co-occurrence Matrix 92.2 Singular Value Decomposition 102.3 Skip-gram Model with Negative Sampling 103 Combination of SVD and SGNS 124 Experimental Setup 144.1 Hyperparameters 144.2 Training Details 164.3 Test Data Set 175 Results and Discussion 185.1 Main Results 185.2 Error Analysis 256 Conclusion 27Bibliography 28 zh_TW dc.format.extent 876806 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0103354027 en_US dc.subject (關鍵詞) 矩陣分解 zh_TW dc.subject (關鍵詞) 初始值 zh_TW dc.subject (關鍵詞) 自然語言處理 zh_TW dc.subject (關鍵詞) 神經網絡 zh_TW dc.subject (關鍵詞) Matrix factorization en_US dc.subject (關鍵詞) Initalization en_US dc.subject (關鍵詞) Natural language processing en_US dc.subject (關鍵詞) Neural network en_US dc.title (題名) 詞彙向量的理論與評估基於矩陣分解與神經網絡 zh_TW dc.title (題名) Theory and evaluation of word embedding based on matrix factorization and neural network en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) Marco Baroni, Georgiana Dinu, and German Kruszewski. Don`t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1), pages 238-247, 2014.Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1-127, 2009.Christophe Biernacki, Gilles Celeux, and Gerard Govaert. Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models. Computational Statistics & Data Analysis, 41(3):561-575, 2003.Paul S Bradley and Usama M Fayyad. Refining initial points for k-means clustering. In ICML, volume 98, pages 91-99. Citeseer, 1998.Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics, 2012.John Caron. Experiments with lsa scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391, 1990.Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.`s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL (2), pages 302-308, 2014a.Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization.In Advances in Neural Information Processing Systems, pages 2177-2185, 2014b.Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171-180, 2014.Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225, 2015.Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104-113, 2013.Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. Evaluation in information retrieval. Introduction to information retrieval, pages 151-175, 2008.Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ecient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013b.Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pages 746-751, 2013c.Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346. ACM, 2011.Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014.Gerard Salton and Michael J McGill. Introduction to modern information retrieval. 1986.Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In ACL (1), pages 455-465, 2013a.Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer, 2013b.Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 41-47. ACM, 2003.Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013. zh_TW
