關鍵詞偵測方法的比較與應用

Abstract

近年來由於文本被大量數位化，使得文字探勘（Text Mining）成為熱門研究領域，愈來愈多研究藉由量化技術找出文字涵意，提供專家意見不同角度的語意解讀。文本在經過結構化（Structurization）後，根據不同需求如關鍵詞擷取、尋找潛在文本主題、情感分析、輿情分析等，建立統計及機器學習等數位模型。其中關鍵詞擷取可用於解讀作者想法、提升閱讀效率、掌握寫作風格以及文章出版時空背景的變化。本研究也以決定關鍵詞為研究目標，除了提出一種非監督學習的統計方法，也使用中文文本評估新方法與幾種常見關鍵詞偵測方法，包括網路流行的TF-IDF (Term Frequency Inverse Document Frequency；詞頻與文本頻率)、統計分析的羅吉斯迴歸（Logistic Regression）、常見的機器學習模型。實證分析採用《人民日報》、《新青年雜誌》兩個白話文的文本，其中《人民日報》為1971-1989年與人權有關的514篇報導，《新青年》則是第七卷（1919年）、第八卷（1920年），這些文本的篇幅大約都介於40～60萬字。先由人文學者標記出各文本的關鍵詞，將其視為標準答案，再套用上述三種方法選取可能的關鍵詞，再比較上述方法與專家意見的差異及準確率；另外，我們也將比較人工挑選、自動挑選關鍵詞的差異，並探索兼具兩種方法優點的可能。

Text Mining has become one of the popular research areas after the IBM proposed the term Big Data in 2010. Since then many texts are being digitalized and more scholars are devoted in developing quantitative tools for giving texts semantic meaning without the help of human experts. This greatly increases the efficiency of reading a hugh amount of texts provided that the texts are properly structurized. The structurization of texts includes quite a few steps, such as keyword extraction and sentiment analysis. The keyword extraction is critical and the keywords can be used to summarize an article and compare two authors’ writing styles.\nThe goal of this study is to propose a new unsupervised method for extracting keywords and compare it to some frequently used methods, including term frequency inverse document frequency (TF-IDF), logistic regression, machine learning models. In the empirical analysis, we considered three modern Chinese texts, one from People’s Daily (514 articles in 1971-1989) and two from New Youth Magazine (volumes 7 and 8 in 1919-1920). The numbers of words in all texts are approximately 400,000 to 600,000. We asked historical scholars to pick up keywords from these three texts and treat them as the true keywords. Then, we applied different keyword extraction methods to these texts and compared their results. We found that the proposed method has the best performance among all supervised methods and it is competitive to the supervised methods.