Transformer 應用於中文文章摘要

Abstract

自從Transformer 發表後，無疑為自然語言處理領域的立下新的里程碑，許多的模型也因應而起，分別在各自然語言處理項目有傑出的表現。如此強大的模型多數背後依靠巨量的參數運算，但各模型皆以英文為發展主軸，我們很難訓練一個一樣強的中文模型，在缺乏原生中文模型的情況下，我們利用現有的資源及模型訓練機器做中文文章摘要，使用BERT 及GPT-2，搭配中研院中文詞知識庫小組的中文模型，並採用新聞資料進行訓練。先透過BERT 從原文章獲得抽取式摘要，使文章篇幅縮短並保留住重要資訊，接著使用GPT-2 從抽取過的摘要中再進行生成式摘要，去除掉重複的資訊並使語句更平順。在我們的實驗中，我們獲得了不錯的中文文章摘要，證明這個方法是有效的。

Since the publication of Transformer, it has undoubtedly set a new milestone in the field of Natural Language Processing, and many models have also been released depending on it and performed outstandingly in various Natural Language Processing tasks. Most of such powerful models rely on a large number of parameter operations, but most of them are developed in English, and it is difficult for us to train a Chinese model that is equally strong. In the absence of native Chinese models, we use existing resources and model to train the machine to make Chinese article summaries: using BERT and GPT-2 model, with the Chinese model of the Chinese Knowledge and Information Processing of the Academia Sinica of Taiwan, and using news datasets for training. First, use BERT to obtained an extractive summarization from the original article, so that the length of the article is shortened and important information is retained, then use GPT-2 to generate a summarization from the extracted summary to remove duplicate information and make the sentence smoother. In our experiments, we obtained decent Chinese article summaries, proving that this method is effective.