1. NCCU Academic Hub
  2. Publications
  3. College of Informatics
  4. Proceedings
  5. Item

Publications-Proceedings

Article View/Open

Publication Export

Google ScholarTM

NCCU Library

Citation Infomation

Related Publications in TAIR

題名 MSD-1030: A Well-built Multi-Sense Evaluation Dataset for Sense Representation Models
作者 黃瀚萱
Huang, Hen-Hsen
Yen, Ting-Yu
Lee, Yang-Yin
Shiue, Yow-Ting
Shiue, Yow-Ting
貢獻者 資科系
關鍵詞 semantics ; evaluation methodologies ; crowdsourcing
日期 2020-05
上傳時間 4-Jun-2021 14:42:41 (UTC+8)
摘要 Sense embedding models handle polysemy by giving each distinct meaning of a word form a separate representation. They are considered improvements over word models, and their effectiveness is usually judged with benchmarks such as semantic similarity datasets. However, most of these datasets are not designed for evaluating sense embeddings. In this research, we show that there are at least six concerns about evaluating sense embeddings with existing benchmark datasets, including the large proportions of single-sense words and the unexpected inferior performance of several multi-sense models to their single-sense counterparts. These observations call into serious question whether evaluations based on these datasets can reflect the sense model’s ability to capture different meanings. To address the issues, we propose the Multi-Sense Dataset (MSD-1030), which contains a high ratio of multi-sense word pairs. A series of analyses and experiments show that MSD-1030 serves as a more reliable benchmark for sense embeddings. The dataset is available at http://nlg.csie.ntu.edu.tw/nlpresource/MSD-1030/.
關聯 Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), European Language Resources Association, pp.5802-5809
資料類型 conference
dc.contributor 資科系
dc.creator (作者) 黃瀚萱
dc.creator (作者) Huang, Hen-Hsen
dc.creator (作者) Yen, Ting-Yu
dc.creator (作者) Lee, Yang-Yin
dc.creator (作者) Shiue, Yow-Ting
dc.creator (作者) Shiue, Yow-Ting
dc.date (日期) 2020-05
dc.date.accessioned 4-Jun-2021 14:42:41 (UTC+8)-
dc.date.available 4-Jun-2021 14:42:41 (UTC+8)-
dc.date.issued (上傳時間) 4-Jun-2021 14:42:41 (UTC+8)-
dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/135527-
dc.description.abstract (摘要) Sense embedding models handle polysemy by giving each distinct meaning of a word form a separate representation. They are considered improvements over word models, and their effectiveness is usually judged with benchmarks such as semantic similarity datasets. However, most of these datasets are not designed for evaluating sense embeddings. In this research, we show that there are at least six concerns about evaluating sense embeddings with existing benchmark datasets, including the large proportions of single-sense words and the unexpected inferior performance of several multi-sense models to their single-sense counterparts. These observations call into serious question whether evaluations based on these datasets can reflect the sense model’s ability to capture different meanings. To address the issues, we propose the Multi-Sense Dataset (MSD-1030), which contains a high ratio of multi-sense word pairs. A series of analyses and experiments show that MSD-1030 serves as a more reliable benchmark for sense embeddings. The dataset is available at http://nlg.csie.ntu.edu.tw/nlpresource/MSD-1030/.
dc.format.extent 1750851 bytes-
dc.format.mimetype application/pdf-
dc.relation (關聯) Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), European Language Resources Association, pp.5802-5809
dc.subject (關鍵詞) semantics ; evaluation methodologies ; crowdsourcing
dc.title (題名) MSD-1030: A Well-built Multi-Sense Evaluation Dataset for Sense Representation Models
dc.type (資料類型) conference