1. NCCU Academic Hub
  2. Publications
  3. College of Informatics
  4. Theses
  5. Item

Publications-Theses

Article View/Open

Publication Export

Google ScholarTM

NCCU Library

Citation Infomation

Related Publications in TAIR

題名 基於圖預訓練與提示詞學習於推薦系統
Graph-based Pre-training and Prompting for Recommendation Systems
作者 張立暘
Chang, Li-Yang
貢獻者 蔡銘峰
Tsai, Ming-Feng
張立暘
Chang, Li-Yang
關鍵詞 推薦系統
預訓練模型
多模態推薦系統
提示詞學習
圖神經網路
冷啟動推薦系統
日期 2024
上傳時間 4-Sep-2024 14:58:56 (UTC+8)
摘要 本研究旨在探討基於多模態預訓練模型在推薦系統中的應用，特別是利用圖預訓練和提示詞學習技術來提升推薦系統的性能。我們提出了一種創新的方法，結合了圖神經網絡（Graph Neural Networks, GNNs）來捕捉圖的信息和自然語言處理（NLP）學習文本的長距離相依性。這樣的預訓練模型能夠有效捕捉文本和圖結構等多模態數據中的深層次語義和結構信息，從而為推薦系統提供有質量的預訓練模，供下游推薦任務使用。 我們的研究重點之一是提示學習技術，它包括離散提示（硬提示）和連續提示（軟提示）兩種方法。離散提示通過設計固定的詞語或短語來引導模型生成特定輸出，而連續提示則通過學習得到的嵌入向量與預訓練模型的輸入層結合，實現模型的微調。我們發現，連續提示在靈活性和適應性方面具有顯著優勢，特別是在處理複雜多變的推薦場景中，這樣的參數高效微調技術（PEFT）的應用，減少了模型微調的資源需求；並透過遷移學習(transfer learning) 有效的利用預訓練模型中的通用知識，並將其應用於推薦系統任務中。這些技術的結合使我們的系統能夠在冷啟動和一般推薦場景中均展現出優異的表現。 實驗結果顯示，基於圖預訓練和提示詞學習技術的推薦系統在多個評估指標上有不錯的成績，相較於傳統的推薦系統模型無法在冷啟動模型中作使用。特別是在冷啟動場景中，我們的方法顯著提升了多種評估指標，像是命中率（Hit Rate）、平均準確率（Mean Average Precision）、召回率（Recall）和標準折扣累積增益（NCDG）等，顯示出其在處理新用戶和新物品時的強大適應能力。同時，我們的方法在一般推薦場景中也展示了良好的性能，特別是使用圖編碼器時，顯示出圖結構數據在捕捉用戶和物品關係方面的潛力。 總結來說，本研究通過結合圖預訓練與提示詞學習技術，實現了一種創新的多模態推薦系統，並展示了這些技術在提升推薦質量和適應性方面的潛力。未來的工作將集中於進一步優化這些技術，探索更多應用場景，以期為推薦系統的發展提供更強大的技術支持和理論指導。
參考文獻 [1] J. Atwood and D. Towsley. Diffusion-convolutional neural networks, 2016. [2] O. Barkan and N. Koenigstein. Item2vec: Neural item embedding for collaborative filtering, 2017. [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCan- dlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, 2020. [4] C.-M. Chen, M.-F. Tsai, Y.-C. Lin, and Y.-H. Yang. Query-based music recom- mendations via preference embedding. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys ’16, page 79–82, New York, NY, USA, 2016. Association for Computing Machinery. [5] C.-M. Chen, T.-H. Wang, C.-J. Wang, and M.-F. Tsai. Smore: modularize graph embedding for recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, page 582–583, New York, NY, USA, 2019. Association for Computing Machinery. [6] C.-M. Chen, T.-H. Wang, C.-J. Wang, and M.-F. Tsai. Smore: modularize graph embedding for recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, page 582–583, New York, NY, USA, 2019. Association for Computing Machinery. [7] G. de Souza Pereira Moreira, S. Rabhi, J. M. Lee, R. Ak, and E. Oldridge. Trans- formers4rec: Bridging the gap between nlp and sequential / session-based recom- mendation. page 143–153, 2021. [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019. [9] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar ́e, M. Lomeli, L. Hosseini, and H. J ́egou. The faiss library. 2024. [10] S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang. Recommendation as language process- ing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). 2023. [11] A. Grover and J. Leskovec. node2vec: Scalable feature learning for networks, 2016. [12] W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs, 2018. [13] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation, 2020. [14] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua. Neural collaborative filtering, 2017. [15] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks, 2017. [16] O. Kuchaiev and B. Ginsburg. Training deep autoencoders for collaborative filtering, 2017. [17] J. Li, M. Wang, J. Li, J. Fu, X. Shen, J. Shang, and J. McAuley. Text is all you need: Learning language representations for sequential recommendation. 2023. [18] W. Li, Y. Zhang, Y. Sun, W. Wang, W. Zhang, and X. Lin. Approximate nearest neighbor search on high dimensional data — experiments, analyses, and improve- ment (v1.0). 2016. [19] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word repre- sentations in vector space. 2013. [20] J. Ni, J. Li, and J. McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. pages 188–197, Nov. 2019. [21] B. Perozzi, R. Al-Rfou, and S. Skiena. Deepwalk: online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’14. ACM, Aug. 2014. [22] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. 2021. [23] A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. [24] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. [25] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2023. [26] S. Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining, pages 995–1000, 2010. [27] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. 2012. [28] S. Ruder. An overview of gradient descent optimization algorithms, 2017. [29] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conferenceon World Wide Web, WWW ’01, page 285–295, New York, NY, USA, 2001. Association for Computing Machinery. [30] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. 2019. [31] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei. Line: Large-scale infor- mation network embedding. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15. International World Wide Web Conferences Steering Committee, May 2015. [32] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. 2019. [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2023. [34] Z. Wen and Y. Fang. Augmenting low-resource text classification with graph- grounded pre-training and prompting. 2023. [35] J. Weston, H. Yee, and R. J. Weiss. Learning to rank recommendations with the k-order statistic loss. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, page 245–248, New York, NY, USA, 2013. Association for Computing Machinery. [36] J.-H. Yang, C.-M. Chen, C.-J. Wang, and M.-F. Tsai. Hop-rec: high-order prox- imity for implicit recommendation. In Proceedings of the 12th ACM Conference yon Recommender Systems, RecSys ’18, page 140–144, New York, NY, USA, 2018. Association for Computing Machinery. [37] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, July 2022.
描述 碩士
國立政治大學
資訊科學系
110753140
資料來源 http://thesis.lib.nccu.edu.tw/record/#G0110753140
資料類型 thesis
dc.contributor.advisor 蔡銘峰zh_TW
dc.contributor.advisor Tsai, Ming-Fengen_US
dc.contributor.author (Authors) 張立暘zh_TW
dc.contributor.author (Authors) Chang, Li-Yangen_US
dc.creator (作者) 張立暘zh_TW
dc.creator (作者) Chang, Li-Yangen_US
dc.date (日期) 2024en_US
dc.date.accessioned 4-Sep-2024 14:58:56 (UTC+8)-
dc.date.available 4-Sep-2024 14:58:56 (UTC+8)-
dc.date.issued (上傳時間) 4-Sep-2024 14:58:56 (UTC+8)-
dc.identifier (Other Identifiers) G0110753140en_US
dc.identifier.uri (URI) https://nccur.lib.nccu.edu.tw/handle/140.119/153374-
dc.description (描述) 碩士zh_TW
dc.description (描述) 國立政治大學zh_TW
dc.description (描述) 資訊科學系zh_TW
dc.description (描述) 110753140zh_TW
dc.description.abstract (摘要) 本研究旨在探討基於多模態預訓練模型在推薦系統中的應用，特別是利用圖預訓練和提示詞學習技術來提升推薦系統的性能。我們提出了一種創新的方法，結合了圖神經網絡（Graph Neural Networks, GNNs）來捕捉圖的信息和自然語言處理（NLP）學習文本的長距離相依性。這樣的預訓練模型能夠有效捕捉文本和圖結構等多模態數據中的深層次語義和結構信息，從而為推薦系統提供有質量的預訓練模，供下游推薦任務使用。 我們的研究重點之一是提示學習技術，它包括離散提示（硬提示）和連續提示（軟提示）兩種方法。離散提示通過設計固定的詞語或短語來引導模型生成特定輸出，而連續提示則通過學習得到的嵌入向量與預訓練模型的輸入層結合，實現模型的微調。我們發現，連續提示在靈活性和適應性方面具有顯著優勢，特別是在處理複雜多變的推薦場景中，這樣的參數高效微調技術（PEFT）的應用，減少了模型微調的資源需求；並透過遷移學習(transfer learning) 有效的利用預訓練模型中的通用知識，並將其應用於推薦系統任務中。這些技術的結合使我們的系統能夠在冷啟動和一般推薦場景中均展現出優異的表現。 實驗結果顯示，基於圖預訓練和提示詞學習技術的推薦系統在多個評估指標上有不錯的成績，相較於傳統的推薦系統模型無法在冷啟動模型中作使用。特別是在冷啟動場景中，我們的方法顯著提升了多種評估指標，像是命中率（Hit Rate）、平均準確率（Mean Average Precision）、召回率（Recall）和標準折扣累積增益（NCDG）等，顯示出其在處理新用戶和新物品時的強大適應能力。同時，我們的方法在一般推薦場景中也展示了良好的性能，特別是使用圖編碼器時，顯示出圖結構數據在捕捉用戶和物品關係方面的潛力。 總結來說，本研究通過結合圖預訓練與提示詞學習技術，實現了一種創新的多模態推薦系統，並展示了這些技術在提升推薦質量和適應性方面的潛力。未來的工作將集中於進一步優化這些技術，探索更多應用場景，以期為推薦系統的發展提供更強大的技術支持和理論指導。zh_TW
dc.description.tableofcontents 第一章 緒論 1 1.1 前言 1 第二章 相關文獻探討 3 2.1 傳統推薦系統算法 3 2.1.1 基於協同過濾的推薦系統算法 4 2.1.2 圖表示學習法的推薦系統算法 5 2.1.3 冷啟動推薦系統算法 6 2.2 預訓練模型 6 2.2.1 多模態預訓練模型應用於推薦系統 6 2.3 提示學習技術 7 2.3.1 離散提示 7 2.3.2 連續提示 8 第三章 研究方法 10 3.1 預訓練階段 10 3.1.1 圖編碼器 11 3.1.2 文字編碼器 12 3.1.3 基於文字與圖神經網路之對比式學習 12 3.2 提示學習技術應用在推薦系統 15 3.2.1 個人化推薦系統提示學習之設計 15 第四章 實驗結果與討論 19 4.1 實驗設計 19 4.1.1 資料集介紹 19 4.1.2 資料前處理 20 4.1.3 預訓練階段模型與訓練調適 21 4.1.4 下游任務模型建構與訓練調適 23 4.2 評估指標 24 4.3 實驗過程 25 4.4 結果分析 26 4.4.1 一般推薦場景結果分析 26 4.4.2 冷啟動場景結果分析 28 4.4.3 綜合分析 28 4.5 消融研究 29 4.6 案例研究 32 4.6.1 音樂與樂器資料集 . 32 第五章 結論 35 5.1 結論 35 5.2 未來工作展望 36 參考文獻 38zh_TW
dc.format.extent 2512994 bytes-
dc.format.mimetype application/pdf-
dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0110753140en_US
dc.subject (關鍵詞) 推薦系統zh_TW
dc.subject (關鍵詞) 預訓練模型zh_TW
dc.subject (關鍵詞) 多模態推薦系統zh_TW
dc.subject (關鍵詞) 提示詞學習zh_TW
dc.subject (關鍵詞) 圖神經網路zh_TW
dc.subject (關鍵詞) 冷啟動推薦系統zh_TW
dc.title (題名) 基於圖預訓練與提示詞學習於推薦系統zh_TW
dc.title (題名) Graph-based Pre-training and Prompting for Recommendation Systemsen_US
dc.type (資料類型) thesisen_US
dc.relation.reference (參考文獻) [1] J. Atwood and D. Towsley. Diffusion-convolutional neural networks, 2016. [2] O. Barkan and N. Koenigstein. Item2vec: Neural item embedding for collaborative filtering, 2017. [3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCan- dlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, 2020. [4] C.-M. Chen, M.-F. Tsai, Y.-C. Lin, and Y.-H. Yang. Query-based music recom- mendations via preference embedding. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys ’16, page 79–82, New York, NY, USA, 2016. Association for Computing Machinery. [5] C.-M. Chen, T.-H. Wang, C.-J. Wang, and M.-F. Tsai. Smore: modularize graph embedding for recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, page 582–583, New York, NY, USA, 2019. Association for Computing Machinery. [6] C.-M. Chen, T.-H. Wang, C.-J. Wang, and M.-F. Tsai. Smore: modularize graph embedding for recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, page 582–583, New York, NY, USA, 2019. Association for Computing Machinery. [7] G. de Souza Pereira Moreira, S. Rabhi, J. M. Lee, R. Ak, and E. Oldridge. Trans- formers4rec: Bridging the gap between nlp and sequential / session-based recom- mendation. page 143–153, 2021. [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019. [9] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar ́e, M. Lomeli, L. Hosseini, and H. J ́egou. The faiss library. 2024. [10] S. Geng, S. Liu, Z. Fu, Y. Ge, and Y. Zhang. Recommendation as language process- ing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). 2023. [11] A. Grover and J. Leskovec. node2vec: Scalable feature learning for networks, 2016. [12] W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs, 2018. [13] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation, 2020. [14] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua. Neural collaborative filtering, 2017. [15] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks, 2017. [16] O. Kuchaiev and B. Ginsburg. Training deep autoencoders for collaborative filtering, 2017. [17] J. Li, M. Wang, J. Li, J. Fu, X. Shen, J. Shang, and J. McAuley. Text is all you need: Learning language representations for sequential recommendation. 2023. [18] W. Li, Y. Zhang, Y. Sun, W. Wang, W. Zhang, and X. Lin. Approximate nearest neighbor search on high dimensional data — experiments, analyses, and improve- ment (v1.0). 2016. [19] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word repre- sentations in vector space. 2013. [20] J. Ni, J. Li, and J. McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. pages 188–197, Nov. 2019. [21] B. Perozzi, R. Al-Rfou, and S. Skiena. Deepwalk: online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’14. ACM, Aug. 2014. [22] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. 2021. [23] A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. [24] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. [25] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2023. [26] S. Rendle. Factorization machines. In 2010 IEEE International Conference on Data Mining, pages 995–1000, 2010. [27] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. 2012. [28] S. Ruder. An overview of gradient descent optimization algorithms, 2017. [29] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conferenceon World Wide Web, WWW ’01, page 285–295, New York, NY, USA, 2001. Association for Computing Machinery. [30] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. 2019. [31] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei. Line: Large-scale infor- mation network embedding. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15. International World Wide Web Conferences Steering Committee, May 2015. [32] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. 2019. [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2023. [34] Z. Wen and Y. Fang. Augmenting low-resource text classification with graph- grounded pre-training and prompting. 2023. [35] J. Weston, H. Yee, and R. J. Weiss. Learning to rank recommendations with the k-order statistic loss. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ’13, page 245–248, New York, NY, USA, 2013. Association for Computing Machinery. [36] J.-H. Yang, C.-M. Chen, C.-J. Wang, and M.-F. Tsai. Hop-rec: high-order prox- imity for implicit recommendation. In Proceedings of the 12th ACM Conference yon Recommender Systems, RecSys ’18, page 140–144, New York, NY, USA, 2018. Association for Computing Machinery. [37] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, July 2022.zh_TW