學術產出-Theses
Article View/Open
Publication Export
-
題名 透過強化學習設計合作遊戲的夥伴
Designing Game Companions in Cooperative Games Using Reinforcement Learning作者 吳宥衡
Wu, You-Heng貢獻者 李蔡彥
Li, Tsai-Yen
吳宥衡
Wu, You-Heng關鍵詞 人工智慧
強化學習
非玩家角色
遊戲夥伴
Artificial Intelligent
Reinforcement learning
Non-player character
Game companions日期 2022 上傳時間 2-Dec-2022 15:19:54 (UTC+8) 摘要 電玩遊戲中與玩家互動的非玩家角色（Non-Player Character, NPC）一直是影響玩家遊戲體驗的要素，如何設計一個行為自然又能讓遊戲更加好玩的NPC遊戲夥伴不僅是遊戲業者一直以來努力的方向，也是許多玩家長年來的期待。本研究整理過去設計遊戲夥伴的相關文獻，探討玩家在玩遊戲過程中覺得好玩的原因，以及對遊戲夥伴的期待，發現大多數的玩家期望遊戲夥伴能觀察環境變化，並與玩家相互依賴合作。因此我們於Unity3D遊戲引擎設計一款雙人合作射擊遊戲，有別於過往使用強化學習設計完美通關遊戲的AI，本研究採用ML-Agents套件中近端策略優化（PPO）強化學習演算法的方式，一步一步讓遊戲夥伴學會新的遊戲技術，最後引導遊戲夥伴學會與玩家合作通關遊戲。本研究實驗請20名受試者分別與合作版本與非合作版本的遊戲夥伴一同闖關，透過受試者在實驗後給予的回饋，實驗結果也顯示了大多數玩家認為若遊戲夥伴能在遊戲過程中關注自身的狀態，並且在雙方有難時互相合作，可以更加有助於遊戲的正向體驗。
The non-player character (NPC) that interacts with players in video games has al-ways been an element that affects the players` game experience. How to design an NPC game companion that behaves naturally and makes the games more interesting is not only what the game designers’ striving for, but also the expectation of many players for a long time. This study tries to figure out the reasons why players feel interested in playing games and their expectations of game companions. We have found that most players look forward to game companions to observe changes in the environment and to rely on and cooperate with players. Therefore, we designed a two-player cooperative shooting game in the Unity3D game engine. Differing from using traditional reinforce-ment learning to design game agents in the past, we use proximal policy optimization (PPO) algorithm with ML-Agents toolkit to design our game companions. We try to make game companions learn game skills step by step, and finally learn how to cooper-ate with players to clear the game. We invited twenty participants to participate in our experiment. The participants were asked to play our shooting games in the cooperative version and the non-cooperative version with game companions, respectively. Through the feedback given by the participants, the experimental results show that most players believe that if the game companions can pay attention to players’ state during the games, and cooperate with each other in trouble, it will contribute to more positive playing ex-periences of the game.參考文獻 [1] E. Bouquet, V. Mäkelä, and A. Schmidt, "Exploring the Design of Companions in Video Games," in Academic Mindtrek 2021, 2021, pp. 145-153.[2] K. Emmerich, P. Ring, and M. Masuch, "I`m Glad You Are on My Side: How to Design Compelling Game Companions," in Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, 2018, pp. 141-152.[3] N. Afonso and R. Prada, "Agents that relate: Improving the social believability of non-player characters in role-playing games," in Proceedings of International Conference on Entertainment Computing, 2008: Springer, pp. 34-45.[4] A. Chowanda, M. Flintham, P. Blanchfield, and M. Valstar, "Playing with social and emotional game companions," in Proceedings of International Conference on Intelligent Virtual Agents, 2016: Springer, pp. 85-95.[5] M. Csikszentmihalyi and M. Csikzentmihaly, Flow: The psychology of optimal experience. Harper & Row New York, 1990.[6] J. Chen, "Flow in games (and everything else)," Communications of the ACM, vol. 50, no. 4, pp. 31-34, 2007.[7] M. P. Silva, V. do Nascimento Silva, and L. Chaimowicz, "Dynamic difficulty adjustment through an adaptive AI," in 2015 14th Brazilian symposium on computer games and digital entertainment (SBGames), 2015: IEEE, pp. 173-182.[8] M. P. Silva, V. do Nascimento Silva, and L. Chaimowicz, "Dynamic difficulty adjustment on MOBA games," Entertainment Computing, vol. 18, pp. 103-123, 2017.[9] J. Tremblay and C. Verbrugge, "Adaptive companions in FPS games, "in Proceedings of International Conference on Foundations of Digital Games , vol. 13, pp. 229-236, 2013..[10] A. Sharifi, R. Zhao, and D. Szafron, "Learning companion behaviors using reinforcement learning in games," in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2010, vol. 5, no. 1.[11] M. Smith, S. Lee-Urban, and H. Muñoz-Avila, "RETALIATE: Learning winning policies in first-person shooter games," in Proceedings of the AAAI Conference on Artificial Intelligence, 2007, pp. 1801-1806.[12] 王宇軒, "多重代理人之策略競爭遊戲之強化學習方法," 碩士論文, 國立東海大學資訊科學系, 2019.[13] D. Piergigli, L. A. Ripamonti, D. Maggiorini, and D. Gadia, "Deep Reinforcement Learning to train agents in a multiplayer First Person Shooter: some preliminary results," in Proceedings of 2019 IEEE Conference on Games (CoG), 2019: IEEE, pp. 1-8.[14] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.[15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017.[16] W. IJsselsteijn et al., "Measuring the Experience of Digital Game Enjoyment," in Proceedings of measuring behavior Conference, 2008, Noldus Maastricht, the Netherlands, pp. 88-89.[17] Poels, K., de Kort, Y. A. W., & IJsselsteijn, W. A. (2007). D3.3 : Game Experience Questionnaire: development of a self-report measure to assess the psychological impact of digital games. Technische Universiteit Eindhoven.[18] R. Likert, "A technique for the measurement of attitudes," Archives of psychology, 1932.[19] S. S. Shapiro and M. B. Wilk, "An analysis of variance test for normality (complete samples)," Biometrika, vol. 52, no. 3/4, pp. 591-611, 1965.[20] F. Wilcoxon, "Individual comparisons by ranking methods," Biometrics Bulletin vol. 1, no. 6 (Dec., 1945), pp. 80-83 描述 碩士
國立政治大學
資訊科學系
108753123資料來源 http://thesis.lib.nccu.edu.tw/record/#G0108753123 資料類型 thesis dc.contributor.advisor 李蔡彥 zh_TW dc.contributor.advisor Li, Tsai-Yen en_US dc.contributor.author (Authors) 吳宥衡 zh_TW dc.contributor.author (Authors) Wu, You-Heng en_US dc.creator (作者) 吳宥衡 zh_TW dc.creator (作者) Wu, You-Heng en_US dc.date (日期) 2022 en_US dc.date.accessioned 2-Dec-2022 15:19:54 (UTC+8) - dc.date.available 2-Dec-2022 15:19:54 (UTC+8) - dc.date.issued (上傳時間) 2-Dec-2022 15:19:54 (UTC+8) - dc.identifier (Other Identifiers) G0108753123 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/142638 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 資訊科學系 zh_TW dc.description (描述) 108753123 zh_TW dc.description.abstract (摘要) 電玩遊戲中與玩家互動的非玩家角色（Non-Player Character, NPC）一直是影響玩家遊戲體驗的要素，如何設計一個行為自然又能讓遊戲更加好玩的NPC遊戲夥伴不僅是遊戲業者一直以來努力的方向，也是許多玩家長年來的期待。本研究整理過去設計遊戲夥伴的相關文獻，探討玩家在玩遊戲過程中覺得好玩的原因，以及對遊戲夥伴的期待，發現大多數的玩家期望遊戲夥伴能觀察環境變化，並與玩家相互依賴合作。因此我們於Unity3D遊戲引擎設計一款雙人合作射擊遊戲，有別於過往使用強化學習設計完美通關遊戲的AI，本研究採用ML-Agents套件中近端策略優化（PPO）強化學習演算法的方式，一步一步讓遊戲夥伴學會新的遊戲技術，最後引導遊戲夥伴學會與玩家合作通關遊戲。本研究實驗請20名受試者分別與合作版本與非合作版本的遊戲夥伴一同闖關，透過受試者在實驗後給予的回饋，實驗結果也顯示了大多數玩家認為若遊戲夥伴能在遊戲過程中關注自身的狀態，並且在雙方有難時互相合作，可以更加有助於遊戲的正向體驗。 zh_TW dc.description.abstract (摘要) The non-player character (NPC) that interacts with players in video games has al-ways been an element that affects the players` game experience. How to design an NPC game companion that behaves naturally and makes the games more interesting is not only what the game designers’ striving for, but also the expectation of many players for a long time. This study tries to figure out the reasons why players feel interested in playing games and their expectations of game companions. We have found that most players look forward to game companions to observe changes in the environment and to rely on and cooperate with players. Therefore, we designed a two-player cooperative shooting game in the Unity3D game engine. Differing from using traditional reinforce-ment learning to design game agents in the past, we use proximal policy optimization (PPO) algorithm with ML-Agents toolkit to design our game companions. We try to make game companions learn game skills step by step, and finally learn how to cooper-ate with players to clear the game. We invited twenty participants to participate in our experiment. The participants were asked to play our shooting games in the cooperative version and the non-cooperative version with game companions, respectively. Through the feedback given by the participants, the experimental results show that most players believe that if the game companions can pay attention to players’ state during the games, and cooperate with each other in trouble, it will contribute to more positive playing ex-periences of the game. en_US dc.description.tableofcontents 致謝 i摘要 iiAbstract iii目錄 iv圖目錄 vi表目錄 viii第1章 緒論 11.1 研究背景 11.2 研究動機與目標 21.3 論文貢獻 21.4 論文架構 3第2章 背景知識與相關研究 42.1 傳統的NPC設計 42.2 遊戲中的NPC夥伴設計概述 52.3 心流理論與電子遊戲 52.4 使用行為樹設計遊戲夥伴 62.5 使用強化學習設計的遊戲夥伴 72.6 使用深度強化學習設計多位NPC 92.7 強化學習演算法 102.7.1傳統強化學習演算法 102.7.2近端策略優化演算法 122.8 小結 14第3章 研究問題與系統設計 163.1 問題定義 163.2 遊戲規則與遊戲操作 173.2.1遊戲規則及系統介紹 173.2.2遊戲操作 193.3 系統架構及流程 193.4 遊戲夥伴的設計階段 263.4.1尋物階段 263.4.2抗敵階段 313.4.3合作階段 36第4章 實驗方法與結果分析 404.1 實驗目標 414.2 實驗對象 414.3 實驗流程與實驗問卷 434.3.1 實驗方式及實驗流程 434.3.2 實驗問卷設計 454.4 實驗結果與分析 484.4.1 問卷前半部分的結果 484.4.2 問卷後半部分的結果 514.4.3 問卷開放式問題的回應 53第5章 結論與未來發展 565.1 研究結論 565.2 改進與未來展望 565.2.1 改進 565.2.2 未來展望 575.2.3 未來應用 57參考文獻 59附錄一：實驗同意書 61附錄二：射擊遊戲實驗問卷 62 zh_TW dc.format.extent 2851498 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0108753123 en_US dc.subject (關鍵詞) 人工智慧 zh_TW dc.subject (關鍵詞) 強化學習 zh_TW dc.subject (關鍵詞) 非玩家角色 zh_TW dc.subject (關鍵詞) 遊戲夥伴 zh_TW dc.subject (關鍵詞) Artificial Intelligent en_US dc.subject (關鍵詞) Reinforcement learning en_US dc.subject (關鍵詞) Non-player character en_US dc.subject (關鍵詞) Game companions en_US dc.title (題名) 透過強化學習設計合作遊戲的夥伴 zh_TW dc.title (題名) Designing Game Companions in Cooperative Games Using Reinforcement Learning en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) [1] E. Bouquet, V. Mäkelä, and A. Schmidt, "Exploring the Design of Companions in Video Games," in Academic Mindtrek 2021, 2021, pp. 145-153.[2] K. Emmerich, P. Ring, and M. Masuch, "I`m Glad You Are on My Side: How to Design Compelling Game Companions," in Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, 2018, pp. 141-152.[3] N. Afonso and R. Prada, "Agents that relate: Improving the social believability of non-player characters in role-playing games," in Proceedings of International Conference on Entertainment Computing, 2008: Springer, pp. 34-45.[4] A. Chowanda, M. Flintham, P. Blanchfield, and M. Valstar, "Playing with social and emotional game companions," in Proceedings of International Conference on Intelligent Virtual Agents, 2016: Springer, pp. 85-95.[5] M. Csikszentmihalyi and M. Csikzentmihaly, Flow: The psychology of optimal experience. Harper & Row New York, 1990.[6] J. Chen, "Flow in games (and everything else)," Communications of the ACM, vol. 50, no. 4, pp. 31-34, 2007.[7] M. P. Silva, V. do Nascimento Silva, and L. Chaimowicz, "Dynamic difficulty adjustment through an adaptive AI," in 2015 14th Brazilian symposium on computer games and digital entertainment (SBGames), 2015: IEEE, pp. 173-182.[8] M. P. Silva, V. do Nascimento Silva, and L. Chaimowicz, "Dynamic difficulty adjustment on MOBA games," Entertainment Computing, vol. 18, pp. 103-123, 2017.[9] J. Tremblay and C. Verbrugge, "Adaptive companions in FPS games, "in Proceedings of International Conference on Foundations of Digital Games , vol. 13, pp. 229-236, 2013..[10] A. Sharifi, R. Zhao, and D. Szafron, "Learning companion behaviors using reinforcement learning in games," in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2010, vol. 5, no. 1.[11] M. Smith, S. Lee-Urban, and H. Muñoz-Avila, "RETALIATE: Learning winning policies in first-person shooter games," in Proceedings of the AAAI Conference on Artificial Intelligence, 2007, pp. 1801-1806.[12] 王宇軒, "多重代理人之策略競爭遊戲之強化學習方法," 碩士論文, 國立東海大學資訊科學系, 2019.[13] D. Piergigli, L. A. Ripamonti, D. Maggiorini, and D. Gadia, "Deep Reinforcement Learning to train agents in a multiplayer First Person Shooter: some preliminary results," in Proceedings of 2019 IEEE Conference on Games (CoG), 2019: IEEE, pp. 1-8.[14] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.[15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017.[16] W. IJsselsteijn et al., "Measuring the Experience of Digital Game Enjoyment," in Proceedings of measuring behavior Conference, 2008, Noldus Maastricht, the Netherlands, pp. 88-89.[17] Poels, K., de Kort, Y. A. W., & IJsselsteijn, W. A. (2007). D3.3 : Game Experience Questionnaire: development of a self-report measure to assess the psychological impact of digital games. Technische Universiteit Eindhoven.[18] R. Likert, "A technique for the measurement of attitudes," Archives of psychology, 1932.[19] S. S. Shapiro and M. B. Wilk, "An analysis of variance test for normality (complete samples)," Biometrika, vol. 52, no. 3/4, pp. 591-611, 1965.[20] F. Wilcoxon, "Individual comparisons by ranking methods," Biometrics Bulletin vol. 1, no. 6 (Dec., 1945), pp. 80-83 zh_TW dc.identifier.doi (DOI) 10.6814/NCCU202201687 en_US
