1. NCCU Academic Hub
  2. Publications
  3. College of Informatics
  4. Theses
  5. Item

Publications-Theses

Article View/Open

Publication Export

Google ScholarTM

NCCU Library

Citation Infomation

Related Publications in TAIR

題名 基於強化學習下的餐廳機器人— 接待與送餐之應用
Restaurant Robot Based on Reinforcement Learning—Application of Reception and Delivery
作者 鄭玉筠
Cheng, Yu-Yun
貢獻者 蔡子傑
Tsai, Tzu-Chieh
鄭玉筠
Cheng, Yu-Yun
關鍵詞 強化學習近端策略優化（RL-PPO）演算法
馬可夫決策過程
局部觀測
餐廳機器人
接待與送餐
Reinforcement Learning—Proximal Policy Optimization Algorithm
Markov Decision Process
Partially Observable
Robot of Restaurant
Reception and Delivery
日期 2023
上傳時間 9-Mar-2023 18:26:04 (UTC+8)
摘要 台灣在2020年人口首度呈現負成長，少子化已經成為產業人力缺口的重大問題；又或是有高度傳染性疾病流行時，人與人之間可能也不適合有太多接觸。餐飲服務業面臨上述的問題，若是能導入自動化AI系統，使用服務機器人來取代部分的人力，負責接待與送餐任務，就可以減輕人力不足與減少傳染性疾病的感染風險。餐廳若是有多個機器人的服務系統，透過工作排程，可以同時去完成不同的任務，不但可以減少人力運用，也可以具有較高的顧客滿意度的優勢。\n本文提出基於強化學習近端策略優化（RL-PPO）演算法的多個機器人服務系統的訓練框架，探索用於建構能夠減少人力的自動智慧餐廳的可能性。系統整合OpenAI Gym與Pygame 做為模擬環境，運用RL-PPO演算法的技術，並在最終階段類比成效。在本文中，我們對餐廳服務機器人系統進行建立模型，我們是以增加服務顧客的數量與減少顧客等待的時間為評估指標，而這與路徑規劃的距離會有正相關，在這樣的框架下，還可以進一步優化其他的指標：例如顧客的滿意度、員工每工時的勞動生產率等。我們針對這二項評估指標優化，因為問題涉及順序決策，同時也需要實時決策，所以我們將二項服務任務建模為馬可夫決策過程，採用RL-PPO演算法來解決該問題。\n本文模擬系統針對服務顧客數量與顧客等待時間二項指標的優化，證明經過本系統RL-PPO演算法架構下訓練的機器人系統，只需要餐廳的局部觀測資訊，通過自我學習，即可以維持餐廳服務機器人的服務效能。意即在餐廳臨時因應服務硬體佈局有所調整時，餐廳機器人從事接待與送餐工作時，也不需要更改系統或架構，餐廳機器人還是可以運作。這樣的框架系統，更具有靈活性、泛化性與穩定性，可以做為未來次世代的餐廳服務機器人系統的應用。
In 2020, Taiwan`s population showed negative growth for the first time, and the declining birth rate has become a major problem for the industry`s manpower shortage; or when there are highly contagious diseases, it may not be suitable for too much contact between people. The catering service industry is facing the above-mentioned problems. If an automated AI system can be introduced with service robots for reception and delivery tasks, the shortage of manpower and the risk of infection of epidemic diseases can be alleviated. If a restaurant has such a service system with multiple robots, different tasks can be completed at the same time through appropriate job scheduling. Thus it can not only reduce the use of manpower, but also have the advantage of higher customer satisfaction.\nThis thesis proposes a training framework for the multiple robot service system based on the Reinforcement Learning—Proximal Policy Optimization (RL-PPO) algorithm. It explores the possibility of constructing an automatic smart restaurant that can reduce manpower. We use OpenAI Gym and Pygame as the simulation environment. We build a model for the restaurant service robot system to evaluate the performance. The waiting time of customers versus number of serving customers is considered, which will be positively correlated with the robot working distance of path planning. Other indicators can also be further optimized, such as customer satisfaction, employee productivity per working hour, etc. In order to optimize the two evaluation indicators, sequential and real-time decision-makings are required. We model it as a Markov Decision Process, and use the RL-PPO algorithm to solve this problem.\nWe also prove that the robot system trained under the RL-PPO algorithm framework of this system only needs part of the observation information of the restaurant, and can maintain the efficiency through robot self-learning. That is to say, when the restaurant temporarily adjusts the service hardware layout, the restaurant robot can still operate without changing the system. Such a framework system is more stable, flexible and generalizable, and can be used as an application in the next generation of restaurant service robot systems in the future.
參考文獻 [1] A. M. Turing (1950). Computing Machinery and Intelligence. Mind, New Series, 59(236), 433-460.\n[2] David Silver (2016). Tutorial: Deep Reinforcement Learning\n[3] Chathurangi Shyalika, Thushari Silva, Asoka Karunananda (2020). Reinforcement Learning in Dynamic Task Scheduling: A Review. SN Computer Science, 1(6), 306\n[4] Byrd, K.、Fan, A. et al. (2021). Robot vs human: expectations, performances and gaps in off-premise restaurant service modes. International Journal of Contemporary Hospitality Management, 11(33), 3996-4016\n[5] Jun Yang, Xinghui You et al. (2019). Application of reinforcement learning in UAV cluster task scheduling, Future Generation Computer Systems, 95, 140-148\n[6] Tingxiang Fan, Pinxin Long et al. (2020). Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios. SAGE Journals\n[7] Takeshi Shimmura, Ryosuke Ichikari et al. (2020). Service robot introduction to a restaurant enhances both labor productivity and service quality. Procedia CIRP, 88, 589-594\n[8] Ruijun Yang, Liang Cheng, (2019). Path Planning of Restaurant Service Robot Based on A-star Algorithms with Updated Weights. 2019 12th International Symposium on Computational Intelligence and Design (ISCID)\n[9] Thanh Thi Nguyen, Ngoc Duy Nguyen et al. (2020). Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications. IEEE Transactions on Cybernetics (Volume: 50, Issue: 9)\n[10] Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction. MIT press.\n[11] Thorndike, E. L. (1898). Animal intelligence: an experimental study of the associate processes in animals. American Psychologist, 53(10), 1125-1127.\n[12] Deng, L., and Yu, D. (2014). Deep learning: methods and applications. Foundations and Trends in Signal Processing, 7(34), 197-387.\n[13] Min-Gyu Kim, Heeyoon Yoon et al. (2021).Investigating Frontline Service Employees to Identify Behavioral Goals of Restaurant Service Robot: An Exploratory Study. 2021 18th International Conference on Ubiquitous Robots (UR)\n[14] Prejitha.CT, Vikram Raj.N et al. (2020). Design of Restaurant Service Robot for Contact less and Hygienic Eating Experience. International Research Journal of Engineering and Technology (IRJET), 07(08), 2938-2943\n[15] OpenAI (Christopher Berner, Greg Brockman, et al. (2021). Dota 2 with Large Scale Deep Reinforcement Learning. arVix:1912.06680v1\n[16] K. Lakshmi Narayanan, et al. (2021). Fuzzy Guided Autonomous Nursing Robot through Wireless Beacon Network. Multimedia Tools and Applications, doi: 10.1007/s11042-021-11264-6\n[17] Lai, Chien-Jung; Tsai, Ching-Pei (2018). Design of Introducing Service Robot into Catering Services. Proceedings of the 2018 International Conference on Service Robotics Technologies, 62-66, doi:10.1145/3208833.3208837\n[18] Osman El-Said, Sara Al Hajri. (2022). Are customers happy with robot service? Investigating satisfaction with robot service restaurants during the COVID-19 pandemic. Heliyon 8(10), doi:10.1016/j.heliyon.2022.e08986\n[19] Hideharu Ouchi, Ryosuke Ueno et al. (2019). Development of Robot Restaurant Simulator. 2019 16th International Conference on Ubiquitous Robots\n[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347\n[21] Vanessa Hayes et al. (2019). Human origins in a southern African palaeo-wetland and first migrations. Nature\n[22] Thorndike, E. L. (1898). Animal intelligence: an experimental study of the associate processes in animals. American Psychologist, 53(10), 1125-1127.\n\n[23] Minsky, M. L. (1954). Theory of neural-analog reinforcement systems and its application to the brain model problem. Princeton University.\n[24] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. nature 529, 484.\n[25] Beakcheol Jang, Myeonghwi Kim, et al. (2019). Q-Learning Algorithms: A Comprehensive Classification and Applications. IEEE Access, 7\n[26] Schulman, John, et al. (2015). Trust Region Policy Optimization. arXiv:1502.05477
描述 碩士
國立政治大學
資訊科學系碩士在職專班
109971017
資料來源 http://thesis.lib.nccu.edu.tw/record/#G0109971017
資料類型 thesis
dc.contributor.advisor 蔡子傑zh_TW
dc.contributor.advisor Tsai, Tzu-Chiehen_US
dc.contributor.author (Authors) 鄭玉筠zh_TW
dc.contributor.author (Authors) Cheng, Yu-Yunen_US
dc.creator (作者) 鄭玉筠zh_TW
dc.creator (作者) Cheng, Yu-Yunen_US
dc.date (日期) 2023en_US
dc.date.accessioned 9-Mar-2023 18:26:04 (UTC+8)-
dc.date.available 9-Mar-2023 18:26:04 (UTC+8)-
dc.date.issued (上傳時間) 9-Mar-2023 18:26:04 (UTC+8)-
dc.identifier (Other Identifiers) G0109971017en_US
dc.identifier.uri (URI) https://ah.lib.nccu.edu.tw/item?item_id=163162-
dc.description (描述) 碩士zh_TW
dc.description (描述) 國立政治大學zh_TW
dc.description (描述) 資訊科學系碩士在職專班zh_TW
dc.description (描述) 109971017zh_TW
dc.description.abstract (摘要) 台灣在2020年人口首度呈現負成長，少子化已經成為產業人力缺口的重大問題；又或是有高度傳染性疾病流行時，人與人之間可能也不適合有太多接觸。餐飲服務業面臨上述的問題，若是能導入自動化AI系統，使用服務機器人來取代部分的人力，負責接待與送餐任務，就可以減輕人力不足與減少傳染性疾病的感染風險。餐廳若是有多個機器人的服務系統，透過工作排程，可以同時去完成不同的任務，不但可以減少人力運用，也可以具有較高的顧客滿意度的優勢。\n本文提出基於強化學習近端策略優化（RL-PPO）演算法的多個機器人服務系統的訓練框架，探索用於建構能夠減少人力的自動智慧餐廳的可能性。系統整合OpenAI Gym與Pygame 做為模擬環境，運用RL-PPO演算法的技術，並在最終階段類比成效。在本文中，我們對餐廳服務機器人系統進行建立模型，我們是以增加服務顧客的數量與減少顧客等待的時間為評估指標，而這與路徑規劃的距離會有正相關，在這樣的框架下，還可以進一步優化其他的指標：例如顧客的滿意度、員工每工時的勞動生產率等。我們針對這二項評估指標優化，因為問題涉及順序決策，同時也需要實時決策，所以我們將二項服務任務建模為馬可夫決策過程，採用RL-PPO演算法來解決該問題。\n本文模擬系統針對服務顧客數量與顧客等待時間二項指標的優化，證明經過本系統RL-PPO演算法架構下訓練的機器人系統，只需要餐廳的局部觀測資訊，通過自我學習，即可以維持餐廳服務機器人的服務效能。意即在餐廳臨時因應服務硬體佈局有所調整時，餐廳機器人從事接待與送餐工作時，也不需要更改系統或架構，餐廳機器人還是可以運作。這樣的框架系統，更具有靈活性、泛化性與穩定性，可以做為未來次世代的餐廳服務機器人系統的應用。zh_TW
dc.description.abstract (摘要) In 2020, Taiwan`s population showed negative growth for the first time, and the declining birth rate has become a major problem for the industry`s manpower shortage; or when there are highly contagious diseases, it may not be suitable for too much contact between people. The catering service industry is facing the above-mentioned problems. If an automated AI system can be introduced with service robots for reception and delivery tasks, the shortage of manpower and the risk of infection of epidemic diseases can be alleviated. If a restaurant has such a service system with multiple robots, different tasks can be completed at the same time through appropriate job scheduling. Thus it can not only reduce the use of manpower, but also have the advantage of higher customer satisfaction.\nThis thesis proposes a training framework for the multiple robot service system based on the Reinforcement Learning—Proximal Policy Optimization (RL-PPO) algorithm. It explores the possibility of constructing an automatic smart restaurant that can reduce manpower. We use OpenAI Gym and Pygame as the simulation environment. We build a model for the restaurant service robot system to evaluate the performance. The waiting time of customers versus number of serving customers is considered, which will be positively correlated with the robot working distance of path planning. Other indicators can also be further optimized, such as customer satisfaction, employee productivity per working hour, etc. In order to optimize the two evaluation indicators, sequential and real-time decision-makings are required. We model it as a Markov Decision Process, and use the RL-PPO algorithm to solve this problem.\nWe also prove that the robot system trained under the RL-PPO algorithm framework of this system only needs part of the observation information of the restaurant, and can maintain the efficiency through robot self-learning. That is to say, when the restaurant temporarily adjusts the service hardware layout, the restaurant robot can still operate without changing the system. Such a framework system is more stable, flexible and generalizable, and can be used as an application in the next generation of restaurant service robot systems in the future.en_US
dc.description.tableofcontents 目次\n致謝 i\n摘要 ii\nABSTRACT iii\n目次 v\n表次 vi\n圖次 vii\n第一章 緒論 1\n第一節 論文介紹 1\n第二節 研究動機與目的 2\n第三節 文獻探討 4\n第四節 論文架構 8\n第二章 基礎理論介紹 10\n第一節 強化學習 10\n第二節 近端策略優化演算法（PPO） 14\n第三節 最短路徑規畫：A-Star演算法 17\n第三章 模擬系統架構 19\n第一節 模擬環境 19\n第二節 模擬系統架構 22\n第四章 實驗設計與結果分析 31\n第一節 實驗設計 31\n第二節 結果分析 33\n第五章 結論與未來展望 48\n參考文獻 49\n\n表次\n表1 系統環境模型參數表 24\n表2 系統中PPO演算法的參數設定表 26\n表3 模擬系統中隨機環境參數表 28\n表4 模擬系統中障礙物隨機參數表 29\n表5 餐廳固定模式下平均顧客等待時間統計表 35\n表6 餐廳固定模式下服務顧客數量與機器人平均移動距離表 36\n表7 餐廳固定模式下中央排程系統之效能比較表 38\n表8 餐廳隨機模式下平均顧客等待時間統計表 42\n表9 餐廳隨機模式下服務顧客數量與機器人平均移動距離統計表 42\n表10 餐廳隨機障礙物模式下平均顧客等待時間比較表 44\n表11 餐廳隨機模式下服務顧客數量與機器人平均移動距離統計表 44\n\n圖次\n圖1 人工智慧與機器學習與強化學習的關係圖 11\n圖2 強化學習方法圖 12\n圖3 強化學習演算法的分類圖 12\n圖4 Q Learning 演算法原型 13\n圖5 Actor-Critic方法 14\n圖6 PPO演算法 15\n圖7 TRPO[26]與PPO[20]演算法 16\n圖8 PPO2[20]演算法 16\n圖9 PPO演算法[20] 17\n圖10 A*演算法公式計算推導圖 18\n圖11 模擬系統與角色間的互動關係圖 19\n圖12 餐廳配置圖 20\n圖13 顧客與中央排程系統互動關係圖 21\n圖14 餐廳服務機器人與中央排程系統互動關係圖 22\n圖15 餐廳服務機器人與顧客與中央排程狀態轉換模型圖 23\n圖16 PPO架構下-中央排程系統的深度神經網路圖 25\n圖17 PPO架構下-服務機器人系統的深度神經網路圖 25\n圖18 Agent, Environment, Reward的關係圖 27\n圖19 模擬系統強化學習PPO演算法 27\n圖20 餐廳固定模式下中央排程系統的平均Rewards 34\n圖21 餐廳固定模式下服務機器人路徑規劃系統之機器人平均Rewards 34\n圖22 餐廳固定模式下RL-PPO組之模擬系統影片截圖 35\n圖23 餐廳固定模式下平均顧客等待時間比較圖 36\n圖24 餐廳固定模式下服務顧客數量與機器人平均移動距離圖 36\n圖25 RL-PPO組的機器人足跡圖 37\n圖26 類比實驗組-FIFO+A*的機器人足跡圖 37\n圖27 餐廳固定模式下中央排程系統之效能比較圖 38\n圖28 餐廳隨機模式下中央排程系統的平均Rewards 39\n圖29 餐廳隨機模式下服務機器人路徑規劃系統之機器人平均Rewards 40\n圖30 餐廳隨機模式下A*組之模擬系統影片截圖 41\n圖31 餐廳隨機模式下RL-PPO組之模擬系統影片截圖 41\n圖32 餐廳隨機模式下RL-PPO組之模擬系統影片截圖 41\n圖33 餐廳隨機模式下平均顧客等待時間比較圖 42\n圖34 餐廳隨機模式下服務顧客數量與機器人平均移動距離圖 42\n圖35 餐廳隨機障礙物模式下平均顧客等待時間比較圖 44\n圖36 餐廳隨機障礙物模式下服務顧客數量與機器人平均移動距離圖 44\n圖37 餐廳隨機障礙物模式下RL-PPO組之模擬系統影片截圖 45\n圖38 違法行為對機器人平均顧客等待時間影響 46\n圖39 共享模型與否對機器人的平均顧客等待時間之影響 47zh_TW
dc.format.extent 3089861 bytes-
dc.format.mimetype application/pdf-
dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0109971017en_US
dc.subject (關鍵詞) 強化學習近端策略優化（RL-PPO）演算法zh_TW
dc.subject (關鍵詞) 馬可夫決策過程zh_TW
dc.subject (關鍵詞) 局部觀測zh_TW
dc.subject (關鍵詞) 餐廳機器人zh_TW
dc.subject (關鍵詞) 接待與送餐zh_TW
dc.subject (關鍵詞) Reinforcement Learning—Proximal Policy Optimization Algorithmen_US
dc.subject (關鍵詞) Markov Decision Processen_US
dc.subject (關鍵詞) Partially Observableen_US
dc.subject (關鍵詞) Robot of Restauranten_US
dc.subject (關鍵詞) Reception and Deliveryen_US
dc.title (題名) 基於強化學習下的餐廳機器人— 接待與送餐之應用zh_TW
dc.title (題名) Restaurant Robot Based on Reinforcement Learning—Application of Reception and Deliveryen_US
dc.type (資料類型) thesisen_US
dc.relation.reference (參考文獻) [1] A. M. Turing (1950). Computing Machinery and Intelligence. Mind, New Series, 59(236), 433-460.\n[2] David Silver (2016). Tutorial: Deep Reinforcement Learning\n[3] Chathurangi Shyalika, Thushari Silva, Asoka Karunananda (2020). Reinforcement Learning in Dynamic Task Scheduling: A Review. SN Computer Science, 1(6), 306\n[4] Byrd, K.、Fan, A. et al. (2021). Robot vs human: expectations, performances and gaps in off-premise restaurant service modes. International Journal of Contemporary Hospitality Management, 11(33), 3996-4016\n[5] Jun Yang, Xinghui You et al. (2019). Application of reinforcement learning in UAV cluster task scheduling, Future Generation Computer Systems, 95, 140-148\n[6] Tingxiang Fan, Pinxin Long et al. (2020). Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios. SAGE Journals\n[7] Takeshi Shimmura, Ryosuke Ichikari et al. (2020). Service robot introduction to a restaurant enhances both labor productivity and service quality. Procedia CIRP, 88, 589-594\n[8] Ruijun Yang, Liang Cheng, (2019). Path Planning of Restaurant Service Robot Based on A-star Algorithms with Updated Weights. 2019 12th International Symposium on Computational Intelligence and Design (ISCID)\n[9] Thanh Thi Nguyen, Ngoc Duy Nguyen et al. (2020). Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications. IEEE Transactions on Cybernetics (Volume: 50, Issue: 9)\n[10] Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: An introduction. MIT press.\n[11] Thorndike, E. L. (1898). Animal intelligence: an experimental study of the associate processes in animals. American Psychologist, 53(10), 1125-1127.\n[12] Deng, L., and Yu, D. (2014). Deep learning: methods and applications. Foundations and Trends in Signal Processing, 7(34), 197-387.\n[13] Min-Gyu Kim, Heeyoon Yoon et al. (2021).Investigating Frontline Service Employees to Identify Behavioral Goals of Restaurant Service Robot: An Exploratory Study. 2021 18th International Conference on Ubiquitous Robots (UR)\n[14] Prejitha.CT, Vikram Raj.N et al. (2020). Design of Restaurant Service Robot for Contact less and Hygienic Eating Experience. International Research Journal of Engineering and Technology (IRJET), 07(08), 2938-2943\n[15] OpenAI (Christopher Berner, Greg Brockman, et al. (2021). Dota 2 with Large Scale Deep Reinforcement Learning. arVix:1912.06680v1\n[16] K. Lakshmi Narayanan, et al. (2021). Fuzzy Guided Autonomous Nursing Robot through Wireless Beacon Network. Multimedia Tools and Applications, doi: 10.1007/s11042-021-11264-6\n[17] Lai, Chien-Jung; Tsai, Ching-Pei (2018). Design of Introducing Service Robot into Catering Services. Proceedings of the 2018 International Conference on Service Robotics Technologies, 62-66, doi:10.1145/3208833.3208837\n[18] Osman El-Said, Sara Al Hajri. (2022). Are customers happy with robot service? Investigating satisfaction with robot service restaurants during the COVID-19 pandemic. Heliyon 8(10), doi:10.1016/j.heliyon.2022.e08986\n[19] Hideharu Ouchi, Ryosuke Ueno et al. (2019). Development of Robot Restaurant Simulator. 2019 16th International Conference on Ubiquitous Robots\n[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347\n[21] Vanessa Hayes et al. (2019). Human origins in a southern African palaeo-wetland and first migrations. Nature\n[22] Thorndike, E. L. (1898). Animal intelligence: an experimental study of the associate processes in animals. American Psychologist, 53(10), 1125-1127.\n\n[23] Minsky, M. L. (1954). Theory of neural-analog reinforcement systems and its application to the brain model problem. Princeton University.\n[24] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of Go with deep neural networks and tree search. nature 529, 484.\n[25] Beakcheol Jang, Myeonghwi Kim, et al. (2019). Q-Learning Algorithms: A Comprehensive Classification and Applications. IEEE Access, 7\n[26] Schulman, John, et al. (2015). Trust Region Policy Optimization. arXiv:1502.05477zh_TW