Publications-Theses
Article View/Open
Publication Export
-
題名 基於深度強化學習的智能存貨控制：以高科技供應鏈為例
A Deep Reinforcement Learning approach for Intelligent Inventory Control in high-tech supply chains作者 廖信堯
Liao, Hsin-Yao貢獻者 莊皓鈞
Chuang, Hao-Chun
廖信堯
Liao, Hsin-Yao關鍵詞 強化學習
深度學習
庫存最佳化
模擬
作業管理
Reinforcement learning
Deep learning
Inventory optimization
Simulation
Operations management日期 2020 上傳時間 6-Apr-2020 14:44:04 (UTC+8) 摘要 Machine learning is revolutionizing business operations across industry sectors. Among different learning techniques, deep reinforcement learning (DRL) has received broad attention in recent years due to the salient performance of AlphaGo, an artificial intelligence (AI) system empowered by DRL. DRL is a model-free and data-driven approach to develop near-optimal policies for sequential decision-making problems. Intrigued by the success of DRL in various fields, we, in this study, assess the applicability of DRL to multi-period inventory control under stochastic demand, which is a classical Markov Decision Process problem. Working with the largest distributor of electronics manufacturing services (EMS) in the world, we propose deep Q-networks (DQN) for intelligent inventory control (IIC). Facing erratic and non-stationary demand for electronic components with limited market life cycle, the distributor could not infer the exact demand distribution and solve the inventory optimization problem analytically in a finite-horizon with lost sales setting. Hence, we develop DQN by specifying relevant state and decision inputs, and then designing a data-driven simulation environment, in which the agent is trained over thousands of episodes. For trained items, DQN outperforms the benchmark in a few ways. First, DQN can reduce the total inventory by at least 40% while achieving better service level. Second, when penalty parameter increases, DQN can effectively reduce the amount of out-of-stock. While we transfer trained DQN into testing sets, within the same item, the out-of-sample performance is excellent. For other unseen items, we use the Maximum Entropy Bootstrap to train ensemble DDQN and make our DRL agent more robust. Given the promising results in our experiments, we discuss implications, limitations, and further directions for applying DRL/DQN to business decision-making problems. 參考文獻 Arulkumaran, K., Deisenroth, P. M., Brundage, M., & Bharath, A. A. (2017) A brief survey of deep reinforcement learning. ArXiv: 1708.05866v2.Chaharsooghi, S., Heydari, J., & Zegordi, S. (2008) A reinforcement learning model for supply chain ordering management: An application to the beer game. Decision Support Systems, 45(4): 949-959.Chollet, F. (2017) Deep Learning with Python. Manning Publications.Giannoccaro, I., & Pontrandolfo, P. (2002) Inventory management in supply chains: A reinforcement learning approach. International Journal of Production Economics, 78(2): 153-161.Gijsbrechts, J., Boute, R., Zhang, D., & Van Mieghem, J. (2019) Can Deep Reinforcement Learning Improve Inventory Management? Performance on Dual Sourcing, Lost Sales and Multi-Echelon Problems. Available at SSRN https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3302881.Gosavi, A. (2009) Reinforcement learning: A tutorial survey and recent advances. INFORMS Journal on Computing, 21(3): 177-345.Kingma, D. P., & Ba, J. (2017). Adam: A Method for Stochastic Optimization. ArXiv:1412.6980v9.Lin, L. J. (1992) Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3-4): 293-321.Mnih V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) Playing Atari with deep reinforcement learning. ArXiv:1312.5602v1.Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., & Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015) Human-level control through deep reinforcement learning. Nature, 518: 529-533.Oroojlooyjadid, A., Snyder, L., & Takáč, M. (2019) Applying deep learning to the newsvendor problem. IISE Transactions, in press.Oroojlooyjadid, A., Nazari, M., Snyder, L., & Takáč, M. (2019b) A deep Q-Network for the beer game: A deep reinforcement learning algorithm to solve inventory optimization problems. ArXiv:1708.05924v3.Porteus, E. L. (2002) Foundations of Stochastic Inventory Theory. Stanford University Press, California.Qi, X., Wu, G., Boriboonsomsin, K., Barth, M. J., & Gonder, J. (2016) Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles. Transportation Research Record, 2572(1), 1-8.Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized Experience Replay. ArXiv: 1511.05952v4.Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., Hassabis, D. (2017) Mastering the game of Go without human knowledge. Nature 550: 354-359.Sutton, R. S. and Barto, A. G. (2018) Reinforcement Learning: An Introduction. Second edition, MIT Press, Cambridge.Van Hasselt, H., Guez, A., and Silver, D. (2015). Deep Reinforcement Learning with Double Q-Learning. ArXiv:1509.06461.Vinod, H.D. and Lòpez-de-Lacalle, J. (2009). Maximum entropy bootstrap for time series. The meboot R Package. J. Stat. Softw., 29 (2009), pp. 1-19 描述 碩士
國立政治大學
資訊管理學系
107356003資料來源 http://thesis.lib.nccu.edu.tw/record/#G0107356003 資料類型 thesis dc.contributor.advisor 莊皓鈞 zh_TW dc.contributor.advisor Chuang, Hao-Chun en_US dc.contributor.author (Authors) 廖信堯 zh_TW dc.contributor.author (Authors) Liao, Hsin-Yao en_US dc.creator (作者) 廖信堯 zh_TW dc.creator (作者) Liao, Hsin-Yao en_US dc.date (日期) 2020 en_US dc.date.accessioned 6-Apr-2020 14:44:04 (UTC+8) - dc.date.available 6-Apr-2020 14:44:04 (UTC+8) - dc.date.issued (上傳時間) 6-Apr-2020 14:44:04 (UTC+8) - dc.identifier (Other Identifiers) G0107356003 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/129213 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 資訊管理學系 zh_TW dc.description (描述) 107356003 zh_TW dc.description.abstract (摘要) Machine learning is revolutionizing business operations across industry sectors. Among different learning techniques, deep reinforcement learning (DRL) has received broad attention in recent years due to the salient performance of AlphaGo, an artificial intelligence (AI) system empowered by DRL. DRL is a model-free and data-driven approach to develop near-optimal policies for sequential decision-making problems. Intrigued by the success of DRL in various fields, we, in this study, assess the applicability of DRL to multi-period inventory control under stochastic demand, which is a classical Markov Decision Process problem. Working with the largest distributor of electronics manufacturing services (EMS) in the world, we propose deep Q-networks (DQN) for intelligent inventory control (IIC). Facing erratic and non-stationary demand for electronic components with limited market life cycle, the distributor could not infer the exact demand distribution and solve the inventory optimization problem analytically in a finite-horizon with lost sales setting. Hence, we develop DQN by specifying relevant state and decision inputs, and then designing a data-driven simulation environment, in which the agent is trained over thousands of episodes. For trained items, DQN outperforms the benchmark in a few ways. First, DQN can reduce the total inventory by at least 40% while achieving better service level. Second, when penalty parameter increases, DQN can effectively reduce the amount of out-of-stock. While we transfer trained DQN into testing sets, within the same item, the out-of-sample performance is excellent. For other unseen items, we use the Maximum Entropy Bootstrap to train ensemble DDQN and make our DRL agent more robust. Given the promising results in our experiments, we discuss implications, limitations, and further directions for applying DRL/DQN to business decision-making problems. en_US dc.description.tableofcontents Introduction 1Section2. Literature Review 3Section2.1 Reinforcement Learning 3Section2.2 Deep Reinforcement Learning for Inventory Control 6Section3. A Deep Reinforcement Learning Agent 8Section3.1 Problem Situation and Simulation Environment 8Section3.2 Deep Q-Networks for Inventory Control 10Section4. Training Performance and Comparisons 15Section4.1 RL Simulation Design and Neural Net Architecture Tuning 15Section4.2 Comparisons Between DRL and Benchmark 18Section5. The Applicability of Transferring Trained Agents 21Section5.1 Transfer Learning Performance on Latter Period of the Same Item 21Section5.2 Transfer Learning Performance on New Unseen Items 22Section5.3 DDQN with Ensemble Learning Method 24Section6. Conclusion and Discussion 25References 27 zh_TW dc.format.extent 1802849 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0107356003 en_US dc.subject (關鍵詞) 強化學習 zh_TW dc.subject (關鍵詞) 深度學習 zh_TW dc.subject (關鍵詞) 庫存最佳化 zh_TW dc.subject (關鍵詞) 模擬 zh_TW dc.subject (關鍵詞) 作業管理 zh_TW dc.subject (關鍵詞) Reinforcement learning en_US dc.subject (關鍵詞) Deep learning en_US dc.subject (關鍵詞) Inventory optimization en_US dc.subject (關鍵詞) Simulation en_US dc.subject (關鍵詞) Operations management en_US dc.title (題名) 基於深度強化學習的智能存貨控制：以高科技供應鏈為例 zh_TW dc.title (題名) A Deep Reinforcement Learning approach for Intelligent Inventory Control in high-tech supply chains en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) Arulkumaran, K., Deisenroth, P. M., Brundage, M., & Bharath, A. A. (2017) A brief survey of deep reinforcement learning. ArXiv: 1708.05866v2.Chaharsooghi, S., Heydari, J., & Zegordi, S. (2008) A reinforcement learning model for supply chain ordering management: An application to the beer game. Decision Support Systems, 45(4): 949-959.Chollet, F. (2017) Deep Learning with Python. Manning Publications.Giannoccaro, I., & Pontrandolfo, P. (2002) Inventory management in supply chains: A reinforcement learning approach. International Journal of Production Economics, 78(2): 153-161.Gijsbrechts, J., Boute, R., Zhang, D., & Van Mieghem, J. (2019) Can Deep Reinforcement Learning Improve Inventory Management? Performance on Dual Sourcing, Lost Sales and Multi-Echelon Problems. Available at SSRN https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3302881.Gosavi, A. (2009) Reinforcement learning: A tutorial survey and recent advances. INFORMS Journal on Computing, 21(3): 177-345.Kingma, D. P., & Ba, J. (2017). Adam: A Method for Stochastic Optimization. ArXiv:1412.6980v9.Lin, L. J. (1992) Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3-4): 293-321.Mnih V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) Playing Atari with deep reinforcement learning. ArXiv:1312.5602v1.Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., & Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015) Human-level control through deep reinforcement learning. Nature, 518: 529-533.Oroojlooyjadid, A., Snyder, L., & Takáč, M. (2019) Applying deep learning to the newsvendor problem. IISE Transactions, in press.Oroojlooyjadid, A., Nazari, M., Snyder, L., & Takáč, M. (2019b) A deep Q-Network for the beer game: A deep reinforcement learning algorithm to solve inventory optimization problems. ArXiv:1708.05924v3.Porteus, E. L. (2002) Foundations of Stochastic Inventory Theory. Stanford University Press, California.Qi, X., Wu, G., Boriboonsomsin, K., Barth, M. J., & Gonder, J. (2016) Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles. Transportation Research Record, 2572(1), 1-8.Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized Experience Replay. ArXiv: 1511.05952v4.Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., Hassabis, D. (2017) Mastering the game of Go without human knowledge. Nature 550: 354-359.Sutton, R. S. and Barto, A. G. (2018) Reinforcement Learning: An Introduction. Second edition, MIT Press, Cambridge.Van Hasselt, H., Guez, A., and Silver, D. (2015). Deep Reinforcement Learning with Double Q-Learning. ArXiv:1509.06461.Vinod, H.D. and Lòpez-de-Lacalle, J. (2009). Maximum entropy bootstrap for time series. The meboot R Package. J. Stat. Softw., 29 (2009), pp. 1-19 zh_TW dc.identifier.doi (DOI) 10.6814/NCCU202000375 en_US
