學術產出-Theses
Article View/Open
Publication Export
-
題名 分散式系統Kafka的延展性研究：以支付應用為例
An empirical study on the scalability of Kafka in a scenario of online payments作者 江品萱
Chiang, Pin-Hsuan貢獻者 陳恭<br>郭桐惟
Chen, Kung<br>Kuo, Tung-Wei
江品萱
Chiang, Pin-Hsuan關鍵詞 分散式系統
系統延展性
支付清算
帳務驗證
Kafka
Distributed system
System scalability
Online payments
Account verification日期 2023 上傳時間 1-Sep-2023 14:56:05 (UTC+8) 摘要 近年來，科技的發展使得金融支付越來越方便，在金流快速流通的現代社會中，銀行每日接收許多轉帳、匯款等多樣的交易需求，交易量龐大且複雜，一但發生系統主機設備故障即有可能造成交易無法即時進行。在過去，銀行仰賴集中式主機系統來處理交易清算，然而集中式主機容易發生系統癱瘓、安全風險、量能極限等問題，且擴展成本及維護成本大，在設備及人力上皆須負擔相對較高的費用。本研究以分散式系統Kafka為基礎，欲探討在金融應用場景中，透過分散式系統的延展性特性和可靠性，是否適合應用於銀行的交易清算場景中。分散式系統的應用能否能因應交易量增加，而設計出一具可擴展性、容錯性且保持系統效能良好的帳務驗證系統，解決以往集中式主機面臨的問題。研究結果顯示透過Kafka的延展性特性可以使系統突破單一節點在交易量增加時可承受之極限，並經由兩階段帳務驗證方式確保銀行餘額的正確性和批次處理多筆小額交易達到效率提升。此外，透過Kafka機制確保所有進入系統的交易都能完整處理，有效防止因系統異常而導致的交易資料處理不完整，進而引發銀行餘額錯誤的情況發生。同時，透過分散式系統的備援機制，在故障發生時快速處理異常，保證交易處理即時性。總結而言，透過運用Kafka系統的延展性及備援機制，實作系統不僅能應對交易量的增加，更能夠確保銀行清算餘額的正確性、完整性以及即時性，解決了過去集中式系統可能碰到的問題，且維持系統的運作效率。
In recent years, the advancement of technology has made financial payments increasingly convenient. In the fast-paced modern society where money flows rapidly, banks receive numerous transaction requests daily, including transfers and remittances. The transaction volume is enormous and complex. However, any system failure can potentially cause delays in processing transactions and result in substantial costs.This study is based on the distributed system Kafka and aims to explore the application of distributed systems in the financial domain. In my research, I investigate whether the increasing transaction volume can be handled by designing a scalable, fault-tolerant, and high-performance account verification system based on Kafka. The goal is to address the challenges faced by centralized systems in the past.The result of the designed system demonstrates that the scalability of Kafka enables the system to surpass the limits of processing and increased transaction volume. By designing a two-phase account verification and batch processing approach, the accuracy of bank balances is ensured and system efficiency is increased. Additionally, the Kafka mechanism guarantees the complete processing of all transactions, avoiding discrepancies in bank balances due to system anomalies. It also provides fast backup and handling of abnormal situations, ensuring real-time transaction processing in the event of failures.參考文獻 Gururajapathy, S. S., Mokhlis, H., & Illias, H. A. (2017). Fault location and detection techniques in power distribution systems with distributed generation: A review. Renewable and Sustainable Energy Reviews, 74, 949–958.Kersting, W. H. (2017). Distribution system modeling and analysis. CRC press.Kumar, M., & Singh, C. (2017). Building Data Streaming Applications with Apache Kafka. Packt Publishing Ltd.Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: the definitive guide: real-time data and stream processing at scale. “ O’Reilly Media, Inc.”Shapira, G., Palino, T., Sivaram, R., & Petty, K. (2021). Kafka: the definitive guide. “ O’Reilly Media, Inc.”Shree, R., Choudhury, T., Gupta, S. C., & Kumar, P. (2017). KAFKA: The modern platform for data management and analysis in big data domain. 2017 2nd International Conference on Telecommunication and Networks (TEL-NET), 1–5.Vohra, D., & Vohra, D. (2016). Apache kafka. Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools, 339–347.Wang, G., Chen, L., Dikshit, A., Gustafson, J., Chen, B., Sax, M. J., Roesler, J., Blee-Goldman, S., Cadonna, B., & Mehta, A. (2021). Consistency and completeness: Rethinking distributed stream processing in apache kafka. Proceedings of the 2021 International Conference on Management of Data, 2602–2613.Wu, H. (2019). Research proposal: Reliability evaluation of the apache kafka streaming system. 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 112–113.Wu, H., Shang, Z., & Wolter, K. (2019). Performance prediction for the apache kafka messaging system. 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 154–161. 描述 碩士
國立政治大學
資訊管理學系
111356023資料來源 http://thesis.lib.nccu.edu.tw/record/#G0111356023 資料類型 thesis dc.contributor.advisor 陳恭<br>郭桐惟 zh_TW dc.contributor.advisor Chen, Kung<br>Kuo, Tung-Wei en_US dc.contributor.author (Authors) 江品萱 zh_TW dc.contributor.author (Authors) Chiang, Pin-Hsuan en_US dc.creator (作者) 江品萱 zh_TW dc.creator (作者) Chiang, Pin-Hsuan en_US dc.date (日期) 2023 en_US dc.date.accessioned 1-Sep-2023 14:56:05 (UTC+8) - dc.date.available 1-Sep-2023 14:56:05 (UTC+8) - dc.date.issued (上傳時間) 1-Sep-2023 14:56:05 (UTC+8) - dc.identifier (Other Identifiers) G0111356023 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/146899 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 資訊管理學系 zh_TW dc.description (描述) 111356023 zh_TW dc.description.abstract (摘要) 近年來，科技的發展使得金融支付越來越方便，在金流快速流通的現代社會中，銀行每日接收許多轉帳、匯款等多樣的交易需求，交易量龐大且複雜，一但發生系統主機設備故障即有可能造成交易無法即時進行。在過去，銀行仰賴集中式主機系統來處理交易清算，然而集中式主機容易發生系統癱瘓、安全風險、量能極限等問題，且擴展成本及維護成本大，在設備及人力上皆須負擔相對較高的費用。本研究以分散式系統Kafka為基礎，欲探討在金融應用場景中，透過分散式系統的延展性特性和可靠性，是否適合應用於銀行的交易清算場景中。分散式系統的應用能否能因應交易量增加，而設計出一具可擴展性、容錯性且保持系統效能良好的帳務驗證系統，解決以往集中式主機面臨的問題。研究結果顯示透過Kafka的延展性特性可以使系統突破單一節點在交易量增加時可承受之極限，並經由兩階段帳務驗證方式確保銀行餘額的正確性和批次處理多筆小額交易達到效率提升。此外，透過Kafka機制確保所有進入系統的交易都能完整處理，有效防止因系統異常而導致的交易資料處理不完整，進而引發銀行餘額錯誤的情況發生。同時，透過分散式系統的備援機制，在故障發生時快速處理異常，保證交易處理即時性。總結而言，透過運用Kafka系統的延展性及備援機制，實作系統不僅能應對交易量的增加，更能夠確保銀行清算餘額的正確性、完整性以及即時性，解決了過去集中式系統可能碰到的問題，且維持系統的運作效率。 zh_TW dc.description.abstract (摘要) In recent years, the advancement of technology has made financial payments increasingly convenient. In the fast-paced modern society where money flows rapidly, banks receive numerous transaction requests daily, including transfers and remittances. The transaction volume is enormous and complex. However, any system failure can potentially cause delays in processing transactions and result in substantial costs.This study is based on the distributed system Kafka and aims to explore the application of distributed systems in the financial domain. In my research, I investigate whether the increasing transaction volume can be handled by designing a scalable, fault-tolerant, and high-performance account verification system based on Kafka. The goal is to address the challenges faced by centralized systems in the past.The result of the designed system demonstrates that the scalability of Kafka enables the system to surpass the limits of processing and increased transaction volume. By designing a two-phase account verification and batch processing approach, the accuracy of bank balances is ensured and system efficiency is increased. Additionally, the Kafka mechanism guarantees the complete processing of all transactions, avoiding discrepancies in bank balances due to system anomalies. It also provides fast backup and handling of abnormal situations, ensuring real-time transaction processing in the event of failures. en_US dc.description.tableofcontents 第一章 緒論 1第一節 研究背景與動機 1第二節 研究目的 4第三節 論文結構 5第二章 相關研究與技術背景 6第一節 集中式系統與分散式系統 6第二節 系統延展性 8第三節 Kafka相關名詞 10第三章 系統架構設計與實驗 13第一節 系統設計理念 13第二節 系統整體架構 14第三節 系統設計細節 15第四節 實驗設計 23第四章 實驗結果與分析 27第一節 實驗工具介紹 27第二節 結果分析 28第五章 結論與未來發展 43第一節 研究結論 43第二節 研究限制與未來方向 44參考文獻 46 zh_TW dc.format.extent 2022672 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0111356023 en_US dc.subject (關鍵詞) 分散式系統 zh_TW dc.subject (關鍵詞) 系統延展性 zh_TW dc.subject (關鍵詞) 支付清算 zh_TW dc.subject (關鍵詞) 帳務驗證 zh_TW dc.subject (關鍵詞) Kafka en_US dc.subject (關鍵詞) Distributed system en_US dc.subject (關鍵詞) System scalability en_US dc.subject (關鍵詞) Online payments en_US dc.subject (關鍵詞) Account verification en_US dc.title (題名) 分散式系統Kafka的延展性研究：以支付應用為例 zh_TW dc.title (題名) An empirical study on the scalability of Kafka in a scenario of online payments en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) Gururajapathy, S. S., Mokhlis, H., & Illias, H. A. (2017). Fault location and detection techniques in power distribution systems with distributed generation: A review. Renewable and Sustainable Energy Reviews, 74, 949–958.Kersting, W. H. (2017). Distribution system modeling and analysis. CRC press.Kumar, M., & Singh, C. (2017). Building Data Streaming Applications with Apache Kafka. Packt Publishing Ltd.Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: the definitive guide: real-time data and stream processing at scale. “ O’Reilly Media, Inc.”Shapira, G., Palino, T., Sivaram, R., & Petty, K. (2021). Kafka: the definitive guide. “ O’Reilly Media, Inc.”Shree, R., Choudhury, T., Gupta, S. C., & Kumar, P. (2017). KAFKA: The modern platform for data management and analysis in big data domain. 2017 2nd International Conference on Telecommunication and Networks (TEL-NET), 1–5.Vohra, D., & Vohra, D. (2016). Apache kafka. Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools, 339–347.Wang, G., Chen, L., Dikshit, A., Gustafson, J., Chen, B., Sax, M. J., Roesler, J., Blee-Goldman, S., Cadonna, B., & Mehta, A. (2021). Consistency and completeness: Rethinking distributed stream processing in apache kafka. Proceedings of the 2021 International Conference on Management of Data, 2602–2613.Wu, H. (2019). Research proposal: Reliability evaluation of the apache kafka streaming system. 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), 112–113.Wu, H., Shang, Z., & Wolter, K. (2019). Performance prediction for the apache kafka messaging system. 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 154–161. zh_TW
