學術產出-Theses
Article View/Open
Publication Export
-
題名 基於支持向量機計算的相互熵之特徵選取
A Feature Selection Study based on SVM and Mutual Entropy作者 游上葦
Yu, Shang-Wei貢獻者 周珮婷
游上葦
Yu, Shang-Wei關鍵詞 機器學習
特徵選取
維度縮減
支持向量機
支援向量機
熵
相互熵
Machine learning
Feature selection
Dimension reduction
Support Vector Machine
SVM
Entropy
Shannon Entropy
Mutual Entropy日期 2019 上傳時間 5-Sep-2019 15:42:06 (UTC+8) 摘要 特徵選取為機器學習領域中一重要部分，適當的選取特徵(變數)，除了減少機器運算時間、人力與金錢外，也可以避免模型過度配適或是欠擬和的情況發生。雖然經過多年發展已有很多特徵選取的方法，但同一種模型，不一定適用所有資料情況，因此提出新方法希望在特徵選取上會有更多選擇。 本文提出一新方法衡量變數之間的關係，使用資訊理論中熵的概念，結合分類器支持向量機，獲取變數間關係，並將變數分群，給予每群適當的相對應分數，以此篩選變數。本文採用半監督式學習，計算敏感度、特異度與準確度之平均及所使用變數數量，並以高斯混合模型搭配EM演算法利用KS檢定之檢定統計量定義資料重要變數，評估方法能否選取重要變數，本文一共使用2筆模擬資料與5筆真實資料，並將結果與各大方法比較，結果顯示在各資料集中皆有穩定表現，即使在變數少的情況下也能有不錯表現。
Feature selection technique plays a significant role in machine learning. Selecting features (variables) adequately can not only reduce the expenditure, operating time in machine and the cost of labor but also prevent under fitting or overfitting. Although lots of feature selection methods have been developed for decades, it is impossible to apply a unique method to all types of data sets. In this study, we propose a new method to calculate the correlation between variables based on the Shannon entropy from information theory and SVM classifier. Variables are grouped into several clusters and selected by the new correlation measurement. Besides, we define the importance of variable by the test statistic of KS test using Gaussian mixed model and E-M algorithm for the propose of result assessment. The performance of proposed method on two simulated data and five real data are demonstrated and compared with other feature selection methods. The predicted results are stable through the proposed method with a reduced dataset.參考文獻 Akay, M.F., (2009). Support vector machines combined with feature selection for breast cancer diagnosis. Expert Systems with Applications, 36 (2), 3240–3247.Arunasakthi, K., KamatchiPriya, & L., Askerunisa, A., (2014). Fisher Score Dimensionality Reduction for SVM Classification. 2014 International Conference on Innovations in Engineering and Technology, 3(3), 1900-1904.Blum, A., Langley, P., Selection of relevant features and examples in machine learning. Artificial Intelligence, 97(1-2), 245-271.Bennasar, M., Hicks, Y., & Setchi, R., (2015). Feature selection using Joint Mutual Information Maximisation. Expert Systems with Applications, 42, 8520-8532.Boser, B.E., Guyon, I.M., & Vapnil, V.N., (1992). A Training Algorithm for Optimal Margin Classiers. Proceedings of the fifth annual workshop on Computational learning theory, 144-152.Breiman, L., (1996). Bagging Predictors. Machine Learning, 24, 123–140.Breiman, L., (2001). Random Forests. Machine Learning, 45(1), 5-32. doi:10.1023/a:1010933404324Chen, Y.W., & Lin, C.J., (2005). Combining SVMs with Various Feature Selection Strategies. Department of Computer Science, National Taiwan University.https://www.csie.ntu.edu.tw/~cjlin/papers/features.pdfCortes, C., & Vapnik, V., (1995). Support-vector networks. Machine Learning, 20(3), 273-297. doi:10.1007/bf00994018Dash, M., & Liu, H., (1997). Feature selection for classification. Intelligent Data Analysis, 1(1-4), 131-156.Díaz-Uriarte, R., & Alvarez de Andrés, S., (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics, 7(1), 3.doi:10.1186/1471-2105-7-3Guyon, I., & Elisseeff, A., (2003). An introduction to variable and feature selection. Journal of machine learning research, 3(March), 1157-1182.Haldurai, H., Madhubala, T., & Rajalakshmi, R., (2016). A Study on Genetic Algorithm and its Applications. International Journal of Computer Sciences and Engineering, 4(10), 139-143.Ho, T.K., (1995). Random decision forests. Proceedings of the Third International Conference on Document Analysis and Recognition, 1, 278.Ho, T.K., (1998). The Random Subspace Method for Constructing Decision Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832-844.Holland, J.H., (1992). Genetic Algorithms. Scientific American, 267(1), 66-73.Hsieh, F., Liu, S.Y., Hsieh, Y.C., McCowan, B., (2018). From patterned response dependency to structured covariate dependency: Entropy based categorical- pattern-matching. PLoS ONE, 13(6).Hsu, C.W., Chang, C.C., & Lin, C.J., (2003). Machine Learning with Applications in Breast Cancer Diagnosis and Prognosis. Department of Computer Science, National Taiwan University. https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdfHuang, C.L., & Wang, C.J., (2006). A GA-based feature selection and parameters optimization for support vector machines, Expert Systems with Applications, 31, 231-240.Jolliffe, I.T., & Cadima, J., (2016). Principal component analysis: A review and recent developments. Philosophical Transactions of The Royal Society A Mathematical Physical and Engineering Sciences, 374(2065), 20150202.Kumbhar, P., & Mali, M., (2016). A Survey on Feature Selection Techniques and Classification Algorithms for Efficient Text Classification. International Journal of Science and Research, 5(5), 1267-1275.Popovic, M., (2018). Researchers in an Entropy Wonderland: A Review of the Entropy Concept. Thermal Science, 22(2), 1163-1178.Raymer, M.L., Punch, W.F., Goodman, E.D., Kuhn, L.A., & Jain, A.K., (2000). Dimensionality reduction using genetic algorithms. IEEE Transactions on Evolutionary Computation, 4(2), 164-171.Kohavi, R., John, G., (1997). Wrappers for feature selection. Artificial Intelligence, 97(1-2), 273-324.Genuer, R., Poggi, J. M., & Tuleau-Malot, C., (2010). Variable selection using Random Forests. Pattern Recognition Letters, 31(14), 2225-2236.Saeys, Y., Inza, I., & Larrañaga, P., (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507-2517.Scho ̈lkopf, B., Sung, K.K., Burges, Chris J.C., Girosi, F., Niyogi, P., Poggio, T., & Vapnik, V., (1997). Comparing support vector machines with Gaussian kernels to radial basis function classifiers. IEEE Transactions on Signal Processing, 45(11), 2758-2765.Shannon, C. E., (1948). A Mathematical Theory of Communication. The Bell System Technical Journal. 27, 379-432, 623-656.Srivastava, S., Joshi, N., & Gaur, M., (2013). A Review Paper on Feature Selection Methodologies and Their Applications. International Journal of Engineering Research and Development, 7(6), 57-61.Tibshirani, R., (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267-288.Xu, X., (2006). Adaptive Intrusion Detection Based on Machine Learning: Feature Extraction, Classifier Construction and Sequential Pattern Prediction. International Journal of Web Services Practices, 2(1-2), 49-58.Yu, L., & Liu, H., (2003). Feature selection for high-dimensional data: A fast correlation based filter solution. Department of Computer Science & Engineering, Arizona State University. 描述 碩士
國立政治大學
統計學系
106354024資料來源 http://thesis.lib.nccu.edu.tw/record/#G0106354024 資料類型 thesis dc.contributor.advisor 周珮婷 zh_TW dc.contributor.author (Authors) 游上葦 zh_TW dc.contributor.author (Authors) Yu, Shang-Wei en_US dc.creator (作者) 游上葦 zh_TW dc.creator (作者) Yu, Shang-Wei en_US dc.date (日期) 2019 en_US dc.date.accessioned 5-Sep-2019 15:42:06 (UTC+8) - dc.date.available 5-Sep-2019 15:42:06 (UTC+8) - dc.date.issued (上傳時間) 5-Sep-2019 15:42:06 (UTC+8) - dc.identifier (Other Identifiers) G0106354024 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/125517 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 統計學系 zh_TW dc.description (描述) 106354024 zh_TW dc.description.abstract (摘要) 特徵選取為機器學習領域中一重要部分，適當的選取特徵(變數)，除了減少機器運算時間、人力與金錢外，也可以避免模型過度配適或是欠擬和的情況發生。雖然經過多年發展已有很多特徵選取的方法，但同一種模型，不一定適用所有資料情況，因此提出新方法希望在特徵選取上會有更多選擇。 本文提出一新方法衡量變數之間的關係，使用資訊理論中熵的概念，結合分類器支持向量機，獲取變數間關係，並將變數分群，給予每群適當的相對應分數，以此篩選變數。本文採用半監督式學習，計算敏感度、特異度與準確度之平均及所使用變數數量，並以高斯混合模型搭配EM演算法利用KS檢定之檢定統計量定義資料重要變數，評估方法能否選取重要變數，本文一共使用2筆模擬資料與5筆真實資料，並將結果與各大方法比較，結果顯示在各資料集中皆有穩定表現，即使在變數少的情況下也能有不錯表現。 zh_TW dc.description.abstract (摘要) Feature selection technique plays a significant role in machine learning. Selecting features (variables) adequately can not only reduce the expenditure, operating time in machine and the cost of labor but also prevent under fitting or overfitting. Although lots of feature selection methods have been developed for decades, it is impossible to apply a unique method to all types of data sets. In this study, we propose a new method to calculate the correlation between variables based on the Shannon entropy from information theory and SVM classifier. Variables are grouped into several clusters and selected by the new correlation measurement. Besides, we define the importance of variable by the test statistic of KS test using Gaussian mixed model and E-M algorithm for the propose of result assessment. The performance of proposed method on two simulated data and five real data are demonstrated and compared with other feature selection methods. The predicted results are stable through the proposed method with a reduced dataset. en_US dc.description.tableofcontents 第一章 緒論 1第二章 文獻探討 3第三章 研究目的 5一、特徵選取(feature selection) 5第四章 研究方法 6第一節 ENTROPY特徵選取 6一、Shannon Entropy 6二、Mutual Entropy 7三、Support Vector Machine(SVM) 10四、Feature selection via Mutual Entropy based on classified model 11五、分群規則 13六、方法步驟 15第二節 其他特徵選取方法 17一、Random forest(RF) 17二、F-score 18三、Least Absolute Shrinkage and Selection Operation(LASSO) 19四、Fast Correlation-Based Filter(FCBF) 20第五章 資料介紹 22一、模擬資料一 22二、模擬資料二 24三、Wisconsin Diagnostic Breast Cancer 26四、Connectionist Bench(Sonar, Mines vs. Rocks) 27五、Credit Card Fraud Detection 28六、Oxford Parkinson’s Disease 29七、APS Failure at Scania Trucks 30第六章 研究結果 33第一節 資料結果 35一、模擬資料一及模擬資料二 35二、Wisconsin Diagnostic Breast Cancer 38三、Connectionist Bench(Sonar, Mines vs. Rocks) 41四、Credit Card Fraud Detection 44五、Oxford Parkinson’s Disease 47六、APS Failure at Scania Trucks 50第二節 結論 53第七章 未來展望 54第八章 參考資料 55 zh_TW dc.format.extent 1809860 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0106354024 en_US dc.subject (關鍵詞) 機器學習 zh_TW dc.subject (關鍵詞) 特徵選取 zh_TW dc.subject (關鍵詞) 維度縮減 zh_TW dc.subject (關鍵詞) 支持向量機 zh_TW dc.subject (關鍵詞) 支援向量機 zh_TW dc.subject (關鍵詞) 熵 zh_TW dc.subject (關鍵詞) 相互熵 zh_TW dc.subject (關鍵詞) Machine learning en_US dc.subject (關鍵詞) Feature selection en_US dc.subject (關鍵詞) Dimension reduction en_US dc.subject (關鍵詞) Support Vector Machine en_US dc.subject (關鍵詞) SVM en_US dc.subject (關鍵詞) Entropy en_US dc.subject (關鍵詞) Shannon Entropy en_US dc.subject (關鍵詞) Mutual Entropy en_US dc.title (題名) 基於支持向量機計算的相互熵之特徵選取 zh_TW dc.title (題名) A Feature Selection Study based on SVM and Mutual Entropy en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) Akay, M.F., (2009). Support vector machines combined with feature selection for breast cancer diagnosis. Expert Systems with Applications, 36 (2), 3240–3247.Arunasakthi, K., KamatchiPriya, & L., Askerunisa, A., (2014). Fisher Score Dimensionality Reduction for SVM Classification. 2014 International Conference on Innovations in Engineering and Technology, 3(3), 1900-1904.Blum, A., Langley, P., Selection of relevant features and examples in machine learning. Artificial Intelligence, 97(1-2), 245-271.Bennasar, M., Hicks, Y., & Setchi, R., (2015). Feature selection using Joint Mutual Information Maximisation. Expert Systems with Applications, 42, 8520-8532.Boser, B.E., Guyon, I.M., & Vapnil, V.N., (1992). A Training Algorithm for Optimal Margin Classiers. Proceedings of the fifth annual workshop on Computational learning theory, 144-152.Breiman, L., (1996). Bagging Predictors. Machine Learning, 24, 123–140.Breiman, L., (2001). Random Forests. Machine Learning, 45(1), 5-32. doi:10.1023/a:1010933404324Chen, Y.W., & Lin, C.J., (2005). Combining SVMs with Various Feature Selection Strategies. Department of Computer Science, National Taiwan University.https://www.csie.ntu.edu.tw/~cjlin/papers/features.pdfCortes, C., & Vapnik, V., (1995). Support-vector networks. Machine Learning, 20(3), 273-297. doi:10.1007/bf00994018Dash, M., & Liu, H., (1997). Feature selection for classification. Intelligent Data Analysis, 1(1-4), 131-156.Díaz-Uriarte, R., & Alvarez de Andrés, S., (2006). Gene selection and classification of microarray data using random forest. BMC Bioinformatics, 7(1), 3.doi:10.1186/1471-2105-7-3Guyon, I., & Elisseeff, A., (2003). An introduction to variable and feature selection. Journal of machine learning research, 3(March), 1157-1182.Haldurai, H., Madhubala, T., & Rajalakshmi, R., (2016). A Study on Genetic Algorithm and its Applications. International Journal of Computer Sciences and Engineering, 4(10), 139-143.Ho, T.K., (1995). Random decision forests. Proceedings of the Third International Conference on Document Analysis and Recognition, 1, 278.Ho, T.K., (1998). The Random Subspace Method for Constructing Decision Forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832-844.Holland, J.H., (1992). Genetic Algorithms. Scientific American, 267(1), 66-73.Hsieh, F., Liu, S.Y., Hsieh, Y.C., McCowan, B., (2018). From patterned response dependency to structured covariate dependency: Entropy based categorical- pattern-matching. PLoS ONE, 13(6).Hsu, C.W., Chang, C.C., & Lin, C.J., (2003). Machine Learning with Applications in Breast Cancer Diagnosis and Prognosis. Department of Computer Science, National Taiwan University. https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdfHuang, C.L., & Wang, C.J., (2006). A GA-based feature selection and parameters optimization for support vector machines, Expert Systems with Applications, 31, 231-240.Jolliffe, I.T., & Cadima, J., (2016). Principal component analysis: A review and recent developments. Philosophical Transactions of The Royal Society A Mathematical Physical and Engineering Sciences, 374(2065), 20150202.Kumbhar, P., & Mali, M., (2016). A Survey on Feature Selection Techniques and Classification Algorithms for Efficient Text Classification. International Journal of Science and Research, 5(5), 1267-1275.Popovic, M., (2018). Researchers in an Entropy Wonderland: A Review of the Entropy Concept. Thermal Science, 22(2), 1163-1178.Raymer, M.L., Punch, W.F., Goodman, E.D., Kuhn, L.A., & Jain, A.K., (2000). Dimensionality reduction using genetic algorithms. IEEE Transactions on Evolutionary Computation, 4(2), 164-171.Kohavi, R., John, G., (1997). Wrappers for feature selection. Artificial Intelligence, 97(1-2), 273-324.Genuer, R., Poggi, J. M., & Tuleau-Malot, C., (2010). Variable selection using Random Forests. Pattern Recognition Letters, 31(14), 2225-2236.Saeys, Y., Inza, I., & Larrañaga, P., (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507-2517.Scho ̈lkopf, B., Sung, K.K., Burges, Chris J.C., Girosi, F., Niyogi, P., Poggio, T., & Vapnik, V., (1997). Comparing support vector machines with Gaussian kernels to radial basis function classifiers. IEEE Transactions on Signal Processing, 45(11), 2758-2765.Shannon, C. E., (1948). A Mathematical Theory of Communication. The Bell System Technical Journal. 27, 379-432, 623-656.Srivastava, S., Joshi, N., & Gaur, M., (2013). A Review Paper on Feature Selection Methodologies and Their Applications. International Journal of Engineering Research and Development, 7(6), 57-61.Tibshirani, R., (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267-288.Xu, X., (2006). Adaptive Intrusion Detection Based on Machine Learning: Feature Extraction, Classifier Construction and Sequential Pattern Prediction. International Journal of Web Services Practices, 2(1-2), 49-58.Yu, L., & Liu, H., (2003). Feature selection for high-dimensional data: A fast correlation based filter solution. Department of Computer Science & Engineering, Arizona State University. zh_TW dc.identifier.doi (DOI) 10.6814/NCCU201900996 en_US
