Publications-Theses
Article View/Open
Publication Export
-
題名 基於深度學習框架之夜晚霧霾圖像模擬與復原方法評估
Nighttime Haze Images Simulation and Restoration Using Deep Learning Frameworks作者 鄭可昕
Cheng, Ko-Hsin貢獻者 廖文宏
Liao, Wen-Hung
鄭可昕
Cheng, Ko-Hsin關鍵詞 深度學習
夜晚圖像
霧霾模擬
圖像去霧
圖像復原
Deep learning
Nighttime images
Fog simulation
Haze removal
Image restoration日期 2021 上傳時間 2-Mar-2021 14:57:22 (UTC+8) 摘要 近年來氣候異常、空氣污染問題日漸嚴重，使得日常中發生霧霾現象的次數越來越多，在霧霾環境中拍攝的圖像，會使得圖像的清晰度與對比度大幅降低，當霧霾現象發生在夜晚，伴隨燈光的干擾，其圖像品質更差。隨著深度學習在圖像領域研究成果的突破，如何將深度學習方法應用於霧霾圖像的復原與去霧，逐漸成為研究者感興趣的主題之一。本研究以霧霾圖像形成原理與圖像深度為概念，結合生成對抗網路、大氣散射模型與圖像深度估計等方法，在清晰的夜晚圖像上，疊加霧霾效果，模擬出夜晚霧霾圖像，並透過深度學習方法，將模擬的圖像作為訓練資料，訓練一組模型能夠應用在復原模擬的夜晚霧霾圖像。為了對模型進一步評估與分析，本研究亦使用真實夜晚霧霾圖像做測試，檢驗模型的泛化能力。此外，為能更客觀地確認去霧成效，我們計算並比較復原前與復原後圖像之圖像品質指標，以及使用YOLOv5目標偵測方法，以所得之mAP作為衡量基準，均可觀察到處理前後的明顯差異。
In recent years, extreme weather and air pollution problems have become serious, causing the frequent formation of haze in our daily life. Images taken in a haze environment will significantly lose sharpness and contrast. When haze occurs at nighttime, the image quality worsens due to the interference of light. With the rapid progress of deep learning in the field of computer vision, applying deep neural networks to the restoration of images degraded by haze has become one of the topics of interest to researchers.This research employs the concept of fog and haze image formation and image depth, in combination with generative adversarial network, atmospheric scattering model, and image depth estimation, to simulate nighttime haze images by superimposing the haze on clear nighttime images. Based on deep learning methods, the simulated images are used as training data to build a model that can successfully restore the simulated nighttime haze images.To evaluate the effectiveness of the proposed approach, we also use real nighttime haze images to observe the generalization ability of the model. To examine the effect of dehazing in an objective manner, several image quality indices have been computed and compared. Additionally, the YOLOv5 object detection method has been utilized to calculate the mAP of the detection before and after restoration. All results indicate improved performance after image dehazing.參考文獻 [1] 中央氣象局數位科普網:迷茫之城-是霧，或是霾? https://pweb.cwb.gov.tw/PopularScience/index.php/weather/277[2] ImageNet Large Scale Visual Recognition Competition (ILSVRC). http://www.image-net.org/challenges/LSVRC/[3] 國立台灣大學計算機及資訊網路中心電子報，第 0038 期，ISSN:2077- 8813, http://www.cc.ntu.edu.tw/chinese/epaper/0038/20160920_3805.html[4] Michael A. Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015. http://neuralnetworksanddeeplearning.com/index.html[5] ImageNet Winning CNN Architectures (ILSVRC). https://www.kaggle.com/getting-started/149448[6] McCartney, E. J. (1976). Optics of the atmosphere: scattering by molecules and particles. nyjw.[7] He, K., Sun, J., & Tang, X. (2010). Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2341-2353.[8] Zhang, T., Shao, C., & Wang, X. (2011, October). Atmospheric scattering- based multiple images fog removal. In 2011 4th International Congress on Image and Signal Processing (Vol. 1, pp. 108-112). IEEE.[9] Zhu, J. X., Meng, L. L., Wu, W. X., Choi, D., & Ni, J. J. (2020). Generative adversarial network-based atmospheric scattering model for image dehazing. Digital Communications and Networks.[10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).[11] Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).[12] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to- image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).[13] Anoosheh, A., Sattler, T., Timofte, R., Pollefeys, M., & Van Gool, L. (2019, May). Night-to-day image translation for retrieval-based localization. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 5958-5964). IEEE.[14] Li, Z., & Snavely, N. (2018). Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2041-2050).[15] Godard, C., Mac Aodha, O., Firman, M., & Brostow, G. J. (2019). Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE international conference on computer vision (pp. 3828-3838).[16] Sakaridis, C., Dai, D., & Van Gool, L. (2018). Semantic foggy scene understanding with synthetic data.International Journal of Computer Vision, 126(9), 973-992.[17] Zhang, N., Zhang, L., & Cheng, Z. (2017, November). Towards simulating foggy and hazy images and evaluating their authenticity. InInternational Conference on Neural Information Processing (pp. 405-415). Springer, Cham.[18] W. Maddern, G. Pascoe, C. Linegar and P. Newman, "1 Year, 1000km: The Oxford RobotCar Dataset", The International Journal of Robotics Research (IJRR), 2016.[19] He, K., Sun, J., & Tang, X. (2010). Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2341-2353.[20] Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016). Dehazenet: An end-to- end system for single image haze removal. IEEE Transactions on Image Processing, 25(11), 5187-5198.[21] Li, Y., Tan, R. T., & Brown, M. S. (2015). Nighttime haze removal with glow and multiple light colors. In Proceedings of the IEEE international conference on computer vision (pp. 226-234).[22] Dong, H., Pan, J., Xiang, L., Hu, Z., Zhang, X., Wang, F., & Yang, M. H. (2020). Multi-Scale Boosted Dehazing Network with Dense Feature Fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2157-2167).[23] Qin, X., Wang, Z., Bai, Y., Xie, X., & Jia, H. (2020). FFA-Net: Feature Fusion Attention Network for Single Image Dehazing. In AAAI (pp. 11908-11915).[24] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600-612.[25] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).[26] Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012). No-reference image quality assessment in the spatial domain.IEEE Transactions on image processing, 21(12), 4695-4708.[27] Mittal, A., Soundararajan, R., & Bovik, A. C. (2012). Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 20(3), 209-212.[28] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., ... & Darrell, T. (2020). BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2636-2645).[29] Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., & Wang, Z. (2017). Reside: A benchmark for single image dehazing. arXiv preprint arXiv:1712.04143, 1.[30] Jocher, G. (2020). Yolov5. Code repository https://github.com/ultralytics/yolov5.[31] Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImg[32] IEEE International Conference on Multimedia and Expo 2020 Grand Challenge – Embedded Deep Learning Object Detection Model Compression Competition for Traffic in Asian Countries. http://2020.ieeeicme.org/www.2020.ieeeicme.org/index.php/grand- challenges/index.html[33] Oxford Robotics Institute. Software Development Kit for the Oxford Robotcar Dataset. Git code repository. https://github.com/ori-mrg/robotcar-dataset- sdk[34] 交通部高速公路局「交通資料庫」CCTV 動態資訊(v1.1) https://tisvcloud.freeway.gov.tw/ 描述 碩士
國立政治大學
資訊科學系碩士在職專班
107971017資料來源 http://thesis.lib.nccu.edu.tw/record/#G0107971017 資料類型 thesis dc.contributor.advisor 廖文宏 zh_TW dc.contributor.advisor Liao, Wen-Hung en_US dc.contributor.author (Authors) 鄭可昕 zh_TW dc.contributor.author (Authors) Cheng, Ko-Hsin en_US dc.creator (作者) 鄭可昕 zh_TW dc.creator (作者) Cheng, Ko-Hsin en_US dc.date (日期) 2021 en_US dc.date.accessioned 2-Mar-2021 14:57:22 (UTC+8) - dc.date.available 2-Mar-2021 14:57:22 (UTC+8) - dc.date.issued (上傳時間) 2-Mar-2021 14:57:22 (UTC+8) - dc.identifier (Other Identifiers) G0107971017 en_US dc.identifier.uri (URI) http://nccur.lib.nccu.edu.tw/handle/140.119/134204 - dc.description (描述) 碩士 zh_TW dc.description (描述) 國立政治大學 zh_TW dc.description (描述) 資訊科學系碩士在職專班 zh_TW dc.description (描述) 107971017 zh_TW dc.description.abstract (摘要) 近年來氣候異常、空氣污染問題日漸嚴重，使得日常中發生霧霾現象的次數越來越多，在霧霾環境中拍攝的圖像，會使得圖像的清晰度與對比度大幅降低，當霧霾現象發生在夜晚，伴隨燈光的干擾，其圖像品質更差。隨著深度學習在圖像領域研究成果的突破，如何將深度學習方法應用於霧霾圖像的復原與去霧，逐漸成為研究者感興趣的主題之一。本研究以霧霾圖像形成原理與圖像深度為概念，結合生成對抗網路、大氣散射模型與圖像深度估計等方法，在清晰的夜晚圖像上，疊加霧霾效果，模擬出夜晚霧霾圖像，並透過深度學習方法，將模擬的圖像作為訓練資料，訓練一組模型能夠應用在復原模擬的夜晚霧霾圖像。為了對模型進一步評估與分析，本研究亦使用真實夜晚霧霾圖像做測試，檢驗模型的泛化能力。此外，為能更客觀地確認去霧成效，我們計算並比較復原前與復原後圖像之圖像品質指標，以及使用YOLOv5目標偵測方法，以所得之mAP作為衡量基準，均可觀察到處理前後的明顯差異。 zh_TW dc.description.abstract (摘要) In recent years, extreme weather and air pollution problems have become serious, causing the frequent formation of haze in our daily life. Images taken in a haze environment will significantly lose sharpness and contrast. When haze occurs at nighttime, the image quality worsens due to the interference of light. With the rapid progress of deep learning in the field of computer vision, applying deep neural networks to the restoration of images degraded by haze has become one of the topics of interest to researchers.This research employs the concept of fog and haze image formation and image depth, in combination with generative adversarial network, atmospheric scattering model, and image depth estimation, to simulate nighttime haze images by superimposing the haze on clear nighttime images. Based on deep learning methods, the simulated images are used as training data to build a model that can successfully restore the simulated nighttime haze images.To evaluate the effectiveness of the proposed approach, we also use real nighttime haze images to observe the generalization ability of the model. To examine the effect of dehazing in an objective manner, several image quality indices have been computed and compared. Additionally, the YOLOv5 object detection method has been utilized to calculate the mAP of the detection before and after restoration. All results indicate improved performance after image dehazing. en_US dc.description.tableofcontents 謝辭 i摘要 iiAbstract iii目錄 iv圖目錄 vii表目錄 ix第一章 緒論 11.1 研究背景與動機 11.2 研究目的 21.3 論文架構 3第二章 技術背景與相關研究 42.1 深度學習的發展背景 42.2 霧霾圖像形成原理 62.3 生成對抗網路 72.3.1 pix2pix 82.3.2 CycleGAN 92.3.3 ToDayGAN 92.4 基於深度學習的圖像深度資訊估計 102.5 合成霧霾方法 112.6 圖像去霧方法 122.6.1 基於傳統圖像處理 122.6.2 基於深度神經網路 132.7 圖像品質評估指標 132.7.1 峰值訊噪比(PSNR) 142.7.2 結構相似性(SSIM) 142.7.3 均方誤差(MSE) 152.7.4 學習感知圖像塊相似度(LPIPS) 152.7.5 無參考圖像空間品質評估(BRISQUE) 152.7.6 自然圖像品質評估(NIQE) 152.8 目標偵測評估指標 162.8.1 精確率與召回率 162.8.2 IoU 與 mAP 17第三章 研究方法 183.1 基本構想 183.2 前期研究 193.2.1 夜晚霧霾圖像一階段生成 203.2.2 夜晚霧霾圖像二階段生成 223.2.3 資料標註 243.2.4 夜晚霧霾圖像去霧 253.2.5 圖像去霧結果評估 263.3 研究架構設計 263.3.1 問題陳述 263.3.2 研究架構 263.4 目標設定 29第四章 研究過程與結果分析 304.1 實驗環境 304.2 研究過程 314.2.1 夜晚無霧圖像資料集 314.2.2 白天圖像生成 324.2.3 深度資訊估計 344.2.4 夜晚霧霾圖像模擬 354.2.5 去霧預訓練模型評估 374.2.6 去霧模型重新訓練 384.2.7 去霧效果評估 424.3 結果分析 434.3.1 圖像品質評估分析與討論 434.3.2 霧霾圖像與去霧圖像之目標偵測結果分析 444.3.3 真實夜晚霧霾圖像套用去霧模型結果分析 504.4 研究結果之應用 51第五章 結論與未來研究方向 525.1 結論 525.2 未來研究方向 53參考文獻 54 zh_TW dc.format.extent 71099837 bytes - dc.format.mimetype application/pdf - dc.source.uri (資料來源) http://thesis.lib.nccu.edu.tw/record/#G0107971017 en_US dc.subject (關鍵詞) 深度學習 zh_TW dc.subject (關鍵詞) 夜晚圖像 zh_TW dc.subject (關鍵詞) 霧霾模擬 zh_TW dc.subject (關鍵詞) 圖像去霧 zh_TW dc.subject (關鍵詞) 圖像復原 zh_TW dc.subject (關鍵詞) Deep learning en_US dc.subject (關鍵詞) Nighttime images en_US dc.subject (關鍵詞) Fog simulation en_US dc.subject (關鍵詞) Haze removal en_US dc.subject (關鍵詞) Image restoration en_US dc.title (題名) 基於深度學習框架之夜晚霧霾圖像模擬與復原方法評估 zh_TW dc.title (題名) Nighttime Haze Images Simulation and Restoration Using Deep Learning Frameworks en_US dc.type (資料類型) thesis en_US dc.relation.reference (參考文獻) [1] 中央氣象局數位科普網:迷茫之城-是霧，或是霾? https://pweb.cwb.gov.tw/PopularScience/index.php/weather/277[2] ImageNet Large Scale Visual Recognition Competition (ILSVRC). http://www.image-net.org/challenges/LSVRC/[3] 國立台灣大學計算機及資訊網路中心電子報，第 0038 期，ISSN:2077- 8813, http://www.cc.ntu.edu.tw/chinese/epaper/0038/20160920_3805.html[4] Michael A. Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015. http://neuralnetworksanddeeplearning.com/index.html[5] ImageNet Winning CNN Architectures (ILSVRC). https://www.kaggle.com/getting-started/149448[6] McCartney, E. J. (1976). Optics of the atmosphere: scattering by molecules and particles. nyjw.[7] He, K., Sun, J., & Tang, X. (2010). Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2341-2353.[8] Zhang, T., Shao, C., & Wang, X. (2011, October). Atmospheric scattering- based multiple images fog removal. In 2011 4th International Congress on Image and Signal Processing (Vol. 1, pp. 108-112). IEEE.[9] Zhu, J. X., Meng, L. L., Wu, W. X., Choi, D., & Ni, J. J. (2020). Generative adversarial network-based atmospheric scattering model for image dehazing. Digital Communications and Networks.[10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).[11] Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).[12] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to- image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).[13] Anoosheh, A., Sattler, T., Timofte, R., Pollefeys, M., & Van Gool, L. (2019, May). Night-to-day image translation for retrieval-based localization. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 5958-5964). IEEE.[14] Li, Z., & Snavely, N. (2018). Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2041-2050).[15] Godard, C., Mac Aodha, O., Firman, M., & Brostow, G. J. (2019). Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE international conference on computer vision (pp. 3828-3838).[16] Sakaridis, C., Dai, D., & Van Gool, L. (2018). Semantic foggy scene understanding with synthetic data.International Journal of Computer Vision, 126(9), 973-992.[17] Zhang, N., Zhang, L., & Cheng, Z. (2017, November). Towards simulating foggy and hazy images and evaluating their authenticity. InInternational Conference on Neural Information Processing (pp. 405-415). Springer, Cham.[18] W. Maddern, G. Pascoe, C. Linegar and P. Newman, "1 Year, 1000km: The Oxford RobotCar Dataset", The International Journal of Robotics Research (IJRR), 2016.[19] He, K., Sun, J., & Tang, X. (2010). Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12), 2341-2353.[20] Cai, B., Xu, X., Jia, K., Qing, C., & Tao, D. (2016). Dehazenet: An end-to- end system for single image haze removal. IEEE Transactions on Image Processing, 25(11), 5187-5198.[21] Li, Y., Tan, R. T., & Brown, M. S. (2015). Nighttime haze removal with glow and multiple light colors. In Proceedings of the IEEE international conference on computer vision (pp. 226-234).[22] Dong, H., Pan, J., Xiang, L., Hu, Z., Zhang, X., Wang, F., & Yang, M. H. (2020). Multi-Scale Boosted Dehazing Network with Dense Feature Fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2157-2167).[23] Qin, X., Wang, Z., Bai, Y., Xie, X., & Jia, H. (2020). FFA-Net: Feature Fusion Attention Network for Single Image Dehazing. In AAAI (pp. 11908-11915).[24] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600-612.[25] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).[26] Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012). No-reference image quality assessment in the spatial domain.IEEE Transactions on image processing, 21(12), 4695-4708.[27] Mittal, A., Soundararajan, R., & Bovik, A. C. (2012). Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 20(3), 209-212.[28] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., ... & Darrell, T. (2020). BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2636-2645).[29] Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., & Wang, Z. (2017). Reside: A benchmark for single image dehazing. arXiv preprint arXiv:1712.04143, 1.[30] Jocher, G. (2020). Yolov5. Code repository https://github.com/ultralytics/yolov5.[31] Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImg[32] IEEE International Conference on Multimedia and Expo 2020 Grand Challenge – Embedded Deep Learning Object Detection Model Compression Competition for Traffic in Asian Countries. http://2020.ieeeicme.org/www.2020.ieeeicme.org/index.php/grand- challenges/index.html[33] Oxford Robotics Institute. Software Development Kit for the Oxford Robotcar Dataset. Git code repository. https://github.com/ori-mrg/robotcar-dataset- sdk[34] 交通部高速公路局「交通資料庫」CCTV 動態資訊(v1.1) https://tisvcloud.freeway.gov.tw/ zh_TW dc.identifier.doi (DOI) 10.6814/NCCU202100281 en_US
