基於轉換器之跨語言語者辨識強健性分析

Abstract

語者辨識廣泛運用於生活之中，小至語音助理，大至犯罪鑑識。隨著深度學習技術進展，語者辨識正確率逐步提升，不過多數研究專注於單一語言，鮮少處理跨語言的語者辨識任務，跨語言資料集亦相當稀少。本研究錄製跨語言資料集MET-40，其中每位參與者皆以三種語言（包含華語、英語和台語）錄音，當中有40位參與者，男性與女性人數各半，總和時長為740分鐘。華語、英語與台語文本主要取自小學教材，部分英語文本採用TIMIT資料集，錄製後會評估每位參與者運用個別語言之流暢度。\n 本論文採用轉換器與卷積網路為概念的網路架構，探討單一語言、混合語言、跨語言之語者辨識，訓練模型包含ResNet、Vision Transformer（ViT）和Convolutional Vision Transformer（CvT），搭配三種常用語音特徵（頻譜圖、梅爾頻譜圖和梅爾倒頻譜係數圖）進行實驗。混合語言與跨語言差異在於識別語言是否加入訓練資料，混合語言是測試之語言已存在於訓練集，反之跨語言則是識別語言不在訓練集。實驗結果在MET-40單一語言模型下，最高可達97.16%準確率。與單一語言模型相比，混合兩種語言以上之模型最高識別率為99.17%。跨語言識別中，語者混合越多種類語言有助於提升模型之泛化性。在我們實驗中，單一語言模型在跨語言識別最高準確率為79.64%，混合兩種語言以上之模型於跨語言辨識最高準確率為90.92%。實驗結果顯示CvT不易受抽取特徵影響，且具有較佳泛化性，不論是單一語言、混合語言或跨語言，整體識別度最佳。\n 模型的強健度，攸關應用時安全性，因此本論文分析不同模型的語者辨識遭致對抗例攻擊影響程度。從對抗例攻擊實驗結果，訓練與測試選擇相同語言資料集，可利用FGSM與PGD產生有效攻擊。進而探討跨語言攻擊之可轉移性，其中特徵使用頻譜圖或梅爾頻譜圖之模型攻擊不具可轉移性，而梅爾倒頻譜係數圖特徵雖在語者識別任務有卓越表現，但易受對抗例攻擊影響使得識別率降低。即使沒有訓練資料依然能產生攻擊，在FGSM跨語言攻擊中平均下降31.57%識別率，因此採用梅爾倒頻譜係數圖特徵之模型需要更加小心保護。

Speaker recognition is widely used in daily life, ranging from voice assistants to criminal forensics. With the rapid progress in deep learning technology, accuracy of speaker recognition has increased accordingly. However, most studies focus on a single language. Cross-language speaker recognition is rarely investigated. Cross-language data sets are also quite scarce. This study collected trilingual (including Mandarin, English, and Taiwanese) cross-language recordings named MET-40. A total of 40 participants (20 male, 20 female) contribute to the dataset which contains 740 minutes of audio. Mandarin, English and Taiwanese texts are mainly taken from elementary school textbooks, and some English texts use TIMIT. The fluency of individual participant in each language is also evaluated.\n\nWe employ ResNet, vision transformer (ViT), and convolutional vision transformer (CvT) in combination with three acoustic features, namely, spectrogram, Mel spectrogram and Mel frequency cepstral coefficient) for single, mixed and cross-language speaker recognition tasks. In the mixed-language setting, the language to be tested is included in the training set, while in the cross-language scenario the language to be tested is not used for training. Experimental results show that the highest accuracy is 97.16% for single language models. Mixture of two languages improves the performance to 99.17%. In cross-language situations, the accuracy drops significantly to 79.64%, as the spoken language is not present in the training data. When two languages are employed for training, the accuracy increased to 90.92%. In general, CvT-based models demonstrate best generalization ability in all cases.\n\nThe robustness of the model is critical to security in practical applications. Therefore, we analyze how adversarial attacks impact different speaker identification models. Experimental results reveal that if the same language dataset is selected for training and testing, FGSM and PGD attacks can be effectively generated. In the case of cross-language models, however, adversarial attacks using spectrogram or Mel-spectrogram are not transferable. Finally, when MFCC is chosen to be the acoustic feature, extra caution needs to be taken as attacks can still take place without training data, and the recognition rate is reduced by 31.57% using FGSM cross-language attack.