CycleCoopNet: 基於合作學習的神經網路進行圖片轉換

Abstract

本文提出了一種新的圖像到圖像轉換方法，CycleCoopNet。圖像到圖像的轉換是一種將圖片從一種樣式更改為另一種樣式的方法，透過該方法，我們可以創建不存在的新穎圖片。 CycleCoopNet採用CoopNet框架，具有兩個主要模型，稱為generator和descriptor。generator生成圖片，該圖片由descriptor通過MCMC（Markov Chain Monte Carlo）採樣進行修訂，因此可以從descriptor指導的監督式學習中讓generator學習。另一方面，descriptor透過 modified contrastive divergence從數據中學習，使得descriptor被調整為與修改後的數據和實數據輸出相同的結果。\n先前的一些論文也有實作圖像到圖像的轉換方法。 CycleGAN是與我們的作品類似的著名作品之一，它使用GAN（生成對抗網絡）的概念來演示此方法。它演示了進行圖像到圖像轉換的良好性能。但是，CycleGAN通過無監督學習生成圖片，也就是說，generator的結果在學習過程中沒有標準的生成圖片答案。 CycleGAN僅使用discriminator來確定結果是正確還是不正確。每個結果僅需要通過discriminator測試，這可以使generator僅需要找到如何通過discriminator測試，而無需嘗試找到正確的生成答案或更多可能的答案。這個問題我們稱為Mode collapse，它導致結果的變異性較小，也就是說，generator始終生成相同的圖片，以獲得更好的分數。\n在我們的實驗中，我們使用edge2handbags數據集來觀察圖片如何從草圖更改為背包。我們發現我們的模型可以產生更多不同的結果。並且這些結果可以被另一個相反的generator模型穩定地恢復到原始圖片。另一個實驗我們使用vangogh2photo數據集來觀察圖片如何從照片變為VanGogh風格的圖片。我們展示了我們的模型可以做出更好的變化。\n我們的目標是透過將discriminator更改為descriptor來升級該網路。descriptor模型改編自CoopNet（合作神經網絡）。這個想法改變discriminator（descriptor）卷積網絡的輸出尺寸。使用descriptor可以讓我們的generator標記答案來調整其模型參數，並具有將此問題更改為監督式學習問題的能力。另外，使用descriptor可以防止Mode collapse。避免generator始終生成相似的結果。

This paper proposes a new Image-to-Image translation method, CycleCoopNet. The image-to-image translation is a method of changing pictures from one style to another style, with which we can create novel pictures that do not exist. CycleCoopNet adopts the CoopNet framework with two main models called generator and descriptor. The generator generates pictures that are revised by the descriptor with MCMC (Markov Chain Monte Carlo) sampling, thus the generator is learned from supervised learning guided by the descriptor. On the other hand, the descriptor learns from real data by modified contrastive divergence, such that the descriptor is adjusted to output the same vector from the revised data and the real data.\nSeveral previous works are doing the Image-to-Image translation method. CycleGAN is one of the famous work doing similar working as our work, it used the concept of GAN (generative adversarial network) to demonstrate this method. It demonstrates the nice performance of doing Image-to-Image translation. However, CycleGAN generating pictures by unsupervised learning, that is, the results of the generator does NOT have a standard generated pictures answer in the learning process. CycleGAN only uses the discriminator to decide the results are correct or incorrect. Every result only needs to pass the discriminator testing, this can make the generator only need to find how to pass the discriminator testing and NOT trying to find the correct generated answer or more possible answers. This problem we called Mode collapse, that causes the results with less variability, that is, the generator always generates the same picture cheating discriminator to getting a better score.\nIn our experiments, we use the edges2handbags dataset to observe how does the picture change from sketches to bags. We found that our model can generate more diverse results. And these results can be recovered to the origin picture by another opposite generator model stably. Another experiment we use vangogh2photo dataset to observe how does the picture change from photos to VanGogh-style pictures. We show our model can make a better variety.\nOur goal is to upgrade this network by changing the discriminator to the descriptor. The descriptor model is adapted from the CoopNet(Cooperative Neural Network). The idea is changing the output dimension of the discriminator (descriptor) convolutional network. Using the descriptor can let our generator have labeled answer to adjust its model parameters, and change this problem to supervised learning. Also, using a descriptor can prevent from the Mode collapse. Avoid the generator always generate similar patterns.