We’ll talk about GANs (generative adversarial network) today, because it is kind of interesting, at least to me. GANs are basically made up of 2 things - generator and discriminator. That’s all. Now there are many fancier versions of GANs in itself, but in this blog, we’re going to talk only about simple GAN.
The two main components here are Generator and Discriminator.
Discriminator - Nothing but a classifier. Decides whether an image is real or fake.
Generator - Basically takes in random noise and creates images that will look similar to the real images. It will try to fool the discriminator by making it believe that the images generated by generator are real when in reality, they’re fake.
Loss function -
It took a while to me to understand why one is maximising and other is minimising, but I’ll try to simplify.
The discriminator wants to classify correctly, meaning real as real and fake as fake. For a real image, D(x) is very close to 1, hence the log term will be very close to 0. For a fake image, D(G(z)) = 0, hence the term will be log (1 - 0) ⇒ 0 Hence, total loss becomes 0. This is the maximum it can get (theoretically). Why? let’s say D(x) = 0.8 ⇒ log (0.8) = -0.223 and let’s say D(fake) = 0.4, then log(1 - 0.4) = -0.510 hence, total = -0.733, and if you’ll compute for any value, it will be negative, the max value it can achieve is 0. Hence, we say that the discriminator is trying to max out the loss function.
Now, the generator affects the second term only. It will try to make D(G(z)) as close as possible to 1 to make the discriminator believe that the images are real, and if D(G(z)) becomes 1 , then it is log(0), basically very negative, let’s say D(G(z)) is 0.5, hence the term will be -0.693, and if its 0.9, then its -2.30, and if D(G(z)) is 0, it means discriminator caught the image to be fake. Hence, we say, the generator tries to minimise the loss because it wants D(G(z)) to be as close to 1 as possible, to fool the discriminator. But in practice this loss causes gradient saturation, so we use the non-saturating generator loss: -log(D(G(z))), because when D(G(z)) is close to 0, the gradient of log(1 − D(G(z))) becomes very small, so the generator learns slowly.
Overall, this becomes a min - max game type of situation, where one player is trying to max and other is trying to min.
I’ve also implemented a Simple GAN from scratch and you can check it out - https://github.com/Moksh-10/ML-Projects/blob/main/GAN/Simple-GAN/model.py
References -