Generative Adversarial Networks refer to a family of generative models that seek to discover the underlying distribution behind a certain data generating process. This distribution is discovered through an adversarial competition between a generator and a discriminator. As we saw in an earlier introductory post on GANs, the two models are trained such that the discriminator strives to distinguish between generated and true examples, while the generator seeks to confuse the discriminator by producing data that are as realistic and compelling as possible.
In this post, we’ll take a deep dive into the math behind GANs. My primary source of reference is Generative Adversarial Nets by Ian Goodfellow, et al. It is in this paper that Goodfellow first outlined the concept of a GAN, which is why it only makes sense that we commence from the analysis of this paper. Let’s begin!
Motivating the Loss Function
GAN can be seen as an interplay between two different models: the generator and the discriminator. Therefore, each model will have its own loss function. In this section, let’s try to motivate an intuitive understanding of the loss function for each.
Notation
To minimize confusion, let’s define some notation that we will be using throughout this post.
The Discriminator
\[\begin{multline} \shoveleft x: \text{Real data} \\ \shoveleft z: \text{Latent vector} \\ \shoveleft G(z): \text{Fake data} \\ \shoveleft D(x): \text{Discriminator's evaluation of real data} \\ \shoveleft D(G(z)): \text{Discriminator's evaluation of fake data} \\ \shoveleft \text{Error}(a, b): \text{Error between } a \text{ and } b\\ \end{multline}\]
The goal of the discriminator is to correctly label generated images as false and empirical data points as true. Therefore, we might consider the following to be the loss function of the discriminator:
\[L_D = \text{Error}(D(x), 1) + \text{Error}(D(G(z)), 0) \tag{1}\]
... continue reading