Introduction - Discriminative vs Generative models

A father has two kids, Kid A and Kid B. Kid A has a special character whereas he can learn everything in depth. Kid B have a special character, whereas he can only learn the differences between what he saw.

One fine day, The father takes his 2 kids (A and B) to a zoo. This zoo is a very small one and has only two kinds of animals, say a lion and an elephant. After they came out of the zoo, the father showed them an animal and asked both of them, “is this animal a lion or an elephant?”

The Kid A, the kid suddenly draw the image of lion and elephant in a piece of paper based on what he saw inside the zoo. He compared both the images with the animal standing before and answered based on the closest match ****of image & animal, he answered: “The animal is Lion”.

The Kid B knows only the differences, based on different properties learned, he answered: “The animal is a Lion”.

Here, we can see both of them is finding the kind of animal, but the way of learning and the way of finding answer is entirely different. In Machine Learning, we generally call Kid A as a Generative Model & Kid B as a Discriminative Model.

A Generative model models the joint probability function $p(x|y)$ and $p(y)$, or $p(x,y)$ to derive $p(y|x)$ by using the Bayes’ theorem. A Generative Model ‌explicitly models the actual distribution of each class.

With Discriminative models, we model the conditional probability distribution $p(y|x)$ or $h(x)$ directly.

For example, Parzen classifier by definition estimates the data distribution, therefore it's generative. Decision tree does not estimate data distribution, therefore it is discriminative.

Until now, we have looked into generative classifiers:

In this lecture, we will look into Linear Discriminative classifiers: Logistic regression and SVC (and later on, non-linear classifiers).

<aside> 📝 Which of the following best describes what probabilistic discriminative approaches try to model? ($w$ are the parameters of the model)

• $p(y|x,w)$
• $p(y|w)$
• $p(x|y,w)$
• $p(w|x,y)$
• Answer </aside>

<aside> 📝 Let $x$ be the data, and $y$ the labels. Which are true?

1. Discriminative classifiers learn $p(x|y)$
2. Generative classifiers learn $p(x|y)$
3. Generative classifiers make stronger assumptions about the class-conditional data distributions than Discriminative classifiers
4. Generative models do not optimize a loss
5. If your test data has a different but known class distribution than your training data, you can easily adapt a trained Generative model accordingly to reduce the expected error

• Answer </aside>

How to define a classifier/regressor through a loss function and hypothesis class

Suppose I have a problem and I gather an infinitely amount of data. These data can be of 2 classes, blue or orange. I calculate the joint conditional probability and plot it: