Models

Up to now, we assumed that we now the true $p(y|x)$ (or $p(x|y),p(y)$).

In practice, we only get a sample (training set), drawn from this distribution.

Then, we have to approximate $p(y|x)$ (or $p(x|y),p(y)$). We will consider:

• Discriminative models. Directly estimate $p(y|x)$, i.e., compute $\hat{p}(y|x)$.

  Model where we forget about the densities and just focus on the decision boundaries.

  When we know the posterior probability densities, we can directly classify objects. However, it is really hard to estimate these posterior probabilities. Hence, strong assumptions are needed.

• Generative models. Model the actual distribution of each class, and then learn the class conditional probability $p(x|y)$. Then apply Bayes’ rule to obtain $p(y|x)$.

  If we are using Bayes’ rule in order to get the posterior probabilities, we need the distribution of the classes. If we do, or at least estimate of one the classes well, you could be able to generate new samples from the class, making your own datasets.

  When we know the prior and conditional densities, we know everything about the data for classification. Density has to be estimated. Given examples from different classes, ‘standard’ density estimation is sufficient.

  Generative models assume you can generate new samples from the distributions.

This models at the same time can also be categorized in:

• Parametric models

  

  Generative model where you assume you really know the shape of the full distribution. You only do not know the parameters of this model. Hence, strong assumptions.

• Nonparametric models

  

  Generative model where you may have some vague assumptions, but it is not restricted to some specific shape. Hence, weak assumptions.

Classifiers we will learn in this course

Now, we will focus on the generative models: parametric and nonparametric.

<aside> 📝 Below follow four statements regarding discriminative and generative models of Machine Learning. Which statement is false?

1. Generative models are able to create new data samples from classes while Discriminative models cannot.
2. Discriminative models estimate $p(X|Y)$ in order to predict which class label a test sample should get.
3. Generative models learn the conditional class probability $p(X|Y)$ from the data.
4. Discriminative models estimate $p(Y|X)$ directly from the data in order to predict which class label a test sample should get.

• Answer </aside>

Histogram-based Density Estimation

The easiest way of estimating the density is by using a histogram. The downside to this is that this requires a lot of samples (both in the amount of objects and the amount of repetitions) in order to generate a reliable reading.

Terminology:

• Parameters: the estimates of the true distribution parameters, e.g. mean, standard deviation etc.
• Dimension of data: the amount of features we wish to estimate
• Sample point: points of data in our dataset

Let’s start with a simple example. Say we have 10.000 throws of a dice, with 6 possible outcomes.

If you have a really large sample, and if you repeat the experiment enough times, the relative frequencies of how often you get each outcome will appear. In the above example, as we have a large sample (of 10.000 objects), we get around 0.16 probability for each outcome (the small black vertical line above it is the standard deviation). This looks like a quite reliable estimate of our densities.