Introduction

In the previous lectures, we talked about parametric density estimations:

• Known distribution, e.g. “assume a single Gaussian distribution for each of the classes $\hat{p}(x|y_i)=N(x|\mu_i,\Sigma_i)$”.
• Estimate the global parameters on training set, e.g., “Estimate $\mu_i$ and $\Sigma_i$ for each of the classes”.
• For classification use Bayes rule ($p(x|y_1)p(y_1)>p(x|y_2)p(y_2)$)

An important limitation of this approach is that the chosen density might be a poor model of the distribution that generates the data, which can result in poor predictive performance. For instance, if the process that generates the data is multimodal, then this aspect of the distribution can never be captured by a Gaussian, which is necessarily unimodal.

In this lesson, we consider some non-parametric density estimations that make few assumptions about the form of the distribution:

• no knowledge on the distribution
• manage the smoothness of the distribution (use contributions of local densities)

Popular non-parametric algorithms are Parzen, k-Nearest Neighbours, Naïve Bayes.

Assume we have a 1D dataset, with just one feature, of 40 samples:

How could we estimate the probability density? One of the simplest ways is using histograms, which allow you to locally estimate the densities for a single point in your feature space.

Remember that a histogram is a graph in which values of observations are plotted on the horizontal axis, and the frequency with which each value occurs in the data set is plotted on the vertical axis.

So, what we do is:

• Split the feature in sub-regions (called bins or buckets) of width $h$.
• Count the number of objects in each bin: $k_x$ and fill the buckets.
• Probability density estimate at point $x$:
  • fraction of samples in a bin (containing sample x) normalized by the size of the bin
  • $\hat{p}(x)=\hat{p}(\hat x)\approx\frac{1}{h}\frac{k_x}{N}$, where $N$ is the total amount of data points in the dataset, and $|x-\hat x|\le h/2$.