We can interpret the measurements as a vector in a vector space:
$$ x=\begin{bmatrix} x_1,x_2,\ \ldots,x_p\end{bmatrix}^T. $$
This originates in principle from a probability density over the whole feature space $p(x,y)$.
Above you can see a visual representation of a dataset. All possible objects we can consider are contained in the feature space. At every location of this feature space we will get a probability density of how likely is to get some type of numerical range or label, etc., at this point. Looking at the graph we can get a simple idea of which are the features of each class: for example, Iris Setosa normally has a low sepal length and a high sepal width.
As you can see, the other 2 classes seem to be in the same area. Probability densities are needed to be able to somehow determine if a new data point is from one class or the other.
Given labelled data $x$, we assign to each object a class label $y$. In effect, it splits the feature space in separate regions:
Our objective is to find a function $f$, which should the predicted output:
This function $f$, also denoted as $h$, is called hypothesis, which is a “good” predictor for the corresponding value of $y$.
For each object in the feature space, we should find $p(y|x)$. However, normally we need to estimate this for a sampled data set (denoted as $\hat{p}(y|x)$), with just a few points, so we don’t get to know the perfect distribution; we need to guess what will happen to the points not in our dataset.
Note that in every point all classes has some posterior probability. As a result, if we are able to find these posterior probabilities for all classes, then you should have several functions in all your feature space:
Note that in the above picture instead of using $\omega$ to indicate the class, $y$ is used.
Again, remember these functions $f(x)$ will be approximated, as we depend on the given data.
First, let’s remember how the Pattern Recognition Pipeline looks like: