Could someone provide differences between these two techniques? Also, I would very much appreciate if you could give an example of a ML method that falls into discriminative model or gnerative one. For instance, is Perceptron discriminative? K-Means?
generative vs discriminative machine learning
2 Answers
Let $X$ be your observed data and let $Y$ be their unobserved/hidden properties. In a ML setting, $Y$ usually holds the categories of $X$.
A generative model models their joint distribution, $P(X,Y)$.
A discriminative model models the posterior probability of the categories, $P(Y|X)$.
Depending on what you want to do, you choose between generative versus discriminative modeling. For example, if you are interested in doing classification, a discriminative model would be your choice because you are interested in $P(Y|X)$. However, you can use a generative model, $P(X,Y)$, for classification, too.
A generative model allows you to generate synthetic data ($X$) using the joint but you cannot do this with a discriminative model.
Consider a classification problem at the simplest approach one can get to its goal by designing a discriminant function. for example: a perceptron $y=sign(\theta^Tx)$ you can learn the parameters by a algorithm. in this case you should use Perceptron Learning Rule.
the above approach is not probabilistic and is called 'discriminant function' approach.
a better goal is trying to predict the labels probabilistically. the best goal is to have P(y|X) that is called the posterior.
Example of a discriminative approach: logistic regression
$P(y=1|X)=g(\theta^Tx)$,where g is the sigmoid function.
Example of a generative model:
$P(y|X)=P(X|y)P(y)$
you can model $P(X|y=1)=N(\mu_1,\Sigma_1)$ and $P(X|y=0)=N(\mu_0,\Sigma_0)$ and estimates its parameters from data. (the simple approach is Maximum likelihood).