6 ms·

Maximum likelihood estimation and loss functions

119 points by snprajwal 2y ago

skzv 2y ago
To bring things full circle: the cross-entropy loss is the KL divergence. So intuitively, when you're minimizing cross-entropy loss, you're trying to minimize the "divergence" between the true distribution and your model distribution. This intuition really helped me understand CE loss.
- sidr 2y ago
  Cross-entropy is not the KL divergence. There is an additional term in cross-entropy which is the entropy of the data distribution (i.e., independent of the model). So, you're right in that minimizing one is equivalent to minimizing the other. https://stats.stackexchange.com/questions/357963/what-is-the-difference-between-cross-entropy-and-kl-divergence https://stats.stackexchange.com/questions/357963/what-is-the...
  - skzv 2y ago
    Yes, you are totally correct, but I believe this term is omitted from the cross-entropy loss function that is used in machine learning? Because it is a constant which does not contribute to the optimization. Please correct me if I'm wrong.
Onavo 2y ago
The next step is ELBO — the evidence lower bound.
abdljasser2 2y ago
Excellent post
sidravi1 2y ago
This reminds me that David Mackay’s book and his lectures are so excellent on these topics.
levocardia 2y ago
Somehow I've never found an explanation for MLE that's intuitive and suitable for someone who didn't take graduate-level statistics already. I'm 100% on board with the introduction (MSE and cross-entropy make total intuitive sense; you can see how they penalize 'wrongness' to an increasing degree) but in the next paragraph we jump right to: >Let pmodel(x;θ) be a parametric family of distributions over a space of parameters ... and it's straight to the grad-level textbook stuff that breezily assumes familiarity with advanced mathematical notation. One of the reasons I loved Andrew Ng's machine learning course so much is that it eased you into understanding the notation, terminology, and signposted things like "hey this is really important" vs. "hey this is just a weird notational quirk that mathematicians have, don't worry about it too much."
- jey 2y ago
  This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. It's totally practical to learn this stuff on your own from videos, books, and PDFs. First you need to get a solid conceptual grasp on probability distributions and their associated concepts like random variables, conditional probability, and joint probability. Then you'll be ready to learn some mathematical statistics and follow along with all the notation. Note that you don't need to go deep into measure-theoretic probability or any of that stuff that requires more advanced prior education in math.
  - PittleyDunkin 2y ago
    > This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. My courses definitely used different notation for the same semantics.
  - user_7832 2y ago
    > This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. Perhaps a first course at grad level, but my engineering bachelors covered MLEs but we didn’t learn/use any of those formal language things. I think the core mathematics (and likely other pure science) cohorts were the only people who learnt it.
deleted 2y ago
[deleted]
patrick451 2y ago
If you have a parametrized functions that imperfectly models a real phenomenon, of course there are errors. Why assume they are random? A better assumption is that your model is just poor. Assuming deterministic modeling errors are due to randomness has always struck me as bizarre.
- exhibit001 2y ago
  In the context of MLE, random has a formal definition. What you describe as poor would be included in the mathematics as a factor outside the deterministic parameters that are modeled. E.g. Y = aFactor1 + bFactor2 + ... + constant + 'poor model correction factor'. To solve the equation, we have to make assumptions of the poor correction factor. These assumptions about the error generally have some 'mathematically nice' qualities. For example it's not predictable or has a trend relating to any other factors. An concrete example is having a mean of zero. If it had a non-zero mean, it should be accounted in the constant factor of the model. All these mathematically nice assumptions can be summed up be calling the 'poor model correction' factor as random.
  - patrick451 2y ago
    This doesn't make sense to me. Any time you want a reduced order model, you will get error. For example: - Say I have data which is perfectly sinusoidal, with an dc bias. I can fit a line a to this data, which will approximate the bias (or be exactly the bias if the data is over an integer number of cycles). - I want to fit a plane to a curved surface - I want to fit a low order transfer function to a high order system. - I want to model a system with friction as a system with no friction. Fitting parameters in all of these situations will result in a non-zero residual. But assuming that is due to randomness is not useful.
anon946 2y ago
My favorite MLE example: Suppose you walk into a bank and ask them to give you a quarter. You flip the quarter twice and get two heads. Given this experiment, what do you estimate to be the probability p of getting a heads when you flip this coin? Using MLE, you would get p = 1. In other words, this coin will always give you a heads when you flip it! (According to MLE.)
- truculent 2y ago
  The example only seems ridiculous because you've deliberately excluded relevant knowledge about the world from the model. Add a prior to the model and you'll have a much more reasonable function to maximise.
  - anon946 2y ago
    Yes, but then it is no longer MLE, but rather MAP: [https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf](https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf) https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lect....
- radarsat1 2y ago
  Are you just demonstrating overfitting when estimating using too little data? Or is there something deeper going on in your example? What does the bank have to do with anything?
  - anon946 2y ago
    The bank is context that gives us a prior probability. However, MLE does not consider a prior. So MLE can give results that are not very helpful in the real world. All it does is answer: What parameter value (in case the probability) of a head, makes the observed outcome most likely? But it considers all parameter values equally likely. In reality, we know that it is highly likely that a random coin from a bank is a fair coin. Thus, if we flip two heads, we are almost certain that it's still a fair coin. If, on the other hand, we flipped 10 heads in a row, we might start to wonder if somehow the bank gave you a trick coin. MAP is an alternative to MLE, arguably better in many situations: [https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf](https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf) https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lect....

119 points by snprajwal 2y ago

skzv 2y ago

To bring things full circle: the cross-entropy loss is the KL divergence. So intuitively, when you're minimizing cross-entropy loss, you're trying to minimize the "divergence" between the true distribution and your model distribution. This intuition really helped me understand CE loss.

sidr 2y ago
Cross-entropy is not the KL divergence. There is an additional term in cross-entropy which is the entropy of the data distribution (i.e., independent of the model). So, you're right in that minimizing one is equivalent to minimizing the other. https://stats.stackexchange.com/questions/357963/what-is-the-difference-between-cross-entropy-and-kl-divergence https://stats.stackexchange.com/questions/357963/what-is-the...
- skzv 2y ago
  Yes, you are totally correct, but I believe this term is omitted from the cross-entropy loss function that is used in machine learning? Because it is a constant which does not contribute to the optimization. Please correct me if I'm wrong.

Onavo 2y ago

The next step is ELBO — the evidence lower bound.

abdljasser2 2y ago

Excellent post

sidravi1 2y ago

This reminds me that David Mackay’s book and his lectures are so excellent on these topics.

levocardia 2y ago

Somehow I've never found an explanation for MLE that's intuitive and suitable for someone who didn't take graduate-level statistics already. I'm 100% on board with the introduction (MSE and cross-entropy make total intuitive sense; you can see how they penalize 'wrongness' to an increasing degree) but in the next paragraph we jump right to: >Let pmodel(x;θ) be a parametric family of distributions over a space of parameters ... and it's straight to the grad-level textbook stuff that breezily assumes familiarity with advanced mathematical notation. One of the reasons I loved Andrew Ng's machine learning course so much is that it eased you into understanding the notation, terminology, and signposted things like "hey this is really important" vs. "hey this is just a weird notational quirk that mathematicians have, don't worry about it too much."

jey 2y ago
This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. It's totally practical to learn this stuff on your own from videos, books, and PDFs. First you need to get a solid conceptual grasp on probability distributions and their associated concepts like random variables, conditional probability, and joint probability. Then you'll be ready to learn some mathematical statistics and follow along with all the notation. Note that you don't need to go deep into measure-theoretic probability or any of that stuff that requires more advanced prior education in math.
- PittleyDunkin 2y ago
  > This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. My courses definitely used different notation for the same semantics.
- user_7832 2y ago
  > This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. Perhaps a first course at grad level, but my engineering bachelors covered MLEs but we didn’t learn/use any of those formal language things. I think the core mathematics (and likely other pure science) cohorts were the only people who learnt it.

deleted 2y ago

[deleted]

patrick451 2y ago

If you have a parametrized functions that imperfectly models a real phenomenon, of course there are errors. Why assume they are random? A better assumption is that your model is just poor. Assuming deterministic modeling errors are due to randomness has always struck me as bizarre.

exhibit001 2y ago
In the context of MLE, random has a formal definition. What you describe as poor would be included in the mathematics as a factor outside the deterministic parameters that are modeled. E.g. Y = aFactor1 + bFactor2 + ... + constant + 'poor model correction factor'. To solve the equation, we have to make assumptions of the poor correction factor. These assumptions about the error generally have some 'mathematically nice' qualities. For example it's not predictable or has a trend relating to any other factors. An concrete example is having a mean of zero. If it had a non-zero mean, it should be accounted in the constant factor of the model. All these mathematically nice assumptions can be summed up be calling the 'poor model correction' factor as random.
- patrick451 2y ago
  This doesn't make sense to me. Any time you want a reduced order model, you will get error. For example: - Say I have data which is perfectly sinusoidal, with an dc bias. I can fit a line a to this data, which will approximate the bias (or be exactly the bias if the data is over an integer number of cycles). - I want to fit a plane to a curved surface - I want to fit a low order transfer function to a high order system. - I want to model a system with friction as a system with no friction. Fitting parameters in all of these situations will result in a non-zero residual. But assuming that is due to randomness is not useful.

anon946 2y ago

My favorite MLE example: Suppose you walk into a bank and ask them to give you a quarter. You flip the quarter twice and get two heads. Given this experiment, what do you estimate to be the probability p of getting a heads when you flip this coin? Using MLE, you would get p = 1. In other words, this coin will always give you a heads when you flip it! (According to MLE.)

truculent 2y ago
The example only seems ridiculous because you've deliberately excluded relevant knowledge about the world from the model. Add a prior to the model and you'll have a much more reasonable function to maximise.
- anon946 2y ago
  Yes, but then it is no longer MLE, but rather MAP: [https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf](https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf) https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lect....
radarsat1 2y ago
Are you just demonstrating overfitting when estimating using too little data? Or is there something deeper going on in your example? What does the bank have to do with anything?
- anon946 2y ago
  The bank is context that gives us a prior probability. However, MLE does not consider a prior. So MLE can give results that are not very helpful in the real world. All it does is answer: What parameter value (in case the probability) of a head, makes the observed outcome most likely? But it considers all parameter values equally likely. In reality, we know that it is highly likely that a random coin from a bank is a fair coin. Thus, if we flip two heads, we are almost certain that it's still a fair coin. If, on the other hand, we flipped 10 heads in a row, we might start to wonder if somehow the bank gave you a trick coin. MAP is an alternative to MLE, arguably better in many situations: [https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf](https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lecture4_inked.pdf) https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lect....