4 ms·

> Because there's bias Not as much as you suggest. A lot of people are too dismissive of NFL because they read somewhere that some people use it incorrectly. I

> Because there's bias Not as much as you suggest. A lot of people are too dismissive of NFL because they read somewhere that some people use it incorrectly. If I am wrong please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers

tbalsam 4y ago
I'm a practicioner with a number of years of experience in the field who has contributed meaningful research to it. I've also spent thousands of hours working with models like this. It's about basic mathematical principles, not something someone read somewhere. That's why a number of people have attempted to explain this in a few different ways here. > please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion. Measuring it will take quite a long while depending upon how low the snr is, but yes, that's how it works. I do not mean to be condescending but I would really encourage you back to Shannon and some of the statistics underlying this principle. I feel you have something close to the principle but don't feel I can provide the "aha" moment that makes it click. When a number of experienced people in the field are trying to demonstrate a concept to you and explain why a difference in opinion is merited, they could all be wrong (after all -- humanity + statistics), but it does bear taking a solid look at the fundamentals otherwise. Because if they are all wrong, there are some implications in that as well. Hopefully that helps, I'll probably have to move on from this conversation at this point. Best of luck and care. :thumbsup:
- medo-bear 4y ago
  > Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion I understand that LLMs seem impressive but they still fail on human behaviour that is sufficiently outside of training sample And to be sure, I agree that we can model human behaviour in various succesful ways. But I think it is wrong to dismiss NFL. Instead the real challange is defining and structuring the problem domain. If you do that successfuly then all talk about NFL becomes irrelevant
  - tbalsam 4y ago
    Again, you've got to look at the mathematical foundations here. Shannon and frequentist statistics will help in particular. I'd encourage you to go back to the mathematical roots and take a good gander over it. I find it to be a very rewarding journey in and of itself.
    - medo-bear 4y ago
      I read through few simple explanations that might help you. This seems like a good candidate https://towardsdatascience.com/what-no-free-lunch-really-means-in-machine-learning-85493215625d https://towardsdatascience.com/what-no-free-lunch-really-mea...
    - medo-bear 4y ago
      I work in the field

tbalsam 4y ago

I'm a practicioner with a number of years of experience in the field who has contributed meaningful research to it. I've also spent thousands of hours working with models like this. It's about basic mathematical principles, not something someone read somewhere. That's why a number of people have attempted to explain this in a few different ways here. > please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion. Measuring it will take quite a long while depending upon how low the snr is, but yes, that's how it works. I do not mean to be condescending but I would really encourage you back to Shannon and some of the statistics underlying this principle. I feel you have something close to the principle but don't feel I can provide the "aha" moment that makes it click. When a number of experienced people in the field are trying to demonstrate a concept to you and explain why a difference in opinion is merited, they could all be wrong (after all -- humanity + statistics), but it does bear taking a solid look at the fundamentals otherwise. Because if they are all wrong, there are some implications in that as well. Hopefully that helps, I'll probably have to move on from this conversation at this point. Best of luck and care. :thumbsup:

medo-bear 4y ago
> Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion I understand that LLMs seem impressive but they still fail on human behaviour that is sufficiently outside of training sample And to be sure, I agree that we can model human behaviour in various succesful ways. But I think it is wrong to dismiss NFL. Instead the real challange is defining and structuring the problem domain. If you do that successfuly then all talk about NFL becomes irrelevant
- tbalsam 4y ago
  Again, you've got to look at the mathematical foundations here. Shannon and frequentist statistics will help in particular. I'd encourage you to go back to the mathematical roots and take a good gander over it. I find it to be a very rewarding journey in and of itself.
  - medo-bear 4y ago
    I read through few simple explanations that might help you. This seems like a good candidate https://towardsdatascience.com/what-no-free-lunch-really-means-in-machine-learning-85493215625d https://towardsdatascience.com/what-no-free-lunch-really-mea...
  - medo-bear 4y ago
    I work in the field