4 ms·

The most common statement I make is about the misapplication of the NFL. This is not an appropriate use of the NFL. The original commenter is asking about a g

The most common statement I make is about the misapplication of the NFL. This is not an appropriate use of the NFL. The original commenter is asking about a general solution that will work well for all problems presented at it. The NFL details fine-grained tradeoffs in _ideal solutions_ for specific traits in certain areas. Additionally, we're not operating in an unbiased space here, so trivially there is a best estimator without bias. So by that very fact alone the NFL is invalidated in terms of the method it is being applied in here. This is something I feel frustrated seeing a lot of younger people entering the field do (not saying you are young or new, it's just the trend). If this was the case in this kind of a way we never would have gotten beyond MLPs. Yes, indeed there is in fact a set of general solutions that works roughly well over everything and is biased towards the situations where people will need it the most. No, it will likely not be the technically best performing solution. What the OP is looking for I believe is convenience, stability, and reliably good performance. Hence, the NFL is not applicable or relevant to this matter in the way it is being used.

medo-bear 3y ago
> The original commenter is asking about a general solution that will work well for all problems presented at it solution of what? nfl is actually pretty simple. it says that if you have a black box optimization technique then it will perform no better than a random function. a black box function is one that assumes no domain knowledge on the problem it is being applied to. since the op used the term 'any task' im pretty sure it applies to his statement otherwise, please give me an example of an algorithm that goes against what im saying
- tbalsam 3y ago
  The primary poster with a high degree of likelihood means "an reasonable human task". Unless in the future we need models that make summarized briefs from hyperintelligent dogs that communicate only in a language embedded in the Fourier Domain with an XOR-like mapping and zero redundancy as well as summarize a news article _with exactly equal probability to each other in the same model_, then the NFL theorem does not generally apply. Because there's bias. If it helps, put another way, the previous poster isn't being literal. They're using vernacular meaning most expected outcomes, i e. Generally whatever minimizes the empirical risk via a log likelihood criterion. The expected nonliterality of English at all of the wrong times can be quite bizzare.
  - medo-bear 3y ago
    > Because there's bias Not as much as you suggest. A lot of people are too dismissive of NFL because they read somewhere that some people use it incorrectly. If I am wrong please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers
    - tbalsam 3y ago
      I'm a practicioner with a number of years of experience in the field who has contributed meaningful research to it. I've also spent thousands of hours working with models like this. It's about basic mathematical principles, not something someone read somewhere. That's why a number of people have attempted to explain this in a few different ways here. > please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion. Measuring it will take quite a long while depending upon how low the snr is, but yes, that's how it works. I do not mean to be condescending but I would really encourage you back to Shannon and some of the statistics underlying this principle. I feel you have something close to the principle but don't feel I can provide the "aha" moment that makes it click. When a number of experienced people in the field are trying to demonstrate a concept to you and explain why a difference in opinion is merited, they could all be wrong (after all -- humanity + statistics), but it does bear taking a solid look at the fundamentals otherwise. Because if they are all wrong, there are some implications in that as well. Hopefully that helps, I'll probably have to move on from this conversation at this point. Best of luck and care. :thumbsup:
      - medo-bear 3y ago
        > Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion I understand that LLMs seem impressive but they still fail on human behaviour that is sufficiently outside of training sample And to be sure, I agree that we can model human behaviour in various succesful ways. But I think it is wrong to dismiss NFL. Instead the real challange is defining and structuring the problem domain. If you do that successfuly then all talk about NFL becomes irrelevant

medo-bear 3y ago

> The original commenter is asking about a general solution that will work well for all problems presented at it solution of what? nfl is actually pretty simple. it says that if you have a black box optimization technique then it will perform no better than a random function. a black box function is one that assumes no domain knowledge on the problem it is being applied to. since the op used the term 'any task' im pretty sure it applies to his statement otherwise, please give me an example of an algorithm that goes against what im saying

tbalsam 3y ago
The primary poster with a high degree of likelihood means "an reasonable human task". Unless in the future we need models that make summarized briefs from hyperintelligent dogs that communicate only in a language embedded in the Fourier Domain with an XOR-like mapping and zero redundancy as well as summarize a news article _with exactly equal probability to each other in the same model_, then the NFL theorem does not generally apply. Because there's bias. If it helps, put another way, the previous poster isn't being literal. They're using vernacular meaning most expected outcomes, i e. Generally whatever minimizes the empirical risk via a log likelihood criterion. The expected nonliterality of English at all of the wrong times can be quite bizzare.
- medo-bear 3y ago
  > Because there's bias Not as much as you suggest. A lot of people are too dismissive of NFL because they read somewhere that some people use it incorrectly. If I am wrong please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers
  - tbalsam 3y ago
    I'm a practicioner with a number of years of experience in the field who has contributed meaningful research to it. I've also spent thousands of hours working with models like this. It's about basic mathematical principles, not something someone read somewhere. That's why a number of people have attempted to explain this in a few different ways here. > please provide an example of an algorithm that has success rate greater than 50% on all practical human tasks that have true/false answers Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion. Measuring it will take quite a long while depending upon how low the snr is, but yes, that's how it works. I do not mean to be condescending but I would really encourage you back to Shannon and some of the statistics underlying this principle. I feel you have something close to the principle but don't feel I can provide the "aha" moment that makes it click. When a number of experienced people in the field are trying to demonstrate a concept to you and explain why a difference in opinion is merited, they could all be wrong (after all -- humanity + statistics), but it does bear taking a solid look at the fundamentals otherwise. Because if they are all wrong, there are some implications in that as well. Hopefully that helps, I'll probably have to move on from this conversation at this point. Best of luck and care. :thumbsup:
    - medo-bear 3y ago
      > Yes, any language model with a true/false output trained for one step with a successful decrease in the loss on the full spectra of text satisfies this criterion I understand that LLMs seem impressive but they still fail on human behaviour that is sufficiently outside of training sample And to be sure, I agree that we can model human behaviour in various succesful ways. But I think it is wrong to dismiss NFL. Instead the real challange is defining and structuring the problem domain. If you do that successfuly then all talk about NFL becomes irrelevant