4 ms·

the question was about "objective that works well in practice for any task". thats pretty global in my books

karpierz 4y ago
If you take "any task" to mean literally any conceivable task, then sure. If you take "any task" to mean "any practical task" or "any task a human would conceivably want to have done", then no free lunch doesn't apply.
tbalsam 4y ago
Another way of looking at karpierz's comment is through the incompressibility of pure noise at scale. As soon as some infinitely generated sort of noise is from some subset of possible noise, there is indeed (AFAIK) some kind of an ideal estimator that appropriately compresses that noise source with no bias and less entropy that the full space of possible noise. I hope this shines an additional alternative light on the topic.
- mitthrowaway2 4y ago
  I deeply suspect that this has more than surface similarities with the observation that you cannot extract work from a uniform heat bath, but you can extract work from a temperature gradient. There's probably not much an intelligence could learn to predict about its future environment living inside the core of the sun, even if it could withstand those temperatures -- just randomness everywhere in every direction. But on Earth, with the sun shining in and the coolness of space behind us, there are a lot of causal relationships that can be compressed to make useful predictions.
  - tbalsam 4y ago
    The flow of entropy is a marvelous, strange, bizarre, and beautiful thing.
nl 4y ago
The "in practice" thing is pretty important. In the no-free-lunch version of the world memory is infinite, all mathematical operations are the same performance wise and things the size of your floating point numbers are ignored. But "in practice" these things matter a lot. For practical purposes the no-free-lunch theorem is a distraction, or at best something to encourage people to try alternate approaches.
- tbalsam 4y ago
  A bumpy landscape indeedy.

If you take "any task" to mean literally any conceivable task, then sure. If you take "any task" to mean "any practical task" or "any task a human would conceivably want to have done", then no free lunch doesn't apply.

tbalsam 4y ago

Another way of looking at karpierz's comment is through the incompressibility of pure noise at scale. As soon as some infinitely generated sort of noise is from some subset of possible noise, there is indeed (AFAIK) some kind of an ideal estimator that appropriately compresses that noise source with no bias and less entropy that the full space of possible noise. I hope this shines an additional alternative light on the topic.

mitthrowaway2 4y ago
I deeply suspect that this has more than surface similarities with the observation that you cannot extract work from a uniform heat bath, but you can extract work from a temperature gradient. There's probably not much an intelligence could learn to predict about its future environment living inside the core of the sun, even if it could withstand those temperatures -- just randomness everywhere in every direction. But on Earth, with the sun shining in and the coolness of space behind us, there are a lot of causal relationships that can be compressed to make useful predictions.
- tbalsam 4y ago
  The flow of entropy is a marvelous, strange, bizarre, and beautiful thing.

nl 4y ago

The "in practice" thing is pretty important. In the no-free-lunch version of the world memory is infinite, all mathematical operations are the same performance wise and things the size of your floating point numbers are ignored. But "in practice" these things matter a lot. For practical purposes the no-free-lunch theorem is a distraction, or at best something to encourage people to try alternate approaches.

tbalsam 4y ago
A bumpy landscape indeedy.