3 ms·

Benchmaxxing is the default case, and always has been. It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a

by ben_w 1mo ago

Benchmaxxing is the default case, and always has been. It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a problem in machine learning, it's the standard failure mode of all minds capable of learning, human, animal, artificial. Even pure genetics has this problem. Viruses and cancers also demonstrate this behaviour, with the bench being evolution's only option: reproductive success.

conmod278 1mo ago
Why would I adapt myself to unseen environments unnecessarily?
- TedDoesntTalk 1mo ago
  Propagation
- Muromec 1mo ago
  Because then you could sit under the palm tree and enjoy your free bananas and relax.
- NewJazz 1mo ago
  You can see possible existing environments even if you explicitly blind yourself to them, through reflections off of environments that you do not blind yourself to. Even with the benchmark excluded, the social zeitgeist that has considered the benchmark and included it or ideas from the benchmark either implicitly or explicitly in their code, documentation, et cetera is still part of your training data.
podocarp 1mo ago
You can't justify it being OK just because it's common. Here, this just makes benchmarks into a low signal and useless marketing number once people get numb to all the 99%s. Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts.
- injidup 1mo ago
  Cancer is simply cells breaking free of the cooperative jail. Essentially the grey goo scenario of nano machines. Instead of cooperation they just do their own thing. Cells have a certain optimised DNA mutation rate kept in check by various machinery. Multi cellular life expects each of these little replication machine to co-operate in the grand scheme of running a body. But it's also required in the grand scheme for DNA to mutate a little bit to ensure population variance. So you could say that cancer is the tax paid for having cooperative yet flexible and adaptive nano machinery. So yes the propensity for cancer developed under evolutioniary pressure towards a non zero level. The population could have optimised for zero cancer but it would not have paid for itself in terms of overall population adaptability and survival.
- ben_w 1mo ago
  > Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts The host's survival is not the benchmark of the reproductive unit, which are the cancer's cells short-term reproduction. The distinction is the reason benchmaxxing is in fact not good: the benchmark is only approximately related to what people actually care about. You do want your cells to reproduce sucessfully, after all; you just also want some emergency stop buttons for when they go wrong, and those things failing is your body's benchmark rather than your cell's benchmark. (There's at least two examples of cancers that can be spread from host to host; lupine genital and taxmanian devil nasal, IIRC)
muddi900 1mo ago
Cancers being distinct living beings by itself is controversial, but you unnecessarily anthropomorphizing. The truth is much more mundane. It's just Goodhart's Law.
- ben_w 1mo ago
  If people already knew about Goodhart's Law, I wouldn't feel it necessay to give illustrated examples.
  - muddi900 1mo ago
    Most of the crowd here is pretty familiar. Some might say their job is optimizing ways to make numbers appear better without substantial change in input or output.

by ben_w 1mo ago

conmod278 1mo ago

Why would I adapt myself to unseen environments unnecessarily?

TedDoesntTalk 1mo ago
Propagation
Muromec 1mo ago
Because then you could sit under the palm tree and enjoy your free bananas and relax.
NewJazz 1mo ago
You can see possible existing environments even if you explicitly blind yourself to them, through reflections off of environments that you do not blind yourself to. Even with the benchmark excluded, the social zeitgeist that has considered the benchmark and included it or ideas from the benchmark either implicitly or explicitly in their code, documentation, et cetera is still part of your training data.

podocarp 1mo ago

You can't justify it being OK just because it's common. Here, this just makes benchmarks into a low signal and useless marketing number once people get numb to all the 99%s. Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts.

injidup 1mo ago
Cancer is simply cells breaking free of the cooperative jail. Essentially the grey goo scenario of nano machines. Instead of cooperation they just do their own thing. Cells have a certain optimised DNA mutation rate kept in check by various machinery. Multi cellular life expects each of these little replication machine to co-operate in the grand scheme of running a body. But it's also required in the grand scheme for DNA to mutate a little bit to ensure population variance. So you could say that cancer is the tax paid for having cooperative yet flexible and adaptive nano machinery. So yes the propensity for cancer developed under evolutioniary pressure towards a non zero level. The population could have optimised for zero cancer but it would not have paid for itself in terms of overall population adaptability and survival.
ben_w 1mo ago
> Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts The host's survival is not the benchmark of the reproductive unit, which are the cancer's cells short-term reproduction. The distinction is the reason benchmaxxing is in fact not good: the benchmark is only approximately related to what people actually care about. You do want your cells to reproduce sucessfully, after all; you just also want some emergency stop buttons for when they go wrong, and those things failing is your body's benchmark rather than your cell's benchmark. (There's at least two examples of cancers that can be spread from host to host; lupine genital and taxmanian devil nasal, IIRC)

muddi900 1mo ago

Cancers being distinct living beings by itself is controversial, but you unnecessarily anthropomorphizing. The truth is much more mundane. It's just Goodhart's Law.

ben_w 1mo ago
If people already knew about Goodhart's Law, I wouldn't feel it necessay to give illustrated examples.
- muddi900 1mo ago
  Most of the crowd here is pretty familiar. Some might say their job is optimizing ways to make numbers appear better without substantial change in input or output.