3 ms·

"Consensus is extremely common, and it's almost universally a good thing." isn't about saying things are assumed when they are not, just that the community is a

by jpgvm 5y ago

"Consensus is extremely common, and it's almost universally a good thing." isn't about saying things are assumed when they are not, just that the community is able to quickly adapt to changes in fundamental theory in order to explore the new ground opened up as a result. Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. I think the problem here is conflating consensus with faith, that is assuming truth in absence of data. Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.

sillysaurusx 5y ago
> Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there. In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948 https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897cc61e44479267968716b/training/networks_stylegan.py#L310 https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...). So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138b8b1c08b6cf7ed36a1fc6/training/networks_stylegan2.py#L159 https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138...). I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default. It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch. Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces." However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers... But, this all does tie in to your final point: > Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit. It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different... The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic. But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.) As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkReelTV https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.
- jpgvm 5y ago
  Yeah I think everything you wrote here is on-point, nothing here happens instantaneously and with the current velocity of ML it's even less clear that it's taking place. Thanks for your really in depth and informative reply though, I'm only cursorily interested in ML through its relation to data, which is my speciality. My exposure to ML is mostly through said ML teams making requests for data and systems like feature stores to run their models so I have some understanding of how they construct their overall architecture but not the specifics so this was super interesting.
- glenstein 5y ago
  >But I think the "no" is because of the uniqueness of ML In a thread where people are parsing the meaning of consensus with respect to climate change, citing the atypical example of the state of consensus in a unique field is going to contribute to the loss of context that makes people confused about the original subject at issue in this thread. It's great to know about BigGAN vs StyleGAN, but I'm almost positive people will skim this detour into increasingly detailed elaboration on machine learning and think the upshot is supposed to be "and therefore we shouldn't put stock in climate science consensus."

by jpgvm 5y ago

sillysaurusx 5y ago

> Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there. In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948 https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897cc61e44479267968716b/training/networks_stylegan.py#L310 https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...). So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138b8b1c08b6cf7ed36a1fc6/training/networks_stylegan2.py#L159 https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138...). I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default. It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch. Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces." However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers... But, this all does tie in to your final point: > Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit. It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different... The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic. But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.) As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkReelTV https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.

jpgvm 5y ago
Yeah I think everything you wrote here is on-point, nothing here happens instantaneously and with the current velocity of ML it's even less clear that it's taking place. Thanks for your really in depth and informative reply though, I'm only cursorily interested in ML through its relation to data, which is my speciality. My exposure to ML is mostly through said ML teams making requests for data and systems like feature stores to run their models so I have some understanding of how they construct their overall architecture but not the specifics so this was super interesting.
glenstein 5y ago
>But I think the "no" is because of the uniqueness of ML In a thread where people are parsing the meaning of consensus with respect to climate change, citing the atypical example of the state of consensus in a unique field is going to contribute to the loss of context that makes people confused about the original subject at issue in this thread. It's great to know about BigGAN vs StyleGAN, but I'm almost positive people will skim this detour into increasingly detailed elaboration on machine learning and think the upshot is supposed to be "and therefore we shouldn't put stock in climate science consensus."