4 ms·

> Consensus is the mechanism used to translate scientific findings into actionable advice/recommendations/engineering. Without it nothing would ever happen. Sci

> Consensus is the mechanism used to translate scientific findings into actionable advice/recommendations/engineering. Without it nothing would ever happen. Science is always full of unknowns, consensus allows us to move forward in absence of complete information (because such a thing often doesn't exist) FWIW, my experiences have been the precise opposite. My colleagues never seem to have consensus about anything. But companies move forward with our recommendations, both because we're more likely to be right, and if our recommendations happen to serve the company's goals. I'm sure a bunch of people will read into this comment, but I am speaking purely about my experiences as an ML scientist. It would be valid to say that ML represents a very small fraction of the global scientific apparatus, and I certainly don't claim to know what it's like to work at CERN. It surprised me, though, how often you were right -- there often is consensus, in the sense that everyone reads about a certain technique in papers and automatically assumes it's applicable to their own situation. For example, everyone in language modeling almost universally agrees that "lower loss is better," ("loss" is a specific ML term,"). But that's only true for general language models trained on large quantities of data. If you're fine-tuning a language model, you almost never want the steepest loss slope, because it means your learning rate is too high and you'll destroy the existing knowledge. I would say that experience matters, and each situation is different. It's super tricky to make an argument that consensus is both universally common and universally good.

jpgvm 5y ago
ML has gone through a significant enough transformation in the last few years that it's velocity essentially makes it very difficult to form consensus about anything right now as everything feels up for replacement in short order.
- sillysaurusx 5y ago
  That's probably true, which is why I love it so much. :) You're right that it might not be the best example. Still... in particle physics, for example, I've heard that there was consensus in favor of certain particles existing, even to the extent of "multiple experiments have achieved an N-sigma result showing that this particle exists," for some fairly high N. And then a later experiment showed that the particle simply didn't exist, and that the previous results were experimental error. (Specifically, experimental bias – they were expecting to see certain results, and so they were probably subconsciously combing the data for such results.) My knowledge on that subject comes from Veritasium, not from reading physics papers or following along with experimental breakthroughs. But I've seen similar things happen in ML, where people are pretty convinced a certain technique is good (hello, style mixing in StyleGAN) and no one ever bothers to check whether it's true in reality (style mixing is what prevents StyleGAN from being as cohesive as BigGAN). I guess I broadly agree with your point. There is consensus that black holes exist, and that we've even observed one, so it's pretty likely that if you were to travel there, you'd find a black hole. But given how much science itself has changed throughout history, alarm bells start going off when hearing things like "Consensus is extremely common, and it's almost universally a good thing."
  - jpgvm 5y ago
    "Consensus is extremely common, and it's almost universally a good thing." isn't about saying things are assumed when they are not, just that the community is able to quickly adapt to changes in fundamental theory in order to explore the new ground opened up as a result. Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. I think the problem here is conflating consensus with faith, that is assuming truth in absence of data. Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.
    - sillysaurusx 5y ago
      > Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there. In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948 https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897cc61e44479267968716b/training/networks_stylegan.py#L310 https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...). So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138b8b1c08b6cf7ed36a1fc6/training/networks_stylegan2.py#L159 https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138...). I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default. It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch. Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces." However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers... But, this all does tie in to your final point: > Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit. It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different... The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic. But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.) As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkReelTV https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.
addicted 5y ago
Also the vast majority of companies fail. I’m not sure what your point is, unless it’s saying that a lack of consensus is indicative of being wrong. Which would support the commenter you replied to’s argument.

jpgvm 5y ago

ML has gone through a significant enough transformation in the last few years that it's velocity essentially makes it very difficult to form consensus about anything right now as everything feels up for replacement in short order.

sillysaurusx 5y ago
That's probably true, which is why I love it so much. :) You're right that it might not be the best example. Still... in particle physics, for example, I've heard that there was consensus in favor of certain particles existing, even to the extent of "multiple experiments have achieved an N-sigma result showing that this particle exists," for some fairly high N. And then a later experiment showed that the particle simply didn't exist, and that the previous results were experimental error. (Specifically, experimental bias – they were expecting to see certain results, and so they were probably subconsciously combing the data for such results.) My knowledge on that subject comes from Veritasium, not from reading physics papers or following along with experimental breakthroughs. But I've seen similar things happen in ML, where people are pretty convinced a certain technique is good (hello, style mixing in StyleGAN) and no one ever bothers to check whether it's true in reality (style mixing is what prevents StyleGAN from being as cohesive as BigGAN). I guess I broadly agree with your point. There is consensus that black holes exist, and that we've even observed one, so it's pretty likely that if you were to travel there, you'd find a black hole. But given how much science itself has changed throughout history, alarm bells start going off when hearing things like "Consensus is extremely common, and it's almost universally a good thing."
- jpgvm 5y ago
  "Consensus is extremely common, and it's almost universally a good thing." isn't about saying things are assumed when they are not, just that the community is able to quickly adapt to changes in fundamental theory in order to explore the new ground opened up as a result. Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. I think the problem here is conflating consensus with faith, that is assuming truth in absence of data. Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit.
  - sillysaurusx 5y ago
    > Take your example about StyleGAN vs BigGan, I assume once it became clear that the latter was superior to the former that likely resulted in changes to existing architectures that then found additional improvements. This change in consensus is what enables that and is a good thing. Well, no. :) But I think the "no" is because of the uniqueness of ML rather than a "no" to your point in general. You might be right about other fields; I don't have experience there. In ML, there an enormous number of techniques. Style mixing was presented as a core feature of StyleGAN (https://arxiv.org/abs/1812.04948 https://arxiv.org/abs/1812.04948) and was enabled by default in the codebase (https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897cc61e44479267968716b/training/networks_stylegan.py#L310 https://github.com/NVlabs/stylegan/blob/03563d18a0cf8d67d897...). So there's a lot of "inertia" -- for example, when StyleGAN 2 came out, style mixing was still the default (https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138b8b1c08b6cf7ed36a1fc6/training/networks_stylegan2.py#L159 https://github.com/NVlabs/stylegan2/blob/f2f751cdc7f996e3138...). I haven't had time to dig into StyleGAN 3, but I suspect that style mixing might still be enabled by default. It wasn't until we did a detailed, methodical analysis side-by-side with BigGAN, specifically to answer the question "Why is BigGAN so much better for diverse datasets?" that, on a whim, I turned off style mixing and was astonished to see BigGAN type quality pop out of a StyleGAN type arch. Discoveries like that usually go unnoticed, frankly because it's a lot of effort to write a paper specifically to say "Hey, if you're training StyleGAN, definitely turn off style mixing. It only seems to work well on faces." However, if such a paper were to be written, and accepted into a peer-reviewed journal, then your original point would probably be valid. So I don't even know if it's worth writing all of this -- I just thought it'd be interesting to point out the "Well, not really" in this case. The knowledge ends up floating around on Twitter and Discord rather than being transmitted via scientific papers... But, this all does tie in to your final point: > Consensus is easily changed with the introduction of new data, faith hangs on no matter how much evidence is put forward that it's horseshit. It's remarkably easy for old, accepted ideas to hang around. You'd think it'd just be a matter of "Run the experiment; experiment proves thing; thing becomes accepted." But in practice it's felt quite different... The thing is, everything you're saying is true in general. As t approaches infinity, there tends to be more and more consensus about older ideas, like the existence of black holes, or the validity of laws like F=ma. So we should probably pay attention when there is 99% consensus on a particular topic. But, for example, one reason I wouldn't want to publish a paper claiming style mixing was bad, is because it would contradict the results of Karras, who is famous. I'd better be very certain about my claim! So there's sometimes a reluctance to contradict the consensus, too, which ends up equivalent to "faith" in your example -- we have faith that famous scientists are correct. (They usually are.) As a cherry on top, I'll just leave a link to Feynman's messenger lectures: https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkReelTV https://www.youtube.com/watch?v=-kFOXP026eE&ab_channel=TalkR... ... the history of science is fascinating. I'd dreamt for years of becoming a scientist, but the actual experience turned out to be surprisingly different than what I thought it'd be. I love it though -- all these weird corner cases are the spice of life.

addicted 5y ago

Also the vast majority of companies fail. I’m not sure what your point is, unless it’s saying that a lack of consensus is indicative of being wrong. Which would support the commenter you replied to’s argument.