16 ms·

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. Many "state of the art models" are simply a bunch of common strategies glued together in a way researchers found worked the best (by trying a bunch of different ones). An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models. Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. It's still the wild west

The_rationalist 6y ago
While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic. While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...
- throwaway189262 6y ago
  Interesting, but in not sure you're completely right about XLNET. I heard it takes an absurd amount of resources to train. Even more than the BERT variations. And this is likely why there's not a ton of interest in it
  - The_rationalist 6y ago
    https://github.com/renatoviolin/xlnet https://github.com/renatoviolin/xlnet XLnet running on very low end hardware (a single 8GB 2080 non ti) significantly outperform BERT large on e.g the reference question answering benchmarck: SQUAD 2 86% vs 81% Nobody has even tried to create a spanXLnet (akin to spanBERT) How many years will be wasted before researchers get out of the BERT local minima? I'm a afraid it might last a decade
- Der_Einzige 6y ago
  Not sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it. Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.
  - The_rationalist 6y ago
    Honestly my comments that have negative Karma have generally much more useful truths than my comments that have positive Karma, this is almost systemic. This show how low quality the HN community is, epistemologically speaking. There are much less lazy communities out there like lesswrong.com but unfortunately they don't talk much about computer science
sillysaurusx 6y ago
Apply to TFRC! https://www.tensorflow.org/tfrc https://www.tensorflow.org/tfrc They are very permissive. And you get to play with $500k worth of hardware. Been a member for over a year now. Jonathan is singlehandedly the best support person I've ever worked with, or perhaps ever will work with. I would've completely agreed with you if not for TFRC. And I couldn't resist the opportunity of playing with some big metal, even if it's hard to work with.
- panabee 6y ago
  just applied. thanks for sharing!
nickhuh 6y ago
There's a lot of interest in various ML communities on more efficient training and inference. Both vision and NLP have had a growing focus on these problems in recent years. I think you make a good observation that much of ML progress is driven by tinkering with existing models, though instead of describing it as more "alchemy than science" it's probably more accurate to say it's very experimental right now. Being very experimental is neither unscientific nor unusual in the development of knowledge. James Watt worked as an instrument maker (not a theoretician) when he invented the Watt steam engine in 1776 [1], and at the time the idea of heat as Phlogiston [2] was still more prevalent than anything that looks like modern thermodynamics. Theory and practice naturally take turns outpacing each other, which is part of why we need both. I'd also caution against the belief that experimental work doesn't require "particularly demanding thought". There are many things one can tweak in current ML models (the search space is exponential) and, as you point out, the experiments are expensive. Having a solid understanding of the system, great intuition, and good heuristics is necessary to reliably make progress. For those who are interested in the theory of deep learning, the community has recently made great strides on developing a mathematical understanding of neural networks. The research is still very cutting edge, but the following PDF helps introduce the topic [3]. [1]: https://en.wikipedia.org/wiki/James_Watt https://en.wikipedia.org/wiki/James_Watt [2]: https://en.wikipedia.org/wiki/Phlogiston_theory https://en.wikipedia.org/wiki/Phlogiston_theory [3]: https://www.cs.princeton.edu/courses/archive/fall19/cos597B/lecnotes/bookdraft.pdf https://www.cs.princeton.edu/courses/archive/fall19/cos597B/...
- ghufran_syed 6y ago
  that course from princeton looks great! This paper is a nice short read and gives some geometric insight: https://arxiv.org/abs/1805.10451 https://arxiv.org/abs/1805.10451
tbalsam 6y ago
This feels much like the sentiment in the field about two years ago or so. While I feel like the "alchemy" storyline is still somewhat in play, most of the big important parts of the deep learning process have enough ideological linear approximators stacked around them that if you know what you're doing or looking at, you can jump to an unexplored trench with some reasonable feeling about whether you'll get something good or not. I feel like the "alchemy" approach is when people new to the field are innundated with information about it, and while I think that still holds, there very much is a well-understood science of principles in most parts of it. There's the neural tangent kernel work that's achieved a lot, and the transformers themselves are really taking off a lot as the blockwise/lower rank approximation algorithms look more and more like circuits built off of basic, more well-established components. "An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models." This feels like it trivializes a lot of the work and collapses some of the major advancements in training at scale down to a more one-dimensional outlook. Companies are doing both, but it's easy to throw money and compute at an absolutely guaranteed logarithmic improvement in results. It's not stupidity, it's just reducing variance in scaling known laws as we work on making things more efficient, which weirdly enough starts the iterative process of academics frantically trying to mine the expensive, inefficient compute tactics to flag plant their own materials. With respect to you comment on protein folding and such, I feel you might have missed a lot of the major work in that aren more recently. There really and truly been some field-shattering work on that in combining deep learning systems with last-mile supervision and refinement systems. I'd posit that we're very much out of the wild west and in the mild, but still rambunctious west, if I were to put terms on it. With reference to guess and check -- yes, that especially was prevalent and worked 2-3 years ago and I'd be in favor of advocating that it does still happen somewhat in a more refined fashion, but I personally believe we'd not get too far beyond the SOTA if we're not working (effectively) with your data manifold now and tightly incorporating whatever projections/constraints of that data distillation process into your network training procedure. I really do agree with you in that I think average Joe breakthroughs will happen and continue to benefit the the field, and I'd certainly agree that there's always going to be the mediocre paper churn of paper mills I think that you alluded to trying to justify their own existence as academics/paper writers, but I really do legitimately think there's enough precedent set in most parts of the field that you need to have some kind of thoughtful improvement to move forward (like AdaBelief, which is still terrible because they straight up lie about what they do in the abstract, even though the improvement of debiasing the variance estimates during training is an exceptionally good idea). Just my 2c, hope this helps. I think we may have a similar end perspective from two different sides, like two explorers looking at the same peak from the different side of the mountain. :thumbsup:
jgalt212 6y ago
> The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. "guess and check" is terribly ineffective with multi-day training runs. Brings us right back to the batch processing paradigm of the 1960s.
sjg007 6y ago
>We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. Have you not heard of alphafold?
- solarengineer 6y ago
  Thanks for mentioning Alphafold. This article by the DeepMind team is very insightful. https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-...
eachro 6y ago
> An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. Right, but you have to remember there are legions of grad students doing exactly this so it ends up being quite competitive to churn out papers this way.
visarga 6y ago
> Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. I get that you'd like to have a clear theoretical basis for what works and we're far from there. But in the meantime we're stumbling in the dark, discovering tricks and forming intuitions, not knowing even where the road is going to lead us. This is an evolutionary process of ideas, similar to biological evolution that managed to make us. If you know where you're going you can optimise your actions but when you don't even know what might be useful later on, then all attempts are good. They increase diversity and discover blind spots. Some of them will be the stepping stones for the future, but we can't say in advance which and how. Link to a long discussion about the evolution of ML ideas and the book "Why greatness cannot be planned" by Kenneth Stanley - https://youtu.be/lhYGXYeMq_E?t=416 https://youtu.be/lhYGXYeMq_E?t=416
woko 6y ago
I see your point when it comes to paper publication in general, but I feel that your post is very unwarranted with respect to the original post: DeepMind has recruited top-quality theoretical researchers from public institutions, it is not just experimental work which your average Joe could do with a few guesses. These researchers published a lot of theoretical papers before they were recruited, and they still publish a lot of them now that they are working at DeepMind, but they have more computational hardware to apply their ideas. Here is just one uncurated example of a publication: - https://deepmind.com/research/publications/Taylor-Expansion-Policy-Optimization https://deepmind.com/research/publications/Taylor-Expansion-... - https://arxiv.org/pdf/2003.06259.pdf https://arxiv.org/pdf/2003.06259.pdf
physicsguy 6y ago
Minor variations on top of existing stuff and occasional leaps forward is most research though, it's not surprising ML research follows a similar pattern.

The_rationalist 6y ago

While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic. While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...

throwaway189262 6y ago
Interesting, but in not sure you're completely right about XLNET. I heard it takes an absurd amount of resources to train. Even more than the BERT variations. And this is likely why there's not a ton of interest in it
- The_rationalist 6y ago
  https://github.com/renatoviolin/xlnet https://github.com/renatoviolin/xlnet XLnet running on very low end hardware (a single 8GB 2080 non ti) significantly outperform BERT large on e.g the reference question answering benchmarck: SQUAD 2 86% vs 81% Nobody has even tried to create a spanXLnet (akin to spanBERT) How many years will be wasted before researchers get out of the BERT local minima? I'm a afraid it might last a decade
Der_Einzige 6y ago
Not sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it. Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.
- The_rationalist 6y ago
  Honestly my comments that have negative Karma have generally much more useful truths than my comments that have positive Karma, this is almost systemic. This show how low quality the HN community is, epistemologically speaking. There are much less lazy communities out there like lesswrong.com but unfortunately they don't talk much about computer science

sillysaurusx 6y ago

Apply to TFRC! https://www.tensorflow.org/tfrc https://www.tensorflow.org/tfrc They are very permissive. And you get to play with $500k worth of hardware. Been a member for over a year now. Jonathan is singlehandedly the best support person I've ever worked with, or perhaps ever will work with. I would've completely agreed with you if not for TFRC. And I couldn't resist the opportunity of playing with some big metal, even if it's hard to work with.

panabee 6y ago
just applied. thanks for sharing!

nickhuh 6y ago

There's a lot of interest in various ML communities on more efficient training and inference. Both vision and NLP have had a growing focus on these problems in recent years. I think you make a good observation that much of ML progress is driven by tinkering with existing models, though instead of describing it as more "alchemy than science" it's probably more accurate to say it's very experimental right now. Being very experimental is neither unscientific nor unusual in the development of knowledge. James Watt worked as an instrument maker (not a theoretician) when he invented the Watt steam engine in 1776 [1], and at the time the idea of heat as Phlogiston [2] was still more prevalent than anything that looks like modern thermodynamics. Theory and practice naturally take turns outpacing each other, which is part of why we need both. I'd also caution against the belief that experimental work doesn't require "particularly demanding thought". There are many things one can tweak in current ML models (the search space is exponential) and, as you point out, the experiments are expensive. Having a solid understanding of the system, great intuition, and good heuristics is necessary to reliably make progress. For those who are interested in the theory of deep learning, the community has recently made great strides on developing a mathematical understanding of neural networks. The research is still very cutting edge, but the following PDF helps introduce the topic [3]. [1]: https://en.wikipedia.org/wiki/James_Watt https://en.wikipedia.org/wiki/James_Watt [2]: https://en.wikipedia.org/wiki/Phlogiston_theory https://en.wikipedia.org/wiki/Phlogiston_theory [3]: https://www.cs.princeton.edu/courses/archive/fall19/cos597B/lecnotes/bookdraft.pdf https://www.cs.princeton.edu/courses/archive/fall19/cos597B/...

ghufran_syed 6y ago
that course from princeton looks great! This paper is a nice short read and gives some geometric insight: https://arxiv.org/abs/1805.10451 https://arxiv.org/abs/1805.10451

tbalsam 6y ago

This feels much like the sentiment in the field about two years ago or so. While I feel like the "alchemy" storyline is still somewhat in play, most of the big important parts of the deep learning process have enough ideological linear approximators stacked around them that if you know what you're doing or looking at, you can jump to an unexplored trench with some reasonable feeling about whether you'll get something good or not. I feel like the "alchemy" approach is when people new to the field are innundated with information about it, and while I think that still holds, there very much is a well-understood science of principles in most parts of it. There's the neural tangent kernel work that's achieved a lot, and the transformers themselves are really taking off a lot as the blockwise/lower rank approximation algorithms look more and more like circuits built off of basic, more well-established components. "An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models." This feels like it trivializes a lot of the work and collapses some of the major advancements in training at scale down to a more one-dimensional outlook. Companies are doing both, but it's easy to throw money and compute at an absolutely guaranteed logarithmic improvement in results. It's not stupidity, it's just reducing variance in scaling known laws as we work on making things more efficient, which weirdly enough starts the iterative process of academics frantically trying to mine the expensive, inefficient compute tactics to flag plant their own materials. With respect to you comment on protein folding and such, I feel you might have missed a lot of the major work in that aren more recently. There really and truly been some field-shattering work on that in combining deep learning systems with last-mile supervision and refinement systems. I'd posit that we're very much out of the wild west and in the mild, but still rambunctious west, if I were to put terms on it. With reference to guess and check -- yes, that especially was prevalent and worked 2-3 years ago and I'd be in favor of advocating that it does still happen somewhat in a more refined fashion, but I personally believe we'd not get too far beyond the SOTA if we're not working (effectively) with your data manifold now and tightly incorporating whatever projections/constraints of that data distillation process into your network training procedure. I really do agree with you in that I think average Joe breakthroughs will happen and continue to benefit the the field, and I'd certainly agree that there's always going to be the mediocre paper churn of paper mills I think that you alluded to trying to justify their own existence as academics/paper writers, but I really do legitimately think there's enough precedent set in most parts of the field that you need to have some kind of thoughtful improvement to move forward (like AdaBelief, which is still terrible because they straight up lie about what they do in the abstract, even though the improvement of debiasing the variance estimates during training is an exceptionally good idea). Just my 2c, hope this helps. I think we may have a similar end perspective from two different sides, like two explorers looking at the same peak from the different side of the mountain. :thumbsup:

jgalt212 6y ago

> The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. "guess and check" is terribly ineffective with multi-day training runs. Brings us right back to the batch processing paradigm of the 1960s.

sjg007 6y ago

>We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. Have you not heard of alphafold?

solarengineer 6y ago
Thanks for mentioning Alphafold. This article by the DeepMind team is very insightful. https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-...

eachro 6y ago

> An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. Right, but you have to remember there are legions of grad students doing exactly this so it ends up being quite competitive to churn out papers this way.

visarga 6y ago

> Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. I get that you'd like to have a clear theoretical basis for what works and we're far from there. But in the meantime we're stumbling in the dark, discovering tricks and forming intuitions, not knowing even where the road is going to lead us. This is an evolutionary process of ideas, similar to biological evolution that managed to make us. If you know where you're going you can optimise your actions but when you don't even know what might be useful later on, then all attempts are good. They increase diversity and discover blind spots. Some of them will be the stepping stones for the future, but we can't say in advance which and how. Link to a long discussion about the evolution of ML ideas and the book "Why greatness cannot be planned" by Kenneth Stanley - https://youtu.be/lhYGXYeMq_E?t=416 https://youtu.be/lhYGXYeMq_E?t=416

woko 6y ago

I see your point when it comes to paper publication in general, but I feel that your post is very unwarranted with respect to the original post: DeepMind has recruited top-quality theoretical researchers from public institutions, it is not just experimental work which your average Joe could do with a few guesses. These researchers published a lot of theoretical papers before they were recruited, and they still publish a lot of them now that they are working at DeepMind, but they have more computational hardware to apply their ideas. Here is just one uncurated example of a publication: - https://deepmind.com/research/publications/Taylor-Expansion-Policy-Optimization https://deepmind.com/research/publications/Taylor-Expansion-... - https://arxiv.org/pdf/2003.06259.pdf https://arxiv.org/pdf/2003.06259.pdf

physicsguy 6y ago

Minor variations on top of existing stuff and occasional leaps forward is most research though, it's not surprising ML research follows a similar pattern.