4 ms·

>What does that tell us? A few things, first the authors are completely ignoring the danger of multiple hypothesis testing and generally piss on any statistical

>What does that tell us? A few things, first the authors are completely ignoring the danger of multiple hypothesis testing and generally piss on any statistical foundations of their "research" This is a really poor take, considering I make the same gut feel choices professionally and they generally work in production. This is still a brand new field and we fundamentally don't have answ re as to why many of these tweaks work better than others. That shouldn't stop someone from publishing a novel architecture or improvement.

7532yahoogmail 6y ago
I do not object to publishing a better result. Like the paper said serious craft goes into that and there's nothing wrong with typing it up and having it published. But it's not science; one day one time some how some way the black boxes have got to open up to address why. This is a very valid question and on point criticism. Even in basic software development why (eg requirements) should be known. It's not consequence free free to proceed otherwise.
- mtgp1000 6y ago
  >But it's not science That's just not true. We are probing a novel domain. In fact how else would you expect this to proceed? We've discovered a new phenomenon, which likely requires novel mathematics, yet through this exact kind of experimentation we are building the intuition that will guide more rigorous formalization later. Sure, I get it, the quality on arxiv isn't the same as some physics journal; but to dismiss this as unscientific is not only wrong but very much unfair. I'm working the cutting edge at work, and we're documenting our discoveries as we map the structure of a new frontier - if that isn't science, I don't know what is. tldr this is how science progresses in new domains before novel mathematics and formalisms are developed to address the new class of problems. This is a really exciting time if neural nets don't hit any serious blocks. Edit: and by the way, my coworkers are all graduated educated scientists from various backgrounds. What else are they doing if not science?
YeGoblynQueenne 6y ago
Machine learning is not a brand new field. The name was first used in 1959 by Arthur Samuel. The first "artificial neuron", the Pitts & McCulloch neuron, was described in 1938. Rosenblat described the Perceptron in 1958. Backpropagation was first propsed to train neural networks by Werbos in 1976 and then again in 1986 by Rumelhart, Hinton and Williams. Even "modern" deep learning architectures like LSTMs and CNNs are already more than 20 years old. LSTMs were first described by Hochreiter and Schmidhuber in 1997. The Neocognitron was described by Fukushima in 1980. And so on. Machine learning did not start in 2012.
- mtgp1000 6y ago
  Yes, the foundations for ML are somewhat old, but the explosion of progress has put everyone on the edge of a fresh domain for which we are only now developing a theoretical framework. At the very least you must concede that applied machine learning is effectively a new field. Yes I'm sure a handful of neuron-like components have probably been assembled here or there in the past to do something interesting, but no one was going to college trying to make it big with a career in ML.
  - YeGoblynQueenne 6y ago
    >> Yes I'm sure a handful of neuron-like components have probably been assembled here or there in the past to do something interesting, but no one was going to college trying to make it big with a career in ML. I'm at a looss as to how to respond to your comment. You seem to really believe that before the big success of CNNs in 2012 there were no neural netwroks to speak of. That could not be farther from the truth. Wikipedia has a decent introduction to the history of neural networks: https://en.wikipedia.org/wiki/History_of_artificial_neural_networks https://en.wikipedia.org/wiki/History_of_artificial_neural_n... I suggest you start from there, then follow the links etc.
    - mtgp1000 6y ago
      You're underappreciating how much more we are doing with neural networks now. Image recognition, general purpose contextual searching, NLP, accelerated 3D modeling, abstract problem solving (applied derivatives of alphago and other RL), and we're on the cusp of image synthesis, music synthesis, news summarization, not to mention translation... All of this powered by dozens of exotic architectures and hundreds of discovered tweaks and optimizations - we are finally digging into the iceberg of which research before the 10s only scratch the surface. That's not to discount the visionary work of the past! But if you plotted some general measure of progress in this field, you would see an enormous discontinuity in the derivative starting sometime in the 10s with GPUs and the open source/open science initiatives. For all the shit I give to Google and Facebook and FAANG in general, they absolutely brought immense good to society by democratizing practical general function approximators. If there aren't any major roadblocks ML will touch every aspect of our near future lives, much like the internet. There's just too much potential - the cutting edge is just hitting the industry, our tech is already proven and it goes far deeper than classifying cats and dogs or digits. This is real science and it's huge, and it's a fact that even the money has never been remotely there for the kind of progress we are making; not to even mention the drastic difference between running ML on 1980s hardware vs a modern GPU. The kind of science we can now do from the comfort of our homes was impossible until very recently. It's like coming out of the bronze age and starting to work with steel. We build the tools as we go along. This is part of the process and we're actually doing a great job.

7532yahoogmail 6y ago

I do not object to publishing a better result. Like the paper said serious craft goes into that and there's nothing wrong with typing it up and having it published. But it's not science; one day one time some how some way the black boxes have got to open up to address why. This is a very valid question and on point criticism. Even in basic software development why (eg requirements) should be known. It's not consequence free free to proceed otherwise.

mtgp1000 6y ago
>But it's not science That's just not true. We are probing a novel domain. In fact how else would you expect this to proceed? We've discovered a new phenomenon, which likely requires novel mathematics, yet through this exact kind of experimentation we are building the intuition that will guide more rigorous formalization later. Sure, I get it, the quality on arxiv isn't the same as some physics journal; but to dismiss this as unscientific is not only wrong but very much unfair. I'm working the cutting edge at work, and we're documenting our discoveries as we map the structure of a new frontier - if that isn't science, I don't know what is. tldr this is how science progresses in new domains before novel mathematics and formalisms are developed to address the new class of problems. This is a really exciting time if neural nets don't hit any serious blocks. Edit: and by the way, my coworkers are all graduated educated scientists from various backgrounds. What else are they doing if not science?

YeGoblynQueenne 6y ago

Machine learning is not a brand new field. The name was first used in 1959 by Arthur Samuel. The first "artificial neuron", the Pitts & McCulloch neuron, was described in 1938. Rosenblat described the Perceptron in 1958. Backpropagation was first propsed to train neural networks by Werbos in 1976 and then again in 1986 by Rumelhart, Hinton and Williams. Even "modern" deep learning architectures like LSTMs and CNNs are already more than 20 years old. LSTMs were first described by Hochreiter and Schmidhuber in 1997. The Neocognitron was described by Fukushima in 1980. And so on. Machine learning did not start in 2012.

mtgp1000 6y ago
Yes, the foundations for ML are somewhat old, but the explosion of progress has put everyone on the edge of a fresh domain for which we are only now developing a theoretical framework. At the very least you must concede that applied machine learning is effectively a new field. Yes I'm sure a handful of neuron-like components have probably been assembled here or there in the past to do something interesting, but no one was going to college trying to make it big with a career in ML.
- YeGoblynQueenne 6y ago
  >> Yes I'm sure a handful of neuron-like components have probably been assembled here or there in the past to do something interesting, but no one was going to college trying to make it big with a career in ML. I'm at a looss as to how to respond to your comment. You seem to really believe that before the big success of CNNs in 2012 there were no neural netwroks to speak of. That could not be farther from the truth. Wikipedia has a decent introduction to the history of neural networks: https://en.wikipedia.org/wiki/History_of_artificial_neural_networks https://en.wikipedia.org/wiki/History_of_artificial_neural_n... I suggest you start from there, then follow the links etc.
  - mtgp1000 6y ago
    You're underappreciating how much more we are doing with neural networks now. Image recognition, general purpose contextual searching, NLP, accelerated 3D modeling, abstract problem solving (applied derivatives of alphago and other RL), and we're on the cusp of image synthesis, music synthesis, news summarization, not to mention translation... All of this powered by dozens of exotic architectures and hundreds of discovered tweaks and optimizations - we are finally digging into the iceberg of which research before the 10s only scratch the surface. That's not to discount the visionary work of the past! But if you plotted some general measure of progress in this field, you would see an enormous discontinuity in the derivative starting sometime in the 10s with GPUs and the open source/open science initiatives. For all the shit I give to Google and Facebook and FAANG in general, they absolutely brought immense good to society by democratizing practical general function approximators. If there aren't any major roadblocks ML will touch every aspect of our near future lives, much like the internet. There's just too much potential - the cutting edge is just hitting the industry, our tech is already proven and it goes far deeper than classifying cats and dogs or digits. This is real science and it's huge, and it's a fact that even the money has never been remotely there for the kind of progress we are making; not to even mention the drastic difference between running ML on 1980s hardware vs a modern GPU. The kind of science we can now do from the comfort of our homes was impossible until very recently. It's like coming out of the bronze age and starting to work with steel. We build the tools as we go along. This is part of the process and we're actually doing a great job.