7 ms·

I think it often goes understated just how much the emergence of massive datasets led to the successes of neural networks. I once heard Daphne Koller say that

I think it often goes understated just how much the emergence of massive datasets led to the successes of neural networks. I once heard Daphne Koller say that before big data, neural networks were always the second best way to do anything.

bmitc 5y ago
What was the first best way?
- visarga 5y ago
  SVMs and Random Forests.
otabdeveloper4 5y ago
> successes of neural networks Too early to say that with a straght face yet. "Deep learning" produced some amazing generative art, and some very flashy research papers, but it failed to drive business decisions. (And not for the lack of trying, that's for sure.)
- joefourier 5y ago
  What about machine translation, speech recognition and transcription, speech synthesis, image classification, segmentation and other computer vision tasks, facial recognition, image and audio denoising, beating world champions at Go along with chess engines, language models for text generation, image editing (e.g. FaceApp)... You don’t have to fall for the hype and think deep learning will lead us to AGI in the next 5 years, but dismissing it as generative art and flashy papers isn’t any more accurate.
  - otabdeveloper4 5y ago
    Did you read my post? None of that drives business decisions. As for Alexa, et al - these things fall into the "generative art" bucket. The search results they give aren't any better than the Eliza-tier expert systems of yore. They just feel much better and more human when you use them. (Which is also important, but doesn't drive business decisions, except as part of a marketing strategy.)
    - joefourier 5y ago
      What do you mean by doesn’t drive business decisions? ~65 million Alexas were sold last year, and the product would be fairly useless without very high performance speech recognition powered by deep learning. I personally own a startup that uses deep neural networks as part of its core product functionality and our business decisions would be drastically different if it weren’t for modern machine learning. We could technically ship some similar products, but they would either be far inferior or take orders of magnitude longer to engineer by hand.
      - otabdeveloper4 5y ago
        Most business requires data analysis and planning. Under the hood regression models and/or some sort of component analysis is used. Deep learning is a cool feature to differentiate Alexa from competing home electronic gadgets. But if you want to forecast Alexa sales, or understand market segmentation, or the portrait of a typical Alexa buyer then you need something other than deep learning because neural nets utterly failed in this domain. The business metrics problem is vastly, vastly more important than the "making cool gadgets" problem, and huge resources were poured into making deep learning a thing in this space. The money was mostly wasted. (A negative result is still a result, but still the misallocation of resources is staggering.)
        joefourier 5y ago
        I get that it didn’t work in your particular field and that obviously that’s what matters the most to you, but I would suggest reading up on its wider applications before dismissing the whole technology. Entire classes of products suddenly being possible is not just a “cool feature”. FYI, deep learning isn’t a differentiating feature for the Alexa, it powers essentially all modern voice applications.
- snowgrove 5y ago
  As the other commenter mentions, neural network-driven speech recognition represents the current state of the art and powers products like Alexa and Google Home. https://ai.googleblog.com/2020/11/improving-on-device-speech-recognition.html https://ai.googleblog.com/2020/11/improving-on-device-speech...
thomasahle 5y ago
LeCun said that on this recent podcast too: https://www.listennotes.com/podcasts/machine-learning-street-talk-mlst-machine-exNG8XK0CFn/ https://www.listennotes.com/podcasts/machine-learning-street... Before the ImageNet dataset, CNNs weren't worth it.
6gvONxR4sf7o 5y ago
Agreed. There’s a question of ownership in that too. It’s frequently brought up in regards to Copilot. If companies had had to pay to license all the photos or code or writing they train on, a lot of these datasets wouldn’t exist, and then neither would the models. Which is why I wish there was a copyleft open source data analogue. If you train on everyone’s public data, your model should have to be just as publicly available.
- stickfigure 5y ago
  I hereby request that you submit your brain for... public availability.
- endgame 5y ago
  I don't understand why these models are not "derived works" under the GPL family of licences.
  - glandium 5y ago
    Especially when you consider that on one end some companies forbid their employees to even read GPL source to alleviate the legal risks of accidental copy, and on the other hand, there's the common practice of clean room implementations for more or less the same reason. If brains can be tainted by reading code, how does that not apply to Copilot?
    - pc86 5y ago
      > some companies forbid their employees to even read GPL source Is this really common enough it's even worth mentioning?
  - PeterisP 5y ago
    Licenses don't get to define what is a derived work which needs a license from the author, that's defined solely by copyright law - and it's a tricky issue because the definitions used in copyright law aren't intended for a scenario like this and are rather ambiguous (at least in US) and IMHO does not have any reasonable case law to inform which plausible interpretations would be valid and which would not. It's not clear whether the model trained on some data is copyrightable at all (and if it isn't, it can't be a derived work and gets no protection nor restrictions from copyright law) as in general facts about a work - including things like word frequency statistics, which was a popular type of trained language models not that long ago (e.g. n-gram models used in statistical MT systems) - are not copyrightable and in that case making/distributing those is not an exclusive right of the author and needs no license or permission, that was settled long ago between publishers and e.g. dictionary makers. There is also the notion that mechanistic transformations or difficult labor can't result in copyrightable work, there needs to be human creativity involved; in US copyright law (as in Fiest Publications Inc. v. Rural phone service Co, also see https://www.gutenberg.org/help/no_sweat_copyright.html https://www.gutenberg.org/help/no_sweat_copyright.html) the fact that making some work required lots of work and cost you lots of money does not imply that it deserves copyright protection, no matter how much work was required; so it's irrelevant that someone spent millions of dollars worth of GPU time, and it could certainly be argued (case law would be useful here!) that the software used to train the model is copyrightable, but the model output by that software is not. Perhaps case law or some new explicit law will settle otherwise, but currently all the research and industry is proceeding with the assumption that a trained model is not a derived work (in the copyright law sense) from the training data, and as far as I see this assumption is not being challenged in courts.
bigcat123 5y ago

bmitc 5y ago

What was the first best way?

visarga 5y ago
SVMs and Random Forests.

otabdeveloper4 5y ago

> successes of neural networks Too early to say that with a straght face yet. "Deep learning" produced some amazing generative art, and some very flashy research papers, but it failed to drive business decisions. (And not for the lack of trying, that's for sure.)

joefourier 5y ago
What about machine translation, speech recognition and transcription, speech synthesis, image classification, segmentation and other computer vision tasks, facial recognition, image and audio denoising, beating world champions at Go along with chess engines, language models for text generation, image editing (e.g. FaceApp)... You don’t have to fall for the hype and think deep learning will lead us to AGI in the next 5 years, but dismissing it as generative art and flashy papers isn’t any more accurate.
- otabdeveloper4 5y ago
  Did you read my post? None of that drives business decisions. As for Alexa, et al - these things fall into the "generative art" bucket. The search results they give aren't any better than the Eliza-tier expert systems of yore. They just feel much better and more human when you use them. (Which is also important, but doesn't drive business decisions, except as part of a marketing strategy.)
  - joefourier 5y ago
    What do you mean by doesn’t drive business decisions? ~65 million Alexas were sold last year, and the product would be fairly useless without very high performance speech recognition powered by deep learning. I personally own a startup that uses deep neural networks as part of its core product functionality and our business decisions would be drastically different if it weren’t for modern machine learning. We could technically ship some similar products, but they would either be far inferior or take orders of magnitude longer to engineer by hand.
    - otabdeveloper4 5y ago
      Most business requires data analysis and planning. Under the hood regression models and/or some sort of component analysis is used. Deep learning is a cool feature to differentiate Alexa from competing home electronic gadgets. But if you want to forecast Alexa sales, or understand market segmentation, or the portrait of a typical Alexa buyer then you need something other than deep learning because neural nets utterly failed in this domain. The business metrics problem is vastly, vastly more important than the "making cool gadgets" problem, and huge resources were poured into making deep learning a thing in this space. The money was mostly wasted. (A negative result is still a result, but still the misallocation of resources is staggering.)
      - joefourier 5y ago
        I get that it didn’t work in your particular field and that obviously that’s what matters the most to you, but I would suggest reading up on its wider applications before dismissing the whole technology. Entire classes of products suddenly being possible is not just a “cool feature”. FYI, deep learning isn’t a differentiating feature for the Alexa, it powers essentially all modern voice applications.
snowgrove 5y ago
As the other commenter mentions, neural network-driven speech recognition represents the current state of the art and powers products like Alexa and Google Home. https://ai.googleblog.com/2020/11/improving-on-device-speech-recognition.html https://ai.googleblog.com/2020/11/improving-on-device-speech...

thomasahle 5y ago

LeCun said that on this recent podcast too: https://www.listennotes.com/podcasts/machine-learning-street-talk-mlst-machine-exNG8XK0CFn/ https://www.listennotes.com/podcasts/machine-learning-street... Before the ImageNet dataset, CNNs weren't worth it.

6gvONxR4sf7o 5y ago

Agreed. There’s a question of ownership in that too. It’s frequently brought up in regards to Copilot. If companies had had to pay to license all the photos or code or writing they train on, a lot of these datasets wouldn’t exist, and then neither would the models. Which is why I wish there was a copyleft open source data analogue. If you train on everyone’s public data, your model should have to be just as publicly available.

stickfigure 5y ago
I hereby request that you submit your brain for... public availability.
endgame 5y ago
I don't understand why these models are not "derived works" under the GPL family of licences.
- glandium 5y ago
  Especially when you consider that on one end some companies forbid their employees to even read GPL source to alleviate the legal risks of accidental copy, and on the other hand, there's the common practice of clean room implementations for more or less the same reason. If brains can be tainted by reading code, how does that not apply to Copilot?
  - pc86 5y ago
    > some companies forbid their employees to even read GPL source Is this really common enough it's even worth mentioning?
- PeterisP 5y ago
  Licenses don't get to define what is a derived work which needs a license from the author, that's defined solely by copyright law - and it's a tricky issue because the definitions used in copyright law aren't intended for a scenario like this and are rather ambiguous (at least in US) and IMHO does not have any reasonable case law to inform which plausible interpretations would be valid and which would not. It's not clear whether the model trained on some data is copyrightable at all (and if it isn't, it can't be a derived work and gets no protection nor restrictions from copyright law) as in general facts about a work - including things like word frequency statistics, which was a popular type of trained language models not that long ago (e.g. n-gram models used in statistical MT systems) - are not copyrightable and in that case making/distributing those is not an exclusive right of the author and needs no license or permission, that was settled long ago between publishers and e.g. dictionary makers. There is also the notion that mechanistic transformations or difficult labor can't result in copyrightable work, there needs to be human creativity involved; in US copyright law (as in Fiest Publications Inc. v. Rural phone service Co, also see https://www.gutenberg.org/help/no_sweat_copyright.html https://www.gutenberg.org/help/no_sweat_copyright.html) the fact that making some work required lots of work and cost you lots of money does not imply that it deserves copyright protection, no matter how much work was required; so it's irrelevant that someone spent millions of dollars worth of GPU time, and it could certainly be argued (case law would be useful here!) that the software used to train the model is copyrightable, but the model output by that software is not. Perhaps case law or some new explicit law will settle otherwise, but currently all the research and industry is proceeding with the assumption that a trained model is not a derived work (in the copyright law sense) from the training data, and as far as I see this assumption is not being challenged in courts.

bigcat123 5y ago