9 ms·

Untapped opportunities in AI

164 points by dennybritz 12y ago

striglia 12y ago
Cool article. I really like repetition that model complexity is not a pancea. Seems like the industrial AI/ML movement as a whole has gone down a road where practitioners will, by default, throw the most powerful model they know at a problem and see how it pans out. Works well on benchmarks(if you regularize/validate carefully) but isn't a very sustainable way to engineer a system. Separately, I do find it curious that his list of "pretty standard machine-learning methods" included Logistic Regression, K-means and....deep neural nets? Sure they're white hot in terms of popularity and the experts have done astounding things, but unless I've missed some major improvements in their off-the-shelf usability they strike me as out of place in this list.
- beaucronin 12y ago
  That's fair pushback re how "standard" deep learning is. That said, those methods are rapidly establishing themselves as the go-to for applications in speech, vision, and text. At least for those outfits that can afford the substantial dev costs.
  - agibsonccc 12y ago
    This is what I am hoping to launch with the launch of skymind[1]. I believe in the potential of deep learning, especially in the ways deep learning can learn better representations for feature extraction than some hand curated models. Practicality has always been a problem with deep learning, and I think enabling access to this powerful technology will be a great enabler for many people in the long term. [1] http://wired.com/2014/06/skymind-deep-learning/ http://wired.com/2014/06/skymind-deep-learning/
- dgreensp 12y ago
  Agreed on both points. In addition, I don't know why anyone would think Google is going to make all the AI, just because today's most notable, state-of-the-art AI systems are made by a few big companies with the resources to fund large teams of experts for years. Fifty years ago this article could have been called "Untapped opportunities in software" (or operating systems) and talked about IBM -- is there software IBM can't or won't write for us?
  - nostrademons 12y ago
    I think the key difference between Google and IBM is that Google has shown a clear interest in AI, and practice of it is pretty pervasive throughout the company. There's a world of difference between "The largest tech company that can fund a team of experts" and "The largest tech company that is funding multiple teams of experts to actively work on the problem."
    - xerophtye 12y ago
      >I think the key difference between Google and IBM is that Google has shown a clear interest in AI Umm... What about Watson?
      - DanBC 12y ago
        Google needs AI for its core product. Better search is an AI problem. What did the user ask for versus what did the user want to ask for?
- craigching 12y ago
  In comments he mentions two "high quality open-source packages": http://torch.ch/ http://torch.ch/ (used widely at large companies) http://deeplearning4j.org/ http://deeplearning4j.org/ (newer) I have no idea if they are quality or open source, just posting the information.
  - agibsonccc 12y ago
    Hi, creator of deeplearning4j here. I'd like to add this IS newer. That being said, I am going to be focusing on ease of use here. Currently a lot of the deep learning frameworks out there aren't focused on a lot of practicality. I am also hoping to add sdks for different languages trying to make this a fast general purposed deep learning core that allows for people to do neural nets in different languages while also benefiting from a fast runtime. I'm going to be opening up a good contribution pipeline here soon and would love to answer questions where possible. I'm looking for help on everything from documentation to feature requests.
- fchollet 12y ago
  Deep convolutional neural nets are the staple method for companies doing computer vision at scale. Google, Facebook, Yahoo, and Baidu make extensive use of them. In 2014 they definitely deserve their place on the shortlist of "pretty standard machine-learning methods". They are the current state of the art for visual recognition in particular (see the results on ImageNet from the past few years). They have also been commoditized by libraries such as Theano (Python) and Torch (Lua). Google and Facebook use their own tools based on Torch. My own version of the shortlist would be: Logistic regression, KNN, random forests, SVM, and deep convnets.
  - agibsonccc 12y ago
    This is why I'm starting skymind[1] to address. Many of these companies are using cutting edge techniques without a focus on usability (since they know this stuff). The hope here is to create an off the shelf package people can just use with only knowledge of fairly conventional machine learning techniques while also avoiding the problems of having to program in java (which isn't realistic in data science) or even lua. Many people could benefit from a neural nets ability to create good features for itself, but it's hard to use in a practical setting. That being said over the next few years I believe this can change. [1]: http://wired.com/2014/06/skymind-deep-learning/ http://wired.com/2014/06/skymind-deep-learning/
    - fchollet 12y ago
      I like the idea a lot. Just trying to understand this better: it seems like your company is entirely about selling consulting services, yet your stated goal is to "give people machine learning without them having to hire a data scientist". What's your path to that goal?
      - agibsonccc 12y ago
        In this case, being that on staff data scientist for them. Many companies only need a one off model to set themselves up for some sort of baseline data product. This can also be training for them on using machine learning for their problem solving. The goal isn't necessarily to totally supplant data scientists (love press sensationalism), but to help enable companies to build easy to use models in their apps. This can also map to saving data scientists time by not necessarily "skipping" the feature extraction part (which they can with deep learning and still do reasonably well) but allowing them to just use a fairly good machine learning model out of the box to use as a baseline. The great thing about machine learning is the ability to mix different techniques. Google's voice detection is a great example of this. They use neural nets for feature extraction and hidden markov models for final translation of speech to text. I think deep learning (if wrapped in the right apps or sdks) with the auto feature extraction alongside then specifying say: a "task". This task could be predicting a value, labeling things, or even compression of data[1] would allow companies to not focus on machine learning, but on straight problem solving. The idea would be once they are familiar enough with how to feed data in to the system, and specifying a "task", they can do a lot of machine learning by themselves without having to think too much about the problem they are solving (what features work best given the data I have?) [1] http://www.slideshare.net/agibsonccc/ir-34811120 http://www.slideshare.net/agibsonccc/ir-34811120
- hershel 12y ago
  http://www.ersatzlabs.com/ http://www.ersatzlabs.com/ claim to offer intuitive deep learning.
- dkersten 12y ago
  I do find it curious that his list of "pretty standard machine-learning methods" included Logistic Regression, K-means and....deep neural nets? The next sentence is much more curious to me: "the point is that they’re widely available in high-quality open source packages" because I have yet to find a proper well documented well maintained non-toy open source deep neural network implementation.
araes 12y ago
I can honestly say that this post has revolutionized my thoughts on AI. Primarily this is because of what I perceive as the thesis statement, which is: "<AI> is the construction of weighted tables (choices, data, meta relations, whatever) from large sets of <prior data> by <method>" This is kind of crazy, because I think it says you could make a Turing AI by using large datasets of prior life data for humans. In essence, "<my life> is the construction of weighted tables from large sets of <life experience> by <human learning>." For example, if you had an AI that could learn through text, you could have extensive transcribed conversation logs of people and then large time-activity logs to use as your inputs. If it could learn through video (IE, it could view images, understand objects, object relations, events in time, and assign will to the person behind actions / events) then you could instead just feed it huge video logs of people's lives. If you wanted a copy of a person, you could feed it only a single individual, and if you wanted a more general AI, then you could feed it cross sections of the population. In addition, there's a very cool meta aspect to the large dataset concept, in that it can be large datasets for when to use, or to feed data to, specialized sub-AI's. For example, you might have a math sub-AI that has been trained by feeding it massive sets of math problems (or perhaps it can learn math through the video life logs of a person?). If its then being used as a part of a larger piece, then you'd want to know when to use it to solve problems, or when to feed it experience inputs for further learning. In essence, its tables of categories for experience types, and then grown / paired sub-AI's for those types. I would wager that it is possible, right now, to create a chatbot that can pass Turing using the above by feeding it the equivalent of mass IRC chat or somesuch huge, human interaction by text dataset over a variety of topics. This would naturally need sub-AI's for mechanical things like grammar or parts of speech, and then possibly higher level meta-AI's for interpreting intent, orchestrating long form thought, or planning. In a way, its layers of AI based on level of thought abstraction. If it were a human, the high intensity portions of sub-AI would occupy space relative to intensity within reconfigurable co-processor zones (sight:visual cortex, sight:face recognition:occipital and temporal lobes, executive functions:frontal lobes, ect...)
- sp332 12y ago
  It doesn't explain any personality differences. Or chemical changes like whether a person is hungry, or drunk.
hyp0 12y ago
Massive datasets do outperform clever theories... but I think that's just because no one has yet worked out the theories that work best with the data. This requires insight, in addition to data, and could come from anyone. The alternative - that massively complex probabilistic models are the best theory of the data - is hopefully not true. Especially not of our minds. But it could be true, and if so, it would mean that our intelligence is irreducible, and we are forever beyond our own self-understanding (even in principle). Our history is full of inexplicable mysteries that were eventually understood. But not all: quantum randomness. I really hope intelligence is will be one of the former.
- rwissmann 12y ago
  No reason to be so pessimistic about quantum randomness. Quantum theory is barely 100 years old and our understanding of it is still evolving massively. Though the latter is not always appreciated by the public.
  - kansface 12y ago
    Well, local hidden variables are out the window... do you see any indication this will change?
    - rwissmann 12y ago
      Personally I am also not convinced that LHV theories will every describe the effects correctly. But whether the successor of QM ultimately contains true randomness or not, there is no reason to think it will forever stay an 'inexplicable mystery'.
- iandanforth 12y ago
  There are a lot of AI problems that can be solved with less than human intelligence but some human numbers for reference: By the time you're 30 you have been exposed to: ~1.4 petabytes of visual information ~1.8 terabytes of auditory information Touch and proprioceptive bandwidth is harder to calculate but the ascending pathway through the spinal cord is about 10 million fibers, which is 10x the optic nerve (Or 5x the number of fibers from both eyes). So: Between 1.4 and 14 petabytes of touch and proprioceptive information. So we're a fairly large data problem on top of millions of years of evolution that have baked in some knowledge and abilities.
jostmey 12y ago
As a postdoctoral candidate in biology, I can say that my approach to problem solving is exactly the opposite: My job is to infer as much as I can from the scant amount of data I can obtain. The goals outlined in this article are to collect as much data as you can, creating what is essentially a glorified lookup table of results. I must say the latter approach seems a hell of a lot easier.

164 points by dennybritz 12y ago

Cool article. I really like repetition that model complexity is not a pancea. Seems like the industrial AI/ML movement as a whole has gone down a road where practitioners will, by default, throw the most powerful model they know at a problem and see how it pans out. Works well on benchmarks(if you regularize/validate carefully) but isn't a very sustainable way to engineer a system. Separately, I do find it curious that his list of "pretty standard machine-learning methods" included Logistic Regression, K-means and....deep neural nets? Sure they're white hot in terms of popularity and the experts have done astounding things, but unless I've missed some major improvements in their off-the-shelf usability they strike me as out of place in this list.

beaucronin 12y ago
That's fair pushback re how "standard" deep learning is. That said, those methods are rapidly establishing themselves as the go-to for applications in speech, vision, and text. At least for those outfits that can afford the substantial dev costs.
- agibsonccc 12y ago
  This is what I am hoping to launch with the launch of skymind[1]. I believe in the potential of deep learning, especially in the ways deep learning can learn better representations for feature extraction than some hand curated models. Practicality has always been a problem with deep learning, and I think enabling access to this powerful technology will be a great enabler for many people in the long term. [1] http://wired.com/2014/06/skymind-deep-learning/ http://wired.com/2014/06/skymind-deep-learning/
dgreensp 12y ago
Agreed on both points. In addition, I don't know why anyone would think Google is going to make all the AI, just because today's most notable, state-of-the-art AI systems are made by a few big companies with the resources to fund large teams of experts for years. Fifty years ago this article could have been called "Untapped opportunities in software" (or operating systems) and talked about IBM -- is there software IBM can't or won't write for us?
- nostrademons 12y ago
  I think the key difference between Google and IBM is that Google has shown a clear interest in AI, and practice of it is pretty pervasive throughout the company. There's a world of difference between "The largest tech company that can fund a team of experts" and "The largest tech company that is funding multiple teams of experts to actively work on the problem."
  - xerophtye 12y ago
    >I think the key difference between Google and IBM is that Google has shown a clear interest in AI Umm... What about Watson?
    - DanBC 12y ago
      Google needs AI for its core product. Better search is an AI problem. What did the user ask for versus what did the user want to ask for?
craigching 12y ago
In comments he mentions two "high quality open-source packages": http://torch.ch/ http://torch.ch/ (used widely at large companies) http://deeplearning4j.org/ http://deeplearning4j.org/ (newer) I have no idea if they are quality or open source, just posting the information.
- agibsonccc 12y ago
  Hi, creator of deeplearning4j here. I'd like to add this IS newer. That being said, I am going to be focusing on ease of use here. Currently a lot of the deep learning frameworks out there aren't focused on a lot of practicality. I am also hoping to add sdks for different languages trying to make this a fast general purposed deep learning core that allows for people to do neural nets in different languages while also benefiting from a fast runtime. I'm going to be opening up a good contribution pipeline here soon and would love to answer questions where possible. I'm looking for help on everything from documentation to feature requests.
fchollet 12y ago
Deep convolutional neural nets are the staple method for companies doing computer vision at scale. Google, Facebook, Yahoo, and Baidu make extensive use of them. In 2014 they definitely deserve their place on the shortlist of "pretty standard machine-learning methods". They are the current state of the art for visual recognition in particular (see the results on ImageNet from the past few years). They have also been commoditized by libraries such as Theano (Python) and Torch (Lua). Google and Facebook use their own tools based on Torch. My own version of the shortlist would be: Logistic regression, KNN, random forests, SVM, and deep convnets.
- agibsonccc 12y ago
  This is why I'm starting skymind[1] to address. Many of these companies are using cutting edge techniques without a focus on usability (since they know this stuff). The hope here is to create an off the shelf package people can just use with only knowledge of fairly conventional machine learning techniques while also avoiding the problems of having to program in java (which isn't realistic in data science) or even lua. Many people could benefit from a neural nets ability to create good features for itself, but it's hard to use in a practical setting. That being said over the next few years I believe this can change. [1]: http://wired.com/2014/06/skymind-deep-learning/ http://wired.com/2014/06/skymind-deep-learning/
  - fchollet 12y ago
    I like the idea a lot. Just trying to understand this better: it seems like your company is entirely about selling consulting services, yet your stated goal is to "give people machine learning without them having to hire a data scientist". What's your path to that goal?
    - agibsonccc 12y ago
      In this case, being that on staff data scientist for them. Many companies only need a one off model to set themselves up for some sort of baseline data product. This can also be training for them on using machine learning for their problem solving. The goal isn't necessarily to totally supplant data scientists (love press sensationalism), but to help enable companies to build easy to use models in their apps. This can also map to saving data scientists time by not necessarily "skipping" the feature extraction part (which they can with deep learning and still do reasonably well) but allowing them to just use a fairly good machine learning model out of the box to use as a baseline. The great thing about machine learning is the ability to mix different techniques. Google's voice detection is a great example of this. They use neural nets for feature extraction and hidden markov models for final translation of speech to text. I think deep learning (if wrapped in the right apps or sdks) with the auto feature extraction alongside then specifying say: a "task". This task could be predicting a value, labeling things, or even compression of data[1] would allow companies to not focus on machine learning, but on straight problem solving. The idea would be once they are familiar enough with how to feed data in to the system, and specifying a "task", they can do a lot of machine learning by themselves without having to think too much about the problem they are solving (what features work best given the data I have?) [1] http://www.slideshare.net/agibsonccc/ir-34811120 http://www.slideshare.net/agibsonccc/ir-34811120
hershel 12y ago
http://www.ersatzlabs.com/ http://www.ersatzlabs.com/ claim to offer intuitive deep learning.
dkersten 12y ago
I do find it curious that his list of "pretty standard machine-learning methods" included Logistic Regression, K-means and....deep neural nets? The next sentence is much more curious to me: "the point is that they’re widely available in high-quality open source packages" because I have yet to find a proper well documented well maintained non-toy open source deep neural network implementation.

araes 12y ago

I can honestly say that this post has revolutionized my thoughts on AI. Primarily this is because of what I perceive as the thesis statement, which is: "<AI> is the construction of weighted tables (choices, data, meta relations, whatever) from large sets of <prior data> by <method>" This is kind of crazy, because I think it says you could make a Turing AI by using large datasets of prior life data for humans. In essence, "<my life> is the construction of weighted tables from large sets of <life experience> by <human learning>." For example, if you had an AI that could learn through text, you could have extensive transcribed conversation logs of people and then large time-activity logs to use as your inputs. If it could learn through video (IE, it could view images, understand objects, object relations, events in time, and assign will to the person behind actions / events) then you could instead just feed it huge video logs of people's lives. If you wanted a copy of a person, you could feed it only a single individual, and if you wanted a more general AI, then you could feed it cross sections of the population. In addition, there's a very cool meta aspect to the large dataset concept, in that it can be large datasets for when to use, or to feed data to, specialized sub-AI's. For example, you might have a math sub-AI that has been trained by feeding it massive sets of math problems (or perhaps it can learn math through the video life logs of a person?). If its then being used as a part of a larger piece, then you'd want to know when to use it to solve problems, or when to feed it experience inputs for further learning. In essence, its tables of categories for experience types, and then grown / paired sub-AI's for those types. I would wager that it is possible, right now, to create a chatbot that can pass Turing using the above by feeding it the equivalent of mass IRC chat or somesuch huge, human interaction by text dataset over a variety of topics. This would naturally need sub-AI's for mechanical things like grammar or parts of speech, and then possibly higher level meta-AI's for interpreting intent, orchestrating long form thought, or planning. In a way, its layers of AI based on level of thought abstraction. If it were a human, the high intensity portions of sub-AI would occupy space relative to intensity within reconfigurable co-processor zones (sight:visual cortex, sight:face recognition:occipital and temporal lobes, executive functions:frontal lobes, ect...)

sp332 12y ago
It doesn't explain any personality differences. Or chemical changes like whether a person is hungry, or drunk.

hyp0 12y ago

Massive datasets do outperform clever theories... but I think that's just because no one has yet worked out the theories that work best with the data. This requires insight, in addition to data, and could come from anyone. The alternative - that massively complex probabilistic models are the best theory of the data - is hopefully not true. Especially not of our minds. But it could be true, and if so, it would mean that our intelligence is irreducible, and we are forever beyond our own self-understanding (even in principle). Our history is full of inexplicable mysteries that were eventually understood. But not all: quantum randomness. I really hope intelligence is will be one of the former.

rwissmann 12y ago
No reason to be so pessimistic about quantum randomness. Quantum theory is barely 100 years old and our understanding of it is still evolving massively. Though the latter is not always appreciated by the public.
- kansface 12y ago
  Well, local hidden variables are out the window... do you see any indication this will change?
  - rwissmann 12y ago
    Personally I am also not convinced that LHV theories will every describe the effects correctly. But whether the successor of QM ultimately contains true randomness or not, there is no reason to think it will forever stay an 'inexplicable mystery'.
iandanforth 12y ago
There are a lot of AI problems that can be solved with less than human intelligence but some human numbers for reference: By the time you're 30 you have been exposed to: ~1.4 petabytes of visual information ~1.8 terabytes of auditory information Touch and proprioceptive bandwidth is harder to calculate but the ascending pathway through the spinal cord is about 10 million fibers, which is 10x the optic nerve (Or 5x the number of fibers from both eyes). So: Between 1.4 and 14 petabytes of touch and proprioceptive information. So we're a fairly large data problem on top of millions of years of evolution that have baked in some knowledge and abilities.

jostmey 12y ago

As a postdoctoral candidate in biology, I can say that my approach to problem solving is exactly the opposite: My job is to infer as much as I can from the scant amount of data I can obtain. The goals outlined in this article are to collect as much data as you can, creating what is essentially a glorified lookup table of results. I must say the latter approach seems a hell of a lot easier.