5 ms·

Neural Networks making a come-back?

64 points by mariorz 15y ago

ohashi 15y ago
Is there much commercialization of neural networks going on? I own NeuralNetwork.com but am not suited to turn it into a business. If anyone is in the space and is interested in using the category killer name, I'd love to talk.
- jal278 15y ago
  I work with NNs in a research context and am working on a startup commercializing some research-related NN technology. What price range were you thinking?
  - ohashi 15y ago
    You don't have any contact info in your profile. Could you contact me via mine?
    - jal278 15y ago
      my email is jlehman at eecs dot cs dot ucf dot edu. the captcha on your blog wouldn't work for me -- I see a recaptcha in the html, but it wasn't displaying in either firefox or chrome for me
      - ohashi 15y ago
        Your email bounced for me. Try my username at gmail.
- ComputerGuru 15y ago
  Seeing as its without the trailing 's', I think a lot of people will end up going elsewhere......
ignifero 15y ago
Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in ANNs lately
- larryfreeman 15y ago
  Are you sure about this? I'm not an expert by any means but when I participated in the Netflix Prize a few years, there seemed to be lots of investigation about different variations of neural networks. For example, the Restricted Boltzmann Machine (http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Boltzmann_Machine http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Bo...), as far as I understand it, seems to be a variation of neural networks. If you can post a link to an article that covers recent work in the area and explains why they none of them are breakthroughs, I'd love to read it.
  - yid 15y ago
    The backpropagation neural network itself is a variation of many nonlinear, additive statistical models. The field emerged as an independent entity largely because it had "cool" neurological connotations and was largely ignored by statisticians. The Netflix prize was ultimately won using linear algebra with sensible starting values (the KDD paper that describes the winning method is remarkably easy to read).
    - pdhborges 15y ago
      Is http://www.commendo.at/references/files/kdd08.pdf http://www.commendo.at/references/files/kdd08.pdf (this) the paper you are talking about?
      - yid 15y ago
        Nope, it was by Bell and Koren. This one is related, but not the one I originally mentioned: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197... I was at the talk but can't for the life of me remember the title, sorry.
hooande 15y ago
The OP is including a lot of different concepts under the neural network umbrella. Things like restricted boltzmann machines and hierarchical temporal memory are technically neural networks, but many computer scientists would consider them to be different enough in approach to think of them separately. ie, you wouldn't say "let's use a type of neural network to solve this problem" you would probably say "let's use a restricted boltzmann machine". It is true that these things are becoming more popular. I've found in practice that a modern computer scientist is still more likely to solve a simple learning problem with some form of regression, if only because it's faster than training a NN.
- srean 15y ago
  Restricted Boltzmann machines are as bonafide neural networks (NN) as you can get and has been around since the golden age of neural networks. You have the same layered structure, feed-forward connections, same "squashing function". The only "restriction" is that the unknown and the known nodes must live on different layers so that the connections are only between a known and an unknown node. It has been called different names and the theory explaining them has had different names too, for example "Harmony Theory" of Smolensky. I think NN is a broad enough category that no matter what you want to use or describe, you will have to qualify your "lets use blah" statements with a particular kind of neural network. Similar in spirit to statements like "lets use a parser" vs "lets use a LALR parser". But back to the topic of new found interest on NNs, part of the reason is that there have been new developments in training algorithms which work significantly better than what were used traditionally. With these methods NNs require far less baby-sitting. NNs traditionally really required a huge lot of that. The other reason is that sheer scale and size of the data sets that are available now, have forced machine learners to move from powerful but batch optimization algorithms (quadratic programming for instance) to simple and online gradient based algorithms that have been the forte of the NN community all along. Training a NN is no different than regression. It is another name/technique for (some what systematically) creating a tower of increasingly complex regression functions. If the simplest(linear) one works, its imperative that one uses the simplest one in the interest of good predictive accuracy on unseen data. Bundled together with the low training time that parent mentioned, its a win win.
- 15y ago
aksbhat 15y ago
Rather than using Google scholar, I would suggest looking at papers in ICML, NIPS and Journal of Machine Learning Research. For vision based research I would suggest CVPR and ICCV conference and IEEE Pattern Analysis and Machine Intelligence journal.
- yaroslavvb 15y ago
  Make these stats and I'll link them from the page :)
daoudc 15y ago
Interesting that the trend isn't replicated on the web... yet: http://www.google.com/trends?q=neural+networks http://www.google.com/trends?q=neural+networks I wonder if there is a lag between academia and the web?
- FrojoS 15y ago
  I also find it surprising to see in which countries the term is most popular according to g-trends. None of the top 10 is part of the G8. Also English is only the third most frequent language. I suppose they are lagging behind, but I might be wrong. At my Uni, the only course that had some focus on NN felt pretty outdated.
csomar 15y ago
I don't think this is significant at all because of the information boom. As the Internet and Storage has boomed and become more cheap and available, more content and data is hosted on the web. More and more is indexed by Google. May be we need to adjust the stats graph that are based on Google Scholar with the inflation of data in the last decade.
- mreid 15y ago
  Maybe I missed something but I don't understand this criticism. The author states that 'The number of hits for each year was divided by the number of hits for "machine learning".' Wouldn't that control for the inflation of data in exactly the way you are proposing?
geuis 15y ago
I have this impression that there are many more startups popping up that generally operating under the AI umbrella. Companies are doing things relating to medicine, finance, etc. There's even a couple on the May Who's Hiring post here on HN. Quite an exciting time. Seems like the AI Spring is in progress and we're coming up on Summer.

64 points by mariorz 15y ago

Is there much commercialization of neural networks going on? I own NeuralNetwork.com but am not suited to turn it into a business. If anyone is in the space and is interested in using the category killer name, I'd love to talk.

jal278 15y ago
I work with NNs in a research context and am working on a startup commercializing some research-related NN technology. What price range were you thinking?
- ohashi 15y ago
  You don't have any contact info in your profile. Could you contact me via mine?
  - jal278 15y ago
    my email is jlehman at eecs dot cs dot ucf dot edu. the captcha on your blog wouldn't work for me -- I see a recaptcha in the html, but it wasn't displaying in either firefox or chrome for me
    - ohashi 15y ago
      Your email bounced for me. Try my username at gmail.
ComputerGuru 15y ago
Seeing as its without the trailing 's', I think a lot of people will end up going elsewhere......

ignifero 15y ago

Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in ANNs lately

larryfreeman 15y ago
Are you sure about this? I'm not an expert by any means but when I participated in the Netflix Prize a few years, there seemed to be lots of investigation about different variations of neural networks. For example, the Restricted Boltzmann Machine (http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Boltzmann_Machine http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Bo...), as far as I understand it, seems to be a variation of neural networks. If you can post a link to an article that covers recent work in the area and explains why they none of them are breakthroughs, I'd love to read it.
- yid 15y ago
  The backpropagation neural network itself is a variation of many nonlinear, additive statistical models. The field emerged as an independent entity largely because it had "cool" neurological connotations and was largely ignored by statisticians. The Netflix prize was ultimately won using linear algebra with sensible starting values (the KDD paper that describes the winning method is remarkably easy to read).
  - pdhborges 15y ago
    Is http://www.commendo.at/references/files/kdd08.pdf http://www.commendo.at/references/files/kdd08.pdf (this) the paper you are talking about?
    - yid 15y ago
      Nope, it was by Bell and Koren. This one is related, but not the one I originally mentioned: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197... I was at the talk but can't for the life of me remember the title, sorry.

hooande 15y ago

The OP is including a lot of different concepts under the neural network umbrella. Things like restricted boltzmann machines and hierarchical temporal memory are technically neural networks, but many computer scientists would consider them to be different enough in approach to think of them separately. ie, you wouldn't say "let's use a type of neural network to solve this problem" you would probably say "let's use a restricted boltzmann machine". It is true that these things are becoming more popular. I've found in practice that a modern computer scientist is still more likely to solve a simple learning problem with some form of regression, if only because it's faster than training a NN.

srean 15y ago
Restricted Boltzmann machines are as bonafide neural networks (NN) as you can get and has been around since the golden age of neural networks. You have the same layered structure, feed-forward connections, same "squashing function". The only "restriction" is that the unknown and the known nodes must live on different layers so that the connections are only between a known and an unknown node. It has been called different names and the theory explaining them has had different names too, for example "Harmony Theory" of Smolensky. I think NN is a broad enough category that no matter what you want to use or describe, you will have to qualify your "lets use blah" statements with a particular kind of neural network. Similar in spirit to statements like "lets use a parser" vs "lets use a LALR parser". But back to the topic of new found interest on NNs, part of the reason is that there have been new developments in training algorithms which work significantly better than what were used traditionally. With these methods NNs require far less baby-sitting. NNs traditionally really required a huge lot of that. The other reason is that sheer scale and size of the data sets that are available now, have forced machine learners to move from powerful but batch optimization algorithms (quadratic programming for instance) to simple and online gradient based algorithms that have been the forte of the NN community all along. Training a NN is no different than regression. It is another name/technique for (some what systematically) creating a tower of increasingly complex regression functions. If the simplest(linear) one works, its imperative that one uses the simplest one in the interest of good predictive accuracy on unseen data. Bundled together with the low training time that parent mentioned, its a win win.
15y ago

aksbhat 15y ago

Rather than using Google scholar, I would suggest looking at papers in ICML, NIPS and Journal of Machine Learning Research. For vision based research I would suggest CVPR and ICCV conference and IEEE Pattern Analysis and Machine Intelligence journal.

yaroslavvb 15y ago
Make these stats and I'll link them from the page :)

daoudc 15y ago

Interesting that the trend isn't replicated on the web... yet: http://www.google.com/trends?q=neural+networks http://www.google.com/trends?q=neural+networks I wonder if there is a lag between academia and the web?

FrojoS 15y ago
I also find it surprising to see in which countries the term is most popular according to g-trends. None of the top 10 is part of the G8. Also English is only the third most frequent language. I suppose they are lagging behind, but I might be wrong. At my Uni, the only course that had some focus on NN felt pretty outdated.

csomar 15y ago

I don't think this is significant at all because of the information boom. As the Internet and Storage has boomed and become more cheap and available, more content and data is hosted on the web. More and more is indexed by Google. May be we need to adjust the stats graph that are based on Google Scholar with the inflation of data in the last decade.

mreid 15y ago
Maybe I missed something but I don't understand this criticism. The author states that 'The number of hits for each year was divided by the number of hits for "machine learning".' Wouldn't that control for the inflation of data in exactly the way you are proposing?

geuis 15y ago

I have this impression that there are many more startups popping up that generally operating under the AI umbrella. Companies are doing things relating to medicine, finance, etc. There's even a couple on the May Who's Hiring post here on HN. Quite an exciting time. Seems like the AI Spring is in progress and we're coming up on Summer.