3 ms·

Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in A

Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in ANNs lately

larryfreeman 16y ago
Are you sure about this? I'm not an expert by any means but when I participated in the Netflix Prize a few years, there seemed to be lots of investigation about different variations of neural networks. For example, the Restricted Boltzmann Machine (http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Boltzmann_Machine http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Bo...), as far as I understand it, seems to be a variation of neural networks. If you can post a link to an article that covers recent work in the area and explains why they none of them are breakthroughs, I'd love to read it.
- yid 16y ago
  The backpropagation neural network itself is a variation of many nonlinear, additive statistical models. The field emerged as an independent entity largely because it had "cool" neurological connotations and was largely ignored by statisticians. The Netflix prize was ultimately won using linear algebra with sensible starting values (the KDD paper that describes the winning method is remarkably easy to read).
  - pdhborges 16y ago
    Is http://www.commendo.at/references/files/kdd08.pdf http://www.commendo.at/references/files/kdd08.pdf (this) the paper you are talking about?
    - yid 16y ago
      Nope, it was by Bell and Koren. This one is related, but not the one I originally mentioned: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197... I was at the talk but can't for the life of me remember the title, sorry.
  - dododo 16y ago
    backpropagation is just the chain rule of differentiation. each layer of an ANN is logistic regression, which statisticians have been and continue to be interested in. i'm not sure your caricature of the netflix winning solution is correct: i believe it was a blending of around 25 different models (including, i think, the RBM someone pointed about above) each in themselves quite varied from one another. this is typically how these challenges are won.
    - yid 16y ago
      A lot of the top teams had blended submissions, but my characterization was not incorrect: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197...
      - elq 16y ago
        I work on the cinematch team at netflix... I've reviewed all of the winning code... your characterization is incorrect. The netflix prize would not have been won when it was without rbm.
- deleted 16y ago
  [deleted]
- ignifero 16y ago
  not an expert either, and indeed, the Hinton papers on RBMs are the last thing i remember reading about. But their invention dates back to the 80s. I 'm just noticing that if you search "neural network" on google scholar for the recent years, you will find many neuroscientific results. Of course i dont have the data to back it up.
albertzeyer 16y ago
Also, take a look at some of the work by J.Schmidhuber (http://www.idsia.ch/~juergen/ http://www.idsia.ch/~juergen/). For example, the LSTM architecture is quite powerful. And he very successfully applied those RNN models to a wide number of areas (handwriting recognition, robotics, ...).
dododo 16y ago
in machine learning, the 'big' thing about neural networks at the moment is deep learning via some sort of unsupervised pre-training. there's be a lot of work published on this recently. http://deeplearningworkshopnips2010.wordpress.com/ http://deeplearningworkshopnips2010.wordpress.com/
- yid 16y ago
  Hmm, could you give us a TL;DR on how this differs from model selection in statistics, e..g, choosing the structure of an HMM?
  - apu 16y ago
    A good intro page with lots of links: http://deeplearning.net/tutorial/ http://deeplearning.net/tutorial/ A lot of the newest work is under a few different names: "deep learning", "convolutional deep networks", "unsupervised feature learning", etc. Here's a great talk by Andrew Ng of Stanford, who's a recent convert to this area: http://www.youtube.com/watch?v=ZmNOAtZIgIk http://www.youtube.com/watch?v=ZmNOAtZIgIk (Note that this is quite a one-sided view of things, but it does convey the excitement of deep-learning researchers and the potential of what might be possible.) Although I'm not in deep learning myself (I'm a computer vision researcher), here's a TL;DR as I understand it: rather than having people in specific domains such as computer vision or speech processing create their own features, the idea is to take raw inputs (pixels in the case of images) and train multi-layer neural net architectures that "learn" the relevant higher-level features in an unsupervised way (i.e., without labeled training data). Some of these seem to be pulling out interesting features and perform competitively on a few benchmarks in vision and other fields. I'm not sold on this yet, because it seems like the complexity of designing features has merely been traded in for the complexity of designing different learning architectures, but it's certainly becoming quite popular these days (mostly led by Geoff Hinton of Toronto, Yoshua Bengio of Montreal, and Yann LeCunn of NYU).
    - yid 16y ago
      Thanks for that, definitely some prominent folks involved.
jkan 15y ago
Well, but he's conditioning on "machine learning" already (the little caption on the histogram says "P(neural network | machine learning"), so that most likely filters out the straight neuroscience papers.

Might this be due to the explosion on neuroscience publications that are not related to artificial neural networks? There haven't been major breakthroughs in ANNs lately

larryfreeman 16y ago

Are you sure about this? I'm not an expert by any means but when I participated in the Netflix Prize a few years, there seemed to be lots of investigation about different variations of neural networks. For example, the Restricted Boltzmann Machine (http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Boltzmann_Machine http://en.wikipedia.org/wiki/Boltzmann_machine#Restricted_Bo...), as far as I understand it, seems to be a variation of neural networks. If you can post a link to an article that covers recent work in the area and explains why they none of them are breakthroughs, I'd love to read it.

yid 16y ago
The backpropagation neural network itself is a variation of many nonlinear, additive statistical models. The field emerged as an independent entity largely because it had "cool" neurological connotations and was largely ignored by statisticians. The Netflix prize was ultimately won using linear algebra with sensible starting values (the KDD paper that describes the winning method is remarkably easy to read).
- pdhborges 16y ago
  Is http://www.commendo.at/references/files/kdd08.pdf http://www.commendo.at/references/files/kdd08.pdf (this) the paper you are talking about?
  - yid 16y ago
    Nope, it was by Bell and Koren. This one is related, but not the one I originally mentioned: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197... I was at the talk but can't for the life of me remember the title, sorry.
- dododo 16y ago
  backpropagation is just the chain rule of differentiation. each layer of an ANN is logistic regression, which statisticians have been and continue to be interested in. i'm not sure your caricature of the netflix winning solution is correct: i believe it was a blending of around 25 different models (including, i think, the RBM someone pointed about above) each in themselves quite varied from one another. this is typically how these challenges are won.
  - yid 16y ago
    A lot of the top teams had blended submissions, but my characterization was not incorrect: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197422 http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5197...
    - elq 16y ago
      I work on the cinematch team at netflix... I've reviewed all of the winning code... your characterization is incorrect. The netflix prize would not have been won when it was without rbm.
deleted 16y ago
[deleted]
ignifero 16y ago
not an expert either, and indeed, the Hinton papers on RBMs are the last thing i remember reading about. But their invention dates back to the 80s. I 'm just noticing that if you search "neural network" on google scholar for the recent years, you will find many neuroscientific results. Of course i dont have the data to back it up.

albertzeyer 16y ago

Also, take a look at some of the work by J.Schmidhuber (http://www.idsia.ch/~juergen/ http://www.idsia.ch/~juergen/). For example, the LSTM architecture is quite powerful. And he very successfully applied those RNN models to a wide number of areas (handwriting recognition, robotics, ...).

dododo 16y ago

in machine learning, the 'big' thing about neural networks at the moment is deep learning via some sort of unsupervised pre-training. there's be a lot of work published on this recently. http://deeplearningworkshopnips2010.wordpress.com/ http://deeplearningworkshopnips2010.wordpress.com/

yid 16y ago
Hmm, could you give us a TL;DR on how this differs from model selection in statistics, e..g, choosing the structure of an HMM?
- apu 16y ago
  A good intro page with lots of links: http://deeplearning.net/tutorial/ http://deeplearning.net/tutorial/ A lot of the newest work is under a few different names: "deep learning", "convolutional deep networks", "unsupervised feature learning", etc. Here's a great talk by Andrew Ng of Stanford, who's a recent convert to this area: http://www.youtube.com/watch?v=ZmNOAtZIgIk http://www.youtube.com/watch?v=ZmNOAtZIgIk (Note that this is quite a one-sided view of things, but it does convey the excitement of deep-learning researchers and the potential of what might be possible.) Although I'm not in deep learning myself (I'm a computer vision researcher), here's a TL;DR as I understand it: rather than having people in specific domains such as computer vision or speech processing create their own features, the idea is to take raw inputs (pixels in the case of images) and train multi-layer neural net architectures that "learn" the relevant higher-level features in an unsupervised way (i.e., without labeled training data). Some of these seem to be pulling out interesting features and perform competitively on a few benchmarks in vision and other fields. I'm not sold on this yet, because it seems like the complexity of designing features has merely been traded in for the complexity of designing different learning architectures, but it's certainly becoming quite popular these days (mostly led by Geoff Hinton of Toronto, Yoshua Bengio of Montreal, and Yann LeCunn of NYU).
  - yid 16y ago
    Thanks for that, definitely some prominent folks involved.

jkan 15y ago

Well, but he's conditioning on "machine learning" already (the little caption on the histogram says "P(neural network | machine learning"), so that most likely filters out the straight neuroscience papers.