3 ms·

Extreme Learning Machine

50 points by sytelus 8y ago

stochastic_monk 8y ago
I really enjoyed the discussion here [0] on Yann LeCun’s comments. In some ways, it’s like random Fourier features/kitchen sinks (Cf. Ben Recht). They’re more insightful than they are useful. [0] https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/ https://www.reddit.com/r/MachineLearning/comments/34u0go/yan...
bigred100 8y ago
Is this an example of reservoir computing?
- T-A 8y ago
  The two first sentences of the respective Wikipedia article answer that: https://en.wikipedia.org/wiki/Extreme_learning_machine https://en.wikipedia.org/wiki/Extreme_learning_machine "Extreme learning machines are feedforward neural networks" https://en.wikipedia.org/wiki/Reservoir_computing https://en.wikipedia.org/wiki/Reservoir_computing "The reservoir consists of a collection of recurrently connected units" So, no.
  - bigred100 8y ago
    The ideas seem quite relate. A common reservoir computing setup involves learning a linear map from a back box dynamical system (which can be a feedforward network if we really want it to) to some output. The only significant distinction I see from a short observation is that the input and output size in reservoir models are the same from what I’ve seen
finndark 8y ago
"ELM" = "Extreme Learning Machine" is apparently heavily based upon previous work, if not outright plagiarism. The Wikipedia article on ELM discusses this only briefly: https://en.wikipedia.org/wiki/Extreme_learning_machine#Controversy https://en.wikipedia.org/wiki/Extreme_learning_machine#Contr... Yann LeCun' opinion on ELM: 'What's so great about "Extreme Learning Machines"?': https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/ https://www.reddit.com/r/MachineLearning/comments/34u0go/yan... A sort-of "How-to plagiarize successfully" guide including "Easy but Proven 5 Steps to Academic Fame" with Huang's work as a gleaming example: https://elmorigin.wixsite.com/originofelm https://elmorigin.wixsite.com/originofelm Two very detailed plagiarism analyses of two of Huang's papers. They are quite sarcastic, funny and enlightening: https://docs.wixstatic.com/ugd/5256f1_ac3a0f6b08524a3ca6101d4f3b74f54e.pdf https://docs.wixstatic.com/ugd/5256f1_ac3a0f6b08524a3ca6101d... And finally, as one poster says, the provenance of "Deep Learning" itself has some controversy: https://www.researchgate.net/post/Why_Extreme_Learning_machine_is_not_so_popular_as_Deep_Learning https://www.researchgate.net/post/Why_Extreme_Learning_machi... "But, just recently I also found that there is also a problem with the inventorship of Deep Learning. http://people.idsia.ch/~juergen/deep-learning-conspiracy.html http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm... For this, I post a question for discussing these issues. https://www.researchgate.net/post/Any_enlightment_on_the_origin_or_inventorships_of_ELM_and_Deep_Learning https://www.researchgate.net/post/Any_enlightment_on_the_ori... https://docs.wixstatic.com/ugd/5256f1_556b42c00d0d4d5bb199fb2e45def79f.pdf"* https://docs.wixstatic.com/ugd/5256f1_556b42c00d0d4d5bb199fb...
- screye 8y ago
  As much as people dislike Schumidhuber for being bit of a curmudgeon, his contributions have been undermined in the Deep Learning community at large. He did not deserve to be shafted for the Turing award. His contributions were as impressive as the other 3, and a lot of it was done in isolation away from the support of the NA AI/ML community.
lenticular 8y ago
ELMs are just neural nets with random hidden weights. This idea has been around since the 60's before being rebranded to get unoriginal papers published. This is equivalent to a low-rank Fourier approximation of Gaussian process regression (also known as random features) for some kernel. Random kitchen sink approximations are similar, but better performing since the weight value distribution actually has theoretical justification. RKS is a really great way to use approximate Gaussian process regression on larger datasets.

50 points by sytelus 8y ago

I really enjoyed the discussion here [0] on Yann LeCun’s comments. In some ways, it’s like random Fourier features/kitchen sinks (Cf. Ben Recht). They’re more insightful than they are useful. [0] https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/ https://www.reddit.com/r/MachineLearning/comments/34u0go/yan...

bigred100 8y ago

Is this an example of reservoir computing?

T-A 8y ago
The two first sentences of the respective Wikipedia article answer that: https://en.wikipedia.org/wiki/Extreme_learning_machine https://en.wikipedia.org/wiki/Extreme_learning_machine "Extreme learning machines are feedforward neural networks" https://en.wikipedia.org/wiki/Reservoir_computing https://en.wikipedia.org/wiki/Reservoir_computing "The reservoir consists of a collection of recurrently connected units" So, no.
- bigred100 8y ago
  The ideas seem quite relate. A common reservoir computing setup involves learning a linear map from a back box dynamical system (which can be a feedforward network if we really want it to) to some output. The only significant distinction I see from a short observation is that the input and output size in reservoir models are the same from what I’ve seen

finndark 8y ago

"ELM" = "Extreme Learning Machine" is apparently heavily based upon previous work, if not outright plagiarism. The Wikipedia article on ELM discusses this only briefly: https://en.wikipedia.org/wiki/Extreme_learning_machine#Controversy https://en.wikipedia.org/wiki/Extreme_learning_machine#Contr... Yann LeCun' opinion on ELM: 'What's so great about "Extreme Learning Machines"?': https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/ https://www.reddit.com/r/MachineLearning/comments/34u0go/yan... A sort-of "How-to plagiarize successfully" guide including "Easy but Proven 5 Steps to Academic Fame" with Huang's work as a gleaming example: https://elmorigin.wixsite.com/originofelm https://elmorigin.wixsite.com/originofelm Two very detailed plagiarism analyses of two of Huang's papers. They are quite sarcastic, funny and enlightening: https://docs.wixstatic.com/ugd/5256f1_ac3a0f6b08524a3ca6101d4f3b74f54e.pdf https://docs.wixstatic.com/ugd/5256f1_ac3a0f6b08524a3ca6101d... And finally, as one poster says, the provenance of "Deep Learning" itself has some controversy: https://www.researchgate.net/post/Why_Extreme_Learning_machine_is_not_so_popular_as_Deep_Learning https://www.researchgate.net/post/Why_Extreme_Learning_machi... "But, just recently I also found that there is also a problem with the inventorship of Deep Learning. http://people.idsia.ch/~juergen/deep-learning-conspiracy.html http://people.idsia.ch/~juergen/deep-learning-conspiracy.htm... For this, I post a question for discussing these issues. https://www.researchgate.net/post/Any_enlightment_on_the_origin_or_inventorships_of_ELM_and_Deep_Learning https://www.researchgate.net/post/Any_enlightment_on_the_ori... https://docs.wixstatic.com/ugd/5256f1_556b42c00d0d4d5bb199fb2e45def79f.pdf"* https://docs.wixstatic.com/ugd/5256f1_556b42c00d0d4d5bb199fb...

screye 8y ago
As much as people dislike Schumidhuber for being bit of a curmudgeon, his contributions have been undermined in the Deep Learning community at large. He did not deserve to be shafted for the Turing award. His contributions were as impressive as the other 3, and a lot of it was done in isolation away from the support of the NA AI/ML community.

lenticular 8y ago

ELMs are just neural nets with random hidden weights. This idea has been around since the 60's before being rebranded to get unoriginal papers published. This is equivalent to a low-rank Fourier approximation of Gaussian process regression (also known as random features) for some kernel. Random kitchen sink approximations are similar, but better performing since the weight value distribution actually has theoretical justification. RKS is a really great way to use approximate Gaussian process regression on larger datasets.