4 ms·

Not citing prior work is a grave issue in the academic world. Papers can get accepted/rejected based on their novelty status. Missing citations of central topic

by fnl 10y ago

Not citing prior work is a grave issue in the academic world. Papers can get accepted/rejected based on their novelty status. Missing citations of central topics of a work can therefore lead to retractions. I am not an expert on the issue in this particular case, but if Schmidhuber is right, from an academic perspective, he has all the right to be pretty pissed off. I certainly would recommend to reject a paper if the author were to refuse to cite prior art.

visarga 10y ago
In that case the technique was similar but the way it was used (purpose) was different. > The new approach seems similar in many ways. Both approaches use "adversarial" MLPs to estimate certain probabilities and to learn to encode distributions. A difference is that the new system learns to generate a non-trivial distribution in response to statistically independent, random inputs, while good old PM learns to generate statistically independent, random outputs in response to a non-trivial distribution (by extracting mutually independent, factorial features encoding the distribution). Hence the new system essentially inverts the direction of PM - is this the main difference? Should it perhaps be called "inverse PM"? http://media.nips.cc/nipsbooks/nipspapers/paper_files/nips27/reviews/1384.html http://media.nips.cc/nipsbooks/nipspapers/paper_files/nips27... And this is from Ian Goodfellow himself, giving 3 counter arguments: > Some previous work has used the general concept of having two neural networks compete. The most relevant work is predictability minimization [Schmidhuber, J. 1992]. In predictability minimization, each hidden unit in a neural network is trained to be different from the output of a second network, which predicts the value of that hidden unit given the value of all of the other hidden units. This work differs from predictability minimization in three important ways: 1) in this work, the competition between the networks is the sole training criterion, and is sufficient on its own to train the network. Predictability minimization is only a regularizer that encourages the hidden units of a neural network to be statistically independent while they accomplish some other task; it is not a primary training criterion.2) The nature of the competition is different. In predictability minimization, two networks’ outputs are compared, with one network trying to make the outputs similar and the other trying to make the outputs different. The output in question is a single scalar. In GANs, one network produces a rich, high dimensional vector that is used as the input to another network, and attempts to choose an input that the other network does not know how to process. 3) The specification of the learning process is different. Predictability minimization is described as an optimization problem with an objective function to be minimized, and learning approaches the minimum of the objective function. GANs are based on a minimax game rather than an optimization problem, and have a value function that one agent seeks to maximize and the other seeks to minimize. The game terminates at a saddle point that is a minimum with respect to one player’s strategy and a maximum with respect to the other player’s strategy. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf http://papers.nips.cc/paper/5423-generative-adversarial-nets...
- fnl 10y ago
  Thanks for the details, very enlightening! But its a bit one-sided, still: What are Schmidhuber's arguments on this issue?
deepnotderp 10y ago
This had been twisted by the nytimes, Bloomberg and everyone else. The majority of the ml community agrees that schmidhuber is in the wrong here. And this isn't group think either, he interrupted a tutorial presentation to argue about something that the presenter (Goodfellow) had already discussed with schmidhuber.

by fnl 10y ago

visarga 10y ago

In that case the technique was similar but the way it was used (purpose) was different. > The new approach seems similar in many ways. Both approaches use "adversarial" MLPs to estimate certain probabilities and to learn to encode distributions. A difference is that the new system learns to generate a non-trivial distribution in response to statistically independent, random inputs, while good old PM learns to generate statistically independent, random outputs in response to a non-trivial distribution (by extracting mutually independent, factorial features encoding the distribution). Hence the new system essentially inverts the direction of PM - is this the main difference? Should it perhaps be called "inverse PM"? http://media.nips.cc/nipsbooks/nipspapers/paper_files/nips27/reviews/1384.html http://media.nips.cc/nipsbooks/nipspapers/paper_files/nips27... And this is from Ian Goodfellow himself, giving 3 counter arguments: > Some previous work has used the general concept of having two neural networks compete. The most relevant work is predictability minimization [Schmidhuber, J. 1992]. In predictability minimization, each hidden unit in a neural network is trained to be different from the output of a second network, which predicts the value of that hidden unit given the value of all of the other hidden units. This work differs from predictability minimization in three important ways: 1) in this work, the competition between the networks is the sole training criterion, and is sufficient on its own to train the network. Predictability minimization is only a regularizer that encourages the hidden units of a neural network to be statistically independent while they accomplish some other task; it is not a primary training criterion.2) The nature of the competition is different. In predictability minimization, two networks’ outputs are compared, with one network trying to make the outputs similar and the other trying to make the outputs different. The output in question is a single scalar. In GANs, one network produces a rich, high dimensional vector that is used as the input to another network, and attempts to choose an input that the other network does not know how to process. 3) The specification of the learning process is different. Predictability minimization is described as an optimization problem with an objective function to be minimized, and learning approaches the minimum of the objective function. GANs are based on a minimax game rather than an optimization problem, and have a value function that one agent seeks to maximize and the other seeks to minimize. The game terminates at a saddle point that is a minimum with respect to one player’s strategy and a maximum with respect to the other player’s strategy. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf http://papers.nips.cc/paper/5423-generative-adversarial-nets...

fnl 10y ago
Thanks for the details, very enlightening! But its a bit one-sided, still: What are Schmidhuber's arguments on this issue?

deepnotderp 10y ago

This had been twisted by the nytimes, Bloomberg and everyone else. The majority of the ml community agrees that schmidhuber is in the wrong here. And this isn't group think either, he interrupted a tutorial presentation to argue about something that the presenter (Goodfellow) had already discussed with schmidhuber.