5 ms·
I find the nomenclature in this article a bit weird. > Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss
by cl3misch 6y ago
I find the nomenclature in this article a bit weird.
> Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss function. Mathematically, the goal in training a model is converging on the global minimum, the point in the loss function where the model has optimized its ability to make predictions.
"Backpropagation" is the method how to compute the gradient of the weights with respect to a loss function. But the article repeatedly uses the term as if it was the whole optimization algorithm, running into local minima.
- savant_penguin 6y agoSome people use backpropagation and gradient decent interchangeably. It's really confusing though
- jhrmnn 6y agoI wouldn’t say it’s confusing—it’s wrong and suggests at the least sloppiness in terminology
- ethbr0 6y ago> sloppiness in terminology venturebeat.com Not the worst, but we're not talking Nature or Spectrum here.
- option 6y agoof course it is wrong. One nice thing about math is that thing are defined precisely and back propagation and, say, SGD or Adam are different things
- dnautics 6y agoThat's bad. There are gradient descent techniques that are feed-forward, and understand the domains where those are appropriate and the domains where backprop is appropriate is I think important, especially as you try to do things like mix machine learning techniques with other differentiable programming strategies.
- joe_the_user 6y agoWikipedia: "The term backpropagation strictly refers only to the algorithm for computing the gradient, not how the gradient is used; however, the term is often used loosely to refer to the entire learning algorithm, including how the gradient is used, such as by stochastic gradient descent." -- Meaning follows usage. https://en.wikipedia.org/wiki/Backpropagation https://en.wikipedia.org/wiki/Backpropagation
- enriquto 6y agoThis is wrong, stupid and extremely confusing.
- edoceo 6y agoCould you provide a definition correct, smart and sorta helpful?
- atty 6y agoAs someone who uses this in their day job, I have no problem using loose terminology to describe a well known procedure. Most of the time when I am referring to the act of optimization, it doesn’t matter exactly what method I am using, and I can use backprop as a stand in. If I’m talking about the technical details of my work, I will state the specific optimization strategy. Everyone on my team does similar things, and no one is confused or misled. Use rigorous language when necessary, and use colloquial language when appropriate.
- enriquto 6y ago> Use rigorous language when necessary, and use colloquial language when appropriate. Do you think a peer-reviewd publication is formal enough to warrant precise language? These publications are not only read by specialists in the field. I use automatic differentiation in my daily work, but I'm not familiar with machine learning. Thus I am very confused when "backpropagation" is used to mean an optimization algorithm. EDIT: It is as if physicists used the term "special relativity" to talk about "quantum mechanics" because, after all, quantum mechanics happens in Lorentzian spacetime. Now for specialists of quantum physics it may make sense, since they are using "special relativity" to distinguish it from fancier quantum theories that combine field theory with GR. But for normal people it would be certainly misleading. Using "backpropagation" to include optimization has the same feeling.
- bonoboTP 6y agoDeep learning terminology is often loose like that, see "softmax loss" or "deconvolution layer" etc.