3 ms·

What's wrong with the name "loss"? I like the idea of calling it "kerfuffle", but loss seems like a neutral term to me. I agree with you about fucked-up shit.

What's wrong with the name "loss"? I like the idea of calling it "kerfuffle", but loss seems like a neutral term to me. I agree with you about fucked-up shit. I had a strong optimization background, and a traditional statistics background, and from that point everything you do with neural networks is just crazy.

sillysaurusx 6y ago
We have an opportunity here to define the terms that our descendants will be using 50 years from now. It won't come again. Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them. Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible. I would s/loss/penalty/ and s/learning rate/step size/, after giving it much thought. At least, I haven't thought of better names yet. The reason "step size" is important is because it represents what's actually going on. You don't increase the learning rate to make it learn faster. You increase the step size to make it take longer steps towards a goal. And when it's close to the goal, it circles around the goal, like water down a drain. You decrease the step size (learning rate) towards the end of training so that it doesn't keep dancing around the bowl, and can finally reach its target in the middle. Slight modifications like that can give lots of insights. For example, now that you're thinking of learning rate in terms of water spiraling down a drain, you can see why averaging the last N model checkpoints increases accuracy: if you're spinning in a circle around a target, then the average of your last 5 positions must bring you closer to the center. In fact, that's true of any convex shape. Therefore the loss landscape seems mostly convex. And so it goes. It's very much like compound interest. The more you understand, the more you can understand. That's why it's so important to be determined and patient. Also, ask lots of questions on Twitter. In my opinion it's one of the most crucial resources for learning ML. The ML community there is phenomenal, and I don't know why. All I know is that everyone is super friendly and eager to help you out. Start with @pbaylies, @jonathanfly, @aydaoai, and @arfafax.
- notretarded 6y ago
  It's not penalty or step size. It's loss as in amount of information lossd (not encoded in your network) compared to one perfectly encoding ground truth. Learning rate, as in what is the maximum amount of delta you are allowed to change your inputs to minimise your information loss analogous to how quickly you can possibly learn in one experiment.
  - sillysaurusx 6y ago
    Fair. I’ll keep that in mind. On the other hand, it went way over my head, and I’m not afraid to admit it. One of the nice things about ML (and math, for that matter) is that there are multiple mathematically equivalent ways of looking at a thing.
- rseed42 6y ago
  That's really funny, 50 years from now? Backpropagation is not a law of physics, you know. Thanks for making me laugh ;).

sillysaurusx 6y ago

We have an opportunity here to define the terms that our descendants will be using 50 years from now. It won't come again. Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them. Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible. I would s/loss/penalty/ and s/learning rate/step size/, after giving it much thought. At least, I haven't thought of better names yet. The reason "step size" is important is because it represents what's actually going on. You don't increase the learning rate to make it learn faster. You increase the step size to make it take longer steps towards a goal. And when it's close to the goal, it circles around the goal, like water down a drain. You decrease the step size (learning rate) towards the end of training so that it doesn't keep dancing around the bowl, and can finally reach its target in the middle. Slight modifications like that can give lots of insights. For example, now that you're thinking of learning rate in terms of water spiraling down a drain, you can see why averaging the last N model checkpoints increases accuracy: if you're spinning in a circle around a target, then the average of your last 5 positions must bring you closer to the center. In fact, that's true of any convex shape. Therefore the loss landscape seems mostly convex. And so it goes. It's very much like compound interest. The more you understand, the more you can understand. That's why it's so important to be determined and patient. Also, ask lots of questions on Twitter. In my opinion it's one of the most crucial resources for learning ML. The ML community there is phenomenal, and I don't know why. All I know is that everyone is super friendly and eager to help you out. Start with @pbaylies, @jonathanfly, @aydaoai, and @arfafax.

notretarded 6y ago
It's not penalty or step size. It's loss as in amount of information lossd (not encoded in your network) compared to one perfectly encoding ground truth. Learning rate, as in what is the maximum amount of delta you are allowed to change your inputs to minimise your information loss analogous to how quickly you can possibly learn in one experiment.
- sillysaurusx 6y ago
  Fair. I’ll keep that in mind. On the other hand, it went way over my head, and I’m not afraid to admit it. One of the nice things about ML (and math, for that matter) is that there are multiple mathematically equivalent ways of looking at a thing.
rseed42 6y ago
That's really funny, 50 years from now? Backpropagation is not a law of physics, you know. Thanks for making me laugh ;).