4 ms·

> Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Penalty already has a meaning in machine learning so this substitutio

> Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Penalty already has a meaning in machine learning so this substitution just adds more confusion instead clarifying things. Loss seems descriptive enough to me.

sillysaurusx 6y ago
Perhaps, but regularization is a better name for that term anyway. How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe. The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it. Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain kinds of games. Minimizing penalty, on the other hand, is perfectly clear. If you want the neural network to do something less, add a penalty term.
- superbcarrot 6y ago
  So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just important to keep in mind that the rest of the world isn't on board.
  - sillysaurusx 6y ago
    Nah, I’ll bend the world to my way of doing things. It’s better. Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary. But the thing is, it’s extremely easy to remember to say “loss” instead of “penalty” when I’m talking to someone. But it was extremely hard for me to even understand what the heck a loss was. What is it, exactly? What’s it doing and why? How should I think about it — and more importantly, how can I extrapolate that thinking to take advantage of it? Maybe it’s a personal quirk, but I simply couldn’t understand loss. I know penalty though. Ditto tor learning rate vs step size. So it’s more of “internal advice” rather than me saying that you should rewrite your papers with the new names. EDIT: By the way, I wasn't proposing that "penalty be renamed to "regularization". I was under the impression that what the parent comment was calling penalty" was normally called "regularization", i.e. that regularization was the formal name for it. If that's not true, it's possible my understanding is incomplete -- what is penalty? I haven't heard of it till now, to be honest. And googling for "machine learning penalty" pops up 5 articles on regularization. So I was proposing two changes: loss -> penalty, and learning rate -> step size.
    - jmmcd 6y ago
      Yes, I think you're confused about regularisation. A regularisation is (usually) a component in the overall loss, which has the goal of simplifying or preventing overfitting, as opposed to the main component which has the goal of fitting. It's not another term or a formal term for penalty.
      - sillysaurusx 6y ago
        Thanks for explaining that. It seems that penalty is already used, which is unfortunate. One of the interesting things about ML is that you can learn for a year and a half and still uncover more things you didn't know, which I love. I guess I'll call loss "punishment." It matches how it feels to make progress in ML anyway.
  - tpoacher 6y ago
    Just because terms become established doesn't mean the way they got about becoming established was through clarity and careful deliberation. In fact I'd go as far as saying that more than half of the terms/notation in such fields sound like they were created as silly placeholder names which then stuck. So much so that we need a translation of their actual meaning each time they're used. Even something as basic as p(x).
    - pugio 6y ago
      I still feel mental friction when contemplating anything to do with "regression" because the word doesn't seem to capture what the technique(s) (e.g. linear, logistic) actually do. I have looked into the historical context and reason for the use of the word (the technique was first popularized in something which "regressed to the mean"), as well as its development, and it still bugs me any time.
jl2718 6y ago
“error”? “resisidual”? “objective”? “cost”? “Penalty function” is commonly used in operations research/optimization, and the parameter correction, the “penalty”. There are some conceptual differences with RL as “penalty” is more like a data input, but I think that should be a participle like “punishment” because it implies action by the trainer.

sillysaurusx 6y ago

Perhaps, but regularization is a better name for that term anyway. How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe. The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it. Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain kinds of games. Minimizing penalty, on the other hand, is perfectly clear. If you want the neural network to do something less, add a penalty term.

superbcarrot 6y ago
So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just important to keep in mind that the rest of the world isn't on board.
- sillysaurusx 6y ago
  Nah, I’ll bend the world to my way of doing things. It’s better. Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary. But the thing is, it’s extremely easy to remember to say “loss” instead of “penalty” when I’m talking to someone. But it was extremely hard for me to even understand what the heck a loss was. What is it, exactly? What’s it doing and why? How should I think about it — and more importantly, how can I extrapolate that thinking to take advantage of it? Maybe it’s a personal quirk, but I simply couldn’t understand loss. I know penalty though. Ditto tor learning rate vs step size. So it’s more of “internal advice” rather than me saying that you should rewrite your papers with the new names. EDIT: By the way, I wasn't proposing that "penalty be renamed to "regularization". I was under the impression that what the parent comment was calling penalty" was normally called "regularization", i.e. that regularization was the formal name for it. If that's not true, it's possible my understanding is incomplete -- what is penalty? I haven't heard of it till now, to be honest. And googling for "machine learning penalty" pops up 5 articles on regularization. So I was proposing two changes: loss -> penalty, and learning rate -> step size.
  - jmmcd 6y ago
    Yes, I think you're confused about regularisation. A regularisation is (usually) a component in the overall loss, which has the goal of simplifying or preventing overfitting, as opposed to the main component which has the goal of fitting. It's not another term or a formal term for penalty.
    - sillysaurusx 6y ago
      Thanks for explaining that. It seems that penalty is already used, which is unfortunate. One of the interesting things about ML is that you can learn for a year and a half and still uncover more things you didn't know, which I love. I guess I'll call loss "punishment." It matches how it feels to make progress in ML anyway.
- tpoacher 6y ago
  Just because terms become established doesn't mean the way they got about becoming established was through clarity and careful deliberation. In fact I'd go as far as saying that more than half of the terms/notation in such fields sound like they were created as silly placeholder names which then stuck. So much so that we need a translation of their actual meaning each time they're used. Even something as basic as p(x).
  - pugio 6y ago
    I still feel mental friction when contemplating anything to do with "regression" because the word doesn't seem to capture what the technique(s) (e.g. linear, logistic) actually do. I have looked into the historical context and reason for the use of the word (the technique was first popularized in something which "regressed to the mean"), as well as its development, and it still bugs me any time.

jl2718 6y ago

“error”? “resisidual”? “objective”? “cost”? “Penalty function” is commonly used in operations research/optimization, and the parameter correction, the “penalty”. There are some conceptual differences with RL as “penalty” is more like a data input, but I think that should be a participle like “punishment” because it implies action by the trainer.