4 ms·

A big problem with scoring systems is that it incentivises risk taking. If you enter one of the stock picking competitions, for example, you rational choice is

by xorcist 12y ago

A big problem with scoring systems is that it incentivises risk taking. If you enter one of the stock picking competitions, for example, you rational choice is to take extreme risks. You probably won't pick a winning stock, but if you do you're likely to have a good chance at winning the competition. This is the opposite of what an investor wants to do with his own money, which is to manage the risk taking. This is a common problem in designing scoring systems for measuring performance, even after the more ovious problems with natural variations.

learnstats2 12y ago
That's true of stock picking competitions/tournament-style gambling, yes, but it wouldn't apply here. If I'm a surgeon with a good score, I make bad judgements and they start coming off poorly, I won't have a good score any more. Am I motivated to take a bad risk? Not unless I have some other reason to.
- xorcist 12y ago
  It's not directly applicable because it is a different problem domain, but the general reasoning holds. The problem is that the optimum strategy for maximizing your score is not to be as good a surgeon as possible. (Which I stupidly say without having the slighest idea how this works beyond what is described in the article. But I have yet to see a scoring system which aligns where the optimum strategy aligns perfectly with the desired outcome, and this holds for everything from grades in school to karma systems.)
  - learnstats2 12y ago
    No: the general reasoning doesn't hold. The reasoning for tournaments is that you have to be first at all cost, so you should consider taking the largest risk available to avoid coming second. That reasoning doesn't hold for surgeons: there's room for more than one surgeon. There are problems with scoring systems, yes, but a complete lack of information for consumers is considered terrible in any other industry.

by xorcist 12y ago

learnstats2 12y ago

That's true of stock picking competitions/tournament-style gambling, yes, but it wouldn't apply here. If I'm a surgeon with a good score, I make bad judgements and they start coming off poorly, I won't have a good score any more. Am I motivated to take a bad risk? Not unless I have some other reason to.

xorcist 12y ago
It's not directly applicable because it is a different problem domain, but the general reasoning holds. The problem is that the optimum strategy for maximizing your score is not to be as good a surgeon as possible. (Which I stupidly say without having the slighest idea how this works beyond what is described in the article. But I have yet to see a scoring system which aligns where the optimum strategy aligns perfectly with the desired outcome, and this holds for everything from grades in school to karma systems.)
- learnstats2 12y ago
  No: the general reasoning doesn't hold. The reasoning for tournaments is that you have to be first at all cost, so you should consider taking the largest risk available to avoid coming second. That reasoning doesn't hold for surgeons: there's room for more than one surgeon. There are problems with scoring systems, yes, but a complete lack of information for consumers is considered terrible in any other industry.