4 ms·

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outc

by imajes 15y ago

what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regularly end up in situations where the human component of a transaction may look different than the data, and react accordingly...

davidw 15y ago
> It just requires better educators to explain it, with analogies and metaphor. So... anyone want to take a stab at explaining that equation to those of us who don't really get it?
- dpritchett 15y ago
  If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom. [1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
  - davidw 15y ago
    That much was fairly clear on the site. I'm talking about the math itself.
    - dxbydt 15y ago
      With zero knowledge of statistics, here's how to think about it - Wilson wants to construct some score. That score can be as low as 0 and as high as 1. So he wants some interval [x,y]. The center of that interval would obviously be c = (x+y)/2. Wilson first decides what c to pick. So Wilson says, the center c must be decided by the proportion of upvotes. But then he thinks, c must also be close to a half. So he says, okay, lets figure out c using some sort of an average. So he chooses two weights a and b. The weighted average would then be a times half plus b times the upvote proportion. He chooses those weights in such a fashion that if you have lots of data, one of the weights vanish & the other becomes unity. So the weights matter only if you have too few upvotes & downvotes. Having figured out the midpoint c, Wilson has to actually figure out the lower bound x and upper bound y. Now he draws a distribution centered at c....ok so at this point you would need to know what is a distribution, and why you would need one, whether that distribution has a skew & whether its homoskedastic & so on...which is stats 101, so I won't go there. But if you've gotten this far, you should be able to atleast see the intuition behind Wilson's procedure.
    - dpritchett 15y ago
      Wilson's 1927 paper is freely available [1]. I can't say I have brushed up on my statistics enough this decade to verify the math but the basics as I read them are as follows: - You have a normal distribution (bell curve) of data points, in this case quality scores. - You wish to sort these points based on their respective vote totals. Any given data point has pos positive votes out of n total votes for that item. - You have a confidence interval, e.g. 95%. This confidence is expressed in terms of the bell curve, so a 95% confidence is within 1.96 standard deviations of the mean [2]. - You have a Ruby function accepting the aforementioned n, pos, and confidence variables and returning a decimal value representing the normalized confidence_interval_lower_bound, that is the quality score that our input data point has a 95% chance of meeting or exceeding. - Given a set of data points, evaluate the ci_lower_bound for each, and then sort them accordingly. The results will give you a best-guess sorting that accounts for the fact that some data points will have more votes cast for/against them than others. [1] http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/wilson_jasa_1927.pdf http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/... [2] http://en.wikipedia.org/wiki/1.96 http://en.wikipedia.org/wiki/1.96
- ealloc 15y ago
  Say the 'acutal' rating for an item is p. (ie, if you got an infinite number of people voting, the ratio of upvotes to total votes is p). Now say your users vote, and they upvote with that probability p. The number of upvotes k you get out of n votes will follow a binomial distribution B(k;n,p). The binomial distribution has mean np and stdev sqrt(n p(1-p)), and is very close to gaussian in shape. Since the stdev is a rough measure of the 'width' of the distribution, common way to describe the error is (mean +/- lambda*stdev), where you can tune lambda to your desire. If you increase lambda you get a wider confidence interval, and therefore more certainty that a measurement will be within that confidence interval. Now, say you measure k upvotes out of n votes. You can divide by n to get p0, your estimated rating based on those votes. An easy estimate for the error of this measurement is to assume that p0 is approximately correct and equal to p. Then the expected number of upvotes would be n p0 with stdev(n p0(1-p0)). Divide this by n to get the fraction of upvotes, to give a final estimate for p of p0 +- lambda sqrt(p0(1-p0)/n) Now, your estimate (p0) of p is not quite right, and therefore your estimate of the error (which depends on p0) is not quite right either. The wilson score attempts to correct for that. We don't know p, but imagine if we did, we would expect any measurement p0 to be in the range p +- lambda sqrt(p(1-p)/n). That is, we expect abs(p - p0) < lambda sqrt(p (1-p)/n) If you solve this equation for p in terms of p0, you get a formula given in the article, ie, the confidence limits for p given p0.
edw519 15y ago
There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the problem? Actually, simplying the problem actually reduces the compromise when humans are involved. So, no. ...wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Only if it made economic sense to do so. I am not going to replace 800 workers earning $12/hour with "the best people who understand the domain" because the computer suddenly has formulas that people don't understand. Experience has shown repeatedly that workers, at any level, simply stop caring when they feel powerless by "solutions" like OP's Wilson formula. That abducation of responsibility almost always far outweighs any incremental benefit that a more sophisticated but incomprehensible formula introduces. Good points, nice discussion, but please, for the sake of this community, don't reply with words like "what?", "Further:", or "Hint:". You made your point without the snarkiness.
- adamio 15y ago
  Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them. Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job. You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You replace your inventory manager with someone who understands the math (if you want to manage your inventory that way)
  - brazzy 15y ago
    Sufficiently advanced sophistication is indistinguishable from obfuscation. And obfuscation is a problem that's not ideally solved by hiring smarter people. I think that's the point edw519 was trying to make.
- imajes 15y ago
deleted 15y ago
[deleted]
IanDrake 15y ago
I thought he was talking about the end user. They may not understand why something with 209 thumbs up and 100 thumbs down ranks better than one with 5 thumbs up (100% positive).

by imajes 15y ago

davidw 15y ago

> It just requires better educators to explain it, with analogies and metaphor. So... anyone want to take a stab at explaining that equation to those of us who don't really get it?

dpritchett 15y ago
If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom. [1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
- davidw 15y ago
  That much was fairly clear on the site. I'm talking about the math itself.
  - dxbydt 15y ago
    With zero knowledge of statistics, here's how to think about it - Wilson wants to construct some score. That score can be as low as 0 and as high as 1. So he wants some interval [x,y]. The center of that interval would obviously be c = (x+y)/2. Wilson first decides what c to pick. So Wilson says, the center c must be decided by the proportion of upvotes. But then he thinks, c must also be close to a half. So he says, okay, lets figure out c using some sort of an average. So he chooses two weights a and b. The weighted average would then be a times half plus b times the upvote proportion. He chooses those weights in such a fashion that if you have lots of data, one of the weights vanish & the other becomes unity. So the weights matter only if you have too few upvotes & downvotes. Having figured out the midpoint c, Wilson has to actually figure out the lower bound x and upper bound y. Now he draws a distribution centered at c....ok so at this point you would need to know what is a distribution, and why you would need one, whether that distribution has a skew & whether its homoskedastic & so on...which is stats 101, so I won't go there. But if you've gotten this far, you should be able to atleast see the intuition behind Wilson's procedure.
  - dpritchett 15y ago
    Wilson's 1927 paper is freely available [1]. I can't say I have brushed up on my statistics enough this decade to verify the math but the basics as I read them are as follows: - You have a normal distribution (bell curve) of data points, in this case quality scores. - You wish to sort these points based on their respective vote totals. Any given data point has pos positive votes out of n total votes for that item. - You have a confidence interval, e.g. 95%. This confidence is expressed in terms of the bell curve, so a 95% confidence is within 1.96 standard deviations of the mean [2]. - You have a Ruby function accepting the aforementioned n, pos, and confidence variables and returning a decimal value representing the normalized confidence_interval_lower_bound, that is the quality score that our input data point has a 95% chance of meeting or exceeding. - Given a set of data points, evaluate the ci_lower_bound for each, and then sort them accordingly. The results will give you a best-guess sorting that accounts for the fact that some data points will have more votes cast for/against them than others. [1] http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/wilson_jasa_1927.pdf http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/... [2] http://en.wikipedia.org/wiki/1.96 http://en.wikipedia.org/wiki/1.96
ealloc 15y ago
Say the 'acutal' rating for an item is p. (ie, if you got an infinite number of people voting, the ratio of upvotes to total votes is p). Now say your users vote, and they upvote with that probability p. The number of upvotes k you get out of n votes will follow a binomial distribution B(k;n,p). The binomial distribution has mean np and stdev sqrt(n p(1-p)), and is very close to gaussian in shape. Since the stdev is a rough measure of the 'width' of the distribution, common way to describe the error is (mean +/- lambda*stdev), where you can tune lambda to your desire. If you increase lambda you get a wider confidence interval, and therefore more certainty that a measurement will be within that confidence interval. Now, say you measure k upvotes out of n votes. You can divide by n to get p0, your estimated rating based on those votes. An easy estimate for the error of this measurement is to assume that p0 is approximately correct and equal to p. Then the expected number of upvotes would be n p0 with stdev(n p0(1-p0)). Divide this by n to get the fraction of upvotes, to give a final estimate for p of p0 +- lambda sqrt(p0(1-p0)/n) Now, your estimate (p0) of p is not quite right, and therefore your estimate of the error (which depends on p0) is not quite right either. The wilson score attempts to correct for that. We don't know p, but imagine if we did, we would expect any measurement p0 to be in the range p +- lambda sqrt(p(1-p)/n). That is, we expect abs(p - p0) < lambda sqrt(p (1-p)/n) If you solve this equation for p in terms of p0, you get a formula given in the article, ie, the confidence limits for p given p0.

edw519 15y ago

There's nothing wrong with the math. I never said there was. In fact, I praised it as an elegant solution. It just requires better educators to explain it, with analogies and metaphor. You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000. Why would you compromise the data/outcome in a hope to simplify the problem? Actually, simplying the problem actually reduces the compromise when humans are involved. So, no. ...wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Only if it made economic sense to do so. I am not going to replace 800 workers earning $12/hour with "the best people who understand the domain" because the computer suddenly has formulas that people don't understand. Experience has shown repeatedly that workers, at any level, simply stop caring when they feel powerless by "solutions" like OP's Wilson formula. That abducation of responsibility almost always far outweighs any incremental benefit that a more sophisticated but incomprehensible formula introduces. Good points, nice discussion, but please, for the sake of this community, don't reply with words like "what?", "Further:", or "Hint:". You made your point without the snarkiness.

adamio 15y ago
Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them. Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job. You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You replace your inventory manager with someone who understands the math (if you want to manage your inventory that way)
- brazzy 15y ago
  Sufficiently advanced sophistication is indistinguishable from obfuscation. And obfuscation is a problem that's not ideally solved by hiring smarter people. I think that's the point edw519 was trying to make.
imajes 15y ago

deleted 15y ago

[deleted]

IanDrake 15y ago

I thought he was talking about the end user. They may not understand why something with 209 thumbs up and 100 thumbs down ranks better than one with 5 thumbs up (100% positive).