7 ms·

I love it and I hate it. Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application f

• tel 15y ago
  I hate it exclusively. It's clearly better than (1) and (2), but... I talk about this with some of my med school friends that are interested in/want to create/hate/fear automated diagnosis. At one level, having appropriate statistics to make use of a wealth of prior experience, worldwide prevalence, epidemiological data, &c is basically a requirement for the future of proper healthcare. It's also an obvious terrible mistake to ever convince people that these statistics are anything besides a decision making tool. I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route. For this reason, I think proper statistical application is an intersection between mathematics, computer science, and UI design.
  • raldi 15y ago
    No matter how much statistical data you reveal, you still need to present the results in some particular order. And this formula is the best ordering function I know of for things that are rated.
    • tel 15y ago
      I think the situation is much more informatively modeled by a partial order though. You can induce a total order on a partial order and you can use this formula to do so conservatively, but maybe the real way to solve the problem is to create an intuitive way for users to appreciate partial orders.
  • radarsat1 15y ago
    > I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route. You're right, but I would go further. I would argue that when statistics are presented as "the answer," in many cases they are being abused. A good scientist recognizes the limits of his dataset. Like how a computer program can only do what it is written to do, statistics can only tell you what is in the data, they can't tell you what is _not_.
• m_for_monkey 15y ago
  You have overlooked a critical point: no one has to mess with this (otherwise quite simple) formula anymore, except the programmer, and even he already got a working piece of Ruby code.
• imajes 15y ago
  what? There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem? Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not? Hint: insurance/loan underwriters regularly end up in situations where the human component of a transaction may look different than the data, and react accordingly...
  • davidw 15y ago
    > It just requires better educators to explain it, with analogies and metaphor. So... anyone want to take a stab at explaining that equation to those of us who don't really get it?
    • dpritchett 15y ago
      If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom. [1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-system.html http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
      • davidw 15y ago
        That much was fairly clear on the site. I'm talking about the math itself.
        • 15y ago
• brown9-2 15y ago
  And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why." And somehow the solution isn't to have the computer output a detailed description explaining why it arrived at the decision that it did?
• raldi 15y ago
  I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.
  • bri3d 15y ago
    Amusingly the proposed solution in this blog post is almost line-for-line exactly the method Reddit uses to rank comments - proof, I think, that the method isn't "too complicated." When you're working with a system managing orders, where there's a direct human interaction with the algorithm (like the top-level poster is), simplicity is somewhat important. When you're working with a system that's designed to look like magic to the end user, making the system more magical is often better, because it makes it less intuitive to game.
  • trebor 15y ago
    The math doesn't make the comments good. The math makes good comments easier to find, and buries comments with inadequate favor or confidence. Besides, the math made no effort to write the sentences. ;)
  • true_religion 15y ago
    I think you're overlooking a critical component of the Google equation: the supply. Webmasters * constantly* complain about the impenetrability of the 'algorithm' and how it constantly changes for secret reasons. So you're right in a way---the demand side doesn't care how you manage to curate supply, but supply is intimately aware of your curation and worries about being unfairly demerited.
  • dailyllama 15y ago
    I used the wilson score method to generate star ratings off of individual user up/down reviews on a commerce site thirteen years ago. You can do something very similar with plaintext reviews by doing sentiment analysis to mark them as a yay, a nay, or a no opinion, and generate a similar score. There are other, more complicated, ways of doing things, but this is fine for somebody who only has a modest knowledge of stat.
• tripzilch 15y ago
  Math is hard, let's go shopping! If you think this is too difficult then maybe you shouldn't try to design scoring/voting systems.
• NathanRice 15y ago
  Operations research and hard systems modeling are neat. The only gripe I have is that often times the models make simplifying assumptions which hold tenuously at best. One example of this is the Gaussian copula used in VAR. In actuality, neither the marginals nor the copula are Gaussian in the large majority of cases; they ARE stable distributions, but the tails can be absolutely HUGE. The Gaussian distribution is attractive because it is well behaved, and in most cases we don't have enough observations to properly understand the full tail structure of the underlying distribution. As a result, you get quants that make a model and call a small subset of points (which are completely valid) "outliers". Then when we observe the tails of the distribution, things fall apart because over-leveraged bankers made decisions based on the notion that the model accurately reflected reality.
• Shorel 15y ago
  Put this code in a class or a library and let a smarter programmer maintain it.
• JumpCrisscross 15y ago
  »The computer decided not to and I have no idea why This just means you need a better notification system, not a dumber model. I am fairly sure Netflix and Pandora use complicated recommendation algorithms - we don't complain because the complexity is abstracted away behind labels like "dark cerebral thrillers" or "songs with rhythmic gyrations like Pretty Lights".
• efsavage 15y ago
  Right, the solution here is that if you're going to sort by Bayesian average or Wilson score confidence interval or something besides what we're familiar with, reframe it to avoid my "something's wrong" detector. Hence the fact that people accept "trending" or "popular" but if I sort by "rating" and the results don't make sense because I expected an average, I don't think "I should probably understand this algorithm". I think "this site is broken and dumb".
• jlarocco 15y ago
  Meh. "Why did you return the list of reviews in this order?" - "Here's the link to Wikipedia explaining it". The formulas aren't "too complex to understand", the people are being too lazy to take the time and understand them. No reason to dumb it down and ruin it for the rest of us.
• dxbydt 15y ago
  >What human can understand it? Lets start with Wilson's midpoint, since that's just high school math. def mid(upvotes:Int, downvotes:Int) = { val total = upvotes+downvotes+0.0 val up = upvotes/total val half = 0.5 val a = total/(4+total) val b = 4/(4+total) a * up + b * half } So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes. It should be very clear that if the total becomes large, a goes to 1 and b to zero. At that point you end up using the proportion of upvotes, just like Amazon. Now, lets bring in the entire confidence interval. def wilson(upvotes:Int, downvotes:Int) = { val z = 1.96 val n = upvotes+downvotes+0.0d val phat = upvotes/n val lower = (phat + z*z/(2*n) - z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n) val upper = (phat + z*z/(2*n) + z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n) (lower,upper) } Using the author's data points, the results look like this: val itemVotes = List((600,400),(5500,4500),(2,0), (100,1)) itemVotes.foreach( x => { val s = mid(x._1,x._2) val w = wilson(x._1,x._2) printf("Up:%5d\tDown:%5d\tMid:%.3f\tW:[%.3f,%.3f]\n", x._1, x._2,s,w._1,w._2) }) scala> Up: 600 Down: 400 Mid:0.600 W:[0.569,0.630] Up: 5500 Down: 4500 Mid:0.550 W:[0.540,0.560] Up: 2 Down: 0 Mid:0.667 W:[0.342,1.000] Up: 100 Down: 1 Mid:0.971 W:[0.946,0.998] Basically, we prefer the lower bound of the confidence interval instead of the midpoint of that same interval.
  • DanI-S 15y ago
    I think the original post still stands - try explaining the above to an irate customer.
    • 27182818284 15y ago
      I've dealt with people who have wanted a retraction of negative comments to their business on the web, I explained it didn't work that way as best I could, but there isn't much you can do. Same with this, if they are irate about a low rating, you play nice, but you don't expect them to learn the ins and outs.
      • 15y ago
• Eliezer 15y ago
  It's neither precise, nor elegant, and certainly not rigorous. On this approach, an item with 1 upvote and 2 downvotes will be ranked below an item with 1000 upvotes and 2000 downvotes, as one commenter pointed out; and in general, bright new items will almost never be presented to anyone. Search on "Bayesian" in the comments for the precise, elegant, rigorous solution below. I won't claim that it's easy to understand, but it's a lot easier to understand than the frequentist ad-hoc version.
• gus_massa 15y ago
  To get a general idea, you can compare this formula with the simple version, that is * mu - 3 sigma* (be careful when the number of samples is small), where mu is the average and sigma is the standard variation, and 3 is a parameter that depends on the confidence (~93%). When the number of samples (n) is big, then the two formulas give similar results.
• jfarmer 15y ago
  The way I think about it is this: comprehensibility is a feature. You have to weigh it against accuracy, effectiveness, etc., but it should be taken into account.
• rwallace 15y ago
  > "Why didn't we place an order?" "Because the computer decided not to and I have no idea why." This may or may not be a problem. Remember, human intuition evolved to be able to give a fast answer to any question in the face of insufficient data, not an optimal answer to specific questions with sufficient data. In many domains, once you have appropriate data and the means to statistically analyze it, an algorithm using such analysis can reliably outperform human experts - and in some cases, the problem was the political one of how to get the human experts to accept this, take their hands off the wheel and stop second-guessing the algorithm. In other words, if you are dealing with such a domain, you may be better off to count your blessings, let the computer do its job and save the humans for tasks that can't be handled by a formula.