13 ms·
Reddit’s empire is founded on a flawed algorithm
- deleted 13y ago[deleted]
- fleitz 13y agoIt's theoretically broken, in practice it works quite well.
- maxerickson 13y agoIs it even true that it is theoretically broken? The blog sketches out a corner case that maybe isn't handled well, but posts with net negative votes probably aren't "hot", and I'm pretty sure they have mechanisms in there to make sure that bad voters are at least eventually ignored.
- DrStalker 13y agoI don;t think anyone really cares how submissions are ranked once they are obviously net negative. The concern is for the first few votes; a submissions should not be effectively discarded because the first person to look at it did not like it.
- Grue3 13y agoIt doesn't work well at all, which is apparent if you look at any small subreddit.
- rhizome 13y agoWait, so the point is that negative vote-score articles don't show up in "Hot?" Seems reasonable.
- dwild 13y agoSo -10 is higher than -5 now?
- DrStalker 13y agoMore that if the first vote a submission gets is negative it can't recover and vanishes.
- rhizome 13y agoI didn't get that `sign` was not ever re-evaluated from the writeup, can you elaborate?
- sesqu 13y agoThe first downvote pushes the submission down below every positive submission ever, not just below the last 12 hours' worth. It can only recover if upvoted from other views.
- rhizome 13y agoSo, in the minimum case, two upvotes? Like any post? Seems there would have to be an appearance cutoff somewhere.
- randomwalker 13y agotl;dr: Posts whose net score ever becomes negative essentially vanish permanently due to a quirk in the algorithm. So an attacker can disappear posts he doesn't like by constantly watching the "New" page and downvoting them as soon as they appear.
- amerika_blog 13y agoGood synopsis. Check out the discussion on Reddit itself: http://www.reddit.com/r/programming/comments/1si31j/reddits_empire_is_founded_on_a_flawed_algorithm/ http://www.reddit.com/r/programming/comments/1si31j/reddits_...
- dshanahan 13y agoParticularly the reddit dev comments here: http://www.reddit.com/r/programming/comments/1si31j/reddits_empire_is_founded_on_a_flawed_algorithm/cdy2jhh http://www.reddit.com/r/programming/comments/1si31j/reddits_...
- raldi 13y agoNo, they just vanish from the hot page, and it's not necessarily permanent. They're still visible on /new, which allows for upvotes-of-resurrection.
- yen223 13y agoBut as the author pointed out, nobody visits /new.
- __Joker 13y agoLots do. At least two reasons, I can thinks of, 1. If you are a regular to a sub, new will act like RSS feed. You know what to read and where to stop. 2. If you believe in reddit karma, better chance of your comments to be recognised. By the time it hits the front page, more or less it is a comment muddle.
- 13y ago
- jbigelow76 13y agoReddit's algorithm is its community, the rest is just math.
- snowwrestler 13y agoThis got downvoted but I think it is essentially the right answer. I don't see how Reddit's success is based on the amazing efficacy of its algorithm. For example, I don't think fixing this bug and using the fixed code to launch a competing site would beat Reddit. Its success is based on attracting an engaged audience, who participated heavily, in turn attracting a larger audience, whose participation further attracted even more people... etc. The algorithm may have mattered very early, in the beginning, when it was first attracting people who were evaluating it for the first time. But even then, I think that the content that Reddit's staff continuously posted was a bigger factor than the algorithm.
- frozenport 13y agoThis is a feature that promotes controversial posts.
- recuter 13y agoSo the gist of this is: "I found a recent post in a fairly inactive subreddit and downvoted it, bringing its total vote score negative. Sure enough, that post not only dropped off the first page (a first page which contained month-old submissions), but it was effectively banished from the “Hot” ranking entirely. I felt bad and removed my downvote, but that post never really recovered... While testing, I noticed a number of odd phenomena surounding Reddit’s vote scores. Scores would often fluctuate each time I refreshed the page, even on old posts in low-activity subreddits. I suspect they have something more going on, perhaps at the infrastructure level – a load balancer, perhaps, or caching issues." This is partially due to vote fuzzing. More to the point, votes go into a queue and the removal of the downvote might not cancel out the previous action for some time. As a result, this suggested flaw will supposedly let somebody successfully snipe puffins from the new page of a small sized birdwatching subreddit before they ever get a fair shake. I think if somebody would attempt this sort of manipulation further they would find it an ineffective strategy, there have been (probably constantly are) attempts to game Reddit before and this seems like an excellent honeypot. Beyond the narrow set of circumstances during a very small time window the flaw disappears, yet if you try to abuse this you'll stick out like a sore thumb. The true horror expressed in the OP is that the ordering of posts in the purgatory is not strictly logical - the post ranked 10042 should really be ranked 10041. Gasp. Twitch. This is a very lovable brand of OCD to my eyes. :)
- Sous-Vir 13y agoI think you'd be surprised on how much on a knife edge the whole process works. It's perfectly plausible that something that ends up on the front page gets only a handful of upvotes in the first twenty minutes, or half an hour. Moreover, once an article has been submitted, you're not supposed to resubmit it, and many moderators will remove duplicates. I'm actually involved in moderating a fairly large subreddit, and we have periodic waves of neo-Nazi posters gaming the subreddit, and they are surprisingly effective at altering the general mood. You can also see some genuinely shocking opinions as top posts on r/worldnews. These are subreddits with hundreds of thousands of daily visitors. If reddit is operating a system which can easily be gamed, it matters a lot. In this case, with enough proxy accounts, and a modicum of programming experience, you could anonymously supress stories you don't like, with some ease. Do you not think that matters?
- amerika_blog 13y agoReddit is designed for SEO gaming. The point is to have your bots/friends downvote everything but your submission. It works every time.
- woah 13y agoMaybe this flaw is, through some strange and circuitous social mechanism, the precise thing that has made Reddit so popular in the first place.
- aston 13y agoThere's a Reddit thread between various users and keltranis, (one of the more senior coders Reddit ever had) explaining the code here: http://www.reddit.com/r/programming/comments/td4tz/reddits_actual_story_ranking_algorithm_explained/ http://www.reddit.com/r/programming/comments/td4tz/reddits_a... And a quotation for those not wanting to click: *ProfDrMorph* 2 points 1 year ago So that means all posts in all subreddits (when browsing 'hot') are sorted this way: 1. all posts with more upvotes than downvotes with the order determined by age (newer posts are preferred) and popularity 2. all posts with the same number of up- and downvotes in whatever order the database returns them 3. all posts with less upvotes than downvotes with the order determined by age (older posts are preferred) and popularity (posts with a lot more downvotes are preferred) Because that's what the _hot() function implies if the sorting algorithm uses it as a 'key'. *ketralnis* 2 points 1 year ago Yes that's accurate
- Danieru 13y agoI doubt keltranis is missing the issue. Instead let's assume reddit's devs know what's what think of a why. My guess is they want to fast bury spam. Spam in old posts will have been deleted and thus old posts are more trustworthy. Voting brigades are a smaller risk than the constant flood of spam.
- ketralnis 13y agoNo, spam doesn't have anything to do with it. And actually, older (high-scoring, anyway) links are more likely to have spam comments because the moderators aren't hanging around there anymore.
- kennywinker 13y agoI believe they have other, closed source, anti-spam measures. This might be the reason, but if it is 'tis a silly reason. If I understand correctly, it means that if 75% of people like a post, and 25% dislike it, that means it has a 25% chance of not being successful because the first vote is negative. Overall, this measure is effectively lowering the quality of all content on the site.
- 13y ago
- joseph_cooney 13y agoSeems about right vis a vis the need for algorithms to be correct. Good enough is, by definition, good enough.
- Brakenshire 13y agoGood enough to be plausible to a casual user is not good enough to prevent the algorithm from being damaging. For a lot of people, reddit is their only source of news, if that can be easily gamed, it has serious consequences. Developers have responsibilities beyond increasing traffic.
- pippy 13y agoI've been very interested in this problem. I ran a community for a few months that wound up being quite popular (40k uniques a day before I closed it down). My attempt to address with problem was to have a min + max time, but most interestingly count the number of posts as well. Even if an opinion was popular if it got a response out of people it would stay around longer before dropping of quicker. I prioritised community engagement over the communities quality of content. This turned out to be slightly more effective way of ranking content.
- mafuyu 13y agoInterestingly enough, the HN ranking algorithm takes a bit of an opposite stance, punishing posts that generate too much discussion in proportion to upvotes. Both approaches are valid, depending on what your goals for the community are. HN's system would rather quickly derank a post that is potentially inflammatory and keep good content on top rather than using comments as a heuristic for community involvement.
- AndrewKemendo 13y agoI really really really wish there was a website that broke down code like this into explained text. I can grok a lot of code regardless of language somewhat intuitively because there is so much crossover - but I still have issues often when it comes to breaking down complex and unique segments. This would really help the learning process but I appreciate how time intensive it is.
- girvo 13y agoHave a look at literate programming (I think that's what it's called). It's not exactly what you're looking for (in that the project must be done that way from the beginning) but you'll find it interesting I think!
- Jtsummers 13y agoThat is what it's called. A site [1] with some examples. Donald Knuth developed the term and concept. It seems fairly popular in the haskell community, and to a lesser extent the scheme/racket communities. [1] http://en.literateprograms.org/LiteratePrograms:Welcome http://en.literateprograms.org/LiteratePrograms:Welcome
- youngian 13y agoYes, that would be a great place to find more like this. CoffeeScript has added support for literate programming, so that might be a good place to go for modern, web-oriented code samples.
- benihana 13y ago>Maybe it’s that a good technical implementation is a distant second to a good product This is what computer scientists should take away from this.
- brudgers 13y agoRegular internet using people are not amazed by Reddit's ranking algorithm, they just enjoy it's anti-trolling technology and critical mass.
- sytelus 13y agoNope. This is just survivor ship bias. You look at one successful product and you propose a rule that technical implementation doesn't matter. You need to look at all the dead startups which had a good product but not good implementations (ex, friendster and other FB competitors which were ahead of the game but couldn't scale). In reality, Reddit is successful by a pure chance. In its initial days it was pretty much barren wasteland for fringe people. Most people had written of it as another me-too without much of a differentiation and Digg was the place to be. Then Digg screwed up and people wanted alternative and suddenly Reddit was overnight lord of link submission evolving in to discussion forums.
- ilaksh 13y agoThis is a good example of where something that is fundamentally flawed becomes accepted and popular and then a huge amount of effort goes into rationalizing it. Which goes to show you that things are the way they are not because that's the way things should be, but just because that's the way things are. Which is a very stupid way to run things, but that is the way our 'society' works.
- raldi 13y agoThe real flawed reddit algorithm is "controversy". It's basically: SORT ABS(ups - downs) ASCENDING ...which means something with 1000 upvotes and 999 downvotes will be considered less controversial than something with 2 upvotes and 2 downvotes. A much better algorithm for controversy would be: SORT MIN(ups, downs) DESCENDING
- recuter 13y agoHow many people use the different alternative ranking options do you, err, reckon, if you had to guess? I figure all the different options to slice and dice are fun for information junkies but most users won't bother or even know about them.
- raldi 13y agoI answered that over here: http://www.reddit.com/r/programming/comments/1si31j/reddits_empire_is_founded_on_a_flawed_algorithm/cdy26bf http://www.reddit.com/r/programming/comments/1si31j/reddits_...
- philh 13y agoI'm confused, because as far as I can tell you didn't answer that at all.
- raldi 13y agoThe question (both here and above the reddit comment I linked to) was effectively, "Isn't it pointless to spend time working on the controversy sort, because very few redditors ever sort by it?", and my reply is effectively, "No, it isn't pointless, because if it worked well, it could be used to surface controversial posts on the default view."
- recuter 13y agoWhy is that desirable for the default view?
- dionidium 13y agoAn argument for the proposition that this behavior was intended is that if the purpose of sign was to get the sign for order, then it was actually entirely unnecessary and they could have just done something like this: order = log(max(abs(s), 1)) * ((s) / max(abs(s), 1)) I'd prefer the benefit of the doubt, especially given their previous responses.
- dionidium 13y agoMade a mistake when rearranging parens. It's not wrong, just a little ugly. That should of course be: order = log(max(abs(s), 1)) * (s / max(abs(s), 1)) Or, better: ns = max(abs(s), 1) order = log(ns) * (s / ns)
- makomk 13y agoThat's not a very good argument - they could just as easily have calculated sign as ((s) / max(abs(s), 1)) if this was intentional, so the fact they didn't probably just means whoever wrote the code didn't think of that trick.
- dionidium 13y agoThat a good point. Of course, I'm reasoning backward to justify something that's already there, so it started its life as a bad argument :)
- lnanek2 13y agoJust goes to prove you only need to get the parts users care about right. Treatment of some negative score posts just isn't too important and may even help remove spam ASAP at the cost of some good posts. If they had sacrificed some other aspect of the site to get this right, they probably would have been worse off.
- shadowmint 13y agoSo, a massively, massively popular site that makes it business by ranking the user generated content on it by importance... is wrong. > Maybe there is no moral. Reddit screwed up. ...or maybe, they know what they're doing. Maybe not. ...but when you supply a bugfix, the onus is on the submitter to demonstrate that 1) the fix fixes the problem and 2) that it doesn't break anything else. It would appear that no effort has been made at (2), to demonstrate that the proposed change would not have an adverse affect on other high-vote rankings. To be fair, it would have been nice to see the pull request response (https://github.com/reddit/reddit/pull/583 https://github.com/reddit/reddit/pull/583) mention that an alternative algorithm choice would have to be demonstrably better in a large scale analysis before they would even dream of changing their core ranking algorithm, but it's not unfair for them to take that stance. It's like asking Google to change their page rank algorithm because you don't like it.
- lectrick 13y ago> the onus is on the submitter to demonstrate that 1) the fix fixes the problem and 2) that it doesn't break anything else. No. The onus is on Reddit's test suite which, ostensibly, would cover voting (one of the core features/functionality of the site!) to demonstrate this. Or are you suggesting that he didn't run the full build?
- ryanobjc 13y agoAh yes the tests pass. Now it is time to relax and veg out. Oh wait, the test suite doesnt actually guarantee bug free software? Worse, test suites dont protect you from bugs in the specs? Well I can tell we are going to have a rough night ahead...
- shadowmint 13y agoDid /r2/tests suddenly get a hundred new awesome tests in the last 6 months? It's pretty... bare. Or it was last time I looked. Pull request -> should include tests if relevant ones don't already exist.
- 13y ago
- kylelibra 13y agoIf the same post is at the top of reddit and hackernews at the same time I'm pretty sure the internet implodes and life as we know it comes to an end.
- interstitial 13y agoSo relevant it hurts: https://news.ycombinator.com/item?id=5927904 https://news.ycombinator.com/item?id=5927904
- sesqu 13y agoThe allegation there, from what I can see, was that a someone had a bot controlling five accounts. That's enough to impact even the corrected version of this ranking, and as such is only moderately relevant.
- maskoliunas 13y agoIf I was a salesman, I would explain that easily: 1. If the material is newer and already has attracted the same amount of negative votes in shorter period than another one in longer period -- the first is worse. Push it down. 2. If people suddenly started hating something very much, that might mean the content is hot and attracts a lot of attention. So pull it up. "thinking out of the... emm, where is my box???" ------------- Imagine two submissions, submitted 5 seconds apart. Each receives two downvotes. seconds is larger for the newer submission, but because of a negative sign, the newer submission is actually rated lower than the older submission. Imagine two more submissions, submitted at exactly the same time. One receives 10 downvotes, the other 5 downvotes. seconds is the same for both, sign is -1 for both, but order is higher for the -10 submission. So it actually ranks higher than the -5 submission, even though people hate it twice as much.
- petepete 13y agoIf it's stupid, but it works, it ain't stupid.
- sesqu 13y ago> And notably, they are sorted oldest first, just as I predicted. This bit is actually misstated. Those posts all have a comparison value of 0 (assuming score is simplistic), and are not affected by the oldest-first ranking of negative submissions. The ordering here is likely insertion order, which just happens to be the same as oldest-first.
- Semaphor 13y agoFor those of you who never are on Reddit, both keltranis and raldi posting here are former Reddit admins.
- davidgerard 13y agoI think the key point is: Reddit works well enough that they don't, and quite possibly shouldn't, care.
- arca_vorago 13y agoThe real flaw with reddit is moderator abuse.
- quokka 13y agoI don't understand. The definition of order is s = score(ups, downs) order = log10(max(abs(s), 1)) and the poster says that "order will always be positive". But that isn't true. It is the logarithm of a number in (0,1], and so is negative or zero. Since we cut the value off at 1 I assume that the score function does something to the votes beyond (ups - downs), scaling the value in a way that makes the logarithm of the score interesting.
- youngian 13y ago`ups` and `downs` are whole numbers, so `abs(s)` will usually be >= 1, like 2654 or something. The log is there to reduce the influence of additional votes after a certain level of popularity. See footnote ^2: http://technotes.iangreenleaf.com/posts/2013-12-09-reddits-empire-is-built-on-a-flawed-algorithm.html#fn2 http://technotes.iangreenleaf.com/posts/2013-12-09-reddits-e...
- quokka 13y agoOh, good lord. The definition uses max, but I was thinking of min. Face palm.
- youngian 13y agoAuthor here. I posted a quick follow-up with some corrections and other items of interest that came out of the discussion: http://technotes.iangreenleaf.com/posts/2013-12-10-the-reddit-algorithm-a-recap.html http://technotes.iangreenleaf.com/posts/2013-12-10-the-reddi.... And of course, if you would like more articles written by me and an extremely high signal-to-noise ratio (because I post so rarely...), consider subscribing: http://technotes.iangreenleaf.com http://technotes.iangreenleaf.com. RSS is not dead, dammit.