7 ms·
Why HN Should Use Randomized Algorithms
- Udo 13y agoI think it's a great idea that solves an actual problem and it would probably help smooth out voting rings that slip past the detector. However, that would probably mean HN couldn't optimize or cache the front page. On the other hand, this is a feature for logged-in users only, so it might be feasible.
- albertzeyer 13y agoYou could have two versions of the site, one with random noise, one without. Or maybe even specify the noise.
- Udo 13y agoThat's what I meant when I said it was for logged-in users only.
- dangoldin 13y agoIt could also be done via client side JS. Load the first 2 pages of results with the true scores and then use JS to add the randomness factor. This way the page can be cached.
- 1200bps 13y agoAs far as I can see, HN already has randomized algorithms. I just had to create a new account because my six-month old 3,000+ karma account just got slowbanned and silentbanned for apparently no reason. Out of nowhere, anywhere from 6 to 14 seconds to load the front page (or any other HN page) when logged into that account.
- Udo 13y agoWhat does silentbanned mean? I'm not sure what the offense is for which people get slowbanned on HN, but I also don't understand how that counts as a randomized algorithm. As an aside, it amazes me that people who notice being hellbanned (which is really the ultimate way of HN telling you not to post here anymore) make 2nd or 3rd accounts and just continue. I'd want to find out why I was being banned and then, if I didn't agree with the reason, I'd just leave.
- 1200bps 13y agoI was not hellbanned - where everything I posted was instantly dead. To be honest the only thing I'm sure of is that I was slowbanned. I timed page loads 10 times with the slowbanned account and they averaged about 11 seconds. Then page loads with not being logged in averaged about 1 second. Then page loads with a new account averaged about 1 second as well. The junky thing is that the account was almost 7 months old and had 3,324 positive karma but I guess I said something that ticked someone off: https://news.ycombinator.com/user?id=300bps https://news.ycombinator.com/user?id=300bps
- Udo 13y agoThere isn't anything in that account's immediate history that looks like grounds for bannination, on the contrary. Are you sure it's a slowban, not some technical problem? I'd try to email the admins to find out more.
- 300bps 13y agoI'm pretty confident it's a slowban. Logging in with this account again, I go from instant page loads to 12 second page loads - only on HN of course. After re-reading the terms of use of the site, I emailed pg on 11/22 stating that I believed I was slowbanned and that I didn't believe I had violated any terms of use. I hadn't spammed ever (I have nothing to sell), I don't participate in flame wars. I didn't get a response so I emailed him again on 11/27 just saying it's clear I was slowbanned and that it would be nice to know why but that I guess that's not going to happen. I figure there are a few possibilities: 1. I said one thing one time that rubbed someone the wrong way 2. An admin thought it would be funny to give someone who uses the username 300bps a slowban thereby effectively causing their web experience to be equivalent to a 300 bps modem Who knows? Seems random to me to be honest.
- jds375 13y agoI would still be interested in maybe a weeklong trial run and see how I felt it went.
- alan_cx 13y agoGreat idea. Would be interesting to see it trialed.
- tehwalrus 13y agoIt is unfortunate that you have to read the footnotes to understand the article - on a first read of the text I was completely lost. This is, though, a good idea - at the moment, many interesting articles are lost because not enough people visit the new page.
- ColinWright 13y ago> It is unfortunate that you have to read the footnotes > to understand the article - on a first read of the text > I was completely lost. I'm not the author, but I'm intensely interested in effective communication. You've taken the trouble to comment, and I was wondering if you'd be willing to walk through the item and explain where you got to in the main body that you thought was unclear, and where you were lost. I assume the first paragraph was self-evident - where did you find it started to become impossible to follow? > ... at the moment, many interesting articles are lost > because not enough people visit the new page. I've always done this, but it's becoming increasingly pointless. My impression is that the number of really interesting articles remains roughly constant, but the total volume increases. It's getting more and more depressing to wade through the uninteresting (to me) to find the rare gems. I still do it though.
- tehwalrus 13y agoThe first three are in English, sure, I'm talking about the last two paragraphs. I understood the problem, almost from the title - I was interested mostly in the proposed solution. In particular, the last 2-3 sentences where a complex procedure is defined in a few inline code snippets - and then the author continues as if the result is obvious. It reminds me very much of when I read overly mathematical scientific publications (which I do a lot, being a PhD student.) In such papers, the authors often expect you to stop and spend 5 minutes reading a few symbols (perhaps reading a couple of Wikipedia articles along the way) before continuing. This makes the articles highly unreadable - I sometimes spend 2-3 days slowing working through the particularly poor ones. (Obviously, I only do this for highly relevant/useful papers!) This article isn't in the same level of awful, obviously, but it did give me the same "I'm lost now, I should go back" feeling - which I thought was worth feeding back to the author, especially since they had provided the explanation, but they'd hidden it in a footnote.
- aragot 13y ago...or we could have a much larger home page, so we don't have to deal with the end-of-page-1 discontinuity, and we rely on the end-of-reader's-attention-span discontinuity, which is much smoother.
- deleted 13y ago[deleted]
- Houshalter 13y agoIt's a good idea, though I wonder if there is a better way to do this without randomness.[1] I mean it's just an exploitation vs exploration problem. You want to maximize the number of articles the average user will upvote (essentially avoiding wasting our time.) You want to use the time and number of votes to predict the probability that a user will upvote it. But you also want to do "experiments" to find even better articles even if it means wasting a few people's time. I'm certain there is an elegant math equation that does this perfectly but I can't figure it out. http://lesswrong.com/lw/vp/worse_than_random/ http://lesswrong.com/lw/vp/worse_than_random/
- yummyfajitas 13y agoI definitely agree with this. I'm not sure what HN's algorithm is actually supposed to optimize, but apart from that I think it's pretty straightforward how to optimize it with non-random systems. http://www.bayesianwitch.com/blog/2013/why_hn_shouldnt_use_randomized_algorithms.html http://www.bayesianwitch.com/blog/2013/why_hn_shouldnt_use_r...
- danmaz74 13y agoGiving a randomness to the whole sorting of the articles could have a "noisy" effect on users who update the home page often. On the other end, PG could just add a line at the end of the home page to show one post randomly selected from the new articles. This way we could all contribute to the selection of new articles, instead of only relying on those brave souls who regularly wade through the "new" page.
- pmiller2 13y ago>Giving a randomness to the whole sorting of the articles could have a "noisy" effect on users who update the home page often. Yes, but not much. Provided the amount of noise is small enough, the order of articles on the front page may change a bit, but the articles themselves won't change much. I don't know much about the typical distribution of scores of articles on the front page at any given moment, but I suspect even the order of most of the page probably won't be affected. Think about it this way: pushing an article that's already high up on the home page up or down a little in ranking won't push it off the front page. It's only the bottom of the page (and the top of page 2) that would even be materially affected.
- danmaz74 13y agoAgreed on that, but you know, just not knowing if you are "really" #3 or not is bad, if you have your post in the home page. IMHO it would detract some from the value of the website.
- read 13y agoThe idea is not new. For building systems, tlb argues there should be a random component in all arbitrary decisions. http://tlb.org/tlbImages/thesis.ps http://tlb.org/tlbImages/thesis.ps
- taspeotis 13y agoPrevious discussion: https://news.ycombinator.com/item?id=6498992 https://news.ycombinator.com/item?id=6498992 86 points by luu 58 days ago | flag | 41 comments
- netforay 13y agoInstead of randomness, rank should be used as probability of getting selected on the first page. Not sure how to do it, and how to maintain the cache.
- kens 13y agoThis sounds like a cool idea that solves a real problem, but I'm not convinced it will work. The main blocker for an article is getting the first two or three votes fast enough to get from the new page to the front page. If you look at my article on HN scoring [1], most successful articles shoot up quickly and then slowly drop, so randomization is mostly going to just put declining articles back on the front page for a bit, which doesn't really help the reader or articles. To be effective, the randomization needs to be focused on newish articles. I like danmaz's idea of adding a random new article to the front page. Best would be combining ideas and using weighted randomness to add a new article to the front page, so a vote or two would boost an article's chance of getting the random slot. (Another thing that would help is if the More link didn't time out after a couple minutes, so readers could go to the second page more reliably.) [1] http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.html http://www.righto.com/2013/11/how-hacker-news-ranking-really...
- baby 13y agoI've put some thoughts into it as well. By doing that you create a new problem : pushing crappy stories to the frontpage and scaring new users away. For a website like HN that doesn't need or want new users that's a good thing. But for a reddit, hypem, etc... the best solution, IMO, is to trust people browsing /r/new to do the work for the others.
- jedberg 13y agoI think this is the wrong solution. I wrote my thoughts here: https://news.ycombinator.com/item?id=6834705 https://news.ycombinator.com/item?id=6834705