6 ms·
Are HN submissions dominated by a small elite of users?
I was wondering the same thing when I recently saw this blog post: https://news.ycombinator.com/item?id=6956690
I thought it would be easy to find out how karma is distributed among users using the HNsearch API.
Unfortunately, it's not that easy because this API is not optimized for fetching massive amounts of data. What I could do was looking at the 1000 users with the most karma. I separated those into ten chunks and looked at the total sum of Karma per chunk of 100 users. This is the result:
http://www.chartgo.com/share.do?id=6cca3d09aa
(Data:
[ (0, 2941707),
(1, 1314760),
(2, 936226),
(3, 745944),
(4, 636533),
(5, 557441),
(6, 493996),
(7, 449999),
(8, 413963),
(9, 384491)]
)
So as you can see the 1st chunk is pretty dominant but then it kind of flattens out. So are HN submissions dominated by a small elite? It obviously depends on your definitions, but I guess not.
It would be interesting to have better data available. Did I miss a better way to query the API? Here's the Python script which I used to query the API: https://gist.github.com/johannes-gehrs/77a92284e3509f2ae960
- maaku 13y agoThere's a book you should read called "The 80/20 Principle". It might help you understand better how this sort of thing naturally happens, and why it is to be expected and nothing wrong about it.
- minimaxir 13y agoAny Karma measure analysis would be biased since Karma is accumulated through both Links and Comments (in contrast with Reddit, which splits both measurements). "Dominated" is a flawed qualifier since Karma is not a zero-sum game.
- sheetjs 13y agoKeep in mind that karma includes both submissions and comments (unlike reddit, where they are separate). For comments, at least, there is a clear bias where comments from users with higher karma tend to stay at the top of the comment page longer, regardless of the votes for other comments. This is intentional and definitely contributes to the effect
- interstitial 13y agoSort of a "founder effect" built into a site for founders. How meta.
- nemothekid 13y agohttp://en.wikipedia.org/wiki/1%25_rule_(Internet_culture) http://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)
- tptacek 13y agoIf you're talking about submission karma, clearly no. I follow the submissions of a selection of HN users I've known for awhile, all of whom are karma leaders, and their submissions are often as not buried on page 4 or 5 with 2-3 votes. If you're talking about comment karma, clearly yes. I could write a comment consisting of randomly chosen names from the Chicago phone book and someone would upvote it. Name recognition has that effect, as does a small group of people who seem to follow my comments deliberately (as I do for the list of people in my profile).
- soneca 13y agoNot just name recognition, but a tptacek/pg/patio11 comment is landed #1 on the comment page and stays there a long time just because of karma. Then, occupying the most privileged real state on the page, after thousands of views, even if it is a small percentage that likes the comment, it is a lot of upvotes. Not that I disagree with this algorithm, I actually agree. Just saying that I think the position of the comment is more important than name recognition in this case.
- bananacurve 13y agogrellas posts are particularly buried because they often don't subscribe to the hivemind, but they are always enlightening.
- mathattack 13y agoI suspect that you'll find it to be some kind of power law distribution, which implies that measures like Standard Deviation won't be so useful. A quick look at the top 1000 that you provided suggests this might be the case. But the top 100 users don't dominate the top 1000, let alone the entire population. There does seem to be some leveling off, though it's unclear how steady that will be. Net - domination is too strong a word. Dominated might imply 20% of the karma in 20 users, and this clearly isn't the case. Even defining the Elite as 100 doesn't get us this.
- mixmastamyk 13y agoYes, take a look at the "new" page. On it you'll see 30 recent submissions, most of which are just as appropriate as those on the home page. But, they don't get any votes, why not? I don't think it typically matters whether the submitter has name recognition or karma, but rather how many friends and colleagues the person can rally to their cause. All it takes is about 10 points in the first hour to get to the front page and then it can "ride the wave" from there. This is an easy threshold for people with large professional networks, companies, with friends in SV et al. Many people here know each other. Yes, there are some technical counter-measures, but there is only so much that can be done. This is not to say that a user acting alone has no chance of reaching the home page. However, after a few years of frequenting this site, I believe that most of the posts hitting the front page have had a boost such as I described above. My personal experience anecdote is that, over several years (also considering prior accounts), working as a "loner", that I've never been able to get more than a few meager votes on a submission. Even if they were highly on topic, even if I got up at 6am Pacific, supposedly the best time to do it. Maybe I'm unlucky or my submissions suck, but I rarely bother any more.
- tptacek 13y agoMy impression is that suppression of voting rings, which is what you're describing with your "group of 10 friends" example, is something that the HN maintainers actually spend a lot of effort on; I get this both firsthand from the maintainers that I actually know, and from the fact that I feel like I regularly manage to do (innocuous) things that trip the detector.
- mixmastamyk 13y agoThey do, but avoiding the counter-measures isn't that hard if you're aware of them, and importantly, only needs to be done a few times a year for each combination of friends. I could be mistaken, but the hypothesis fits so far.
- judk 13y agoHN voting is a trivial example of selection bias. If frontpage was determined by a random sampling of the user base (weighted by karma or age if you like), instead of whoever may go looking for the post, it would be more fair. One of the reddit devs suggested something like this recently, putting a New post on every page.
- amerika_blog 13y agoAll social media sites are susceptible to this simple hack: 1. Set up IRC channel. 2. Wait for friends to arrive. 3. When there's ten, post new content to social media. 4. Post link to the IRC. 5. Have you and 1-2 close friends downvote everything else. All of the algorithms I've seen so far are susceptible to this. I do not engage in this practice for many reasons, some ethical and some practical. On an ethical level, it's toying with people's faith in society; some trolls argue successfully that this faith is misplaced and should be shaken; I counterargue that without strong signaling to that effect, it becomes vandalism. The practical is that anyone who trolls around the sites can see who is engaging in this practice and rapidly realizes they're behaving badly. I remember the crucifixion of davisreis666 on Reddit as an example.
- pearjuice 13y agoNo, they are dominated by the common denonimator and that is a massive flaw of vote-based user input. You will only see politically correct, left winged and in-tone liberal opinions and support everywhere. Mutual backpatting is encouraged and because of these traits, the circle continues.
- AutoCorrect 13y agoecho chamber depending on first (maybe most numerous) adopters that self-reinforces by driving away dissenting voices. I agree 100%
- peterwwillis 13y agoLook through the submission list of each user whose posts make it to the top, or of all of the users submitting to new. You'll see if the stories each user posts gains lots of points; if they don't, it's all comment karma.
- hnletter 13y agoA little late to this, but for my Hacker Newsletter project I keep track of of the number of times a user gets on the top three pages: 337 shawndumas 312 ColinWright 298 iProject 291 Libertatea 214 danso 197 llambda 165 Lightning 163 tokenadult 158 sethbannon 147 mtgx 144 evo_9 120 DanielRibeiro 115 denzil_correa, jonbaer 111 lelf 102 001sky 101 dmor 98 coloneltcb 97 Tsiolkovsky, craigkerstiens, dsr12, mh_ 95 ssclafani 90 jseliger 89 jamesbritt 87 JumpCrisscross 85 Garbage 83 wallflower 82 luu 81 antr, protomyth 77 anigbrowl, codegeek 74 aaronbrethorst 73 jgrahamc, pg 71 uladzislau 70 sinak, steveklabnik 69 RougeFemme 68 cleverjake, derpenxyne, swohns 67 _pius 66 jamesjyu, rosser, wslh 64 Pr0, eplanit 63 Brajeshwar, sk2code, uptown 62 bane, stfu 61 teawithcarl 59 ghosh, joeyespo, ph0rque 58 gnosis 57 r0h1n, scholia 56 AndrewDucker, cpeterso, ohjeez 55 bpierre, cyphersanctus, gruseom 54 conductor 52 nikunjk 51 Suraj-Sun, kirillzubovsky 50 lispython 49 MikeCapone, WestCoastJustin, co_pl_te, duck, robin_reala 48 Anon84, ca98am79, taylorbuley 47 fraqed, morphics, nkurz, rpm4321 46 jmduke, kumarski, nreece 45 georgecmu 44 daegloe, rdl 43 donohoe, kunle, olivercameron 42 tanglesome 41 austengary, ck2, jnazario 40 interconnector, rnyman, sonabinu, wikiburner, zoowar 39 cdvonstinkpot, pwg, revorad 38 1337biz, hepha1979, jeffbarr, mhb, zdw 37 fogus, kjhughes, vectorbunny 36 Cbasedlifeform, DanBC, DiabloD3, Reltair, acremades, geetarista 35 AndreyKarpov, jessaustin, mxfh, prostoalex, raganwald, rbanffy 34 wglb 33 CrankyBear, cryptoz, hackhackhack, kposehn, nvk 32 MarlonPro, bcn, npguy, qubitsam, techinsidr 31 Ashuu, Baustin, caffeinewriter, drucken, edent, iamtechaddict, ilamont, mariuz, mikecane, rmah, rohshall, shill, whoishiring 30 eguizzo, eksith, gklein, lukashed, talhof8 29 Sami_Lehtinen, ari_elle, cwan, espeed, gwern, jacquesm, mitmads, sgdesign, superchink, velodrome, wyclif 28 Quekster, X4, clicks, esalazar, iamwil, jaf12duke, javinpaul, laurent123456, recoiledsnake, vanwilder77, zt 27 6thSigma, ValentineC, aespinoza, ananyob, anu_gupta, awwstn, bavidar, dave1010uk, ekm2, pajju, rkudeshi, sdoering, yapcguy 26 DanielBMarkham, bitsweet, brudgers, hornokplease, jcr, tptacek, whalesalad 25 EzGraphs, Hirvesh, adventured, albertzeyer, barredo, johns, jpadilla_, kevin_morrill, kmfrk, mmastrac, mooreds, mrb, niggler, ot, prateekj, saurabh, shrikant, tambourine_man, thibaut_barrere, tjaerv, usaphp 24 amerf1, coffeemug, dangoldin, neya, qiqing, signa11, twapi, w1ntermute, zrail 23 Dekku, Shivetya, duggieawesome, ingve, jalanco, mikeevans, ninthfrank07, npalli, olalonde, playhard, rachbelaid, rb2e, subsystem 22 adamnemecek, amarsahinovic, aritraghosh007, chmars, colinprince, ctoth, dutchbrit, eranation, jashkenas, jmacd, joshfraser, kine, lowglow, mike_esspe, muratmutlu, napolux, redDragon, rrhoover, tareqak, youngerdryas, yread, ytNumbers 21 T-A, akandiah, anon1385, breck, dpaluy, glazskunrukitis, gridscomputing, homakov, ianstormtaylor, jusben1369, kunai, mindcrime, msvan, onosendai, peterkchen, prajjwal, pron, schrofer, sp332, treskot, ttunguz, urlwolf, ximeng 20 BerislavLopac, BruceM, Jaigus, a3voices, abdophoto, angersock, boh, brandnewlow, chinmoy, chrisacky, cperciva, fekberg, ivoflipse, jasonshen, julien, llamataboot, mjn, mmahemoff, platz, shakes, smit, soundsop, speeder, tmoretti, tocomment, vellum 19 RobAley, SanderMak, WadeF, bencevans, bitcartel, clarkm, declan, experiment0, gasull, goronbjorn, hartleybrody, jejune06, jkuria, k-mcgrady, kevingibbon, mijustin, minimaxir, njoglekar, obilgic, ohadfrankfurt, pclark, rainmaker23, robdoherty2, rubikscube, sciwiz, stevewilhelm, tdrnd, timf, twakefield, wamatt 18 10char, ForFreedom, Pasanpr, aynlaplant, bsg75, cs702, deviceguru, dotmanish, ezl, jheitzeb, jstreebin, julien421, orrsella, pain_perdu, philfreo, polskibus, rblion, ridruejo, rmason, sew, swombat 17 MaysonL, ananddass, barmstrong, bconway, bergie, crabasa, dmmalam, dsego, dshankar, equilibrium, fejr, ferdo, gebe, jkopelman, josephby, mikeleeorg, mkrecny, nashequilibrium, petenixey, petercooper, petrel, profquail, ra, replicatorblog, songzme, spking, taytus, ujeezy, wrongc0ntinent 16 Adrock, Impossible, LiveTheDream, StavrosK, aashaykumar92, afshinmeh, ajaymehta, cdl, choult, damian2000, danboarder, daw___, electic, esolyt, followmylee, franze, giis, greenyoda, gregpurtell, hunvreus, jdorfman, jdp23, jellyksong, jorde, jordn, karamazov, linux_devil, lleims, mahmoudimus, maskofsanity, mjhea0, narad, nate, p4bl0, peter123, rfreytag, sciurus, sebg, sharkweek, spindritf, stevewillensky, tosh, tshtf, xmpir, zengr 15 Ataub24, FredericJ, MIT_Hacker, Mithrandir, adulau, apress, areski, austenallred, babawere, bhauer, bitops, bmmayer1, bradleybuda, ckelly, cooldeal, czr80, dan1234, danielpal, gits1225, grey-area, groundCode, hamidr, hawkharris, jeffreyfox, lucb1e, mayop100, mikeknoop, pdknsk, relation, robbiet480, rodriguezcommaj, rpsubhub, sasvari, sheri, smacktoward, thejteam, thinkcomp, vinhnx, xtraclass 14 0cool, Alex3917, ComputerGuru, Peroni, SparksZilla, amazedsaint, anons2011, apoorvamehta, bifrost, blacktulip, bradly, brandonb, bslatkin, c-oreills, carlosgg, charlieirish, chwolfe, codelion, davidroberts, dbaupp, dendory, dennybritz, draegtun, dsl, e1ven, edwintorok, enneff, glazemaster, grej, harryzhang, ishener, ivankirigin, japhyr, jarederondu, jkaljundi, johndcook, joshuacc, justincormack, kamaal, kanamekun, lisper, mathattack, networked, nickmain, pjvds, pragmatictester, pykello, robg, sabalaba, samspenc, sinnerswing, technologizer, ternaryoperator, thewarrior, timr, titlex, watermel0n, wlll 13 6ren, AlexMuir, CrunchyJams, DavidChouinard, Kopion, acav, alter8, asanwal, benjlang, bra-ket, brettcvz, bryanh, bussetta, chrismealy, cinquemb, cpleppert, creamyhorror, cwilson, davidw, detcader, dfc, digisth, ekianjo, endtwist, gatsby, glaugh, gulbrandr, ics, jfaucett, jlongster, kn0thing, larrys, lenkendall, libovness, lukaseder, mactitan, marban, mgunes, michaelrbock, neeee, noinput, palidanx, paulschlacter, plessthanpt05, porker, pytrin, rjsamson, rohin, rpledge, samsolomon, sebkomianos, selmnoo, sethev, smaili, srathi, t0dd, techaddict009, tellarin, tilt, tlrobinson, tomasien, trendspotter, turoczy, vyrotek, waffle_ss, waterlesscloud, weu 12 31reasons, ChuckMcM, CrazedGeek, DaNmarner, Ecio78, Fletch137, GuiA, ISL, IgorPartola, JacksonGariety, Kilo-byte, Rickasaurus, RockyMcNuts, Sealy, SteliE, aaronpk, abraham, achalkley, akos, alexholehouse, ankitoberoi, anujkk, bberson, bcl, bjansn, bjonathan, brianchu, btilly, bwertz, casca, cgi_man, changdizzle, chaz, chriscampbell, cocoflunchy, cramforce, d4vlx, daigoba66, dandrewsen, dctoedt, deusclovis, dmoney67, doh, dwynings, eladgil, enmaku, evolve2k, feelthepain, flippyhead, ghshephard, googletron, gz5, hansy, hkmurakami, hornbaker, irollboozers, jnoller, joxie, juliangamble, kareemm, kitcar, metajack, microwise, mindstab, mmariani, morisy, mpweiher, negrit, nekojima, nqureshi, paraschopra, pascal07, pauljonas, pbiggar, pearjuice, peterkelly, poissonpie, robheaton, seminatore, sherm8n, spooneybarger, stesch, taigeair, thegarside, theoutlander, tikhonj, tnorthcutt, uvdiv, vog, vu0tran, weisser, yeleti, zan2434 11 Amadou, BIackSwan, Cieplak, Swizec, TDL, adrianhoward, aelaguiz, afschwartz, alok-g, aram, benhowdle89, benwerd, cstross, darxius, davidsmith8900, dboles99, decklin, devx, diggan, dkasper, dohertyjf, eduardordm, edw519, eibrahim, ekpyrotic, elleferrer, epenn, fatiherikli, frankdenbow, friism, guiseppecalzone, haven, hudibras, iand, igrigorik, immad, infoman, jacoblyles, jamest, jbaudanza, jdmitch, jgv, jipumarino, johnjlocke, jordanmessina, jpmc, judegomila, kloncks, knes, kogir, ldayley, leephillips, lest, lettergram, luigi, marcieoum, mazsa, memoryfailure, moonboots, morefranco, nbashaw, ndesaulniers, nonrecursive, nsns, nsp, null_ptr, octopus, oleganza, patrickaljord, pavel_lishin, philip1209, rasengan, rdemmer, remi, rfnslyr, rvivek, salimmadjd, sgrove, shared4you, sillysaurus2, simonreed, spdy, speednoise, stollercyrus, swannodette, teamgb, thehodge, tippytop, tlongren, tomse, tortilla, trevin, twog, ukd1, whit537, willvarfar, wiradikusuma, xijuan, yitchelle, zachinglis, zhs 10 AirbnbNerds, Avalaxy, CaptainZapp, Danieru, JanLaussmann, MattRogish, Maven911, Mitt, SonicSoul, Xcelerate, andreiursan, andrewnez, antman, austinhallock, bdehaaff, bearwithclaws, bevenky, binarybits, blackhole, brown9-2, ceeK, chaostheory, cheeaun, chewxy, chrislloyd, clarky07, cobrausn, cyang08, dazbradbury, dcope, denysonique, dhotson, diminium, dirkk0, dmitri1981, dsowers, dylangs1030, dz0ny, edtechdev, fecak, felipebueno, fmavituna, giorgiofontana, gkuan, gregman, hboon, hippo33, hodgesmr, imkevinxu, jacobwg, jashmenn, jayadevan, jazzychad, jballanc, jf, jkarneges, josh2600, jpatokal, jvns, jwallaceparker, k33l0r, kafkaesque, kinlan, kintamanimatt, krat0sprakhar, kyledrake, lukedeering, maccman, malloc47, marojejian, matt1, maudlinmau5, mdturnerphys, mecredis, milesf, misiti3780, mjfern, molecule, mparramon, mschonfeld, nathanbarry, nileshd, ovechtrick, pavs, pearkes, petethomas, phenylene, philk10, pkrein, r4vik, radley, ramisms, robhawkes, rocky1138, rsobers, sidcool, siong1987, smanuel, stefan_kendall, sw007, swah, swampthing, swanson, sweis, techdog, thingsilearned, timjahn, tylermenezes, ukdm, will_brown, wsieroci, yesplorer, yonasb, zacharyvoase