5 ms·

Ask PG: Statistics

14 points by tel 19y ago

There have been a few posts (one being mine) around concerning fears about "fluff" posts taking over. Solutions are ranging from allowing downmods, adjusting weighting algorithms, and even just blacklisting Reddit posts. One thing I'd like to see before considering adjustments like this would be more detailed statistics about how people vote, how leaders vote, how karma is distributed, &c. I feel like interesting data sets would be: Across all users Karma Number of votes Number of submissions Times downmodded ... Across all posts Karma Comments Flag for whether it's considered "fluffy" (mod's discretion) Flag for Reddit submission ... Really, it could be a pretty big problem. It does seem like a potential playground of hacks and at least some of those statistics are certainly not difficult to mine (probably Arc one-liners). So, if it seems interesting and there's not any issue with privacy (strip out usernames and it'd be hard to correlate anything beyond the leaderboard) is there any chance of seeing a hn-stats tarball?

andreyf 19y ago
In general, Paul, is there a reason not to open up a machine-readable HN platform for people to tinker on top of (other than the time it'll take to code)?
- ed 19y ago
  Exposing voting history strikes me as an obviously bad thing to do given that votes have always been anonymous. Even if you tried to scrub personal information from the data, it'd be fairly easy to match anonymous ID's to HN nicks.
  - tel 19y ago
    Full voting history, probably, yes. Number of votes (some relation to activity, perhaps) not so much. Anything that's obviously going to make it easy to reverse engineer anonymous things isn't a good idea, but that doesn't mean you can't still find interesting gems.
    - yters 19y ago
      I agree. Aggregate stats such as whether a small group does most of the voting, whether the same people seem to vote for the highest ranked items, etc. would be pretty interesting while not giving away too much information. I am also curious whether a completely open system would work. Has that ever been tried? On a system the size of Reddit or Digg it would be intriguing to see how groups cluster.
iamdave 19y ago
I'm in favor most of the flag option. I understand how users with a high degree of karma can prove themselves worthy by virtue of what karma even means, but combining flags and the number of times downmodded creates a more level playing field for the process.
bigtoga 19y ago
Anyone so concerned with stats that does not work for HN is just looking for more places to waste their time instead of working on their hack/startup. This isn't your community; it's a website. You may think of it as your community but where will you be two years from now? Still wasting time on this site asking for stuff like this? No. Google "This too shall pass" for it is the lesson of life. Really - the post today, "Do It Fucking Now", applies here as well lol. Figure out what's important and do that; having stats on this site is not important in your or my life.
- rms 19y ago
  Your objection here doesn't make any sense. Of course people could be spending more time doing work and less time on news.yc. But asking for stats isn't any better or worse than any other use of this site. Obviously this site isn't a community to you, but it is to some people. It's also entertainment. I'd rather have stats on this website than watch whatever is on network television right now.
- bayareaguy 19y ago
  Some of us aren't interested in the numbers but we are interested in the patterns behind them, especially if they could be used to identify destructive trends.
  - fiaz 19y ago
    Organizations that think like this: DHS FBI CIA DOD KGB Thought Police Please don't add YC.News to this list. It's one of the few places I enjoy on the net at the moment (and of those few, this is my favorite). I don't like the notion of having some sort policing activity to identify "destructive trends" as you call it. Sorry for putting it so bluntly.
    - rrival 19y ago
      Having observed Digg and Reddit devolve, solving this problem has nothing to do with conspiracy theories. This is more about addressing Eternal September concerns (brilliant reference by someone yesterday). History will repeat itself until a clever, llama-free community ahem addresses these problems intelligently. I can't think of a better place for it.
fiaz 19y ago
I have the following question for those that are advocating a tech/statistical/mathematical solution: How many of you are individuals who are in the process of creating a startup in the hopes of getting YC funding? The reason why I'm asking this question is because I had the impression that the articles here are more reflective of the VC/hacker roots of YCombinator (silly me!). I also feel that somehow alot of the up-votes for this perspective is reflective of the notion that if you complain to PG enough, he'll swoop in and fix things so that all is well. Hint: if you continue to have this attitude, then you will see your chances of getting YC funding (or even have them bat an eyelash at your application) decrease rapidly because your objection to "fluff" is so strong; do you really think anybody in their right mind would fund somebody who can't handle the "fluff"?!?!?!?! I recognize that there are many who have been suggesting solutions that PG implement (and to his great credit he has pleased many of you by expanding the leaders list to 100 now!); but I also want to know if your energies would be better suited in actually finding articles that you like and submitting those (in other words, being competitive), and perhaps even EARNING karma points (which translates into recognition as well). Not everything is solvable in code and not everything that you do not like is a "problem". I would argue that the diversity (which can be read as the mix of hard-core geekstuff along with fluff) is reflective of the VC/hacker roots of Hacker News. Or in other words: break out of the box and have a multifaceted view of the world.
- iamelgringo 19y ago
  I'm probably going to regret responding to this, but here goes... Many of us are also very invested in this site, and are interested in keeping it close to its roots. That's why there's been a lot of alarm regarding the content on the site. I'm sure that some of it will blow over. Some of it will cause changes. I don't really think that people are trying to get PG to automagically fix the problem for us. I think that we're trying to have an open discussion about solving the Eternal September problem with social sites. This is a recurring problem that occurs on pretty much any site that has a user-contributed component and gets popular. And, regarding earning karma points: If you take a look at the people frequently involved in the discussion on fluff, trolls and quality slides, I think that you'll find that many of the people that are concerned about this are actually quite active on the leader board. I think it's because we're on the leader board, that we're so invested in keeping Hacker News on the straight and narrow. We've invested a lot of time and effort into the site, and are willing to continue that investment.
ghiotion 19y ago
The original question aside (for the record, I'd find it interesting as well), does anyone else think this comment thread is kinda nasty? Ugh.

Ask PG: Statistics

14 points by tel 19y ago

andreyf 19y ago

In general, Paul, is there a reason not to open up a machine-readable HN platform for people to tinker on top of (other than the time it'll take to code)?

ed 19y ago
Exposing voting history strikes me as an obviously bad thing to do given that votes have always been anonymous. Even if you tried to scrub personal information from the data, it'd be fairly easy to match anonymous ID's to HN nicks.
- tel 19y ago
  Full voting history, probably, yes. Number of votes (some relation to activity, perhaps) not so much. Anything that's obviously going to make it easy to reverse engineer anonymous things isn't a good idea, but that doesn't mean you can't still find interesting gems.
  - yters 19y ago
    I agree. Aggregate stats such as whether a small group does most of the voting, whether the same people seem to vote for the highest ranked items, etc. would be pretty interesting while not giving away too much information. I am also curious whether a completely open system would work. Has that ever been tried? On a system the size of Reddit or Digg it would be intriguing to see how groups cluster.

iamdave 19y ago

I'm in favor most of the flag option. I understand how users with a high degree of karma can prove themselves worthy by virtue of what karma even means, but combining flags and the number of times downmodded creates a more level playing field for the process.

bigtoga 19y ago

Anyone so concerned with stats that does not work for HN is just looking for more places to waste their time instead of working on their hack/startup. This isn't your community; it's a website. You may think of it as your community but where will you be two years from now? Still wasting time on this site asking for stuff like this? No. Google "This too shall pass" for it is the lesson of life. Really - the post today, "Do It Fucking Now", applies here as well lol. Figure out what's important and do that; having stats on this site is not important in your or my life.

rms 19y ago
Your objection here doesn't make any sense. Of course people could be spending more time doing work and less time on news.yc. But asking for stats isn't any better or worse than any other use of this site. Obviously this site isn't a community to you, but it is to some people. It's also entertainment. I'd rather have stats on this website than watch whatever is on network television right now.
bayareaguy 19y ago
Some of us aren't interested in the numbers but we are interested in the patterns behind them, especially if they could be used to identify destructive trends.
- fiaz 19y ago
  Organizations that think like this: DHS FBI CIA DOD KGB Thought Police Please don't add YC.News to this list. It's one of the few places I enjoy on the net at the moment (and of those few, this is my favorite). I don't like the notion of having some sort policing activity to identify "destructive trends" as you call it. Sorry for putting it so bluntly.
  - rrival 19y ago
    Having observed Digg and Reddit devolve, solving this problem has nothing to do with conspiracy theories. This is more about addressing Eternal September concerns (brilliant reference by someone yesterday). History will repeat itself until a clever, llama-free community ahem addresses these problems intelligently. I can't think of a better place for it.

fiaz 19y ago

I have the following question for those that are advocating a tech/statistical/mathematical solution: How many of you are individuals who are in the process of creating a startup in the hopes of getting YC funding? The reason why I'm asking this question is because I had the impression that the articles here are more reflective of the VC/hacker roots of YCombinator (silly me!). I also feel that somehow alot of the up-votes for this perspective is reflective of the notion that if you complain to PG enough, he'll swoop in and fix things so that all is well. Hint: if you continue to have this attitude, then you will see your chances of getting YC funding (or even have them bat an eyelash at your application) decrease rapidly because your objection to "fluff" is so strong; do you really think anybody in their right mind would fund somebody who can't handle the "fluff"?!?!?!?! I recognize that there are many who have been suggesting solutions that PG implement (and to his great credit he has pleased many of you by expanding the leaders list to 100 now!); but I also want to know if your energies would be better suited in actually finding articles that you like and submitting those (in other words, being competitive), and perhaps even EARNING karma points (which translates into recognition as well). Not everything is solvable in code and not everything that you do not like is a "problem". I would argue that the diversity (which can be read as the mix of hard-core geekstuff along with fluff) is reflective of the VC/hacker roots of Hacker News. Or in other words: break out of the box and have a multifaceted view of the world.

iamelgringo 19y ago
I'm probably going to regret responding to this, but here goes... Many of us are also very invested in this site, and are interested in keeping it close to its roots. That's why there's been a lot of alarm regarding the content on the site. I'm sure that some of it will blow over. Some of it will cause changes. I don't really think that people are trying to get PG to automagically fix the problem for us. I think that we're trying to have an open discussion about solving the Eternal September problem with social sites. This is a recurring problem that occurs on pretty much any site that has a user-contributed component and gets popular. And, regarding earning karma points: If you take a look at the people frequently involved in the discussion on fluff, trolls and quality slides, I think that you'll find that many of the people that are concerned about this are actually quite active on the leader board. I think it's because we're on the leader board, that we're so invested in keeping Hacker News on the straight and narrow. We've invested a lot of time and effort into the site, and are willing to continue that investment.

ghiotion 19y ago

The original question aside (for the record, I'd find it interesting as well), does anyone else think this comment thread is kinda nasty? Ugh.