6 ms·

It uses submission titles AND comments. How would you suggest going about normalization?

retube 15y ago
In that case I would do it by number of submissions (posts) + number of comments for each period (3 months I think, looking at your graphs?). I would normalised all quarters to the last complete quarter. So if say in q1 2011 there were 1m posts + comments, and in q1 2007 10k, multiply the count for q1 2007 by 100. It would be truly great to have this normalisation optional, e.g option to normalise by comment+post count and maybe also normalise by number of users (could be an interesting comparison). And perhaps have them super-imposed on each other!
- sant0sk1 15y ago
  Thanks, I will definitely implement the multiplier as you suggested. I like this better than showing as a % of total. I'm storing the total hits so I will do the calculations on the client-side, which means I just might be able to add some toggles for the normalization schemes ;)
  - retube 15y ago
    Thinking a little bit more about this, there are of course lots of submissions that don't get voted up or commented on - spam, off-topic, not interesting etc. Maybe you should only use posts that have either at least one upvote or comment, just to filter out the crap.
- sant0sk1 15y ago
  A first version of this has been implemented. You can't toggle the normalizations just yet, but I figured I should get the fix out there as soon as I could. Check the "Learn More" section where I give you props for the help :)
  - retube 15y ago
    Awesome, and thanks :) Also, see my other comment below for a suggestion re filtering out crappy posts.
topomorph 15y ago
Coincidentally, I was also working on an HN Trends app a couple weeks ago (though I got distracted and haven't finished), and mine is normalized (by number of comments). For example: http://hn-trends.heroku.com/trends?q=scala%2Cclojure http://hn-trends.heroku.com/trends?q=scala%2Cclojure Your app looks a lot better, though! (And my dataset ends around Sept. of last year.)
byrneseyeview 15y ago
Divide by the weighted average change in the number of hits for the 100 most common English words? Basically, add up all the uses of "and," "or," "a," "the," etc. in each period. If Q3 2009 has 4X the sum of those words as Q1 2007, then divide all Q3 2009 results by 4.
- sqrt2 15y ago
  Unfortuantely, many common words seem to be filtered by the API. Some however are not, and they show a common growth pattern: http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here

In that case I would do it by number of submissions (posts) + number of comments for each period (3 months I think, looking at your graphs?). I would normalised all quarters to the last complete quarter. So if say in q1 2011 there were 1m posts + comments, and in q1 2007 10k, multiply the count for q1 2007 by 100. It would be truly great to have this normalisation optional, e.g option to normalise by comment+post count and maybe also normalise by number of users (could be an interesting comparison). And perhaps have them super-imposed on each other!

sant0sk1 15y ago
Thanks, I will definitely implement the multiplier as you suggested. I like this better than showing as a % of total. I'm storing the total hits so I will do the calculations on the client-side, which means I just might be able to add some toggles for the normalization schemes ;)
- retube 15y ago
  Thinking a little bit more about this, there are of course lots of submissions that don't get voted up or commented on - spam, off-topic, not interesting etc. Maybe you should only use posts that have either at least one upvote or comment, just to filter out the crap.
sant0sk1 15y ago
A first version of this has been implemented. You can't toggle the normalizations just yet, but I figured I should get the fix out there as soon as I could. Check the "Learn More" section where I give you props for the help :)
- retube 15y ago
  Awesome, and thanks :) Also, see my other comment below for a suggestion re filtering out crappy posts.

topomorph 15y ago

Coincidentally, I was also working on an HN Trends app a couple weeks ago (though I got distracted and haven't finished), and mine is normalized (by number of comments). For example: http://hn-trends.heroku.com/trends?q=scala%2Cclojure http://hn-trends.heroku.com/trends?q=scala%2Cclojure Your app looks a lot better, though! (And my dataset ends around Sept. of last year.)

byrneseyeview 15y ago

Divide by the weighted average change in the number of hits for the 100 most common English words? Basically, add up all the uses of "and," "or," "a," "the," etc. in each period. If Q3 2009 has 4X the sum of those words as Q1 2007, then divide all Q3 2009 results by 4.

sqrt2 15y ago
Unfortuantely, many common words seem to be filtered by the API. Some however are not, and they show a common growth pattern: http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here http://hntrends.jerodsanto.net/?q=what%2C+you%2C+here