5 ms·

Slightly off topic but I’ve been playing with this dataset for a while now to learn ML. It’s remarkably humbling. No NLP approach I’ve tried has been able to p

Slightly off topic but I’ve been playing with this dataset for a while now to learn ML. It’s remarkably humbling. No NLP approach I’ve tried has been able to predict question score based on content better than my baseline of “choose the mean”. (I’ve tried random forest on bag of words, AWD-LSTM, and Google AutoML so far). The author of this post tried score prediction as well and pivoted to tag prediction after she couldn’t find anything that worked well: https://twitter.com/srobtweets/status/1125860523377979398?s=21 https://twitter.com/srobtweets/status/1125860523377979398?s=... It’s so crazy to me that which posts get popular might just be random. And makes me wonder about the correlation between post content and popularity on other social sites like HN and reddit.

existencebox 7y ago
Quick question: Did you try/find any correlation with post time/seasonality? I recall that in previous examinations of HN/Reddit top/viral posts, there was some amount of signal along that dimension. (I notice someone mentions this in the twitter thread too but I didn't see a response) Additionally, you found average (all up) to be a better predictor than average per user or per category? Apologies for grilling you here, I should frankly dig in myself, but if you happen to feel like indulging me it's much appreciated :)
- yeldarb 7y ago
  I didn’t try adding in any metadata to the models yet (including time, date, and author). I was just trying to work with the text content of the posts.
AlexCoventry 7y ago
It's not that it's random, it's that the signal is too abstract for such simple methods to grasp it. Machine Learning which can predict Stack Overflow question scores is going to have to understand what's being asked, and how useful the responses are going to be for people interested in the question.
- djakjxnanjak 7y ago
  It may seem obvious with the benefit of hindsight, but I would have guessed that the better answers might be different in terms of word choice or style.
  - AlexCoventry 7y ago
    I'm not saying it was a silly thing to try, just that it's wrong to conclude from the result that the scores are random. I.e., I was mostly responding to "It’s so crazy to me that which posts get popular might just be random."
    - yeldarb 7y ago
      I’m not convinced they are randomly distributed but before I started playing with it I was fairly certain they weren’t. Now all I know is I haven’t seen evidence that they aren’t. I could see how there could be some element of luck in whether a post gets attention before getting buried. It depends who’s online when it’s posted and whether it piques their interest, and many other random factors that don’t have anything to do with the content of the post like how quickly other posts come in afterwards to bury yours, and how any automated algorithms decide to surface it.
- yeldarb 7y ago
  I’m not convinced. I tried manually predicting high score vs low score as well and couldn’t do so reliably. The language model (viewable at https://stackroboflow.com https://stackroboflow.com ) scores much higher accuracy predicting the next word on Stack Overflow than other datasets like IMDb. But on the IMDb dataset this approach led to state of the art sentiment analysis results by pulling out the encoder and adding a custom head whereas on Stack Overflow it didn’t grok anything about the score.
jzwinck 7y ago
I have some unproven factors for predicting certain popular low-scoring questions: https://meta.stackoverflow.com/questions/373412/require-confirmation-before-allowing-mostly-image-or-mostly-link-posts https://meta.stackoverflow.com/questions/373412/require-conf... A lot of questions that should have a very low score get fixed by people other than the original author. This may make your modeling more difficult because not only are there multiple authors, but the expected score of a question is not time-invariant. In fact, the final score of a question is a path dependent composite score of potentially several intermediate formulations.
raz32dust 7y ago
Just curious - did you include only the title and text of the question, or also the tags and author history?
- yeldarb 7y ago
  Only title, and text. My original goal was to make a Chrome extension that would give you tips to help you craft a better question. (I added tags to the language model later.)
  - baroffoos 7y ago
    Date also plays a huge part. Most stackoverflow posts get very little votes in the month they were posted but slowly grow as people search a problem and find an answer. The oldest posts have the highest score so something like "How do I change branch in git" will get 4000 points but posting the same thing or a similar thing today will get you a negative score. Generally very short questions score badly because they will be stuff like "How do I make a video sharing website" but a very similar question like "How do I copy the current line in vim" will score well as long as its not a duplicate. Both of those questions look similar when you just look the words and sentence structure. To know the difference you have to understand what a video sharing website is and know what kind of task copying a line in vim is so you can know which one is a reasonable question and which one is likely to help more future readers.
    - yeldarb 7y ago
      I did try to rule out question age as a major source of error in the model by training on a sample of ~1 million questions all from ~2012. But it didn’t do any better on those. And (theoretically) duplicates are deleted by the moderators.

existencebox 7y ago

Quick question: Did you try/find any correlation with post time/seasonality? I recall that in previous examinations of HN/Reddit top/viral posts, there was some amount of signal along that dimension. (I notice someone mentions this in the twitter thread too but I didn't see a response) Additionally, you found average (all up) to be a better predictor than average per user or per category? Apologies for grilling you here, I should frankly dig in myself, but if you happen to feel like indulging me it's much appreciated :)

yeldarb 7y ago
I didn’t try adding in any metadata to the models yet (including time, date, and author). I was just trying to work with the text content of the posts.

AlexCoventry 7y ago

It's not that it's random, it's that the signal is too abstract for such simple methods to grasp it. Machine Learning which can predict Stack Overflow question scores is going to have to understand what's being asked, and how useful the responses are going to be for people interested in the question.

djakjxnanjak 7y ago
It may seem obvious with the benefit of hindsight, but I would have guessed that the better answers might be different in terms of word choice or style.
- AlexCoventry 7y ago
  I'm not saying it was a silly thing to try, just that it's wrong to conclude from the result that the scores are random. I.e., I was mostly responding to "It’s so crazy to me that which posts get popular might just be random."
  - yeldarb 7y ago
    I’m not convinced they are randomly distributed but before I started playing with it I was fairly certain they weren’t. Now all I know is I haven’t seen evidence that they aren’t. I could see how there could be some element of luck in whether a post gets attention before getting buried. It depends who’s online when it’s posted and whether it piques their interest, and many other random factors that don’t have anything to do with the content of the post like how quickly other posts come in afterwards to bury yours, and how any automated algorithms decide to surface it.
yeldarb 7y ago
I’m not convinced. I tried manually predicting high score vs low score as well and couldn’t do so reliably. The language model (viewable at https://stackroboflow.com https://stackroboflow.com ) scores much higher accuracy predicting the next word on Stack Overflow than other datasets like IMDb. But on the IMDb dataset this approach led to state of the art sentiment analysis results by pulling out the encoder and adding a custom head whereas on Stack Overflow it didn’t grok anything about the score.

jzwinck 7y ago

I have some unproven factors for predicting certain popular low-scoring questions: https://meta.stackoverflow.com/questions/373412/require-confirmation-before-allowing-mostly-image-or-mostly-link-posts https://meta.stackoverflow.com/questions/373412/require-conf... A lot of questions that should have a very low score get fixed by people other than the original author. This may make your modeling more difficult because not only are there multiple authors, but the expected score of a question is not time-invariant. In fact, the final score of a question is a path dependent composite score of potentially several intermediate formulations.

raz32dust 7y ago

Just curious - did you include only the title and text of the question, or also the tags and author history?

yeldarb 7y ago
Only title, and text. My original goal was to make a Chrome extension that would give you tips to help you craft a better question. (I added tags to the language model later.)
- baroffoos 7y ago
  Date also plays a huge part. Most stackoverflow posts get very little votes in the month they were posted but slowly grow as people search a problem and find an answer. The oldest posts have the highest score so something like "How do I change branch in git" will get 4000 points but posting the same thing or a similar thing today will get you a negative score. Generally very short questions score badly because they will be stuff like "How do I make a video sharing website" but a very similar question like "How do I copy the current line in vim" will score well as long as its not a duplicate. Both of those questions look similar when you just look the words and sentence structure. To know the difference you have to understand what a video sharing website is and know what kind of task copying a line in vim is so you can know which one is a reasonable question and which one is likely to help more future readers.
  - yeldarb 7y ago
    I did try to rule out question age as a major source of error in the model by training on a sample of ~1 million questions all from ~2012. But it didn’t do any better on those. And (theoretically) duplicates are deleted by the moderators.