7 ms·

Predicting Stack Overflow Tags with Google’s Cloud AI

163 points by amrrs 7y ago

aficionado 7y ago
Links to dataset do not work.
- turtlegrids 7y ago
  Not sure why you're being downvoted. I also cannot get to at least one dataset -https://storage.googleapis.com/cloudml-demo-lcm/SO_ml_tags_avocado_188k_v2.csv https://storage.googleapis.com/cloudml-demo-lcm/SO_ml_tags_a... which is linked to from the fourth paragraph directly above the "What Is The Bag Of Words Model?" section heading.
  - boulos 7y ago
    I pinged Sara. Probably just forgot to make it publicly accessible.
    - turtlegrids 7y ago
      Thanks!
- sararob 7y ago
  Article author here. To access the dataset you need to be logged in with a Google account. Details here: https://github.com/GoogleCloudPlatform/ai-platform-text-classifier-shap/issues/1 https://github.com/GoogleCloudPlatform/ai-platform-text-clas...
onurcel 7y ago
If you want to experiment with question tag prediction on your laptop, you can also play with fastText : https://fasttext.cc/docs/en/supervised-tutorial.html https://fasttext.cc/docs/en/supervised-tutorial.html
gridlockd 7y ago
Out of curiosity, does anyone ever use tags on Stackoverflow? Does anybody use search on Stackoverflow? Does anybody ever use in-site search on any website, instead of just using Google?
- jodrellblank 7y ago
  Yes, tags are mandatory when posting a question and useful when looking for questions to answer. And yes, Google is worse than useless, it doesn't even search for what I type in, and when I force it to do that, it returns SEO spam websites, bot-farmed-content, clickbait and advertising ahead of useful content. Or instead of useful content.
  - gridlockd 7y ago
    That's weird, Stackoverflow is almost always my first result. Perhaps by accident you clicked on those spam websites too often and now Google thinks you love them. As for asking, I suppose if my question has never been asked before by someone else, I'd rather not know the answer.
- amrrs 7y ago
  I had a daily practice of looking at the tags of my language of desire `R` and `Python` to see the kind of incoming questions and the variety of answers coming at different instance of time. This is like a break from my full time work let's say after I finish a task or something. Somehow this has helped me improve my coding.
- w-m 7y ago
  Stackoverflow may be the only site where I ever use tags. Tags are useful there for finding questions you can answer. You can watch tags of topics that you probably can reply to and browser the front page with a collection of interesting questions. Or look up the new questions page of a specific tag. The same could probably done with search terms saved to your profile, but the tags are a much more organized alternative.
- tyingq 7y ago
  "Does anybody ever use in-site search on any website, instead of just using Google?" It's certainly common for e-commerce. Especially when you can search by specific product attributes, shipping options, etc.
yeldarb 7y ago
Slightly off topic but I’ve been playing with this dataset for a while now to learn ML. It’s remarkably humbling. No NLP approach I’ve tried has been able to predict question score based on content better than my baseline of “choose the mean”. (I’ve tried random forest on bag of words, AWD-LSTM, and Google AutoML so far). The author of this post tried score prediction as well and pivoted to tag prediction after she couldn’t find anything that worked well: https://twitter.com/srobtweets/status/1125860523377979398?s=21 https://twitter.com/srobtweets/status/1125860523377979398?s=... It’s so crazy to me that which posts get popular might just be random. And makes me wonder about the correlation between post content and popularity on other social sites like HN and reddit.
- existencebox 7y ago
  Quick question: Did you try/find any correlation with post time/seasonality? I recall that in previous examinations of HN/Reddit top/viral posts, there was some amount of signal along that dimension. (I notice someone mentions this in the twitter thread too but I didn't see a response) Additionally, you found average (all up) to be a better predictor than average per user or per category? Apologies for grilling you here, I should frankly dig in myself, but if you happen to feel like indulging me it's much appreciated :)
  - yeldarb 7y ago
    I didn’t try adding in any metadata to the models yet (including time, date, and author). I was just trying to work with the text content of the posts.
- AlexCoventry 7y ago
  It's not that it's random, it's that the signal is too abstract for such simple methods to grasp it. Machine Learning which can predict Stack Overflow question scores is going to have to understand what's being asked, and how useful the responses are going to be for people interested in the question.
  - djakjxnanjak 7y ago
milad_nazari 7y ago
Similarly, predicting issue tags on Github would be interesting.
Darkphibre 7y ago
I've used simple TF/IDF with a custom stemmer as a similarity scoring tool against the XDK documentation. It worked pretty well, I'd be curious to know how that'd fare vs. the neural model (i.e. I suspect it'd be better in identifying the rare, high-signal words that were excluded by the author's 400-most-common word limit). For my side project: As we received emails from developers asking for clarification/help with the APIs, the system would provide relevant documentation URLs so that anyone could pick up an inquiry and brush up on the API (and have a handy link if the the docs could be leveraged in the response).
btgeekboy 7y ago
Between this and the Pi calculation "achievement" it appears Google's all-in on the grassroots marketing these days.
- zitterbewegung 7y ago
  By grassroots marketing you mean paying people to use their services?
  - nullwasamistake 7y ago
    I'm thinking more blatant, like paying people to write about them, then paying for internet points on HN
- boulos 7y ago
  Disclosure: I work on Google Cloud. Both Emma and Sara are in our Developer Relations (aka Dev Rel) organization, like Kelsey Hightower and Felipe Hoffa. They’re explicitly not in Sales, and their job is focused on explaining and demonstrating stuff to Developers. They go to meetups, give invited talks, write blog posts, and so on. Sorry if that doesn’t come across as clear. They work for Google, are paid by Google for that work, but aren’t measured by revenue or anything.
  - seizethecheese 7y ago
    The most effective sales is disguised as “not sales”.
  - ACow_Adonis 7y ago
    I'm not trying to be snarky (I swear every second sentence I say starts that way these days, sad spirit of the times), but to me, that sounds like sales...
  - manigandham 7y ago
    That’s called marketing. Also sometimes pre-sales with solutions engineering and onboarding.
lettergram 7y ago
For those interested, I’ve done similar work in the past. I’ve also written a guide along with explanations of how it works (on sentence classification): https://github.com/lettergram/sentence-classification https://github.com/lettergram/sentence-classification It uses Keras and goes through everything from encodings, to the way various networks function, to hyperparameter tuning.
ACow_Adonis 7y ago
Kaggle ran a facebook recruitment challenge trying to do something similar a few years ago for those interested (I am because I participated in it): https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction https://www.kaggle.com/c/facebook-recruiting-iii-keyword-ext... I'm in a waiting room right now, but is anyone interested in summarising or commenting on the differences/ gains/ losses between the performance and aspects of comparing the two? :)

163 points by amrrs 7y ago

aficionado 7y ago

Links to dataset do not work.

turtlegrids 7y ago
Not sure why you're being downvoted. I also cannot get to at least one dataset -https://storage.googleapis.com/cloudml-demo-lcm/SO_ml_tags_avocado_188k_v2.csv https://storage.googleapis.com/cloudml-demo-lcm/SO_ml_tags_a... which is linked to from the fourth paragraph directly above the "What Is The Bag Of Words Model?" section heading.
- boulos 7y ago
  I pinged Sara. Probably just forgot to make it publicly accessible.
  - turtlegrids 7y ago
    Thanks!
sararob 7y ago
Article author here. To access the dataset you need to be logged in with a Google account. Details here: https://github.com/GoogleCloudPlatform/ai-platform-text-classifier-shap/issues/1 https://github.com/GoogleCloudPlatform/ai-platform-text-clas...

onurcel 7y ago

If you want to experiment with question tag prediction on your laptop, you can also play with fastText : https://fasttext.cc/docs/en/supervised-tutorial.html https://fasttext.cc/docs/en/supervised-tutorial.html

gridlockd 7y ago

Out of curiosity, does anyone ever use tags on Stackoverflow? Does anybody use search on Stackoverflow? Does anybody ever use in-site search on any website, instead of just using Google?

jodrellblank 7y ago
Yes, tags are mandatory when posting a question and useful when looking for questions to answer. And yes, Google is worse than useless, it doesn't even search for what I type in, and when I force it to do that, it returns SEO spam websites, bot-farmed-content, clickbait and advertising ahead of useful content. Or instead of useful content.
- gridlockd 7y ago
  That's weird, Stackoverflow is almost always my first result. Perhaps by accident you clicked on those spam websites too often and now Google thinks you love them. As for asking, I suppose if my question has never been asked before by someone else, I'd rather not know the answer.
amrrs 7y ago
I had a daily practice of looking at the tags of my language of desire `R` and `Python` to see the kind of incoming questions and the variety of answers coming at different instance of time. This is like a break from my full time work let's say after I finish a task or something. Somehow this has helped me improve my coding.
w-m 7y ago
Stackoverflow may be the only site where I ever use tags. Tags are useful there for finding questions you can answer. You can watch tags of topics that you probably can reply to and browser the front page with a collection of interesting questions. Or look up the new questions page of a specific tag. The same could probably done with search terms saved to your profile, but the tags are a much more organized alternative.
tyingq 7y ago
"Does anybody ever use in-site search on any website, instead of just using Google?" It's certainly common for e-commerce. Especially when you can search by specific product attributes, shipping options, etc.

yeldarb 7y ago

Slightly off topic but I’ve been playing with this dataset for a while now to learn ML. It’s remarkably humbling. No NLP approach I’ve tried has been able to predict question score based on content better than my baseline of “choose the mean”. (I’ve tried random forest on bag of words, AWD-LSTM, and Google AutoML so far). The author of this post tried score prediction as well and pivoted to tag prediction after she couldn’t find anything that worked well: https://twitter.com/srobtweets/status/1125860523377979398?s=21 https://twitter.com/srobtweets/status/1125860523377979398?s=... It’s so crazy to me that which posts get popular might just be random. And makes me wonder about the correlation between post content and popularity on other social sites like HN and reddit.

existencebox 7y ago
Quick question: Did you try/find any correlation with post time/seasonality? I recall that in previous examinations of HN/Reddit top/viral posts, there was some amount of signal along that dimension. (I notice someone mentions this in the twitter thread too but I didn't see a response) Additionally, you found average (all up) to be a better predictor than average per user or per category? Apologies for grilling you here, I should frankly dig in myself, but if you happen to feel like indulging me it's much appreciated :)
- yeldarb 7y ago
  I didn’t try adding in any metadata to the models yet (including time, date, and author). I was just trying to work with the text content of the posts.
AlexCoventry 7y ago
It's not that it's random, it's that the signal is too abstract for such simple methods to grasp it. Machine Learning which can predict Stack Overflow question scores is going to have to understand what's being asked, and how useful the responses are going to be for people interested in the question.
- djakjxnanjak 7y ago

milad_nazari 7y ago

Similarly, predicting issue tags on Github would be interesting.

Darkphibre 7y ago

I've used simple TF/IDF with a custom stemmer as a similarity scoring tool against the XDK documentation. It worked pretty well, I'd be curious to know how that'd fare vs. the neural model (i.e. I suspect it'd be better in identifying the rare, high-signal words that were excluded by the author's 400-most-common word limit). For my side project: As we received emails from developers asking for clarification/help with the APIs, the system would provide relevant documentation URLs so that anyone could pick up an inquiry and brush up on the API (and have a handy link if the the docs could be leveraged in the response).

btgeekboy 7y ago

Between this and the Pi calculation "achievement" it appears Google's all-in on the grassroots marketing these days.

zitterbewegung 7y ago
By grassroots marketing you mean paying people to use their services?
- nullwasamistake 7y ago
  I'm thinking more blatant, like paying people to write about them, then paying for internet points on HN
boulos 7y ago
Disclosure: I work on Google Cloud. Both Emma and Sara are in our Developer Relations (aka Dev Rel) organization, like Kelsey Hightower and Felipe Hoffa. They’re explicitly not in Sales, and their job is focused on explaining and demonstrating stuff to Developers. They go to meetups, give invited talks, write blog posts, and so on. Sorry if that doesn’t come across as clear. They work for Google, are paid by Google for that work, but aren’t measured by revenue or anything.
- seizethecheese 7y ago
  The most effective sales is disguised as “not sales”.
- ACow_Adonis 7y ago
  I'm not trying to be snarky (I swear every second sentence I say starts that way these days, sad spirit of the times), but to me, that sounds like sales...
- manigandham 7y ago
  That’s called marketing. Also sometimes pre-sales with solutions engineering and onboarding.

lettergram 7y ago

For those interested, I’ve done similar work in the past. I’ve also written a guide along with explanations of how it works (on sentence classification): https://github.com/lettergram/sentence-classification https://github.com/lettergram/sentence-classification It uses Keras and goes through everything from encodings, to the way various networks function, to hyperparameter tuning.

ACow_Adonis 7y ago

Kaggle ran a facebook recruitment challenge trying to do something similar a few years ago for those interested (I am because I participated in it): https://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction https://www.kaggle.com/c/facebook-recruiting-iii-keyword-ext... I'm in a waiting room right now, but is anyone interested in summarising or commenting on the differences/ gains/ losses between the performance and aspects of comparing the two? :)