5 ms·

How? just get started working on a fun problem. A good place to start is keyword extraction. You don't need a PhD or expensive tools. All you need is some free

How? just get started working on a fun problem. A good place to start is keyword extraction. You don't need a PhD or expensive tools. All you need is some free time and willingness to read some cool stuff. Copy a few articles into text files and get working on implementing some of these methods until you have enough of an understanding to construct your own methods for the fun of it. Here's some good reading material: https://www.facebook.com/notes/facebook-engineering/under-the-hood-the-entities-graph/10151490531588920/ https://www.facebook.com/notes/facebook-engineering/under-th... https://www.researchgate.net/profile/Stuart_Rose/publication/227988510_Automatic_Keyword_Extraction_from_Individual_Documents/links/55071c570cf27e990e04c8bb.pdf https://www.researchgate.net/profile/Stuart_Rose/publication... http://cdn.intechopen.com/pdfs/5338.pdf http://cdn.intechopen.com/pdfs/5338.pdf https://arxiv.org/pdf/1603.03827v1.pdf https://arxiv.org/pdf/1603.03827v1.pdf https://www.quora.com/Sentiment-Analysis-What-are-the-good-ways-to-extract-topics-keywords-from-a-text-paragraph-article https://www.quora.com/Sentiment-Analysis-What-are-the-good-w... http://hrcak.srce.hr/file/207669 http://hrcak.srce.hr/file/207669 http://nlp.stanford.edu/fsnlp/promo/colloc.pdf http://nlp.stanford.edu/fsnlp/promo/colloc.pdf https://arxiv.org/ftp/cs/papers/0410/0410062.pdf https://arxiv.org/ftp/cs/papers/0410/0410062.pdf http://delivery.acm.org/10.1145/1120000/1119383/p216-hulth.pdf?ip=90.152.24.166&id=1119383&acc=OPEN&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E6D218144511F3437&CFID=890263035&CFTOKEN=90663898&__acm__=1484938921_1ba7ae06773c926d1f8efae701324613 http://delivery.acm.org/10.1145/1120000/1119383/p216-hulth.p... Edit: Don't get deterred by the math formulas in these papers. They look far more complicated than they actually are.

garysieling 10y ago
Another fun thing is to paste article text into some API, like the Watson demo, so you can see what kinds of things are possible: https://alchemy-language-demo.mybluemix.net/ https://alchemy-language-demo.mybluemix.net/ I played around with this a bit to develop https://www.findlectures.com https://www.findlectures.com, so knowing what works/doesn't work there I'm developing some NLP scripts to support my use cases.
- demonshalo 10y ago
  I never thought about this particular use-case. The subtitle for TED talks should be an ocean of info for you to extract keywords from :D Pretty neat site you got there. I will be using it. Thanks!
bosie 10y ago
Thank you very much for the reading material
- demonshalo 10y ago
  u welcome!
jventura 10y ago
I would say that a good example for starting in this field would be to implement something like Tf-Idf [0] for identifying keywords on a set of documents. I don't know where one can find current datasets for this, but I made WikiCorpusExtractor [1] to build sets of documents from the Wikipedia. The only thing one really needs is to count the frequency of words in each document and do very simple math. Tf-Idf is still very relevant today and provides you with a very good idea on how statistics is used on text-mining. [0] https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf [1] https://github.com/joaoventura/WikiCorpusExtractor https://github.com/joaoventura/WikiCorpusExtractor
- demonshalo 10y ago
  I started even simpler than that. I started by just eliminating stopwords and count the frequency in each word in the document itself. I did not use a set of documents as the goal was for the algorithm to be used on the spot for a single block of text. A few months later and after many iterations + a whole lot of testing, the algorithm now can extract super relevant keywords 90%+ of the time! I wish I knew about the WikiCorpusExtractor. Thanks for the link!

garysieling 10y ago

Another fun thing is to paste article text into some API, like the Watson demo, so you can see what kinds of things are possible: https://alchemy-language-demo.mybluemix.net/ https://alchemy-language-demo.mybluemix.net/ I played around with this a bit to develop https://www.findlectures.com https://www.findlectures.com, so knowing what works/doesn't work there I'm developing some NLP scripts to support my use cases.

demonshalo 10y ago
I never thought about this particular use-case. The subtitle for TED talks should be an ocean of info for you to extract keywords from :D Pretty neat site you got there. I will be using it. Thanks!

bosie 10y ago

Thank you very much for the reading material

demonshalo 10y ago
u welcome!

jventura 10y ago

I would say that a good example for starting in this field would be to implement something like Tf-Idf [0] for identifying keywords on a set of documents. I don't know where one can find current datasets for this, but I made WikiCorpusExtractor [1] to build sets of documents from the Wikipedia. The only thing one really needs is to count the frequency of words in each document and do very simple math. Tf-Idf is still very relevant today and provides you with a very good idea on how statistics is used on text-mining. [0] https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf [1] https://github.com/joaoventura/WikiCorpusExtractor https://github.com/joaoventura/WikiCorpusExtractor

demonshalo 10y ago
I started even simpler than that. I started by just eliminating stopwords and count the frequency in each word in the document itself. I did not use a set of documents as the goal was for the algorithm to be used on the spot for a single block of text. A few months later and after many iterations + a whole lot of testing, the algorithm now can extract super relevant keywords 90%+ of the time! I wish I knew about the WikiCorpusExtractor. Thanks for the link!