7 ms·
Not sure if this is the right thread to ask, wanted to ask in https://news.ycombinator.com/item?id=13445255 https://news.ycombinator.com/item?id=13445255 but I
by wiradikusuma 10y ago
Not sure if this is the right thread to ask, wanted to ask in https://news.ycombinator.com/item?id=13445255 https://news.ycombinator.com/item?id=13445255 but I missed the boat:
How difficult is it to use NLP to do keyword extraction for non-English languages? Any pointer to get started? (not NLP in general, but how to tailor it for non-English)
- gattilorenz 10y agoAs long as you have a tokenizer, i.e. you can split words in a sentence, language doesn't matter for some basic techniques. Two examples that easily come to mind: TF-IDF [1] and TextRank [2,3]. These are good to get started, but if you want to know more about the state of the art, searching on Google Scholar for "keyword extraction $language" is your best bet (and maybe "keyword extraction overview", also in Scholar). [1] http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extraction/ http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extrac... [2] http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1/high_res_d/Mihalcea-2004-TextRank-Bringing_Order_into_Texts.pdf http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1... [3] https://github.com/davidadamojr/TextRank https://github.com/davidadamojr/TextRank
- EmilStenstrom 10y agoKeyword extraction is not easy for other languages, but POS tagging and parsing is easy. There's a project called UDPipe which has pretrained models for most languages. That's the best "general" solution I've found.