8 ms·

Stanford Natural Language Parser

142 points by aphextron 10y ago

brudgers 10y ago
Parser home: http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml
m0th87 10y ago
Looks like it can even handle garden path sentences [1] like "the complex houses married and single soldiers and their families." 1: https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence
- charlieegan3 10y ago
  Do any dependency parsers do this well? I think 'houses' throws them off, e.g.: https://demos.explosion.ai/displacy/?text=The%20complex%20houses%20married%20and%20single%20soldiers%20and%20their%20families.&model=en&cpu=1&cph=1 https://demos.explosion.ai/displacy/?text=The%20complex%20ho... vs. https://demos.explosion.ai/displacy/?text=The%20complex%20contains%20married%20and%20single%20soldiers%20and%20their%20families.&model=en&cpu=1&cph=1 https://demos.explosion.ai/displacy/?text=The%20complex%20co...
- ma2rten 10y ago
  I don't think this parser parses left-to-right, like humans and other parsers like Parsley MacParseface do.
  - botexpert 10y ago
    It does. left-to-right linear time parsing is the idea that's about 15 years old and turned out to be most efficient and most performant.
    - ma2rten 10y ago
      I might be wrong, but based on the parser home page it looks to me like they provide a Shift-reduce parser but it is not the default. I even says on the page Parser: englishPCFG.ser.gz.
      - botexpert 10y ago
        True, it might not be the case, although they do have the exact same beam search shift-reduce parser that mcparseface has (without the neural net).
  - kuschku 10y ago
    I’m currently writing a paper on improving natural language interfaces for databases, and I’ve ran both Parsey McParseface and Stanford’s CoreNLP for that, and both have horrible performance. Their results for even the most simple english sentences are horrible, often with multiple mistagged words. As soon as you add relative clauses, they break down entirely.
    - YeGoblynQueenne 10y ago
      Well, it's not really a solved problem, is it? Saying the performance is "horrible" does some injustice to it. It's actually the state of the art in solutions to a very difficult problem. I've used the Stanford parser for university work (at Masters level). It was part of the pipeline for a sentiment extractor. I didn't get the feeling it performed badly, quite the contrary- the parses it generated were quite useful as features in my extractor. It had this rare feeling of a tool that you can actually use to do something interesting and useful. Then again, I do have my expectations set very lowly, for this sort of thing, especially after completing my Masters thesis (on grammar induction). Language learning is hard.
      - kuschku 10y ago
        I mean, it's surprising how well it works, but if you come in, and expect to get a perfect parse tree out of everything, so you can then easily pattern match on top of that, well, you'll be in for a rouch surprise. Any every slightly more complicated sentence was completely misparsed by CoreNLP and ParseyMcParseface. As I mentioned before, as soon as you introduce relative clauses it breaks down. It's hard when your research was supposed to discuss how to better handle the common knowledge problem of NLIDBs, but you have to spend a lot of time just to get a kinda useful parse out in the first place.
- malaya_zemlya 10y ago
  "The horse raced past the barn fell" still trips it.
- invaliduser 10y ago
  You usually can't tell from one single sentence if a parser handles garden path sentences, which involve backtracking (hence the name). Plus, there is a part-of-speech tagger that provides an intermediate layer of nlp analysis, which makes a backtracking involving a change of part-of-speech («houses» being either a verb of a noun) quite harder (yet not impossible). POS taggers improve parsers performance dramatically but usually are an indication that the parser have limited backtracking capabilities. You can try it with «the complex houses married soldiers», houses is erroneously marked as a noun.
  - YeGoblynQueenne 10y ago
    Yup. It gets it fundamentally wrong, thinking "complex" is an adjective and "houses" is a noun: the/DT complex/JJ houses/NNS married/JJ and/CC single/JJ soldiers/NNS and/CC their/PRP$ families/NNS But, honest, that's a rotten way to try out NL parsers. Nobody claims Stanford's, or anyone's, parser can handle that sort of thing. It throws human language organs because it's specially engineered to do that. And if it can confuse humans, there's no program that it can't confuse, except for one where the right parse is hard-coded. It's an unfair test, is what I'm saying.
- kwelstr 10y ago
  Try: Time flies like the wind but fruit flies like bananas.
charlieegan3 10y ago
I always thought http://corenlp.run http://corenlp.run was a better demo of the tool. Sadly it's no longer available: https://github.com/stanfordnlp/CoreNLP/issues/273 https://github.com/stanfordnlp/CoreNLP/issues/273 You could use this Dockerfile to run a local copy if you're interested: https://gist.github.com/charlieegan3/910276eef0f8658b44b42af268599931 https://gist.github.com/charlieegan3/910276eef0f8658b44b42af...
aminorex 10y ago
This software kinda sucks. I guess you get what you pay for, sometimes.
- aminorex 10y ago
  It is crazy slow and uses absurd amounts of memory, as well as being simultaneously an object lesson in both over-engineering, and under-engineering. Built by academics, so it's not unexpected. Nice example of the application of recent NLP thesis topics to parsing problems, though. Quite suitable as documentation and reference for, e.g. future work on SpaCy - which is much more usable in productive contexts.
  - stephenr 10y ago
    Yes, it does use a reasonable chunk of memory (I wouldn't run in with less than 3GB available) However, speed wise, its only slow in my experience, if you try to integrate via a shell script - if you run it as a service and make http calls it's been amazingly fast in my usage. I believe they include a "service" mode as part of the package now, but for a project a little while ago I forked & extended a project that provides a JSON/XML http service: https://github.com/Koalephant/StanfordCoreNLPHTTPServer https://github.com/Koalephant/StanfordCoreNLPHTTPServer
  - ronack 10y ago
    In my testing it struck a good balance between accuracy and performance. I was also impressed by the lengths the group goes to help people on their mailing list. That said, their licensing is not friendly so I generally have to rule it out.
jackfoxy 10y ago
Thanks to Sergey Tihon in Minsk the Stanford NLP library gets made available as a .NET library about as fast as new releases come out. http://sergey-tihon.github.io/Stanford.NLP.NET/ http://sergey-tihon.github.io/Stanford.NLP.NET/
- fowlerpower 10y ago
  Just checked that out. It looks fantastic, big thanks for sharing. I haven't seen too many things for NLP on .NET. I have run across a ton things on Python or C++/C. Would love to see more inn .NET.
EternalData 10y ago
Ah, but can it tell when I'm trying to be witty?
- deleted 10y ago
  [deleted]
- kobeya 10y ago
  That's not its job.
  - EternalData 10y ago
    I understand that -- as a parser, of course it's not its job to understand sarcasm or wit. Just referencing the old Chomskian bit about statistical correlates and how relevant it is to understanding the fundamental principles behind the expression of language (rather than merely its syntax). I guess it was a bad joke gone over. ah well
amelius 10y ago
Previous discussion: https://news.ycombinator.com/item?id=11273240 https://news.ycombinator.com/item?id=11273240 Unfortunately with zero comments.
YeGoblynQueenne 10y ago
My favourite example phrase to mess with NL parsers: Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo Stanford fails miserably on this one, even at the tagging step: Buffalo/NNP buffalo/NNP Buffalo/NNP buffalo/JJ buffalo/JJ buffalo/NN Buffalo/NNP buffalo/NNP ... but then, most humans also find that sentence really hard to parse. Check it out: https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...
- xtiansimon 10y ago
  That's great. But let me ask, technically, how is the sentence "a grammatically correct sentence in American English" without punctuation? For NLP, shouldn't this first be regarded as a 'special case' of unpunctuated sentences? (and BTW, the parser doesn't handle the punctuated version any better).
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
  - sthatipamala 10y ago
    It is in fact grammatically correct without punctuation, just somewhat awkward. Consider this analogous sentence with the same meaning, with optional parentheticals added to help you parse: New York oxen (that) New York oxen bully (themselves) bully New York oxen.
    - gcb0 10y ago
      the fact that nobody can understand without a preposition and a pronoun even after you replaced the uncommon words with common ones tells me this is a bunch a buffalo excrement that barely entertain linguist phds.
  - jcranmer 10y ago
mrcactu5 10y ago
Generic sentence from newspaper article. I heard Stanford uses Java? (ROOT (S (PP (IN In) (S (VP (VBG keeping) (PP (IN with) (NP (PRP$ his) (JJ insurgent) (NN campaign)))))) (, ,) (NP (NNP President) (NNP Trump)) (VP (VP (VBD dispensed) (PP (IN with) (NP (NNS appeals))) (PP (TO to) (NP (NN unity)))) (CC or) (VP (VBZ attempts) (S (VP (TO to) (VP (VB build) (NP (NNS bridges)) (PP (TO to) (NP (NP (PRP$ his) (NNS opponents)) (PP (IN in) (NP (PRP$ his) (JJ inaugural) (NN address)))))))))) (. .)))
mark_l_watson 10y ago
It is a nice statistical parser. I did note that it does not do anaphora resolution (map pronouns to original referred to nouns, etc.)
- gtani 10y ago
  did you try http://nlp.stanford.edu/projects/coref.shtml http://nlp.stanford.edu/projects/coref.shtml
  - mark_l_watson 10y ago
    Thanks, I have tried that online but it is not available as a library. My kbsportal library does collocation, but not well.
ma2rten 10y ago
Does anyone here use parsers and know of practical applications for them?
- victor9000 10y ago
  My main use case is unsupervised feature extraction from blob text. One application may be to extract all noun entities from a news article, so you can automatically figure out who the article is about.
  - ma2rten 10y ago
    Good idea! However, chunking (shallow parsing) might be enough for that purpose.
- garysieling 10y ago
  Features you can use for other stuff, like entity detection. E.g. AlchemyAPI has a cool demo, where they can detect names of diseases, cities, etc. I suspect the parts of speech would improve the accuracy for tagging words that have multiple meanings. I used the disease bit on AlchemyAPI for https://www.findlectures.com https://www.findlectures.com, to make facets for a health topics: https://www.findlectures.com/?p=1&class1=Science&category_l2_Science=Health https://www.findlectures.com/?p=1&class1=Science&category_l2...
- trumpeta 10y ago
  I use this parser in particular at work designing a closed-domain chatbot.
  - ma2rten 10y ago
    Do you mind going into more detail, since most chatbots just use rules and keyword words.
bpodgursky 10y ago
Taking the opportunity to plug my own D3 visualization using CoreNLP : ) http://nlpviz.bpodgursky.com/ http://nlpviz.bpodgursky.com/
douche 10y ago
I really wanted to use the .NET port of this, but unfortunately it is GPL... dead in the water for commercial software...
- deleted 10y ago
  [deleted]
congerous 10y ago
The assumptions we make, the more performance goes down in tasks like machine translation, and that includes the assumption that parts of speech matter to machines.
wiradikusuma 10y ago
Not sure if this is the right thread to ask, wanted to ask in https://news.ycombinator.com/item?id=13445255 https://news.ycombinator.com/item?id=13445255 but I missed the boat: How difficult is it to use NLP to do keyword extraction for non-English languages? Any pointer to get started? (not NLP in general, but how to tailor it for non-English)
- gattilorenz 10y ago
  As long as you have a tokenizer, i.e. you can split words in a sentence, language doesn't matter for some basic techniques. Two examples that easily come to mind: TF-IDF [1] and TextRank [2,3]. These are good to get started, but if you want to know more about the state of the art, searching on Google Scholar for "keyword extraction $language" is your best bet (and maybe "keyword extraction overview", also in Scholar). [1] http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extraction/ http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extrac... [2] http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1/high_res_d/Mihalcea-2004-TextRank-Bringing_Order_into_Texts.pdf http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1... [3] https://github.com/davidadamojr/TextRank https://github.com/davidadamojr/TextRank
- EmilStenstrom 10y ago
  Keyword extraction is not easy for other languages, but POS tagging and parsing is easy. There's a project called UDPipe which has pretrained models for most languages. That's the best "general" solution I've found.
0x54MUR41 10y ago
My undergraduate research used Stanford Parser for getting word tags in a sentence before calculating its similarity with another sentence. Stanford Parser results are good, but it fails in some sentences. If you need a better alternative than this (for JVM languages), I would recommend NLP4J [1]. [1]: https://emorynlp.github.io/nlp4j/ https://emorynlp.github.io/nlp4j/

142 points by aphextron 10y ago

Parser home: http://nlp.stanford.edu/software/lex-parser.shtml http://nlp.stanford.edu/software/lex-parser.shtml

Looks like it can even handle garden path sentences [1] like "the complex houses married and single soldiers and their families." 1: https://en.wikipedia.org/wiki/Garden_path_sentence https://en.wikipedia.org/wiki/Garden_path_sentence

charlieegan3 10y ago
Do any dependency parsers do this well? I think 'houses' throws them off, e.g.: https://demos.explosion.ai/displacy/?text=The%20complex%20houses%20married%20and%20single%20soldiers%20and%20their%20families.&model=en&cpu=1&cph=1 https://demos.explosion.ai/displacy/?text=The%20complex%20ho... vs. https://demos.explosion.ai/displacy/?text=The%20complex%20contains%20married%20and%20single%20soldiers%20and%20their%20families.&model=en&cpu=1&cph=1 https://demos.explosion.ai/displacy/?text=The%20complex%20co...
ma2rten 10y ago
I don't think this parser parses left-to-right, like humans and other parsers like Parsley MacParseface do.
- botexpert 10y ago
  It does. left-to-right linear time parsing is the idea that's about 15 years old and turned out to be most efficient and most performant.
  - ma2rten 10y ago
    I might be wrong, but based on the parser home page it looks to me like they provide a Shift-reduce parser but it is not the default. I even says on the page Parser: englishPCFG.ser.gz.
    - botexpert 10y ago
      True, it might not be the case, although they do have the exact same beam search shift-reduce parser that mcparseface has (without the neural net).
- kuschku 10y ago
  I’m currently writing a paper on improving natural language interfaces for databases, and I’ve ran both Parsey McParseface and Stanford’s CoreNLP for that, and both have horrible performance. Their results for even the most simple english sentences are horrible, often with multiple mistagged words. As soon as you add relative clauses, they break down entirely.
  - YeGoblynQueenne 10y ago
    Well, it's not really a solved problem, is it? Saying the performance is "horrible" does some injustice to it. It's actually the state of the art in solutions to a very difficult problem. I've used the Stanford parser for university work (at Masters level). It was part of the pipeline for a sentiment extractor. I didn't get the feeling it performed badly, quite the contrary- the parses it generated were quite useful as features in my extractor. It had this rare feeling of a tool that you can actually use to do something interesting and useful. Then again, I do have my expectations set very lowly, for this sort of thing, especially after completing my Masters thesis (on grammar induction). Language learning is hard.
    - kuschku 10y ago
      I mean, it's surprising how well it works, but if you come in, and expect to get a perfect parse tree out of everything, so you can then easily pattern match on top of that, well, you'll be in for a rouch surprise. Any every slightly more complicated sentence was completely misparsed by CoreNLP and ParseyMcParseface. As I mentioned before, as soon as you introduce relative clauses it breaks down. It's hard when your research was supposed to discuss how to better handle the common knowledge problem of NLIDBs, but you have to spend a lot of time just to get a kinda useful parse out in the first place.
malaya_zemlya 10y ago
"The horse raced past the barn fell" still trips it.
invaliduser 10y ago
You usually can't tell from one single sentence if a parser handles garden path sentences, which involve backtracking (hence the name). Plus, there is a part-of-speech tagger that provides an intermediate layer of nlp analysis, which makes a backtracking involving a change of part-of-speech («houses» being either a verb of a noun) quite harder (yet not impossible). POS taggers improve parsers performance dramatically but usually are an indication that the parser have limited backtracking capabilities. You can try it with «the complex houses married soldiers», houses is erroneously marked as a noun.
- YeGoblynQueenne 10y ago
  Yup. It gets it fundamentally wrong, thinking "complex" is an adjective and "houses" is a noun: the/DT complex/JJ houses/NNS married/JJ and/CC single/JJ soldiers/NNS and/CC their/PRP$ families/NNS But, honest, that's a rotten way to try out NL parsers. Nobody claims Stanford's, or anyone's, parser can handle that sort of thing. It throws human language organs because it's specially engineered to do that. And if it can confuse humans, there's no program that it can't confuse, except for one where the right parse is hard-coded. It's an unfair test, is what I'm saying.
kwelstr 10y ago
Try: Time flies like the wind but fruit flies like bananas.

charlieegan3 10y ago

I always thought http://corenlp.run http://corenlp.run was a better demo of the tool. Sadly it's no longer available: https://github.com/stanfordnlp/CoreNLP/issues/273 https://github.com/stanfordnlp/CoreNLP/issues/273 You could use this Dockerfile to run a local copy if you're interested: https://gist.github.com/charlieegan3/910276eef0f8658b44b42af268599931 https://gist.github.com/charlieegan3/910276eef0f8658b44b42af...

aminorex 10y ago

This software kinda sucks. I guess you get what you pay for, sometimes.

aminorex 10y ago
It is crazy slow and uses absurd amounts of memory, as well as being simultaneously an object lesson in both over-engineering, and under-engineering. Built by academics, so it's not unexpected. Nice example of the application of recent NLP thesis topics to parsing problems, though. Quite suitable as documentation and reference for, e.g. future work on SpaCy - which is much more usable in productive contexts.
- stephenr 10y ago
  Yes, it does use a reasonable chunk of memory (I wouldn't run in with less than 3GB available) However, speed wise, its only slow in my experience, if you try to integrate via a shell script - if you run it as a service and make http calls it's been amazingly fast in my usage. I believe they include a "service" mode as part of the package now, but for a project a little while ago I forked & extended a project that provides a JSON/XML http service: https://github.com/Koalephant/StanfordCoreNLPHTTPServer https://github.com/Koalephant/StanfordCoreNLPHTTPServer
- ronack 10y ago
  In my testing it struck a good balance between accuracy and performance. I was also impressed by the lengths the group goes to help people on their mailing list. That said, their licensing is not friendly so I generally have to rule it out.

jackfoxy 10y ago

Thanks to Sergey Tihon in Minsk the Stanford NLP library gets made available as a .NET library about as fast as new releases come out. http://sergey-tihon.github.io/Stanford.NLP.NET/ http://sergey-tihon.github.io/Stanford.NLP.NET/

fowlerpower 10y ago
Just checked that out. It looks fantastic, big thanks for sharing. I haven't seen too many things for NLP on .NET. I have run across a ton things on Python or C++/C. Would love to see more inn .NET.

EternalData 10y ago

Ah, but can it tell when I'm trying to be witty?

deleted 10y ago
[deleted]
kobeya 10y ago
That's not its job.
- EternalData 10y ago
  I understand that -- as a parser, of course it's not its job to understand sarcasm or wit. Just referencing the old Chomskian bit about statistical correlates and how relevant it is to understanding the fundamental principles behind the expression of language (rather than merely its syntax). I guess it was a bad joke gone over. ah well

amelius 10y ago

Previous discussion: https://news.ycombinator.com/item?id=11273240 https://news.ycombinator.com/item?id=11273240 Unfortunately with zero comments.

YeGoblynQueenne 10y ago

My favourite example phrase to mess with NL parsers: Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo Stanford fails miserably on this one, even at the tagging step: Buffalo/NNP buffalo/NNP Buffalo/NNP buffalo/JJ buffalo/JJ buffalo/NN Buffalo/NNP buffalo/NNP ... but then, most humans also find that sentence really hard to parse. Check it out: https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...

xtiansimon 10y ago
That's great. But let me ask, technically, how is the sentence "a grammatically correct sentence in American English" without punctuation? For NLP, shouldn't this first be regarded as a 'special case' of unpunctuated sentences? (and BTW, the parser doesn't handle the punctuated version any better).
- deleted 10y ago
  [deleted]
- deleted 10y ago
  [deleted]
- sthatipamala 10y ago
  It is in fact grammatically correct without punctuation, just somewhat awkward. Consider this analogous sentence with the same meaning, with optional parentheticals added to help you parse: New York oxen (that) New York oxen bully (themselves) bully New York oxen.
  - gcb0 10y ago
    the fact that nobody can understand without a preposition and a pronoun even after you replaced the uncommon words with common ones tells me this is a bunch a buffalo excrement that barely entertain linguist phds.
- jcranmer 10y ago

mrcactu5 10y ago

Generic sentence from newspaper article. I heard Stanford uses Java? (ROOT (S (PP (IN In) (S (VP (VBG keeping) (PP (IN with) (NP (PRP$ his) (JJ insurgent) (NN campaign)))))) (, ,) (NP (NNP President) (NNP Trump)) (VP (VP (VBD dispensed) (PP (IN with) (NP (NNS appeals))) (PP (TO to) (NP (NN unity)))) (CC or) (VP (VBZ attempts) (S (VP (TO to) (VP (VB build) (NP (NNS bridges)) (PP (TO to) (NP (NP (PRP$ his) (NNS opponents)) (PP (IN in) (NP (PRP$ his) (JJ inaugural) (NN address)))))))))) (. .)))

mark_l_watson 10y ago

It is a nice statistical parser. I did note that it does not do anaphora resolution (map pronouns to original referred to nouns, etc.)

gtani 10y ago
did you try http://nlp.stanford.edu/projects/coref.shtml http://nlp.stanford.edu/projects/coref.shtml
- mark_l_watson 10y ago
  Thanks, I have tried that online but it is not available as a library. My kbsportal library does collocation, but not well.

ma2rten 10y ago

Does anyone here use parsers and know of practical applications for them?

victor9000 10y ago
My main use case is unsupervised feature extraction from blob text. One application may be to extract all noun entities from a news article, so you can automatically figure out who the article is about.
- ma2rten 10y ago
  Good idea! However, chunking (shallow parsing) might be enough for that purpose.
garysieling 10y ago
Features you can use for other stuff, like entity detection. E.g. AlchemyAPI has a cool demo, where they can detect names of diseases, cities, etc. I suspect the parts of speech would improve the accuracy for tagging words that have multiple meanings. I used the disease bit on AlchemyAPI for https://www.findlectures.com https://www.findlectures.com, to make facets for a health topics: https://www.findlectures.com/?p=1&class1=Science&category_l2_Science=Health https://www.findlectures.com/?p=1&class1=Science&category_l2...
trumpeta 10y ago
I use this parser in particular at work designing a closed-domain chatbot.
- ma2rten 10y ago
  Do you mind going into more detail, since most chatbots just use rules and keyword words.

bpodgursky 10y ago

Taking the opportunity to plug my own D3 visualization using CoreNLP : ) http://nlpviz.bpodgursky.com/ http://nlpviz.bpodgursky.com/

douche 10y ago

I really wanted to use the .NET port of this, but unfortunately it is GPL... dead in the water for commercial software...

deleted 10y ago
[deleted]

congerous 10y ago

The assumptions we make, the more performance goes down in tasks like machine translation, and that includes the assumption that parts of speech matter to machines.

wiradikusuma 10y ago

Not sure if this is the right thread to ask, wanted to ask in https://news.ycombinator.com/item?id=13445255 https://news.ycombinator.com/item?id=13445255 but I missed the boat: How difficult is it to use NLP to do keyword extraction for non-English languages? Any pointer to get started? (not NLP in general, but how to tailor it for non-English)

gattilorenz 10y ago
As long as you have a tokenizer, i.e. you can split words in a sentence, language doesn't matter for some basic techniques. Two examples that easily come to mind: TF-IDF [1] and TextRank [2,3]. These are good to get started, but if you want to know more about the state of the art, searching on Google Scholar for "keyword extraction $language" is your best bet (and maybe "keyword extraction overview", also in Scholar). [1] http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extraction/ http://www.joyofdata.de/blog/tf-idf-statistic-keyword-extrac... [2] http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1/high_res_d/Mihalcea-2004-TextRank-Bringing_Order_into_Texts.pdf http://digital.library.unt.edu/ark%3A/67531/metadc30962/m2/1... [3] https://github.com/davidadamojr/TextRank https://github.com/davidadamojr/TextRank
EmilStenstrom 10y ago
Keyword extraction is not easy for other languages, but POS tagging and parsing is easy. There's a project called UDPipe which has pretrained models for most languages. That's the best "general" solution I've found.

0x54MUR41 10y ago

My undergraduate research used Stanford Parser for getting word tags in a sentence before calculating its similarity with another sentence. Stanford Parser results are good, but it fails in some sentences. If you need a better alternative than this (for JVM languages), I would recommend NLP4J [1]. [1]: https://emorynlp.github.io/nlp4j/ https://emorynlp.github.io/nlp4j/