5 ms·
I use Stanford NER to extract book titles and authors from HN comments. Previous thread: https://news.ycombinator.com/item?id=14202557 https://news.ycombinator.
by jmstfv 9y ago
I use Stanford NER to extract book titles and authors from HN comments. Previous thread: https://news.ycombinator.com/item?id=14202557 https://news.ycombinator.com/item?id=14202557
- wyldfire 9y agoHow do you disambiguate similar/identical titles? Did you evaluate other NER against CoreNLP? e.g. how does spacy compare? EDIT: answering my own question -- research here [1] ranks CoreNLP as the winner among CoreNLP, NLTK, spaCy, Lingpipe but AFAICT spaCy is competitive. [1] https://aclweb.org/anthology/W/W16/W16-2703.pdf https://aclweb.org/anthology/W/W16/W16-2703.pdf
- jmstfv 9y agoI found spacy's documentation lacking in details on how to train NER from scratch (at least I couldn't make sense of it). That's why I decided to stick with Stanford's NER. Regarding disambiguation: People usually mention books with their respective authors, so it takes one query against the local database to check whether entity is indeed a particular book by a particular author. When no author is mentioned, I check parent comment (if any) to find if the given book is mentioned there. Well, and if comment turns out to be standalone then I query Goodreads / Google Books API. Given books with identical titles, these APIs return the most popular option.