3 ms·
There are many comments remarking on how Jordan because of the large number of stories mentioning Michael Jordan (both of them) is causing problems. Apparently
by nl 11y ago
There are many comments remarking on how Jordan because of the large number of stories mentioning Michael Jordan (both of them) is causing problems.
Apparently people are unaware that this problem is at least partially solved in modern named entity recognition systems.
The first story from a search for "Jordan"[1] is "Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others".
Stanford NLP[2] tags "Michael Jordan" as PERSON.
The first story mentioning Jordan as a country is "Internet Blocking Begins In Jordan". It tags Jordan as LOCATION
Spacy's POS tagging tags "Michael" and "Jordan" as a compound person in the first example[3] and "Jordan" as GPE (geo-poltical entity) in the second[4]
[1] https://hn.algolia.com/?query=Jordan&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=Jordan&sort=byPopularity&prefi...
[2] http://corenlp.run/ http://corenlp.run/ (annoyingly the state can't be passed via URL parameters)
[3] https://api.spacy.io/displacy/index.html?full=Machine-Learning%20Maestro%20Michael%20Jordan%20on%20the%20Delusions%20of%20Big%20Data%20and%20Others https://api.spacy.io/displacy/index.html?full=Machine-Learni...
[4] https://api.spacy.io/displacy/index.html?full=Internet%20Blocking%20Begins%20In%20Jordan https://api.spacy.io/displacy/index.html?full=Internet%20Blo...
- deleted 11y ago[deleted]
- syllogism 11y agoYou can see that the tagger basically works on this specific case by looking at the distributional similarity of the tagged tokens, as well: https://sense2vec.spacy.io/?Jordan%7CGPE https://sense2vec.spacy.io/?Jordan%7CGPE https://sense2vec.spacy.io/?Jordan%7CPERSON https://sense2vec.spacy.io/?Jordan%7CPERSON (This specific word works well because there are plenty of examples of both entities in the training data, which has a lot of text from the 1980s Wall Street Journal. Other examples are much worse. I have high hopes that distributional similarity can help surface these errors, and fix them.)
- lmm 11y agoI imagine it's a lot easier to tag "Michael Jordan" (correctly spelled) as a person than "Micheal Jordan".
- nl 11y agoHa. Fixed, thanks. (Note that the headlines were correctly spelled, so the computer part worked. The failure was at the human end..)