4 ms·
Accuracy still seems to leave a fair bit to be desired. For example, when parsing "Blue, Brown, Orange, Green and Red Lines were running normally" from this ne
by nostrademons 10y ago
Accuracy still seems to leave a fair bit to be desired. For example, when parsing "Blue, Brown, Orange, Green and Red Lines were running normally" from this news article [1], "Blue" was interpreted as the color [2], "Brown" was interpreted as the Brown Bears football team [3], "Orange" was interpreted as the Orange Line Washington (not Chicago) Metro [4], "Green" was interpreted as "environmentally friendly" [5], and "Red lines" was interpreted as the organization (but without a Wikipedia page).
Works pretty well on places, though.
[1] http://chicago.cbslocal.com/2011/02/02/dangerous-blizzards-wrath-continues/ http://chicago.cbslocal.com/2011/02/02/dangerous-blizzards-w...
[2] http://en.wikipedia.org/wiki/Blue http://en.wikipedia.org/wiki/Blue
[3] http://en.wikipedia.org/wiki/Brown_Bears_football http://en.wikipedia.org/wiki/Brown_Bears_football
[4] http://en.wikipedia.org/wiki/Orange_Line_(Washington_Metro) http://en.wikipedia.org/wiki/Orange_Line_(Washington_Metro)
[5] http://en.wikipedia.org/wiki/Environmentally_friendly http://en.wikipedia.org/wiki/Environmentally_friendly
- bduerst 10y agoHmm, which NLP request did you run it as? It seemed to run better when I ran it through analyzeEntities: http://pastebin.com/raw/AfXvKAzw http://pastebin.com/raw/AfXvKAzw
- nostrademons 10y agoIt was in context - I pasted the whole text of the CBS Local article I linked to into the "Try it out!" text field on the product home page, and these were the entities it identified.
- dmorr 10y agoGoogle Research PM here -- my team built the language understanding tech that powers the API. Thanks for checking it out! You picked a really interesting, and really hard, sentence to use to test us with. It has a couple of interesting phenomena: a reduced conjunction ("Line" goes with each color to make a name, like "Blue Line" even though "Blue" and "Line" are far apart), and high ambiguity ("Green" could be the color, the environmental movement, the political party, one of several people, or lots of other things (https://en.wikipedia.org/wiki/Green_(disambiguation) https://en.wikipedia.org/wiki/Green_(disambiguation) ). These are hard! So hard, in fact, that I'm reasonably sure that there's no system in existence that would get these ones right. (I hope I'm wrong, actually, I'd love to see approaches that can solve problems like this generally.) Our systems are state-of-the-art, or in some cases better than any other published system. But language is really hard, and even the world's best systems are way worse than any human at understanding language. That's what makes working on this stuff so much fun and so challenging. It feels so easy for us as humans, but we just haven't figured out how to model all this so that the computers can do as well. (I'm going to steal this sentence to use internally as a great "NLP is hard" example, thanks!)
- dominotw 10y ago>"Green" could be the color, the environmental movement, the political party, one of several people, or lots of other things It says 'train line' right in that sentence. Where is the ambiguity? even if it missed that why didn't any of the 'cta' , 'chicago' , 'train', 'run' or 'line' nudge it in the right direction. It seems to have identified 'cta' entity correctly but completely ignored that context for the next words in the sentence.
- kafkaesq 10y agoIt says 'train line' right in that sentence. Where is the ambiguity? That's the thing -- in your (wetware) mind it says "train line." But the sentence itself it just says "line", which can mean a whole much of things.
- dominotw 10y agoBut the sentence itself it just says "line", which can mean a whole much of things. Ok i changed the sentence to include 'train lines' CTA buses were moving, but slowly, and ‘L’ train lines were operating, but with delays. Blue, Brown, Orange, Green and Red Lines were running normally. Even this is no good CTA buses were moving, but slowly, and ‘L’ train lines were operating, but with delays. Green train line was operating normally. didn't make any difference. Were you able to get it to parse correctly somehow?
- kafkaesq 10y agoNow we have a different problem -- in that the new examples aren't idiomatic. There's no reason we should expect it to identify the phrases 'L' train line and 'L' line with equal accuracy, because the former is something basically nobody says -- while the latter is something everyone (even someone unfamiliar with that particular city) understands. But hey -- I'm not defending the API's accuracy ;)
- nostrademons 10y agoIt does say "'L' trains" in the previous sentence and "trains" in the following sentence. And from bduerst's post, it appears that GCNLP does use non-local information to determine an interpretation. So in theory, there should be enough information within the paragraph to identify these. The tricky part is that I don't really see a way that a computer could disambiguate between "train line" and "bus line" given the information in that paragraph; the only way that you could do this is to have knowledge of Chicago's transit system. (Indeed, as a human, I didn't know for sure that it was talking about a train line rather than some other mass transit system until Googling.)
- euske 10y agoLet's ask a rhetorical question: Who'd use this kind of information? My answer: It's mostly search/listing apps. The biggest merit of tagging/parsing for now is that you can search the sentences who might possibly refer to some specific subway line. Toward this goal, you don't need an exact interpretation of each phrase. What you'd need is a high-level function that asks if "sentence X".refers_to("Blue Line") and the API should be designed for this kind of goals in mind. I think the current premise that you can/should get a exact parse/tagging for a given text is kinda misleading, because an interpretation of natural language is fundamentally vague. IMO, we should go after the performance of these high-level tasks (e.g. searching or translation) rather than microscopic accuracy. It sucks that we wouldn't be able to parse the text once for all and cache the canonical representation for future use, and we'd always have to run the matching API or some sort. But I think it's kinda the nature of NLP.