3 ms·
Wow. That's like saying a person's name isn't important. That's like saying Latin words aren't relevant or Spanish words aren't relevant for processing English.
by freehunter 9y ago
Wow. That's like saying a person's name isn't important. That's like saying Latin words aren't relevant or Spanish words aren't relevant for processing English. To explain why this is bad since if the author is confused(see edit), others may be as well: Twitter mentions are contextual data. They're either addressing someone by name or they're being used in a cheeky way to replace another word.
I could quite easily say "I'm going to @chicago this weekend to see @taylor_swift". Now see if this sentence makes any sense: "I'm going to this weekend to see". Nope. What you need to do is translate them, like any other word. Sure it's not an English word, but you wouldn't ignore the word "hola" just because it's not an English word. Now, if your NLP application doesn't rely on this data, sure, throw it away. But if you're looking at Twitter and throwing away any mentions, you're not really processing natural language, are you?
Sure, it's really hard to translate. Am I talking about Chicago the city or Chicago the band? Maybe Chicago the movie?
Well that's why NLP is one of the most challenging areas of research, and nothing in this article will help solve even 0.009% of those challenges.
(edit) Maybe confused isn't the right word... they may just have other priorities for their use case. However, that doesn't excuse the title.
- minimaxir 9y agoThe approach I've seen to handling Twitter mentions is to replace all mentions with a "<userid>" token, which handles that potential loss of context, although not perfectly.
- ppod 9y ago>Now, if your NLP application doesn't rely on this data, sure, throw it away. But if you're looking at Twitter and throwing away any mentions, you're not really processing natural language, are you? The data is natural language, and your system is processing it. NLP doesn't mean "using computation to model exactly and completely the entire human language faculty". NLP is distinct from text processing in that often text files may contain forms of unstructured information other than natural language. The data being processed here is natural language.