3 ms·

I ran it through our stuff we do fine with segmentation of your comments. We still have trouble with certain poorly formatted numbers, or if you do something l

by drakaal 14y ago

I ran it through our stuff we do fine with segmentation of your comments. We still have trouble with certain poorly formatted numbers, or if you do something like for get a space after a sentence that ends in a number. "I'll take 2.In case I need one later." Typos are really hard for rules systems. PS Glad you didn't hate it.

arafalov 14y ago
Hah, you thought that was hard! Try one of these: http://www.undocs.org/A/Res/66/2 http://www.undocs.org/A/Res/66/2 (start from finding the end of the first sentence). Or, you can have a go at a whole corpus of these broken at paragraph level at http://www.uncorpora.org/ http://www.uncorpora.org/
mchaver 14y ago
Aren't you arguing against prescriptivism and not linguists (as in people with a background in Linguistics)? My background is Linguistics, Computer Science and NLP. From what I was taught, and what seems to be the norm, is linguists tend to do descriptive linguistics. You learn from your data and change the rules and methods accordingly, not the other way around. Anyway, interesting read. I am curious about one thing. Are you just handling white spaced words, or do you handle lexical items as well such like polywords (inside out), phrasal verbs (put up with, beat up), idioms, etc.?
- drakaal 14y ago
  We handle polywords. We had to. Noun adjuncts being one of the worst in terms of how often they show up, and I'll be honest I don't even know what you call Gerunds as Adjectives like "Running shorts" but those are a pain too. But you can kind of find ways to get lists of those. Parts catalogs, shopping sites. People want to buy polyword nouns. There are fewer of the verbs but they are harder to find lists of. Idioms we haven't addressed much. Fortunately in news they come up less than other things. Sarcasm we deal with in most cases, but not all. There are hints that things are sarcastic, but if they are written like "the Onion" and they don't change tone because they are really a parody rather than sarcasm we can't tell that. I like to say that if your 5 year old would figure it out we will figure it out. If your 5 year old won't probably we won't either. But this is a huge leap forward since the competition is about the level where your dog would understand.

by drakaal 14y ago

arafalov 14y ago

Hah, you thought that was hard! Try one of these: http://www.undocs.org/A/Res/66/2 http://www.undocs.org/A/Res/66/2 (start from finding the end of the first sentence). Or, you can have a go at a whole corpus of these broken at paragraph level at http://www.uncorpora.org/ http://www.uncorpora.org/

mchaver 14y ago

Aren't you arguing against prescriptivism and not linguists (as in people with a background in Linguistics)? My background is Linguistics, Computer Science and NLP. From what I was taught, and what seems to be the norm, is linguists tend to do descriptive linguistics. You learn from your data and change the rules and methods accordingly, not the other way around. Anyway, interesting read. I am curious about one thing. Are you just handling white spaced words, or do you handle lexical items as well such like polywords (inside out), phrasal verbs (put up with, beat up), idioms, etc.?

drakaal 14y ago
We handle polywords. We had to. Noun adjuncts being one of the worst in terms of how often they show up, and I'll be honest I don't even know what you call Gerunds as Adjectives like "Running shorts" but those are a pain too. But you can kind of find ways to get lists of those. Parts catalogs, shopping sites. People want to buy polyword nouns. There are fewer of the verbs but they are harder to find lists of. Idioms we haven't addressed much. Fortunately in news they come up less than other things. Sarcasm we deal with in most cases, but not all. There are hints that things are sarcastic, but if they are written like "the Onion" and they don't change tone because they are really a parody rather than sarcasm we can't tell that. I like to say that if your 5 year old would figure it out we will figure it out. If your 5 year old won't probably we won't either. But this is a huge leap forward since the competition is about the level where your dog would understand.