3 ms·
>> Thus far, these perturbations have been limited to: + the subtraction of articles (a, an, the) + the subtraction of the second part of a verb contractio
by YeGoblynQueenne 10y ago
>> Thus far, these perturbations have been limited to:
+ the subtraction of articles (a, an, the)
+ the subtraction of the second part of a verb contraction (e.g. “‘ve”, “‘ll”, “‘s”, “‘m”)
+ the replacement of a few common homophones with one of their counterparts (e.g. replacing “their” with “there”, “then” with “than”)
Oooh, that's _very_ tricky what they're trying to do there.
"Perturbations" that cause grammatical sentences to become ungrammatical are
_very_ hard to create, for the absolutely practical reason that the only way
to know whether a sentence is ungrammatical is to check that a grammar rejects
it. And, for English (and generally natural languages) we have no (complete)
such grammars. In fact, that's the whole point of language modelling-
everyone's trying to "model" (i.e. approximate, i.e. guess at) the structure
of English (etc)... because nobody has a complete grammar of it!
Dropping a few bits off sentences may sound like a reasonable alternative (an
approximation of an ungrammaticalising perturbation) but, unfortunately, it's
really, really not that simple.
For instance, take the removal of articles: consider the sentence: "Give him
the flowers". Drop the "the". Now you have "Give him flowers". Which is
perfectly correct and entirely plausible, conversational, everyday English.
In fact, dropping words is de rigeur in language modelling, either to generate
skip-grams for training, or to clean up a corpus by removing "stop words"
(uninformative words like the the and and's) or generally, cruft.
For this reason you'll notice that the NUCLE corpus used in the CoNLL-2014
error correction task mentioned in the OP is not auto-generated, and instead consists of student
essays corrected by professors of English.
tl;dr: You can't rely on generating ungrammaticality unless you can generate
grammaticallity.
- Houshalter 10y agoSo what? Who says the training data has to be perfect? Your comment reminds me of an essay by Peter Norvig. Where he showed that a simple statistical model could tell the difference between a nonsense sentence that was grammatically correct and one that wasn't, but gave them both very low probability of occuring naturally. Similarly, the machine may learn that "Give him the flowers" has higher probability than "give him flowers". Or it may learn that both are possible sentences and not be able to correct it. Which is also OK, we can't expect any system to be perfect.
- YeGoblynQueenne 10y agoYou're talking about the grammatical, but nonsensical sentence "colorless green ideas sleep furiously" proposed by Chomsky, along with its ungrammatical counterpart, "Furiously sleep ideas green colorless." [1] Chomsky famously said that one of the two sentences is ungrammatical, but both would be assigned a low probability by a statistical model trained on English corpora. The one who showed that the ungrammatical sentence was (much) less likely than the grammatical one was Fernando Pereira, not Peter Norvig. Peter Norvig has argued against Chomsky in a different instance, and cited Pereira during that (one-sided) exchange, if memory serves. This is completely unrelated to what I'm saying, of course. The fact of the matter is that you can't just swap some words out of a sentence and call it "ungrammatical". You can call it a "perturbed" sentence. You can call it a "sentence with some words dropped". You can call it Daisy Duck for all I care. But it's not an ungrammatical sentence, because that sort of thing has a very precise definition, which I gave above. _______________ [1] The sentence is famous enough to have a wikipedia page all its own: https://en.wikipedia.org/wiki/Colorless_green_ideas_sleep_furiously https://en.wikipedia.org/wiki/Colorless_green_ideas_sleep_fu...
- unhammer 10y agohttps://encrypted.google.com/search?q=%22give%20him%20flowers%22 https://encrypted.google.com/search?q=%22give%20him%20flower... – if your data set is good enough, you won't be suggesting insertion there. Maybe calling the perturbed (and model-filtered!) sentence "ungrammatical" is too strong a statement, but that doesn't stop the system from being useful. We don't have a perfect model of NL grammar, but nor do we have complete dictionaries of any NL lexicons – we still rely on spell checkers because they have fairly acceptable false positive rates. Spell checkers also do these perturbations: Levenshtein-edits. And some times they generate sequences that the model (dictionary) will remove because it's already in there with some frequency and they generate a true negative, some times they generate sequences that were simply missing from their data set and they generate a false positive, and some times they generate a sequence that is neither in their model nor in the language and you get a true positive. The same general principle applies to grammar checkers (but they of course are much harder to make with an acceptable false positive rate, and it's much more difficult to generate confusion sets).
- YeGoblynQueenne 10y ago>> – if your data set is good enough, you won't be suggesting insertion there. Generally, I doubt you should ever suggest any corrections to the use of articles. This tends to be colloquial, or even personal and there's no point trying to force one style on your users. Unless you like pissing them off (MS Word sure does). As to spellcheckers- I think that goes the other way, doesn't it? You apply some edits to an out-of-vocabulary word you found, to see whether you can arrive at an in-vocabulary word to suggest as a correction. In any case, I'm not disagreeing with anything you say. I'm just saying that "grammatical" and "ungrammatical" have strict definitions and you can't just throw them around like skittles.