4 ms·

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the

Interesting idea. I went ahead and tested: > Alex went to the kitchen to store the milk in the fridge. Corrected: > Alex went to the kitchen to the store the milk in the fridge. Gathering a large, high quality dataset from the internet is probably not so easy. A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Movie dialogues don't necessarily consist of grammatically correct sentences like the ones you'd write in a letter. Perhaps there is some public domain contemporary literature available that could be used instead or alongside the dialogues? EDIT: Unrelated to this project, I have this general fear of language recommendation tools trained on just low-quality comments or emails. A simple thesaurus and a grammar-checker are often enough to find the right words when writing. But a tool that could understand my intent and then propose restructured or similar sentences and words that convey the same meaning could be a true killer application.

babuskov 10y ago
> A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Corrected: > A lot of the content on the HN/Reddit/forums is of the quality grammatically and non-native written by written English speakers (such myself). as UNK Yeah. It's got a long way to go. No idea where "as UNK" came from.
- pilooch 10y ago
  UNK usually comes from the final sampling step: the distribution of words contains a special token UNK to represent all words with insufficient statistics in the training corpus.
- pizza 10y ago
  It's like the equivalent of NaN; what you put when you don't know what should be there. coincidentally, the op corrected this as: It's like the equivalent of NaN
tannerc 10y ago
> A tool that could understand my intent and then propose structure or similar sentences and words I'm working on this exact thing right now fwiw, in my app called Prompts. I imagine machine assisted apps are going to percolate up over the coming years, or months, in a way we haven't seen yet. It's pretty exciting, if we can get it right. Written language seems like a decent place to start.
atpaino 10y ago
Yep, it definitely has room to improve. The work thus far has primarily been a proof-of-concept for the methodology used to generate training samples (i.e. starting with grammatically correct text and introducing errors). Next step is to try to include more high quality data, after which I may try out comment data from HN, etc. I think it would be interesting to see what the effect of somewhat noisier data like that would have on the model.
- walrus1066 10y ago
  Programmatically generating incorrect grammar from correct sentences must be really tough. There are so many more ways to incorrectly structure a sentence than there are correct ones. Random idea: what happens if you use Google translate to generate the incorrect sentence, I.e. Translate it to other languages and then back again. If the resulting sentence doesn't match the original, add it to the dataset.
aembleton 10y ago
Dropping `in the fridge` works. "Alex went to kitchen to store the milk" corrects to "Alex went to the kitchen to store the milk"
- lern_too_spel 10y ago
  On the other hand, it never removes unnecessary articles, demonstrating one of the deficiencies of the training set.
  - hashhar 10y ago
    Couldn't that be rectified with a Parts of Speech tagger? Or am I overlooking something?
    - lern_too_spel 10y ago
      The training data never says to remove any articles, so a PoS tagger won't help.
walrus1066 10y ago
Interesting example, seems the algo cannot infer whether 'store' is the verb or the noun. It would need to understand the meaning and context of the sentence to do this. Maybe it's also to do with how they generate the training data. Author did say removal of articles was one thing they used to generate the 'incorrect' sentence in the training data. If the film data uses 'the store' much more than 'store', you can imagine how the algo could get biased to thinking store is always preceded with 'the'.
jwong_ 10y ago
I also got some weird results in the small sample of testing I did. > The player chose to go to the good team rather than the sucky one. > The player chose to go to the good team than rather the sucky one. And it seems to like duplicating adjectives: > Being worse off than even worse off makes one feel sad. > Being worse worse than even worse worse makes one feel sad.
Paul-ish 10y ago
Technical documentation may be dry, but it is generally well read and edited, grammatically correct, and easily obtained.
YeGoblynQueenne 10y ago
>> But a tool that could understand my intent and then propose restructured or similar sentences and words that convey the same meaning could be a true killer application. Yeah but that's hard though. Who knows what is your intent when you're generating an utterance, written or spoken? Sometimes you yourself may even read what you wrote (or hear what you said) and wonder what you meant. Not to mention what an absolute nightmare it would be, trying to compile data on the linguistic intent behind utterances! How do you even start to collect that? Ask people to say things, then ask them what they meant... but what did they mean when they explain what they meant in the first instance? The worse thing is that the very notion of what is grammatical changes with context. For instance, to go back to dropping the "the"'s: imagine you read the phrase "eat soup with spoon". Is that an ungrammatical form of "eat the soop with the spoon", or is it an instruction, perhaps something you'd find in a soup-eating manual, which therefore is perfectly valid as a terse form of English? What you intend an utterance to mean affects whether it is grammatical and the grammaticality of the utterance affects its meaning. Nice, eh?
- nmstoker 10y ago
  You make a great point. Presumably biasing the training sets to those built from corrected sources would help, but what about using n-grams to throw out poor examples of ungrammatical sentences? It wouldn't be perfect, but some kind of score based on a much lower prevlence of n-grams from the generated sentence vs the original might indicate acceptable cases and those where they were fairly similar might indicate the generated sentence could in fact be valid (and thus should be discarded)
- deleted 10y ago
  [deleted]

babuskov 10y ago

> A lot of the content on HN/Reddit/forums is of low quality grammatically and often written by non-native English speakers (such as myself). Corrected: > A lot of the content on the HN/Reddit/forums is of the quality grammatically and non-native written by written English speakers (such myself). as UNK Yeah. It's got a long way to go. No idea where "as UNK" came from.

pilooch 10y ago
UNK usually comes from the final sampling step: the distribution of words contains a special token UNK to represent all words with insufficient statistics in the training corpus.
pizza 10y ago
It's like the equivalent of NaN; what you put when you don't know what should be there. coincidentally, the op corrected this as: It's like the equivalent of NaN

tannerc 10y ago

> A tool that could understand my intent and then propose structure or similar sentences and words I'm working on this exact thing right now fwiw, in my app called Prompts. I imagine machine assisted apps are going to percolate up over the coming years, or months, in a way we haven't seen yet. It's pretty exciting, if we can get it right. Written language seems like a decent place to start.

atpaino 10y ago

Yep, it definitely has room to improve. The work thus far has primarily been a proof-of-concept for the methodology used to generate training samples (i.e. starting with grammatically correct text and introducing errors). Next step is to try to include more high quality data, after which I may try out comment data from HN, etc. I think it would be interesting to see what the effect of somewhat noisier data like that would have on the model.

walrus1066 10y ago
Programmatically generating incorrect grammar from correct sentences must be really tough. There are so many more ways to incorrectly structure a sentence than there are correct ones. Random idea: what happens if you use Google translate to generate the incorrect sentence, I.e. Translate it to other languages and then back again. If the resulting sentence doesn't match the original, add it to the dataset.

aembleton 10y ago

Dropping `in the fridge` works. "Alex went to kitchen to store the milk" corrects to "Alex went to the kitchen to store the milk"

lern_too_spel 10y ago
On the other hand, it never removes unnecessary articles, demonstrating one of the deficiencies of the training set.
- hashhar 10y ago
  Couldn't that be rectified with a Parts of Speech tagger? Or am I overlooking something?
  - lern_too_spel 10y ago
    The training data never says to remove any articles, so a PoS tagger won't help.

walrus1066 10y ago

Interesting example, seems the algo cannot infer whether 'store' is the verb or the noun. It would need to understand the meaning and context of the sentence to do this. Maybe it's also to do with how they generate the training data. Author did say removal of articles was one thing they used to generate the 'incorrect' sentence in the training data. If the film data uses 'the store' much more than 'store', you can imagine how the algo could get biased to thinking store is always preceded with 'the'.

jwong_ 10y ago

I also got some weird results in the small sample of testing I did. > The player chose to go to the good team rather than the sucky one. > The player chose to go to the good team than rather the sucky one. And it seems to like duplicating adjectives: > Being worse off than even worse off makes one feel sad. > Being worse worse than even worse worse makes one feel sad.

Paul-ish 10y ago

Technical documentation may be dry, but it is generally well read and edited, grammatically correct, and easily obtained.

YeGoblynQueenne 10y ago

>> But a tool that could understand my intent and then propose restructured or similar sentences and words that convey the same meaning could be a true killer application. Yeah but that's hard though. Who knows what is your intent when you're generating an utterance, written or spoken? Sometimes you yourself may even read what you wrote (or hear what you said) and wonder what you meant. Not to mention what an absolute nightmare it would be, trying to compile data on the linguistic intent behind utterances! How do you even start to collect that? Ask people to say things, then ask them what they meant... but what did they mean when they explain what they meant in the first instance? The worse thing is that the very notion of what is grammatical changes with context. For instance, to go back to dropping the "the"'s: imagine you read the phrase "eat soup with spoon". Is that an ungrammatical form of "eat the soop with the spoon", or is it an instruction, perhaps something you'd find in a soup-eating manual, which therefore is perfectly valid as a terse form of English? What you intend an utterance to mean affects whether it is grammatical and the grammaticality of the utterance affects its meaning. Nice, eh?

nmstoker 10y ago
You make a great point. Presumably biasing the training sets to those built from corrected sources would help, but what about using n-grams to throw out poor examples of ungrammatical sentences? It wouldn't be perfect, but some kind of score based on a much lower prevlence of n-grams from the generated sentence vs the original might indicate acceptable cases and those where they were fairly similar might indicate the generated sentence could in fact be valid (and thus should be discarded)
deleted 10y ago
[deleted]