3 ms·

One of the Senior Thesis presentations at my college last year was someone working on a context spell-checker. I don't know if his demo could suggest a better w

by dkuntz2 13y ago

One of the Senior Thesis presentations at my college last year was someone working on a context spell-checker. I don't know if his demo could suggest a better word, but it would highlight words that seemed wrong. It was based off of Google's NGrams, I think he used 3-grams, and checked to see how frequently a word showed up between the two words next to it. The problem with that was it required a HUGE amount of data. Like several hundred gigabytes worth of space just to store the 3-grams (compressed down to one instance of each 3-gram coupled with the number of times it showed up in the original dataset).

pbhjpbhj 13y ago
Average vocabulary isn't going to require using that massive a set of 3-grams though. If we started with homophones (https://en.wikipedia.org/wiki/Homophone#English https://en.wikipedia.org/wiki/Homophone#English) that would seem to make a large difference; add in ability to easily get a definition (or list of synonyms) for words/phrases (like in Google Translate). Google suggest already does a lot of what is required.
- dkuntz2 13y ago
  Google also has a ton of computing power and disk space... That said, homophones and homonyms only would make the 3-gram sets smaller, but it would only detect homophones and homonyms being used incorrectly. I could still use a word incorrectly, and if I'm banking on that software to detect my mistakes it wouldn't.
  - pbhjpbhj 13y ago
    Indeed. My false positive rate [flagged errors that are really omissions from the dict.] on spell-check in-browser appears to have been about 95% over the past couple of months. I wouldn't be expecting perfection; surely some mistakes corrected is better than none. How I envision it is also a learning tool - "can here you" would pop up a "can {here} you [here refers to location, hear to hearing sound; 'can hear you']" allowing the author to click the 'corrected' phrase. At the same time you're learning the distinctions. Thanks for the input.

by dkuntz2 13y ago

pbhjpbhj 13y ago

Average vocabulary isn't going to require using that massive a set of 3-grams though. If we started with homophones (https://en.wikipedia.org/wiki/Homophone#English https://en.wikipedia.org/wiki/Homophone#English) that would seem to make a large difference; add in ability to easily get a definition (or list of synonyms) for words/phrases (like in Google Translate). Google suggest already does a lot of what is required.

dkuntz2 13y ago
Google also has a ton of computing power and disk space... That said, homophones and homonyms only would make the 3-gram sets smaller, but it would only detect homophones and homonyms being used incorrectly. I could still use a word incorrectly, and if I'm banking on that software to detect my mistakes it wouldn't.
- pbhjpbhj 13y ago
  Indeed. My false positive rate [flagged errors that are really omissions from the dict.] on spell-check in-browser appears to have been about 95% over the past couple of months. I wouldn't be expecting perfection; surely some mistakes corrected is better than none. How I envision it is also a learning tool - "can here you" would pop up a "can {here} you [here refers to location, hear to hearing sound; 'can hear you']" allowing the author to click the 'corrected' phrase. At the same time you're learning the distinctions. Thanks for the input.