4 ms·
Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spellingsociety.org/jour
by DanBC 6y ago
Here's an article that people might be interested in. It gives a bit more detail: https://web.archive.org/web/20100706052342/http://www.spellingsociety.org/journals/j20/spellchecking.php https://web.archive.org/web/20100706052342/http://www.spelli...
I'm particularly interested in this one, and I'm curious about how useful something like this would be to use.
> The second does not use a dictionary at all (Morris & Cherry 1975). Like the previous method, it divides the text into trigrams, but it creates a table of these, noting how often each one occurs in this particular piece of text. It then goes through the text again calculating an index of peculiarity for each word on the basis of the trigrams it contains. Given pkxie, for instance, it would probably find that this was the only word in the text containing pkx and kxi (and possibly xie too), so it would rate it highly peculiar. The word fairy, by contrast, would get a low rating since fai, air and iry probably all occur elsewhere, perhaps quite often, in the passage being analysed. Having completed its analysis, it draws the user's attention to any words with a high peculiarity index. Like the previous method, it would fail to spot a high proportion of ordinary spelling errors, but it is quite good at spotting typing errors, which it was designed for. An advantage that it has over all dictionary-based methods is that it is not tied to English; it will work on passages of, say, French, German or Greek.
(The Morris there is Bob Morris).
- geoduck14 6y agoA while ago, I was considering building a text editor that ran SQL for a very specific purpose at work. I mused about adding a syntax checker/suggested. I wonder if the above approach might be a better approach to a look up table.
- powersnail 6y agoConsider that the most often typo I make is "ture" for "true", and "flase" for "false", I'd say this isn't going to catch some common mistakes.
- warent 6y agoFrom what they said above, it sounds like that's exactly the kind of thing it would catch unless you consistently wrote "ture" and "flase" many times in the same document
- vageli 6y ago> From what they said above, it sounds like that's exactly the kind of thing it would catch unless you consistently wrote "ture" and "flase" many times in the same document The trigrams tur and ure are not that uncommon, nor fla or ase (though maybe they are less common in a programming context, by but even then it might depend on domain).
- lowbloodsugar 6y agoflag case future
- OJFord 6y ago'tur' & 'ure' must be pretty common (turnip, turn, turing, nurture, century; urethra, pure, cure, lure, endure) trigrams though.
- powersnail 6y agoFrom a trigram point of view, they do appear many times.
- adrianN 6y agoI have a bunch of iab in my vimrc for typos like that because I make them so often.
- NegativeLatency 6y agoCare to share some? I use vim and have been meaning to try this for years
- bjoli 6y agoI have abbrev mode i Emacs for that. No more "teh" instead of the. I found most of them went away when I switched to Dvorak, since most of my errors like that are when the letters are in one hand. "teh" however became the all time high. 4000 corrections this year according to abbrev mode. And English isn't even the language I write the most.
- efitz 6y agoIf you just want to fix typos then it seems to me that Damerau-Levenshtein distance would be the best approach - a DL distance of 1 from a dictionary word is a high likelihood of a transposition or a missing or extra character.
- jka 6y agoGreat reference. This highlights an interesting property of 'reasonably-solved' problems like spell checking and why they might be so much more straightforward now. It's often not that we've developed groundbreaking algorithms that make solving the underlying problems intrinsically easier - the techniques we're using (like ngram modeling, in this case) may be the result of research work decades ago. Instead the difference is that the fruit of that prior work has been implemented and made available in more accessible forms (libraries and source code) - and becomes easier to re-use, reason about, and modify thanks to abstraction and languages that have evolved to handle similar problems in a more expressible manner. (upgrades in hardware and resources certainly help advancement too, but spell checking's probably a good example of a situation where an efficiently-designed implementation's likely to be noticeably more responsive, whether it's 1980 or 2020)
- lizardmancan 6y agothats such lovely example of working with what you have
- WalterBright 6y agoI've thought of a variation on that, if you're spell checking a large text file. Create an index of all the unique words in the file, along with how many each appears. Any unique words with a count of 1 are likely to be misspelled.
- ivanhoe 6y agoIt'd probably be better to look for words with small Levenshtein distance that look like a case of typo, because people tend to miss-spell some words repeatedly, especially in larger texts.