5 ms·

I know it's HN folklore to claim "Trivial! Would do over a weekend!", but please... Doing a half-decent production spell checker is STILL a major feat. Same as

by Radim 15y ago

I know it's HN folklore to claim "Trivial! Would do over a weekend!", but please... Doing a half-decent production spell checker is STILL a major feat. Same as "just crawling the web" (further down the discussion). Both require problem understanding and engineering you can't see and appreciate at a glance. And no, looking up individual words in some predefined dictionary doesn't qualify as half-decent spell checking, especially for non-English languages. Spelling correction is another step. "Their coming too sea if its reel."

rcfox 15y ago
Maybe it's not fair to cite the work of super heroes, but Peter Norvig wrote a spelling corrector in 21 lines of Python: http://norvig.com/spell-correct.html http://norvig.com/spell-correct.html
- Radim 15y ago
  It is fair, it is a first step in that direction :-) That corrector has no context though, so it will not correct misspelled words that happen to come out as other words (incl. the "Their coming too sea if its reel."). This is especially important on the web, where pretty much every conceivable word is "correct" (a name of a company or otherwise). False positives are costly, and context disambiguation critical. Let the fun begin.
  - dextorious 15y ago
    You keep using this word "spelling". I don't think it means what you think it means. This is GRAMMAR checking, or at least grammar-assisted spell checking. Very few, if any, shipping mainstream spelling correctors do that.
    - Radim 15y ago
      People want their documents (or queries) free of spelling errors. That is their pain and that is the challenge. The example sentence has misspelled words, hence is within the domain of spell checking. This type of misspelled words are called "homonyms", which are one very common spelling problem. The academic terminology is uninteresting to most users, however. Or if you mean to posit that "spell checking really means looking up if a word exists in a static dictionary of English", then yes, that's easy and solved, no argument there.
      - tincholio 15y ago
        > The example sentence has misspelled words, hence is within the domain of spell checking. No, it doesn't . It is grammatically incorrect, but all the words are spelled correctly. You're definitely talking about a grammar checker, not a spelling checker.
        Radim 15y ago
        I understand this is a mismatch in terminology. In your view, "spelling check" is applied to individual words, to see if they appear in a fixed dictionary (see my other comment about difficulties with choosing this "correct" dictionary in reality, though). I can imagine this view is inviting for programmers, because it's easy to implement, but I doubt anyone else finds useful a definition that says there are no misspellings in "Their coming too sea if its reel." In my view, "spell check" applies to utterances and roughly means "all words are spelled as per the norm of the language; I can send this document to my boss/customer and they won't laugh at my spelling." It is a more user-centric view, and more complex too, because it covers intent and norms, as opposed to the comforting lookup table for a few hand-picked strings. Modern spell checkers make heavy use of statistical analysis of large text corpora, to reasonably approximate context needed to model such intent. Once we agree on the terminology, I believe we are in agreement, so let's not split hairs. The "correctly spelled" sentence under question comes from the Wikipedia article on spell checking, by the way.
        bermanoid 15y ago
        I can imagine this view is inviting for programmers, because it's easy to implement, but I doubt anyone else finds useful a definition that says there are no misspellings in "Their coming too sea if its reel." I realize this is just a debate over a definition, so it's not very meaningful, but the fact is, you're on the wrong side of the common definition here. I just tested, and every spell checker that I just checked (Chrome, Firefox, MS Word, TextMate, TextEdit - perhaps some of these rely on the same underlying engine, I'm not sure?) accepts that sentence as not having a spelling error, so clearly there's some use for such a definition. The grammar checkers, on the other hand, don't like it, but by changing "their" to "they're", they all accept it, despite the fact that it's still garbage. So don't overestimate how good "modern" spell checkers are...though there may be techniques to do a better job, they're not in common use, at least in the most common spell-checking contexts (which, lets be honest, pretty much means MS Word).
    - regularfry 15y ago
      No, it's spell-checking. Use of the word "reel" in this sentence, for instance, is definitely a spelling error. There's no grammatically valid form of the word "real" spelled with two 'e's. The fact that "reel" happens to be a valid word doesn't mean that its presence in the sentence is due to a grammatical error - it's just an accidental collision. That no spell checker might be able to catch this specific class of error doesn't change the type of error it is.
  - Radim 15y ago
    Incidentally, the linked article by Peter Norvig also mentions that adding context is the best way to improve spelling correction. If you think this claim is so controversial, explain yourself, don't just downvote please.
SquareWheel 15y ago
That'd be a grammar checker, that's a whole 'nother beast. That should spell check just fine.
- plq 15y ago
  spell checking for languages that sport a high rate of agglutination where you can't (feasibly) enumerate every possible form of a stem is still a major feat. languages like turkish, finnish and hungarian are prime examples of this.
nandemo 15y ago
That's not the point of the article. He's not talking about writing a state-of-the-art spelling-and-grammar checker. > And no, looking up individual words in some predefined dictionary doesn't qualify as half-decent spell checking, Well, but the author is talking about that problem! Even if you don't consider that real spell-checking, his point still stands. Let's define crappy-spell-checking as "looking up individual words in some predefined dictionary"; that problem used to be hard and now it's very easy, as in, you could write one in 15 minutes using Python.
- Radim 15y ago
  Ok, fair point -- I blame the misleading title :) I read the point of the article to compare "spell-checking then (80s) and now", whereas others read it more along the lines of "looking up static English words then and now". Your nickname sounds japanese, but I assume you're talking about English as well, with those 15 minutes.

by Radim 15y ago

rcfox 15y ago

Maybe it's not fair to cite the work of super heroes, but Peter Norvig wrote a spelling corrector in 21 lines of Python: http://norvig.com/spell-correct.html http://norvig.com/spell-correct.html

Radim 15y ago
It is fair, it is a first step in that direction :-) That corrector has no context though, so it will not correct misspelled words that happen to come out as other words (incl. the "Their coming too sea if its reel."). This is especially important on the web, where pretty much every conceivable word is "correct" (a name of a company or otherwise). False positives are costly, and context disambiguation critical. Let the fun begin.
- dextorious 15y ago
  You keep using this word "spelling". I don't think it means what you think it means. This is GRAMMAR checking, or at least grammar-assisted spell checking. Very few, if any, shipping mainstream spelling correctors do that.
  - Radim 15y ago
    People want their documents (or queries) free of spelling errors. That is their pain and that is the challenge. The example sentence has misspelled words, hence is within the domain of spell checking. This type of misspelled words are called "homonyms", which are one very common spelling problem. The academic terminology is uninteresting to most users, however. Or if you mean to posit that "spell checking really means looking up if a word exists in a static dictionary of English", then yes, that's easy and solved, no argument there.
    - tincholio 15y ago
      > The example sentence has misspelled words, hence is within the domain of spell checking. No, it doesn't . It is grammatically incorrect, but all the words are spelled correctly. You're definitely talking about a grammar checker, not a spelling checker.
      - Radim 15y ago
        I understand this is a mismatch in terminology. In your view, "spelling check" is applied to individual words, to see if they appear in a fixed dictionary (see my other comment about difficulties with choosing this "correct" dictionary in reality, though). I can imagine this view is inviting for programmers, because it's easy to implement, but I doubt anyone else finds useful a definition that says there are no misspellings in "Their coming too sea if its reel." In my view, "spell check" applies to utterances and roughly means "all words are spelled as per the norm of the language; I can send this document to my boss/customer and they won't laugh at my spelling." It is a more user-centric view, and more complex too, because it covers intent and norms, as opposed to the comforting lookup table for a few hand-picked strings. Modern spell checkers make heavy use of statistical analysis of large text corpora, to reasonably approximate context needed to model such intent. Once we agree on the terminology, I believe we are in agreement, so let's not split hairs. The "correctly spelled" sentence under question comes from the Wikipedia article on spell checking, by the way.
        bermanoid 15y ago
        I can imagine this view is inviting for programmers, because it's easy to implement, but I doubt anyone else finds useful a definition that says there are no misspellings in "Their coming too sea if its reel." I realize this is just a debate over a definition, so it's not very meaningful, but the fact is, you're on the wrong side of the common definition here. I just tested, and every spell checker that I just checked (Chrome, Firefox, MS Word, TextMate, TextEdit - perhaps some of these rely on the same underlying engine, I'm not sure?) accepts that sentence as not having a spelling error, so clearly there's some use for such a definition. The grammar checkers, on the other hand, don't like it, but by changing "their" to "they're", they all accept it, despite the fact that it's still garbage. So don't overestimate how good "modern" spell checkers are...though there may be techniques to do a better job, they're not in common use, at least in the most common spell-checking contexts (which, lets be honest, pretty much means MS Word).
  - regularfry 15y ago
    No, it's spell-checking. Use of the word "reel" in this sentence, for instance, is definitely a spelling error. There's no grammatically valid form of the word "real" spelled with two 'e's. The fact that "reel" happens to be a valid word doesn't mean that its presence in the sentence is due to a grammatical error - it's just an accidental collision. That no spell checker might be able to catch this specific class of error doesn't change the type of error it is.
- Radim 15y ago
  Incidentally, the linked article by Peter Norvig also mentions that adding context is the best way to improve spelling correction. If you think this claim is so controversial, explain yourself, don't just downvote please.

SquareWheel 15y ago

That'd be a grammar checker, that's a whole 'nother beast. That should spell check just fine.

plq 15y ago
spell checking for languages that sport a high rate of agglutination where you can't (feasibly) enumerate every possible form of a stem is still a major feat. languages like turkish, finnish and hungarian are prime examples of this.

nandemo 15y ago

That's not the point of the article. He's not talking about writing a state-of-the-art spelling-and-grammar checker. > And no, looking up individual words in some predefined dictionary doesn't qualify as half-decent spell checking, Well, but the author is talking about that problem! Even if you don't consider that real spell-checking, his point still stands. Let's define crappy-spell-checking as "looking up individual words in some predefined dictionary"; that problem used to be hard and now it's very easy, as in, you could write one in 15 minutes using Python.

Radim 15y ago
Ok, fair point -- I blame the misleading title :) I read the point of the article to compare "spell-checking then (80s) and now", whereas others read it more along the lines of "looking up static English words then and now". Your nickname sounds japanese, but I assume you're talking about English as well, with those 15 minutes.