4 ms·

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is pe

A good spell checker is still a hard engineering problem, despite the hardware progress. Just a hash map ain't gonna work. The only reason spell checking is perceived as a solved problem is availability of libraries. Here's an open source example https://github.com/hunspell/hunspell https://github.com/hunspell/hunspell way above 10k lines of code. I speak 4 languages, and in my experience what's in Microsoft Office is the best one I used so far.

jiggawatts 6y ago
To be fair, Hunspell was written specifically for Hungarian because it's such a difficult language to handle efficiently in a spell checker. It just happens to also work for other languages. If you only need English, the complexity of Hunspell is not required. I still haven't found a decent Hungarian spell checker, they get confused by rare words that have the same letters as a very common word, but different accents. E.g.: The word "és" means "and", which makes it much more common than "es", which is the word root for "fall" and is rarely used without a suffix like "esett" or "estek". I'm yet to see a spellchecker that's smart enough to fix "something es something" but not "le es".
- unpopularopp 6y ago
  >but not "le es" First of all it's a verb with -ik ending so you can't write "es", the root lexical word is "esik". Second you never use anything like "le es" because generaly you always have to use the verb and prefix together ("leesett", "leesik" etc.) unless you use a commanding form ("ess le") or a modal verb ("le akart esni, "le fog esni" etc.) "le es" is strictly wrong because it's the wrong verb (should be "esik") and you have to write together ("leesik") so obviously a spellchecker picks that up.
  - jiggawatts 6y ago
    Okay, I admit, my Hungarian is not so good these days. But if you're right (seems like you are), then why do spell checkers not pick up on "es" as misspelled?
    - unpopularopp 6y ago
      Most likely because it's used as a suffix too. Mostly for numerals: "I arrive with the train at 7" [as in 7 o'clock] = "A 7-es vonattal érkezem" "M7 motorway" = "M7-es autópálya" "50m² room" = "50m²-es szoba" But it's also used with foreign names and full names.
oneearedrabbit 6y ago
I suppose, it is worth giving a shout-out to a recent Hunspell port to Python by Zverok: https://github.com/zverok/spylls https://github.com/zverok/spylls, this description from Github sums it up nicely: > Hunspell is a long-living, complicated, almost undocumented piece of software, and it was our feeling that the significant part of human knowledge is somehow "locked" in a form of a large C++ project. That's how Spylls was born: as an attempt to "unlock" it, via well-structured and well-documented implementation in a high-level language. It's incredible how much work has been done (along with documenting algorithms!) in this one-man project.
- indeyets 6y ago
  Exactly. It's a very important argument against calling it a solved problem. I believe the problem with hunspell is that it's very hard to get into fixing it (it's good, but it's not perfect) and spylls makes it feasible again
- mellavora 6y ago
  my hovercraft is full of eels.
Aissen 6y ago
If french is one of your language, state of the art has been Druide's Antidote for more than 20 years. And it has had a Linux version available for almost 15 years, too.
- jeltz 6y ago
  It is similar for Swedish. As far as I know nobody has surpassed stava despite it being about 25 years old.
vbezhenar 6y ago
I would argue that it's not a hard engineering problem. It's just a problem that should be solved. Linguists made a lot of formalizations about language. There are rules, there are known lists of exceptions, there are dictionaries. You just have to implement those things carefully with lots of edge-cases, implement it for every language, but it's just mundane engineering work, nothing that requires breakthrough. Currently spell checking just checks words from a dictionary. Its 1% of work. Or even less. Microsoft Word used to have awesome spell checking, grammar checking, etc back in 2000 for Russian language. It seems to be degraded since then. But the fact is, this problem was solved. It checked for spelling, for grammar, for punctuation. I would expect that kind of functionality working in every OS textbox by now.
- anthk 6y ago
  >You just have to implement those things carefully with lots of edge-cases, Say hello to homophones, ambiguous phrases and double meaning. In Spanish a lot of verbs share the singular first and third persons. "Estaba" can mean "I was" or "He/she was.". Or "fuimos", (we were); but also "we went".

jiggawatts 6y ago

To be fair, Hunspell was written specifically for Hungarian because it's such a difficult language to handle efficiently in a spell checker. It just happens to also work for other languages. If you only need English, the complexity of Hunspell is not required. I still haven't found a decent Hungarian spell checker, they get confused by rare words that have the same letters as a very common word, but different accents. E.g.: The word "és" means "and", which makes it much more common than "es", which is the word root for "fall" and is rarely used without a suffix like "esett" or "estek". I'm yet to see a spellchecker that's smart enough to fix "something es something" but not "le es".

unpopularopp 6y ago
>but not "le es" First of all it's a verb with -ik ending so you can't write "es", the root lexical word is "esik". Second you never use anything like "le es" because generaly you always have to use the verb and prefix together ("leesett", "leesik" etc.) unless you use a commanding form ("ess le") or a modal verb ("le akart esni, "le fog esni" etc.) "le es" is strictly wrong because it's the wrong verb (should be "esik") and you have to write together ("leesik") so obviously a spellchecker picks that up.
- jiggawatts 6y ago
  Okay, I admit, my Hungarian is not so good these days. But if you're right (seems like you are), then why do spell checkers not pick up on "es" as misspelled?
  - unpopularopp 6y ago
    Most likely because it's used as a suffix too. Mostly for numerals: "I arrive with the train at 7" [as in 7 o'clock] = "A 7-es vonattal érkezem" "M7 motorway" = "M7-es autópálya" "50m² room" = "50m²-es szoba" But it's also used with foreign names and full names.

oneearedrabbit 6y ago

I suppose, it is worth giving a shout-out to a recent Hunspell port to Python by Zverok: https://github.com/zverok/spylls https://github.com/zverok/spylls, this description from Github sums it up nicely: > Hunspell is a long-living, complicated, almost undocumented piece of software, and it was our feeling that the significant part of human knowledge is somehow "locked" in a form of a large C++ project. That's how Spylls was born: as an attempt to "unlock" it, via well-structured and well-documented implementation in a high-level language. It's incredible how much work has been done (along with documenting algorithms!) in this one-man project.

indeyets 6y ago
Exactly. It's a very important argument against calling it a solved problem. I believe the problem with hunspell is that it's very hard to get into fixing it (it's good, but it's not perfect) and spylls makes it feasible again
mellavora 6y ago
my hovercraft is full of eels.

Aissen 6y ago

If french is one of your language, state of the art has been Druide's Antidote for more than 20 years. And it has had a Linux version available for almost 15 years, too.

jeltz 6y ago
It is similar for Swedish. As far as I know nobody has surpassed stava despite it being about 25 years old.

vbezhenar 6y ago

I would argue that it's not a hard engineering problem. It's just a problem that should be solved. Linguists made a lot of formalizations about language. There are rules, there are known lists of exceptions, there are dictionaries. You just have to implement those things carefully with lots of edge-cases, implement it for every language, but it's just mundane engineering work, nothing that requires breakthrough. Currently spell checking just checks words from a dictionary. Its 1% of work. Or even less. Microsoft Word used to have awesome spell checking, grammar checking, etc back in 2000 for Russian language. It seems to be degraded since then. But the fact is, this problem was solved. It checked for spelling, for grammar, for punctuation. I would expect that kind of functionality working in every OS textbox by now.

anthk 6y ago
>You just have to implement those things carefully with lots of edge-cases, Say hello to homophones, ambiguous phrases and double meaning. In Spanish a lot of verbs share the singular first and third persons. "Estaba" can mean "I was" or "He/she was.". Or "fuimos", (we were); but also "we went".