3 ms·
I get the point of the article that today's improved hardware allows things that weren't possible before. But neither spell checking nor its more advanced sibl
by wolfgarbe 6y ago
I get the point of the article that today's improved hardware allows things that weren't possible before.
But neither spell checking nor its more advanced siblings spelling correction or automatic spelling correction is solved.
Here is an example:
"I want to by Apple"
"I want to be Apple"
"I want to buy Apple"
Because "by" is a genuine English word, a spellchecker based on a simple hashtable will not detect the error, which obviously exists.
We need to look at the context of the whole sentence or even document. Today we can use deep learning and word embedding to solve/improve that problem.
A simple spelling corrector that is based on edit distance would always prefer "be" over "buy" because "be" is the more frequent word of both.
So the risk of all the easy accessible libraries for a certain task is, that you think the problem is solved, but you overlook many problems because the topic is much more diverse and deep than you would think at the first glance
Another point is that while hardware improved, also the expectations have risen. We expect milliseconds latency instead of seconds, we often require the software to work in a multi-tenant architecture with thousands or millions of concurrent users.
So even for seemingly simple and straightforward tasks, and despite all the exciting advances in hardware, always new challenges present themselves.
- MHard 6y agoThat would be a grammar checker though. And there are many solutions available today to do that. Even open source solutions. Grammarly would be one example, language tool another. The ones I looked at work by encoding common spelling mistakes into a grammar (as in Chomsky grammar) and then running that over the text.
- wolfgarbe 6y agoNo, that's not grammar checking, but spelling mistakes that turn out to be real words! If you misspelled "college" as "collage", or you misspelled "three" as "tree" the word you typed incorrectly happens to be an actual word itself! Correcting these types of errors is called "real word spelling correction". Real-word spelling errors are words in a text that, although correctly spelled words in the dictionary, are not the words that the writer intended https://www.researchgate.net/publication/221628953_Real-Word_Spelling_Correction_with_Trigrams_A_Reconsideration_of_the_Mays_Damerau_and_Mercer_Model https://www.researchgate.net/publication/221628953_Real-Word... And even if there are several approaches evaluated, the problem is far from solved.
- pbhjpbhj 6y agoIt's so weird that spellchecking isn't context sensitive: both locality and use. Locality: The pair "buy Apple" must be vastly preferred, like millions of times more. Use: I'm writing this and my swipe keyboard offers "spellcasting" when I want "spellchecking". The page I'm on is about spellchecking and the other word isn't one I've ever used until now. You can split these down in to prior-use and use-context, I guess: the former is most annoying, always having to correct in the same manner.