3 ms·
This is a really good introduction to some natural language processing concepts, and the code as other people mentions is very beautiful and easy to read too.
by slx26 9y ago
This is a really good introduction to some natural language processing concepts, and the code as other people mentions is very beautiful and easy to read too.
Just as a curiosity: I once did some text correction applied to tweets, using word embeddings (word vectors, word2vec), which are a language model too. In real life contexts, where you don't limit yourself to a dictionary, it's really interesting because word vectors can include many foreignisms and other "incorrect" words that you might want to keep in the text.
The article also mentions edit distances, the distances between words (in this case, the "incorrect" word and its possible corrections). In general, algorithms are based on common errors at the keyboard. Missing or adding a letter, transposing two of them, etc. Algorithms like Damerau-Levenshtein (pretty much the one applied in the article, even though it's not explicitly mentioned I think, and there are many variations) keep it simple by doing this, but it's interesting to notice that you can also fine-tune distances by considering which keys are closer to other keys in the keyboard (it's easier to accidentally type "a" instead of "s" in qwerty than "u"), but when doing more general text correction you can also use phonetic distances or even more sophisticated techniques like finite-state transducers that apply language-specific rules to suggest corrections. (I know the article is only meant as an introduction, but thought someone might find it interesting)