3 ms·
This sounds very similar to: https://github.com/wolfgarbe/SymSpell https://github.com/wolfgarbe/SymSpell Or maybe I am confused what you're saying. Are you try
by itake 3y ago
This sounds very similar to:
https://github.com/wolfgarbe/SymSpell https://github.com/wolfgarbe/SymSpell
Or maybe I am confused what you're saying. Are you trying to find more creative ways of saying the same phrase?
- zerojames 3y agoI definitely need to write more docs on this project! I will share them when I'm done. It looks like SymSpell is doing spelling correction, which is part of what my code does. The "surprisal" (information theory: Shannon information) component is doing statistical next word prediction. For example, given a corpus of data in which "data augmentation" is a common phrase, "da" could complete to "data augmentation". "data augmentation" starts with "da", and would be determined as a likely candidate for the next word because it was common in the dataset on which the statistical model was "fine-tuned". This Wikipedia page covers the concept in more depth: https://en.wikipedia.org/wiki/Information_content https://en.wikipedia.org/wiki/Information_content
- zerojames 3y agoBy the way, this project is _awesome_! My programming language [1] does OCR correction via heuristics (possible because there is a restricted grammar) but this project looks like it could help _a lot_ with improving the accuracy of the OCR output. Thank you! [1] https://visionscript.dev/paper/ https://visionscript.dev/paper/