In some ways, I think computational linguistics has missed a mark. We have dictionaries, lexicons, grammar engines, spell checkers, pluralization rules, tries,

In some ways, I think computational linguistics has missed a mark. We have dictionaries, lexicons, grammar engines, spell checkers, pluralization rules, tries, quotation disambiguation, sentiment analyzers, word segmentation, and more. You'd think we could roll all these into a unified, standard definition format. My KeenWrite editor, for instance, uses: * https://github.com/DaveJarvis/KeenSpell https://github.com/DaveJarvis/KeenSpell (spell check, lexicon) * https://github.com/DaveJarvis/KeenQuotes https://github.com/DaveJarvis/KeenQuotes (curls straight quotes) * https://github.com/DaveJarvis/KeenWrite/blob/main/R/pluralize.R https://github.com/DaveJarvis/KeenWrite/blob/main/R/pluraliz... (pluralization) * https://lucene.apache.org/ https://lucene.apache.org/ (word segmentation) I was looking at integrating LanguageTool[0] for grammar and realized that it has partial functionality for KeenQuotes (lexing and tokenization), duplicates the SymSpell algorithm used by KeenSpell, and because it offers grammar corrections it likely can pluralize words, as well. Unifying those for English alone would be a massive undertaking. At present, KeenWrite parses prose in multiple, redundant passes. While this works, I think it'd be great to start with a basic lexer/tokenizer that can emit tokens at blazing speeds[1], which feeds into higher-level abstractions. [0]: https://github.com/languagetool-org/languagetool https://github.com/languagetool-org/languagetool [1]: https://github.com/DaveJarvis/KeenQuotes/blob/main/src/main/java/com/whitemagicsoftware/keenquotes/lex/Lexer.java https://github.com/DaveJarvis/KeenQuotes/blob/main/src/main/...