4 ms·
Chinese Text Analyzer https://www.chinesetextanalyser.com/ https://www.chinesetextanalyser.com/ Tokenizes chinese text into "words" for learning purposes and t
by JCharante 3y ago
Chinese Text Analyzer https://www.chinesetextanalyser.com/ https://www.chinesetextanalyser.com/
Tokenizes chinese text into "words" for learning purposes and then renders the text in a GUI where you can click on a word to get the definition. It's not perfect, but a LLM fine tuned for it will eventually result in much better "tokenization".
- majewsky 3y agoThe same thing for Japanese: https://ichi.moe/ https://ichi.moe/