3 ms·
My algorithm ( https://ichi.moe/ https://ichi.moe/ ) assumes almost nothing about the grammar and simply divides the sentences into the most valuable chunks (ba
by Grue3 7y ago
My algorithm ( https://ichi.moe/ https://ichi.moe/ ) assumes almost nothing about the grammar and simply divides the sentences into the most valuable chunks (based on the word's length and how common it is), and then some sequences get bonus scores (for example noun followed by a particle). It performs surprisingly well in real life cases (where strict grammar is unlikely to be followed). I think it's the best algorithm that's currently available.
Example text from the blog post:
https://ichi.moe/cl/qr/?q=%E3%81%B2%E3%81%A8%E3%81%8C%E3%81%99%E3%81%94%E3%81%84%E3%80%82%E3%81%92%E3%82%93%E3%81%8D%E3%81%AA%E3%81%B2%E3%81%A8%E3%81%8C%E3%81%84%E3%82%8B%E3%80%82%E3%83%A2%E3%83%B3%E3%82%BF%E3%83%8A%E3%81%8C%E3%81%86%E3%81%A4%E3%81%8F%E3%81%97%E3%81%84%E3%80%82%E3%81%92%E3%82%93%E3%81%8D%E3%81%AA%E3%81%B2%E3%81%A8%E3%81%8C%E3%81%86%E3%81%A4%E3%81%8F%E3%81%97%E3%81%84%E3%83%A2%E3%83%B3%E3%82%BF%E3%83%8A%E3%81%AB%E3%81%84%E3%82%8B%E3%80%82%E3%81%84%E3%82%8B%E3%81%AE%E3%81%8C%E3%81%86%E3%81%A4%E3%81%8F%E3%81%97%E3%81%84%E3%83%A2%E3%83%B3%E3%82%BF%E3%83%8A%E3%81%A0%E3%80%82&r=htr https://ichi.moe/cl/qr/?q=%E3%81%B2%E3%81%A8%E3%81%8C%E3%81%...
- GolDDranks 7y agoWhat you are doing is essentially different: you are doing word sequencing and possibly part-of-speech tagging from a character stream. What is being done in this post, is parsing, which imposes a tree-like structure on the input sequence. These two techniques complement each other, and it might be beneficial to first do the word sequencing (the Viterbi algorithm is a good choice) and parsing after that.