4 ms·
You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the tr
by yorwba 6mo ago
You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the training data to count possible continuations on the fly: https://arxiv.org/abs/2401.17377 https://arxiv.org/abs/2401.17377 That paper uses suffix arrays, but more compact indices are possible: https://arxiv.org/abs/2506.12229 https://arxiv.org/abs/2506.12229
- anthk 6mo agoCould that be implemented in Hailo for Perl (it's two commands away from a base install:) - Install Cpanminus for Perl, some C compiler and sqlite3 just to be sure. cpanm -n local::lib cpanm -n Hailo ~/perl5/bin/hailo -t corpus.txt -b chatbot.brn ~/perl5/bin/hailo -b chatbot.brn
- yorwba 6mo agoYou would probably have to completely rewrite its internals.