4 ms·

How does it compare to a Markov chain generator I wonder.

• jll29 6mo ago
  The Transformer is the more powerful model than Markov chain, but on such a weak machine as the C64, a MC could output text faster - but it surely would sound "psychedelic", as the memory limits a MC to a first-order or second-order model, so to predict one word, only the two words before would be taken into account as context (and no attention). On a plain vanilla C64, the Transformer cannot really show what it's capable of doing. An implementation using 2 bit per weight (vectorized) could be slightly better, perhaps.
  • yorwba 6mo ago
    You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the training data to count possible continuations on the fly: https://arxiv.org/abs/2401.17377 https://arxiv.org/abs/2401.17377 That paper uses suffix arrays, but more compact indices are possible: https://arxiv.org/abs/2506.12229 https://arxiv.org/abs/2506.12229
    • anthk 6mo ago
      Could that be implemented in Hailo for Perl (it's two commands away from a base install:) - Install Cpanminus for Perl, some C compiler and sqlite3 just to be sure. cpanm -n local::lib cpanm -n Hailo ~/perl5/bin/hailo -t corpus.txt -b chatbot.brn ~/perl5/bin/hailo -b chatbot.brn
      • yorwba 6mo ago
        You would probably have to completely rewrite its internals.