3 ms·
Output text is word salad every few words apart. You can't scale n-gram counting enough to make it work.
by visarga 1y ago
Output text is word salad every few words apart. You can't scale n-gram counting enough to make it work.
- sadiq 1y agoYou might find https://arxiv.org/abs/2401.17377v3 https://arxiv.org/abs/2401.17377v3 interesting..
- JPLeRouzic 1y agoOnly if you have access to corporate-level hardware: "It took us 48 hours to build the suffix array for RedPajama on a single node with 128 CPUs and 1TiB RAM"
- protomikron 1y agoIt's okayish. Considering 64G to 128G are available for (nerd) high-end consumers you're just off with a factor 5 (if we can squeeze out a little bit more performance). Thas is pretty astonishing in my opinion.
- JPLeRouzic 1y agoNot exactly a few words in my experience, I would say every 100 words, if you sophisticate your Markov Chain (n-gram = 3 at minimum, using a good tokenizer, making it tailored to the training data, large training set (500Kbytes or +), intelligent fallback instead of random, etc.).