3 ms·
> They just usually don't gain much traction because I suspect you're overstating their general utility. :-) I find myself desiring a DFA over LLM tokens right
by LoganDark 3y ago
> They just usually don't gain much traction because I suspect you're overstating their general utility. :-)
I find myself desiring a DFA over LLM tokens right now, which are 16 bits each, at least for this model.
I, personally, am very pleased with the idea of attempting to build a dense DFA with that vocabulary. Maybe your computer would explode!
- telotortium 3y agoCould you not encode the tokens into 8-bit bytes? Rust regex supports byte strings: https://docs.rs/regex/latest/regex/bytes/struct.Regex.html https://docs.rs/regex/latest/regex/bytes/struct.Regex.html
- LoganDark 3y agoI'm dumb. Thank you, I'll look into it.