It looks like this thing doesn't tokenize into anything with any semantic meaning, but rather just a sequence of bytes that match some sort of information theor

It looks like this thing doesn't tokenize into anything with any semantic meaning, but rather just a sequence of bytes that match some sort of information theoretic criteria. It doesn't appear to have any linguistic (written, nor verbal) pattern. I guess it's fine for their specific use case, but whatever. Tokenization is such a basic and domain specific operation, it feels like someone had to demo something. Bonus (code) points for just saying "fuck it" on emojis. They didn't even split it into code points.