BPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it

BPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it uses an existing semantic model for ‘what normal text looks like’ to choose potentially useful tokens (even for nonsensical text, which is why it works on stuff like code) And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.) Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…