correct, I worked with the tokenizer quite a lot, and I should've clearly made the distinction between semantic splitting and just this statistically-convenient

correct, I worked with the tokenizer quite a lot, and I should've clearly made the distinction between semantic splitting and just this statistically-convenient splitting. I got carried out by the greater picture. thanks for pointing this, I would've updated the post if I could I didn't say we should drop LLMs, your point about them still getting better and better is one reason why. they're very useful and we'll only ditch them after something better comes out