> The tokenizer covers the entire dataset. Well, this is only trivially true. You can feed binary data to the LLM and it probably has tokens that only cover si

> The tokenizer covers the entire dataset. Well, this is only trivially true. You can feed binary data to the LLM and it probably has tokens that only cover single bytes of that.