4 ms·Yes, but doesn't the token change mean that?by lkbm 6mo agoYes, but doesn't the token change mean that?clickety_clack 6mo agoYou can train a tokenizer on old data just like you can train a model on old data.wongarsu 6mo agoBut you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratchdannyw 6mo agoA little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.