4 ms·

Ok so to get a little weirder, could you use a normal tokenizer for pretraining and then go back to bytes for fine tuning along with or before the length extens

• Fripplebubby 3y ago
  Do the bytes represent natural language text, or something else? If they do represent text, then it is not so weird to me.
• kristjansson 3y ago
  Tokens (be they tokens or bytes) basically atoms of input to a LM. So you'd have to figure out how to express the relationship between tokens and bytes to the LM. I guess you could do an incremental thing e.g. - make vocab ~40k tokens + 256 tokens (for each possible byte), - start training with 100% tokens, - then after some portion of train budget has elapsed, randomly replace some tokens with corresponding byte-tokens, - then ramp the fraction of replacements up so you're training on 100% byte tokens for the last xx% of training, without ever exceeding an 8k (or whatever) sequence length - then apply the trick from TFA to get xk -> 8xk/64xk bytes of context? but I'd guess the interesting part of a byte-transformer is multimodality, and we'd need more than a few tricks to get from ^^ to there.