3 ms·

Does this mean we can skip "tokenization" and feed raw bytes into our models now?

by sp332 3y ago

kristjansson 3y ago
That's been on the table e.g. [0][1]. But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...) [0]: https://arxiv.org/abs/2305.07185 https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660 https://arxiv.org/abs/2401.13660
- sp332 3y ago
  Ok so to get a little weirder, could you use a normal tokenizer for pretraining and then go back to bytes for fine tuning along with or before the length extension?
  - Fripplebubby 3y ago
    Do the bytes represent natural language text, or something else? If they do represent text, then it is not so weird to me.
  - kristjansson 3y ago
    Tokens (be they tokens or bytes) basically atoms of input to a LM. So you'd have to figure out how to express the relationship between tokens and bytes to the LM. I guess you could do an incremental thing e.g. - make vocab ~40k tokens + 256 tokens (for each possible byte), - start training with 100% tokens, - then after some portion of train budget has elapsed, randomly replace some tokens with corresponding byte-tokens, - then ramp the fraction of replacements up so you're training on 100% byte tokens for the last xx% of training, without ever exceeding an 8k (or whatever) sequence length - then apply the trick from TFA to get xk -> 8xk/64xk bytes of context? but I'd guess the interesting part of a byte-transformer is multimodality, and we'd need more than a few tricks to get from ^^ to there.
lettergram 3y ago
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

Does this mean we can skip "tokenization" and feed raw bytes into our models now?

by sp332 3y ago

Does this mean we can skip "tokenization" and feed raw bytes into our models now?

That's been on the table e.g. [0][1]. But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...) [0]: https://arxiv.org/abs/2305.07185 https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660 https://arxiv.org/abs/2401.13660

sp332 3y ago
Ok so to get a little weirder, could you use a normal tokenizer for pretraining and then go back to bytes for fine tuning along with or before the length extension?
- Fripplebubby 3y ago
  Do the bytes represent natural language text, or something else? If they do represent text, then it is not so weird to me.
- kristjansson 3y ago
  Tokens (be they tokens or bytes) basically atoms of input to a LM. So you'd have to figure out how to express the relationship between tokens and bytes to the LM. I guess you could do an incremental thing e.g. - make vocab ~40k tokens + 256 tokens (for each possible byte), - start training with 100% tokens, - then after some portion of train budget has elapsed, randomly replace some tokens with corresponding byte-tokens, - then ramp the fraction of replacements up so you're training on 100% byte tokens for the last xx% of training, without ever exceeding an 8k (or whatever) sequence length - then apply the trick from TFA to get xk -> 8xk/64xk bytes of context? but I'd guess the interesting part of a byte-transformer is multimodality, and we'd need more than a few tricks to get from ^^ to there.

lettergram 3y ago

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.