3 ms·

My notes: It's a 3 component model. - Encoder: Takes byte groupings and outputs a hidden state/encoding called patches - Transformer: Takes these encodings o

My notes: It's a 3 component model. - Encoder: Takes byte groupings and outputs a hidden state/encoding called patches - Transformer: Takes these encodings of patches in autoregressive fashion - Decoder: Takes processed encodings by transformers and outputs bytes Loss is on byte to byte crossentropy (Next byte prediction) How they group bytes. - Use entropy thresholds: If a sequence of bytes have entropy lower than a threshold, group them - This is a learned model (from data) Why this helps over current byte-pair tokenization in LLMs. - Encoder/decoder essentially act as “learnable” tokenization scheme - Better efficiency tradeoffs (as for highly predictable sequence of bytes, encoder can “offload” computation effort from the main transformer) - History teaches us that end to end learned system beats human designed mechanisms

CuriousSkeptic 2y ago
> History teaches us that end to end learned system beats human designed mechanisms I think this may need some qualifiers Even byte representations are human designed encodings. I would think a human designed decoder of such encodings must be more efficient than learning. Sure bytes encoding a stream of unicode code points maps fairly easy to useful information. But bytes representing a zip compressed collection of PDF files? I did wonder though, training on text encoding vs pixel encoding, perhaps brute forcing OCR, like humans, will be more flexible in the end then being limited to text encodings.
- paraschopra 2y ago
  >Even byte representations are human designed encodings The point is that it can model any sequence of bytes. It's what-follows-what that matters, not how we're encoding it.
hiddencost 2y ago
> History teaches us that end to end learned system beats human designed mechanisms Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere. That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly. I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in. BPE was a huge advancement over fixed vocab, e.g.
- entilzha 2y ago
  (Author Here) Related thought, I think BPE is quite a good, cheap inductive bias to have in a model, which is part of what made it challenging to scale better against. I also suspect this is part of why with less training FLOPs BPE is better (left side of figure 1), BLT has to expend some of its FLOPs budget to recover/learn some of this useful bias. With more training FLOPs this becomes a smaller fraction of the budget though leading to better scaling.
deleted 2y ago
[deleted]

CuriousSkeptic 2y ago

> History teaches us that end to end learned system beats human designed mechanisms I think this may need some qualifiers Even byte representations are human designed encodings. I would think a human designed decoder of such encodings must be more efficient than learning. Sure bytes encoding a stream of unicode code points maps fairly easy to useful information. But bytes representing a zip compressed collection of PDF files? I did wonder though, training on text encoding vs pixel encoding, perhaps brute forcing OCR, like humans, will be more flexible in the end then being limited to text encodings.

paraschopra 2y ago
>Even byte representations are human designed encodings The point is that it can model any sequence of bytes. It's what-follows-what that matters, not how we're encoding it.

hiddencost 2y ago

> History teaches us that end to end learned system beats human designed mechanisms Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere. That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly. I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in. BPE was a huge advancement over fixed vocab, e.g.

entilzha 2y ago
(Author Here) Related thought, I think BPE is quite a good, cheap inductive bias to have in a model, which is part of what made it challenging to scale better against. I also suspect this is part of why with less training FLOPs BPE is better (left side of figure 1), BLT has to expend some of its FLOPs budget to recover/learn some of this useful bias. With more training FLOPs this becomes a smaller fraction of the budget though leading to better scaling.

deleted 2y ago

[deleted]