8 ms·

Byte Latent Transformer: Patches Scale Better Than Tokens

378 points by zxexz 2y ago

bloomingkales 2y ago
I thought we’re supposed to be plateauing!?
- ArnoVW 2y ago
  We are. Plateauing doesnt mean you don't book progress. Arguably that is what you would call "plateaud". The argument of plateauing is not that AI is fundamentally impossible. The argument is that just dumping more data and more compute on the problem, using the same approach, has diminishing returns. It's that statistical inference is not how the human mind works (not exclusively) and thus that we are not guaranteed to be able to replicate all traits of human intelligence by brute forcing. Of course we can and will still improve the algorithms. But the question remains whether tweaks like these, as cool and useful they may be to solve certain issues, will be enough by themselves. Since it remains statistical in nature, my position is "no".
  - logicchains 2y ago
    > that we are not guaranteed to be able to replicate all traits of human intelligence by brute forcing. We know from complexity theory that transformers with chain of thought are guaranteed to be able to reproduce a significant fraction of human reasoning, anything in the complexity class PTIME: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923
    - fl0id 2y ago
      I don’t think this paper says what you claim. It says chain of reasoning and its length can improve transformer performance. Not that this represent a significant fraction of human reasoning or that it’s even reasoning
      - logicchains 2y ago
        It says it can represent all programs in P, i.e. all reasoning that produces an output in no more than polynomial time with respect to the amount of inputs. Most human reasoning is presumably in P, not NP, because we generally don't find ourselves needing to think exponentially long about things.
- random3 2y ago
  who's "we"?
qouteall 2y ago
Related quote from Karpathy: Tokenization is at the heart of much weirdness of LLMs. Do not brush it off. • Why can't LLM spell words? Tokenization. • Why can't LLM do super simple string processing tasks like reversing a string? Tokenization. • Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization. • Why is LLM bad at simple arithmetic? Tokenization. • Why did GPT-2 have more than necessary trouble coding in Python? Tokenization. • Why did my LLM abruptly halt when it sees the string "<|endoftext|>"? Tokenization. • What is this weird warning I get about a "trailing whitespace"? Tokenization. • Why the LLM break if I ask it about "SolidGoldMagikarp"? Tokenization. • Why should I prefer to use YAML over JSON with LLMs? Tokenization. • Why is LLM not actually end-to-end language modeling? Tokenization. • What is the real root of suffering? Tokenization.
- withinboredom 2y ago
  It’s weird because I’m pretty sure my brain does something similar when I speed read. I don’t actually, usually, read the words; instead I recognize the shape of the words (most common words) then I jump to the subject of the paragraphs and break down the meaning of the whole page in a second or so.
  - Timwi 2y ago
    That's generally true, but you also have the ability to stop and look closer if you want to. If someone asks you to count the letters in a word, you will stop to look at the letters individually. If you see an unfamiliar word like SolidGoldMagikarp, you can stop and break it apart. Tokenization prevents LLMs from doing this.
    - kimixa 2y ago
      Generally the current crop of LLMs seem pretty good analogues of the "scan reading" immediate instinctual response to stimulus, but seems to completely lack the higher level that can then go "Wait, that doesn't seem right, let's go back over that again". Like hallucinations and seeing "Faces" in dark shadows until you look again, it's like it's doing a pretty good emulation of some level of consciousness. Is that a fundamental difference to the level of processing? I haven't seen that sort of second-tier logic pop up from any emergence behaviors from increasing scale yet, but will that come with time? I'm not sure.
fabmilo 2y ago
I am gonna read this paper and the other latent sentence later today. I always advocated for this kind of solutions together with latent sentence search should get to the next level of AI. Amazing work from Meta
- CuriousSkeptic 2y ago
  Sentence thing being this one? https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode... I don’t get it, isn’t this concept modelling exactly whats going on in the deeper layers of current LLMs?
  - kmacdough 2y ago
    Perhaps it does some similar grouping of content, but this more directly incentivizes longer term gripping of tokens into abstract concepts. I agree that it's not obvious this would perform better than letting the model build it's own structures for grouping tokens, but the proof is in the pudding; the technique led to improved results for a given model & training size. This newer approach gives the model the freedom to build it's own breakpoints, but still bakes the idea into the algorithm itself. What it means is a harder question. Perhaps transformers are simply an inefficient computational structure for this process? Perhaps a more flexible computational structure would integrate this step more efficiently? Perhaps Transformers are efficient enough, but our learning/densifying isn't? Or perhaps it's such a core powerful step that it might as well be built into the algo regardless? Much to learn.
flimflamm 2y ago
To create a patch, a small model is used to predict the likelihood for the next character in the input string. Input string: 'Lazy dog jumped over a fence.' Use the model to predict the likelihood of each character. For example: 100% sure the next character is 'a'. Or maybe it's 10% sure it's 'a', 10% sure it's 'b', and so on. Then we chunk character estimates together. How many characters? Enough characters so that the total uncertainty (entropy) in each chunk is about the same. And there you have your 'patch' (or 'token').
- yorwba 2y ago
  > How many characters? Enough characters so that the total uncertainty (entropy) in each chunk is about the same. That's not how it's described in Section 2.3 of the paper. They only use the entropy of the next byte and whether it exceeds a threshold (Global Constraint) or is larger than the preceding byte's entropy by another threshold (Approx. Monotonic Constraint). That does mean that long repetitive sequences can result in pathologically long patches, as demonstrated in Appendix E. But what I'm really curious about is the "small CNN byte-level model with 2-byte context" in Figure 3 (f), because it's never mentioned in any other part of the paper.
  - flimflamm 2y ago
    "That's not how it's described" - Thanks for the correction!
  - entilzha 2y ago
    (Author Here) Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tried a bunch of patching schemes, EG, keep a running total of entropy until the total exceeds a threshold, but ended up finding simple things worked better. I’ll see if we can add more information about the small CNN in a next update to arXiv paper.
    - psb217 2y ago
      One way of thinking about the "Approximate Monotonic Constraint" is that you're running a quick and dirty edge detector on the entropy. Ie, you're clipping based on the gradient of per-byte entropy wrt timestep compared to detecting an edge based on gradient of per-pixel intensity wrt pixel coordinates. It would be interesting to look at the raw sequences of per-byte entropies to see how strongly these sorts of "edges" correlate with human interpretable boundaries (words, prefixes, suffixes, etc).
paraschopra 2y ago
My notes: It's a 3 component model. - Encoder: Takes byte groupings and outputs a hidden state/encoding called patches - Transformer: Takes these encodings of patches in autoregressive fashion - Decoder: Takes processed encodings by transformers and outputs bytes Loss is on byte to byte crossentropy (Next byte prediction) How they group bytes. - Use entropy thresholds: If a sequence of bytes have entropy lower than a threshold, group them - This is a learned model (from data) Why this helps over current byte-pair tokenization in LLMs. - Encoder/decoder essentially act as “learnable” tokenization scheme - Better efficiency tradeoffs (as for highly predictable sequence of bytes, encoder can “offload” computation effort from the main transformer) - History teaches us that end to end learned system beats human designed mechanisms
- CuriousSkeptic 2y ago
  > History teaches us that end to end learned system beats human designed mechanisms I think this may need some qualifiers Even byte representations are human designed encodings. I would think a human designed decoder of such encodings must be more efficient than learning. Sure bytes encoding a stream of unicode code points maps fairly easy to useful information. But bytes representing a zip compressed collection of PDF files? I did wonder though, training on text encoding vs pixel encoding, perhaps brute forcing OCR, like humans, will be more flexible in the end then being limited to text encodings.
  - paraschopra 2y ago
    >Even byte representations are human designed encodings The point is that it can model any sequence of bytes. It's what-follows-what that matters, not how we're encoding it.
- hiddencost 2y ago
  > History teaches us that end to end learned system beats human designed mechanisms Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere. That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly. I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in. BPE was a huge advancement over fixed vocab, e.g.
dr_dshiv 2y ago
Does this mean AI can pre-train on binaries?
- bloomingkales 2y ago
  Some believe AI can now output compiled binaries (e.g update Notepad.exe with this feature). We all think AI writing code for us will be the end, but it might be an even simpler take over.
  - deleted 2y ago
    [deleted]
  - 8n4vidtmkvmk 2y ago
    That just sounds worse though? We can't validate the change is correct if we can't read the code. It is interesting though
    - hackernewds 2y ago
      at some point you can't or won't be allowed to do any validations
    - refulgentis 2y ago
      Idk what they mean, I've never seen anyone claim, or come close to claiming, it could alter an executable binary by itself. Chose to interpret it as "some people think an llm can code well enough to add features on top of a ~50KLOC codebase automatically"
      - famouswaffles 2y ago
        I think he's saying people believe a LLM trained with this architecture would be able to do something like that.
dewijones92 2y ago
notebooklm https://notebooklm.google.com/notebook/77fe83ee-35b3-4a9a-a3d9-e2f1395a3a0f/audio https://notebooklm.google.com/notebook/77fe83ee-35b3-4a9a-a3...
- ricardobeat 2y ago
  Interesting, this is one of the worst NotebookLM examples I've seen so far. They are interjecting way too often and breaking the rhythm. Is generation quality going down due to the popularity of the service?
  - bratao 2y ago
    Yeah, super strange. One cannot finish a sentence without the other interjecting.
  - stuartjohnson12 2y ago
    Big successful launch, hype for the product lead, product lead moves on, product goes to shit. Another classic for the Google graveyard.
    - marviel 2y ago
      Core team just moved on to something else: https://werebuilding.ai/ https://werebuilding.ai/
      - 8n4vidtmkvmk 2y ago
        That has to be the worst landing page ever
        marviel 2y ago
        no comment; not my site, just sharing
    - throwaway20222 2y ago
amelius 2y ago
Why can't the tokenization be implicit, so we only feed bytes (or characters) to the model?
- killerstorm 2y ago
  It can work, but you have more tokens / weaker performance. People tested it and it was worse.
- entilzha 2y ago
  (Author Here) Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which means you’re pretty FLOP inefficient so scale worse. You could make the model smaller to compensate, but then the model isn’t as good.
PaulHoule 2y ago
The summer that BERT came out I was working at a startup that was using character-based CNN models for classification. We were thinking a lot about alternate representations, other members of the team were keen on word vectors but I wasn't, particularly because it seemed the documents were were working on frequently had out-of-dictionary words, because those words were important, and because discarding them would lead to failure. (We were working on "foundation models" too, so it's not just being out-of-dictionary in the final model that's a problem but being out-of-dictionary in the foundation model which is more expensive to train.) We were doing OK with character based models for classification but people believed that storing the "dictionary" inside the neural net was not a good use of the neural net so there was a lot of enthusiasm for tokens. Meanwhile I felt so sure that schemes like Word2Vec were doomed that I had left an earlier project using RNNs where the goal was text understanding with a foundation model made by training an RNN to write fake abstracts for case reports from PubMed. When byte-pair encoding was introduced I remember telling people in a meeting that it was the first tokenization scheme we'd looked at that I could endorse. I have to admit though that I wish we could work at the character label.
- yndoendo 2y ago
  Do you mean that all produced output must be a chain or words found in a dictionary? The real-world for humans has them creating and using non-dictionary words to communicate daily. A good example is "notify", defined in the dictionary. "notifier", which is not and is used to describe "a means to notify someone". The code to send an email notification is an "email notifier", then there is text message, voice call, call center call back notifiers .... All industries and organizations have jargon, custom defined words not found in a dictionary and use non distinctive acronyms. How would a ML output be useful if it cannot handle real world commutation and only lab based sanitization of in-dictionary only responses?
  - phh 2y ago
    That's the OP's point. At the time, the community was split between word-level, which has the shortcomings you're describing, and byte-level which is uselessly compute intensive. BPE was the first reasonable in-between. BLT improves on BPE by having the the compression learnable rather than precomputed
nodja 2y ago
From my understanding this not only removes tokenization but also sampling correct? Sampling can be a pain point of LLMs, but they also can enable interesting usages, like forcing grammar so the model always outputs valid JSON or tuning temperature to get more varied distribution, XTC sampling, etc. What would be the equivalent of these in a BLT? I can only think of providing the decoder an extra input of allowed/prohibited bytes and run the decode over and over until it outputs something valid, maybe there's a simpler and more obvious approach.
- yorwba 2y ago
  It doesn't remove sampling, and forcing grammar by specifying allowed/prohibited bytes doesn't require running the decoder over and over, you just compute the softmax at the output layer over allowed bytes only and sample from those accordingly, same as with BPE-based models.
modeless 2y ago
I really hope this works out. Death to tokenizers! Interesting that it's a hierarchical structure but only two levels of hierarchy. Stacking more levels seems like an obvious direction for further research. Note: I posted this comment on another related story[1] and the author replied: "Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But ya, there are many directions to go from here!" [1] https://news.ycombinator.com/item?id=42413430 https://news.ycombinator.com/item?id=42413430
- smaddox 2y ago
  Agree more levels seems like it could be beneficial. And another Meta paper published a day later shows how that might work: https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode...
boulos 2y ago
This is neat work, but I also love the (presumably intentional?) backronym of BLT.
dang 2y ago
Recent and related: Sharing new research, models, and datasets from Meta FAIR - https://news.ycombinator.com/item?id=42412360 https://news.ycombinator.com/item?id=42412360 - Dec 2024 (61 comments)
dang 2y ago
The paper: https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/470135129_1314438233309836_4712217603129928862_n.pdf?_nc_cat=111&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=WqSN1qsot3oQ7kNvgFWGG4j&_nc_zt=14&_nc_ht=scontent-sjc3-1.xx&_nc_gid=A2yO-vwOF4w2PIUX2gHIbXD&oh=00_AYBAR_B1_9ewVRJM5VYbJbdfm4Uk5INZY0t67hlpNccpAA&oe=676400C8 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/470135129_...
RandyOrion 2y ago
An interesting read on alternative tokenization methods. Questions: 1. What's the goal of entropy based byte token grouping as tokenization? Is this tokenization method best suited for the goal? 2. What about simply using byte level sequence to sequence autoencoder with down sampling for tokenization?
DerSaidin 2y ago
> Unlike tokenization, BLT has no fixed vocabulary for patches. iiuc this means: the vocabulary of patches is not known prior to training. I guess once training has established a vocabulary of patches, that same fixed vocabulary is used for inference (if this is not true I don't see how it could work). Right?
macrolime 2y ago
I wonder whether llama 4 will use this
iandanforth 2y ago
I find it interesting how far linguistic, and experienced based approaches have fallen out of fashion. Humans don't read character by character, even if we can it's not a standard operating mode. We have word stems and understand modifications by endings. Tokenization doesn't replicate this experience (seriously, look at the tokens that appear in LLM vocabularies), nor does character or byte encoding. Humans have multiple ways to parse words. You can grok a full sentence, read a phrase, read word by word, or sound out a new word character by character. Very few papers explicitly claim that a method is good because it replicates the way a human would perform a task, or perceive the world. I suspect as LLM reliance increases we'll want to align the models to our experience more closely. I further suspect this will make the errors that models make more comprehensible.
vishpr 2y ago
So only thing teaching model (loss) is probability prediction in single byte space. And that is enough? Looks very promising, if I am not misunderstanding.

378 points by zxexz 2y ago

bloomingkales 2y ago

I thought we’re supposed to be plateauing!?

ArnoVW 2y ago
We are. Plateauing doesnt mean you don't book progress. Arguably that is what you would call "plateaud". The argument of plateauing is not that AI is fundamentally impossible. The argument is that just dumping more data and more compute on the problem, using the same approach, has diminishing returns. It's that statistical inference is not how the human mind works (not exclusively) and thus that we are not guaranteed to be able to replicate all traits of human intelligence by brute forcing. Of course we can and will still improve the algorithms. But the question remains whether tweaks like these, as cool and useful they may be to solve certain issues, will be enough by themselves. Since it remains statistical in nature, my position is "no".
- logicchains 2y ago
  > that we are not guaranteed to be able to replicate all traits of human intelligence by brute forcing. We know from complexity theory that transformers with chain of thought are guaranteed to be able to reproduce a significant fraction of human reasoning, anything in the complexity class PTIME: https://arxiv.org/abs/2310.07923 https://arxiv.org/abs/2310.07923
  - fl0id 2y ago
    I don’t think this paper says what you claim. It says chain of reasoning and its length can improve transformer performance. Not that this represent a significant fraction of human reasoning or that it’s even reasoning
    - logicchains 2y ago
      It says it can represent all programs in P, i.e. all reasoning that produces an output in no more than polynomial time with respect to the amount of inputs. Most human reasoning is presumably in P, not NP, because we generally don't find ourselves needing to think exponentially long about things.
random3 2y ago
who's "we"?

qouteall 2y ago

Related quote from Karpathy: Tokenization is at the heart of much weirdness of LLMs. Do not brush it off. • Why can't LLM spell words? Tokenization. • Why can't LLM do super simple string processing tasks like reversing a string? Tokenization. • Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization. • Why is LLM bad at simple arithmetic? Tokenization. • Why did GPT-2 have more than necessary trouble coding in Python? Tokenization. • Why did my LLM abruptly halt when it sees the string "<|endoftext|>"? Tokenization. • What is this weird warning I get about a "trailing whitespace"? Tokenization. • Why the LLM break if I ask it about "SolidGoldMagikarp"? Tokenization. • Why should I prefer to use YAML over JSON with LLMs? Tokenization. • Why is LLM not actually end-to-end language modeling? Tokenization. • What is the real root of suffering? Tokenization.

withinboredom 2y ago
It’s weird because I’m pretty sure my brain does something similar when I speed read. I don’t actually, usually, read the words; instead I recognize the shape of the words (most common words) then I jump to the subject of the paragraphs and break down the meaning of the whole page in a second or so.
- Timwi 2y ago
  That's generally true, but you also have the ability to stop and look closer if you want to. If someone asks you to count the letters in a word, you will stop to look at the letters individually. If you see an unfamiliar word like SolidGoldMagikarp, you can stop and break it apart. Tokenization prevents LLMs from doing this.
  - kimixa 2y ago
    Generally the current crop of LLMs seem pretty good analogues of the "scan reading" immediate instinctual response to stimulus, but seems to completely lack the higher level that can then go "Wait, that doesn't seem right, let's go back over that again". Like hallucinations and seeing "Faces" in dark shadows until you look again, it's like it's doing a pretty good emulation of some level of consciousness. Is that a fundamental difference to the level of processing? I haven't seen that sort of second-tier logic pop up from any emergence behaviors from increasing scale yet, but will that come with time? I'm not sure.

fabmilo 2y ago

I am gonna read this paper and the other latent sentence later today. I always advocated for this kind of solutions together with latent sentence search should get to the next level of AI. Amazing work from Meta

CuriousSkeptic 2y ago
Sentence thing being this one? https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode... I don’t get it, isn’t this concept modelling exactly whats going on in the deeper layers of current LLMs?
- kmacdough 2y ago
  Perhaps it does some similar grouping of content, but this more directly incentivizes longer term gripping of tokens into abstract concepts. I agree that it's not obvious this would perform better than letting the model build it's own structures for grouping tokens, but the proof is in the pudding; the technique led to improved results for a given model & training size. This newer approach gives the model the freedom to build it's own breakpoints, but still bakes the idea into the algorithm itself. What it means is a harder question. Perhaps transformers are simply an inefficient computational structure for this process? Perhaps a more flexible computational structure would integrate this step more efficiently? Perhaps Transformers are efficient enough, but our learning/densifying isn't? Or perhaps it's such a core powerful step that it might as well be built into the algo regardless? Much to learn.

flimflamm 2y ago

To create a patch, a small model is used to predict the likelihood for the next character in the input string. Input string: 'Lazy dog jumped over a fence.' Use the model to predict the likelihood of each character. For example: 100% sure the next character is 'a'. Or maybe it's 10% sure it's 'a', 10% sure it's 'b', and so on. Then we chunk character estimates together. How many characters? Enough characters so that the total uncertainty (entropy) in each chunk is about the same. And there you have your 'patch' (or 'token').

yorwba 2y ago
> How many characters? Enough characters so that the total uncertainty (entropy) in each chunk is about the same. That's not how it's described in Section 2.3 of the paper. They only use the entropy of the next byte and whether it exceeds a threshold (Global Constraint) or is larger than the preceding byte's entropy by another threshold (Approx. Monotonic Constraint). That does mean that long repetitive sequences can result in pathologically long patches, as demonstrated in Appendix E. But what I'm really curious about is the "small CNN byte-level model with 2-byte context" in Figure 3 (f), because it's never mentioned in any other part of the paper.
- flimflamm 2y ago
  "That's not how it's described" - Thanks for the correction!
- entilzha 2y ago
  (Author Here) Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tried a bunch of patching schemes, EG, keep a running total of entropy until the total exceeds a threshold, but ended up finding simple things worked better. I’ll see if we can add more information about the small CNN in a next update to arXiv paper.
  - psb217 2y ago
    One way of thinking about the "Approximate Monotonic Constraint" is that you're running a quick and dirty edge detector on the entropy. Ie, you're clipping based on the gradient of per-byte entropy wrt timestep compared to detecting an edge based on gradient of per-pixel intensity wrt pixel coordinates. It would be interesting to look at the raw sequences of per-byte entropies to see how strongly these sorts of "edges" correlate with human interpretable boundaries (words, prefixes, suffixes, etc).

paraschopra 2y ago

My notes: It's a 3 component model. - Encoder: Takes byte groupings and outputs a hidden state/encoding called patches - Transformer: Takes these encodings of patches in autoregressive fashion - Decoder: Takes processed encodings by transformers and outputs bytes Loss is on byte to byte crossentropy (Next byte prediction) How they group bytes. - Use entropy thresholds: If a sequence of bytes have entropy lower than a threshold, group them - This is a learned model (from data) Why this helps over current byte-pair tokenization in LLMs. - Encoder/decoder essentially act as “learnable” tokenization scheme - Better efficiency tradeoffs (as for highly predictable sequence of bytes, encoder can “offload” computation effort from the main transformer) - History teaches us that end to end learned system beats human designed mechanisms

CuriousSkeptic 2y ago
> History teaches us that end to end learned system beats human designed mechanisms I think this may need some qualifiers Even byte representations are human designed encodings. I would think a human designed decoder of such encodings must be more efficient than learning. Sure bytes encoding a stream of unicode code points maps fairly easy to useful information. But bytes representing a zip compressed collection of PDF files? I did wonder though, training on text encoding vs pixel encoding, perhaps brute forcing OCR, like humans, will be more flexible in the end then being limited to text encodings.
- paraschopra 2y ago
  >Even byte representations are human designed encodings The point is that it can model any sequence of bytes. It's what-follows-what that matters, not how we're encoding it.
hiddencost 2y ago
> History teaches us that end to end learned system beats human designed mechanisms Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere. That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly. I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in. BPE was a huge advancement over fixed vocab, e.g.

dr_dshiv 2y ago

Does this mean AI can pre-train on binaries?

bloomingkales 2y ago
Some believe AI can now output compiled binaries (e.g update Notepad.exe with this feature). We all think AI writing code for us will be the end, but it might be an even simpler take over.
- deleted 2y ago
  [deleted]
- 8n4vidtmkvmk 2y ago
  That just sounds worse though? We can't validate the change is correct if we can't read the code. It is interesting though
  - hackernewds 2y ago
    at some point you can't or won't be allowed to do any validations
  - refulgentis 2y ago
    Idk what they mean, I've never seen anyone claim, or come close to claiming, it could alter an executable binary by itself. Chose to interpret it as "some people think an llm can code well enough to add features on top of a ~50KLOC codebase automatically"
    - famouswaffles 2y ago
      I think he's saying people believe a LLM trained with this architecture would be able to do something like that.

dewijones92 2y ago

notebooklm https://notebooklm.google.com/notebook/77fe83ee-35b3-4a9a-a3d9-e2f1395a3a0f/audio https://notebooklm.google.com/notebook/77fe83ee-35b3-4a9a-a3...

ricardobeat 2y ago
Interesting, this is one of the worst NotebookLM examples I've seen so far. They are interjecting way too often and breaking the rhythm. Is generation quality going down due to the popularity of the service?
- bratao 2y ago
  Yeah, super strange. One cannot finish a sentence without the other interjecting.
- stuartjohnson12 2y ago
  Big successful launch, hype for the product lead, product lead moves on, product goes to shit. Another classic for the Google graveyard.
  - marviel 2y ago
    Core team just moved on to something else: https://werebuilding.ai/ https://werebuilding.ai/
    - 8n4vidtmkvmk 2y ago
      That has to be the worst landing page ever
      - marviel 2y ago
        no comment; not my site, just sharing
  - throwaway20222 2y ago

amelius 2y ago

Why can't the tokenization be implicit, so we only feed bytes (or characters) to the model?

killerstorm 2y ago
It can work, but you have more tokens / weaker performance. People tested it and it was worse.
entilzha 2y ago
(Author Here) Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular token LLM, you can’t pack as many bytes in a batch, which means you’re pretty FLOP inefficient so scale worse. You could make the model smaller to compensate, but then the model isn’t as good.

PaulHoule 2y ago

The summer that BERT came out I was working at a startup that was using character-based CNN models for classification. We were thinking a lot about alternate representations, other members of the team were keen on word vectors but I wasn't, particularly because it seemed the documents were were working on frequently had out-of-dictionary words, because those words were important, and because discarding them would lead to failure. (We were working on "foundation models" too, so it's not just being out-of-dictionary in the final model that's a problem but being out-of-dictionary in the foundation model which is more expensive to train.) We were doing OK with character based models for classification but people believed that storing the "dictionary" inside the neural net was not a good use of the neural net so there was a lot of enthusiasm for tokens. Meanwhile I felt so sure that schemes like Word2Vec were doomed that I had left an earlier project using RNNs where the goal was text understanding with a foundation model made by training an RNN to write fake abstracts for case reports from PubMed. When byte-pair encoding was introduced I remember telling people in a meeting that it was the first tokenization scheme we'd looked at that I could endorse. I have to admit though that I wish we could work at the character label.

yndoendo 2y ago
Do you mean that all produced output must be a chain or words found in a dictionary? The real-world for humans has them creating and using non-dictionary words to communicate daily. A good example is "notify", defined in the dictionary. "notifier", which is not and is used to describe "a means to notify someone". The code to send an email notification is an "email notifier", then there is text message, voice call, call center call back notifiers .... All industries and organizations have jargon, custom defined words not found in a dictionary and use non distinctive acronyms. How would a ML output be useful if it cannot handle real world commutation and only lab based sanitization of in-dictionary only responses?
- phh 2y ago
  That's the OP's point. At the time, the community was split between word-level, which has the shortcomings you're describing, and byte-level which is uselessly compute intensive. BPE was the first reasonable in-between. BLT improves on BPE by having the the compression learnable rather than precomputed

nodja 2y ago

From my understanding this not only removes tokenization but also sampling correct? Sampling can be a pain point of LLMs, but they also can enable interesting usages, like forcing grammar so the model always outputs valid JSON or tuning temperature to get more varied distribution, XTC sampling, etc. What would be the equivalent of these in a BLT? I can only think of providing the decoder an extra input of allowed/prohibited bytes and run the decode over and over until it outputs something valid, maybe there's a simpler and more obvious approach.

yorwba 2y ago
It doesn't remove sampling, and forcing grammar by specifying allowed/prohibited bytes doesn't require running the decoder over and over, you just compute the softmax at the output layer over allowed bytes only and sample from those accordingly, same as with BPE-based models.

modeless 2y ago

I really hope this works out. Death to tokenizers! Interesting that it's a hierarchical structure but only two levels of hierarchy. Stacking more levels seems like an obvious direction for further research. Note: I posted this comment on another related story[1] and the author replied: "Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But ya, there are many directions to go from here!" [1] https://news.ycombinator.com/item?id=42413430 https://news.ycombinator.com/item?id=42413430

smaddox 2y ago
Agree more levels seems like it could be beneficial. And another Meta paper published a day later shows how that might work: https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/ https://ai.meta.com/research/publications/large-concept-mode...

boulos 2y ago

This is neat work, but I also love the (presumably intentional?) backronym of BLT.

dang 2y ago

Recent and related: Sharing new research, models, and datasets from Meta FAIR - https://news.ycombinator.com/item?id=42412360 https://news.ycombinator.com/item?id=42412360 - Dec 2024 (61 comments)

dang 2y ago

The paper: https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/470135129_1314438233309836_4712217603129928862_n.pdf?_nc_cat=111&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=WqSN1qsot3oQ7kNvgFWGG4j&_nc_zt=14&_nc_ht=scontent-sjc3-1.xx&_nc_gid=A2yO-vwOF4w2PIUX2gHIbXD&oh=00_AYBAR_B1_9ewVRJM5VYbJbdfm4Uk5INZY0t67hlpNccpAA&oe=676400C8 https://scontent-sjc3-1.xx.fbcdn.net/v/t39.2365-6/470135129_...

RandyOrion 2y ago

An interesting read on alternative tokenization methods. Questions: 1. What's the goal of entropy based byte token grouping as tokenization? Is this tokenization method best suited for the goal? 2. What about simply using byte level sequence to sequence autoencoder with down sampling for tokenization?

DerSaidin 2y ago

> Unlike tokenization, BLT has no fixed vocabulary for patches. iiuc this means: the vocabulary of patches is not known prior to training. I guess once training has established a vocabulary of patches, that same fixed vocabulary is used for inference (if this is not true I don't see how it could work). Right?

macrolime 2y ago

I wonder whether llama 4 will use this

iandanforth 2y ago

I find it interesting how far linguistic, and experienced based approaches have fallen out of fashion. Humans don't read character by character, even if we can it's not a standard operating mode. We have word stems and understand modifications by endings. Tokenization doesn't replicate this experience (seriously, look at the tokens that appear in LLM vocabularies), nor does character or byte encoding. Humans have multiple ways to parse words. You can grok a full sentence, read a phrase, read word by word, or sound out a new word character by character. Very few papers explicitly claim that a method is good because it replicates the way a human would perform a task, or perceive the world. I suspect as LLM reliance increases we'll want to align the models to our experience more closely. I further suspect this will make the errors that models make more comprehensible.

vishpr 2y ago

So only thing teaching model (loss) is probability prediction in single byte space. And that is enough? Looks very promising, if I am not misunderstanding.