5 ms·

I think the fact alone that distillation and quantization are techniques that can produce substantial improvements is a strong sign that we still have no real c

by xg15 2y ago

I think the fact alone that distillation and quantization are techniques that can produce substantial improvements is a strong sign that we still have no real comprehensive understanding how the models work. If we had, there would be no reason to train a model with more parameters than are strictly necessary to represent the space's semantic structure. But then it should be impossible for distilled models with less parameters to come close to the performance of the original model. Yet this is what happens - the distilled or quantized models often come very close to the original model. So I think there are still many low-hanging fruits to pick.

teruakohatu 2y ago
> still have no real comprehensive understanding how the models work. We do understand how they work, we just have not optimised their usage. For example someone who has a good general understanding of how an ICE or EV car works. Even if the user interface is very unfamiliar, they can figure out how to drive any car within a couple of minutes. But that does not mean they can race a car, drift a car or drive a car on challenging terrain even if the car is physically capable of all these things.
- spiorf 2y ago
  We know how the next token is selected, but not why doing that repeatedly brings all the capabilities it does. We really don't understand how the emergent behaviours emerge.
  - Valgrim 2y ago
    It feels less like a word prediction algorithm and more like a world model compression algorithm. Maybe we tried to create one and accidentaly created the other?
    - codeulike 2y ago
      Its almost like a Model of Language, but very Large
    - bloomingkales 2y ago
      Why would asking a question about ice cream trigger a consideration about all possible topics? As in, to formulate the answer, the LLM will consider the origin of Elephants even. It won’t be significant, but it will be factored in. Why? In the spiritual realm, many postulated that even the Elephant you never met is part of your life. None of this is a coincidence.
  - fennecfoxy 2y ago
    Eh I feel like that mostly just down to; yes transformers are a "next token predictor" but during fine tuning for instruct the attention related wagon slapped on the back is partially hijacked as a bridge from input token->sequences of connections in the weights. For example if I ask "If I have two foxes and I take away one, how many foxes do I have?" I reckon attention has been hijacked to essentially highlight the "if I have x and take away y then z" portion of the query to connect to a learned sequence from readily available training data (apparently the whole damn Internet) where there are plenty of examples of said math question trope, just using some other object type than foxes. I think we could probably prove it by tracing the hyperdimensional space the model exists in and ask it variants of the same question/find hotspots in that space that would indicate it's using those same sequences (with attention branching off to ensure it replies with the correct object type that was referenced).
- gessha 2y ago
  Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would training the model for 5000 days help? I dunno, try it. Car technology is the opposite of that - it’s a white box. It’s composed of very well defined elements whose interactions are defined and explained by laws of thermodynamics and whatnot.
  - brookst 2y ago
    Isn't that just scale? Even small LLMs have more parts than any car. LLMs are more analogous to economics, psychology, politics -- it is possible there's a core science with explicability, but the systems are so complex that even defining the question is hard.
    - ChymeraXYZ 2y ago
      Could be, but it does not change the fact that we do not understand them as of now.
    - gessha 2y ago
      You can make a bigger ICE engine (like a container ship engine) and still understand how the whole thing works. Maybe there’s more parts moving but it still has the structure of an ICE engine. With neural networks big or small, we got no clue what’s going on. You can observe the whole system, from the weights and biases, to the activations, gradients, etc and still get nothing. On the other hand, one of the reasons why economics, psychology and politics are hard is because we can’t open up people’s heads and define and measure what they’re thinking.
      - ijk 2y ago
        One way I've heard it summarized: Computer Science as a field is used to things being like physics or chemistry, but we've suddenly encountered something that behaves more like biology.
- adamc 2y ago
  The "Wait" vs. "Hmm" discussion in the paper does not suggest we know how they work. If we knew, we wouldn't have to try things and measure to figure out the best prompt.
pertymcpert 2y ago
For quantization I don't think that's really true. Quantization is just making more efficient use of bits in memory to represent numbers.
ZeljkoS 2y ago
We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note that similar process happens with human brain, it is called Synaptic pruning (https://en.wikipedia.org/wiki/Synaptic_pruning https://en.wikipedia.org/wiki/Synaptic_pruning). Relevant quote from Wikipedia (https://en.wikipedia.org/wiki/Neuron#Connectivity https://en.wikipedia.org/wiki/Neuron#Connectivity): "It has been estimated that the brain of a three-year-old child has about 10^15 synapses (1 quadrillion). This number declines with age, stabilizing by adulthood. Estimates vary for an adult, ranging from 10^14 to 5x10^14 synapses (100 to 500 trillion)."
- 3abiton 2y ago
  So more 'mature' models might arise in the near future with less params and better benchmarks?
  - raducu 2y ago
    "Better", but not better than the model they were distilled from, at least that's how I understand it.
    - salemba 2y ago
      I think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)
      - cristiancavalli 2y ago
        Not at all — how many people were geniuses and their parents not? I can name several and I’m sure with a quick search you can too.
MR4D 2y ago
I like the analogy of compression, in that a distilled model of an LLM is like a JPEG of a photo. Pretty good, maybe very good, but still lossy. The question I hear you raising seems to be along the lines of, can we use a new compression method to get better resolution (reproducibility of the original) in a much smaller size.
- umeshunni 2y ago
  > in that a distilled model of an LLM is like a JPEG of a photo That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.
  - kedarkhand 2y ago
    Well, JPEG can be thought of as an compression of the natural world of whose photograph was taken
    - bloomingkales 2y ago
      And we can answer the question why quantization works with a lossy format, since quantization just drops accuracy for space but still gives us a good enough output, just like a lossy jpeg. Reiterating again, we can lose a lot of data (have incomplete data) and have a perfectly visible jpeg (or MP3, same thing).
  - homarp 2y ago
    hence https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-... (by Ted Chiang) (discussed here: https://news.ycombinator.com/item?id=34724477 https://news.ycombinator.com/item?id=34724477 )
  - timschmidt 2y ago
    And what is compression but finding the minimum amount of information required to reproduce a phenomena? I.e. discovering natural laws.
cztomsik 2y ago
Nope, it's quite obvious why distillation works. If you just predict next token, then the only information you can use to compute the loss is THE expected token. Whereas if you distill, you can also use (typically few) logits from the teacher. "My name is <?>" without distillation has only one valid answer (from the dataset) and everything else is wrong. Whereas with distillation, you get lots of other names too (from the teacher), and you can add some weight to them too. That way, model learns faster, because it gets more information in each update. (So instead of "My name is Foo", the model learns "My name is <some name, but in this case Foo>")

by xg15 2y ago

teruakohatu 2y ago

> still have no real comprehensive understanding how the models work. We do understand how they work, we just have not optimised their usage. For example someone who has a good general understanding of how an ICE or EV car works. Even if the user interface is very unfamiliar, they can figure out how to drive any car within a couple of minutes. But that does not mean they can race a car, drift a car or drive a car on challenging terrain even if the car is physically capable of all these things.

spiorf 2y ago
We know how the next token is selected, but not why doing that repeatedly brings all the capabilities it does. We really don't understand how the emergent behaviours emerge.
- Valgrim 2y ago
  It feels less like a word prediction algorithm and more like a world model compression algorithm. Maybe we tried to create one and accidentaly created the other?
  - codeulike 2y ago
    Its almost like a Model of Language, but very Large
  - bloomingkales 2y ago
    Why would asking a question about ice cream trigger a consideration about all possible topics? As in, to formulate the answer, the LLM will consider the origin of Elephants even. It won’t be significant, but it will be factored in. Why? In the spiritual realm, many postulated that even the Elephant you never met is part of your life. None of this is a coincidence.
- fennecfoxy 2y ago
  Eh I feel like that mostly just down to; yes transformers are a "next token predictor" but during fine tuning for instruct the attention related wagon slapped on the back is partially hijacked as a bridge from input token->sequences of connections in the weights. For example if I ask "If I have two foxes and I take away one, how many foxes do I have?" I reckon attention has been hijacked to essentially highlight the "if I have x and take away y then z" portion of the query to connect to a learned sequence from readily available training data (apparently the whole damn Internet) where there are plenty of examples of said math question trope, just using some other object type than foxes. I think we could probably prove it by tracing the hyperdimensional space the model exists in and ask it variants of the same question/find hotspots in that space that would indicate it's using those same sequences (with attention branching off to ensure it replies with the correct object type that was referenced).
gessha 2y ago
Your example is somewhat inadequate. We _fundamentally_ don’t understand how deep learning systems works in the sense that they are more or less black boxes that we train and evaluate. Innovations in ML are a whole bunch of wizards with big stacks of money changing “Hmm” to “Wait” and seeing what happens. Would a different sampler help you? I dunno, try it. Would a smaller dataset help? I dunno, try it. Would training the model for 5000 days help? I dunno, try it. Car technology is the opposite of that - it’s a white box. It’s composed of very well defined elements whose interactions are defined and explained by laws of thermodynamics and whatnot.
- brookst 2y ago
  Isn't that just scale? Even small LLMs have more parts than any car. LLMs are more analogous to economics, psychology, politics -- it is possible there's a core science with explicability, but the systems are so complex that even defining the question is hard.
  - ChymeraXYZ 2y ago
    Could be, but it does not change the fact that we do not understand them as of now.
  - gessha 2y ago
    You can make a bigger ICE engine (like a container ship engine) and still understand how the whole thing works. Maybe there’s more parts moving but it still has the structure of an ICE engine. With neural networks big or small, we got no clue what’s going on. You can observe the whole system, from the weights and biases, to the activations, gradients, etc and still get nothing. On the other hand, one of the reasons why economics, psychology and politics are hard is because we can’t open up people’s heads and define and measure what they’re thinking.
    - ijk 2y ago
      One way I've heard it summarized: Computer Science as a field is used to things being like physics or chemistry, but we've suddenly encountered something that behaves more like biology.
adamc 2y ago
The "Wait" vs. "Hmm" discussion in the paper does not suggest we know how they work. If we knew, we wouldn't have to try things and measure to figure out the best prompt.

pertymcpert 2y ago

For quantization I don't think that's really true. Quantization is just making more efficient use of bits in memory to represent numbers.

ZeljkoS 2y ago

We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note that similar process happens with human brain, it is called Synaptic pruning (https://en.wikipedia.org/wiki/Synaptic_pruning https://en.wikipedia.org/wiki/Synaptic_pruning). Relevant quote from Wikipedia (https://en.wikipedia.org/wiki/Neuron#Connectivity https://en.wikipedia.org/wiki/Neuron#Connectivity): "It has been estimated that the brain of a three-year-old child has about 10^15 synapses (1 quadrillion). This number declines with age, stabilizing by adulthood. Estimates vary for an adult, ranging from 10^14 to 5x10^14 synapses (100 to 500 trillion)."

3abiton 2y ago
So more 'mature' models might arise in the near future with less params and better benchmarks?
- raducu 2y ago
  "Better", but not better than the model they were distilled from, at least that's how I understand it.
  - salemba 2y ago
    I think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)
    - cristiancavalli 2y ago
      Not at all — how many people were geniuses and their parents not? I can name several and I’m sure with a quick search you can too.

MR4D 2y ago

I like the analogy of compression, in that a distilled model of an LLM is like a JPEG of a photo. Pretty good, maybe very good, but still lossy. The question I hear you raising seems to be along the lines of, can we use a new compression method to get better resolution (reproducibility of the original) in a much smaller size.

umeshunni 2y ago
> in that a distilled model of an LLM is like a JPEG of a photo That's an interesting analogy, because I've always thought of the hidden states (and weights and biases) of an LLMs as a compressed version of the training data.
- kedarkhand 2y ago
  Well, JPEG can be thought of as an compression of the natural world of whose photograph was taken
  - bloomingkales 2y ago
    And we can answer the question why quantization works with a lossy format, since quantization just drops accuracy for space but still gives us a good enough output, just like a lossy jpeg. Reiterating again, we can lose a lot of data (have incomplete data) and have a perfectly visible jpeg (or MP3, same thing).
- homarp 2y ago
  hence https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-... (by Ted Chiang) (discussed here: https://news.ycombinator.com/item?id=34724477 https://news.ycombinator.com/item?id=34724477 )
- timschmidt 2y ago
  And what is compression but finding the minimum amount of information required to reproduce a phenomena? I.e. discovering natural laws.

cztomsik 2y ago

Nope, it's quite obvious why distillation works. If you just predict next token, then the only information you can use to compute the loss is THE expected token. Whereas if you distill, you can also use (typically few) logits from the teacher. "My name is <?>" without distillation has only one valid answer (from the dataset) and everything else is wrong. Whereas with distillation, you get lots of other names too (from the teacher), and you can add some weight to them too. That way, model learns faster, because it gets more information in each update. (So instead of "My name is Foo", the model learns "My name is <some name, but in this case Foo>")