3 ms·

But the parent was talking about lossless, GPT presumably is anything but.

d_burfoot 3y ago
GPT, and probably any LLM, can be hooked up to an encoder to produce a lossless data compressor. Patrice Bellard has done this, HN discussion here: https://news.ycombinator.com/item?id=23618465 https://news.ycombinator.com/item?id=23618465
pizza 3y ago
Sure it is, just use a probability distribution over the entire vocabulary of possible tokens to do arithmetic/range coding of the original text.
- _a_a_a_ 3y ago
  Uh? GPT is an neural net, right?Which is a bunch of weights, surely, so how can you get lossless compression from that? (Complete newbie here BTW)
  - pizza 3y ago
    The model can either generate probable outputs or score the likelihood of given inputs.
    - _a_a_a_ 3y ago
      How does one produce 'lossless' from 'probable' or 'likelihood'?
      - sp332 3y ago
        By running the decoder during encoding. If the decoder already produces the right output, which it will most of the time, then you don’t need to store anything in the file. If it produces the wrong output, store a correction, then continue from there. The final file only needs to include the corrections.
      - qazxcvbnm 3y ago
        Arithmetic coding is one of the classic information theory methods that produce essentially-optimal results. Like everywhere else in information theory, the eminence of the probability distribution is fully recognised, and given it, arithmetic coding gives you an information-theoretically optimal lossless compression (to the degree that the probability distribution describes the data well) (by treating the objects in question as numbers and intuitively, like the other post says, only keeping diffs). See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding
      - pizza 3y ago
        Let's say your text is "I like red cats", your current token input is at "I like red". Let's say the model thinks "cats" is 25% likely. It also thinks "dogs" is eg 50% likely for the next token. So rather than storing 1 bit (= log p) to encode dog in the output encoded code, it can use 2 bits to store "cats". Totally lossless. At decode time, you're reading the code, so it's going to just pick the "cats" token anyway, even though it's not the likeliest, because it's not randomly sampling here, it's just using the underlying probabilistic model to do entropy coding. It's not lossy here, since we're not letting the model/sampling process freely choose what the text is made of.

GPT, and probably any LLM, can be hooked up to an encoder to produce a lossless data compressor. Patrice Bellard has done this, HN discussion here: https://news.ycombinator.com/item?id=23618465 https://news.ycombinator.com/item?id=23618465

pizza 3y ago

Sure it is, just use a probability distribution over the entire vocabulary of possible tokens to do arithmetic/range coding of the original text.

_a_a_a_ 3y ago
Uh? GPT is an neural net, right?Which is a bunch of weights, surely, so how can you get lossless compression from that? (Complete newbie here BTW)
- pizza 3y ago
  The model can either generate probable outputs or score the likelihood of given inputs.
  - _a_a_a_ 3y ago
    How does one produce 'lossless' from 'probable' or 'likelihood'?
    - sp332 3y ago
      By running the decoder during encoding. If the decoder already produces the right output, which it will most of the time, then you don’t need to store anything in the file. If it produces the wrong output, store a correction, then continue from there. The final file only needs to include the corrections.
    - qazxcvbnm 3y ago
      Arithmetic coding is one of the classic information theory methods that produce essentially-optimal results. Like everywhere else in information theory, the eminence of the probability distribution is fully recognised, and given it, arithmetic coding gives you an information-theoretically optimal lossless compression (to the degree that the probability distribution describes the data well) (by treating the objects in question as numbers and intuitively, like the other post says, only keeping diffs). See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding
    - pizza 3y ago
      Let's say your text is "I like red cats", your current token input is at "I like red". Let's say the model thinks "cats" is 25% likely. It also thinks "dogs" is eg 50% likely for the next token. So rather than storing 1 bit (= log p) to encode dog in the output encoded code, it can use 2 bits to store "cats". Totally lossless. At decode time, you're reading the code, so it's going to just pick the "cats" token anyway, even though it's not the likeliest, because it's not randomly sampling here, it's just using the underlying probabilistic model to do entropy coding. It's not lossy here, since we're not letting the model/sampling process freely choose what the text is made of.