4 ms·

Uh? GPT is an neural net, right?Which is a bunch of weights, surely, so how can you get lossless compression from that? (Complete newbie here BTW)

pizza 3y ago
The model can either generate probable outputs or score the likelihood of given inputs.
- _a_a_a_ 3y ago
  How does one produce 'lossless' from 'probable' or 'likelihood'?
  - sp332 3y ago
    By running the decoder during encoding. If the decoder already produces the right output, which it will most of the time, then you don’t need to store anything in the file. If it produces the wrong output, store a correction, then continue from there. The final file only needs to include the corrections.
  - qazxcvbnm 3y ago
    Arithmetic coding is one of the classic information theory methods that produce essentially-optimal results. Like everywhere else in information theory, the eminence of the probability distribution is fully recognised, and given it, arithmetic coding gives you an information-theoretically optimal lossless compression (to the degree that the probability distribution describes the data well) (by treating the objects in question as numbers and intuitively, like the other post says, only keeping diffs). See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding
  - pizza 3y ago
    Let's say your text is "I like red cats", your current token input is at "I like red". Let's say the model thinks "cats" is 25% likely. It also thinks "dogs" is eg 50% likely for the next token. So rather than storing 1 bit (= log p) to encode dog in the output encoded code, it can use 2 bits to store "cats". Totally lossless. At decode time, you're reading the code, so it's going to just pick the "cats" token anyway, even though it's not the likeliest, because it's not randomly sampling here, it's just using the underlying probabilistic model to do entropy coding. It's not lossy here, since we're not letting the model/sampling process freely choose what the text is made of.

pizza 3y ago

The model can either generate probable outputs or score the likelihood of given inputs.

_a_a_a_ 3y ago
How does one produce 'lossless' from 'probable' or 'likelihood'?
- sp332 3y ago
  By running the decoder during encoding. If the decoder already produces the right output, which it will most of the time, then you don’t need to store anything in the file. If it produces the wrong output, store a correction, then continue from there. The final file only needs to include the corrections.
- qazxcvbnm 3y ago
  Arithmetic coding is one of the classic information theory methods that produce essentially-optimal results. Like everywhere else in information theory, the eminence of the probability distribution is fully recognised, and given it, arithmetic coding gives you an information-theoretically optimal lossless compression (to the degree that the probability distribution describes the data well) (by treating the objects in question as numbers and intuitively, like the other post says, only keeping diffs). See https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding
- pizza 3y ago
  Let's say your text is "I like red cats", your current token input is at "I like red". Let's say the model thinks "cats" is 25% likely. It also thinks "dogs" is eg 50% likely for the next token. So rather than storing 1 bit (= log p) to encode dog in the output encoded code, it can use 2 bits to store "cats". Totally lossless. At decode time, you're reading the code, so it's going to just pick the "cats" token anyway, even though it's not the likeliest, because it's not randomly sampling here, it's just using the underlying probabilistic model to do entropy coding. It's not lossy here, since we're not letting the model/sampling process freely choose what the text is made of.