4 ms·

All the model's weights are just a compressed representation of data it was trained on That's not remotely accurate. I'd argue it's not even understood what th

All the model's weights are just a compressed representation of data it was trained on That's not remotely accurate. I'd argue it's not even understood what the weights are or how they relate to the training data, and what is understood is not at all trivial.

BaseballPhysics 4y ago
That's... just not true. We know these models capture aspects of the training dataset, otherwise training wouldn't be a thing. That's the entire point. We also know these models create a very compact representation of the data they derive. So it is perfectly reasonable to think of these things as a compressed representation. Is it lossy? Yes, of course. But so what?
- tomrod 4y ago
  A valid model is f(all inputs imaginable)=2. No weights involved. You can apply as much training as you like, but its still just the value "2" as an output. There is no representation of the data. ML models are a mix of two things: 1. Patterns defined by the individuals performing the model training. Commonly this is referred to as model architecture, but ultimately its the cobbling together of various items to create a way to get a signal from data in a certain way. (As a side note, thanks Statisticians and Computer Scientists for leading the way here!) 2. Model weights to minimize the uncertainty in those patterns. How that uncertainty is measured is part of (1).
- pixl97 4y ago
  >Is it lossy? Yes, of course. But so what? "But so what" is missing the entire point of intelligence. You as an intelligent creature only work because of this lossiness. Reality presents far too much information to process and instead brains, especially in higher lifeforms have evolved to filter out as much useless information as possible in the most energy efficient way, while not losing important context.
- Riverheart 4y ago
  Right there's over/under fitting. Train SD on the same data too many times it over produces it in results. Just because they tailor it to get more nebulous results doesn't mean the data doesn't exist in some form.
somenameforme 4y ago
Looking for "meaning" in mathematical representations is often not going to be productive. Even for really simple things this is true. Imagine you give me a series of points, I do a quadratic regression, and give you back f(x) = 3.74x^2 - 8.31x + 13.33 as a means of approximating your points and even predicting future points! What does the 3.74 "mean"? For something this simple you can still create some smart sounding explanations related to the curve, but even that quickly becomes impossible as you bump up the complexity a bit. The reality is that it doesn't really have any "meaning" - certainly not in terms of the input data. It's simply the weighting on the exponential that gives an approximation of your input set. That's all. And it's the exact same with the model weights.
nikisweeting 4y ago
If there are fewer bits of weights than original data, and the weights + an algorithm can be used to reproduce the original data (even lossily), then it's absolutely a compression in the information theory sense. The degree of lossyness needed before it's not considered a copyright violation is what humans determine case-by-case in court, but that doesn't make it any less of a "compression". Just like your neurons "compressing" the gazillions of atoms in the world you experience down to smaller representative models in your head.

BaseballPhysics 4y ago

That's... just not true. We know these models capture aspects of the training dataset, otherwise training wouldn't be a thing. That's the entire point. We also know these models create a very compact representation of the data they derive. So it is perfectly reasonable to think of these things as a compressed representation. Is it lossy? Yes, of course. But so what?

tomrod 4y ago
A valid model is f(all inputs imaginable)=2. No weights involved. You can apply as much training as you like, but its still just the value "2" as an output. There is no representation of the data. ML models are a mix of two things: 1. Patterns defined by the individuals performing the model training. Commonly this is referred to as model architecture, but ultimately its the cobbling together of various items to create a way to get a signal from data in a certain way. (As a side note, thanks Statisticians and Computer Scientists for leading the way here!) 2. Model weights to minimize the uncertainty in those patterns. How that uncertainty is measured is part of (1).
pixl97 4y ago
>Is it lossy? Yes, of course. But so what? "But so what" is missing the entire point of intelligence. You as an intelligent creature only work because of this lossiness. Reality presents far too much information to process and instead brains, especially in higher lifeforms have evolved to filter out as much useless information as possible in the most energy efficient way, while not losing important context.
Riverheart 4y ago
Right there's over/under fitting. Train SD on the same data too many times it over produces it in results. Just because they tailor it to get more nebulous results doesn't mean the data doesn't exist in some form.

somenameforme 4y ago

Looking for "meaning" in mathematical representations is often not going to be productive. Even for really simple things this is true. Imagine you give me a series of points, I do a quadratic regression, and give you back f(x) = 3.74x^2 - 8.31x + 13.33 as a means of approximating your points and even predicting future points! What does the 3.74 "mean"? For something this simple you can still create some smart sounding explanations related to the curve, but even that quickly becomes impossible as you bump up the complexity a bit. The reality is that it doesn't really have any "meaning" - certainly not in terms of the input data. It's simply the weighting on the exponential that gives an approximation of your input set. That's all. And it's the exact same with the model weights.

nikisweeting 4y ago

If there are fewer bits of weights than original data, and the weights + an algorithm can be used to reproduce the original data (even lossily), then it's absolutely a compression in the information theory sense. The degree of lossyness needed before it's not considered a copyright violation is what humans determine case-by-case in court, but that doesn't make it any less of a "compression". Just like your neurons "compressing" the gazillions of atoms in the world you experience down to smaller representative models in your head.