5 ms·

I will nitpick and say that models don't compress data but extract useful information. Compression lossly or lossless is for data reconstruction. Machine learni

I will nitpick and say that models don't compress data but extract useful information. Compression lossly or lossless is for data reconstruction. Machine learning models retain information that generalises https://arxiv.org/abs/1906.05849 https://arxiv.org/abs/1906.05849

jhgb 6y ago
Isn't lossy compression pretty much the same thing as extracting some amount of useful information?
- Retric 6y ago
  No, because lossy compression has the goal of preserving elements for replay. At the extreme you could turn a movie into subtitles, but not a costume from the movie alone. The costume might contain more information from the movie, but you can meaningfully recreate the movie from a single costume.
  - jhgb 6y ago
    If I derive Kepler's laws from positions of planets, I have preserved elements of their motion for replay as well.
    - Retric 6y ago
      The laws alone don’t have enough information to replay their motion without each planets location, a foci, the orbital period, and an associated timestamp or equivalent information. So, include that data and it’s lossy compression, exclude it and it’s not.
- teruakohatu 6y ago
  According to Wikipedia [data] compression "is the process of encoding information using fewer bits than the original representation" [1] while lossy compression is "the class of data encoding methods that uses inexact approximations and partial data discarding to represent the content" [2] There is a difference between feature extraction and compression. The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set. For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression. [1] https://en.wikipedia.org/wiki/Data_compression https://en.wikipedia.org/wiki/Data_compression [2] https://en.wikipedia.org/wiki/Lossy_compression https://en.wikipedia.org/wiki/Lossy_compression
  - miltondts 6y ago
    "The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set." The compressor finds a smaller approximation of the data by finding the most redundant data, and as a byproduct it also finds the most unique. At least from the description you gave there is no difference between the two. "For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression." You can and we do use compressors to do exactly that. Take a painting (or a set) as reference and use it to compress other paintings and the greater the compression the more similar the styles are. For an example see: https://www.sciencedirect.com/science/article/abs/pii/S0031320321000510 https://www.sciencedirect.com/science/article/abs/pii/S00313... EDIT: In fact one step in a compressor is feature extraction (usually called modelling) and the other is coding.
viraptor 6y ago
Is there a meaningful / well defined difference? One could say that lossy compression is extraction of useful information. You need to identify unnecessary information to know what to discard safely.
- mjburgess 6y ago
  One is a subset of the other. Compression being the larger category. A mean is a compression of a dataset and useful information. My issue in my comment is that 'compression' corresponds to a massive class of technquies and there isnt a lot of content in the observation that useful information is compressive. However there is some hypey people out there who think this observation has legs. Precisely the people who think intelligence is a mathematical problem, and not an engineering one -- which is my view. ie., that a body isnt incidental to intelligence, but the heart of it. Or: devices matter.
  - typon 6y ago
    I tend to think about it the same way as you. Having an algorithm for multiplying two numbers is qualitatively different than having a lossy compression of a huge dataset of multiplication tables. The latter is what GPT3 has and it just doesn't scale.
- space_rock 6y ago
  Consider a generative adversarial network for faces. Photos have let's say some unique scars A successful model could create faces with scars in them. But not the exact scar and face and background that they trained on without additional information. What you are looking for is mutual information between the images not compression Yes you could use a very well trained GAN for face compression. But a GAN model itself would not be able to reconstruct its training input without being shown the images again

jhgb 6y ago

Isn't lossy compression pretty much the same thing as extracting some amount of useful information?

Retric 6y ago
No, because lossy compression has the goal of preserving elements for replay. At the extreme you could turn a movie into subtitles, but not a costume from the movie alone. The costume might contain more information from the movie, but you can meaningfully recreate the movie from a single costume.
- jhgb 6y ago
  If I derive Kepler's laws from positions of planets, I have preserved elements of their motion for replay as well.
  - Retric 6y ago
    The laws alone don’t have enough information to replay their motion without each planets location, a foci, the orbital period, and an associated timestamp or equivalent information. So, include that data and it’s lossy compression, exclude it and it’s not.
teruakohatu 6y ago
According to Wikipedia [data] compression "is the process of encoding information using fewer bits than the original representation" [1] while lossy compression is "the class of data encoding methods that uses inexact approximations and partial data discarding to represent the content" [2] There is a difference between feature extraction and compression. The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set. For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression. [1] https://en.wikipedia.org/wiki/Data_compression https://en.wikipedia.org/wiki/Data_compression [2] https://en.wikipedia.org/wiki/Lossy_compression https://en.wikipedia.org/wiki/Lossy_compression
- miltondts 6y ago
  "The former selects the most unique and important elements from a set of data, while the latter attempts, in the case of lossy compression, to find a smaller approximation of the original data set." The compressor finds a smaller approximation of the data by finding the most redundant data, and as a byproduct it also finds the most unique. At least from the description you gave there is no difference between the two. "For example imagine a data set of paintings. Feature extraction might simply identify what color paints were used in each painting. That alone might allow classification of paintings with respect to the painter, style or period. But this would not be an approximation of the original picture, unless you consider resizing a Jackson Pollock into an 8x1 pixel image to be compression." You can and we do use compressors to do exactly that. Take a painting (or a set) as reference and use it to compress other paintings and the greater the compression the more similar the styles are. For an example see: https://www.sciencedirect.com/science/article/abs/pii/S0031320321000510 https://www.sciencedirect.com/science/article/abs/pii/S00313... EDIT: In fact one step in a compressor is feature extraction (usually called modelling) and the other is coding.

viraptor 6y ago

Is there a meaningful / well defined difference? One could say that lossy compression is extraction of useful information. You need to identify unnecessary information to know what to discard safely.

mjburgess 6y ago
One is a subset of the other. Compression being the larger category. A mean is a compression of a dataset and useful information. My issue in my comment is that 'compression' corresponds to a massive class of technquies and there isnt a lot of content in the observation that useful information is compressive. However there is some hypey people out there who think this observation has legs. Precisely the people who think intelligence is a mathematical problem, and not an engineering one -- which is my view. ie., that a body isnt incidental to intelligence, but the heart of it. Or: devices matter.
- typon 6y ago
  I tend to think about it the same way as you. Having an algorithm for multiplying two numbers is qualitatively different than having a lossy compression of a huge dataset of multiplication tables. The latter is what GPT3 has and it just doesn't scale.
space_rock 6y ago
Consider a generative adversarial network for faces. Photos have let's say some unique scars A successful model could create faces with scars in them. But not the exact scar and face and background that they trained on without additional information. What you are looking for is mutual information between the images not compression Yes you could use a very well trained GAN for face compression. But a GAN model itself would not be able to reconstruct its training input without being shown the images again