3 ms·

What about something like image improvement algorithms or NeRFs? They seem to increase information even if some of it is made up.

antihero 2y ago
It isn't real information though. This is effectively a Chinese whispers. The only way AI can create information is by doing something in the real world.
- ofrzeta 2y ago
  Maybe information needs to be understood relationally as in "information for a subject x". So if we have an image with a license plate that is unreadable and there's an algorithm that makes it readable to x, there is an information gain for x, although the information might have been in the image all along.
  - bippihippi1 2y ago
    eliminating the noise makes the useful information clearer, but the information describing the noise is lost
  - vunderba 2y ago
    Sure, but what if the upscaling algorithm misinterpreted a P as an F? Without manual supervision/tagging, there's an inherent risk that this information will have an adverse effect on future models.
  - ablob 2y ago
    If the license plate was not readable, then the additional information is false data. You do not know more about the image than you knew before by definition. Replacing pixels with plausible data does not mean a gain of information. If anything, I'd argue that a loss of information occurs: The fact that x was hardly readable/unreadable before is lost, and any decision later on can not factor this in as "x" is now clearly defined and not fuzzy anymore. Would you accept a system that "enhances" images to find the license plate numbers of cars and fine their owners? If the plate number is unreadable the only acceptable option is to not use it. Inserting a plausible number and rolling with it even means that instead of a range of suspects, only one culprit can be supposed. Would you like to find yourself in court for crimes/offenses you never comitted because some black box decided it was a great idea to pretend it knew it was you? Edit: I think I misunderstood the premise. Nonetheless my comment shall stay.
    - immibis 2y ago
      For an example of this, see "Xerox scanners and photocopiers randomly alter numbers in scanned documents" https://news.ycombinator.com/item?id=6156238 https://news.ycombinator.com/item?id=6156238
- jppittma 2y ago
  It’s information taken from many other photos and embedded into a single one of interest no?
- dragonwriter 2y ago
  > The only way AI can create information is by doing something in the real world. Everything done is done in the real world, but the only way an AI can gather (not create) information about some particular thing is to interact with that thing. Without interacting with anything external to itself, all information it can gather is the information already gathered to create it.
  - tomrod 2y ago
    Is there a formalization of this idea? Would love to read more.
  - antihero 2y ago
    That's a better way of putting it, yes.
- Lerc 2y ago
  It is real information, it is just information that is not targeted at anything in particular. Random passwords are, well, random. That they are random and information is what makes them useful as passwords. As said by others, There is nothing terribly insightful about making something estimate the output of another by a non-perfect reproduction mechanism and noticing the output is different. Absent any particular guidance the difference will not be targeted. That is tautologically obvious. The difference is still information though, and with guidance you can target the difference to perform some goal. This is essentially what the gpt-o1 training was doing. Training on data generated by itself, but only when the generated data produced the correct answer.
dartos 2y ago
Do they gain information, or just have lower loss? Too much information encoded in a model can lower performance (called overfitting) That’s why many NN topologies include dropout layers.
blensor 2y ago
Once more and more new training images are based off of those new upscaled images the training of those upscaling algorithms will tend to generate even more of the same type of information drowning out the other information
vunderba 2y ago
If the goal of an image improvement algorithm is effectively "how would this image have looked IN THE REAL WORLD if it had been taken with a better camera", then training on previous "virtual upscaled images" would be training on the wrong fitness function.
raincole 2y ago
Image improvement algorithms are basically injecting statistical information (collected from other images) into one image. The above statement applies for non-neural-network algorithms as well.
dragonwriter 2y ago
“Made up” information is noise, not signal (OTOH, generated in images are used productively all the time in training, but the information content added is not in the images themselves but in their selection and relation to captions.)

antihero 2y ago

It isn't real information though. This is effectively a Chinese whispers. The only way AI can create information is by doing something in the real world.

ofrzeta 2y ago
Maybe information needs to be understood relationally as in "information for a subject x". So if we have an image with a license plate that is unreadable and there's an algorithm that makes it readable to x, there is an information gain for x, although the information might have been in the image all along.
- bippihippi1 2y ago
  eliminating the noise makes the useful information clearer, but the information describing the noise is lost
- vunderba 2y ago
  Sure, but what if the upscaling algorithm misinterpreted a P as an F? Without manual supervision/tagging, there's an inherent risk that this information will have an adverse effect on future models.
- ablob 2y ago
  If the license plate was not readable, then the additional information is false data. You do not know more about the image than you knew before by definition. Replacing pixels with plausible data does not mean a gain of information. If anything, I'd argue that a loss of information occurs: The fact that x was hardly readable/unreadable before is lost, and any decision later on can not factor this in as "x" is now clearly defined and not fuzzy anymore. Would you accept a system that "enhances" images to find the license plate numbers of cars and fine their owners? If the plate number is unreadable the only acceptable option is to not use it. Inserting a plausible number and rolling with it even means that instead of a range of suspects, only one culprit can be supposed. Would you like to find yourself in court for crimes/offenses you never comitted because some black box decided it was a great idea to pretend it knew it was you? Edit: I think I misunderstood the premise. Nonetheless my comment shall stay.
  - immibis 2y ago
    For an example of this, see "Xerox scanners and photocopiers randomly alter numbers in scanned documents" https://news.ycombinator.com/item?id=6156238 https://news.ycombinator.com/item?id=6156238
jppittma 2y ago
It’s information taken from many other photos and embedded into a single one of interest no?
dragonwriter 2y ago
> The only way AI can create information is by doing something in the real world. Everything done is done in the real world, but the only way an AI can gather (not create) information about some particular thing is to interact with that thing. Without interacting with anything external to itself, all information it can gather is the information already gathered to create it.
- tomrod 2y ago
  Is there a formalization of this idea? Would love to read more.
- antihero 2y ago
  That's a better way of putting it, yes.
Lerc 2y ago
It is real information, it is just information that is not targeted at anything in particular. Random passwords are, well, random. That they are random and information is what makes them useful as passwords. As said by others, There is nothing terribly insightful about making something estimate the output of another by a non-perfect reproduction mechanism and noticing the output is different. Absent any particular guidance the difference will not be targeted. That is tautologically obvious. The difference is still information though, and with guidance you can target the difference to perform some goal. This is essentially what the gpt-o1 training was doing. Training on data generated by itself, but only when the generated data produced the correct answer.

dartos 2y ago

Do they gain information, or just have lower loss? Too much information encoded in a model can lower performance (called overfitting) That’s why many NN topologies include dropout layers.

blensor 2y ago

Once more and more new training images are based off of those new upscaled images the training of those upscaling algorithms will tend to generate even more of the same type of information drowning out the other information

vunderba 2y ago

If the goal of an image improvement algorithm is effectively "how would this image have looked IN THE REAL WORLD if it had been taken with a better camera", then training on previous "virtual upscaled images" would be training on the wrong fitness function.

raincole 2y ago

Image improvement algorithms are basically injecting statistical information (collected from other images) into one image. The above statement applies for non-neural-network algorithms as well.

dragonwriter 2y ago

“Made up” information is noise, not signal (OTOH, generated in images are used productively all the time in training, but the information content added is not in the images themselves but in their selection and relation to captions.)