6 ms·

Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't

Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". You fundamentally can't get back information that has been destroyed/or never captured in the first place. What you can do is fill in the gaps/information with plausible values. I don't know whether this sounds like I'm splitting hairs, but it's really important that the general public not think we're extracting information in these procedures, we're interpolating or projecting information that is not there. Very useful for artificially generating skins for each shoe on a shoe rack in a computer game or simulation, potentially disastrous if the general public starts to think it's applicable to security camera footage or admissible as evidence...

jopsen 9y ago
Sometimes US justice system seems very "approximate". So why not convict people based on interpolated evidence? - I'm joking of course :) hehe
- WillReplyfFood 9y ago
  You seem like a funny guy- the batmaNN interpolates that you are very likely a joker.
- dahart 9y ago
  Sadly, it actually happens sometimes. https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi... This thread a year ago worried about it too, but the paper itself seems implausible and problematic. https://news.ycombinator.com/item?id=12983827 https://news.ycombinator.com/item?id=12983827
ZeroGravitas 9y ago
To give specific examples from their test data, it added stubble to people who didn't have stubble, gave them a different shape of glasses, changed the color of cats, changed the color and brand of sport shoe. And even then, I'm a little suspicious of how close some of the images got to original without being given color information. It appears that info was either hidden in the original in a way not apparent to humans or was implicit in their data set in some way that would make it fail on photos of people with different skin tones.
- omtinez 9y ago
  I haven't read the paper in full detail, but reading between the lines I'm guessing that there's a significant portion of manual processing and hand waving involved. From the abstract, emphasis mine: > the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. My interpretation is that they select training data by hand and generate a bunch of outputs. Repeating the process until they like the final result. From the paper: > we allow a user to have an arbitrarily-fine level of control through on-the-fly editing of the exemplar set (E.g., “resynthesize an image using the eye from this image and the nose from that one”).
  - IshKebab 9y ago
    Could be pretty great for police sketch artists. (Although pretty misleading for juries too.)
    - adrianN 9y ago
      Just train the model with the suspect's Facebook photostream and presto you have convincing evidence.
  - WhitneyLand 9y ago
    There's nothing weak or negative about that, it's exactly what'd you expect. Obviously for a given input there will be multiple plausible outputs. With any such system it would make sense to allow some control in choosing among the outputs.
- chaosite 9y ago
  Sounds similar to the problems with JBIG2 lossy compression. https://en.wikipedia.org/wiki/JBIG2#Disadvantages https://en.wikipedia.org/wiki/JBIG2#Disadvantages
gambiting 9y ago
No, but think of these blurred images as a "hash" - in an ideal situation, you only have one value that encodes to a certain hash value, right? So If you are given a hash X you technically can work out that it was derived from value Y - you're not getting back information that was lost - in a way it was merely encoded into the blurred image, and it should be possible to produce a real image which, when blurred, will match what you have. Don't get me wrong, I think we're still far far far off situation where we can get those reliably, but I can see how you could get the actual face out of a blurred image.
- ComputerGuru 9y ago
  > you only have one value that encodes to a certain hash value, right? Errr wrong. A perfect hash, yes. But they're never perfect. You have a collision domain and you hope that you don't have enough inputs to trigger a birthday paradox. Look at the pictures on the article. It's an outline of the shoe. That's your hash. ANY shoe with that general outline resolves to that same hash. If your input is objects found in the Oxford English Dictionary, you'll have low collisions. An elephant doesn't hash to that outline. But if your inputs is the Kohl's catalog, you'll have an unacceptable collision rate. Hashes are attempts at creating a _truncated_ "unique" representation of an input. They throw away data they hope isn't necessary to uniquely identify between possible inputs (bits). A perfect hash for all possible 32 bit values is 32 bits. You can't even have a collision free 31 bit hash. So back to the blurry security camera footage of a license plate or a face. Sure, that "hash" can reliably tell you that it wasn't a sasquatch that committed the robbery, but it literally doesn't contain the data necessary to _ever_ prove it was the suspect in question, even if the techs _can_ prove that the suspect hashes to the image in the footage.
  - consp 9y ago
    Fortunately there is a limit: the universe (in a practical sense). You cannot encode all states it has in a hash as it would require more states than you want to encode as you already mentioned (pigeon hole). But representing macroscopic data like text (or basically anything bigger than atomic scale) uniquely can be done with 128+ bits. Double that and you are likely safe for collisions, assuming the method you use is uniform and not biased to some input. If you want ease collision examples you can take a look at people using CRC32 as hashes/digests. It is notoriously prone to collisions (since only 32 bits).
usrusr 9y ago
Yeah, replace the training set with cartoon characters and the crime show dialog goes like this: "Zoom! Enhance! Zoom! Enhance! Enhance! Oh my god it's full of Smurfs..."
- deleted 9y ago
  [deleted]
scarface74 9y ago
What you can do though, in limited circumstances, is create a still picture with more detail from a lower quality video. https://photo.stackexchange.com/questions/17098/csi-image-resolution-enhance-how-real-is-it https://photo.stackexchange.com/questions/17098/csi-image-re...
- murkle 9y ago
  It's a well-known technique in astronomy, eg https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz
  - mcbits 9y ago
    For anyone put off by the .ps.gz, it's actually just a normal web page that links to the full article in HTML and PDF. Not sure what they were thinking with that URL. I almost didn't bother to look. (Maybe that's what they were thinking?)
- dahart 9y ago
  I seem to remember from my computer vision class way back when that there's a fundamental theoretical limit to the amount of detail you can get out of a moving sequence. Recovering frequencies a little higher than the pixel sampling is definitely possible, but I feel like it was maybe something like 10x theoretical maximum. I also get the feeling, from looking around at available software, that in practice achieving 2-3x is the most you can get in ideal conditions, and most video is far from ideal.
throwaway613834 9y ago
> I don't know whether this sounds like I'm splitting hairs Somewhat no, but somewhat yes. Thing is, while there can be lots of input images that generate the same output, it could be that only one (or a handful) of them would occur in reality. If this happens to sometimes be the case, and if you could somehow guarantee this was the case in some particular scenario, it could very well make sense to admit it as evidence. Of course, the issue is that figuring this out may not be possible...
sweezyjeezy 9y ago
> Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". I would argue that this is a form of enhancement though, and in some cases will be enough to completely reconstruct the original image. For example, if I give you a scanned PDF, and you know for a fact that it was size 12 black Ariel text on a white background, this can feasibly let you reconstruct the original image perfectly. The 'prior' that has been encoded by the model from the large amount of other images increases the mutual information between grainy image and high-res. The catch is that uncertainty cannot be removed entirely, and you need to know that the target image comes from roughly the same distribution as the training set. But knowing this gives you information that is not encoded in the pixels themselves, so you can't necessarily argue that some enhancement is impossible. For example with celebrity images, if the model is able to figure out who is in the picture, this massively decreases the set of plausible outputs.
- trevyn 9y ago
  > The catch is that you need to know that the target image comes from roughly the same distribution as the training set. When humans think about "enhance", they imagine extracting subtle details that were not obvious from the original, which implies that they know very little about what distribution the original image comes from. If they did, they wouldn't have a need for "enhance" 99% of the time -- the remaining 1% is for artistic purposes, which this is indeed suited for. It'll be interesting to see how society copes with the removal of the "photographs = evidence" prior. > when enhancing celebrity images, if the model is able to figure out who is in the picture this massively decreases the set of plausible outputs. This is an excellent insight.
  - ZenPsycho 9y ago
    Do you think knowing which state the license plate is from is enough prior knowledge?
- yorwba 9y ago
  See also https://dheera.net/projects/blur https://dheera.net/projects/blur
7952 9y ago
The benefit depends on how predictable the phenomenon is that your are interpolating from. Sometimes it will be quantitatively better than a low resolution version, sometimes not. A good example is with compression algorithms for media. They work because the sound or image is predictable. And they are ineffective when the input becomes more unpredictable. But if the output is all you have then running the decompression will probably be better than just reading the raw compressed data. But you have to be aware of the limitations.
jonathanstrange 9y ago
The white shoe output vs black shoe output illustrates this fairly well.
matt4077 9y ago
> You fundamentally can't get back information that has been destroyed/or never captured in the first place. I love this cliché. I've seen it thousands of times, and probably written it myself a few times. We all repeat stuff like that ad nauseam, without ever thinking. Because it's fundamentally flawed, especially in the context that it has usually been applied to, namely criticising the CSI:XYZ trope of "enhancing images". The truth is that there is a lot more information in a low-res image than meets the eye. Even if you can't read the letters on a license plate, it can be recovered by an algorithm. If the Empire State Building is in the background, it's likely to be a US license plate. Maybe only some letters would result in the photo's low-res pattern. If you only see part of a letter, knowing the font may allow you to rule out many letters or numbers etc... It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The error mostly appears to be in assuming that the information has been destroyed, when in reality it's often just obscured. And Neural Nets are excellent in squeezing all the information out noisy data.
- rootw0rm 9y ago
  It's not cliché, it's true. You fundamentally can't get back information that has been destroyed/or never captured in the first place. Yes, a low-res image has lots of information. You can process that information in many ways. Missing data can't just be magically blinked into existence though. Copy/pasting bits of guessed data is NOT getting back information that has been destroyed or never captured. Obscured data is very different from non-existent data. Could the software recreate a destroyed painting of mine based on a simple sketch? Of course not, because it would have to invent details it knows nothing about. I think it's almost dangerous to call this line of thinking cliché. It should be celebrated, not ridiculed.
- amelius 9y ago
  > It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The effect does not only need to be deterministic, but also invertible. A low-res image has multiple "inverses" (yikes), supposedly each with an associated probability (if you would model it that way). So it would be more honest if the algorithm shows them all.
k__ 9y ago
On the other hand, this is what the brain does all the time.
- eternalban 9y ago
  Wouldn't it be ironic if a mystified and superstitious GAI emerges out of all these efforts.
gus_massa 9y ago
In comparison of output vs original it is clear that the skin color is not accurate.
WhitneyLand 9y ago
>we're interpolating or projecting information that is not there But that's not fully accurate either. Sometimes the information in total will really be a more accurate representation of reality than the blurred image. Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. It would be interesting to see the results starting with higher quality images. With the camera quality increasing, many times there should be more data to start with. A
- phkahler 9y ago
  >> Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. When is a guess invaluable?
  - WhitneyLand 9y ago
    When it identifies an established terrorist and prevents a mass casualty event.
kevin_thibedeau 9y ago
"Ladies and gentlemen of the jury, we will definitively prove that the black smudge captured on camera was in fact a gun" Already being done today with DNA.
teekert 9y ago
Exactly, this may be possible: [0] but only of the NN has seen such images before, the output will match the training data but says nothing about reality. [0] https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010644ca848dd75.jpg https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010...

jopsen 9y ago

Sometimes US justice system seems very "approximate". So why not convict people based on interpolated evidence? - I'm joking of course :) hehe

WillReplyfFood 9y ago
You seem like a funny guy- the batmaNN interpolates that you are very likely a joker.
dahart 9y ago
Sadly, it actually happens sometimes. https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/ https://www.wired.com/2017/04/courts-using-ai-sentence-crimi... This thread a year ago worried about it too, but the paper itself seems implausible and problematic. https://news.ycombinator.com/item?id=12983827 https://news.ycombinator.com/item?id=12983827

ZeroGravitas 9y ago

To give specific examples from their test data, it added stubble to people who didn't have stubble, gave them a different shape of glasses, changed the color of cats, changed the color and brand of sport shoe. And even then, I'm a little suspicious of how close some of the images got to original without being given color information. It appears that info was either hidden in the original in a way not apparent to humans or was implicit in their data set in some way that would make it fail on photos of people with different skin tones.

omtinez 9y ago
I haven't read the paper in full detail, but reading between the lines I'm guessing that there's a significant portion of manual processing and hand waving involved. From the abstract, emphasis mine: > the second stage uses a pixel-wise nearest neighbor method to map the smoothed output to multiple high-quality, high-frequency outputs in a controllable manner. My interpretation is that they select training data by hand and generate a bunch of outputs. Repeating the process until they like the final result. From the paper: > we allow a user to have an arbitrarily-fine level of control through on-the-fly editing of the exemplar set (E.g., “resynthesize an image using the eye from this image and the nose from that one”).
- IshKebab 9y ago
  Could be pretty great for police sketch artists. (Although pretty misleading for juries too.)
  - adrianN 9y ago
    Just train the model with the suspect's Facebook photostream and presto you have convincing evidence.
- WhitneyLand 9y ago
  There's nothing weak or negative about that, it's exactly what'd you expect. Obviously for a given input there will be multiple plausible outputs. With any such system it would make sense to allow some control in choosing among the outputs.
chaosite 9y ago
Sounds similar to the problems with JBIG2 lossy compression. https://en.wikipedia.org/wiki/JBIG2#Disadvantages https://en.wikipedia.org/wiki/JBIG2#Disadvantages

gambiting 9y ago

No, but think of these blurred images as a "hash" - in an ideal situation, you only have one value that encodes to a certain hash value, right? So If you are given a hash X you technically can work out that it was derived from value Y - you're not getting back information that was lost - in a way it was merely encoded into the blurred image, and it should be possible to produce a real image which, when blurred, will match what you have. Don't get me wrong, I think we're still far far far off situation where we can get those reliably, but I can see how you could get the actual face out of a blurred image.

ComputerGuru 9y ago
> you only have one value that encodes to a certain hash value, right? Errr wrong. A perfect hash, yes. But they're never perfect. You have a collision domain and you hope that you don't have enough inputs to trigger a birthday paradox. Look at the pictures on the article. It's an outline of the shoe. That's your hash. ANY shoe with that general outline resolves to that same hash. If your input is objects found in the Oxford English Dictionary, you'll have low collisions. An elephant doesn't hash to that outline. But if your inputs is the Kohl's catalog, you'll have an unacceptable collision rate. Hashes are attempts at creating a _truncated_ "unique" representation of an input. They throw away data they hope isn't necessary to uniquely identify between possible inputs (bits). A perfect hash for all possible 32 bit values is 32 bits. You can't even have a collision free 31 bit hash. So back to the blurry security camera footage of a license plate or a face. Sure, that "hash" can reliably tell you that it wasn't a sasquatch that committed the robbery, but it literally doesn't contain the data necessary to _ever_ prove it was the suspect in question, even if the techs _can_ prove that the suspect hashes to the image in the footage.
- consp 9y ago
  Fortunately there is a limit: the universe (in a practical sense). You cannot encode all states it has in a hash as it would require more states than you want to encode as you already mentioned (pigeon hole). But representing macroscopic data like text (or basically anything bigger than atomic scale) uniquely can be done with 128+ bits. Double that and you are likely safe for collisions, assuming the method you use is uniform and not biased to some input. If you want ease collision examples you can take a look at people using CRC32 as hashes/digests. It is notoriously prone to collisions (since only 32 bits).

usrusr 9y ago

Yeah, replace the training set with cartoon characters and the crime show dialog goes like this: "Zoom! Enhance! Zoom! Enhance! Enhance! Oh my god it's full of Smurfs..."

deleted 9y ago
[deleted]

scarface74 9y ago

What you can do though, in limited circumstances, is create a still picture with more detail from a lower quality video. https://photo.stackexchange.com/questions/17098/csi-image-resolution-enhance-how-real-is-it https://photo.stackexchange.com/questions/17098/csi-image-re...

murkle 9y ago
It's a well-known technique in astronomy, eg https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz https://www.aanda.org/articles/aa/ps/2005/22/aa2320-04.ps.gz
- mcbits 9y ago
  For anyone put off by the .ps.gz, it's actually just a normal web page that links to the full article in HTML and PDF. Not sure what they were thinking with that URL. I almost didn't bother to look. (Maybe that's what they were thinking?)
dahart 9y ago
I seem to remember from my computer vision class way back when that there's a fundamental theoretical limit to the amount of detail you can get out of a moving sequence. Recovering frequencies a little higher than the pixel sampling is definitely possible, but I feel like it was maybe something like 10x theoretical maximum. I also get the feeling, from looking around at available software, that in practice achieving 2-3x is the most you can get in ideal conditions, and most video is far from ideal.

throwaway613834 9y ago

> I don't know whether this sounds like I'm splitting hairs Somewhat no, but somewhat yes. Thing is, while there can be lots of input images that generate the same output, it could be that only one (or a handful) of them would occur in reality. If this happens to sometimes be the case, and if you could somehow guarantee this was the case in some particular scenario, it could very well make sense to admit it as evidence. Of course, the issue is that figuring this out may not be possible...

sweezyjeezy 9y ago

> Except, and this is really the fundamental catch, it's not so much "enhance" as it is "project a believable substitute/interpretation". I would argue that this is a form of enhancement though, and in some cases will be enough to completely reconstruct the original image. For example, if I give you a scanned PDF, and you know for a fact that it was size 12 black Ariel text on a white background, this can feasibly let you reconstruct the original image perfectly. The 'prior' that has been encoded by the model from the large amount of other images increases the mutual information between grainy image and high-res. The catch is that uncertainty cannot be removed entirely, and you need to know that the target image comes from roughly the same distribution as the training set. But knowing this gives you information that is not encoded in the pixels themselves, so you can't necessarily argue that some enhancement is impossible. For example with celebrity images, if the model is able to figure out who is in the picture, this massively decreases the set of plausible outputs.

trevyn 9y ago
> The catch is that you need to know that the target image comes from roughly the same distribution as the training set. When humans think about "enhance", they imagine extracting subtle details that were not obvious from the original, which implies that they know very little about what distribution the original image comes from. If they did, they wouldn't have a need for "enhance" 99% of the time -- the remaining 1% is for artistic purposes, which this is indeed suited for. It'll be interesting to see how society copes with the removal of the "photographs = evidence" prior. > when enhancing celebrity images, if the model is able to figure out who is in the picture this massively decreases the set of plausible outputs. This is an excellent insight.
- ZenPsycho 9y ago
  Do you think knowing which state the license plate is from is enough prior knowledge?
yorwba 9y ago
See also https://dheera.net/projects/blur https://dheera.net/projects/blur

7952 9y ago

The benefit depends on how predictable the phenomenon is that your are interpolating from. Sometimes it will be quantitatively better than a low resolution version, sometimes not. A good example is with compression algorithms for media. They work because the sound or image is predictable. And they are ineffective when the input becomes more unpredictable. But if the output is all you have then running the decompression will probably be better than just reading the raw compressed data. But you have to be aware of the limitations.

jonathanstrange 9y ago

The white shoe output vs black shoe output illustrates this fairly well.

matt4077 9y ago

> You fundamentally can't get back information that has been destroyed/or never captured in the first place. I love this cliché. I've seen it thousands of times, and probably written it myself a few times. We all repeat stuff like that ad nauseam, without ever thinking. Because it's fundamentally flawed, especially in the context that it has usually been applied to, namely criticising the CSI:XYZ trope of "enhancing images". The truth is that there is a lot more information in a low-res image than meets the eye. Even if you can't read the letters on a license plate, it can be recovered by an algorithm. If the Empire State Building is in the background, it's likely to be a US license plate. Maybe only some letters would result in the photo's low-res pattern. If you only see part of a letter, knowing the font may allow you to rule out many letters or numbers etc... It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The error mostly appears to be in assuming that the information has been destroyed, when in reality it's often just obscured. And Neural Nets are excellent in squeezing all the information out noisy data.

rootw0rm 9y ago
It's not cliché, it's true. You fundamentally can't get back information that has been destroyed/or never captured in the first place. Yes, a low-res image has lots of information. You can process that information in many ways. Missing data can't just be magically blinked into existence though. Copy/pasting bits of guessed data is NOT getting back information that has been destroyed or never captured. Obscured data is very different from non-existent data. Could the software recreate a destroyed painting of mine based on a simple sketch? Of course not, because it would have to invent details it knows nothing about. I think it's almost dangerous to call this line of thinking cliché. It should be celebrated, not ridiculed.
amelius 9y ago
> It's similar to that guy who used Photoshop's swirl effect to hide his face, not knowing that the effect is deterministic, and can easily be undone. The effect does not only need to be deterministic, but also invertible. A low-res image has multiple "inverses" (yikes), supposedly each with an associated probability (if you would model it that way). So it would be more honest if the algorithm shows them all.

k__ 9y ago

On the other hand, this is what the brain does all the time.

eternalban 9y ago
Wouldn't it be ironic if a mystified and superstitious GAI emerges out of all these efforts.

gus_massa 9y ago

In comparison of output vs original it is clear that the skin color is not accurate.

WhitneyLand 9y ago

>we're interpolating or projecting information that is not there But that's not fully accurate either. Sometimes the information in total will really be a more accurate representation of reality than the blurred image. Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. It would be interesting to see the results starting with higher quality images. With the camera quality increasing, many times there should be more data to start with. A

phkahler 9y ago
>> Maybe it could be described as an educated guess, sometimes wrong, sometimes invaluable. When is a guess invaluable?
- WhitneyLand 9y ago
  When it identifies an established terrorist and prevents a mass casualty event.

kevin_thibedeau 9y ago

"Ladies and gentlemen of the jury, we will definitively prove that the black smudge captured on camera was in fact a gun" Already being done today with DNA.

teekert 9y ago

Exactly, this may be possible: [0] but only of the NN has seen such images before, the output will match the training data but says nothing about reality. [0] https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010644ca848dd75.jpg https://i.pinimg.com/originals/b5/29/1b/b5291bba7250abd12010...