6 ms·

The problem with techniques like this is that they fundamentally amount to ‘making a plausible guess as to what the image would look like’, since essentially th

The problem with techniques like this is that they fundamentally amount to ‘making a plausible guess as to what the image would look like’, since essentially they can’t extract information that is simply not there. There is a Shannon entropy limit here. Machine learning is really machine-enhanced educated-guesswork, which has its place but also has its limits.

oconnor663 7y ago
Counterpoint: The human brain converting a 2D image to a 3D model is educated-guesswork too :)
- war1025 7y ago
  This hits on an interesting point. There is an entropy limit to the message, but the message isn't actually the only data. One thing humans are great at is integrating existing knowledge into a messy situation and intuiting more than is available just from the raw message. I.e. The message has an entropy limit, but the message isn't the whole dataset.
  - MiroF 7y ago
    Yes and that's what this "lossy" conversion to daytime does as well, incorporate prior knowledge, but that prior knowledge is about how images of real world things function during night versus day.
  - gnramires 7y ago
    This doesn't quite match Information Theory. First of all, if you were to use the image as a communication channel, how much you could theoretically communicate is exactly the entropy of messages (by definition), and optimal communication means maximum entropy. Information theory already assumes shared existing knowledge in the form of codes; the codes essentially encode all this knowledge, which you could make an analogy in images to digits and shapes, etc. -- what makes them decodable is there is statistical redundancy, shapes do not occur arbitrarily (i.e. not every possible shape occurs, at least not with equal probability) and exhibit dependence between different pixels of a shape and even between shapes elsewhere in the image. Again the dependence is (generally) based on the statistics of the distribution of all possible images -- it essentially encodes all prior knowledge. This redundancy allows reconstruction of losses in on part of the image from data elsewhere. It's the same principle used in error correcting codes, except codes are designed, while shapes are mostly natural (except things like alphabets, which are designed and indeed follow some principles of codes). But because they're not designed there's not guarantee of having a unique/reliable decoding (i.e. you can get a distribution). I think that's a significant issue, because if your estimate doesn't match reality it could have important consequences for the use of the image: perhaps text goes from 'X is good' to 'X sucks'. In this case a few things could be done: 1) Have some kind of watermark indicating the image was enhanced by a neural network, and possibly contains false information; 2) Have some kind of indication of reliability of the image: it should encode the multimodality/confidence of the decoding distribution -- how many different solutions does this have. If it is more or less unique, it would show as high confidence; otherwise it would show a low confidence indicator; 3) Instead of trying to convey uncertainty, the system could simply give up in cases where there is too much uncertainty, i.e. leaving the image dark. This could be done locally or globally, although locally it introduces a lightning consistency problem. --- There's another important observation w.r.t. Information Theory/Statistics: it essentially assumes unbounded computational power (since this distribution could require analyzing arbitrarily large datasets). Of course this isn't true in reality. For example, the entropy of an encrypted of a redundant text is exactly the entropy of the plaintext string plus the entropy of the key (given an encryption ensamble or encryption prior) -- the process of (e.g. through brute force) finding the key doesn't concern statistics. However, with reasonable computational power, the (properly) encrypted stream is indistinguishable from a random string, hence it would have maximum entropy. So there are further computational limits beside statistical limits. In the case of encryption the function is again designed (to be not tractable), while in natural images the correlations are of simpler and hopefully more tractable nature (although I'm sure not always the case).
- amelius 7y ago
  Yes, therefore it is better to turn on the lights!
- bialpio 7y ago
  The courts & hopefully the jurors should be aware that humans are fallible & capable of lying, but may have hard time believing that cameras can lie as well.
  - gowld 7y ago
    The courts are terrible at that. Eyewitness testimony is awful but given gold status.
- astrobe_ 7y ago
  Two 2D Images. The brain doesn't have to guess much when it can use the parallax effect created by both eyes. That's why quite a few animals have two eyes. And I believe that's why we have two ears, too.
jdmichal 7y ago
I'm also not a fan of how the only part actually readable in the (a) original, which is part of the title in the front book, becomes completely whitewashed in (c). Where the model actually had the most information, it completely removed it in the result...
- edjrage 7y ago
  > the model actually had [...] information Wait, did it? Isn't the middle photo being shown for comparison only, rather than as an input?
  - jdmichal 7y ago
    I guess I might have misinterpreted the goal. If the goal was to make the image look like it was daylight, then maybe whitewashing that light reflection was the correct choice. If the goal was to "see in the dark", then it seems like a very bad choice. EDIT: Finally got the paper to load via the helpful wayback machine link provided in another thread. It looks like the goal was to simulate a long exposure with a short exposure. So whitewashing of "bright" areas in the original might be expected.
    - edjrage 7y ago
      Regardless of the goal, there's no way to get a more readable result if the data just isn't there. Whitewashing might simply be a result of that absence.
      - jdmichal 7y ago
        My point was that it whitewashed exactly those areas which had the most information. However, this is inline with their stated goal of mimicking a long exposure. It's not inline with "seeing in the dark", but that's not their goal.
        edjrage 7y ago
        If you insist on the same point, you didn't understand my initial reply. Again, I'm assuming the input is (a), not (b). But maybe you mean the lighter areas have the most information. If so, why? Just because there's more light? More light != more information. It could just be a bunch of noise.
6gvONxR4sf7o 7y ago
Agreed. It gets to an important point of the purpose of the photo. Photo as a record versus photo as an aesthetic piece. This hurts the photo as a record but improves the photo as an aesthetic piece. This would be a bad addition to a security camera, but perhaps a good addition to an instagram pipeline. There are plenty of other issues there, like is it good/healthy for stuff like instagram to be diverging away from records, creating unrealistic (or perhaps literally unreal) expectations. We're taking what are slowly inching closer to imaginative art pieces and presenting them as records.
- gowld 7y ago
  Photographs have never been faithful records. The map never the territory. There are always judgement calls. The whole concept of JPEG is to throw away information.
  - 6gvONxR4sf7o 7y ago
    Nothing's a perfect record, but a thing's purpose can be to serve as a record. Like if you stick seismometer readings into an academic paper, everyone will know there's noise, but that's okay. You did your best. But there are some changes we can agree are bad there. Now, if you stuck your seismometer into an autotuner and sync'd it with led zeppelin, that's okay. Just not if you're putting that into your paper. For a long time, photographs were typically used as records. Even when they were art, they were typically records of something. Soon, we'll be typically doing so much more with them and will have to accept that photos can, but frequently won't serve as records.
anigbrowl 7y ago
It's more than 'good enough' for most purposes. Matching other shots for Hollywood quality, probably not. For surveillance or the like it's fine. The things it's guessing poorly about are textures or colors. Being able to read the title on the books in the example photo is great; you could rely on the title for evidentiary purposes, the smaller text probably not so much. So for a security camera it would do poorly at identifying the color of a car, but might well be sufficient to read the license plate.
- melq 7y ago
  Using these processed images for evidence purposes sounds like precisely what the parent comment is concerned about.
- jameshart 7y ago
  Reading a license plate seems like precisely the kind of circumstances where spurious 'plausible interpretation' of limited data can cause trouble. You show in a courtroom a CNN-enhanced low light image of a car and it's there, literally 'clear as day' - the jury will find it pretty compelling. But maybe the data really wasn't there in the original image, and the CNN just filled in some blanks based on previous images of license plates, letters, and just random noise it had seen in the past. The worry is when these kinds of algorithms get built in to basic image capture processes, so you never even see the raw data, only data that has already been filtered through the inbuilt prejudices of the CNN enhancement suite. The camera never lies, but now it doesn't have to, because it can convince itself it saw something that wasn't really there...
  - anigbrowl 7y ago
    Of course, but what are the odds that the algorithm just lucked into the correct book title and other cover text? It doesn't have a dictionary or semantic network. You are right that the raw sensor data should always be preserved. But sticking with the license plate example, you could challenge a picture of a single car with a visible license plate far away in a wooded area, but it would be hard to refute a picture of the same car in a parking lot surrounded by other (non-suspect) vehicles whose presence there at the same time could be independently verified. In other words, if I can show that it accurately read the license plates of 9 other cars, the chances that it got yours wrong go way down. That's assuming a single photo taken in the dark by an investigator. With a fixed security camera you would have an even larger basis of comparison, with a population of hundreds or thousands of license plates against which to rate it. I predict that before long we'll see preemptive certification for devices warranting the reliability of their image pipeline out to a certain distance at either the manufacturing or installation stage.
deleted 7y ago
[deleted]
SilasX 7y ago
It's a general principle of information theory that, "to make inferences, you have to make assumptions".
Groxx 7y ago
And you can see some of its biases in the results, yeah. Look at the thick book's spine on the right, and compare the box around the title - "our result" has pretty significant staircasing instead of being a slightly-off-vertical line.
throwlaplace 7y ago
>‘making a plausible guess as to what the image would look like’ people bring this up all the time as hot-takes in these areas. it's conditional inference. it's no more disingenuous than linear regression.

oconnor663 7y ago

Counterpoint: The human brain converting a 2D image to a 3D model is educated-guesswork too :)

war1025 7y ago
This hits on an interesting point. There is an entropy limit to the message, but the message isn't actually the only data. One thing humans are great at is integrating existing knowledge into a messy situation and intuiting more than is available just from the raw message. I.e. The message has an entropy limit, but the message isn't the whole dataset.
- MiroF 7y ago
  Yes and that's what this "lossy" conversion to daytime does as well, incorporate prior knowledge, but that prior knowledge is about how images of real world things function during night versus day.
- gnramires 7y ago
  This doesn't quite match Information Theory. First of all, if you were to use the image as a communication channel, how much you could theoretically communicate is exactly the entropy of messages (by definition), and optimal communication means maximum entropy. Information theory already assumes shared existing knowledge in the form of codes; the codes essentially encode all this knowledge, which you could make an analogy in images to digits and shapes, etc. -- what makes them decodable is there is statistical redundancy, shapes do not occur arbitrarily (i.e. not every possible shape occurs, at least not with equal probability) and exhibit dependence between different pixels of a shape and even between shapes elsewhere in the image. Again the dependence is (generally) based on the statistics of the distribution of all possible images -- it essentially encodes all prior knowledge. This redundancy allows reconstruction of losses in on part of the image from data elsewhere. It's the same principle used in error correcting codes, except codes are designed, while shapes are mostly natural (except things like alphabets, which are designed and indeed follow some principles of codes). But because they're not designed there's not guarantee of having a unique/reliable decoding (i.e. you can get a distribution). I think that's a significant issue, because if your estimate doesn't match reality it could have important consequences for the use of the image: perhaps text goes from 'X is good' to 'X sucks'. In this case a few things could be done: 1) Have some kind of watermark indicating the image was enhanced by a neural network, and possibly contains false information; 2) Have some kind of indication of reliability of the image: it should encode the multimodality/confidence of the decoding distribution -- how many different solutions does this have. If it is more or less unique, it would show as high confidence; otherwise it would show a low confidence indicator; 3) Instead of trying to convey uncertainty, the system could simply give up in cases where there is too much uncertainty, i.e. leaving the image dark. This could be done locally or globally, although locally it introduces a lightning consistency problem. --- There's another important observation w.r.t. Information Theory/Statistics: it essentially assumes unbounded computational power (since this distribution could require analyzing arbitrarily large datasets). Of course this isn't true in reality. For example, the entropy of an encrypted of a redundant text is exactly the entropy of the plaintext string plus the entropy of the key (given an encryption ensamble or encryption prior) -- the process of (e.g. through brute force) finding the key doesn't concern statistics. However, with reasonable computational power, the (properly) encrypted stream is indistinguishable from a random string, hence it would have maximum entropy. So there are further computational limits beside statistical limits. In the case of encryption the function is again designed (to be not tractable), while in natural images the correlations are of simpler and hopefully more tractable nature (although I'm sure not always the case).
amelius 7y ago
Yes, therefore it is better to turn on the lights!
bialpio 7y ago
The courts & hopefully the jurors should be aware that humans are fallible & capable of lying, but may have hard time believing that cameras can lie as well.
- gowld 7y ago
  The courts are terrible at that. Eyewitness testimony is awful but given gold status.
astrobe_ 7y ago
Two 2D Images. The brain doesn't have to guess much when it can use the parallax effect created by both eyes. That's why quite a few animals have two eyes. And I believe that's why we have two ears, too.

jdmichal 7y ago

I'm also not a fan of how the only part actually readable in the (a) original, which is part of the title in the front book, becomes completely whitewashed in (c). Where the model actually had the most information, it completely removed it in the result...

edjrage 7y ago
> the model actually had [...] information Wait, did it? Isn't the middle photo being shown for comparison only, rather than as an input?
- jdmichal 7y ago
  I guess I might have misinterpreted the goal. If the goal was to make the image look like it was daylight, then maybe whitewashing that light reflection was the correct choice. If the goal was to "see in the dark", then it seems like a very bad choice. EDIT: Finally got the paper to load via the helpful wayback machine link provided in another thread. It looks like the goal was to simulate a long exposure with a short exposure. So whitewashing of "bright" areas in the original might be expected.
  - edjrage 7y ago
    Regardless of the goal, there's no way to get a more readable result if the data just isn't there. Whitewashing might simply be a result of that absence.
    - jdmichal 7y ago
      My point was that it whitewashed exactly those areas which had the most information. However, this is inline with their stated goal of mimicking a long exposure. It's not inline with "seeing in the dark", but that's not their goal.
      - edjrage 7y ago
        If you insist on the same point, you didn't understand my initial reply. Again, I'm assuming the input is (a), not (b). But maybe you mean the lighter areas have the most information. If so, why? Just because there's more light? More light != more information. It could just be a bunch of noise.

6gvONxR4sf7o 7y ago

Agreed. It gets to an important point of the purpose of the photo. Photo as a record versus photo as an aesthetic piece. This hurts the photo as a record but improves the photo as an aesthetic piece. This would be a bad addition to a security camera, but perhaps a good addition to an instagram pipeline. There are plenty of other issues there, like is it good/healthy for stuff like instagram to be diverging away from records, creating unrealistic (or perhaps literally unreal) expectations. We're taking what are slowly inching closer to imaginative art pieces and presenting them as records.

gowld 7y ago
Photographs have never been faithful records. The map never the territory. There are always judgement calls. The whole concept of JPEG is to throw away information.
- 6gvONxR4sf7o 7y ago
  Nothing's a perfect record, but a thing's purpose can be to serve as a record. Like if you stick seismometer readings into an academic paper, everyone will know there's noise, but that's okay. You did your best. But there are some changes we can agree are bad there. Now, if you stuck your seismometer into an autotuner and sync'd it with led zeppelin, that's okay. Just not if you're putting that into your paper. For a long time, photographs were typically used as records. Even when they were art, they were typically records of something. Soon, we'll be typically doing so much more with them and will have to accept that photos can, but frequently won't serve as records.

anigbrowl 7y ago

It's more than 'good enough' for most purposes. Matching other shots for Hollywood quality, probably not. For surveillance or the like it's fine. The things it's guessing poorly about are textures or colors. Being able to read the title on the books in the example photo is great; you could rely on the title for evidentiary purposes, the smaller text probably not so much. So for a security camera it would do poorly at identifying the color of a car, but might well be sufficient to read the license plate.

melq 7y ago
Using these processed images for evidence purposes sounds like precisely what the parent comment is concerned about.
jameshart 7y ago
Reading a license plate seems like precisely the kind of circumstances where spurious 'plausible interpretation' of limited data can cause trouble. You show in a courtroom a CNN-enhanced low light image of a car and it's there, literally 'clear as day' - the jury will find it pretty compelling. But maybe the data really wasn't there in the original image, and the CNN just filled in some blanks based on previous images of license plates, letters, and just random noise it had seen in the past. The worry is when these kinds of algorithms get built in to basic image capture processes, so you never even see the raw data, only data that has already been filtered through the inbuilt prejudices of the CNN enhancement suite. The camera never lies, but now it doesn't have to, because it can convince itself it saw something that wasn't really there...
- anigbrowl 7y ago
  Of course, but what are the odds that the algorithm just lucked into the correct book title and other cover text? It doesn't have a dictionary or semantic network. You are right that the raw sensor data should always be preserved. But sticking with the license plate example, you could challenge a picture of a single car with a visible license plate far away in a wooded area, but it would be hard to refute a picture of the same car in a parking lot surrounded by other (non-suspect) vehicles whose presence there at the same time could be independently verified. In other words, if I can show that it accurately read the license plates of 9 other cars, the chances that it got yours wrong go way down. That's assuming a single photo taken in the dark by an investigator. With a fixed security camera you would have an even larger basis of comparison, with a population of hundreds or thousands of license plates against which to rate it. I predict that before long we'll see preemptive certification for devices warranting the reliability of their image pipeline out to a certain distance at either the manufacturing or installation stage.

deleted 7y ago

[deleted]

SilasX 7y ago

It's a general principle of information theory that, "to make inferences, you have to make assumptions".

Groxx 7y ago

And you can see some of its biases in the results, yeah. Look at the thick book's spine on the right, and compare the box around the title - "our result" has pretty significant staircasing instead of being a slightly-off-vertical line.

throwlaplace 7y ago

>‘making a plausible guess as to what the image would look like’ people bring this up all the time as hot-takes in these areas. it's conditional inference. it's no more disingenuous than linear regression.