7 ms·

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality

What I don't like about AI processed images is that they are not real. I can't go past the fact that I am not looking at the picture as it looks like in reality but somehow smart approximation of the world that is not necessarily true.

robotresearcher 6y ago
You'd get used to it very quickly. Like you got used to TV and film, which differ substantially from an in-person view in many ways.
GuB-42 6y ago
Video is not real either. The perspective is different and the colors are wrong. It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference. Using neural networks for compression is far from being a new concept, and the result is not more or less real than any other technique. It is just that Nvidia implementation is really good at keeping the most important details in a small size. If you want a more "real" image, you can just use the AI as a predictor and use the remaining bits to encode the difference, like in a traditional MPEG-style codec.
- agumonkey 6y ago
  I understand what parent means, usually processing was too linear to alter the meaning of what you see, it's distorded, altered, maybe filtered and enhanced but in general, it's the same. With ml/dl you can really be sent .. whatever.
- captainclam 6y ago
  In some ways, of course, I agree, but as some other commenters have pointed out, this is essentially a "deepfake" of yourself...in principle there's no restriction on what one could use as their "avatar." As much was demonstrated in the "Different Output Styles" slide. So I'd have a hard time saying this is not dramatically less "real" than some lossy compression technique. Is there some way to formalize "realness?" Maybe it would be inversely proportional to the hardness of manipulating the medium as a user..?
  - GuB-42 6y ago
    Maybe simply using an objective video quality measurement against the original picture would do the trick. Something like PSNR or SSIM. A "deepfake" is likely to score low on such a score that doesn't depend on high level visual perception. Also know that you can "deepfake" yourself using a traditional video encoder, just change the keyframe to someone else's face. Of course, it will look broken and totally unconvincing but because of motion compensation, you can sort of map the movement of your face on someone else's face. The technique in the paper simply has way better motion compensation, so good that it still works if you change the keyframe. Traditional video compression algorithms don't work like that because they are not just for talking faces and can't use such advanced techniques for performance and ease of implementation reasons.
    - shard 6y ago
      I think the point is that for very low bandwidth, the "deepfake" version will score higher than the heavily compressed video stream for PSNR.
- mrfredward 6y ago
  The distinction that makes ai compression creepy is that it can use prior knowledge of what a face is. Traditional compression has no high level knowledge of what a video call looks like, the algorithms are about patterns in how pixels change in time and space, so the artifacts from the algorithm are pixel-based effects (blockiness, blurriness, etc). A neural net that has been trained on a million faces and is setup to draw a face may draw a perfectly clear image of a face on low bandwidth...but when it doesn't have bandwidth to be accurate, it doesn't blur things, it fills in details learned from looking at strangers who aren't on the call.
  - Traster 6y ago
    You're sitting at your computer, you're watching your colleagues are discussing something in detail, slowly you start to notice - your two black colleagues are starting to literally look the same. That's weird, are you slowly getting more racist? oh right, your bandwith dropped and your AI upscaler has decided to upscale all your black colleagues up to some stereotype it got trained with. I'm afraid I suspect this isn't that far fetched.
- make3 6y ago
  come on man there's a difference between a GAN hallucinating 90 % of the image and a very predictable compression algorithm where both parties understand what's going on
- iandanforth 6y ago
  "the result is not more or less real than any other technique" I disagree. If real / not real is binary, sure, it's not real. But if we allow "real" to be a range of values, it is less real.
ashika 6y ago
your brain does lots of preprocessing to create a stable view of "reality", glossing over blind spots, retinal blood flow, etc. in a way we are always hallucinating, and it seems natural that computers should learn what we are actually paying attention to in video, and focus on that
- wruza 6y ago
  Yes! E.g. AAA games render the entire 4k screen with the highest graphics, but only a small spot hits the part of retina that can perceive jagged edges or lighting details.
arthur2e5 6y ago
It could be bad for telemedicine, since the NN may gloss over some "weird" pathological features not found in its training set. The current setup involving an initial image and keypoints should be fine though, as long as the initial image (keyframe?) is not messed with and is transmitted frequently enough.
deleted 6y ago
[deleted]
op1818 6y ago
any lossy video-compression algorithm face the same challenge. what you are seeing is artificial and is constructed by the algorithm to minimize the perceptual difference between the real and the constructed video feed.
wruza 6y ago
That's how your brain works as well. Adding a secondary, uncontrolled layer of perception is definitely not the same thing, but in general, what you "see" is barely "real".
thrwyoilarticle 6y ago
The raw sensor data for a digital camera looks something like this https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Filter_(120px-Colorful_spring_garden).png https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Fil... In addition to the colour grading and interpolation, lens defects might be 'fixed' before you ever get data on disk. ML is just really good interpolation.
alkonaut 6y ago
If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount of data that makes the receiver see what I want. If that means I send 0.1mbps key point data and 1mbps diff pixels then that’s what I want to send. Is that how this works? Because it should be how it works... Not sure if this is a viable way of compressing the video stream or if a transmission of the “diff” would be too costly. If this method would give 1/2 the original bandwidth I’d think it’s more impressive than cutting 3 orders of magnitude via “avatars”.
- ssivark 6y ago
  You’re thinking along the right lines, but the challenge is that a raw diff will have the same number of pixels as the raw image, so no compression in bandwidth. So, how do we represent the diff/residue also with fewer numbers? At that point it’s the same as choosing better parameters within some clever encoding (be it pre-designed like JPEG or H.264 or learned via ML).
  - alkonaut 6y ago
    I was thinking that subtracting the predicted image would give an image that has more zeroes and compresses better (much like dct+quantization for jpeg). After all, any time the neural network would predict an area of the image almost exactly, it can be omitted from the diff stream completely too.
    - ssivark 6y ago
      The diff will be low norm (for some suitable norm), but it needn’t be sparse in pixel space. It could be sparse in some other spaces (Eg: wavelet basis), but that’s right where the challenge lies — finding a basis where the data (and/or the residue) to be transmitted is sparse.
- 6y ago

robotresearcher 6y ago

You'd get used to it very quickly. Like you got used to TV and film, which differ substantially from an in-person view in many ways.

GuB-42 6y ago

Video is not real either. The perspective is different and the colors are wrong. It is even less real after it went through lossy compression. The whole point of a lossy compression is to remove details that you perceive as unimportant. For example, leaves on a tree may look like a greeny mess, but that's fine, from afar, you don't make a difference. Using neural networks for compression is far from being a new concept, and the result is not more or less real than any other technique. It is just that Nvidia implementation is really good at keeping the most important details in a small size. If you want a more "real" image, you can just use the AI as a predictor and use the remaining bits to encode the difference, like in a traditional MPEG-style codec.

agumonkey 6y ago
I understand what parent means, usually processing was too linear to alter the meaning of what you see, it's distorded, altered, maybe filtered and enhanced but in general, it's the same. With ml/dl you can really be sent .. whatever.
captainclam 6y ago
In some ways, of course, I agree, but as some other commenters have pointed out, this is essentially a "deepfake" of yourself...in principle there's no restriction on what one could use as their "avatar." As much was demonstrated in the "Different Output Styles" slide. So I'd have a hard time saying this is not dramatically less "real" than some lossy compression technique. Is there some way to formalize "realness?" Maybe it would be inversely proportional to the hardness of manipulating the medium as a user..?
- GuB-42 6y ago
  Maybe simply using an objective video quality measurement against the original picture would do the trick. Something like PSNR or SSIM. A "deepfake" is likely to score low on such a score that doesn't depend on high level visual perception. Also know that you can "deepfake" yourself using a traditional video encoder, just change the keyframe to someone else's face. Of course, it will look broken and totally unconvincing but because of motion compensation, you can sort of map the movement of your face on someone else's face. The technique in the paper simply has way better motion compensation, so good that it still works if you change the keyframe. Traditional video compression algorithms don't work like that because they are not just for talking faces and can't use such advanced techniques for performance and ease of implementation reasons.
  - shard 6y ago
    I think the point is that for very low bandwidth, the "deepfake" version will score higher than the heavily compressed video stream for PSNR.
mrfredward 6y ago
The distinction that makes ai compression creepy is that it can use prior knowledge of what a face is. Traditional compression has no high level knowledge of what a video call looks like, the algorithms are about patterns in how pixels change in time and space, so the artifacts from the algorithm are pixel-based effects (blockiness, blurriness, etc). A neural net that has been trained on a million faces and is setup to draw a face may draw a perfectly clear image of a face on low bandwidth...but when it doesn't have bandwidth to be accurate, it doesn't blur things, it fills in details learned from looking at strangers who aren't on the call.
- Traster 6y ago
  You're sitting at your computer, you're watching your colleagues are discussing something in detail, slowly you start to notice - your two black colleagues are starting to literally look the same. That's weird, are you slowly getting more racist? oh right, your bandwith dropped and your AI upscaler has decided to upscale all your black colleagues up to some stereotype it got trained with. I'm afraid I suspect this isn't that far fetched.
make3 6y ago
come on man there's a difference between a GAN hallucinating 90 % of the image and a very predictable compression algorithm where both parties understand what's going on
iandanforth 6y ago
"the result is not more or less real than any other technique" I disagree. If real / not real is binary, sure, it's not real. But if we allow "real" to be a range of values, it is less real.

ashika 6y ago

your brain does lots of preprocessing to create a stable view of "reality", glossing over blind spots, retinal blood flow, etc. in a way we are always hallucinating, and it seems natural that computers should learn what we are actually paying attention to in video, and focus on that

wruza 6y ago
Yes! E.g. AAA games render the entire 4k screen with the highest graphics, but only a small spot hits the part of retina that can perceive jagged edges or lighting details.

arthur2e5 6y ago

It could be bad for telemedicine, since the NN may gloss over some "weird" pathological features not found in its training set. The current setup involving an initial image and keypoints should be fine though, as long as the initial image (keyframe?) is not messed with and is transmitted frequently enough.

deleted 6y ago

[deleted]

op1818 6y ago

any lossy video-compression algorithm face the same challenge. what you are seeing is artificial and is constructed by the algorithm to minimize the perceptual difference between the real and the constructed video feed.

wruza 6y ago

That's how your brain works as well. Adding a secondary, uncontrolled layer of perception is definitely not the same thing, but in general, what you "see" is barely "real".

thrwyoilarticle 6y ago

The raw sensor data for a digital camera looks something like this https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Filter_(120px-Colorful_spring_garden).png https://commons.wikimedia.org/wiki/File:Normal_and_Bayer_Fil... In addition to the colour grading and interpolation, lens defects might be 'fixed' before you ever get data on disk. ML is just really good interpolation.

alkonaut 6y ago

If the sending machine (that films my face) does the same decoding I know will be on the other end, then diffs the raw video with the decoded video and finally sends both things, then the receiver should be able to always piece together a 100% reproduction of the actual video feed on my end. The transmitter can always predict exactly what the receiver will decode, so the correct amount of data to send is the amount of data that makes the receiver see what I want. If that means I send 0.1mbps key point data and 1mbps diff pixels then that’s what I want to send. Is that how this works? Because it should be how it works... Not sure if this is a viable way of compressing the video stream or if a transmission of the “diff” would be too costly. If this method would give 1/2 the original bandwidth I’d think it’s more impressive than cutting 3 orders of magnitude via “avatars”.

ssivark 6y ago
You’re thinking along the right lines, but the challenge is that a raw diff will have the same number of pixels as the raw image, so no compression in bandwidth. So, how do we represent the diff/residue also with fewer numbers? At that point it’s the same as choosing better parameters within some clever encoding (be it pre-designed like JPEG or H.264 or learned via ML).
- alkonaut 6y ago
  I was thinking that subtracting the predicted image would give an image that has more zeroes and compresses better (much like dct+quantization for jpeg). After all, any time the neural network would predict an area of the image almost exactly, it can be omitted from the diff stream completely too.
  - ssivark 6y ago
    The diff will be low norm (for some suitable norm), but it needn’t be sparse in pixel space. It could be sparse in some other spaces (Eg: wavelet basis), but that’s right where the challenge lies — finding a basis where the data (and/or the residue) to be transmitted is sparse.
6y ago