4 ms·

ESRI claims: "Super Resolution is an image transformation technique with the help of which we can improve the quality of image and recover high resolution image

by elil17 4y ago

ESRI claims: "Super Resolution is an image transformation technique with the help of which we can improve the quality of image and recover high resolution image from a given low resolution image." I would argue this is not true - rather, super resolution generates a plausible high resolution image that would look like a given low resolution image if it were downscaled (i.e., it's not going to recover real details, it's just going to sharpen lines and potentially show details that look real but might not be). Edit: As an example, in the lower half of figure 5, the algorithm displays circular white dots on the roof, when in reality they are rectangles. An image analyst using this tool might, for example, incorrectly geolocate an image taken on the ground. This tool probably needs a warning label on it.

littlestymaar 4y ago
I've seen this exact same problem apply to the war in Ukraine for armored vehicle identification. There's a blurred still of a video showing an abandoned unrecognizable tank, and some guys arrives and say: “look I enhanced the image quality and now we can clearly see that it's a T80-BVM because we can see [insert the appropriate detail here]” where the “details” are by design completely made up by the neural net. This kind of image transformation is cool when all you care about is esthetic quality of the image, but if you want to see details, then Mark I Eyeball is the best tool you can hope for, because if the thing is unrecognizable, you'll know it and won't make things up to pretend it is not.
7952 4y ago
It is perfectly possible that the interpolated image is a better statistical fit to reality than the source image. This is more intuitive with height data. For example a 100m grid could have a 10m cell right beside a 20m cell. A point on the boundary between both cells is more likely to be at 15m than either 10m or 20m. And you can improve that estimate using other nuances. Picking a predicted value can be more truthful. I agree that caution is important but that is true with any imagery analysis. A human analysing imagery will already be using a lot of intuition as it is.
- Tostino 4y ago
  But without this, all you have is a blurry image, in which you cannot accurately make out the details. You may wait until you get a better image due to different atmospheric conditions, to determine what they actually were. With this, you may get something hallucinated which looks very unexpected. Or in military contexts, maybe looks like a threat.
  - 7952 4y ago
    Someone with a little training and practice can get lots of information out of these kind of "blurry" images. You can make educated guesses based on understanding of the real world. Imagine you are looking for swimming pools to find water to fight a wild fire. All you have is a "blurry" image. That blue shape on the image could be a weirdly shaped tent or patio. And that would be very obvious on high resolution drone imagery. But 99% of the time it will be a pool and that is good enough. All data has limitations and this is no different.
    - Tostino 4y ago
      That is fair enough, I was weighing the cost of getting it wrong very highly (probably more than I should have).
- mistrial9 4y ago
  shiny plastic has smoother reflections, pleasing mathematically.. reality is noisy and full of exceptional things
agolio 4y ago
This is absolutely true. A very dangerous use-case of DL upscaling. Even without claiming it could "recover" imagery it is a dangerous feature as users could be misconceived, but ArcGIS explicitly claiming that? Outrageous. To make absolutely clear: any details revealed by this upscaling do not exist. They are guesses based upon other imagery
galangalalgol 4y ago
I'd be concerned it might remove data too. Its making a very statistically probable image given the input, so if there is an emergency, which by their nature are uncommon, it might edit it out?
- contravariant 4y ago
  It could but it's easy enough to force it to always agree with the underlying low resolution image (though I'm not sure if that is done here).
  - elil17 4y ago
    I believe CycleGAN does some thing along those lines (it's called "cycle consistency loss")
timthorn 4y ago
> it's not going to recover real details, it's just going to sharpen lines and potentially show details that look real but might not be Super resolution as a name covers a number of techniques, but some absolutely can recover real details. Eg Superresolution from video will integrate over time, allowing sub-pixel accuracy. Imagine a grid of pixels, each of which covering a defined area of the source. Now, if that grid moves, the area covered by each pixel will be slightly different. The differences between frames can then be used to determine with accuracy a higher resolution output image.
- elil17 4y ago
  That's really cool. However, that's not what ESRI is doing. You can see evidence of the mistakes it will make in their own blog post.
  - dr_zoidberg 4y ago
    Indeed, in this case ESRI is talking about deep-learning based single image superresolution. With multiple images, even something as simple as shift-and-add can recover details (and lower noise in the picture), but you do require having multiple images. Video being a sequence of images, with large or small movement in between frames, can be an ideal source for robust SR algorithms. To complicate things further, there are deep-learing based multi-image SR algorithms too.
jofer 4y ago
That's exactly true. It worries me a lot that ESRI (who really should know better) is using the term "resolution" for this. This sort of superresolution is just interpolation. By definition, you're not adding more information density. It cannot increase resolution. It can increase edge response, but that's critically not resolution. There are superresolution methods that _can_ increase resolution, but they're in the form of combining multiple captures closely spaced with each other that are slightly offset. "Drizzle" was the original method in astrophysics, and while that method is long gone, it is common to do in remote sensing imagery for many instrument types. Similarly pansharpening (which interpolates multispectral information using a higher resolution black and white image captured at the same time) can actually improve resolution and is commonly used. This is not improving resolution in any way. Two objects that blur together into one will still appear as one object.
- LASR 4y ago
  This is more like extrapolation rather than interpolation. Seems a bit scary to me if used without caveats.
ska 4y ago
Unfortunately it's an abuse of terminology that "stuck" a long time ago, and I think the ship has sailed for fixing that (at least in the image processing community). These sort of techniques were named in reference to more principled techniques involving multiple measurements and precise control of imaging systems, allowing you to estimate higher resolutions than your sensor could achieve.
- mistrial9 4y ago
  > I think the ship has sailed for fixing that (at least in the image processing community) no - if you relent like that perhaps
  - ska 4y ago
    oh i haven't relented, I just don't think it will do much good :)
renewiltord 4y ago
What is it about tech communities that this sort of clarification needs to be made. It's like if someone were to say "okay, I'll just hang out here, I'm not going anywhere" and someone were to respond "technically, that depends on your frame of reference. we're all orbiting the sun and therefore we are going somewhere". At this point everyone knows the memes about CSI Enhance, the Xerox compression image hallucinations, blah blah. Must we constantly revisit subjects at a grade-school level? https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... 8e2fcd496
- jofer 4y ago
  It's because it's a frequent source of confusion in the actual industry. I _constantly_ hear people claim that this method really does add information. My own company constantly makes similar claims, and internally, even for people who actually do work in the field, they honestly do think it adds resolution.
potatolicious 4y ago
Totally agreed. I take some issue with the verbiage, particularly: > "Figure 1: Recovering high resolution image from low resolution" This is terribly misleading. There is no data being "recovered" here. A ML model is guessing at the result based on other training data. It may (and in fact is likely to) make stuff up entirely based on what it thinks should be there. I'm generally pretty live and let live when it comes to ML-based upscaling, because if some drawing or personal photograph has some artifacting it's pretty harmless. But when you're doing it in a tool whose data will be relied upon for Real Stuff, one needs to be painfully accurate when it comes to what the system does and its limitations.

by elil17 4y ago

littlestymaar 4y ago

I've seen this exact same problem apply to the war in Ukraine for armored vehicle identification. There's a blurred still of a video showing an abandoned unrecognizable tank, and some guys arrives and say: “look I enhanced the image quality and now we can clearly see that it's a T80-BVM because we can see [insert the appropriate detail here]” where the “details” are by design completely made up by the neural net. This kind of image transformation is cool when all you care about is esthetic quality of the image, but if you want to see details, then Mark I Eyeball is the best tool you can hope for, because if the thing is unrecognizable, you'll know it and won't make things up to pretend it is not.

7952 4y ago

It is perfectly possible that the interpolated image is a better statistical fit to reality than the source image. This is more intuitive with height data. For example a 100m grid could have a 10m cell right beside a 20m cell. A point on the boundary between both cells is more likely to be at 15m than either 10m or 20m. And you can improve that estimate using other nuances. Picking a predicted value can be more truthful. I agree that caution is important but that is true with any imagery analysis. A human analysing imagery will already be using a lot of intuition as it is.

Tostino 4y ago
But without this, all you have is a blurry image, in which you cannot accurately make out the details. You may wait until you get a better image due to different atmospheric conditions, to determine what they actually were. With this, you may get something hallucinated which looks very unexpected. Or in military contexts, maybe looks like a threat.
- 7952 4y ago
  Someone with a little training and practice can get lots of information out of these kind of "blurry" images. You can make educated guesses based on understanding of the real world. Imagine you are looking for swimming pools to find water to fight a wild fire. All you have is a "blurry" image. That blue shape on the image could be a weirdly shaped tent or patio. And that would be very obvious on high resolution drone imagery. But 99% of the time it will be a pool and that is good enough. All data has limitations and this is no different.
  - Tostino 4y ago
    That is fair enough, I was weighing the cost of getting it wrong very highly (probably more than I should have).
mistrial9 4y ago
shiny plastic has smoother reflections, pleasing mathematically.. reality is noisy and full of exceptional things

agolio 4y ago

This is absolutely true. A very dangerous use-case of DL upscaling. Even without claiming it could "recover" imagery it is a dangerous feature as users could be misconceived, but ArcGIS explicitly claiming that? Outrageous. To make absolutely clear: any details revealed by this upscaling do not exist. They are guesses based upon other imagery

galangalalgol 4y ago

I'd be concerned it might remove data too. Its making a very statistically probable image given the input, so if there is an emergency, which by their nature are uncommon, it might edit it out?

contravariant 4y ago
It could but it's easy enough to force it to always agree with the underlying low resolution image (though I'm not sure if that is done here).
- elil17 4y ago
  I believe CycleGAN does some thing along those lines (it's called "cycle consistency loss")

timthorn 4y ago

> it's not going to recover real details, it's just going to sharpen lines and potentially show details that look real but might not be Super resolution as a name covers a number of techniques, but some absolutely can recover real details. Eg Superresolution from video will integrate over time, allowing sub-pixel accuracy. Imagine a grid of pixels, each of which covering a defined area of the source. Now, if that grid moves, the area covered by each pixel will be slightly different. The differences between frames can then be used to determine with accuracy a higher resolution output image.

elil17 4y ago
That's really cool. However, that's not what ESRI is doing. You can see evidence of the mistakes it will make in their own blog post.
- dr_zoidberg 4y ago
  Indeed, in this case ESRI is talking about deep-learning based single image superresolution. With multiple images, even something as simple as shift-and-add can recover details (and lower noise in the picture), but you do require having multiple images. Video being a sequence of images, with large or small movement in between frames, can be an ideal source for robust SR algorithms. To complicate things further, there are deep-learing based multi-image SR algorithms too.

jofer 4y ago

That's exactly true. It worries me a lot that ESRI (who really should know better) is using the term "resolution" for this. This sort of superresolution is just interpolation. By definition, you're not adding more information density. It cannot increase resolution. It can increase edge response, but that's critically not resolution. There are superresolution methods that _can_ increase resolution, but they're in the form of combining multiple captures closely spaced with each other that are slightly offset. "Drizzle" was the original method in astrophysics, and while that method is long gone, it is common to do in remote sensing imagery for many instrument types. Similarly pansharpening (which interpolates multispectral information using a higher resolution black and white image captured at the same time) can actually improve resolution and is commonly used. This is not improving resolution in any way. Two objects that blur together into one will still appear as one object.

LASR 4y ago
This is more like extrapolation rather than interpolation. Seems a bit scary to me if used without caveats.

ska 4y ago

Unfortunately it's an abuse of terminology that "stuck" a long time ago, and I think the ship has sailed for fixing that (at least in the image processing community). These sort of techniques were named in reference to more principled techniques involving multiple measurements and precise control of imaging systems, allowing you to estimate higher resolutions than your sensor could achieve.

mistrial9 4y ago
> I think the ship has sailed for fixing that (at least in the image processing community) no - if you relent like that perhaps
- ska 4y ago
  oh i haven't relented, I just don't think it will do much good :)

renewiltord 4y ago

What is it about tech communities that this sort of clarification needs to be made. It's like if someone were to say "okay, I'll just hang out here, I'm not going anywhere" and someone were to respond "technically, that depends on your frame of reference. we're all orbiting the sun and therefore we are going somewhere". At this point everyone knows the memes about CSI Enhance, the Xerox compression image hallucinations, blah blah. Must we constantly revisit subjects at a grade-school level? https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... 8e2fcd496

jofer 4y ago
It's because it's a frequent source of confusion in the actual industry. I _constantly_ hear people claim that this method really does add information. My own company constantly makes similar claims, and internally, even for people who actually do work in the field, they honestly do think it adds resolution.

potatolicious 4y ago

Totally agreed. I take some issue with the verbiage, particularly: > "Figure 1: Recovering high resolution image from low resolution" This is terribly misleading. There is no data being "recovered" here. A ML model is guessing at the result based on other training data. It may (and in fact is likely to) make stuff up entirely based on what it thinks should be there. I'm generally pretty live and let live when it comes to ML-based upscaling, because if some drawing or personal photograph has some artifacting it's pretty harmless. But when you're doing it in a tool whose data will be relied upon for Real Stuff, one needs to be painfully accurate when it comes to what the system does and its limitations.