8 ms·
Super Resolution: Image-to-Image Translation Using Deep Learning in ArcGIS Pro
- dogma1138 4y agoArcGIS is used in a lot of life critical applications including law enforcement, emergency services and national defense, even for less serious applications such as land management I’m not sure that adding LSD mode is ever appropriate for these use cases.
- jph00 4y agoA lot of replies here which are based on out of date information and misunderstandings about deep learning super resolution (DLSR). Firstly, this isn't just doing edge detection etc as happened 20 years ago. It's creating a deep learning model which fills in information based on extrapolating from looking at lots of other images and making an educated guess as to what is most likely to be present. It's a fairly new approach and works much better than previous methods, given enough training data. This is, of course, imperfect, but claims that it's just to "look good" and is of no practical benefit are incorrect. For instance, we published a paper using DLSR in microscopy, to help experts identify synaptic vesicles. In the original images the experts had a 3x higher false negative rate compared to the DLSR images. Finally, claims that this can't be called "super resolution" are ignoring years of peer reviewed published research in which this is exactly the name used for this approach. Yes, super resolution can also be achieved using other methods which take advantage of additional data (such as multiple images), but that does not mean DLSR has to be called something else. (I'm not involved in ArcGIS Pro, but am the lead author of the fastai framework which underpins ArcGIS training behind the scenes, and have published papers and tutorials on super resolution using deep learning.)
- nmaleki 4y agoThis is great! It would be immediately useful to me if the trained dataset was uploaded to ArcGIS online so I can access it as a data layer through the API
- ubicomp 4y agoSuper exciting. Have been wanting to see this for years. Great job, Esri!
- elil17 4y agoESRI claims: "Super Resolution is an image transformation technique with the help of which we can improve the quality of image and recover high resolution image from a given low resolution image." I would argue this is not true - rather, super resolution generates a plausible high resolution image that would look like a given low resolution image if it were downscaled (i.e., it's not going to recover real details, it's just going to sharpen lines and potentially show details that look real but might not be). Edit: As an example, in the lower half of figure 5, the algorithm displays circular white dots on the roof, when in reality they are rectangles. An image analyst using this tool might, for example, incorrectly geolocate an image taken on the ground. This tool probably needs a warning label on it.
- littlestymaar 4y agoI've seen this exact same problem apply to the war in Ukraine for armored vehicle identification. There's a blurred still of a video showing an abandoned unrecognizable tank, and some guys arrives and say: “look I enhanced the image quality and now we can clearly see that it's a T80-BVM because we can see [insert the appropriate detail here]” where the “details” are by design completely made up by the neural net. This kind of image transformation is cool when all you care about is esthetic quality of the image, but if you want to see details, then Mark I Eyeball is the best tool you can hope for, because if the thing is unrecognizable, you'll know it and won't make things up to pretend it is not.
- 7952 4y agoIt is perfectly possible that the interpolated image is a better statistical fit to reality than the source image. This is more intuitive with height data. For example a 100m grid could have a 10m cell right beside a 20m cell. A point on the boundary between both cells is more likely to be at 15m than either 10m or 20m. And you can improve that estimate using other nuances. Picking a predicted value can be more truthful. I agree that caution is important but that is true with any imagery analysis. A human analysing imagery will already be using a lot of intuition as it is.
- Tostino 4y ago
- gcapu 4y agoI like the results, but a part of the improvement is due to the increase in contrast, clarity and sharpness. Photoshop can get you half way through.
- wokwokwok 4y ago> Super Resolution is an image transformation technique with the help of which we can improve the quality of image and recover high resolution image from a given low resolution image Saying you can “recover” high resolution seems like a bit of a stretch. Uh… I mean, it looks pretty but it’s basically just invented a bunch of random probabilistic crap on your map right? Is this a thing? People actually want their maps to be pretty and wrong? O_o Strange times. Particularly blurry blocks that are “maybe cars?” in a backyard being turned into high resolution cars. Or blue splat into “100% a swimming pool” seems… pretty dubious.
- practice9 4y agoWell, if a blue splat is (just an example) 93% statistically close to being a swimming pool, there is nothing wrong with upscaling it for many cases where absolute accuracy is not required. Larger and better quality datasets will decrease the amount of possible failure modes after a few development / training iterations of this algorithm Of course if outliers / anomalies are very important for persons business or use-case they shouldn't use this feature.
- larodi 4y agoSorry but no map is true and all maps are wrong. What you are is an approximation of the underlying data, and if you want to find truth in it, you should use R or SpatialSQL to do inference.
- moffkalast 4y agoI'd rather look at something sharp that's potentially likely to be there than look at a blurry mess and have no idea what it could even be at all. A lot of things on maps repeat themselves a lot. Most roof HVACs look the same, road lines, cars, trees, etc. How likely is it that a car shaped blob that's the same as the other 10 million car shaped blobs in the training data isn't a car? Not very.
- mcdonje 4y agoSo now when they say "Enhance!" in crime shows, it's more plausible. Depending on the application, this is a good model. It makes the maps look sharper. There are photo editing plugins that use similar technology. I would hope courts of law would not admit any image edited in such a manner as evidence, though. The model is making it up.
- flangola7 4y agoEvery smartphone camera applies ML post processing to the image, and have for multiple years now. The newest phones even have purpose-built tensor chips. It's why we can have crisp 100x zoom, night images in near total darkness, and daylight photos that look like they were taken with a much more expensive DSLR. Phone manufacturers haven't been making revolutionary advances in image sensors or the centuries long science of camera optics — they've learned how to take a few noisy signals and extrapolate them into plausible high fidelity data.
- mcdonje 4y agoNot exactly the same because the phone camera can take several sensor scans, bracketed or otherwise, to create one image that the user sees. Not surprised ML is involved, but it's more akin to a photographer taking several images and combining them in lightroom or photoshop than it is to this situation where the model only has one low-grade image to work with.
- RicoElectrico 4y agoBing uses similar treatment on their satellite imagery and OSM mappers (who are permitted to use it) have been complaining it looks more like painting than actual imagery. Satellite images have 0.4 m resolution or better these days, what matters most is good color quality (e.g. hyperspherical pansharpening), sensor dynamic range and good lighting conditions (sun elevation in particular).
- razor_router 4y agoWhat kind of technology does Bing use to treat their satellite imagery?
- shiftpgdn 4y ago30/50cm per pixel are the maximum allowable by law in the US for "civilians". I'm sure we could do even better if allowed.
- TOMDM 4y agoI'm having trouble finding citations on the specific law, but this website states that the US has capped satellite imagery for civilian use at 0.31 m https://pubs.acs.org/doi/10.1021/acs.est.0c02365 https://pubs.acs.org/doi/10.1021/acs.est.0c02365
- m2fkxy 4y agoSome companies are working on 10 centimeters.
- astrange 4y ago"Satellite" images on maps are often not from satellites but rather planes or drones. I guess with this technique you could do ML image synthesis guided by SAR satellites - that way you could sort-of look through clouds from space, as long as you don't mind the image being largely fictional.
- aix1 4y agoI'm probably just being a bit slow, but what's the novelty here? It feels like these models have existed for quite some time now. Is it that this is now available in a particular commercial mapping platform?
- elil17 4y agoAs far as I can tell - yes, it's just that it is in ArcGIS now.
- ecommerceguy 4y agoMaybe I don't recall correctly but wasn't upsampling, pan sharpening, edge analysis and color enhancement available 20 years ago in Erdas Imagine?
- ska 4y agovariations of these algorithmic techniques have been around for decades, yes.
- timmg 4y agoI thought "super resolution" used to mean: construct a high res image from several low res images of the same thing. Which means it isn't making up information (I don't think). ML-based "super resolution" is more trying to "guess" what the extra pixel data is based on images of other things. I thought that was (should be) called "upscaling". Are we changing the meaning of this term? Or do I have it wrong? (Or do they have it wrong?)
- xhkkffbf 4y agoThere are a number of different algorithms that people use. You're right about the algorithms that use several images. They may not be making up anything. But there are others that do their best guess and they might be said to be making things up.
- deleted 4y ago[deleted]
- larodi 4y agoThe QGiS article a week ago was upvoted like hell, but I see no comments from QGiS folks how this is done in it? Please someone advice.
- mthoms 4y agoAre you saying QGiS has a similar feature? I'd love to know more.
- colincooke 4y agoThis whole concept is so reckless in realms where the image content actually matters and people keep doing it anyways. You cannot CREATE information. You can infer it in certain situations, but if you infer the information and then analyze it you are setting yourself up to make mistakes by overextrapolating a bias/trend in your data to images where you have no idea if that inference is valid. This was a big thing in the medical imaging community (where I did my stint as a CV researcher), folks were hallucinating microscope images and CT scans with no information theory justification as to why it worked. Super resolution IS possible, but it must be done by synthesizing new pieces of information, not by inferring based on what other similar looking objects looked like. A cool technique by my former advisor does this with microscopes [1]. Deep learning has a place here, just not as a "lets create information" step, but as a way to learn how to synthesize additional information about images from more sources (i.e. more similar to how Google does Night Sight [2]). Edit: if you want to see (an attempt) at using deep learning in this field you can checkout one of my papers [3]. [1]: https://en.wikipedia.org/wiki/Fourier_ptychography https://en.wikipedia.org/wiki/Fourier_ptychography [2]: http://graphics.stanford.edu/papers/night-sight-sigasia19/night-sight-sigasia19.pdf http://graphics.stanford.edu/papers/night-sight-sigasia19/ni... [3]: https://openaccess.thecvf.com/content/ICCV2021/html/Cooke_Physics-Enhanced_Machine_Learning_for_Virtual_Fluorescence_Microscopy_ICCV_2021_paper.html https://openaccess.thecvf.com/content/ICCV2021/html/Cooke_Ph...
- m2fkxy 4y agoAgree, you can even see on their input/predicted/target examples that the created/invented data is off enough from ground truth to be in some cases unsuitable for photo interpretation.
- sbaiddn 4y agoIn the Rittenhouse trial this issue came up, namely the faithfulness of digital images and how computers manipulated images.
- psychphysic 4y agoThis is all very sensible criticism but a bit generic. Sometimes detail accuracy doesn't matter but the presence does. Just about every image you ever view has had some manipulation applied. Sometimes that results in a "better" image. Consider all astronomical images for human consumption, even smartphones adapt now to skin tone. I'm playing hogwarts legacy, a recent AAA game which is very demanding, and where aesthetics are very important on a mediocre PC precisely because FSR from AMD (and if I had an Nvidea GPU DLSS and DLAA).
- bkmeneguello 4y agoI would love to see that applied to MSFS2020
- deleted 4y ago[deleted]
- jongala 4y ago> Super Resolution is used in many fields like surveillance, medical industry, forensic and satellite imaging applications. Computer, generate a list of applications where technology like this should absolutely not be used. This seems terrifying — wouldn't this, for example, synthesize identifying details about motor vehicles not present in the sample, drawn from training data? About people? Etc etc This sounds awesome for making a consumer mapping product feel higher quality. Hell, it could even serve an anonymizing function that is pro-privacy ("a roof" not "your roof"). But it feels incredibly reckless to direct this toward those listed industries. People in surveillance and forensics etc. should be confronted with the limits of the quality of the data they are using, we should not try to synthesize extra confidence in their analysis by making the images seem higher quality than they are.