8 ms·

One of those annoying things is that the name "super resolution" stuck, here. Originally super-resolution was a hardware technique, and not "guessing". If you

• anon_tor_12345 6y ago
  Citation? I've read over 50 papers in MISR and SISR (I wrote a lit review) and I have never seen mention of actual hardware that would shift the imaging system; indeed such a system would have been my dissertation topic if I hadn't switched areas.
  • CarVac 6y ago
    Certain Hasselblad cameras have had this for a long time, and Pentax, Olympus, Panasonic, and Sony have it on various models, using the sensor shift image stabilization to implement it.
  • galago 6y ago
    I think what he's talking about is what Sony calls Pixel Shift Multi Shooting. Other camera manufacturers do it too. https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7rm3/en/index.html https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r...
    • anon_tor_12345 6y ago
      Wow that is in fact exactly it. I'm actually quite impressed they're able to shift the sensor array by exactly one pixel (or even close to) since that's on the order of microns.
      • brudgers 6y ago
        Pentax did it with the K70 back in 2016.
        • yfkar 6y ago
          They also did it earlier in 2015 with Pentax K-3 II. Also Olympus OM-D E-M5 Mark II had a similar feature in 2015.
      • Stratoscope 6y ago
        In fact the Olympus implementation of this feature moves the sensor by half a photosite diagonally. They do it by re-purposing the existing in-body stabilization mechanism to move the sensor around. User 'twic' posted a link to a very interesting article that describes this and also explains the difference between photosites and pixels: https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-om-d-e-m1-mark-ii/ https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
      • RobLach 6y ago
        Some projector manufacturers are shifting a 4K DMD around 4 times to increase the refresh rate of a 1080P image 4x.
        • deleted 6y ago
          [deleted]
        • dogma1138 6y ago
          All of them are shifting either by 4 or by 2 pixels you don’t have native 4K DMDs yet afaik.
  • bobtail3 6y ago
    Probably not exactly what he is talking about, but this also sounds similar to dithering. Where with repeated measurements and random noise you can statistically estimate the value of a signal below the quantization level.
    • Thrymr 6y ago
      Indeed, it includes things like the Drizzle algorithm that has been used by Hubble space telescope astronomers for a while: https://www.stsci.edu/ftp/science/hdf/combination/drizzle.html https://www.stsci.edu/ftp/science/hdf/combination/drizzle.ht...
  • wongarsu 6y ago
    Synthetic aperture radar [1][2] uses this principle. In that case the imaging system is fixed on a moving aircraft or satellite. I think in general satellite imaging is a good place to look for such implementations, since they have a naturally and predictably moving imaging system. 1: https://en.wikipedia.org/wiki/Synthetic-aperture_radar https://en.wikipedia.org/wiki/Synthetic-aperture_radar 2: https://www.youtube.com/watch?v=u2bUKEi9It4 https://www.youtube.com/watch?v=u2bUKEi9It4
    • anon_tor_12345 6y ago
      SAR is similar but not the same since there you're super-resolving in time rather than space. Also in that instance it's just conventional MISR since you're not driving the imaging system (more information is being passively collected as the targets passes).
      • ska 6y ago
        SAR is similar to the video technique mentioned, agree it's not quite the same but if underlying assumptions hold still more estimate than "guess".
      • oivey 6y ago
        I think it’s more useful to think in terms of how well sampled the observations are relative to the size of the output space. SISR is very undersampled, and MISR is oversampled. SAR reconstruction techniques can fall in either bucket.
    • mNovak 6y ago
      A major difference here is that SAR uses phase information, whereas to my knowledge optical techniques are not doing that.
  • twic 6y ago
    I just bought a camera that does it, so it definitely exists. https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-om-d-e-m1-mark-ii/ https://chriseyrewalker.com/the-hi-res-mode-of-the-olympus-o...
    • Stratoscope 6y ago
      That's a really informative article; thanks for posting it. I also have an Olympus E-M1 MkII, but I haven't tried the high resolution mode yet. You just gave me a TODO item!
      • twic 6y ago
        I haven't tried it yet either. It really needs a tripod, and i don't have one.
  • shock-value 6y ago
    Most modern mirrorless cameras have such hardware (usually denoted "in-body image stabilization"). The "repurposing" of it to acquire multiple captures at various slight offsets and combine them intelligently is maybe a bit more recent, but still pretty common at this point. https://en.wikipedia.org/wiki/Image_stabilization#Sensor-shift https://en.wikipedia.org/wiki/Image_stabilization#Sensor-shi... https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7rm3/en/index.html https://support.d-imaging.sony.co.jp/support/ilc/psms/ilce7r... https://www.nikonimgsupport.com/na/NSG_article?articleNo=000042799&configured=1&lang=en_SG https://www.nikonimgsupport.com/na/NSG_article?articleNo=000... https://www.canon.co.uk/pro/stories/8-stops-image-stabilization/ https://www.canon.co.uk/pro/stories/8-stops-image-stabilizat...
    • avereveard 6y ago
      I think the technique was used a lot prior in astronomy, where I encountered it the first time. There is called stacking, and the subpixel offset comes naturally from path distortion in the atmosphere itself. I remember using registax when the first batch of consumer telezoom point and click camera came out and got a 30x long exposure of the moon, with, well, mediocre results.
  • deleted 6y ago
    [deleted]
  • deleted 6y ago
    [deleted]
  • dev_tty01 6y ago
    The iPhone 12 Pro physically shifts the sensor for image stabilization, so precise positioning is possible.
  • vlovich123 6y ago
    You can also do it in software by just recording IMU with high precision time stamps (a lot of camera sensors have a gpio they can interrupt on even every scan line) and post-processing. There’s cool techniques where they can remove various issues in rolling shutter through this technique to get global-shutter like quality and removing camera-movement induced motion blur. I haven’t heard of it applied to super res but I don’t see why not. I think Google uses similar techniques to implement their software HDR solution which takes 3 back to back snapshots at different exposure levels and merges them.
    • bayindirh 6y ago
      Newer Sony mirrorless cameras embed gyroscope data into videos for further stabilization of the image when IBIS is not enough Theoretically, with 30FPS cameras like Sony A1 and said gyroscope data, you can create super resolution images. IIRC Olympus' some handheld super resolution modes use both shake and sensor shift to increase resolution.
  • sorenjan 6y ago
    Google uses it in Google camera. They're not shifting the sensor themselves, but they take advantage of the camera shake users introduce by taking handheld photos. https://ai.googleblog.com/2018/10/see-better-and-further-with-super-res.html https://ai.googleblog.com/2018/10/see-better-and-further-wit...
    • anon_tor_12345 6y ago
      Yes I'm very familiar with this but this is just MISR i.e. purely a software solution (Peyman Milanfar is one of the original researchers associated with MISR). Fortunately elsewhere in this thread hardware implementations have been demonstrated.
      • sorenjan 6y ago
        There are hardware implementations referenced in the blog post, and in the linked published paper. > In the early 2000s, Farsiu et al. [2006] and Gotoh and Okutomi [2004]formulated superresolution from arbitrary motion as an optimization problem that would be infeasible for interactive rates. Ben-Ezraet al. [2005] created a jitter camera prototype to do super-resolution using controlled subpixel detector shifts. This and other works inspired some commercial cameras (e.g.,Sony A6000,Pentax FF K1,Olympus OM-D E-M1orPanasonic Lumix DC-G9) to adopt multi-frame techniques, using controlled pixel shifting of the physical sensor. However, these approaches require the use of a tripod or a static scene. https://arxiv.org/pdf/1905.03277.pdf https://arxiv.org/pdf/1905.03277.pdf
        • anon_tor_12345 6y ago
          damn the Ben-Ezra paper is exactly what i imagined being the scope of work for my planned dissertation. guess it's a good thing i didn't pursue it lol.
  • gnopgnip 6y ago
    Pentax DLSR have this feature too. The same motors that are used to control the sensor and prevent camera shake are used to offset the sensor slightly while multiple images are taken
• Lio 6y ago
  I seem to remember this being done with the piezoelectric vibrator (stop that sniggering at the back) used to shake the dust off sensors.[1] I think Haselblad had that as well as Olympus but I can't quite what the systems were called. https://www.engadget.com/2018-01-18-hasselblad-h6d-400c-medium-format-400-megapixels.html?guccounter=1&guce_referrer=aHR0cHM6Ly9kdWNrZHVja2dvLmNvbS8&guce_referrer_sig=AQAAAD6RbsLVoGFCSp3Yd2O_p8wXUBRFKNpVuQxZNbPea2UQSAp8tyqpnJ5d8W8HOQ5YBlyfsIk__GZ59SDaYYHa_knuxobuzHgiB_ssh3EQg7ERhRcohiNx2aaPcwqbDcGSi-WzzVIFCXq0oHQP-9ThZB0ybdw3sn7CHfOThhn2xxdn https://www.engadget.com/2018-01-18-hasselblad-h6d-400c-medi...
• light_hue_1 6y ago
  This is the actual original meaning of the word "super resolution". The idea is that you use a prior about a specific image, domain, type of image, etc. to enhance the resolution. Super-resolution was already the name for this field in this 1987 review! https://kmh-lanl.hansonhub.com/publications/imrecov.pdf https://kmh-lanl.hansonhub.com/publications/imrecov.pdf But 1987 is pretty recent as far as this field is considered. A. W. Lohmann and D. P. Paris talked about super-resolution this way in 1964! https://www.osapublishing.org/ao/abstract.cfm?uri=ao-3-9-1037 https://www.osapublishing.org/ao/abstract.cfm?uri=ao-3-9-103... So no. It's not annoying that super-resolution is the name of this field. The name predates the entire field of image processing, it predates the invention of the digital camera! And it is the original use of the title "super resolution", people that do this in hardware adopted the name later.
  • ska 6y ago
    I think you misread me (so clearly I articulated poorly); the thing that is annoying is that super resolution in the sense you refer to is categorically a different thing that the common usage of super resolution in (particularly single image) digital processing. The usage existed, as you and I noted, before it came into digital image processioning. Because of this, conversations around it are often confused, and it would be clearer if there were a different terminology for the latter.
    • meowface 6y ago
      What would be a better term for this type, then? "Inferred/approximated super resolution", or something?
• Izkata 6y ago
  > Originally super-resolution was a hardware technique, and not "guessing". If you can [edit: this was poorly worded "control an imager positioning"] control imaging with finer resolution than the sensor has, you can take multiple images and reconstruct a higher resolution image in a principled way for say 2x resolution gain (cf super-resolution microscopy), also some telescope systems. Isn't this also how our eyes work?
  • Valgrim 6y ago
    Not really. Our visual cortex does something closer to the new fancy photoshop algorithm (filling in imaginary detail from a mental model in memory). Super-resolution techniques such as the one described above is more like a mathemathical function based on a statistical values, so it extracts real information hidden in the difference between each image.
    • BearOso 6y ago
      It does both. Your brain interpolates the image, but the eye also uses interferometry to generate extra detail by moving the eye back and forth really quickly (microtremors). The result is that we glean more true information than can be determined by just measuring rod or cone frequency.
• SiebenHeaven 6y ago
  A ASUS phone (zenfone 2) had this hardware superresolution feature you are talking about
• vvanders 6y ago
  You can generalize this to signals by introducing noise to get higher resolutions from ADCs[1]. Human vision does this too[3], there was a really interesting theory that the reason 48 FPS in the Hobbit looked so bad was due to it falling right near the line Nyquist and the brain being unable to distinguish from a sequence of pictures(24FPS) or continuous frames(60FPS)[2]. [1] http://imajeenyus.com/electronics/20120908_improving_adc_resolution/index.shtml http://imajeenyus.com/electronics/20120908_improving_adc_res... [2] https://accidentalscientist.com/2014/12/why-movies-look-weird-at-48fps-and-games-are-better-at-60fps-and-the-uncanny-valley.html https://accidentalscientist.com/2014/12/why-movies-look-weir... [3] https://en.wikipedia.org/wiki/Ocular_tremor https://en.wikipedia.org/wiki/Ocular_tremor
  • formerly_proven 6y ago
    Doesn't strike me as true, 50 fps video is very common all over the world and doesn't have many issues besides noticeable flicker on CRTs. Note that 24 fps film cinema runs at 72 Hz flicker due to triple-exposing each frame. That's why they could get away with 24 fps: Since they are dealing with stored images, multiple exposures were possible to avoid flickering. Using as low fps as possible was desirable because film costs money, film transports get more finicky as fps are increased and exposure gets tricky (doubling or even tripling the frame rate leaves you with a much shorter exposure time, which you need to compensate either with a larger aperture (which may not be available, or if it is, cost a lot of money), or more sensitive film (which may not have been available, or has much worse quality), or by having more light in the scene -- all of these are either undesirable or expensive). So they chose 24 fps because the motion perception isn't quite in the shitter at that fps, though it's pretty bad (nowadays idolized as "cinematic look"; even more hip and cool if you do it on Youtube and deliver 24p which gets converted to 60 fps by means of 3:2 alternation, resulting in a super janky look known as "That Pro Youtuber Look"). This is the reason why people did not like The Hobbit. Meanwhile video people went on to invent interlacing, because video was fundamentally about a non-stored image. Here the reason why low fps are desirable are again cost, since higher fps needs higher video bandwidth, and that makes everything more expensive and lets you have fewer channels (likely not a concern in the early days). Interlacing lets you have a higher frame rate (a CRT at 25 Hz or even 30 Hz will trigger nausea in everyone) without incurring higher costs, just like multiple exposures in a cinema projector.