3 ms·

Smartphone cameras for the past ~8 years have been using multiple frames to improve noise performance, resolution and dynamic range. Sometimes as many as 128

Smartphone cameras for the past ~8 years have been using multiple frames to improve noise performance, resolution and dynamic range. Sometimes as many as 128 frames are stacked, over many seconds, especially in low light. Gyro and optical flow data is also used to help align frames, and noise models to decide how to blend them (and avoid ghosting when things are changing in the frame). How exactly does one fit 128 frames of lossless data from a 12 megapixel sensor into an 18 megabyte file? Or is ProRAW not as RAW or as lossless as the name would imply?

echoangle 2y ago
Maybe it’s just one frame per file? Also, you could probably achieve some pretty good compression with multiple frames because the frames are probably very correlated, right?
- londons_explore 2y ago
  So if it's one Raw sensor frame, it will be noisy, and you won't get any of the computational photography benefits. It will end up looking like a photo from a 10 year old phone. You will get a little compression storing multiple frames, but not much, because the photon shot noise is independent for each frame - and in lossless images, the noise (which does not compress according to information theory) often has more entropy than the ideal image (which compresses well). Or you could store one post-processed frame, after you have combined 128 frames and done the computational photography stuff. But at that point, it isn't exactly raw data anymore.
  - FollowingTheDao 2y ago
    "So if it's one Raw sensor frame, it will be noisy," That is not why noise is created in digital photographs. And Noise or Grain in a photograph is not always a negative. "Noise in photography is the arbitrary alteration of brightness and color in an image. The onset of this random variation generates what is called “noise”or “grain”, which is basically formed by irregular pixels misrepresenting the luminance and tonality of the photograph. These pixels are visible to the eye due to their large size." A signal RAW image does not have to have noise. Noise can be created by incorrect exposure and low light. >Or you could store one post-processed frame, after you have combined 128 frames and done the computational photography stuff. But at that point, it isn't exactly raw data anymore. Yes, and Apple should not be calling any images it captures from an iPhone "RAW".
    - viraptor 2y ago
      > A signal RAW image does not have to have noise. Noise can be created by incorrect exposure and low light. You always have noise. It's more noticeable in low light, but it never goes away. It's due to physics - there's real world noise in photons hitting the sensors and read noise in the A/D converters behind them. That's long before the image is captured into any format - RAW doesn't help you here.
    - spider-mario 2y ago
      There is always noise. In fact, in absolute terms, there is more of it as the amount of light increases. It just increases more slowly than the signal, so the signal-to-noise ratio increases.
  - CorrectHorseBat 2y ago
    Are there any cameras out there that output real raw data from the senors? There's quite a bit of analog and digital post processing and I doubt you can bypass all of it
    - out_of_protocol 2y ago
      Full-sized mirror/mirrorless cameras do that. Sensor is HUGE, optics is very fine and huge as well. Due to how universe works, you can't get similar results with tiny sensor and optics
    - londons_explore 2y ago
      > Are there any cameras out there that output real raw data from the senors? I suspect not, because the data output rate from a high end sensor can easily be 32 Gbps, and I don't think any phones have fast enough flash storage to write data at 32Gbps. You have to process the data in mostly-realtime or lose the data. Storing it isn't an option.
      - out_of_protocol 2y ago
        > Storing it isn't an option. Full sized cameras do use a buffer to deal with burst activity, take series if shots - process - save to slower storage. Also, modern cameras start switching to CFexpress/SDExpress (which are NVMe) > Version 8.0 was announced on 19 May 2020, with support for two PCIe lanes with an additional row of contacts and PCIe 4.0 transfer rates, for a maximum bandwidth of 3,938 MB/s
  - viraptor 2y ago
    > noise (which does not compress according to information theory) That's true in theory/isolation, but in practice we know the noise has much lower maximum amplitude than the original image. I.e. if you take a difference between two frames, almost all of it will require less bits per pixel than the signal. (And that allows compression in practice)
kalleboo 2y ago
ProRAW is definitely not RAW, the amount of white balance correction you can do is far more limited. It think it's closer to capturing the different HDR layers to allow the dynamic range to be tweaked more in post.
- FollowingTheDao 2y ago
  Ex-Semi-Professional Photographer here...Yes, I think Apple is very misleading, if not being fraudulent, saying you can shoot in "RAW" format. "RAW files contain uncompressed and unprocessed image data, allowing photographers to capture practically every detail they see in their viewfinder. The RAW file format stores the largest amount of detail out of any raster file type, which photographers can then edit, compress, and convert into other formats. Learn more about the benefits, drawbacks, and best ways to work with a RAW image." If Apple is giving you JPEG-XL they are compressing the file, so it is not RAW.JPEG-XL is lossless compression.
  - chx 2y ago
    Lossless compression shouldn't matter. That's why it's called lossless.
    - FollowingTheDao 2y ago
      Lossless compression is still compressed. RAW photos are by definition not compressed. I do not care if you take out redundant bits, you are still taking them out. "Lossless compression uses an algorithm to shrink the image without losing any IMPORTANT data." and the algorithm decides what is important.
      - xrisk 2y ago
        Lossless formats can compressed: take FLAC as an example.
      - grujicd 2y ago
        If it's lossless then there's zero loss and can be returned to 100% original file. However if algorithm decides what's important than it's by definition not lossless. Wouldn't be surprised to get something like that from Apple, they love to twist words into something they're not for marketing reasons. Like when they say that non-pro Iphone 15 has 2x telephoto while it does not have telephoto lens at all, it's just a digital zoom which all other phones have.
grujicd 2y ago
Based on article [1], ProRAW is a half way between plain RAW output from sensor and baked in jpeg. It uses computed multiple-frame result, but stores more data, allowing you to recover details from shadows and do other things you typically use RAW for. Debayering is performed (nobody cares about that anyway), multiple frames are combined but higher bit-depth is preserved (12-bit instead of 8-bit), it also contains separate "tone map" which describes HDR gains (for each pixel?). Disclaimer, zero experience with ProRAW, just glanced at that article, but used RAW on DSLR in the past. [1] https://medium.com/halide/understanding-proraw-4eed556d4c54 https://medium.com/halide/understanding-proraw-4eed556d4c54
amelius 2y ago
Well, I suppose the less noise is in your image, the easier it will be to compress.

echoangle 2y ago

Maybe it’s just one frame per file? Also, you could probably achieve some pretty good compression with multiple frames because the frames are probably very correlated, right?

londons_explore 2y ago
So if it's one Raw sensor frame, it will be noisy, and you won't get any of the computational photography benefits. It will end up looking like a photo from a 10 year old phone. You will get a little compression storing multiple frames, but not much, because the photon shot noise is independent for each frame - and in lossless images, the noise (which does not compress according to information theory) often has more entropy than the ideal image (which compresses well). Or you could store one post-processed frame, after you have combined 128 frames and done the computational photography stuff. But at that point, it isn't exactly raw data anymore.
- FollowingTheDao 2y ago
  "So if it's one Raw sensor frame, it will be noisy," That is not why noise is created in digital photographs. And Noise or Grain in a photograph is not always a negative. "Noise in photography is the arbitrary alteration of brightness and color in an image. The onset of this random variation generates what is called “noise”or “grain”, which is basically formed by irregular pixels misrepresenting the luminance and tonality of the photograph. These pixels are visible to the eye due to their large size." A signal RAW image does not have to have noise. Noise can be created by incorrect exposure and low light. >Or you could store one post-processed frame, after you have combined 128 frames and done the computational photography stuff. But at that point, it isn't exactly raw data anymore. Yes, and Apple should not be calling any images it captures from an iPhone "RAW".
  - viraptor 2y ago
    > A signal RAW image does not have to have noise. Noise can be created by incorrect exposure and low light. You always have noise. It's more noticeable in low light, but it never goes away. It's due to physics - there's real world noise in photons hitting the sensors and read noise in the A/D converters behind them. That's long before the image is captured into any format - RAW doesn't help you here.
  - spider-mario 2y ago
    There is always noise. In fact, in absolute terms, there is more of it as the amount of light increases. It just increases more slowly than the signal, so the signal-to-noise ratio increases.
- CorrectHorseBat 2y ago
  Are there any cameras out there that output real raw data from the senors? There's quite a bit of analog and digital post processing and I doubt you can bypass all of it
  - out_of_protocol 2y ago
    Full-sized mirror/mirrorless cameras do that. Sensor is HUGE, optics is very fine and huge as well. Due to how universe works, you can't get similar results with tiny sensor and optics
  - londons_explore 2y ago
    > Are there any cameras out there that output real raw data from the senors? I suspect not, because the data output rate from a high end sensor can easily be 32 Gbps, and I don't think any phones have fast enough flash storage to write data at 32Gbps. You have to process the data in mostly-realtime or lose the data. Storing it isn't an option.
    - out_of_protocol 2y ago
      > Storing it isn't an option. Full sized cameras do use a buffer to deal with burst activity, take series if shots - process - save to slower storage. Also, modern cameras start switching to CFexpress/SDExpress (which are NVMe) > Version 8.0 was announced on 19 May 2020, with support for two PCIe lanes with an additional row of contacts and PCIe 4.0 transfer rates, for a maximum bandwidth of 3,938 MB/s
- viraptor 2y ago
  > noise (which does not compress according to information theory) That's true in theory/isolation, but in practice we know the noise has much lower maximum amplitude than the original image. I.e. if you take a difference between two frames, almost all of it will require less bits per pixel than the signal. (And that allows compression in practice)

kalleboo 2y ago

ProRAW is definitely not RAW, the amount of white balance correction you can do is far more limited. It think it's closer to capturing the different HDR layers to allow the dynamic range to be tweaked more in post.

FollowingTheDao 2y ago
Ex-Semi-Professional Photographer here...Yes, I think Apple is very misleading, if not being fraudulent, saying you can shoot in "RAW" format. "RAW files contain uncompressed and unprocessed image data, allowing photographers to capture practically every detail they see in their viewfinder. The RAW file format stores the largest amount of detail out of any raster file type, which photographers can then edit, compress, and convert into other formats. Learn more about the benefits, drawbacks, and best ways to work with a RAW image." If Apple is giving you JPEG-XL they are compressing the file, so it is not RAW.JPEG-XL is lossless compression.
- chx 2y ago
  Lossless compression shouldn't matter. That's why it's called lossless.
  - FollowingTheDao 2y ago
    Lossless compression is still compressed. RAW photos are by definition not compressed. I do not care if you take out redundant bits, you are still taking them out. "Lossless compression uses an algorithm to shrink the image without losing any IMPORTANT data." and the algorithm decides what is important.
    - xrisk 2y ago
      Lossless formats can compressed: take FLAC as an example.
    - grujicd 2y ago
      If it's lossless then there's zero loss and can be returned to 100% original file. However if algorithm decides what's important than it's by definition not lossless. Wouldn't be surprised to get something like that from Apple, they love to twist words into something they're not for marketing reasons. Like when they say that non-pro Iphone 15 has 2x telephoto while it does not have telephoto lens at all, it's just a digital zoom which all other phones have.

grujicd 2y ago

Based on article [1], ProRAW is a half way between plain RAW output from sensor and baked in jpeg. It uses computed multiple-frame result, but stores more data, allowing you to recover details from shadows and do other things you typically use RAW for. Debayering is performed (nobody cares about that anyway), multiple frames are combined but higher bit-depth is preserved (12-bit instead of 8-bit), it also contains separate "tone map" which describes HDR gains (for each pixel?). Disclaimer, zero experience with ProRAW, just glanced at that article, but used RAW on DSLR in the past. [1] https://medium.com/halide/understanding-proraw-4eed556d4c54 https://medium.com/halide/understanding-proraw-4eed556d4c54

amelius 2y ago

Well, I suppose the less noise is in your image, the easier it will be to compress.