5 ms·

I feel daft, but: The vast majority of images are jpegs, which are internally 420 YUV, but they get converted to 32 bit RGB for use in apps. Using native YU

I feel daft, but: The vast majority of images are jpegs, which are internally 420 YUV, but they get converted to 32 bit RGB for use in apps. Using native YUV formats would save half the memory and rendering bandwidth, speed loading, and provide a tiny quality improvement. What does he mean by using native YUV formats? Something (I wave my hand) in the rendering pipeline from the JPEG in memory to pixels on the screen?

vlovich123 5y ago
Yup. You typically first decode the JPEG into a raw image format in GPU memory & then that gets transferred to scanout HW that actually sends it to the display (e.g. via HDMI). The argument feels a bit weak though. At that point why not just use JPEG through your entire pipeline & save even more bandwidth. Of course you have to decompress if you’re modifying the image in any way so it doesn’t help there.
- edflsafoiewq 5y ago
  You can't texel-fetch directly from JPEG data, it has to be decompressed first.
  - vlovich123 5y ago
    Yes of course. GPUs already support compressed textures though that get decompressed on the fly. Of course JPEG has a lot of baggage that actually makes it a poor format for this but perhaps newer ones might be useful. What you lose in “optimality” (compression schemes designed for GPUs are better for games), you win in “generality” (ability to use this outside games to, for example, improve web browsing battery life).
    - mrec 5y ago
      There's a very hard distinction between GPU texture compression formats and general image compression formats - the former need to support random access lookup in constant time. Anything meeting that criterion is not going to be generally competitive; it's like the difference between UTF-32 strings and UTF-8 strings.
david-gpu 5y ago
You could do the YUV to RGB conversion operation in your pixel shaders in the GPU. That way you save a bit of bandwidth compared to uncompressed RGB. It's been done. There are even GPUs that support this operation natively, so there's no additional overhead.
mindfulplay 5y ago
I would imagine a shader that converts YUV to ARGB at the time of rendering as opposed to storing it all the way along the pipeline as 32 bit integers. It's a bit tricky because rendering pipelines composite the final image through many layers of offscreen compositing before the pixel hits the screen. The core issue is that the offscreen composited layers would still be 32bit textures which is a bigger issue. I would imagine a Skia-based draw list to encode this through the pipeline which could help preserve this perhaps.
PragmaticPulp 5y ago
> What does he mean by using native YUV formats? Your display uses 3 bytes per pixel. 8 bits for each of the R, G, and B channels. This is known as RGB888. (Ignoring the A or alpha transparency channel for now). YUV420 uses chroma subsampling, which means the color information is stored at a lower resolution than the brightness information. Groups of 4 pixels will have the same color, but each pixel can have a different brightness. Our eyes are more sensitive to brightness changes than color changes, so this is usually unnoticeable. This is very advantageous for compression because YUV420 requires 6 bytes per 4 pixels, or 1.5 bytes per pixel, because groups of pixels share a single color value. That's half as many bytes as RGB888. When you decompress a JPEG, you first get a YUV420 output. Converting from YUV420 to RGB888 doesn't add any information, but it doubles the number of bits used to represent the image because it stores the color value for every individual pixel instead of groups of pixels. This is easier to manipulate in software, but it takes twice as much memory to store and twice as much bandwidth to move around relative to YUV420. The idea is that if your application can work with YUV420 through the render pipeline and then let a GPU shader do the final conversion to RGB888 within the GPU, you cut your memory and bandwidth requirements in half at the expense of additional code complexity. Wikipedia is a good source of diagrams and details that explain this further: https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2%80%B2V12_or_YV12)_to_RGB888_conversion https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2...
- Jare 5y ago
  In Commandos 2 we stored the huge map images in YCbCr 420 in memory, and converted to RGB during rendering via lookup tables rather than shaders (this was back in 1999, all CPU!). We compressed the range of each component below 8 bits, running a histogram so our bits represented values inside the range actually present in the original image. We got it down to 6 bits per pixel or something close to that. Converting via lookup tables, one for each component, made it very cheap to perform palette-style tricks like tonemapping and smooth color shifts from day to night. [Edit: it may have actually been CIE Lab, not YUV/YCbCr, because the a&b tended to have narrower ranges. It's been too long!]
  - rgoulter 5y ago

vlovich123 5y ago

Yup. You typically first decode the JPEG into a raw image format in GPU memory & then that gets transferred to scanout HW that actually sends it to the display (e.g. via HDMI). The argument feels a bit weak though. At that point why not just use JPEG through your entire pipeline & save even more bandwidth. Of course you have to decompress if you’re modifying the image in any way so it doesn’t help there.

edflsafoiewq 5y ago
You can't texel-fetch directly from JPEG data, it has to be decompressed first.
- vlovich123 5y ago
  Yes of course. GPUs already support compressed textures though that get decompressed on the fly. Of course JPEG has a lot of baggage that actually makes it a poor format for this but perhaps newer ones might be useful. What you lose in “optimality” (compression schemes designed for GPUs are better for games), you win in “generality” (ability to use this outside games to, for example, improve web browsing battery life).
  - mrec 5y ago
    There's a very hard distinction between GPU texture compression formats and general image compression formats - the former need to support random access lookup in constant time. Anything meeting that criterion is not going to be generally competitive; it's like the difference between UTF-32 strings and UTF-8 strings.

david-gpu 5y ago

You could do the YUV to RGB conversion operation in your pixel shaders in the GPU. That way you save a bit of bandwidth compared to uncompressed RGB. It's been done. There are even GPUs that support this operation natively, so there's no additional overhead.

mindfulplay 5y ago

I would imagine a shader that converts YUV to ARGB at the time of rendering as opposed to storing it all the way along the pipeline as 32 bit integers. It's a bit tricky because rendering pipelines composite the final image through many layers of offscreen compositing before the pixel hits the screen. The core issue is that the offscreen composited layers would still be 32bit textures which is a bigger issue. I would imagine a Skia-based draw list to encode this through the pipeline which could help preserve this perhaps.

PragmaticPulp 5y ago

> What does he mean by using native YUV formats? Your display uses 3 bytes per pixel. 8 bits for each of the R, G, and B channels. This is known as RGB888. (Ignoring the A or alpha transparency channel for now). YUV420 uses chroma subsampling, which means the color information is stored at a lower resolution than the brightness information. Groups of 4 pixels will have the same color, but each pixel can have a different brightness. Our eyes are more sensitive to brightness changes than color changes, so this is usually unnoticeable. This is very advantageous for compression because YUV420 requires 6 bytes per 4 pixels, or 1.5 bytes per pixel, because groups of pixels share a single color value. That's half as many bytes as RGB888. When you decompress a JPEG, you first get a YUV420 output. Converting from YUV420 to RGB888 doesn't add any information, but it doubles the number of bits used to represent the image because it stores the color value for every individual pixel instead of groups of pixels. This is easier to manipulate in software, but it takes twice as much memory to store and twice as much bandwidth to move around relative to YUV420. The idea is that if your application can work with YUV420 through the render pipeline and then let a GPU shader do the final conversion to RGB888 within the GPU, you cut your memory and bandwidth requirements in half at the expense of additional code complexity. Wikipedia is a good source of diagrams and details that explain this further: https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2%80%B2V12_or_YV12)_to_RGB888_conversion https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2...

Jare 5y ago
In Commandos 2 we stored the huge map images in YCbCr 420 in memory, and converted to RGB during rendering via lookup tables rather than shaders (this was back in 1999, all CPU!). We compressed the range of each component below 8 bits, running a histogram so our bits represented values inside the range actually present in the original image. We got it down to 6 bits per pixel or something close to that. Converting via lookup tables, one for each component, made it very cheap to perform palette-style tricks like tonemapping and smooth color shifts from day to night. [Edit: it may have actually been CIE Lab, not YUV/YCbCr, because the a&b tended to have narrower ranges. It's been too long!]
- rgoulter 5y ago