11 ms·

Vulkan video extensions for accelerated H.264 and H.265 encode

245 points by mshachkov 3y ago

shmerl 3y ago
> In addition, given the high industry demand for AV1 codec support, an AV1 decode extension release is imminent, with an AV1 encode extension development also underway. That's good. By the way, mpv already supports Vulkan video decoding: https://github.com/mpv-player/mpv/issues/11739 https://github.com/mpv-player/mpv/issues/11739 Now we need OBS and Firefox supporting Vulkan video too.
- dist-epoch 3y ago
  On Windows/Mac Firefox uses the system hardware video decoding APIs - DXVA and Video Toolbox.
  - shmerl 3y ago
    On Windows Vulkan video would work. But not on macOS (that mpv thread mentions it). On Linux it should be a better option than VAAPI once drivers will support it. Since Firefox is using ffmpeg anyway, it probably shouldn't be too hard to make it an option.
    - jorvi 3y ago
      Why would it be a better option than VAAPI? Decoding h264/h265 in fixed (CPU) hardware is much more efficient than on the GPU. Don’t get me wrong, its still miles and miles better than software decode.
      - Osiris 3y ago
        Do CPUs have fixed video decoders? I thought those were only included in GPUs, including iGPUs.
        tadfisher 3y ago
        Yes, every Intel consumer CPU has included a hardware video block since Haswell, called Quick Sync Video.
        vetinari 3y ago
        That block is a part of the included iGPU though. Intel CPUs without iGPU, like the F-series, do not support QSV.
      - kllrnohj 3y ago
        VAAPI is also using the GPU. The fixed-function video decoders are on the GPUs, not the CPU. That just happens to be using the iGPU on Intel so you don't need a discreet GPU necessarily, but this is also missing if you have a CPU-only sku like the -f suffix parts The only difference is the vulkan spec is _in theory_ cross-platform. However, since MacOS doesn't support Vulkan and MoltenVK doesn't implement this extension (and probably won't), and this extension also isn't implemented on Android, then your scope of actual availability is quite pathetic. So it probably doesn't make sense to move off of VAAPI regardless unless you really want Linux & Windows to share a codepath, even though you have something OS-specific anyways for MacOS/iOS & Android.
        stefan_ 3y ago
        It's unhelpful to bundle this into the "GPU" at large and is only true for desktop systems anyway. On mobile, video decoding, GPU and display pipeline are typically different hardware blocks often made by different companies.
        brigade 3y ago
        > MoltenVK doesn't implement this extension (and probably won't) Since Khronos appears to have decided to require manual DPB management, this cannot be implemented in HW on macOS by MoltenVK.
        jorvi 3y ago
        Huh, I stand corrected. I always figured they stuffed it in its own little bit of silicon. Thanks for clearing it up!
        tadfisher 3y ago
        Intel does do this, they call it Quick Sync Video. There's support for it in vaapi. AV1 is only if you have a Gen12 iGPU or newer, or Arc.
        jorvi 3y ago
        I might just be bad at Google-fu, but my understanding is that -F processors don’t have Quick Sync, which seems to imply that the iGPU is required in some capacity.
      - deleted 3y ago
        [deleted]
      - brigade 3y ago
        Vulkan Video Extensions use the same physical hardware as VAAPI, but with a little bit less of forcing you to do codec-specific nonsense. It should be better in that it's much easier to get a zero-copy Vulkan texture from the decoded frame.
      - theHamsta 3y ago
        The GPU has also the fixed function video encode like integrated CPUs. Both VAAPI and Vulkan Video address the same hardware. Differences are in the API itself (with Vulkan being more explicit) and potentially in which vendors (or OSS driver developers) support which API in what degree.
      - deskamess 3y ago
        Could anyone elaborate on the difference between software decoding and CPU decoding? Software decoding uses the CPU right?
        mattst88 3y ago
        That's right. "Software decoding" means the decoding algorithms run on the CPU. (As opposed to "hardware decoding" which typically means the work is done by some fixed-function video units on e.g. a GPU)
      - shmerl 3y ago
        I think not being tied to VAAPI reduces the chain of development (like you don't need to wait for Intel to add something first since they are the primary developers of VAAPI in general).
    - slimsag 3y ago
      Beware on Windows, out-of-the-box Microsoft will ship e.g. nvidia drivers that have DirectX 12, but not Vulkan. This can lead to confusing user situations ('DirectX 12 games work fine.. but not Vulkan') and probably means DXVA would be a better choice if targeting Windows
      - my123 3y ago
        > nvidia drivers that have DirectX 12, but not Vulkan not a thing. at worst you get a situation where the ICD loader isn't installed
        slimsag 3y ago
        In Windows 10 and above, windows update provides a default graphics driver for each IHV, which comes from the manufacturer, has DirectX support, but doesn't have Vulkan. Often nvidia users will try to debug it, and discover 'the vulkan checkbox is off but I have the latest driver version!' in gpuZ[0][1] - but it happens across all IHVs, AMD, Intel, etc. I was told by someone in the industry the default Microsoft drivers across all IHVs indeed have DirectX, but not Vulkan. To make matters more annoying, Windows Update often _overwrites_ the user's driver which they installed from the manufacturer directly, leading to Vulkan becoming broken overnight[2][3] Easy to find multiple instances of this happening[4][5][6], I became aware of it because of someone asking on /r/vulkan a few weeks ago. [0] https://i.imgur.com/qj3Rn9p.png https://i.imgur.com/qj3Rn9p.png https://rog-forum.asus.com/t5/rog-strix-series/nvidia-driver-without-vulkan-support-solved/td-p/892751 https://rog-forum.asus.com/t5/rog-strix-series/nvidia-driver... [1] https://forums.x-plane.org/index.php?/forums/topic/228597-vulkan-checkbox-greyed-out/ https://forums.x-plane.org/index.php?/forums/topic/228597-vu... [2] https://www.reddit.com/r/Amd/comments/1145ffp/windows_update_may_have_automatically_replaced/ https://www.reddit.com/r/Amd/comments/1145ffp/windows_update... [3] https://answers.microsoft.com/en-us/windows/forum/all/windows-10-update-replaced-amd-graphics-driver/78c071ed-eaa7-4508-8d94-1121669c39cf https://answers.microsoft.com/en-us/windows/forum/all/window... [4] https://i.imgur.com/PXZRxYh.png https://i.imgur.com/PXZRxYh.png [5] https://www.reddit.com/r/pcmasterrace/comments/qhfesc/vulkan_disabled_after_driver_update/ https://www.reddit.com/r/pcmasterrace/comments/qhfesc/vulkan... [6] https://www.reddit.com/r/techsupport/comments/110bytq/vulkan_api_disabled_on_amd_card/ https://www.reddit.com/r/techsupport/comments/110bytq/vulkan...
    - ComputerGuru 3y ago
      But why would Firefox go out of its way to use Vulkan on Windows or macOS when it doesn’t have to because it can just use the native OS functionality and let the OS worry about implementation details?
      - shmerl 3y ago
        I don't really care much about Windows and macOS. But I'd like to see it using Vulkan video on Linux in addition or instead of VAAPI. As pointed out elsewhere, it's not possible on macOS either way. Apple are too stuck up not supporting Vulkan and it won't work with MoltenVK.
- kllrnohj 3y ago
  > Now we need OBS and Firefox supporting Vulkan video too. Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using. It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.
  - jdiff 3y ago
    Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.
    - mattst88 3y ago
      > Firefox still has no support for hardware video decode on Linux. That's not true at all. See: https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-video-decoding-on-linux https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-vi... https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.omglinux.com/firefox-hardware-acceleration-raspberry-pi/ https://www.omglinux.com/firefox-hardware-acceleration-raspb...
      - cesarb 3y ago
        How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?
- luthMan 3y ago
  What about systems where there is now hardware support for AV1? How would Vulkan implement that? Also, maybe someone knows more about this, but I thought that most software like Firefox, MPV and VLC are using dav1d for AV1 decoding as it seems to be the most performant AV1 decoder (in Software) out there at the moment.
  - shmerl 3y ago
    > How would Vulkan implement that? See the post. They are working on AV1 extensions.
deleted 3y ago
[deleted]
dishsoap 3y ago
So to clarify, this still leverages the fixed function hardware video encoders present alongside many GPUs from eg. Intel/NVIDIA/AMD, and would not (for example) be possible to use on just a GPU itself (without said encoding hardware)?
- shmerl 3y ago
  It is for using GPU video related ASICs (same as VAAPI let's say). If you want to use shaders for some video filters, scaling and such - you don't need those extensions. mpv for example can use Vulkan for both at the same time.
- monocasa 3y ago
  Interestingly, quite a bit of video decode isn't as parallelizable as you might think and isn't a great fit for GPUs. For instance the initial Huffman decoding of the stream is essentially an intrinsically sequential process.
  - 0x1ceb00da 3y ago
    Is there a reason we don't have parallelizable video formats?
    - dagmx 3y ago
      Largely because a lot of compression is contextual to the data around it. That makes most decompression a poor fit for parallelism. The primary choices are space savings and power efficiency. Dedicated hardware and serial decoding/encoding often win out as a result.
    - throwup238 3y ago
      Modern video codecs use motion prediction algorithms to encode the information between key frames which depends on the entire frame and the ones immediately preceding it. You can encode the independent groups of pictures (the key frame and all the subsequent predictive frames until the next key frame) in parallel but at least with 4k video you hit memory bandwidth limitations quite quickly.
dlojudice 3y ago
> vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR how to exceed the character limit on your super wide screen with just one function name
- mft_ 3y ago
  Needs more snake case for added width sillyness :)
- deleted 3y ago
  [deleted]
- suprjami 3y ago
  Eskil Steenberg suggests width of code as a measure of quality. Vulkan wins.
- IPTN 3y ago
  Huh, It only takes up 2/3 of my phone screen width in portrait mode and reads fine at full arms length.
- lostmsu 3y ago
  C needs an extension for import without namespaces purely based on names: import vkGetPhysicalDeviceVideoEncode; QualityLevelPropertiesKHR(...); Or even better: import vk*PhysicalDeviceVideoEncode; Get_QualityLevelPropertiesKHR(...);
temp0826 3y ago
This already works decently well in mpv though you need the latest (from git) mesa/libplacebo/ffmpeg to use it.

245 points by mshachkov 3y ago

> In addition, given the high industry demand for AV1 codec support, an AV1 decode extension release is imminent, with an AV1 encode extension development also underway. That's good. By the way, mpv already supports Vulkan video decoding: https://github.com/mpv-player/mpv/issues/11739 https://github.com/mpv-player/mpv/issues/11739 Now we need OBS and Firefox supporting Vulkan video too.

dist-epoch 3y ago
On Windows/Mac Firefox uses the system hardware video decoding APIs - DXVA and Video Toolbox.
- shmerl 3y ago
  On Windows Vulkan video would work. But not on macOS (that mpv thread mentions it). On Linux it should be a better option than VAAPI once drivers will support it. Since Firefox is using ffmpeg anyway, it probably shouldn't be too hard to make it an option.
  - jorvi 3y ago
    Why would it be a better option than VAAPI? Decoding h264/h265 in fixed (CPU) hardware is much more efficient than on the GPU. Don’t get me wrong, its still miles and miles better than software decode.
    - Osiris 3y ago
      Do CPUs have fixed video decoders? I thought those were only included in GPUs, including iGPUs.
      - tadfisher 3y ago
        Yes, every Intel consumer CPU has included a hardware video block since Haswell, called Quick Sync Video.
        vetinari 3y ago
        That block is a part of the included iGPU though. Intel CPUs without iGPU, like the F-series, do not support QSV.
    - kllrnohj 3y ago
      VAAPI is also using the GPU. The fixed-function video decoders are on the GPUs, not the CPU. That just happens to be using the iGPU on Intel so you don't need a discreet GPU necessarily, but this is also missing if you have a CPU-only sku like the -f suffix parts The only difference is the vulkan spec is _in theory_ cross-platform. However, since MacOS doesn't support Vulkan and MoltenVK doesn't implement this extension (and probably won't), and this extension also isn't implemented on Android, then your scope of actual availability is quite pathetic. So it probably doesn't make sense to move off of VAAPI regardless unless you really want Linux & Windows to share a codepath, even though you have something OS-specific anyways for MacOS/iOS & Android.
      - stefan_ 3y ago
        It's unhelpful to bundle this into the "GPU" at large and is only true for desktop systems anyway. On mobile, video decoding, GPU and display pipeline are typically different hardware blocks often made by different companies.
      - brigade 3y ago
        > MoltenVK doesn't implement this extension (and probably won't) Since Khronos appears to have decided to require manual DPB management, this cannot be implemented in HW on macOS by MoltenVK.
      - jorvi 3y ago
        Huh, I stand corrected. I always figured they stuffed it in its own little bit of silicon. Thanks for clearing it up!
        tadfisher 3y ago
        Intel does do this, they call it Quick Sync Video. There's support for it in vaapi. AV1 is only if you have a Gen12 iGPU or newer, or Arc.
        jorvi 3y ago
        I might just be bad at Google-fu, but my understanding is that -F processors don’t have Quick Sync, which seems to imply that the iGPU is required in some capacity.
    - deleted 3y ago
      [deleted]
    - brigade 3y ago
      Vulkan Video Extensions use the same physical hardware as VAAPI, but with a little bit less of forcing you to do codec-specific nonsense. It should be better in that it's much easier to get a zero-copy Vulkan texture from the decoded frame.
    - theHamsta 3y ago
      The GPU has also the fixed function video encode like integrated CPUs. Both VAAPI and Vulkan Video address the same hardware. Differences are in the API itself (with Vulkan being more explicit) and potentially in which vendors (or OSS driver developers) support which API in what degree.
    - deskamess 3y ago
      Could anyone elaborate on the difference between software decoding and CPU decoding? Software decoding uses the CPU right?
      - mattst88 3y ago
        That's right. "Software decoding" means the decoding algorithms run on the CPU. (As opposed to "hardware decoding" which typically means the work is done by some fixed-function video units on e.g. a GPU)
    - shmerl 3y ago
      I think not being tied to VAAPI reduces the chain of development (like you don't need to wait for Intel to add something first since they are the primary developers of VAAPI in general).
  - slimsag 3y ago
    Beware on Windows, out-of-the-box Microsoft will ship e.g. nvidia drivers that have DirectX 12, but not Vulkan. This can lead to confusing user situations ('DirectX 12 games work fine.. but not Vulkan') and probably means DXVA would be a better choice if targeting Windows
    - my123 3y ago
      > nvidia drivers that have DirectX 12, but not Vulkan not a thing. at worst you get a situation where the ICD loader isn't installed
      - slimsag 3y ago
        In Windows 10 and above, windows update provides a default graphics driver for each IHV, which comes from the manufacturer, has DirectX support, but doesn't have Vulkan. Often nvidia users will try to debug it, and discover 'the vulkan checkbox is off but I have the latest driver version!' in gpuZ[0][1] - but it happens across all IHVs, AMD, Intel, etc. I was told by someone in the industry the default Microsoft drivers across all IHVs indeed have DirectX, but not Vulkan. To make matters more annoying, Windows Update often _overwrites_ the user's driver which they installed from the manufacturer directly, leading to Vulkan becoming broken overnight[2][3] Easy to find multiple instances of this happening[4][5][6], I became aware of it because of someone asking on /r/vulkan a few weeks ago. [0] https://i.imgur.com/qj3Rn9p.png https://i.imgur.com/qj3Rn9p.png https://rog-forum.asus.com/t5/rog-strix-series/nvidia-driver-without-vulkan-support-solved/td-p/892751 https://rog-forum.asus.com/t5/rog-strix-series/nvidia-driver... [1] https://forums.x-plane.org/index.php?/forums/topic/228597-vulkan-checkbox-greyed-out/ https://forums.x-plane.org/index.php?/forums/topic/228597-vu... [2] https://www.reddit.com/r/Amd/comments/1145ffp/windows_update_may_have_automatically_replaced/ https://www.reddit.com/r/Amd/comments/1145ffp/windows_update... [3] https://answers.microsoft.com/en-us/windows/forum/all/windows-10-update-replaced-amd-graphics-driver/78c071ed-eaa7-4508-8d94-1121669c39cf https://answers.microsoft.com/en-us/windows/forum/all/window... [4] https://i.imgur.com/PXZRxYh.png https://i.imgur.com/PXZRxYh.png [5] https://www.reddit.com/r/pcmasterrace/comments/qhfesc/vulkan_disabled_after_driver_update/ https://www.reddit.com/r/pcmasterrace/comments/qhfesc/vulkan... [6] https://www.reddit.com/r/techsupport/comments/110bytq/vulkan_api_disabled_on_amd_card/ https://www.reddit.com/r/techsupport/comments/110bytq/vulkan...
  - ComputerGuru 3y ago
    But why would Firefox go out of its way to use Vulkan on Windows or macOS when it doesn’t have to because it can just use the native OS functionality and let the OS worry about implementation details?
    - shmerl 3y ago
      I don't really care much about Windows and macOS. But I'd like to see it using Vulkan video on Linux in addition or instead of VAAPI. As pointed out elsewhere, it's not possible on macOS either way. Apple are too stuck up not supporting Vulkan and it won't work with MoltenVK.
kllrnohj 3y ago
> Now we need OBS and Firefox supporting Vulkan video too. Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using. It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.
- jdiff 3y ago
  Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.
  - mattst88 3y ago
    > Firefox still has no support for hardware video decode on Linux. That's not true at all. See: https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-video-decoding-on-linux https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-vi... https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.omglinux.com/firefox-hardware-acceleration-raspberry-pi/ https://www.omglinux.com/firefox-hardware-acceleration-raspb...
    - cesarb 3y ago
      How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?
luthMan 3y ago
What about systems where there is now hardware support for AV1? How would Vulkan implement that? Also, maybe someone knows more about this, but I thought that most software like Firefox, MPV and VLC are using dav1d for AV1 decoding as it seems to be the most performant AV1 decoder (in Software) out there at the moment.
- shmerl 3y ago
  > How would Vulkan implement that? See the post. They are working on AV1 extensions.

deleted 3y ago

[deleted]

dishsoap 3y ago

So to clarify, this still leverages the fixed function hardware video encoders present alongside many GPUs from eg. Intel/NVIDIA/AMD, and would not (for example) be possible to use on just a GPU itself (without said encoding hardware)?

shmerl 3y ago
It is for using GPU video related ASICs (same as VAAPI let's say). If you want to use shaders for some video filters, scaling and such - you don't need those extensions. mpv for example can use Vulkan for both at the same time.
monocasa 3y ago
Interestingly, quite a bit of video decode isn't as parallelizable as you might think and isn't a great fit for GPUs. For instance the initial Huffman decoding of the stream is essentially an intrinsically sequential process.
- 0x1ceb00da 3y ago
  Is there a reason we don't have parallelizable video formats?
  - dagmx 3y ago
    Largely because a lot of compression is contextual to the data around it. That makes most decompression a poor fit for parallelism. The primary choices are space savings and power efficiency. Dedicated hardware and serial decoding/encoding often win out as a result.
  - throwup238 3y ago
    Modern video codecs use motion prediction algorithms to encode the information between key frames which depends on the entire frame and the ones immediately preceding it. You can encode the independent groups of pictures (the key frame and all the subsequent predictive frames until the next key frame) in parallel but at least with 4k video you hit memory bandwidth limitations quite quickly.

dlojudice 3y ago

> vkGetPhysicalDeviceVideoEncodeQualityLevelPropertiesKHR how to exceed the character limit on your super wide screen with just one function name

mft_ 3y ago
Needs more snake case for added width sillyness :)
deleted 3y ago
[deleted]
suprjami 3y ago
Eskil Steenberg suggests width of code as a measure of quality. Vulkan wins.
IPTN 3y ago
Huh, It only takes up 2/3 of my phone screen width in portrait mode and reads fine at full arms length.
lostmsu 3y ago
C needs an extension for import without namespaces purely based on names: import vkGetPhysicalDeviceVideoEncode; QualityLevelPropertiesKHR(...); Or even better: import vk*PhysicalDeviceVideoEncode; Get_QualityLevelPropertiesKHR(...);

temp0826 3y ago

This already works decently well in mpv though you need the latest (from git) mesa/libplacebo/ffmpeg to use it.