6 ms·

> Now we need OBS and Firefox supporting Vulkan video too. Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native

> Now we need OBS and Firefox supporting Vulkan video too. Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using. It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.

jdiff 3y ago
Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.
- mattst88 3y ago
  > Firefox still has no support for hardware video decode on Linux. That's not true at all. See: https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-video-decoding-on-linux https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-vi... https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.omglinux.com/firefox-hardware-acceleration-raspberry-pi/ https://www.omglinux.com/firefox-hardware-acceleration-raspb...
  - cesarb 3y ago
    How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?
    - shmerl 3y ago
      At least with AMD GPU, an easy way to do it is to check usage of VCN. Something like this (assuming your GPU index is 0): watch -n 1 sudo cat /sys/kernel/debug/dri/0/amdgpu_pm_info You should see there: VCN: Enabled If GPU accelerated video is being played. I think you might need to set this flag to true in Firefox's about:config (or it might be not needed anymore): media.ffmpeg.vaapi.enabled
    - mattst88 3y ago
      On an Intel GPU, you can check the 'Video' percentage in `sudo intel_gpu_top` (from the igt-gpu-tools package) while playing a video.
    - padenot 3y ago
      We've put a dedicated section called Media in about:support, it has decoding capabilities and other things such as audio IO informations. If you find that it is not accurate, e.g. by cross-checking via other means, please open a ticket at https://bugzilla.mozilla.org/enter_bug.cgi https://bugzilla.mozilla.org/enter_bug.cgi, component "Audio/Video".
      - cesarb 3y ago
        > We've put a dedicated section called Media in about:support, it has decoding capabilities I see, I hadn't found it when searching for the codec name because it only said something like "Information not available. Try again after playing a video." After opening a random YouTube video and refreshing about:support, it did show the hardware decoding information, and it was as I had expected given the hardware on this computer.
  - dhx 3y ago
    Whilst Firefox may support hardware video decoding, Mesa since March 2022 disables patent encumbered codecs by default[1], and distributions such as Fedora and OpenSuse do not explicitly enable these patent encumbered codecs to avoid possible legal problems. Even Gentoo (built from source code by the user) requires the user to explicitly enable a USE flag (proprietary-codes) to use patent encumbered codecs.[2] The thought process is that AMD, NVIDIA, Intel and the likes are not providing a patent license with their hardware.[3] They are instead just supplying part of an overall system that together with operating system kernel, display manager software, video player software, etc allows the decoding and encoding of patent encumbered video files. Open source software projects and distributions are concerned they'd be found to be infringing patents by enabling a complete solution out-of-the-box. Hence they put some hurdles in place so that a user has to go out of their way to separately piece together the various parts to form a complete system capable of encoding and decoding patent encumbered codecs. edit: To clarify, if your Intel or AMD GPU/APU supports a patent-free codec such as AV1 (most GPUs/APUs available for sale?), Firefox on a standard Linux distribution will use hardware video decoding out of the box by default for the patent-free codec. The issue is really one of whether you're sourcing content from a provider that uses a good choice of codec like AV1. The good news is that patent trolls are doing a good job of pushing laggard content providers down this path.[4] [1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15258 https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15... [2] https://github.com/gentoo/gentoo/commit/1265a159743d7f07185acba421e98d46f2cd5fc1 https://github.com/gentoo/gentoo/commit/1265a159743d7f07185a... [3] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ASGIAOCH7XOBX7MN2ZZP2DCYV4MNIYFL/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... [4] https://news.ycombinator.com/item?id=38249527 https://news.ycombinator.com/item?id=38249527
    - Sunspark 3y ago
      I use the flatpak version of Firefox because my Linux machine is an immutable OS. Hardware video decode is working for FF. Not working for Chrome though, sadly.
    - account42 3y ago
      Which is of course a bullshit reason because Linux distributions don't ship the patented hardware decoders (because they are hardwared) so the user already has to assemble the complete system. > a provider that uses a good choice of codec like AV1 Much more limited HW support and slow as molasses encoders. Not exactly a good choice for most.
  - jdiff 3y ago
    My bad, that's what I get for commenting before bed. I meant to tack on an "on Nvidia" in there. AMD and Intel are well supported with VAAPI, but I've never managed to get any of the NV-to-VAAPI wrappers and shims working.
- shmerl 3y ago
  Firefox has both decoding and encoding already supported via ffmpeg+vaapi. What would be nice to see is ffmpeg+vulkan path for it.
shmerl 3y ago
I think a very simple reason is I'd rather rely on Vulkan driver for faster iteration, than on VAAPI. It's historically moving much slower since it first needs Intel to pull weight, then AMD to implement their side for example.
tagrun 3y ago
> They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos One vendor specific API, not "various OS' native APIs". Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA. (A third-party VA-API to NVDEC translation layer for Linux does exist on GitHub, nvidia-vaapi-driver, but it's not yet reliable as the officially supported VDPAU or NVDEC, and is not included in official linux package repositories.) Intel has VA-API, AMD has AMF, and NVIDIA has VDPAU which is being replaced by NVDEC/NVENC. The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.
- mikepavone 3y ago
  > No, Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only, which is not supported by NVIDIA. Intel is behind VA-API originally, but I don't think it's fair to say it's a vendor specific API anymore. It's supported by the open source drivers for GPUs from all 3 vendors. It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
  - tagrun 3y ago
    You can make the same argument for VDPAU. AMD officially supports it, and there is an unofficial translation layer with limited capabilities for some Intel GPUs. Is VDPAU not a vendor specific API anymore then? Intel, AMD and NVIDIA have their own vendor-specific video APIs, and even when they provide official support for the API of another vendor, it tends to expose a limited subset of the full functionality (like the list of available codecs and encoding features). You are free to call these vendor specific APIs for what they are or something else, but the reality has been that there is no single video API officially supported by Intel, AMD and NVIDIA. This changed with Vulkan Video. But Vulkan Video isn't just about desktop: mobile devices, Raspberry Pi, etc. are expected to get on board with it eventually, just like they did with Vulkan. > It's supported by the open source drivers for GPUs from all 3 vendors. Which 3 vendors are you referring to? Intel, AMD, and who? > It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc Why are you bringing up open source drivers, and what is not practical? Both official open source drivers (open-gpu-kernel-module) and unofficial open source drivers (nouveau, through binary firmware) support VDPAU. However, NVIDIA's drivers (open source or binary) does not support VA-API.
geraldhh 3y ago
> Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos they are also plentiful, vendor specific and unstable. a generic solution would be well appreciated

jdiff 3y ago

Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.

mattst88 3y ago
> Firefox still has no support for hardware video decode on Linux. That's not true at all. See: https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-video-decoding-on-linux https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-vi... https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.omglinux.com/firefox-hardware-acceleration-raspberry-pi/ https://www.omglinux.com/firefox-hardware-acceleration-raspb...
- cesarb 3y ago
  How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?
  - shmerl 3y ago
    At least with AMD GPU, an easy way to do it is to check usage of VCN. Something like this (assuming your GPU index is 0): watch -n 1 sudo cat /sys/kernel/debug/dri/0/amdgpu_pm_info You should see there: VCN: Enabled If GPU accelerated video is being played. I think you might need to set this flag to true in Firefox's about:config (or it might be not needed anymore): media.ffmpeg.vaapi.enabled
  - mattst88 3y ago
    On an Intel GPU, you can check the 'Video' percentage in `sudo intel_gpu_top` (from the igt-gpu-tools package) while playing a video.
  - padenot 3y ago
    We've put a dedicated section called Media in about:support, it has decoding capabilities and other things such as audio IO informations. If you find that it is not accurate, e.g. by cross-checking via other means, please open a ticket at https://bugzilla.mozilla.org/enter_bug.cgi https://bugzilla.mozilla.org/enter_bug.cgi, component "Audio/Video".
    - cesarb 3y ago
      > We've put a dedicated section called Media in about:support, it has decoding capabilities I see, I hadn't found it when searching for the codec name because it only said something like "Information not available. Try again after playing a video." After opening a random YouTube video and refreshing about:support, it did show the hardware decoding information, and it was as I had expected given the hardware on this computer.
- dhx 3y ago
  Whilst Firefox may support hardware video decoding, Mesa since March 2022 disables patent encumbered codecs by default[1], and distributions such as Fedora and OpenSuse do not explicitly enable these patent encumbered codecs to avoid possible legal problems. Even Gentoo (built from source code by the user) requires the user to explicitly enable a USE flag (proprietary-codes) to use patent encumbered codecs.[2] The thought process is that AMD, NVIDIA, Intel and the likes are not providing a patent license with their hardware.[3] They are instead just supplying part of an overall system that together with operating system kernel, display manager software, video player software, etc allows the decoding and encoding of patent encumbered video files. Open source software projects and distributions are concerned they'd be found to be infringing patents by enabling a complete solution out-of-the-box. Hence they put some hurdles in place so that a user has to go out of their way to separately piece together the various parts to form a complete system capable of encoding and decoding patent encumbered codecs. edit: To clarify, if your Intel or AMD GPU/APU supports a patent-free codec such as AV1 (most GPUs/APUs available for sale?), Firefox on a standard Linux distribution will use hardware video decoding out of the box by default for the patent-free codec. The issue is really one of whether you're sourcing content from a provider that uses a good choice of codec like AV1. The good news is that patent trolls are doing a good job of pushing laggard content providers down this path.[4] [1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15258 https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15... [2] https://github.com/gentoo/gentoo/commit/1265a159743d7f07185acba421e98d46f2cd5fc1 https://github.com/gentoo/gentoo/commit/1265a159743d7f07185a... [3] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ASGIAOCH7XOBX7MN2ZZP2DCYV4MNIYFL/ https://lists.fedoraproject.org/archives/list/devel@lists.fe... [4] https://news.ycombinator.com/item?id=38249527 https://news.ycombinator.com/item?id=38249527
  - Sunspark 3y ago
    I use the flatpak version of Firefox because my Linux machine is an immutable OS. Hardware video decode is working for FF. Not working for Chrome though, sadly.
  - account42 3y ago
    Which is of course a bullshit reason because Linux distributions don't ship the patented hardware decoders (because they are hardwared) so the user already has to assemble the complete system. > a provider that uses a good choice of codec like AV1 Much more limited HW support and slow as molasses encoders. Not exactly a good choice for most.
- jdiff 3y ago
  My bad, that's what I get for commenting before bed. I meant to tack on an "on Nvidia" in there. AMD and Intel are well supported with VAAPI, but I've never managed to get any of the NV-to-VAAPI wrappers and shims working.
shmerl 3y ago
Firefox has both decoding and encoding already supported via ffmpeg+vaapi. What would be nice to see is ffmpeg+vulkan path for it.

shmerl 3y ago

I think a very simple reason is I'd rather rely on Vulkan driver for faster iteration, than on VAAPI. It's historically moving much slower since it first needs Intel to pull weight, then AMD to implement their side for example.

tagrun 3y ago

> They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos One vendor specific API, not "various OS' native APIs". Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA. (A third-party VA-API to NVDEC translation layer for Linux does exist on GitHub, nvidia-vaapi-driver, but it's not yet reliable as the officially supported VDPAU or NVDEC, and is not included in official linux package repositories.) Intel has VA-API, AMD has AMF, and NVIDIA has VDPAU which is being replaced by NVDEC/NVENC. The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.

mikepavone 3y ago
> No, Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only, which is not supported by NVIDIA. Intel is behind VA-API originally, but I don't think it's fair to say it's a vendor specific API anymore. It's supported by the open source drivers for GPUs from all 3 vendors. It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
- tagrun 3y ago
  You can make the same argument for VDPAU. AMD officially supports it, and there is an unofficial translation layer with limited capabilities for some Intel GPUs. Is VDPAU not a vendor specific API anymore then? Intel, AMD and NVIDIA have their own vendor-specific video APIs, and even when they provide official support for the API of another vendor, it tends to expose a limited subset of the full functionality (like the list of available codecs and encoding features). You are free to call these vendor specific APIs for what they are or something else, but the reality has been that there is no single video API officially supported by Intel, AMD and NVIDIA. This changed with Vulkan Video. But Vulkan Video isn't just about desktop: mobile devices, Raspberry Pi, etc. are expected to get on board with it eventually, just like they did with Vulkan. > It's supported by the open source drivers for GPUs from all 3 vendors. Which 3 vendors are you referring to? Intel, AMD, and who? > It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc Why are you bringing up open source drivers, and what is not practical? Both official open source drivers (open-gpu-kernel-module) and unofficial open source drivers (nouveau, through binary firmware) support VDPAU. However, NVIDIA's drivers (open source or binary) does not support VA-API.

geraldhh 3y ago

> Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos they are also plentiful, vendor specific and unstable. a generic solution would be well appreciated