9 ms·

Firefox 80 and my confusion over its hardware accelerated video on Linux

116 points by eitau_1 6y ago

boring_twenties 6y ago
I've never fully understood just what exactly the problem is? Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to know or care which specific hardware I'm using. What makes it so much more difficult for FF?
- kevin_thibedeau 6y ago
  DRM probably.
  - Delk 6y ago
    YouTube etc. don't have DRM in the video they serve, as far as I know. Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.
- JoshTriplett 6y ago
  Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU. Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video. Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".
  - lcnmrn 6y ago
    I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.
    - JoshTriplett 6y ago
      The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.
  - stefan_ 6y ago
    All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff. Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.
    - heftig 6y ago
      I think the expectation is that WebGL is used to manipulate video frames.
    - vetinari 6y ago
      > Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.
- InfiniteRand 6y ago
  Maybe this is just unwarranted speculation, but I remember a period many years ago (maybe 10+?) when Firefox was very unstable when hardware acceleration was enabled, even on Windows, although it depended on what exactly your hardware was. Eventually the quality of the drivers on Windows made hardware acceleration less unstable on Windows, but that same progress did not happen on Linux (or was happening much slower) so for a while Firefox gave up on Linux hardware acceleration. Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly. Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.
- kevinoid 6y ago
  You can read about the specific reasons that driver versions were blocked on the Mozilla Wiki https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... Most are due to the drivers causing crashes or security issues in Firefox. I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.
- TD-Linux 6y ago
  VLC also has blocklists, e.g. https://code.videolan.org/videolan/vlc/-/blob/master/modules/codec/avcodec/dxva_blocklist.c https://code.videolan.org/videolan/vlc/-/blob/master/modules... https://code.videolan.org/videolan/vlc/-/blob/master/modules/video_output/opengl/interop_vaapi.c#L323 https://code.videolan.org/videolan/vlc/-/blob/master/modules... As does gstreamer (used by Totem): https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/ https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api...
  - Delk 6y ago
    It still seems, at least based on anecdata and a hunch, that decode acceleration through VA-API works more often in VLC than in Firefox. VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever. (Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.) I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.
- Const-me 6y ago
  I think legacy contributed most. On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo https://github.com/Const-me/Vrmac/tree/master/VrmacVideo However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different. 1. They spent much time on that old code (sunk cost fallacy). 2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken). 3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.
  - Jasper_ 6y ago
    > On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers). M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".
    - vetinari 6y ago
      AMD uses VA-API nowadays.
      - selectodude 6y ago
        As does Nvidia.
- ronjouch 6y ago
  The problem is compositing with the rest of the web page. In VLC you don't have much complexity on top of or around the video (a few buttons, sometimes subtitles or a volume overlay), whereas around a video in a browser you have THE INTERWEBS (insert scream emoji here), which themselves also go through hardware acceleration. Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.
- captainmuon 6y ago
  They used to have playback via gstreamer (and before that mplayer plugin). First they castrated it for political reasons (you could only use certain codecs, officially so that codecs wouldn't proliferate and website authors wouldn't start to depend on them) and then they removed it altogether. Now it uses a bundled ffmpeg. I think this switch is one reason it took so long. Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.
  - pjmlp 6y ago
    That is how it works on most desktop platforms with their video frameworks. Naturally Linux desktop being as it is, this solution will never work.
    - PaulDavisThe1st 6y ago
      You mean "naturally, since many of these codecs are completely or partially proprietary and/or not publically documented, this solution will not work on platforms that focus on non-proprietary open standards"?
      - pjmlp 6y ago
        Idealism has a price, but then there is the NIH syndrome on top of that.
verroq 6y ago
This is just one of the times where you’d just have to read the source. No documentation usually means I start diving into the source code.
- tpmx 6y ago
  That attitude/capability is closely related to a job interview question I tend to use. "Could you tell me about a time you delved into the source code of some software product or library to fix a problem you faced?" (My pet theory: I think often it's just the attitude that is missing. So much can be accompished with the right attitude.)
  - missblit 6y ago
    It's impossible to get anything done otherwise, and some cursory investigation is usually a lot faster than emailing some overworked maintainer with nothing but "plz help".
kevinoid 6y ago
I ran into similar issues with driver blocklists on both Linux and Windows. There are workarounds, such as setting MOZ_GFX_SPOOF_* environment variables, but even the suggestions on the Mozilla Wiki at https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers#How_to_force-enable_blocked_graphics_features https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... no longer work, because the spoofed driver version (8.15.10.2302) was blocked in https://bugzilla.mozilla.org/843273 https://bugzilla.mozilla.org/843273 It's a mess. I can understand the desire to avoid using drivers which cause a poor user experience. I don't envy the task of trying to implement and maintain a system to achieve that. I do wish it wasn't so painful and confusing for users when the behavior doesn't match our expectations.
est31 6y ago
IDK why this post is so negative. The post says it's the first release with that feature, and that things are still improving. I'm happy that it's being worked on at all and maybe a couple more releases down the road it's available to me as well. Mozilla/Firefox contributors are adding a feature here in the open but it turn out it's very complicated to add it.
- kelnos 6y ago
  Right, and my feeling is that any feature that requires you to mess around with about:config is inherently an alpha/beta feature. Expecting it to work properly on every setup is expecting too much; once it's enabled by default, then you should expect it to work out of the box, unless your hardware/OS/driver combo has been explicitly blocked due to issues.
- pstrateman 6y ago
  Hardware accelerated video decoding is a feature that is and has been critical for battery life on laptops for years. ffmpeg has supported this for forever and yet firefox doesn't. It's bizarre.
  - heftig 6y ago
    Do you believe enabling hardware video decoding in Firefox was a trivial effort?
    - vetinari 6y ago
      What Firefox 78-80 brought was hardware video decoding under Linux, not hardware video decoding in general. Under Windows and MacOS, it has been implemented years ago. Under Linux, it wasn't. Linux has API for video acceleration and players like VLC, mpv or Kodi have been supporting it for years, as well as ffmpeg, just browsers ignored it. I'm not going to belittle the effort, quite opposite, but it is telling when this feature has been implemented by Redhat employee and not Mozilla.
    - pjmlp 6y ago
novaRom 6y ago
What if: Mozilla is a spooky company. They covardly pretend to be pro-freedom, but in reality they have a shady agreement with Google to look like an alternative, but not better. This is well known pattern. For example what if Nvidia has agreement with Microsoft to provide mediocre experience to average Desktop Linux user.
- boomboomsubban 6y ago
  Then explain the ~five year period where the bulk of their revenue came from Yahoo. Just a ridiculous idea.
boomboomsubban 6y ago
From what I can gather, they're using an Intel card, which I don't think supports VA-API on MPEG-4 videos. Further, I can't tell which distro they use but I'm not sure when the Intel work was added. Their distro provided version would likely iron out some of these issues.
- vetinari 6y ago
  That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding. An extra driver is (intel-vaapi-driver) is needed, it is not distributed and installed together with Mesa as AMD driver is.
  - boomboomsubban 6y ago
    >That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding It does not support MPEG-4, or it's disabled by default. I don't know how widespread that format is, but I believe it was developed with streaming in mind.
    - vetinari 6y ago
      If you mean MPEG 4 ASP (i.e. part 2, as opposed to H.264, which is also MPEG-4, but part 10) I don't believe any Intel hardware did ever support it. Today, it is obsolete.
deleted 6y ago
[deleted]
jml7c5 6y ago
What Firefox really needs is a little icon (preferably displayed prominently in the toolbar) that appears when videos are not being decoded in hardware. I'm sure they've lost users who just think Firefox is "super laggy with YouTube" or that "Firefox really eats battery". An indication that something is wrong and the browser knows about it prevents the assumption that the browser is just buggy/slow/etc; the word-of-mouth changes from "Firefox sucks" to "Firefox told me it couldn't hardware accelerate my videos or something?" Still not positive, but potential users will hear that Firefox might not work with their hardware, rather than that Firefox is buggy, slow, and unreliable. And I know technical users would appreciate it. I still don't know what is and isn't getting hardware video decode, or whether I actually need a particular about:config flag or system package, and I have tried hard to figure out.
- jmisavage 6y ago
  You can go icon blind pretty easily. That being said I know my computer doesn't have VP8/9 hardware support so I installed the h264ify plugin for Firefox so YouTube serves me H.264 videos instead. Maybe Mozilla needs to detect hardware support and offer to install plugins like that or give a setup screen (newer video formats with more features vs old video formats with better battery/less fans)
- stefan_ 6y ago
  The default assumption today and in the past on Linux is that there is zero hardware accelerated video display in any browser.
  - pjmlp 6y ago
    Which is why macOS, Windows and ChromeOS are the platforms for anyone who care about video hardware acceleration without messing with configurations.
    - diamondlovesyou 6y ago
      ChromeOS is Linux.
zanny 6y ago
I did manage to get it working but it has a ton of issues with Youtube vp9 decoding despite Chromium's years-old implementation working fine. This is complex stuff. I spent a day last week trying to figure out why hevc vaapi patches for OBS were failing with mkv containers but not mp4 ones. Its just really ugly C apis all the way down.
ronjouch 6y ago
Yes, the UX is raw for now; hopefully Mozilla will make it simpler to enable after a few versions of testing and confirming it doesn't cause crashes for half the linux userbase. It makes sense to do little early advertising of such a feature, and to initially hide it behind flags & env. vars only within reach of "power users", because these fresh-out-of-the-oven bits are a complex assembly with heavy crash potential, depending on your hardware and drivers up-to-date-ness. As the implementer, you want your initial users to know what they are doing, and you want to maximize the chances they're technically proficient enough to gather the mountain of gpu/driver/environment debug info you'll need for a bug report to be actionable. In the meantime, to save you rummaging around bugzilla, follow https://wiki.archlinux.org/index.php/Firefox#Hardware_video_acceleration https://wiki.archlinux.org/index.php/Firefox#Hardware_video_... , it's the most constantly {up-to-date, concise, reliable} source you'll find. You don't have to be using Arch, the information contained in this section applies to any Linux/Firefox distro. Note that Firefox isn't the only browser to struggle with it; see https://wiki.archlinux.org/index.php/Chromium#Hardware_video_acceleration https://wiki.archlinux.org/index.php/Chromium#Hardware_video... for the similarly non-obvious instructions in Chromium (and by the way, currently it's plain impossible in Chrome). Props to Martin Stransky from RedHat who implemented this for both Wayland and Xorg during the last months.
- test7777 6y ago
  do what chrome does, detect crash and fallback. what are sandboxes even for if you can't crash once in a while without whining too much about it...
  - ronjouch 6y ago
    I doubt things are that simple. See my before-last paragraph above: no, regarding hardware video acceleration, even Chrome doesn't do what you describe. What you describe stands for non-video content crashes, which isn't what's being discussed here. Could the technique you mention be applied to video gpu crashes? I personally don't know and I'd lean towards "nope" (else Mozilla & Google engineers wouldn't be so exceedingly cautious about it). If anyone is knowledgeable enough to precise / contradict this, please chime in. At any rate, even Chrome is built with zero hardware video acceleration support on Linux (hardware acceleration yes, hardware video acceleration no). Chromium does offer the option, just as hidden behind flags as Firefox, and until last month you had to build it yourself with a VAAPI patch left unmerged for years by Chromium devs.
bleepblorp 6y ago
It's not possible go get VA-API to work in FF 88 on X11 or Wayland because VA-API was broken in Firefox 79. The fix is in FF 81.
- bengalister 6y ago
  It seems consistent to what I experimented on Arch LTS/Wayland since FF 79. I have very frequent video player crashes when enabled (like on YT).
- usr1106 6y ago
  Not sure I follow, did you possibly mean: It's not possible to get VA-API to work in FF 80 on X11 or Wayland because VA-API has been broken since FF 79. The fix will be in FF 81. Just guessing, I have not looked into the topic except in this thread. What would be the bugzilla id for that?
  - bleepblorp 6y ago
    Replace the '88' with '80'. VA-API hardware video acceleration was broken in Firefox 79 and won't be fixed until Firefox 81. Therefore, it's not possible to make VA-API work, under either Wayland or X11, in Firefox 80. See: https://bugzilla.mozilla.org/show_bug.cgi?id=1656436 https://bugzilla.mozilla.org/show_bug.cgi?id=1656436
    - usr1106 6y ago
      So to put it cynically: In FF 80 the situation is crystal clear, not at all confusing as the original article claims: It doesn't work. In FF 81 it will be confusing to the non-experienced again: You need to fiddle with config and environment variables. Well, unless a new bug comes up...
bengalister 6y ago
For me, on Arch LTS/wayland/gnome Firefox's video hardware acceleration became buggy since Firefoxy 79 (I followed the Arch wiki recommendations). I had to disable the gfx renderer. And cpu consumption is only marginally better than Chromium. When enabled, cpu consumption is very low <10% watching 720p YT videos, between 10-20% for 1080p but the video player crashes very often.
pjmlp 6y ago
Yeah, since the anti-Flash movement has won, Linux is even less interesting for anyone wanting an out of the box experience regarding video hardware acceleration for Web browsers.
usr1106 6y ago
What would be a poor man's reliable test to see whether acceleration is actually in use or not? If you could just reliably switch it on and off the difference should be obvious in top, possibly by switching to threads view by pressing H (instead of the processes default). But if you cannot, there is nothing to compare.
- sp332 6y ago
  Go to about:support and look in the Graphics section.
  - usr1106 6y ago
    not at my computer now, but if I remember right, that shows you whether HW acceleration could in theory work for some video or not at all. The second case is a clear one. But in the first case you still don't know whether it is actually used for a certain video or not. I understand there can be many parameters preventing usage.

116 points by eitau_1 6y ago

I've never fully understood just what exactly the problem is? Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to know or care which specific hardware I'm using. What makes it so much more difficult for FF?

kevin_thibedeau 6y ago
DRM probably.
- Delk 6y ago
  YouTube etc. don't have DRM in the video they serve, as far as I know. Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.
JoshTriplett 6y ago
Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU. Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video. Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".
- lcnmrn 6y ago
  I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.
  - JoshTriplett 6y ago
    The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.
- stefan_ 6y ago
  All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff. Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.
  - heftig 6y ago
    I think the expectation is that WebGL is used to manipulate video frames.
  - vetinari 6y ago
    > Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.
InfiniteRand 6y ago
Maybe this is just unwarranted speculation, but I remember a period many years ago (maybe 10+?) when Firefox was very unstable when hardware acceleration was enabled, even on Windows, although it depended on what exactly your hardware was. Eventually the quality of the drivers on Windows made hardware acceleration less unstable on Windows, but that same progress did not happen on Linux (or was happening much slower) so for a while Firefox gave up on Linux hardware acceleration. Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly. Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.
kevinoid 6y ago
You can read about the specific reasons that driver versions were blocked on the Mozilla Wiki https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... Most are due to the drivers causing crashes or security issues in Firefox. I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.
TD-Linux 6y ago
VLC also has blocklists, e.g. https://code.videolan.org/videolan/vlc/-/blob/master/modules/codec/avcodec/dxva_blocklist.c https://code.videolan.org/videolan/vlc/-/blob/master/modules... https://code.videolan.org/videolan/vlc/-/blob/master/modules/video_output/opengl/interop_vaapi.c#L323 https://code.videolan.org/videolan/vlc/-/blob/master/modules... As does gstreamer (used by Totem): https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/ https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api...
- Delk 6y ago
  It still seems, at least based on anecdata and a hunch, that decode acceleration through VA-API works more often in VLC than in Firefox. VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever. (Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.) I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.
Const-me 6y ago
I think legacy contributed most. On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo https://github.com/Const-me/Vrmac/tree/master/VrmacVideo However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different. 1. They spent much time on that old code (sunk cost fallacy). 2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken). 3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.
- Jasper_ 6y ago
  > On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers). M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".
  - vetinari 6y ago
    AMD uses VA-API nowadays.
    - selectodude 6y ago
      As does Nvidia.
ronjouch 6y ago
The problem is compositing with the rest of the web page. In VLC you don't have much complexity on top of or around the video (a few buttons, sometimes subtitles or a volume overlay), whereas around a video in a browser you have THE INTERWEBS (insert scream emoji here), which themselves also go through hardware acceleration. Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.
captainmuon 6y ago
They used to have playback via gstreamer (and before that mplayer plugin). First they castrated it for political reasons (you could only use certain codecs, officially so that codecs wouldn't proliferate and website authors wouldn't start to depend on them) and then they removed it altogether. Now it uses a bundled ffmpeg. I think this switch is one reason it took so long. Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.
- pjmlp 6y ago
  That is how it works on most desktop platforms with their video frameworks. Naturally Linux desktop being as it is, this solution will never work.
  - PaulDavisThe1st 6y ago
    You mean "naturally, since many of these codecs are completely or partially proprietary and/or not publically documented, this solution will not work on platforms that focus on non-proprietary open standards"?
    - pjmlp 6y ago
      Idealism has a price, but then there is the NIH syndrome on top of that.

verroq 6y ago

This is just one of the times where you’d just have to read the source. No documentation usually means I start diving into the source code.

tpmx 6y ago
That attitude/capability is closely related to a job interview question I tend to use. "Could you tell me about a time you delved into the source code of some software product or library to fix a problem you faced?" (My pet theory: I think often it's just the attitude that is missing. So much can be accompished with the right attitude.)
- missblit 6y ago
  It's impossible to get anything done otherwise, and some cursory investigation is usually a lot faster than emailing some overworked maintainer with nothing but "plz help".

kevinoid 6y ago

I ran into similar issues with driver blocklists on both Linux and Windows. There are workarounds, such as setting MOZ_GFX_SPOOF_* environment variables, but even the suggestions on the Mozilla Wiki at https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers#How_to_force-enable_blocked_graphics_features https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... no longer work, because the spoofed driver version (8.15.10.2302) was blocked in https://bugzilla.mozilla.org/843273 https://bugzilla.mozilla.org/843273 It's a mess. I can understand the desire to avoid using drivers which cause a poor user experience. I don't envy the task of trying to implement and maintain a system to achieve that. I do wish it wasn't so painful and confusing for users when the behavior doesn't match our expectations.

est31 6y ago

IDK why this post is so negative. The post says it's the first release with that feature, and that things are still improving. I'm happy that it's being worked on at all and maybe a couple more releases down the road it's available to me as well. Mozilla/Firefox contributors are adding a feature here in the open but it turn out it's very complicated to add it.

kelnos 6y ago
Right, and my feeling is that any feature that requires you to mess around with about:config is inherently an alpha/beta feature. Expecting it to work properly on every setup is expecting too much; once it's enabled by default, then you should expect it to work out of the box, unless your hardware/OS/driver combo has been explicitly blocked due to issues.
pstrateman 6y ago
Hardware accelerated video decoding is a feature that is and has been critical for battery life on laptops for years. ffmpeg has supported this for forever and yet firefox doesn't. It's bizarre.
- heftig 6y ago
  Do you believe enabling hardware video decoding in Firefox was a trivial effort?
  - vetinari 6y ago
    What Firefox 78-80 brought was hardware video decoding under Linux, not hardware video decoding in general. Under Windows and MacOS, it has been implemented years ago. Under Linux, it wasn't. Linux has API for video acceleration and players like VLC, mpv or Kodi have been supporting it for years, as well as ffmpeg, just browsers ignored it. I'm not going to belittle the effort, quite opposite, but it is telling when this feature has been implemented by Redhat employee and not Mozilla.
  - pjmlp 6y ago

novaRom 6y ago

What if: Mozilla is a spooky company. They covardly pretend to be pro-freedom, but in reality they have a shady agreement with Google to look like an alternative, but not better. This is well known pattern. For example what if Nvidia has agreement with Microsoft to provide mediocre experience to average Desktop Linux user.

boomboomsubban 6y ago
Then explain the ~five year period where the bulk of their revenue came from Yahoo. Just a ridiculous idea.

boomboomsubban 6y ago

From what I can gather, they're using an Intel card, which I don't think supports VA-API on MPEG-4 videos. Further, I can't tell which distro they use but I'm not sure when the Intel work was added. Their distro provided version would likely iron out some of these issues.

vetinari 6y ago
That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding. An extra driver is (intel-vaapi-driver) is needed, it is not distributed and installed together with Mesa as AMD driver is.
- boomboomsubban 6y ago
  >That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding It does not support MPEG-4, or it's disabled by default. I don't know how widespread that format is, but I believe it was developed with streaming in mind.
  - vetinari 6y ago
    If you mean MPEG 4 ASP (i.e. part 2, as opposed to H.264, which is also MPEG-4, but part 10) I don't believe any Intel hardware did ever support it. Today, it is obsolete.

deleted 6y ago

[deleted]

jml7c5 6y ago

What Firefox really needs is a little icon (preferably displayed prominently in the toolbar) that appears when videos are not being decoded in hardware. I'm sure they've lost users who just think Firefox is "super laggy with YouTube" or that "Firefox really eats battery". An indication that something is wrong and the browser knows about it prevents the assumption that the browser is just buggy/slow/etc; the word-of-mouth changes from "Firefox sucks" to "Firefox told me it couldn't hardware accelerate my videos or something?" Still not positive, but potential users will hear that Firefox might not work with their hardware, rather than that Firefox is buggy, slow, and unreliable. And I know technical users would appreciate it. I still don't know what is and isn't getting hardware video decode, or whether I actually need a particular about:config flag or system package, and I have tried hard to figure out.

jmisavage 6y ago
You can go icon blind pretty easily. That being said I know my computer doesn't have VP8/9 hardware support so I installed the h264ify plugin for Firefox so YouTube serves me H.264 videos instead. Maybe Mozilla needs to detect hardware support and offer to install plugins like that or give a setup screen (newer video formats with more features vs old video formats with better battery/less fans)
stefan_ 6y ago
The default assumption today and in the past on Linux is that there is zero hardware accelerated video display in any browser.
- pjmlp 6y ago
  Which is why macOS, Windows and ChromeOS are the platforms for anyone who care about video hardware acceleration without messing with configurations.
  - diamondlovesyou 6y ago
    ChromeOS is Linux.

zanny 6y ago

I did manage to get it working but it has a ton of issues with Youtube vp9 decoding despite Chromium's years-old implementation working fine. This is complex stuff. I spent a day last week trying to figure out why hevc vaapi patches for OBS were failing with mkv containers but not mp4 ones. Its just really ugly C apis all the way down.

ronjouch 6y ago

Yes, the UX is raw for now; hopefully Mozilla will make it simpler to enable after a few versions of testing and confirming it doesn't cause crashes for half the linux userbase. It makes sense to do little early advertising of such a feature, and to initially hide it behind flags & env. vars only within reach of "power users", because these fresh-out-of-the-oven bits are a complex assembly with heavy crash potential, depending on your hardware and drivers up-to-date-ness. As the implementer, you want your initial users to know what they are doing, and you want to maximize the chances they're technically proficient enough to gather the mountain of gpu/driver/environment debug info you'll need for a bug report to be actionable. In the meantime, to save you rummaging around bugzilla, follow https://wiki.archlinux.org/index.php/Firefox#Hardware_video_acceleration https://wiki.archlinux.org/index.php/Firefox#Hardware_video_... , it's the most constantly {up-to-date, concise, reliable} source you'll find. You don't have to be using Arch, the information contained in this section applies to any Linux/Firefox distro. Note that Firefox isn't the only browser to struggle with it; see https://wiki.archlinux.org/index.php/Chromium#Hardware_video_acceleration https://wiki.archlinux.org/index.php/Chromium#Hardware_video... for the similarly non-obvious instructions in Chromium (and by the way, currently it's plain impossible in Chrome). Props to Martin Stransky from RedHat who implemented this for both Wayland and Xorg during the last months.

test7777 6y ago
do what chrome does, detect crash and fallback. what are sandboxes even for if you can't crash once in a while without whining too much about it...
- ronjouch 6y ago
  I doubt things are that simple. See my before-last paragraph above: no, regarding hardware video acceleration, even Chrome doesn't do what you describe. What you describe stands for non-video content crashes, which isn't what's being discussed here. Could the technique you mention be applied to video gpu crashes? I personally don't know and I'd lean towards "nope" (else Mozilla & Google engineers wouldn't be so exceedingly cautious about it). If anyone is knowledgeable enough to precise / contradict this, please chime in. At any rate, even Chrome is built with zero hardware video acceleration support on Linux (hardware acceleration yes, hardware video acceleration no). Chromium does offer the option, just as hidden behind flags as Firefox, and until last month you had to build it yourself with a VAAPI patch left unmerged for years by Chromium devs.

bleepblorp 6y ago

It's not possible go get VA-API to work in FF 88 on X11 or Wayland because VA-API was broken in Firefox 79. The fix is in FF 81.

bengalister 6y ago
It seems consistent to what I experimented on Arch LTS/Wayland since FF 79. I have very frequent video player crashes when enabled (like on YT).
usr1106 6y ago
Not sure I follow, did you possibly mean: It's not possible to get VA-API to work in FF 80 on X11 or Wayland because VA-API has been broken since FF 79. The fix will be in FF 81. Just guessing, I have not looked into the topic except in this thread. What would be the bugzilla id for that?
- bleepblorp 6y ago
  Replace the '88' with '80'. VA-API hardware video acceleration was broken in Firefox 79 and won't be fixed until Firefox 81. Therefore, it's not possible to make VA-API work, under either Wayland or X11, in Firefox 80. See: https://bugzilla.mozilla.org/show_bug.cgi?id=1656436 https://bugzilla.mozilla.org/show_bug.cgi?id=1656436
  - usr1106 6y ago
    So to put it cynically: In FF 80 the situation is crystal clear, not at all confusing as the original article claims: It doesn't work. In FF 81 it will be confusing to the non-experienced again: You need to fiddle with config and environment variables. Well, unless a new bug comes up...

bengalister 6y ago

For me, on Arch LTS/wayland/gnome Firefox's video hardware acceleration became buggy since Firefoxy 79 (I followed the Arch wiki recommendations). I had to disable the gfx renderer. And cpu consumption is only marginally better than Chromium. When enabled, cpu consumption is very low <10% watching 720p YT videos, between 10-20% for 1080p but the video player crashes very often.

pjmlp 6y ago

Yeah, since the anti-Flash movement has won, Linux is even less interesting for anyone wanting an out of the box experience regarding video hardware acceleration for Web browsers.

usr1106 6y ago

What would be a poor man's reliable test to see whether acceleration is actually in use or not? If you could just reliably switch it on and off the difference should be obvious in top, possibly by switching to threads view by pressing H (instead of the processes default). But if you cannot, there is nothing to compare.

sp332 6y ago
Go to about:support and look in the Graphics section.
- usr1106 6y ago
  not at my computer now, but if I remember right, that shows you whether HW acceleration could in theory work for some video or not at all. The second case is a clear one. But in the first case you still don't know whether it is actually used for a certain video or not. I understand there can be many parameters preventing usage.