5 ms·

I've never fully understood just what exactly the problem is? Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to

I've never fully understood just what exactly the problem is? Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to know or care which specific hardware I'm using. What makes it so much more difficult for FF?

kevin_thibedeau 6y ago
DRM probably.
- Delk 6y ago
  YouTube etc. don't have DRM in the video they serve, as far as I know. Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.
JoshTriplett 6y ago
Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU. Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video. Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".
- lcnmrn 6y ago
  I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.
  - JoshTriplett 6y ago
    The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.
- stefan_ 6y ago
  All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff. Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.
  - heftig 6y ago
    I think the expectation is that WebGL is used to manipulate video frames.
  - vetinari 6y ago
    > Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.
InfiniteRand 6y ago
Maybe this is just unwarranted speculation, but I remember a period many years ago (maybe 10+?) when Firefox was very unstable when hardware acceleration was enabled, even on Windows, although it depended on what exactly your hardware was. Eventually the quality of the drivers on Windows made hardware acceleration less unstable on Windows, but that same progress did not happen on Linux (or was happening much slower) so for a while Firefox gave up on Linux hardware acceleration. Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly. Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.
kevinoid 6y ago
You can read about the specific reasons that driver versions were blocked on the Mozilla Wiki https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... Most are due to the drivers causing crashes or security issues in Firefox. I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.
TD-Linux 6y ago
VLC also has blocklists, e.g. https://code.videolan.org/videolan/vlc/-/blob/master/modules/codec/avcodec/dxva_blocklist.c https://code.videolan.org/videolan/vlc/-/blob/master/modules... https://code.videolan.org/videolan/vlc/-/blob/master/modules/video_output/opengl/interop_vaapi.c#L323 https://code.videolan.org/videolan/vlc/-/blob/master/modules... As does gstreamer (used by Totem): https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/ https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api...
- Delk 6y ago
  It still seems, at least based on anecdata and a hunch, that decode acceleration through VA-API works more often in VLC than in Firefox. VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever. (Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.) I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.
Const-me 6y ago
I think legacy contributed most. On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo https://github.com/Const-me/Vrmac/tree/master/VrmacVideo However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different. 1. They spent much time on that old code (sunk cost fallacy). 2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken). 3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.
- Jasper_ 6y ago
  > On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers). M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".
  - vetinari 6y ago
    AMD uses VA-API nowadays.
    - selectodude 6y ago
      As does Nvidia.
ronjouch 6y ago
The problem is compositing with the rest of the web page. In VLC you don't have much complexity on top of or around the video (a few buttons, sometimes subtitles or a volume overlay), whereas around a video in a browser you have THE INTERWEBS (insert scream emoji here), which themselves also go through hardware acceleration. Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.
captainmuon 6y ago
They used to have playback via gstreamer (and before that mplayer plugin). First they castrated it for political reasons (you could only use certain codecs, officially so that codecs wouldn't proliferate and website authors wouldn't start to depend on them) and then they removed it altogether. Now it uses a bundled ffmpeg. I think this switch is one reason it took so long. Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.
- pjmlp 6y ago
  That is how it works on most desktop platforms with their video frameworks. Naturally Linux desktop being as it is, this solution will never work.
  - PaulDavisThe1st 6y ago
    You mean "naturally, since many of these codecs are completely or partially proprietary and/or not publically documented, this solution will not work on platforms that focus on non-proprietary open standards"?
    - pjmlp 6y ago
      Idealism has a price, but then there is the NIH syndrome on top of that.

kevin_thibedeau 6y ago

DRM probably.

Delk 6y ago
YouTube etc. don't have DRM in the video they serve, as far as I know. Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.

JoshTriplett 6y ago

Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU. Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video. Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".

lcnmrn 6y ago
I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.
- JoshTriplett 6y ago
  The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.
stefan_ 6y ago
All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff. Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.
- heftig 6y ago
  I think the expectation is that WebGL is used to manipulate video frames.
- vetinari 6y ago
  > Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.

InfiniteRand 6y ago

Maybe this is just unwarranted speculation, but I remember a period many years ago (maybe 10+?) when Firefox was very unstable when hardware acceleration was enabled, even on Windows, although it depended on what exactly your hardware was. Eventually the quality of the drivers on Windows made hardware acceleration less unstable on Windows, but that same progress did not happen on Linux (or was happening much slower) so for a while Firefox gave up on Linux hardware acceleration. Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly. Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.

kevinoid 6y ago

You can read about the specific reasons that driver versions were blocked on the Mozilla Wiki https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... Most are due to the drivers causing crashes or security issues in Firefox. I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.

TD-Linux 6y ago

VLC also has blocklists, e.g. https://code.videolan.org/videolan/vlc/-/blob/master/modules/codec/avcodec/dxva_blocklist.c https://code.videolan.org/videolan/vlc/-/blob/master/modules... https://code.videolan.org/videolan/vlc/-/blob/master/modules/video_output/opengl/interop_vaapi.c#L323 https://code.videolan.org/videolan/vlc/-/blob/master/modules... As does gstreamer (used by Totem): https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api-troubleshooting/ https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api...

Delk 6y ago
It still seems, at least based on anecdata and a hunch, that decode acceleration through VA-API works more often in VLC than in Firefox. VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever. (Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.) I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.

Const-me 6y ago

I think legacy contributed most. On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo https://github.com/Const-me/Vrmac/tree/master/VrmacVideo However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different. 1. They spent much time on that old code (sunk cost fallacy). 2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken). 3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.

Jasper_ 6y ago
> On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers). M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".
- vetinari 6y ago
  AMD uses VA-API nowadays.
  - selectodude 6y ago
    As does Nvidia.

ronjouch 6y ago

The problem is compositing with the rest of the web page. In VLC you don't have much complexity on top of or around the video (a few buttons, sometimes subtitles or a volume overlay), whereas around a video in a browser you have THE INTERWEBS (insert scream emoji here), which themselves also go through hardware acceleration. Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.

captainmuon 6y ago

They used to have playback via gstreamer (and before that mplayer plugin). First they castrated it for political reasons (you could only use certain codecs, officially so that codecs wouldn't proliferate and website authors wouldn't start to depend on them) and then they removed it altogether. Now it uses a bundled ffmpeg. I think this switch is one reason it took so long. Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.

pjmlp 6y ago
That is how it works on most desktop platforms with their video frameworks. Naturally Linux desktop being as it is, this solution will never work.
- PaulDavisThe1st 6y ago
  You mean "naturally, since many of these codecs are completely or partially proprietary and/or not publically documented, this solution will not work on platforms that focus on non-proprietary open standards"?
  - pjmlp 6y ago
    Idealism has a price, but then there is the NIH syndrome on top of that.