3 ms·

All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS

All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway. As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff. Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.

heftig 6y ago
I think the expectation is that WebGL is used to manipulate video frames.
vetinari 6y ago
> Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.

heftig 6y ago

I think the expectation is that WebGL is used to manipulate video frames.

vetinari 6y ago

> Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder. > So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.