5 ms·

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific request

Thanks for the nice write up! I work on the WebCodecs team at Chrome. I'm glad to hear it's mostly working for you. If you (or anyone else) has specific requests for new knobs regarding "We may need more encoding options, like non-reference frames or SVC", please file issues at https://github.com/w3c/webcodecs/issues https://github.com/w3c/webcodecs/issues

krebby 3y ago
Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672 https://github.com/w3c/webcodecs/issues/672 Thanks for the great work on WebCodecs!
vlovich123 3y ago
There’s a few that would be neat: * maybe possible already, but it’s not immediately clear how to change the bitrate of the encoder dynamically when doing VBR/CBR (seems like you can only do it with per-frame quantization params which isn’t very friendly) * being able to specify the reference frame to use for encoding p frames * being able to generate slices efficiently / display them easily. For example, Oculus Link encodes 1/n of the video in parallel encoders and decodes similarly. This way your encoding time only contributes 1/n frame encode/decode worth of latency because the rest is amortized with tx+decode of other slices. I suspect the biggest requirement here is to be able to cheaply and easily get N VideoFrames OR be able to cheaply split a VideoFrame into horizontal or vertical slices.
- DaleCurtis 3y ago
  * Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind? * Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285 https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here. * Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.
  - vlovich123 3y ago
    > what kind of scheme are you thinking beyond per frame QP Ideally I'd like to be able to set the CBR / VBR bitrate instead of some vague QP parameter that I manually have to profile to figure out how it corresponds to a bitrate for a given encoder. Of course, maybe encoders don't actually support this? I can't recall. It's been a while. > Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader. I don't think you need a shader. We did it at Oculus Link with existing HW encoders and it worked fine (at least for AMD and NVidia - not 100% sure about Intel's capabilities). It did require some bitmunging to muck with the NVidia H264 bitstream to make the parallel QCOM decoders happy with slices coming from a single encoder session* but it wasn't that significant a problem. For video streaming, supporting a standard for Webcams to be able to deliver slices with timestampped information about the rolling shutter (+ maybe IMU for mobile use cases) would help create a market for premium low-latency webcams. You'd need to figure out how to implement just in time rolling shutter corrections on the display side to mitigate the downsides of rolling shutter but the extra IMU information would be very useful (many mobile camera display packages support this functionality). VR displays often have rolling shutter so a rolling shutter webcam + display together would really make it possible to do "just in time" corrections for where pixels end up to adjust for latency. I'm not sure how much you'd get out of that, but my hunch is that if you knock out all the details you should be able to shave off nearly a frame of latency glass to glass. Speaking of adjustments, extracting motion vectors from the video is also useful, at least for VR, so that you can give the compositor the relevant information to apply last-minute corrections for that "locked to your motion" feeling (counteracts motion sickness). On a related note, with HW GPU encoders, it would be nice to have the webcam frame sent from the webcam directly to the GPU instead of round-tripping into a CPU buffer that you then either transport to the GPU or encode on the CPU - this should save a few ms of latency. Think NVidia's Direct standards but extended so that the GPU can grab the frame from the webcam, encode & maybe even send it out over Ethernet directly (the Ethernet part would be particularly valuable for tech like Stadia / GeForce now). I know the HW standards for that don't actually exist yet, but it might be interesting to explore with NVidia, AMD, and Intel what HW acceleration of that data path might look like. * NVidia's encoder supports slices directly and has an artificial limit on the number of encoder sessions on consumer drivers (they raised it in the past few years but IIRC it's still anemic). That however means that the generated slices have some incorrect parameters in the bitstream if you want to decode them independently. So you have to muck with the bitstream in a trivial way so that the decoders see independent valid H264 bitstreams they can decode. On AMD you don't have a limit to the number of encoder.
kixelated 3y ago
Thanks for WebCodecs! I'm still just trying to get A/V sync working properly because WebAudio makes things annoying. WebCodecs itself is great; I love the simplicity.
- padenot 3y ago
  https://blog.paul.cx/post/audio-video-synchronization-with-the-web-audio-api/ https://blog.paul.cx/post/audio-video-synchronization-with-t... has some background, https://github.com/w3c/webcodecs/blob/main/samples/lib/web_audio_controller.js https://github.com/w3c/webcodecs/blob/main/samples/lib/web_a... is part of a full example that you can run using web codecs, web audio, audioworklet SharedArrayBuffer, and does A/V sync. If it doesn't answer your question let me know because I wrote both (and part of the web audio spec, and part of the webcodecs spec).
  - kixelated 3y ago
    I'm using AudioWorklet and SharedArrayBuffer. Here's my code: https://github.com/kixelated/moq-js/tree/main/lib/playback/webcodecs https://github.com/kixelated/moq-js/tree/main/lib/playback/w... It's just a lot of work to get everything right. It's kind of working, but I removed synchronization because the signaling between the WebWorker and AudioWorklet got too convoluted. It all makes sense; I just wish there was an easier way to emit audio. While you're here, how difficult would it be to implement echo cancellation? The current demo is uni-directional but we'll need to make it bi-directional for conferencing.
    - padenot 3y ago
      Just use getUserMedia as usual and it will just work, nothing special to do.
hokkos 3y ago
I use the WebCodecs API with VideoDecoder with a very specific use case, to get data arrays using the great compression of video codec and the data having temporal coherency. Demo here : https://energygraph.info/d/f487b4fd-45ad-4f94-8e7e-ea32fc280adb/wind-and-solar-charge-factor-map?autofitpanels https://energygraph.info/d/f487b4fd-45ad-4f94-8e7e-ea32fc280... And I have some issues with the copyTo method of VideoFrame, on mobile (Pixel 7 Pro) it is unreliable and output all 0 Uint8Array beyond 20 frames, to the point I am forced to render each frame to an OffscreenCanvas. Also the many formats of frame output around RGBA/R8 with reduced range 16-235 or full range 0-255 makes it hard to use in my convoluted way.
- DaleCurtis 3y ago
  Please file an issue at https://crbug.com/new https://crbug.com/new with the details and we can take a look. Are you rendering frames in order? Android may have some quirks due to legacy MediaCodec restrictions around how we more commonly need frames for video elements, frames only work in sequential order since they must be released to an output texture to access them (and releasing invalidates prior frames to speed up very old MediaCodecs).
  - hokkos 3y ago
    It will try to do a simple reproduction, and yes the frame are decoded in order.
jampekka 3y ago
I'm currently working with WebCodecs to get (the long awaited) frame-by-frame seeking and reverse playback working in the browser. And it even seems to work, albeit the VideoDecoder queuing logic seems to give some grief for this. Any tips on figuring out how many chunks have to be queued for a specific VideoFrame to pop out? An aside: to work with video/container files, be sure to check the libav.js project that can be used to demux streams (WebCodecs don't do this) and even used as a polyfill decoder for browsers without WebCodec support! https://github.com/Yahweasel/libav.js/ https://github.com/Yahweasel/libav.js/
- DaleCurtis 3y ago
  The amount of frames necessary is going to depend on the codec and bitstream parameters. If it's H264 or H265, there's some more discussion and links here: https://github.com/w3c/webcodecs/issues/698#issuecomment-1614900631 https://github.com/w3c/webcodecs/issues/698#issuecomment-161... The optimizeForLatency parameter may also help in some cases: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder/configure#optimizeforlatency https://developer.mozilla.org/en-US/docs/Web/API/VideoDecode...
  - jampekka 3y ago
    Thanks. I appreciate that making an API that can be implemented with the wide variety of decoding implementations is not an easy task. But to be specific, this is a bit problematic with I-frames only videos too, and with optimizeForLatency enabled (that does make the queue shorter). I can of course .flush() to get the frames out but this is too slow for smooth playback. I think I could just keep pushing chunks until I see the frame I want coming out but it will have to be done in an async "busy loop" which feels a bit nasty. But this is done also in the "official" examples I think. Something like "enqueue" event (similarly to dequeue) that more chunks after last .decode() are needed to saturate the decoder would allow for a clean implementation. Don't know if this is possible with all backends though.
    - DaleCurtis 3y ago
      Often Chrome doesn't know when more frames are needed either, so it's not something we could add an API for unfortunately. Yes, just feeding inputs 1 by 1 for each dequeue event until you get the number of outputs you want in your steady state is the best way. It minimizes memory usage. I'll see about updating the MDN documentation to state this better.

krebby 3y ago

Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672 https://github.com/w3c/webcodecs/issues/672 Thanks for the great work on WebCodecs!

vlovich123 3y ago

There’s a few that would be neat: * maybe possible already, but it’s not immediately clear how to change the bitrate of the encoder dynamically when doing VBR/CBR (seems like you can only do it with per-frame quantization params which isn’t very friendly) * being able to specify the reference frame to use for encoding p frames * being able to generate slices efficiently / display them easily. For example, Oculus Link encodes 1/n of the video in parallel encoders and decodes similarly. This way your encoding time only contributes 1/n frame encode/decode worth of latency because the rest is amortized with tx+decode of other slices. I suspect the biggest requirement here is to be able to cheaply and easily get N VideoFrames OR be able to cheaply split a VideoFrame into horizontal or vertical slices.

DaleCurtis 3y ago
* Hmm, what kind of scheme are you thinking beyond per frame QP? Does an abstraction on top of QP work for the case you have in mind? * Reference frame control seems to be https://github.com/w3c/webcodecs/issues/285 https://github.com/w3c/webcodecs/issues/285, there's some interest in this for 2024, so I'd expect progress here. * Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader.
- vlovich123 3y ago
  > what kind of scheme are you thinking beyond per frame QP Ideally I'd like to be able to set the CBR / VBR bitrate instead of some vague QP parameter that I manually have to profile to figure out how it corresponds to a bitrate for a given encoder. Of course, maybe encoders don't actually support this? I can't recall. It's been a while. > Does splitting frames in WebGPU/WebGL work for the use case here? I'm not sure we could do anything internally (we're at the mercy of hardware decode implementations) without implementing such a shader. I don't think you need a shader. We did it at Oculus Link with existing HW encoders and it worked fine (at least for AMD and NVidia - not 100% sure about Intel's capabilities). It did require some bitmunging to muck with the NVidia H264 bitstream to make the parallel QCOM decoders happy with slices coming from a single encoder session* but it wasn't that significant a problem. For video streaming, supporting a standard for Webcams to be able to deliver slices with timestampped information about the rolling shutter (+ maybe IMU for mobile use cases) would help create a market for premium low-latency webcams. You'd need to figure out how to implement just in time rolling shutter corrections on the display side to mitigate the downsides of rolling shutter but the extra IMU information would be very useful (many mobile camera display packages support this functionality). VR displays often have rolling shutter so a rolling shutter webcam + display together would really make it possible to do "just in time" corrections for where pixels end up to adjust for latency. I'm not sure how much you'd get out of that, but my hunch is that if you knock out all the details you should be able to shave off nearly a frame of latency glass to glass. Speaking of adjustments, extracting motion vectors from the video is also useful, at least for VR, so that you can give the compositor the relevant information to apply last-minute corrections for that "locked to your motion" feeling (counteracts motion sickness). On a related note, with HW GPU encoders, it would be nice to have the webcam frame sent from the webcam directly to the GPU instead of round-tripping into a CPU buffer that you then either transport to the GPU or encode on the CPU - this should save a few ms of latency. Think NVidia's Direct standards but extended so that the GPU can grab the frame from the webcam, encode & maybe even send it out over Ethernet directly (the Ethernet part would be particularly valuable for tech like Stadia / GeForce now). I know the HW standards for that don't actually exist yet, but it might be interesting to explore with NVidia, AMD, and Intel what HW acceleration of that data path might look like. * NVidia's encoder supports slices directly and has an artificial limit on the number of encoder sessions on consumer drivers (they raised it in the past few years but IIRC it's still anemic). That however means that the generated slices have some incorrect parameters in the bitstream if you want to decode them independently. So you have to muck with the bitstream in a trivial way so that the decoders see independent valid H264 bitstreams they can decode. On AMD you don't have a limit to the number of encoder.

kixelated 3y ago

Thanks for WebCodecs! I'm still just trying to get A/V sync working properly because WebAudio makes things annoying. WebCodecs itself is great; I love the simplicity.

padenot 3y ago
https://blog.paul.cx/post/audio-video-synchronization-with-the-web-audio-api/ https://blog.paul.cx/post/audio-video-synchronization-with-t... has some background, https://github.com/w3c/webcodecs/blob/main/samples/lib/web_audio_controller.js https://github.com/w3c/webcodecs/blob/main/samples/lib/web_a... is part of a full example that you can run using web codecs, web audio, audioworklet SharedArrayBuffer, and does A/V sync. If it doesn't answer your question let me know because I wrote both (and part of the web audio spec, and part of the webcodecs spec).
- kixelated 3y ago
  I'm using AudioWorklet and SharedArrayBuffer. Here's my code: https://github.com/kixelated/moq-js/tree/main/lib/playback/webcodecs https://github.com/kixelated/moq-js/tree/main/lib/playback/w... It's just a lot of work to get everything right. It's kind of working, but I removed synchronization because the signaling between the WebWorker and AudioWorklet got too convoluted. It all makes sense; I just wish there was an easier way to emit audio. While you're here, how difficult would it be to implement echo cancellation? The current demo is uni-directional but we'll need to make it bi-directional for conferencing.
  - padenot 3y ago
    Just use getUserMedia as usual and it will just work, nothing special to do.

hokkos 3y ago

I use the WebCodecs API with VideoDecoder with a very specific use case, to get data arrays using the great compression of video codec and the data having temporal coherency. Demo here : https://energygraph.info/d/f487b4fd-45ad-4f94-8e7e-ea32fc280adb/wind-and-solar-charge-factor-map?autofitpanels https://energygraph.info/d/f487b4fd-45ad-4f94-8e7e-ea32fc280... And I have some issues with the copyTo method of VideoFrame, on mobile (Pixel 7 Pro) it is unreliable and output all 0 Uint8Array beyond 20 frames, to the point I am forced to render each frame to an OffscreenCanvas. Also the many formats of frame output around RGBA/R8 with reduced range 16-235 or full range 0-255 makes it hard to use in my convoluted way.

DaleCurtis 3y ago
Please file an issue at https://crbug.com/new https://crbug.com/new with the details and we can take a look. Are you rendering frames in order? Android may have some quirks due to legacy MediaCodec restrictions around how we more commonly need frames for video elements, frames only work in sequential order since they must be released to an output texture to access them (and releasing invalidates prior frames to speed up very old MediaCodecs).
- hokkos 3y ago
  It will try to do a simple reproduction, and yes the frame are decoded in order.

jampekka 3y ago

I'm currently working with WebCodecs to get (the long awaited) frame-by-frame seeking and reverse playback working in the browser. And it even seems to work, albeit the VideoDecoder queuing logic seems to give some grief for this. Any tips on figuring out how many chunks have to be queued for a specific VideoFrame to pop out? An aside: to work with video/container files, be sure to check the libav.js project that can be used to demux streams (WebCodecs don't do this) and even used as a polyfill decoder for browsers without WebCodec support! https://github.com/Yahweasel/libav.js/ https://github.com/Yahweasel/libav.js/

DaleCurtis 3y ago
The amount of frames necessary is going to depend on the codec and bitstream parameters. If it's H264 or H265, there's some more discussion and links here: https://github.com/w3c/webcodecs/issues/698#issuecomment-1614900631 https://github.com/w3c/webcodecs/issues/698#issuecomment-161... The optimizeForLatency parameter may also help in some cases: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder/configure#optimizeforlatency https://developer.mozilla.org/en-US/docs/Web/API/VideoDecode...
- jampekka 3y ago
  Thanks. I appreciate that making an API that can be implemented with the wide variety of decoding implementations is not an easy task. But to be specific, this is a bit problematic with I-frames only videos too, and with optimizeForLatency enabled (that does make the queue shorter). I can of course .flush() to get the frames out but this is too slow for smooth playback. I think I could just keep pushing chunks until I see the frame I want coming out but it will have to be done in an async "busy loop" which feels a bit nasty. But this is done also in the "official" examples I think. Something like "enqueue" event (similarly to dequeue) that more chunks after last .decode() are needed to saturate the decoder would allow for a clean implementation. Don't know if this is possible with all backends though.
  - DaleCurtis 3y ago
    Often Chrome doesn't know when more frames are needed either, so it's not something we could add an API for unfortunately. Yes, just feeding inputs 1 by 1 for each dequeue event until you get the number of outputs you want in your steady state is the best way. It minimizes memory usage. I'll see about updating the MDN documentation to state this better.