4 ms·

Very cool. I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. Imagine being able to turn on a brand-new cod

Very cool. I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. Imagine being able to turn on a brand-new codec (or select a customized codec for your particular use) in any browser that supports WebAssembly. Two ends of a WebRTC connection could even negotiate codecs based on power/processing needs, with one sending a WebAssembly codec to the other as a fallback if needed ("hey you don't support this thing that I support in hardware, but you are plugged in, so burn some more electrons on your end"). WebAssembly SIMD [1] will definitely help with performance here. [1] https://www.chromestatus.com/feature/6533147810332672 https://www.chromestatus.com/feature/6533147810332672

markdog12 9y ago
Surprised to see the decoder was not a WebAssembly module.
- mmastrac 9y ago
  I would assume (completely uninformed assumption, however) that WebAssembly without SIMD won't be performant enough to decode this in realtime.
  - markdog12 9y ago
    JavaScript doesn't have SIMD
    - microcolonel 9y ago
      JavaScript was once to have SIMD. ;- ) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SIMD https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
      - markdog12 9y ago
        Yep, but now just WebAssembly
        microcolonel 9y ago
        They're kinda unrelated. I don't think WebAssembly has explicit SIMD operations yet? Then again, maybe WebAssembly could be delivered in a form which makes autovectorization straightforward for some subset of use cases.
        Jasper_ 9y ago
        There's a proposal for SIMD but it's not in any shipping browsers AFAIK. https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md https://github.com/WebAssembly/simd/blob/master/proposals/si...
  - TD-Linux 9y ago
    We already have an enscripten'd decoder, built into a nice analyzer tool that lets you inspect the AV1 bitstream! But you're right, it'll need SIMD to decode at high resolutions in realtime. [1] https://0x0.st/tQ https://0x0.st/tQ
- deleted 9y ago
  [deleted]
mooman219 9y ago
Although there is much to be desired still on the software end, the molasses of the situation is hardware support for the new formats. You mentioned it briefly, but there's only so much you can do with a software decoders. H.264 may not be ideal or technically have the best performance, but most video cards and cpu's have dedicated hardware decoders. Completely unrealistic, but it would be interesting to have on the fly reprogramable FPGAs on die that can be assigned specific tasks on demand. This of course has tons of risks, but the pace that hardware moves is painful.
- StudentStuff 9y ago
  H.265 support is a patchwork in hardware though, look at all the recent Allwinner chips, only 8 bit H.265 playback is supported, despite quite a few H.265 files using 10 bit colors. Worse yet, most Chromecasts and TVs don't support anything newer than H.264. It will be years before H.265 or any new competitor sees hardware support in most homes.
  - DiabloD3 9y ago
    H264 was the same way early on, with several chips doing baseline or main profile, but not high. It's why I said (at the time) that H264 HW support wasn't important... being able to do Bluray profile (a specific variant of high profile) was what was actually important. Now with H265, Bluray 4k profile is important, ergo 10 bit support is mandatory, as well as display output itself being able to handle signalling for Rec2020 HDR10 over HDMI 2.x (or equivalent DP).
  - j1elo 9y ago
    Probably due to H.265 having much more expensive licensing costs than H.264, given enough volume. That alone might make lots of players in the industry to just prefer ignoring the new codec (and joining a joint effort for developing a new codec with more favorable licensing terms...)
    - StudentStuff 9y ago
      Yeah, VP9 will probably monopolize the market as the libre, royalty free codec. With Google and others boosting it, I'd be surprised if it doesn't win like Opus has in the voice codec arena for any new application.
vthriller 9y ago
> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. Oh great, one more thing to help ad networks to crank CPU usage of my browser to the extremes. Hope I'll be able to also disable that through about:config. :/
agnivade 9y ago
> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. I have a n00b question though. How would I display the video in the browser using webassembly ? I mean using webassembly, I can do the decoding of the bits in the stream. But how to display those bits in the browser ? Do I use a canvas and directly dump the pixel buffer there ?

markdog12 9y ago

Surprised to see the decoder was not a WebAssembly module.

mmastrac 9y ago
I would assume (completely uninformed assumption, however) that WebAssembly without SIMD won't be performant enough to decode this in realtime.
- markdog12 9y ago
  JavaScript doesn't have SIMD
  - microcolonel 9y ago
    JavaScript was once to have SIMD. ;- ) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SIMD https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
    - markdog12 9y ago
      Yep, but now just WebAssembly
      - microcolonel 9y ago
        They're kinda unrelated. I don't think WebAssembly has explicit SIMD operations yet? Then again, maybe WebAssembly could be delivered in a form which makes autovectorization straightforward for some subset of use cases.
        Jasper_ 9y ago
        There's a proposal for SIMD but it's not in any shipping browsers AFAIK. https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md https://github.com/WebAssembly/simd/blob/master/proposals/si...
- TD-Linux 9y ago
  We already have an enscripten'd decoder, built into a nice analyzer tool that lets you inspect the AV1 bitstream! But you're right, it'll need SIMD to decode at high resolutions in realtime. [1] https://0x0.st/tQ https://0x0.st/tQ
deleted 9y ago
[deleted]

mooman219 9y ago

Although there is much to be desired still on the software end, the molasses of the situation is hardware support for the new formats. You mentioned it briefly, but there's only so much you can do with a software decoders. H.264 may not be ideal or technically have the best performance, but most video cards and cpu's have dedicated hardware decoders. Completely unrealistic, but it would be interesting to have on the fly reprogramable FPGAs on die that can be assigned specific tasks on demand. This of course has tons of risks, but the pace that hardware moves is painful.

StudentStuff 9y ago
H.265 support is a patchwork in hardware though, look at all the recent Allwinner chips, only 8 bit H.265 playback is supported, despite quite a few H.265 files using 10 bit colors. Worse yet, most Chromecasts and TVs don't support anything newer than H.264. It will be years before H.265 or any new competitor sees hardware support in most homes.
- DiabloD3 9y ago
  H264 was the same way early on, with several chips doing baseline or main profile, but not high. It's why I said (at the time) that H264 HW support wasn't important... being able to do Bluray profile (a specific variant of high profile) was what was actually important. Now with H265, Bluray 4k profile is important, ergo 10 bit support is mandatory, as well as display output itself being able to handle signalling for Rec2020 HDR10 over HDMI 2.x (or equivalent DP).
- j1elo 9y ago
  Probably due to H.265 having much more expensive licensing costs than H.264, given enough volume. That alone might make lots of players in the industry to just prefer ignoring the new codec (and joining a joint effort for developing a new codec with more favorable licensing terms...)
  - StudentStuff 9y ago
    Yeah, VP9 will probably monopolize the market as the libre, royalty free codec. With Google and others boosting it, I'd be surprised if it doesn't win like Opus has in the voice codec arena for any new application.

vthriller 9y ago

> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. Oh great, one more thing to help ad networks to crank CPU usage of my browser to the extremes. Hope I'll be able to also disable that through about:config. :/

agnivade 9y ago

> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. I have a n00b question though. How would I display the video in the browser using webassembly ? I mean using webassembly, I can do the decoding of the bits in the stream. But how to display those bits in the browser ? Do I use a canvas and directly dump the pixel buffer there ?