5 ms·

Web Assembly audio decoders highly optimized for size and performance

145 points by ctoth 2y ago

wg0 2y ago
Interesting. What would be the use case for these? Audio editors?
- duped 2y ago
  Browser support of some codecs is inconsistent, for example Safari's support for opus has been pretty bad/nonexistent (iirc this recently changed)
  - pier25 2y ago
    This. Even FLAC support isn't great.
  - cupofjoakim 2y ago
    I used to work at a place that developed a DAW in the browser, and yeah, we used a WASM Ogg Opus decoder to save on storage costs for our sound library. We earlier had to have mp3s for safari devices. With opus being opus we really wanted to provide it for the majority of devices as well as our server side streaming, and being able to provide opus support for safari ourselves was a godsend. Edit: Should note that our decoder was developed in house and I have nothing to do with the project OP linked to.
    - duped 2y ago
      bandlab?
  - lifthrasiir 2y ago
    Is that true even for mp3 though? Apparently it actually has a universal browser support up to MSIE 9 (!), while Ogg Vorbis and FLAC are now supported by all modern browsers after MSIE [1]. Opus seems to be the earliest format with less support than that, while this library only dates back to 2020. [1] https://caniuse.com/?search=audio%20format https://caniuse.com/?search=audio%20format
    - duped 2y ago
      Sure, but what if you don't want to use mp3... codecs aren't created equal. For anything latency sensitive mp3 is a non starter. For anything under bandwidth constraints, flac isn't great. OPUS is a descendant of CELT (which never caught on, for reasons) which is an open/unhindered by patents low latency lossy codec suitable for VoIP and live streaming. It's relatively recent, but also, those use cases are more popular recently in the browser. And to go even more extreme, some applications really care about bit-exactness in their decoders, which you don't get by relying on other vendors.
      - lifthrasiir 2y ago
        I mean, WebAssembly Opus decoder seems meaningful even to this day, but I wondered why earlier formats are there in the first place. The bit-exactness is a good argument but as you've said that's a very specific requirement, and the library only seems to care about performance and minified size anyway.
        ctoth 2y ago
        Being able to decode audio in a Web Worker rather than the main UI thread is also a nice use case.
  - AshleysBrain 2y ago
    WebM Opus is an open format that plays in all modern browsers now - including recent Safari releases. There are still some bugs though last I checked.
- nsteel 2y ago
  They look great for https://github.com/badaix/snapweb https://github.com/badaix/snapweb which is missing decoders for some formats.
- austin-cheney 2y ago
  So, the JavaScript audio/video APIs already available in the browser are fully competent to build any kind of media player. The major limitation is that media players reliant on the browser APIs are limited to the codecs and containers supported by the browsers, which turns out to be a huge limitation. A media library written for WASM with supplied codecs would not have that limitation, such that you could have an equivalent to VLC directly in a browser supporting immediate playback of any video/audio codec/container supplied in the WASM payload. The major advantage of a browser based application versus a desktop app is immediate playback of media via hyperlink without the soul crushing limitations Apple imposes on iOS.
  - gp 2y ago
    Do you have a recommendation for having frame by frame control of a file? I am trying to synchronize multiple video files that have uneven framerates. I have separate metadata containing the frame timing. The video files are .ts (mpeg stream) files.
    - austin-cheney 2y ago
      Here is the media player I wrote in JavaScript: https://github.com/prettydiff/mp3-master-list/blob/master/lib/browser.ts https://github.com/prettydiff/mp3-master-list/blob/master/li... I suspect there exist filters in the browser APIs to adjust play back timing and rate. I have not attempted to modify the media in that way myself but I did see API filters to alter stero panning, tonal quality, wave shape, biquad, delay, and more.
  - 1oooqooq 2y ago
    what soul crushing limitations? only time i had to resort to those canvas/wasm hacks was to force autoplay video for advertisement, against all users best interest.
    - austin-cheney 2y ago
      iOS imposes substantial limitations to compel app purchases through its app store or through its iTunes stores. I just wanted to listen to music I already have played from my phone while I do dishes. In my case I just wanted to have a media player that could play MP3s and feature a playlist dynamically populated from either the local device file system or, more ideally, from a shared network location. Restricted. There were some media apps that kind of, in a really shitty way, figured this out. Most of those apps, even when shitty, cost money from the app store. Fuck that stupidity. I wrote my own solution in JavaScript that executes in the browser and it does almost everything I want. The playlist data must be statically written into the browser page and cannot be dynamically populated from any file system because Apple disables that part of the browser's FileSystem API in browsers on iOS, though it executes correctly on desktop Safari. I then host the page from a shared network location so that I can access the same content in the same way on any device with access to the network. I adapted this home grown media player to execute video as well, but it can only play videos from codecs/containers already supported in the browser.
xfalcox 2y ago
I wish the author would do the same for image decoding. iOS inability to draw a canvas with 4096x4097 dimensions means you can't use an Off screen canvas to go from file to RGBA array.
- deleted 2y ago
  [deleted]
- lioeters 2y ago
  libav.js is in the direction you're thinking. > compilation of the libraries associated with handling audio and video in FFmpeg — libavformat, libavcodec, libavfilter, libavutil and libswresample — for WebAssembly https://github.com/Yahweasel/libav.js/ https://github.com/Yahweasel/libav.js/ EDIT: Looking closer at the above I don't see image formats, only audio and video. Maybe this gets closer: > Decodes images using Rust's image crate compiled to WebAssembly. > It supports decoding PNG, JPEG, WEBP, GIF, BMP, ICO, TGA, and several others. https://github.com/josephrocca/wasm-image-decoder https://github.com/josephrocca/wasm-image-decoder
nilslice 2y ago
awesome! would love to help get distributions of these as Extism wasm plugins so they can be used in Go/Rust/PHP/Ruby/OCaml/Java/C++/C#/bunch more languages...
jauntywundrkind 2y ago
Probably a good basis for a shoutcast client! So little info on it & I really doubt it's supported in any browsers. Even though it's mp3 with a little metadata sprinkled in every X bytes. https://www.radiotoolbox.com/community/forums/viewtopic.php?t=74 https://www.radiotoolbox.com/community/forums/viewtopic.php?...
- doublerebel 2y ago
  There are lots of docs and info available for shoutcast/icecast servers and clients. Here’s a couple implementations in javascript: https://github.com/livebassmusicrightnow/even-nicercast/blob/master/src/even-nicercast.coffee https://github.com/livebassmusicrightnow/even-nicercast/blob... https://github.com/TooTallNate/node-icy https://github.com/TooTallNate/node-icy WASM would be total overkill. The mpeg stream is readable natively in the browser — that’s how I served the client when I was running livebassmusicrightnow.
- eshaz 2y ago
  Hey, author of the wasm-audio-decoders library here. Actually, I wrote this library for this very reason to supplement my other library, icecast-metadata-player which is an Icecast (Shoutcast) browser client with ICY metadata support. https://github.com/eshaz/icecast-metadata-js https://github.com/eshaz/icecast-metadata-js
hoten 2y ago
If I'm already compiling ogg vorbis/opus/minimp3/etc via emscripten (allegro app), is there still big benefit to using these instead?
beepbooptheory 2y ago
If they are all web workers, how do you use them in an Audio Worklet?
muth02446 2y ago
This scares me a little: There was a security alert on the page with the C-code from mp3.

145 points by ctoth 2y ago

wg0 2y ago

Interesting. What would be the use case for these? Audio editors?

duped 2y ago
Browser support of some codecs is inconsistent, for example Safari's support for opus has been pretty bad/nonexistent (iirc this recently changed)
- pier25 2y ago
  This. Even FLAC support isn't great.
- cupofjoakim 2y ago
  I used to work at a place that developed a DAW in the browser, and yeah, we used a WASM Ogg Opus decoder to save on storage costs for our sound library. We earlier had to have mp3s for safari devices. With opus being opus we really wanted to provide it for the majority of devices as well as our server side streaming, and being able to provide opus support for safari ourselves was a godsend. Edit: Should note that our decoder was developed in house and I have nothing to do with the project OP linked to.
  - duped 2y ago
    bandlab?
- lifthrasiir 2y ago
  Is that true even for mp3 though? Apparently it actually has a universal browser support up to MSIE 9 (!), while Ogg Vorbis and FLAC are now supported by all modern browsers after MSIE [1]. Opus seems to be the earliest format with less support than that, while this library only dates back to 2020. [1] https://caniuse.com/?search=audio%20format https://caniuse.com/?search=audio%20format
  - duped 2y ago
    Sure, but what if you don't want to use mp3... codecs aren't created equal. For anything latency sensitive mp3 is a non starter. For anything under bandwidth constraints, flac isn't great. OPUS is a descendant of CELT (which never caught on, for reasons) which is an open/unhindered by patents low latency lossy codec suitable for VoIP and live streaming. It's relatively recent, but also, those use cases are more popular recently in the browser. And to go even more extreme, some applications really care about bit-exactness in their decoders, which you don't get by relying on other vendors.
    - lifthrasiir 2y ago
      I mean, WebAssembly Opus decoder seems meaningful even to this day, but I wondered why earlier formats are there in the first place. The bit-exactness is a good argument but as you've said that's a very specific requirement, and the library only seems to care about performance and minified size anyway.
      - ctoth 2y ago
        Being able to decode audio in a Web Worker rather than the main UI thread is also a nice use case.
- AshleysBrain 2y ago
  WebM Opus is an open format that plays in all modern browsers now - including recent Safari releases. There are still some bugs though last I checked.
nsteel 2y ago
They look great for https://github.com/badaix/snapweb https://github.com/badaix/snapweb which is missing decoders for some formats.
austin-cheney 2y ago
So, the JavaScript audio/video APIs already available in the browser are fully competent to build any kind of media player. The major limitation is that media players reliant on the browser APIs are limited to the codecs and containers supported by the browsers, which turns out to be a huge limitation. A media library written for WASM with supplied codecs would not have that limitation, such that you could have an equivalent to VLC directly in a browser supporting immediate playback of any video/audio codec/container supplied in the WASM payload. The major advantage of a browser based application versus a desktop app is immediate playback of media via hyperlink without the soul crushing limitations Apple imposes on iOS.
- gp 2y ago
  Do you have a recommendation for having frame by frame control of a file? I am trying to synchronize multiple video files that have uneven framerates. I have separate metadata containing the frame timing. The video files are .ts (mpeg stream) files.
  - austin-cheney 2y ago
    Here is the media player I wrote in JavaScript: https://github.com/prettydiff/mp3-master-list/blob/master/lib/browser.ts https://github.com/prettydiff/mp3-master-list/blob/master/li... I suspect there exist filters in the browser APIs to adjust play back timing and rate. I have not attempted to modify the media in that way myself but I did see API filters to alter stero panning, tonal quality, wave shape, biquad, delay, and more.
- 1oooqooq 2y ago
  what soul crushing limitations? only time i had to resort to those canvas/wasm hacks was to force autoplay video for advertisement, against all users best interest.
  - austin-cheney 2y ago
    iOS imposes substantial limitations to compel app purchases through its app store or through its iTunes stores. I just wanted to listen to music I already have played from my phone while I do dishes. In my case I just wanted to have a media player that could play MP3s and feature a playlist dynamically populated from either the local device file system or, more ideally, from a shared network location. Restricted. There were some media apps that kind of, in a really shitty way, figured this out. Most of those apps, even when shitty, cost money from the app store. Fuck that stupidity. I wrote my own solution in JavaScript that executes in the browser and it does almost everything I want. The playlist data must be statically written into the browser page and cannot be dynamically populated from any file system because Apple disables that part of the browser's FileSystem API in browsers on iOS, though it executes correctly on desktop Safari. I then host the page from a shared network location so that I can access the same content in the same way on any device with access to the network. I adapted this home grown media player to execute video as well, but it can only play videos from codecs/containers already supported in the browser.

xfalcox 2y ago

I wish the author would do the same for image decoding. iOS inability to draw a canvas with 4096x4097 dimensions means you can't use an Off screen canvas to go from file to RGBA array.

deleted 2y ago
[deleted]
lioeters 2y ago
libav.js is in the direction you're thinking. > compilation of the libraries associated with handling audio and video in FFmpeg — libavformat, libavcodec, libavfilter, libavutil and libswresample — for WebAssembly https://github.com/Yahweasel/libav.js/ https://github.com/Yahweasel/libav.js/ EDIT: Looking closer at the above I don't see image formats, only audio and video. Maybe this gets closer: > Decodes images using Rust's image crate compiled to WebAssembly. > It supports decoding PNG, JPEG, WEBP, GIF, BMP, ICO, TGA, and several others. https://github.com/josephrocca/wasm-image-decoder https://github.com/josephrocca/wasm-image-decoder

nilslice 2y ago

awesome! would love to help get distributions of these as Extism wasm plugins so they can be used in Go/Rust/PHP/Ruby/OCaml/Java/C++/C#/bunch more languages...

jauntywundrkind 2y ago

Probably a good basis for a shoutcast client! So little info on it & I really doubt it's supported in any browsers. Even though it's mp3 with a little metadata sprinkled in every X bytes. https://www.radiotoolbox.com/community/forums/viewtopic.php?t=74 https://www.radiotoolbox.com/community/forums/viewtopic.php?...

doublerebel 2y ago
There are lots of docs and info available for shoutcast/icecast servers and clients. Here’s a couple implementations in javascript: https://github.com/livebassmusicrightnow/even-nicercast/blob/master/src/even-nicercast.coffee https://github.com/livebassmusicrightnow/even-nicercast/blob... https://github.com/TooTallNate/node-icy https://github.com/TooTallNate/node-icy WASM would be total overkill. The mpeg stream is readable natively in the browser — that’s how I served the client when I was running livebassmusicrightnow.
eshaz 2y ago
Hey, author of the wasm-audio-decoders library here. Actually, I wrote this library for this very reason to supplement my other library, icecast-metadata-player which is an Icecast (Shoutcast) browser client with ICY metadata support. https://github.com/eshaz/icecast-metadata-js https://github.com/eshaz/icecast-metadata-js

hoten 2y ago

If I'm already compiling ogg vorbis/opus/minimp3/etc via emscripten (allegro app), is there still big benefit to using these instead?

beepbooptheory 2y ago

If they are all web workers, how do you use them in an Audio Worklet?

muth02446 2y ago

This scares me a little: There was a security alert on the page with the C-code from mp3.