6 ms·

I've spent quite a lot of time working with the Web Audio API, and I strongly agree with the author. I got pretty deep into building a modular synthesis enviro

I've spent quite a lot of time working with the Web Audio API, and I strongly agree with the author. I got pretty deep into building a modular synthesis environment using it (https://github.com/rsimmons/plinth https://github.com/rsimmons/plinth) before deciding that working within the constraints of the built-in nodes was ultimately futile. Even building a well-behaved envelope generator (e.g. that handles retriggering correctly) is extremely tricky with what the API provides. How could such a basic use case have been overlooked? I made a library (https://github.com/rsimmons/fastidious-envelope-generator https://github.com/rsimmons/fastidious-envelope-generator) to solve that problem, but it's silly to have to work around the API for basic use cases. Ultimately we have to hold out for the AudioWorklet API (which itself seems potentially over-complicated) to finally get the ability to do "raw" output.

bsder 9y ago
The API is frustrating because it is meant to hide the fact that Android audio sucks giant hairy donkey balls. If you give Web developers access to raw samples, they are going to expect it to work. When it doesn't on Chrome on Android, lots of people are going to start complaining and filing bugs. So, instead of fixing the audio path, they decided to bury its crappiness under a "higher-level" API which has fuzzier latency and can be built with hacks in the audio driver stacks themselves.
- bzbarsky 9y ago
  That... explains why Google was so keen to kill off the Audio Data API Mozilla proposed, I guess.
  - roca 9y ago
    But Chrome for Android didn't come out until 2012 and Chris Rogers started the Web Audio work in 2009. I think someone would have had to have been exceptionally farsighted to think "Android's audio stack is going to suck for several years so we need to design around that now".
    - bsder 9y ago
      At that point in time, though, you could be forgiven with "Sheesh. Javascript is so painfully slow that nobody will ever pass PCM samples around in it." So, at every point in time up to and including now, you've always got something resisting low-latency PCM. Android is just the latest reason. Side note: it looks like Chris Rogers bowed out of Web Audio about 2012/2013 timeframe.
      - roca 9y ago
        You couldn't really, since AudioData worked pretty well. Also, it was obvious JS perf would get better and better.
    - bzbarsky 9y ago
      Ah, good point. I should have checked better on things before commenting!
- code_duck 9y ago
  Android audio is truly terrible for instrument apps. I don't understand how it suffices for things like games. I also don't understand why people even bother to make things like pianos and drum set… The latency is so extreme and inconsistent that even on recent phones they are useless. In contrast, iOS has had excellently playable instruments at least as far back as the iPod Touch 4.
  - dbrgn 9y ago
    Here's an interesting video on this topic back from 2013: https://youtube.com/watch?v=d3kfEeMZ65c https://youtube.com/watch?v=d3kfEeMZ65c
- baybal2 9y ago
  Aha, I had that eerie feeling that I saw those crappy patterns somewhere else. So, Android it was
- j_s 9y ago
  This has been a known issue with Android since at least 2009 (~2,700 stars); work done to address this is starting to trickle out this year. https://issuetracker.google.com/issues/36908622 https://issuetracker.google.com/issues/36908622 AAudio is a new C API. It is designed for high-performance audio applications that require low latency. It is currently in the Android O developer preview and not ready for production use. (Jun 2017)
  - sitkack 9y ago
    There are not a lot of artists at Google. Until this changes, the media apis will lag as G attempts to maintain parity with other orgs. Google makes product for Google devs and incidentally for the world to use.
metajack 9y ago
I think there were alternative designs but either no one cared or people were determined to push WebAudio despite faults. An example alternative that was proposed is http://robert.ocallahan.org/2012/01/mediastreams-processing-demos.html http://robert.ocallahan.org/2012/01/mediastreams-processing-...
akira2501 9y ago
> I strongly agree with the author. I will second this. I wanted to make a live streaming playback feature using the API so I could remotely monitor an audio matrix/routing system that I have in the office. The API has _zero_ provision for streaming MP3. You either load and playback a complete MP3 file or you get corrupted playback because the API simply won't maintain state between decoding calls. What I ended up having to do was write a port of libMAD to JavaScript and then use that to produce a PCM stream, which I _could_ then convert into an AudioBuffer, attach a timer, and then send into the audio API for correct playback. Which is an insane amount of work for a gaping oversight in a common use-case of the API, a simple flag in the browsers native decoder would've sufficed.
- eric_bullington 9y ago
  > The API has _zero_ provision for streaming MP3. Did you look into Media Source Extensions[0,1]? Fetching and playing the various audio formats is a bit outside the purview of Web Audio. But you can feed streaming MSE into Web Audio. If I recall, you use Web Audio's `AudioContext.createMediaElementSource()` to use a (potentially chunked) MSE source with web audio, but it's been a while since I did this. That said, Media Source Extensions (MSE) is only supported on relatively modern browsers (IE11+) but you should be able to use it to stream mp3 to the Web Audio API on supported browsers. There's also a way to do this without using MSE for older browsers. See the 72lions repo below for an example[2]. It's a bit convoluted, but not as much work as your workaround. As described in the README of the 72lions proof-of-concept: "The moment the first part is loaded then the playback starts immediately and it loads the second part. When the second part is loaded then then I create a new AudioBuffer by combining the old and the new, and I change the buffer of the AudioSourceNode with the new one. At that point I start playing again from the new AudioBuffer." 0. https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API https://developer.mozilla.org/en-US/docs/Web/API/Media_Sourc... 1. http://dalecurtis.github.io/llama-demo/index.html http://dalecurtis.github.io/llama-demo/index.html 2. https://github.com/72lions/PlayingChunkedMP3-WebAudioAPI https://github.com/72lions/PlayingChunkedMP3-WebAudioAPI
  - _jal 9y ago
netule 9y ago
Damn, I just played with the Plinth demo for an hour, lost total track of time. Great stuff.
- symstym 9y ago
  Thanks! After a certain point it felt like a dead end to me, so I dropped it in favor of exploring something more along the lines of a JS-based Max/MSP. But it is surprising how much fun can be had with the small number of modules available in Plinth.
throwa34943way 9y ago
(https://github.com/rsimmons/plinth https://github.com/rsimmons/plinth) This demo doesn't keep a straight 120 BPM on my machine, it's incapable of holding the rhythm after 10 seconds of playback ( I tried the first patch on the left , Edge browser).
- symstym 9y ago
  That's unfortunate. I don't have a machine running Edge to test it on. It uses less than 20% of the CPU in Chrome on my 2012 Macbook Air. It's possible that it's a problem with my code, but in general the Web Audio API does not have very good cross-browser support.
fenomas 9y ago
I tend to think of the Web Audio API as the answer to the question: "how much of an audio API can you have if you stipulate that all user-specified code must run in the UI thread?". Within that constraint I don't think it's a terrible API, but it's a big constraint and naturally raw access would be far preferable.
- symstym 9y ago
  Yes.. after I wrote my comment I was feeling a bit bad for sounding like I was just trashing the API. In a world where JS is slow and there is no worker thread machinery, yet you need low latency and flexible processing, the design makes more sense. That being said, the AudioParam "automation" methods still make me want to cry.
  - fenomas 9y ago
    Yeah, AudioParam's refusal to interpolate anything makes it really hairy to work with. Comments on the spec suggest that there was something really complicated about the "cancelAndHold" method (which I guess is still in NYI limbo), but I can't for the life of me figure out what it was.
- shermanyo 9y ago
  "how much of an audio API can you have if you stipulate that all user-specified code must run in the UI thread?" I just threw up in my mouth a little :/

bsder 9y ago

The API is frustrating because it is meant to hide the fact that Android audio sucks giant hairy donkey balls. If you give Web developers access to raw samples, they are going to expect it to work. When it doesn't on Chrome on Android, lots of people are going to start complaining and filing bugs. So, instead of fixing the audio path, they decided to bury its crappiness under a "higher-level" API which has fuzzier latency and can be built with hacks in the audio driver stacks themselves.

bzbarsky 9y ago
That... explains why Google was so keen to kill off the Audio Data API Mozilla proposed, I guess.
- roca 9y ago
  But Chrome for Android didn't come out until 2012 and Chris Rogers started the Web Audio work in 2009. I think someone would have had to have been exceptionally farsighted to think "Android's audio stack is going to suck for several years so we need to design around that now".
  - bsder 9y ago
    At that point in time, though, you could be forgiven with "Sheesh. Javascript is so painfully slow that nobody will ever pass PCM samples around in it." So, at every point in time up to and including now, you've always got something resisting low-latency PCM. Android is just the latest reason. Side note: it looks like Chris Rogers bowed out of Web Audio about 2012/2013 timeframe.
    - roca 9y ago
      You couldn't really, since AudioData worked pretty well. Also, it was obvious JS perf would get better and better.
  - bzbarsky 9y ago
    Ah, good point. I should have checked better on things before commenting!
code_duck 9y ago
Android audio is truly terrible for instrument apps. I don't understand how it suffices for things like games. I also don't understand why people even bother to make things like pianos and drum set… The latency is so extreme and inconsistent that even on recent phones they are useless. In contrast, iOS has had excellently playable instruments at least as far back as the iPod Touch 4.
- dbrgn 9y ago
  Here's an interesting video on this topic back from 2013: https://youtube.com/watch?v=d3kfEeMZ65c https://youtube.com/watch?v=d3kfEeMZ65c
baybal2 9y ago
Aha, I had that eerie feeling that I saw those crappy patterns somewhere else. So, Android it was
j_s 9y ago
This has been a known issue with Android since at least 2009 (~2,700 stars); work done to address this is starting to trickle out this year. https://issuetracker.google.com/issues/36908622 https://issuetracker.google.com/issues/36908622 AAudio is a new C API. It is designed for high-performance audio applications that require low latency. It is currently in the Android O developer preview and not ready for production use. (Jun 2017)
- sitkack 9y ago
  There are not a lot of artists at Google. Until this changes, the media apis will lag as G attempts to maintain parity with other orgs. Google makes product for Google devs and incidentally for the world to use.

metajack 9y ago

I think there were alternative designs but either no one cared or people were determined to push WebAudio despite faults. An example alternative that was proposed is http://robert.ocallahan.org/2012/01/mediastreams-processing-demos.html http://robert.ocallahan.org/2012/01/mediastreams-processing-...

akira2501 9y ago

> I strongly agree with the author. I will second this. I wanted to make a live streaming playback feature using the API so I could remotely monitor an audio matrix/routing system that I have in the office. The API has _zero_ provision for streaming MP3. You either load and playback a complete MP3 file or you get corrupted playback because the API simply won't maintain state between decoding calls. What I ended up having to do was write a port of libMAD to JavaScript and then use that to produce a PCM stream, which I _could_ then convert into an AudioBuffer, attach a timer, and then send into the audio API for correct playback. Which is an insane amount of work for a gaping oversight in a common use-case of the API, a simple flag in the browsers native decoder would've sufficed.

eric_bullington 9y ago
> The API has _zero_ provision for streaming MP3. Did you look into Media Source Extensions[0,1]? Fetching and playing the various audio formats is a bit outside the purview of Web Audio. But you can feed streaming MSE into Web Audio. If I recall, you use Web Audio's `AudioContext.createMediaElementSource()` to use a (potentially chunked) MSE source with web audio, but it's been a while since I did this. That said, Media Source Extensions (MSE) is only supported on relatively modern browsers (IE11+) but you should be able to use it to stream mp3 to the Web Audio API on supported browsers. There's also a way to do this without using MSE for older browsers. See the 72lions repo below for an example[2]. It's a bit convoluted, but not as much work as your workaround. As described in the README of the 72lions proof-of-concept: "The moment the first part is loaded then the playback starts immediately and it loads the second part. When the second part is loaded then then I create a new AudioBuffer by combining the old and the new, and I change the buffer of the AudioSourceNode with the new one. At that point I start playing again from the new AudioBuffer." 0. https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API https://developer.mozilla.org/en-US/docs/Web/API/Media_Sourc... 1. http://dalecurtis.github.io/llama-demo/index.html http://dalecurtis.github.io/llama-demo/index.html 2. https://github.com/72lions/PlayingChunkedMP3-WebAudioAPI https://github.com/72lions/PlayingChunkedMP3-WebAudioAPI
- _jal 9y ago

netule 9y ago

Damn, I just played with the Plinth demo for an hour, lost total track of time. Great stuff.

symstym 9y ago
Thanks! After a certain point it felt like a dead end to me, so I dropped it in favor of exploring something more along the lines of a JS-based Max/MSP. But it is surprising how much fun can be had with the small number of modules available in Plinth.

throwa34943way 9y ago

(https://github.com/rsimmons/plinth https://github.com/rsimmons/plinth) This demo doesn't keep a straight 120 BPM on my machine, it's incapable of holding the rhythm after 10 seconds of playback ( I tried the first patch on the left , Edge browser).

symstym 9y ago
That's unfortunate. I don't have a machine running Edge to test it on. It uses less than 20% of the CPU in Chrome on my 2012 Macbook Air. It's possible that it's a problem with my code, but in general the Web Audio API does not have very good cross-browser support.

fenomas 9y ago

I tend to think of the Web Audio API as the answer to the question: "how much of an audio API can you have if you stipulate that all user-specified code must run in the UI thread?". Within that constraint I don't think it's a terrible API, but it's a big constraint and naturally raw access would be far preferable.

symstym 9y ago
Yes.. after I wrote my comment I was feeling a bit bad for sounding like I was just trashing the API. In a world where JS is slow and there is no worker thread machinery, yet you need low latency and flexible processing, the design makes more sense. That being said, the AudioParam "automation" methods still make me want to cry.
- fenomas 9y ago
  Yeah, AudioParam's refusal to interpolate anything makes it really hairy to work with. Comments on the spec suggest that there was something really complicated about the "cancelAndHold" method (which I guess is still in NYI limbo), but I can't for the life of me figure out what it was.
shermanyo 9y ago
"how much of an audio API can you have if you stipulate that all user-specified code must run in the UI thread?" I just threw up in my mouth a little :/