7 ms·

Building a musical instrument with the Web Audio API

86 points by beardicus 4y ago

gnozzle 4y ago
Diatonic button accordion (or melodeon as we call it here in the UK) player here. I wasn't expecting to see my, rather niche, instrument on the front page of HN! Neat app for experimentation. The bellows control is always the sticking point on electronic DBAs but this works well enough to play a simple tune. I don't think I could do tune, bellows and bass at once on this though! Nice write-up too.
chaosprint 4y ago
very very cool. the visual aspect is very impressing. I also use Svelte for this browser-based music live coding environment I am developing: https://glicol.org https://glicol.org The issue with raw Web Audio API is that when there are some heavy stuff on the main thread (in your case the visual feedback), the audio may get glitches. WASM + AudioWorklet is SOTA the solution for this, and Glicol 's tech stack is Rust->WASM->AudioWorklet + SharedArrayBuffer. I also porting the audio engine of Glicol as an NPM package: https://glicol.js.org https://glicol.js.org So if you wish some better audio performance for future development, perhaps you can take a look on that. It provides better audio performance and friendly APIs. I would be happy to update the audio lib based on your feedback.
- fenomas 4y ago
  > when there are some heavy stuff on the main thread the audio may get glitches Where have you seen that? TFA is just using oscillators, not ScriptProcessor or anything, so one would expect all the heavy audio work to be happening outside the main thread. Personally I use webaudio to do dynamic music for a game, and even though the game is reasonably heavy (3D, physics, etc) I've not noticed any particular glitching.
  - chaosprint 4y ago
    You are right! I should say ScriptProcessor there. But for heavy use of audio, especially when there are lots of interactions (often you want your own abstraction rather than the built-in nodes) and sample-level time should be taken into account, WASM+AudioWorklet should definitely be the first choice: https://youtu.be/GfC2WTStW8M?t=1084 https://youtu.be/GfC2WTStW8M?t=1084
ArtWomb 4y ago
This is a terrific Web Audio tutorial! Real-time sound synth in the browser is here. There are already a few DAWs like GridSound out there. But it's still the Wild West ;) Jim Clark's Synth Programming book from 2003: https://www.cim.mcgill.ca/~clark/nordmodularbook/nm_book_toc.html https://www.cim.mcgill.ca/~clark/nordmodularbook/nm_book_toc... Code: https://github.com/aolney/nord-modular-book https://github.com/aolney/nord-modular-book
- chaosprint 4y ago
  A new video today on Web Audio (Creating Music on the Web - Ben Morss and Hongchan Choi - ADC21): https://youtu.be/GfC2WTStW8M https://youtu.be/GfC2WTStW8M
- deleted 4y ago
  [deleted]
amelius 4y ago
What's the latency?
bambax 4y ago
> Sine and triangle were a bit more subdued but if you don't fade the sound out properly, it makes a really unpleasant kind of cutting sound due to how your ear reacts when a wave gets cut off. The clicking sound happens when a wave is stopped when not at 0. You do need to fade the oscillator down before stopping it. > As I was figuring everything out, I was experimenting with making new AudioContext for every note because I was trying to fade out the sound, but then I kept realizing that after 50 notes, the app would stop working. Fifty is apparently the limit for the browser, so it's better to just make one AudioContext for the entire app. Yeah the reason the app would stop working when doing things this way is, just because the gain of the oscillator is zero, and the oscillator therefore not audible, it's still there! If you put the gain back up you can hear it again. You can't just fade out and be done with it. So the appropriate approach (not explicitly explained in the article IMHO) is to 1/ set the gain to zero: g.gain.setTargetAtTime(0, context.currentTime, 0.1) 2/ then kill the oscillator with stop(), so that it doesn't consume resources anymore: o.stop(context.currentTime + 0.5) (g being the gain and o the oscillator, and context the AudioContext). Note: "then" means the two actions need to happen one after the other, and that is decided by parameters of the two functions; the functions themselves aren't chained and can be written in any order.
cupofjoakim 4y ago
Nice job! The choice of accordion def wanted me to play by resizing the window though :)
PaulDavisThe1st 4y ago
On the one hand: really great write up of a great little project that is great at helping people experiment with synthesis and software instrument design. On the other hand: another example of a development platform ("web browsers") being utilized for something it is fundamentally not designed for, just because it leverages the skills someone gained along the way. I mostly wish this would stop. People are going to build all kinds of awesome toys using web audio APIs, and every single one of those toys is going to be less performant (latency, CPU/DSP load, interface responsiveness, sensor interactivity) than it's native equivalent. That means that its use will face limitations that are a function of the development platform rather than the designer or performer. On the other hand, how can anyone be against people learning more about synthesis and instrument design? My brain hurts.
- chaosprint 4y ago
  Good point. I think you do accept the convenience of browsers (many cool projects such as this one https://learningsynths.ableton.com/ https://learningsynths.ableton.com/). So now the problem is the audio performance. As I post in the comments below, we now have WASM, so C++ and Rust can all run in browsers. This can provide a near-native audio performance. Just take Glicol, the live coding language I design as an example: it runs in browsers (https://glicol.org https://glicol.org) and it also runs as a VST plugin (https://youtu.be/tmmBhBmIEW0 https://youtu.be/tmmBhBmIEW0), or you can use the audio engine to write VST plugin (https://github.com/chaosprint/dattorro-vst-rs https://github.com/chaosprint/dattorro-vst-rs).
  - spacechild1 4y ago
    Yep. Pure Data and SuperCollider can also run in the browser. But there's a difference between C/C++/Rust audio apps compiled to WASM on the one hand and writing audio code in JS on the other hand. Then again, I think it's perfectly ok for people to experiment with audio in browser. "Native" audio desktop applications will always be superior, so there's nothing to be afraid of.
  - PaulDavisThe1st 4y ago
    > As I post in the comments below, we now have WASM, so C++ and Rust can all run in browsers. This can provide a near-native audio performance. Native audio apps require realtime scheduling and memory locking, things you cannot do/control in the browser. The problem is that you can likely get 70-85% of "it" done in the browser, but when the user/performer need the remaining 15-30% for whatever reason, what do they do then?
stevehiehn 4y ago
Great post! I'm very enthusiastic about the WebAudioApi. I'm currently building an audio sample pipeline and enabling users to preview combinations of loops via the api! https://signalsandsorcery.org https://signalsandsorcery.org
jdauriemma 4y ago
If anyone's interested in a Web Audio API trombone, I made one a while back: https://jdauriemma.com/trombone.js/ https://jdauriemma.com/trombone.js/ https://github.com/bignimbus/trombone.js https://github.com/bignimbus/trombone.js
skybrian 4y ago
Headline is misleading - it leaves out that the instrument is an accordion. Talk about burying the lede.
aaaaaaaaata 4y ago
SO cool.

86 points by beardicus 4y ago

Diatonic button accordion (or melodeon as we call it here in the UK) player here. I wasn't expecting to see my, rather niche, instrument on the front page of HN! Neat app for experimentation. The bellows control is always the sticking point on electronic DBAs but this works well enough to play a simple tune. I don't think I could do tune, bellows and bass at once on this though! Nice write-up too.

chaosprint 4y ago

very very cool. the visual aspect is very impressing. I also use Svelte for this browser-based music live coding environment I am developing: https://glicol.org https://glicol.org The issue with raw Web Audio API is that when there are some heavy stuff on the main thread (in your case the visual feedback), the audio may get glitches. WASM + AudioWorklet is SOTA the solution for this, and Glicol 's tech stack is Rust->WASM->AudioWorklet + SharedArrayBuffer. I also porting the audio engine of Glicol as an NPM package: https://glicol.js.org https://glicol.js.org So if you wish some better audio performance for future development, perhaps you can take a look on that. It provides better audio performance and friendly APIs. I would be happy to update the audio lib based on your feedback.

fenomas 4y ago
> when there are some heavy stuff on the main thread the audio may get glitches Where have you seen that? TFA is just using oscillators, not ScriptProcessor or anything, so one would expect all the heavy audio work to be happening outside the main thread. Personally I use webaudio to do dynamic music for a game, and even though the game is reasonably heavy (3D, physics, etc) I've not noticed any particular glitching.
- chaosprint 4y ago
  You are right! I should say ScriptProcessor there. But for heavy use of audio, especially when there are lots of interactions (often you want your own abstraction rather than the built-in nodes) and sample-level time should be taken into account, WASM+AudioWorklet should definitely be the first choice: https://youtu.be/GfC2WTStW8M?t=1084 https://youtu.be/GfC2WTStW8M?t=1084

ArtWomb 4y ago

This is a terrific Web Audio tutorial! Real-time sound synth in the browser is here. There are already a few DAWs like GridSound out there. But it's still the Wild West ;) Jim Clark's Synth Programming book from 2003: https://www.cim.mcgill.ca/~clark/nordmodularbook/nm_book_toc.html https://www.cim.mcgill.ca/~clark/nordmodularbook/nm_book_toc... Code: https://github.com/aolney/nord-modular-book https://github.com/aolney/nord-modular-book

chaosprint 4y ago
A new video today on Web Audio (Creating Music on the Web - Ben Morss and Hongchan Choi - ADC21): https://youtu.be/GfC2WTStW8M https://youtu.be/GfC2WTStW8M
deleted 4y ago
[deleted]

amelius 4y ago

What's the latency?

bambax 4y ago

> Sine and triangle were a bit more subdued but if you don't fade the sound out properly, it makes a really unpleasant kind of cutting sound due to how your ear reacts when a wave gets cut off. The clicking sound happens when a wave is stopped when not at 0. You do need to fade the oscillator down before stopping it. > As I was figuring everything out, I was experimenting with making new AudioContext for every note because I was trying to fade out the sound, but then I kept realizing that after 50 notes, the app would stop working. Fifty is apparently the limit for the browser, so it's better to just make one AudioContext for the entire app. Yeah the reason the app would stop working when doing things this way is, just because the gain of the oscillator is zero, and the oscillator therefore not audible, it's still there! If you put the gain back up you can hear it again. You can't just fade out and be done with it. So the appropriate approach (not explicitly explained in the article IMHO) is to 1/ set the gain to zero: g.gain.setTargetAtTime(0, context.currentTime, 0.1) 2/ then kill the oscillator with stop(), so that it doesn't consume resources anymore: o.stop(context.currentTime + 0.5) (g being the gain and o the oscillator, and context the AudioContext). Note: "then" means the two actions need to happen one after the other, and that is decided by parameters of the two functions; the functions themselves aren't chained and can be written in any order.

cupofjoakim 4y ago

Nice job! The choice of accordion def wanted me to play by resizing the window though :)

PaulDavisThe1st 4y ago

On the one hand: really great write up of a great little project that is great at helping people experiment with synthesis and software instrument design. On the other hand: another example of a development platform ("web browsers") being utilized for something it is fundamentally not designed for, just because it leverages the skills someone gained along the way. I mostly wish this would stop. People are going to build all kinds of awesome toys using web audio APIs, and every single one of those toys is going to be less performant (latency, CPU/DSP load, interface responsiveness, sensor interactivity) than it's native equivalent. That means that its use will face limitations that are a function of the development platform rather than the designer or performer. On the other hand, how can anyone be against people learning more about synthesis and instrument design? My brain hurts.

chaosprint 4y ago
Good point. I think you do accept the convenience of browsers (many cool projects such as this one https://learningsynths.ableton.com/ https://learningsynths.ableton.com/). So now the problem is the audio performance. As I post in the comments below, we now have WASM, so C++ and Rust can all run in browsers. This can provide a near-native audio performance. Just take Glicol, the live coding language I design as an example: it runs in browsers (https://glicol.org https://glicol.org) and it also runs as a VST plugin (https://youtu.be/tmmBhBmIEW0 https://youtu.be/tmmBhBmIEW0), or you can use the audio engine to write VST plugin (https://github.com/chaosprint/dattorro-vst-rs https://github.com/chaosprint/dattorro-vst-rs).
- spacechild1 4y ago
  Yep. Pure Data and SuperCollider can also run in the browser. But there's a difference between C/C++/Rust audio apps compiled to WASM on the one hand and writing audio code in JS on the other hand. Then again, I think it's perfectly ok for people to experiment with audio in browser. "Native" audio desktop applications will always be superior, so there's nothing to be afraid of.
- PaulDavisThe1st 4y ago
  > As I post in the comments below, we now have WASM, so C++ and Rust can all run in browsers. This can provide a near-native audio performance. Native audio apps require realtime scheduling and memory locking, things you cannot do/control in the browser. The problem is that you can likely get 70-85% of "it" done in the browser, but when the user/performer need the remaining 15-30% for whatever reason, what do they do then?

stevehiehn 4y ago

Great post! I'm very enthusiastic about the WebAudioApi. I'm currently building an audio sample pipeline and enabling users to preview combinations of loops via the api! https://signalsandsorcery.org https://signalsandsorcery.org

jdauriemma 4y ago

If anyone's interested in a Web Audio API trombone, I made one a while back: https://jdauriemma.com/trombone.js/ https://jdauriemma.com/trombone.js/ https://github.com/bignimbus/trombone.js https://github.com/bignimbus/trombone.js

skybrian 4y ago

Headline is misleading - it leaves out that the instrument is an accordion. Talk about burying the lede.

aaaaaaaaata 4y ago

SO cool.