11 ms·

Elementary Audio: a modern platform for writing high performance audio software

138 points by davidbarker 5y ago

Severian 5y ago
Is this for Web audio only?
- trypwire 5y ago
  Nope! You can run the same Elementary code in the web browser, at the command line, offline (for file->file processing), or in a native audio plugin
AdmiralAsshat 5y ago
Just a heads-up that I could see both the project name and specifically the domain causing some confusion with the pre-existing ElementaryOS project (whose domain is https://elementary.io https://elementary.io).
deleted 5y ago
[deleted]
ushakov 5y ago
so this is basically a paid Web Audio API wrapper?
- TheOtherHobbes 5y ago
  It does say you can make plugins, but the link doesn't work. It would be huge if you could make cross-platform AU/VST*/AAX without the costs of JUCE, but I suspect that isn't on offer here.
  - moron4hire 5y ago
    Their intro docs page says Windows and macOS, but actually going to the documentation on plugins says "macOS only" https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer
  - ushakov 5y ago
    still not convinced you can make cross-platform plugins without JUCE by using iPlug, Faust or even Rust the selling point of this seems to allow js/web developers to build plugins and we already know what happened when web developers were allowed to build desktop apps (Electron) luckily given real-time and cpu constraints this might never happen to Audio Plug-ins
    - TheOtherHobbes 5y ago
      You can make plugins in Faust, but it's not a super smooth process. iPlug is the nicest option. I agree about the js though. Even if it's a reinvention of web audio with a C++ backend, js and DSP just don't feel like a good fit. And I would be surprised if the UI side wasn't limited in the way that full native isn't.
    - PaulDavisThe1st 5y ago
      Also, DPF. And if you're willing to use D, DPlug.
  - iainctduncan 5y ago
    I find it hard to imagine people the desktop/commercial audio world wanting to write plugins in JS. At a glance, this seems like an interesting webaudio project that is trying to be too much. Plugins are used in a very different situation where low latency and high performance trump pretty much everything. Reconciling that with web dev priorities seems optimistic.
    - sharklazer 5y ago
      Yeah. I got lost at JS. Buffers and buffer manipulation are everything in audio processing, I don’t see anything stealing C/C++’s thunder here except Rust.
  - trypwire 5y ago
    That is indeed on offer :) The new docs structure broke the website links, I'll get that fixed asap. See here in the mean time: https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer Also, as another user here has already pointed out, the "plugin development kit" is still macos only, though I anticipate releasing the windows version within hopefully a week or two.
  - jerrysievert 5y ago
    have you looked at iplug2? https://iplug2.github.io https://iplug2.github.io since I was already making my own knobs and panels for vcvrack, it was an easy fit for building out vst3/au plugins.
  - fxtentacle 5y ago
    Last time I checked, a perpetual JUCE license was priced similar to 3 days from a C++ freelancer. I've never heard it being called out on price before.
    - TheOtherHobbes 5y ago
      It's changed since I last looked. J V4 Used to be $999 for solo devs (2016), now J V6 is $800 with up to $500k annual revenue. And there's a free tier that allows up to $50k annual revenue. That is indeed pretty affordable. J V3 used to be more than that. I remember looking at it and writing it off for cost reasons, but that was quite some time ago.
  - jcelerier 5y ago
    Slowly getting there with https://github.com/celtera/avendish https://github.com/celtera/avendish :)
  - mxmilkiib 5y ago
    Support for LV2 is landing in JUCE 7 apparently.
poisonarena 5y ago
So with this I could make some fun js audio stuff? I dont really get it, why should I use this instead of howler js, or are they even comparable. either way seems pretty interesting and there is a free/lite version
- a-dub 5y ago
  the webpages require a little digging, but they have an examples page. looks like it's a set of dsp functions where the filter graph is constructed by composing the functions in functional style and then passing it to a render function at the end. so kinda like big data style composition of functions (like the scala stuff people often write to run on spark clusters) but with dsp/audio building blocks running in the browser instead.
  - poisonarena 5y ago
    thank you
moron4hire 5y ago
One of the biggest problem I have with WebAudio API is managing nodes. The API is extremely verbose, when you consider that anything interesting will need to be composed of dozens of basic nodes combined in different ways. You're given a very basic set of primitive nodes, from which you can construct pretty much anything else. But those things you construct will never be "nodes" themselves. You can't create synthetic nodes or subclass node types. But then many of the basic nodes are useful on their own. So you end up in a bind of having two different provenance branches of audio "node-like" things and need to know which is which before you can connect any one of Column A to any other of Column B, or vice versa. You're also on your own in terms of keeping track of those connections. Once a node is connected to another node, there is no in-API way of telling what nodes a node is connected to, or what nodes connect to a given node. There's a whole, huge, potentially cyclical graph structure sitting around in memory that you can manipulate but never inspect. With how hard everything is to keep track of what is where, you can easily end up in a situation where old, now-disconnected or unused nodes stay resident in memory and never get garbage collected. Some developers then punt on tracking connections at all and expect the user to reload the page if there is a major change in audio setup (coughcoughGoogleOmnitonecoughcough). So when people talk about how Web-based gaming really hasn't taken off or replaced the haunting specter of Adobe Flash, and then point at WebGL as the problemm, I kind of laugh. WebGL is the least of your worries. Anyway. Elementary Audio seems to end-run around all that by A) only letting you declaritively define your audio setup, and B) running everything in a single Audio Worklet, thereby bypassing the majority of the Web Audio API.
- trypwire 5y ago
  I couldn't agree with this more. Even beyond the web audio API, we have similar graph APIs in the low level/native C++ world that become unmanageable in their complexity simply due to an inability to compose and manage transitions with ease. This is exactly one of the difficulties that Elementary aims to address, whether you're running in the browser (where indeed, Elementary uses almost none of the actual web audio api), or natively.
- fenomas 5y ago
  Yes, managing webaudio nodes is pretty hellish. If you're interested I maintain a small library for abstracting them - basically you pass the library an object describing what oscillators and filters you want, and what timing envelopes you want attached to their params, and then the library creates and destroys everything. It has no concept of plugins or processing, but it makes it convenient to play around with arbitrary configurations of basic nodes. The library is here: https://github.com/fenomas/wasgen https://github.com/fenomas/wasgen
_bohm 5y ago
Heads up (if the OP is related to the project) the docs page for audio plugins 404s
- trypwire 5y ago
  Thanks! The docs just underwent a significant change for v1.0; the new plugins page is here: https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer
iainctduncan 5y ago
Maybe I'm missing something from my quick read, but the idea of using this for plugins seems like a real misfire. I'm curious whether the devs come from the pro audio world at all. It's one area where prioritizing for development time by using high level garbage collected languages just isn't done because your user is always concerned with how many instances they can run and what the lowest latency they can get is, and that latency constraint is going to be applied across your entire system. So if there is one plugin that hoses the latency by needing a time slice big enough to run a GC in the audio dsp routine, pro audio buyers won't touch it. It looks like a nice way to do interactive web audio work, but plugins are a different beast all together. If you want a place where old school skating-with-razor-blades pointer coding is still common practice, it's DSP code.
- ushakov 5y ago
  exactly users often have to bear the costs for developer productivity with audio software the performance aspect is critical to user and thus the cost of increased developer productivity is exceeding the reward you can’t make any compromise, because otherwise users won’t be buying the product and you go bankrupt this is one thing i wish has happened to Electron
- trypwire 5y ago
  Hey, Nick from Elementary Audio here. You're totally right that in this domain you have to be extremely careful with performance and latency. Elementary takes great care to deliver that; it's a JavaScript "frontend" API but all of the actual handling of audio is done natively with high quality realtime constraints. The core engine is all native C/C++, and we run in the web by compiling to wasm and running inside a web worker. For delivering a desktop app or audio plugin, the native engine is compiled directly into the target binary. So, while JavaScript and garbage collectors can bring a performance concern, we're only using JavaScript there for handling the lightweight virtual representation of the underlying engine state, and for that role JavaScript is plenty fast enough and the garbage collector quite helpful!
  - danuker 5y ago
reggieband 5y ago
What a crazy coincidence. I just spent the last couple of days doing a survey of audio engines with a mind to find those that could run in wasm. Did you write your own engine or are you built on top of csound, supercollider, ChucK or something else? How do you compare against the higher level or pattern based implementations like Sonic Pi, Tidal Cycles, Overtone and Cabbage Audio? How about libraries like LibPD or even Max/MSP? I was even considering this space myself and on my list of todo's was checking out the licenses of the above to see what the ramifications of deploying a commercialized version of them would be. However, I must admit that the market for audio software is incredibly split. On the one end are the high-end audio effects with eye-watering prices. On the other OSS end many of the devs are literally begging for scraps on Patreon barely making enough to survive. There doesn't seem to be an in-between. I am very curious what you believe your market will be.
- chaosprint 5y ago
  You should check Glicol: https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol For the marketing part, frankly speaking, I don't know. Since most of these projects you mentioned are based on academics, including myself.
  - reggieband 5y ago
    I definitely did checkout Glicol, sorry I didn't call it out along with the others but I wanted to focus one the "uses existing old-school C/C++ audio engine compiled to wasm as a server" question. I understand you wrote your own custom engine in Rust. I saw your post in the comment section related to Overtone and was actually going to reply saying to you that I appreciated your work. I think I watched all of your Youtube video demos, even the ones on QuaverSeries. The other small project that was interesting to me (that seems to have partially disappeared) was Sporth. The webpage for that has an unfortunate message saying the project was taken down to respect csound community.
    - chaosprint 5y ago
      Wow, thanks for your interest! I haven't maintained QuaverSeries for a while as I kinda feel that there are some issues on Tone.js that cannot be fixed in the foreseeable future. For the topic you are interested (C++ audio engine as WASM), you should check these two: https://github.com/frantic0/sema-engine https://github.com/frantic0/sema-engine https://github.com/micknoise/Maximilian https://github.com/micknoise/Maximilian For those language you mentioned csound, supercollider, Chuck (should add Faust here), I think these languages focus more on different music programming philosophy. Of course, SC is probably the most famous for reusing its audio engine for other high-level langs like the Tidal, Sonic Pi you mentioned. Also, SuperCollider's scsynth can run in browsers now: https://scsynth.org/t/webassembly-support/3037 https://scsynth.org/t/webassembly-support/3037 If you are interested, https://strudel.tidalcycles.org/ https://strudel.tidalcycles.org/ is an ongoing work to port Tidal to browsers. Perhaps at some point it can be connected with Glicol audio engine.
bfrog 5y ago
Another day, another pile of JavaScript eating into an area I care about with a language I despise.
chaosprint 5y ago
Very interesting project. I really like your website design and the documentation. For the audio part, I think it would be very interesting to compare it with my project Glicol (https://glicol.org https://glicol.org) from many aspects. From Web Audio aspect, Glicol recommands users to add only two lines of code to the project: ```html <script src="https://cdn.jsdelivr.net/gh/chaosprint/glicol@latest/js/src/glicol.min.js"></script> https://cdn.jsdelivr.net/gh/chaosprint/glicol@latest/js/src/... ``` Then: ```js run(out: `sin 440`) ``` For params updating, just call the `run` again: ```js run(out: `sin 220`) ``` And Glicol engine will use LCS to analyse the difference, and avoid updating the whole graph. Since Glicol uses Rust, WASM and SharedArrayBuffer, there is no GC, and support sample level control. I haven't checked the source code of your project, but since I know CodePen quite well, it looks like that you need to add SAB, although it requires CORS during deployment. Also, I think it may need more consideration to promote it with "Write once, run anywhere" at this stage. For example, how about Teensy? For Glicol, although it has POC to show that it can run as VST or on Bela board (https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol), there are still a long way to go (for both of us). I need to have Rust no_std from some of Glicol's dependencies. Yet I don't know how JS can be used in Teensy/STM32. But this is definitely not the end of the world. Sometimes, to have a better scope is actually a good thing for a project. If I would be to promote Elementary Audio, I would call it a Tone.js alternative. It would still be a significant contribution, since Tone.js has been so influential in Web Audio, especially for beginners. How do you think? I think for Tone.js, the time management and scheduling is not very satisfying, perhaps you can introduce more on this topic for Elementary? All being said, I think it is very exciting to see a new project in audio community. Good luck and please feel free to discuss more.
Archit3ch 5y ago
Hey Nick! I remember you showcasing this in the audio dev community. It's a paradigm shift in the way audio software is written. Congratulations on your launch and I wish you success!
realYitzi 5y ago
the shell examples are not working for me. `zsh: command not found: elementary` after following install instructions.
PaulDavisThe1st 5y ago
It's hard to know what to say. > The conventional approach to writing audio software is an challenging process requiring expertise in a wide breadth of topics– C++, multi-threading and thread safety, lock-free programming, and realtime thread safety come to quickly to mind, and we haven't even mentioned digital signal processing itself. Moreover, the tooling, workflow, and developer experience that we have in this approach pales in comparison to that of the web software industry, for example. All together, the reality of these complications is that the conventional approach for writing audio software is difficult and time consuming. We believe it doesn't have to be that way. This is then followed by a screenshot of some code that looks like a SuperCollider script from 15 years ago. There are at least a dozen audio "scripting" languages that free someone from all the issues mentioned above. Many have been mentioned in the thread already. The only things that Elementary Audio appears to have in its favor are that it leverages JS and it runs in the browser (sort of). For some people, those two alone will be interesting. Have at it, people! But for people who write plugins designed to run inside DAWs, or modules inside Rack, and in particular for the people who write DAWs or Rack (or Rack-like things), this is of approximately zero interest. It's a web-centric version of SuperCollider, which is cool but ultimately so bounded by the limitations of the platform that for many things, SuperCollider or something similar remains more powerful and more flexible.
- iainctduncan 5y ago
  That was my impression too, it looks very much like an SC style computer music language. Which begs the question, who is this for? Someone who writes plugins doesn't need or want the high level SC style language. Someone seriously into computer music/audio isn't going to want the limitations, and has a lot of options already to choose from that are mature, if relatively unknown outside music circles. If it were my company, I would be marketing this squarely at people with the need for high level procedural audio in a web context. This looks like it could be great for browser-based game devs, for example. I do know that Pure Data and Andy Farnell's great sound design book for it were a hit in the game dev world of procedural audio, FWTW.
  - aaaaaaaaaaab 5y ago
aaaaaaaaaaab 5y ago
>high performance audio software >runs in browser doubt.gif
- subpar 5y ago
  https://doubt.gif.so/ https://doubt.gif.so/
gavinray 5y ago
It's probably helpful to have some background context for the development of this The author of this library is also the author of the React JUCE renderer. It's a toolkit that re-uses React Native's rendering engine to allow you to use JS and React to render JUCE components: https://www.nickwritesablog.com/blueprint-a-juce-rendering-backend-for-react-js/ https://www.nickwritesablog.com/blueprint-a-juce-rendering-b... https://github.com/JoshMarler/react-juce https://github.com/JoshMarler/react-juce So at this point, you have a (presumably) C++ JUCE app that is using JS/TS to render the UI. Why not work to move DSP stuff into JS as well, so that you're gluing minimal bits of C++ together? It's not a far leap from there to here when you follow this line of thinking down the road.
moarsel 5y ago
I see several comments along the lines of "who is this even for", which may be classic HN negativity or maybe the docs don't make it clear enough. I've actually played with this library (and many others), and I'm super psyched to see it ship! To me this is the equivalent of React native for audio: describe your audio declaratively, and let the platform handle the rest. The benefits are: - Write once, deploy everywhere (building DAW plugins sucks, and most web audio is not performant) - Lower barrier to learning DSP (yes, DSP engineers already know low level languages that can do this, AND now more people can build audio software) - Modern dev environment and faster feedback loop. I can now build a React app (or React native) with a high performance audio engine built right in. I'm pretty skeptical of people defending the status quo approach here... Like, where's the Figma for DAWs? Why don't more more people to write their own VST plugins? And why is audio software so UGLY? Audio software design has been stuck in the 90's for too long and I'm psyched to see it open up and become more accessible!
Reubend 5y ago
I was interested but I took a look at the documentation, and it seems like VST plugins using this can only be compiled for Mac right now. Is that right? It doesn't seem very useful for building plugins if that's the case.
dharma1 5y ago
I’m pretty bullish for more and more lightweight use cases of audio running in the browser, and it’s also nice how much of it is open source. Audio editing (audiomass.co), fun little sequencer experiments (https://teetow.github.io/elementary_grid/ https://teetow.github.io/elementary_grid/) - things like this keep getting better. I wouldn’t run a full DAW in the browser yet (too spoiled with Logic) though BandLab is having a good go at it. But makes no sense to poo poo those attempts either. Both have their uses (browser based apps and native apps)
spacemanmatt 5y ago
I wanted to see what it could do, so I ran the demo. I still want to see what it can do.
iuiz 5y ago
Does this run on a STM32?
trypwire 5y ago
Hey all, Nick from Elementary Audio here. I spent some time in the discussion below yesterday and coming back to it again now I can see this continued to get way more attention than I expected! Thanks for all of your thoughts. There's a lot of discussion here, and although I'd love to visit every comment in specific I think it makes sense to speak in summary to a few of the major themes/underlying questions here. In general, the discussion here surrounds totally valid concerns that I'll try to cover, and I understand after this thread that my messaging needs some serious work to speak to those concerns. Before I get to those questions, I want to start with some of the things that Elementary is _not_ well suited for, or some areas where Elementary currently has some limitations. * It's a young project; there are plenty of standard library features that are not implemented yet (oversampling, several more filter topologies, iFFT, etc). That might mean Elementary is currently ill-suited for your project * It uses a generalized, block-based graph rendering algorithm. If you have hard performance requirements or need highly specific and hand tuned DSP optimizations, Elementary is currently not ready for you. * Because of the current rendering algorithm, you cannot specify signal flow graphs that have 1-sample feedback loops. With that out of the way, let's talk through some of the questions here that seem to underly many of the comments. 1. Why JavaScript? * It's inarguably one of the most widely used languages in the world * It's more than fast enough for the role that it plays in Elementary * It's the language of the web; which isn't going anywhere any time soon, and at the same time the industry is showing us demand for audio software that runs in the web. * JavaScript/web tech has pioneered the way that we write app UIs over the last decade, and continues to do so * JavaScript/web tech has pioneered oen of the best developer experience/workflows in software development 2. How can JavaScript possibly be fast enough for this? * Elementary uses JavaScript only to _describe_ audio processes, not execute them * All of the actual audio processing is handled in the underlying C++ audio engine * This engine is compiled to native, platform-specific targets in the appropriate cases (i.e. plugins, cli), and only compiled to WASM when appropriate for running in the browser * The description of the audio process is done with an extremely lightweight virtual representation for which JavaScript is more than fast enough, and for which garbage collection and seamless integration with application logic and user interface programming is desirable 3. What about latency? * The core audio engine adds no audio latency overhead whatsoever; if you're running a platform/native target, you're running with the same latency that your drivers/plugin host are delivering to Elementary. Until, of course, you describe an audio graph which introduces new latency; that's on you. * When compiled to WASM to run in the browser, of course there's additional implications here, but that's a reality of targeting the web in the first place. Elementary doesn't force you to the web, or force you to wasm, but if you want to run in the web, accepting the latency implications yourself for your own apps, Elementary can go there with you 4. What about bottom line performance? Professional, commercial software?? * Elementary will not make the claim that for executing a static graph it will outperform native C/C++ hand-tailored to the application demands. That would be crazy; it's probably fair to say that highly specialized code can always outperform generalized algorithms. * Elementary will make the claim that for plenty of applications, generalized graph processing is surely fast enough. * The generalized graph processing algorithm is fast enough because the audio processing is purely native (or compiled to wasm); we do not execute any JavaScript itself on the audio processing thread 5. What about Faust/SuperCollider/PureData/ChucK/Max...? * I have direct experience with many of these languages, and have reviewed almost all of the others in detail. I think they're excellent pieces of software for prototyping and exploring. I also think they're generally weaker for shipping complete applications. * Of course you can use Faust/Max Gen or similar to generate C++ and then bolt on your own application harness, and I'd encourage you to try. For non-trivial applications, interfacing between the UI and the DSP becomes cumbersome. Elementary aims to provide a better experience here. 6. Why do any of this at all? The current dev process and workflow for writing audio software has stagnated; it takes so much time to turn a prototype into a product and we regularly accept complexities and limitations that for so many types of apps we no longer need to accept. Elementary offers a new way of prototyping and shipping those apps more easily and more quickly. Thanks again for all the discussion, I hope this clears up some of the details. Cheers, Nick

138 points by davidbarker 5y ago

Severian 5y ago

Is this for Web audio only?

trypwire 5y ago
Nope! You can run the same Elementary code in the web browser, at the command line, offline (for file->file processing), or in a native audio plugin

AdmiralAsshat 5y ago

Just a heads-up that I could see both the project name and specifically the domain causing some confusion with the pre-existing ElementaryOS project (whose domain is https://elementary.io https://elementary.io).

deleted 5y ago

[deleted]

ushakov 5y ago

so this is basically a paid Web Audio API wrapper?

TheOtherHobbes 5y ago
It does say you can make plugins, but the link doesn't work. It would be huge if you could make cross-platform AU/VST*/AAX without the costs of JUCE, but I suspect that isn't on offer here.
- moron4hire 5y ago
  Their intro docs page says Windows and macOS, but actually going to the documentation on plugins says "macOS only" https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer
- ushakov 5y ago
  still not convinced you can make cross-platform plugins without JUCE by using iPlug, Faust or even Rust the selling point of this seems to allow js/web developers to build plugins and we already know what happened when web developers were allowed to build desktop apps (Electron) luckily given real-time and cpu constraints this might never happen to Audio Plug-ins
  - TheOtherHobbes 5y ago
    You can make plugins in Faust, but it's not a super smooth process. iPlug is the nicest option. I agree about the js though. Even if it's a reinvention of web audio with a C++ backend, js and DSP just don't feel like a good fit. And I would be surprised if the UI side wasn't limited in the way that full native isn't.
  - PaulDavisThe1st 5y ago
    Also, DPF. And if you're willing to use D, DPlug.
- iainctduncan 5y ago
  I find it hard to imagine people the desktop/commercial audio world wanting to write plugins in JS. At a glance, this seems like an interesting webaudio project that is trying to be too much. Plugins are used in a very different situation where low latency and high performance trump pretty much everything. Reconciling that with web dev priorities seems optimistic.
  - sharklazer 5y ago
    Yeah. I got lost at JS. Buffers and buffer manipulation are everything in audio processing, I don’t see anything stealing C/C++’s thunder here except Rust.
- trypwire 5y ago
  That is indeed on offer :) The new docs structure broke the website links, I'll get that fixed asap. See here in the mean time: https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer Also, as another user here has already pointed out, the "plugin development kit" is still macos only, though I anticipate releasing the windows version within hopefully a week or two.
- jerrysievert 5y ago
  have you looked at iplug2? https://iplug2.github.io https://iplug2.github.io since I was already making my own knobs and panels for vcvrack, it was an easy fit for building out vst3/au plugins.
- fxtentacle 5y ago
  Last time I checked, a perpetual JUCE license was priced similar to 3 days from a C++ freelancer. I've never heard it being called out on price before.
  - TheOtherHobbes 5y ago
    It's changed since I last looked. J V4 Used to be $999 for solo devs (2016), now J V6 is $800 with up to $500k annual revenue. And there's a free tier that allows up to $50k annual revenue. That is indeed pretty affordable. J V3 used to be more than that. I remember looking at it and writing it off for cost reasons, but that was quite some time ago.
- jcelerier 5y ago
  Slowly getting there with https://github.com/celtera/avendish https://github.com/celtera/avendish :)
- mxmilkiib 5y ago
  Support for LV2 is landing in JUCE 7 apparently.

poisonarena 5y ago

So with this I could make some fun js audio stuff? I dont really get it, why should I use this instead of howler js, or are they even comparable. either way seems pretty interesting and there is a free/lite version

a-dub 5y ago
the webpages require a little digging, but they have an examples page. looks like it's a set of dsp functions where the filter graph is constructed by composing the functions in functional style and then passing it to a render function at the end. so kinda like big data style composition of functions (like the scala stuff people often write to run on spark clusters) but with dsp/audio building blocks running in the browser instead.
- poisonarena 5y ago
  thank you

moron4hire 5y ago

One of the biggest problem I have with WebAudio API is managing nodes. The API is extremely verbose, when you consider that anything interesting will need to be composed of dozens of basic nodes combined in different ways. You're given a very basic set of primitive nodes, from which you can construct pretty much anything else. But those things you construct will never be "nodes" themselves. You can't create synthetic nodes or subclass node types. But then many of the basic nodes are useful on their own. So you end up in a bind of having two different provenance branches of audio "node-like" things and need to know which is which before you can connect any one of Column A to any other of Column B, or vice versa. You're also on your own in terms of keeping track of those connections. Once a node is connected to another node, there is no in-API way of telling what nodes a node is connected to, or what nodes connect to a given node. There's a whole, huge, potentially cyclical graph structure sitting around in memory that you can manipulate but never inspect. With how hard everything is to keep track of what is where, you can easily end up in a situation where old, now-disconnected or unused nodes stay resident in memory and never get garbage collected. Some developers then punt on tracking connections at all and expect the user to reload the page if there is a major change in audio setup (coughcoughGoogleOmnitonecoughcough). So when people talk about how Web-based gaming really hasn't taken off or replaced the haunting specter of Adobe Flash, and then point at WebGL as the problemm, I kind of laugh. WebGL is the least of your worries. Anyway. Elementary Audio seems to end-run around all that by A) only letting you declaritively define your audio setup, and B) running everything in a single Audio Worklet, thereby bypassing the majority of the Web Audio API.

trypwire 5y ago
I couldn't agree with this more. Even beyond the web audio API, we have similar graph APIs in the low level/native C++ world that become unmanageable in their complexity simply due to an inability to compose and manage transitions with ease. This is exactly one of the difficulties that Elementary aims to address, whether you're running in the browser (where indeed, Elementary uses almost none of the actual web audio api), or natively.
fenomas 5y ago
Yes, managing webaudio nodes is pretty hellish. If you're interested I maintain a small library for abstracting them - basically you pass the library an object describing what oscillators and filters you want, and what timing envelopes you want attached to their params, and then the library creates and destroys everything. It has no concept of plugins or processing, but it makes it convenient to play around with arbitrary configurations of basic nodes. The library is here: https://github.com/fenomas/wasgen https://github.com/fenomas/wasgen

_bohm 5y ago

Heads up (if the OP is related to the project) the docs page for audio plugins 404s

trypwire 5y ago
Thanks! The docs just underwent a significant change for v1.0; the new plugins page is here: https://www.elementary.audio/docs/packages/plugin-renderer https://www.elementary.audio/docs/packages/plugin-renderer

iainctduncan 5y ago

Maybe I'm missing something from my quick read, but the idea of using this for plugins seems like a real misfire. I'm curious whether the devs come from the pro audio world at all. It's one area where prioritizing for development time by using high level garbage collected languages just isn't done because your user is always concerned with how many instances they can run and what the lowest latency they can get is, and that latency constraint is going to be applied across your entire system. So if there is one plugin that hoses the latency by needing a time slice big enough to run a GC in the audio dsp routine, pro audio buyers won't touch it. It looks like a nice way to do interactive web audio work, but plugins are a different beast all together. If you want a place where old school skating-with-razor-blades pointer coding is still common practice, it's DSP code.

ushakov 5y ago
exactly users often have to bear the costs for developer productivity with audio software the performance aspect is critical to user and thus the cost of increased developer productivity is exceeding the reward you can’t make any compromise, because otherwise users won’t be buying the product and you go bankrupt this is one thing i wish has happened to Electron
trypwire 5y ago
Hey, Nick from Elementary Audio here. You're totally right that in this domain you have to be extremely careful with performance and latency. Elementary takes great care to deliver that; it's a JavaScript "frontend" API but all of the actual handling of audio is done natively with high quality realtime constraints. The core engine is all native C/C++, and we run in the web by compiling to wasm and running inside a web worker. For delivering a desktop app or audio plugin, the native engine is compiled directly into the target binary. So, while JavaScript and garbage collectors can bring a performance concern, we're only using JavaScript there for handling the lightweight virtual representation of the underlying engine state, and for that role JavaScript is plenty fast enough and the garbage collector quite helpful!
- danuker 5y ago

reggieband 5y ago

What a crazy coincidence. I just spent the last couple of days doing a survey of audio engines with a mind to find those that could run in wasm. Did you write your own engine or are you built on top of csound, supercollider, ChucK or something else? How do you compare against the higher level or pattern based implementations like Sonic Pi, Tidal Cycles, Overtone and Cabbage Audio? How about libraries like LibPD or even Max/MSP? I was even considering this space myself and on my list of todo's was checking out the licenses of the above to see what the ramifications of deploying a commercialized version of them would be. However, I must admit that the market for audio software is incredibly split. On the one end are the high-end audio effects with eye-watering prices. On the other OSS end many of the devs are literally begging for scraps on Patreon barely making enough to survive. There doesn't seem to be an in-between. I am very curious what you believe your market will be.

chaosprint 5y ago
You should check Glicol: https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol For the marketing part, frankly speaking, I don't know. Since most of these projects you mentioned are based on academics, including myself.
- reggieband 5y ago
  I definitely did checkout Glicol, sorry I didn't call it out along with the others but I wanted to focus one the "uses existing old-school C/C++ audio engine compiled to wasm as a server" question. I understand you wrote your own custom engine in Rust. I saw your post in the comment section related to Overtone and was actually going to reply saying to you that I appreciated your work. I think I watched all of your Youtube video demos, even the ones on QuaverSeries. The other small project that was interesting to me (that seems to have partially disappeared) was Sporth. The webpage for that has an unfortunate message saying the project was taken down to respect csound community.
  - chaosprint 5y ago
    Wow, thanks for your interest! I haven't maintained QuaverSeries for a while as I kinda feel that there are some issues on Tone.js that cannot be fixed in the foreseeable future. For the topic you are interested (C++ audio engine as WASM), you should check these two: https://github.com/frantic0/sema-engine https://github.com/frantic0/sema-engine https://github.com/micknoise/Maximilian https://github.com/micknoise/Maximilian For those language you mentioned csound, supercollider, Chuck (should add Faust here), I think these languages focus more on different music programming philosophy. Of course, SC is probably the most famous for reusing its audio engine for other high-level langs like the Tidal, Sonic Pi you mentioned. Also, SuperCollider's scsynth can run in browsers now: https://scsynth.org/t/webassembly-support/3037 https://scsynth.org/t/webassembly-support/3037 If you are interested, https://strudel.tidalcycles.org/ https://strudel.tidalcycles.org/ is an ongoing work to port Tidal to browsers. Perhaps at some point it can be connected with Glicol audio engine.

bfrog 5y ago

Another day, another pile of JavaScript eating into an area I care about with a language I despise.

chaosprint 5y ago

Very interesting project. I really like your website design and the documentation. For the audio part, I think it would be very interesting to compare it with my project Glicol (https://glicol.org https://glicol.org) from many aspects. From Web Audio aspect, Glicol recommands users to add only two lines of code to the project: ```html <script src="https://cdn.jsdelivr.net/gh/chaosprint/glicol@latest/js/src/glicol.min.js"></script> https://cdn.jsdelivr.net/gh/chaosprint/glicol@latest/js/src/... ``` Then: ```js run(out: `sin 440`) ``` For params updating, just call the `run` again: ```js run(out: `sin 220`) ``` And Glicol engine will use LCS to analyse the difference, and avoid updating the whole graph. Since Glicol uses Rust, WASM and SharedArrayBuffer, there is no GC, and support sample level control. I haven't checked the source code of your project, but since I know CodePen quite well, it looks like that you need to add SAB, although it requires CORS during deployment. Also, I think it may need more consideration to promote it with "Write once, run anywhere" at this stage. For example, how about Teensy? For Glicol, although it has POC to show that it can run as VST or on Bela board (https://github.com/chaosprint/glicol https://github.com/chaosprint/glicol), there are still a long way to go (for both of us). I need to have Rust no_std from some of Glicol's dependencies. Yet I don't know how JS can be used in Teensy/STM32. But this is definitely not the end of the world. Sometimes, to have a better scope is actually a good thing for a project. If I would be to promote Elementary Audio, I would call it a Tone.js alternative. It would still be a significant contribution, since Tone.js has been so influential in Web Audio, especially for beginners. How do you think? I think for Tone.js, the time management and scheduling is not very satisfying, perhaps you can introduce more on this topic for Elementary? All being said, I think it is very exciting to see a new project in audio community. Good luck and please feel free to discuss more.

Archit3ch 5y ago

Hey Nick! I remember you showcasing this in the audio dev community. It's a paradigm shift in the way audio software is written. Congratulations on your launch and I wish you success!

realYitzi 5y ago

the shell examples are not working for me. `zsh: command not found: elementary` after following install instructions.

PaulDavisThe1st 5y ago

It's hard to know what to say. > The conventional approach to writing audio software is an challenging process requiring expertise in a wide breadth of topics– C++, multi-threading and thread safety, lock-free programming, and realtime thread safety come to quickly to mind, and we haven't even mentioned digital signal processing itself. Moreover, the tooling, workflow, and developer experience that we have in this approach pales in comparison to that of the web software industry, for example. All together, the reality of these complications is that the conventional approach for writing audio software is difficult and time consuming. We believe it doesn't have to be that way. This is then followed by a screenshot of some code that looks like a SuperCollider script from 15 years ago. There are at least a dozen audio "scripting" languages that free someone from all the issues mentioned above. Many have been mentioned in the thread already. The only things that Elementary Audio appears to have in its favor are that it leverages JS and it runs in the browser (sort of). For some people, those two alone will be interesting. Have at it, people! But for people who write plugins designed to run inside DAWs, or modules inside Rack, and in particular for the people who write DAWs or Rack (or Rack-like things), this is of approximately zero interest. It's a web-centric version of SuperCollider, which is cool but ultimately so bounded by the limitations of the platform that for many things, SuperCollider or something similar remains more powerful and more flexible.

iainctduncan 5y ago
That was my impression too, it looks very much like an SC style computer music language. Which begs the question, who is this for? Someone who writes plugins doesn't need or want the high level SC style language. Someone seriously into computer music/audio isn't going to want the limitations, and has a lot of options already to choose from that are mature, if relatively unknown outside music circles. If it were my company, I would be marketing this squarely at people with the need for high level procedural audio in a web context. This looks like it could be great for browser-based game devs, for example. I do know that Pure Data and Andy Farnell's great sound design book for it were a hit in the game dev world of procedural audio, FWTW.
- aaaaaaaaaaab 5y ago

aaaaaaaaaaab 5y ago

>high performance audio software >runs in browser doubt.gif

subpar 5y ago
https://doubt.gif.so/ https://doubt.gif.so/

gavinray 5y ago

It's probably helpful to have some background context for the development of this The author of this library is also the author of the React JUCE renderer. It's a toolkit that re-uses React Native's rendering engine to allow you to use JS and React to render JUCE components: https://www.nickwritesablog.com/blueprint-a-juce-rendering-backend-for-react-js/ https://www.nickwritesablog.com/blueprint-a-juce-rendering-b... https://github.com/JoshMarler/react-juce https://github.com/JoshMarler/react-juce So at this point, you have a (presumably) C++ JUCE app that is using JS/TS to render the UI. Why not work to move DSP stuff into JS as well, so that you're gluing minimal bits of C++ together? It's not a far leap from there to here when you follow this line of thinking down the road.

moarsel 5y ago

I see several comments along the lines of "who is this even for", which may be classic HN negativity or maybe the docs don't make it clear enough. I've actually played with this library (and many others), and I'm super psyched to see it ship! To me this is the equivalent of React native for audio: describe your audio declaratively, and let the platform handle the rest. The benefits are: - Write once, deploy everywhere (building DAW plugins sucks, and most web audio is not performant) - Lower barrier to learning DSP (yes, DSP engineers already know low level languages that can do this, AND now more people can build audio software) - Modern dev environment and faster feedback loop. I can now build a React app (or React native) with a high performance audio engine built right in. I'm pretty skeptical of people defending the status quo approach here... Like, where's the Figma for DAWs? Why don't more more people to write their own VST plugins? And why is audio software so UGLY? Audio software design has been stuck in the 90's for too long and I'm psyched to see it open up and become more accessible!

Reubend 5y ago

I was interested but I took a look at the documentation, and it seems like VST plugins using this can only be compiled for Mac right now. Is that right? It doesn't seem very useful for building plugins if that's the case.

dharma1 5y ago

I’m pretty bullish for more and more lightweight use cases of audio running in the browser, and it’s also nice how much of it is open source. Audio editing (audiomass.co), fun little sequencer experiments (https://teetow.github.io/elementary_grid/ https://teetow.github.io/elementary_grid/) - things like this keep getting better. I wouldn’t run a full DAW in the browser yet (too spoiled with Logic) though BandLab is having a good go at it. But makes no sense to poo poo those attempts either. Both have their uses (browser based apps and native apps)

spacemanmatt 5y ago

I wanted to see what it could do, so I ran the demo. I still want to see what it can do.

iuiz 5y ago

Does this run on a STM32?

trypwire 5y ago

Hey all, Nick from Elementary Audio here. I spent some time in the discussion below yesterday and coming back to it again now I can see this continued to get way more attention than I expected! Thanks for all of your thoughts. There's a lot of discussion here, and although I'd love to visit every comment in specific I think it makes sense to speak in summary to a few of the major themes/underlying questions here. In general, the discussion here surrounds totally valid concerns that I'll try to cover, and I understand after this thread that my messaging needs some serious work to speak to those concerns. Before I get to those questions, I want to start with some of the things that Elementary is _not_ well suited for, or some areas where Elementary currently has some limitations. * It's a young project; there are plenty of standard library features that are not implemented yet (oversampling, several more filter topologies, iFFT, etc). That might mean Elementary is currently ill-suited for your project * It uses a generalized, block-based graph rendering algorithm. If you have hard performance requirements or need highly specific and hand tuned DSP optimizations, Elementary is currently not ready for you. * Because of the current rendering algorithm, you cannot specify signal flow graphs that have 1-sample feedback loops. With that out of the way, let's talk through some of the questions here that seem to underly many of the comments. 1. Why JavaScript? * It's inarguably one of the most widely used languages in the world * It's more than fast enough for the role that it plays in Elementary * It's the language of the web; which isn't going anywhere any time soon, and at the same time the industry is showing us demand for audio software that runs in the web. * JavaScript/web tech has pioneered the way that we write app UIs over the last decade, and continues to do so * JavaScript/web tech has pioneered oen of the best developer experience/workflows in software development 2. How can JavaScript possibly be fast enough for this? * Elementary uses JavaScript only to _describe_ audio processes, not execute them * All of the actual audio processing is handled in the underlying C++ audio engine * This engine is compiled to native, platform-specific targets in the appropriate cases (i.e. plugins, cli), and only compiled to WASM when appropriate for running in the browser * The description of the audio process is done with an extremely lightweight virtual representation for which JavaScript is more than fast enough, and for which garbage collection and seamless integration with application logic and user interface programming is desirable 3. What about latency? * The core audio engine adds no audio latency overhead whatsoever; if you're running a platform/native target, you're running with the same latency that your drivers/plugin host are delivering to Elementary. Until, of course, you describe an audio graph which introduces new latency; that's on you. * When compiled to WASM to run in the browser, of course there's additional implications here, but that's a reality of targeting the web in the first place. Elementary doesn't force you to the web, or force you to wasm, but if you want to run in the web, accepting the latency implications yourself for your own apps, Elementary can go there with you 4. What about bottom line performance? Professional, commercial software?? * Elementary will not make the claim that for executing a static graph it will outperform native C/C++ hand-tailored to the application demands. That would be crazy; it's probably fair to say that highly specialized code can always outperform generalized algorithms. * Elementary will make the claim that for plenty of applications, generalized graph processing is surely fast enough. * The generalized graph processing algorithm is fast enough because the audio processing is purely native (or compiled to wasm); we do not execute any JavaScript itself on the audio processing thread 5. What about Faust/SuperCollider/PureData/ChucK/Max...? * I have direct experience with many of these languages, and have reviewed almost all of the others in detail. I think they're excellent pieces of software for prototyping and exploring. I also think they're generally weaker for shipping complete applications. * Of course you can use Faust/Max Gen or similar to generate C++ and then bolt on your own application harness, and I'd encourage you to try. For non-trivial applications, interfacing between the UI and the DSP becomes cumbersome. Elementary aims to provide a better experience here. 6. Why do any of this at all? The current dev process and workflow for writing audio software has stagnated; it takes so much time to turn a prototype into a product and we regularly accept complexities and limitations that for so many types of apps we no longer need to accept. Elementary offers a new way of prototyping and shipping those apps more easily and more quickly. Thanks again for all the discussion, I hope this clears up some of the details. Cheers, Nick