5 ms·
I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in l
by padenot 5y ago
I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in literate programming in a way that will _never_ glitch: https://blog.paul.cx/post/metronome/ https://blog.paul.cx/post/metronome/. The result is nowhere as interesting as this app, but this is to get the point across.
https://www.html5rocks.com/en/tutorials/audio/scheduling/ https://www.html5rocks.com/en/tutorials/audio/scheduling/ is an article written by my former fellow co-editor of the Web Audio API specification, that talks in length about this.
In 2021, with the AudioWorklet [0] being available in all major browsers (including Safari as of a few weeks ago!), arbitrary computations can happen on the real-time audio thread. It is now possible to send a copy of the sequencer data (which note when, what sample, what velocity, etc.) to the real-time thread, and let it play, unaffected by the load of the machine, be it from the browser or other programs (until everything breaks down because you're also overloading the audio thread).
I think this app is very good and does the best it can under the programming model it's using, considering the environment it's being run in, but if more robustness is needed, there is now a way !
On your last paragraph, it is still _absolutely_ critical. An audio software that doesn't use a real-time thread for rendering cannot render audio without glitches, even under no load, even on a fast machine. You don't need to tune the OS [1] and pick special hardware these days: today's computers (and even computers from 10 years ago or older) are perfectly capable or running reasonably complex audio processing workloads _if the software is written correctly_. This is regardless of the use case. When we disabled the real-time thread on Windows by mistake in Firefox (adjusting priority of the process level, and then that was inherited to all threads, very quickly reverted), we had reports in a matter of _days_ (if not the next day), and the Firefox Nightly population is not that big. This was I think playing regular videos on social media, not even anything complicated.
If the author of the app, or anybody, wants to profile the app with the right tools, I've written a blog post [2] about this as well. I hear this tooling is used by audio professionals, now, and that they're happy with it. Happy to hear about any suggestion though :-).
For your Firefox-specific question, you're right that increasing the content process count will make everything better, and we're constantly improving memory usage so that it's possible to do so even on lower-specs machines, maybe it's worth a try. Please look at the "resident memory" values, and not, say, virtual or shared, because we make heavy use of shared memory and copy on write facilities of the OS to limit the memory usage when using multiple process (which we need to do for origin isolation and avoiding Spectre issues and sandboxing various bits of the browser as hard as we can, for example).
[0]: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle...
[1]: (but you can to improve robustness further, and it helps to have good drivers, such as Jack on Linux, WASAPI with MMCSS on Windows or ASIO, or just the normal stuff on macOS)
[2]: https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ https://blog.paul.cx/post/profiling-firefox-real-time-media-...
- gnrlst 5y agoI experimented 5-6 years ago with a programmable metronome using KnockoutJS and realized quickly how easy it is to cause jitters and unpredictable performance on the main thread. I wish I had come across this post back then! Although a big contributor for performance issues in my project (and especially OP's) is the amount of UI updates that need to happen in succession (lots of event listeners, css transformations and dom changes)
- nyanpasu64 5y agoThanks for the post and your blog! Added to my RSS reader. On the topic of MMCSS, RtAudio is another real-time audio API... except the WASAPI backend had broken real-time thread scheduling (AvSetMmThreadCharacteristicsW, I think this is MMCSS?) on any builds with UNICODE enabled... and somehow literally nobody noticed, until I analyzed RtAudio under Process Monitor and wondered why it was trying to open a random Chinese string as a DLL file. I ended up fixing it at https://github.com/thestk/rtaudio/pull/292/files https://github.com/thestk/rtaudio/pull/292/files . I don't know what that means. Is MMCSS not that important (even though you mentioned missing MMCSS caused severe stuttering in Firefox)? Is RtAudio under-tested/amateurish to the point people didn't notice it was stuttering more than it should be? I assume it's unacceptable to synthesize (or probably time) audio off the main thread. How bad is it to use mutexes and allocations on the audio thread?
- padenot 5y agoI'm not sure how many Windows users RtAudio has, and the kind of machines people are using with it, and their expectations, but in Firefox, the answer is hundreds of millions and "any kind of machine that has existed", such as single or dual-core machines that are running dozens of modern web apps in parallel, so load is high. That might well explain why it was reported so quickly. It's quite noticeable with a standard load testing procedures (e.g. "how many parallel oscillators can this computer/OS/driver do at this latency and be glitch-less under load"). It is acceptable to do audio work off the main thread (as long as buffering is in place to prevent glitches from scheduling delays, and it's prioritized appropriately), but maybe you mean "on the main thread". In this case, it's not really recommended, but sometimes, it's what you need to do, for example when emulating an old console where everything is synchronous in the render loop. Locks and allocations are best avoided, but modern locks and allocators are amazing. If the lock is _not contended_, it's very fast, and it's fine. If the allocation comes from a thread-local pool (some allocators do that), it's also very fast and fine, but that needs to be true across OSes, etc. for us. In other case, which is most of the time, it's best to avoid those. We're not allocation nor lock-free entirely in Firefox though, but it's been thoroughly profiled - when know what we'll optimize when it's time to do so, but for now it's fine under extremely heavy load in a normal situation (we have locks when the audio devices are unplugged for an example of a "non-normal" situation). This is largely due to legacy (the Web Audio API, for example, is about 10 years old).