3 ms·

Running Firefox on Windows 10 x64, on a somewhat under-specced machine that tends to run low on RAM. I'm getting highly inconsistent note timings, with intermi

Running Firefox on Windows 10 x64, on a somewhat under-specced machine that tends to run low on RAM. I'm getting highly inconsistent note timings, with intermittent full 1-second pauses with no new notes being triggered. (FWIW, I also saw something similar in https://news.ycombinator.com/item?id=26870666 https://news.ycombinator.com/item?id=26870666 , and am getting choppy audio in Zoom that disappears as soon as I enable profiling...) Is it my browser's fault for having the JS event loop stall, or the page's fault for driving audio off the page's event loop? Or both? (IDK how to fix Firefox, other than increasing the content processes above 3, which will reduce cross-tab lag but possibly increase RAM, or to reduce my extensions or find the one at fault.) On a related note, how critical is it to follow the "audio on a real-time thread" dogma (http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...)? I've heard that not using Electron, writing real-time code, avoiding mutexes and allocations, mlock to avoid swapping, tuning OSes and picking processors and audio interfaces based on its low-latency properties, etc. is important for live performances, but not necessary for MIDI instrument input for (casual home users? studio work?).

drumnickydrum 5y ago
How many other open tabs and applications? Without a doubt I don't expect excessive overhead from the browser and would encourage users experiencing serious performance issues to wipe out everything else going on. This is still an experiment in just how much we can do in the browser. It would be a fun challenge for me to programmatically figure out if the user is experiencing issues and disabled features to improve performance...
jeroenhd 5y ago
I can reproduce the audio stutter by having Firefox do anything performance intensive (starting recording a performance profile does the trick nicely) and I seem to be getting severe frame drops because of garbage collection. I think the flows of React and Redux are just not built for timing-sensitive applications like these. A lot of allocations and deallocations are taking place without any user input. Looking at the source code, the many selectors and other hooks probably has something to do with that. Redux is also very explicit about treating objects as immutable, causing a lot of duplication as a result. For a simple web application this shouldn't be too much of a problem but again, this application is very sensitive to timing issues where 50ms of GC can cause a noticeable performance degradation. It's not that the code is bad, it seems quite well memo-ised to get decent performance; I think it's a result of using the wrong too for the job, nothing more. Quite impressive that a framework as heavy as React can be made this responsive in my opinion. If you run into any memory troubles, check about:performance and about:memory, you may be able to find what's causing the memory leaks.
- nyanpasu64 5y ago
  In the past I've found about:memory to be almost useless for pinpointing leaks. It's segmented into processes and types of allocations, and only the deeply nested sub-entries say which tab the memory belongs to. And even then, I don't know whether Firefox, the site, or an extension is causing memory bloat. I end up resorting to just blindly closing tabs or restarting the browser. I might try about:performance more, but in the past I recall it being similarly almost useless for pinpointing sites eating CPU.
padenot 5y ago
I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in literate programming in a way that will _never_ glitch: https://blog.paul.cx/post/metronome/ https://blog.paul.cx/post/metronome/. The result is nowhere as interesting as this app, but this is to get the point across. https://www.html5rocks.com/en/tutorials/audio/scheduling/ https://www.html5rocks.com/en/tutorials/audio/scheduling/ is an article written by my former fellow co-editor of the Web Audio API specification, that talks in length about this. In 2021, with the AudioWorklet [0] being available in all major browsers (including Safari as of a few weeks ago!), arbitrary computations can happen on the real-time audio thread. It is now possible to send a copy of the sequencer data (which note when, what sample, what velocity, etc.) to the real-time thread, and let it play, unaffected by the load of the machine, be it from the browser or other programs (until everything breaks down because you're also overloading the audio thread). I think this app is very good and does the best it can under the programming model it's using, considering the environment it's being run in, but if more robustness is needed, there is now a way ! On your last paragraph, it is still _absolutely_ critical. An audio software that doesn't use a real-time thread for rendering cannot render audio without glitches, even under no load, even on a fast machine. You don't need to tune the OS [1] and pick special hardware these days: today's computers (and even computers from 10 years ago or older) are perfectly capable or running reasonably complex audio processing workloads _if the software is written correctly_. This is regardless of the use case. When we disabled the real-time thread on Windows by mistake in Firefox (adjusting priority of the process level, and then that was inherited to all threads, very quickly reverted), we had reports in a matter of _days_ (if not the next day), and the Firefox Nightly population is not that big. This was I think playing regular videos on social media, not even anything complicated. If the author of the app, or anybody, wants to profile the app with the right tools, I've written a blog post [2] about this as well. I hear this tooling is used by audio professionals, now, and that they're happy with it. Happy to hear about any suggestion though :-). For your Firefox-specific question, you're right that increasing the content process count will make everything better, and we're constantly improving memory usage so that it's possible to do so even on lower-specs machines, maybe it's worth a try. Please look at the "resident memory" values, and not, say, virtual or shared, because we make heavy use of shared memory and copy on write facilities of the OS to limit the memory usage when using multiple process (which we need to do for origin isolation and avoiding Spectre issues and sandboxing various bits of the browser as hard as we can, for example). [0]: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle... [1]: (but you can to improve robustness further, and it helps to have good drivers, such as Jack on Linux, WASAPI with MMCSS on Windows or ASIO, or just the normal stuff on macOS) [2]: https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ https://blog.paul.cx/post/profiling-firefox-real-time-media-...

drumnickydrum 5y ago

How many other open tabs and applications? Without a doubt I don't expect excessive overhead from the browser and would encourage users experiencing serious performance issues to wipe out everything else going on. This is still an experiment in just how much we can do in the browser. It would be a fun challenge for me to programmatically figure out if the user is experiencing issues and disabled features to improve performance...

jeroenhd 5y ago

I can reproduce the audio stutter by having Firefox do anything performance intensive (starting recording a performance profile does the trick nicely) and I seem to be getting severe frame drops because of garbage collection. I think the flows of React and Redux are just not built for timing-sensitive applications like these. A lot of allocations and deallocations are taking place without any user input. Looking at the source code, the many selectors and other hooks probably has something to do with that. Redux is also very explicit about treating objects as immutable, causing a lot of duplication as a result. For a simple web application this shouldn't be too much of a problem but again, this application is very sensitive to timing issues where 50ms of GC can cause a noticeable performance degradation. It's not that the code is bad, it seems quite well memo-ised to get decent performance; I think it's a result of using the wrong too for the job, nothing more. Quite impressive that a framework as heavy as React can be made this responsive in my opinion. If you run into any memory troubles, check about:performance and about:memory, you may be able to find what's causing the memory leaks.

nyanpasu64 5y ago
In the past I've found about:memory to be almost useless for pinpointing leaks. It's segmented into processes and types of allocations, and only the deeply nested sub-entries say which tab the memory belongs to. And even then, I don't know whether Firefox, the site, or an extension is causing memory bloat. I end up resorting to just blindly closing tabs or restarting the browser. I might try about:performance more, but in the past I recall it being similarly almost useless for pinpointing sites eating CPU.

padenot 5y ago

I work on the Web Audio API at Mozilla, here is an article that I wrote that explains what goes wrong here, distilled to the core: implementing a metronome in literate programming in a way that will _never_ glitch: https://blog.paul.cx/post/metronome/ https://blog.paul.cx/post/metronome/. The result is nowhere as interesting as this app, but this is to get the point across. https://www.html5rocks.com/en/tutorials/audio/scheduling/ https://www.html5rocks.com/en/tutorials/audio/scheduling/ is an article written by my former fellow co-editor of the Web Audio API specification, that talks in length about this. In 2021, with the AudioWorklet [0] being available in all major browsers (including Safari as of a few weeks ago!), arbitrary computations can happen on the real-time audio thread. It is now possible to send a copy of the sequencer data (which note when, what sample, what velocity, etc.) to the real-time thread, and let it play, unaffected by the load of the machine, be it from the browser or other programs (until everything breaks down because you're also overloading the audio thread). I think this app is very good and does the best it can under the programming model it's using, considering the environment it's being run in, but if more robustness is needed, there is now a way ! On your last paragraph, it is still _absolutely_ critical. An audio software that doesn't use a real-time thread for rendering cannot render audio without glitches, even under no load, even on a fast machine. You don't need to tune the OS [1] and pick special hardware these days: today's computers (and even computers from 10 years ago or older) are perfectly capable or running reasonably complex audio processing workloads _if the software is written correctly_. This is regardless of the use case. When we disabled the real-time thread on Windows by mistake in Firefox (adjusting priority of the process level, and then that was inherited to all threads, very quickly reverted), we had reports in a matter of _days_ (if not the next day), and the Firefox Nightly population is not that big. This was I think playing regular videos on social media, not even anything complicated. If the author of the app, or anybody, wants to profile the app with the right tools, I've written a blog post [2] about this as well. I hear this tooling is used by audio professionals, now, and that they're happy with it. Happy to hear about any suggestion though :-). For your Firefox-specific question, you're right that increasing the content process count will make everything better, and we're constantly improving memory usage so that it's possible to do so even on lower-specs machines, maybe it's worth a try. Please look at the "resident memory" values, and not, say, virtual or shared, because we make heavy use of shared memory and copy on write facilities of the OS to limit the memory usage when using multiple process (which we need to do for origin isolation and avoiding Spectre issues and sandboxing various bits of the browser as hard as we can, for example). [0]: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle... [1]: (but you can to improve robustness further, and it helps to have good drivers, such as Jack on Linux, WASAPI with MMCSS on Windows or ASIO, or just the normal stuff on macOS) [2]: https://blog.paul.cx/post/profiling-firefox-real-time-media-workloads/ https://blog.paul.cx/post/profiling-firefox-real-time-media-...