4 ms·

A tale of two clocks – Scheduling Web Audio with precision

40 points by lioeters 4y ago

errozero 4y ago
I made a library based on this article which I use as a starting point for all my music app projects. It's useful as the main timing code for things like drum machine and sequencers etc. https://github.com/errozero/beatstepper https://github.com/errozero/beatstepper
- bambax 4y ago
  Interesting! The way I understand it, Web Audio API only lets one schedule audio source nodes with start and stop methods. As I needed to schedule something that was not audio related, I ended up creating silent oscillators of almost zero length and relying on the 'ended' event. But running a scheduled callback is better! It's not immediately clear to me how you do it. Can you maybe explain your code a little?
  - errozero 4y ago
    Hey, sure I'll try... if you mean usage of the library: It runs your callback every 16th note, slightly before it is actually due to play, this can vary by a few milliseconds each time but that doesn't matter as the actual audio context start time for the note is passed in to the callback, and you use that to actually schedule the events for that 16th note, eg: osc.start(time). You can schedule 32nd notes etc too by using the stepLength property that is also passed in, time + (stepLength/2) would be a 32nd note. Hope that makes sense? I do need to write a better description on the github page of what it actually is. The inner workings of the library itself are mostly just as described in the article with a few tweaks.
flohofwoe 4y ago
This really needs a (2013) in the title (or at least a "pre-Spectre/Meltdown"). The "up-and-coming" (back in 2013) high-resolution timer performance.now() exists on all browsers for a long time by now, but is clamped to milliseconds on some browsers because of the Spectre/Meltdown mitigations. The web audio timer is also affected by this precision reduction, but I don't know what's the current situation across browsers (I was actually hoping that the article would talk about this). Anti-fingerprinting settings can also heavily reduce timer precision. In short: timing on the web platform is (currently) an incredible mess.
- taf2 4y ago
  Add to that energy saving features that suspend tabs and you can get variable length, setTimeout(() => { console.log('timers'); }, 1); ... ... ... ... timers see: https://developer.chrome.com/blog/timer-throttling-in-chrome-88/ https://developer.chrome.com/blog/timer-throttling-in-chrome...
- pistachiopro 4y ago
  I believe all the major browser vendors now support the notion of "secure context," where you need to use TLS, <Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp> headers, and maybe a couple more requirements. They say you regain access to high precision timers, as well as SharedArrayBuffers and anything else that got lost during the Spectre/Meltdown mitigations. It appears to work in my limited testing. This secure context makes it harder to support various random ad networks, but if you're deploying your app via the web and don't need to monetize it by ads, it's a way forward.
  - flohofwoe 4y ago
    It also doesn't work on popular web hosters that don't allow to change the web server configuration (like Github Pages). (there is a workaround however by injecting the required response headers on the client side, but who knows how long that's going to work: http://stefnotch.github.io/web/COOP%20and%20COEP%20Service%20Worker/ http://stefnotch.github.io/web/COOP%20and%20COEP%20Service%2...)
kierank 4y ago
"The Web Audio API exposes access to the audio subsystem’s hardware clock." I can't speak for Windows and Mac but in ALSA this isn't possible.
- eqvinox 4y ago
  > in ALSA this isn't possible. The ALSA documentation seems to disagree with you? https://www.alsa-project.org/alsa-doc/alsa-lib/group___timer.html https://www.alsa-project.org/alsa-doc/alsa-lib/group___timer... (disclaimer: I have not used this API and don't know if it is missing some functionality — is it?)
aaaaaaaaaaab 4y ago
If it wasn’t clear from the article, the web is not ready for any kind of low-latency interactive audio yet.
- flohofwoe 4y ago
  The article is from 2013 and very outdated, in the meantime there are Audio Worklets which improve the situation somewhat: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle... ...doesn't change the fact that WebAudio is an incredibly overengineered contraption of course.
  - CharlesW 4y ago
    For anyone else curious about browser support for the AudioWorklet API: https://caniuse.com/mdn-api_audioworklet https://caniuse.com/mdn-api_audioworklet
eqvinox 4y ago
> This clock is exposed on the AudioContext object through its .currentTime property, as a floating-point number of seconds since the AudioContext was created. [...] Since there are around 15 decimal digits of precision in a “double”, even if the audio clock has been running for days, it should still have plenty of bits left over to point to a specific sample even at a high sample rate. It is mindboggling to me that a floating-point representation was designed into this. The required precision is well known and won't change. It is also well understood that this precision is required "locally" at possibly large future values. Even having the value wrap is mostly not a problem. Floating point is the opposite of designing for these things… (and no other audio API I'm aware of does this.) (Okay, after doing some quick math, a double precision float should be OK for more than 100 years. The mismatch still irks me…)
- bradknowles 4y ago
  Just because you have lots of digits available doesn't mean you actually have that much precision or accuracy.
- flohofwoe 4y ago
  Javascript doesn't have integer types, only a 'number' type which is equivalent to a 'double', so even if the clock would use an integer representation internally it would need to be exposed as a floating point value in the WebAudio API.
iostream24 4y ago
This is somewhat insane. WHY is no one looking at how protocols explicitly invented to solve this problem work? PTP precision time protocol, albeit implemented on the wrong layer to be directly useful here, solves it by timestamping frames.
- radiowave 4y ago
  It's not clear to me that any of the challenges the article addresses could be improved by the use of a clock synchronization protocol, such as PTP - it is not within the gift of application software to modify the audio sample rate clock, nor the display refresh rate. Beyond what the article discusses, I can imagine long-running audio applications where it might be useful to build an estimate of the rate at which the system time drifts relative to the sample rate clock. Even so, we're still in the realm of the application software having to take these slightly divergent timing sources as given.
  - kierank 4y ago
    Or just have a way for a chipset to expose it's clock instead of all the methods people use to guess using buffer fullness.
- tomduncalf 4y ago
  Isn’t timestamping frames (events) essentially what osc.start/stop (in this example) is doing, by allowing you to pass a timestamp in the future for an event which will then be internally scheduled on the audio thread? The issue really is that all your code is running on the UI thread with JavaScript, so you can’t accurately time stuff, whereas in C++ you’d write this stuff to execute on the audio thread callback. That said, this is now possible with AudioWorklets - but I don’t think you could interact in a sample accurate way with browser-supplied WebAudio nodes from there, so ultimately you still end up needing to schedule them for a future time if accuracy matters.
- flohofwoe 4y ago
  This would be way overengineered, all that's needed is a local monotonic time source with a good enough precision of somewhere between 1 to 10 microseconds. There's no need to synchronize the time with other sources, unless the goal is to control peripheral devices (which is out of scope for WebAudio anyway). The other problem of (traditional) WebAudio is that all work needs to happen on the browser thread which runs in time slices at a very low frequency (usually 60Hz or the display refresh rate). There's now a solution called audio worklets, but those aren't exactly trivial to work with unless all work can happen in the worklet thread (for instance you need to write your own ring buffer to push commands or sample data from the browser to the audio thread).

40 points by lioeters 4y ago

I made a library based on this article which I use as a starting point for all my music app projects. It's useful as the main timing code for things like drum machine and sequencers etc. https://github.com/errozero/beatstepper https://github.com/errozero/beatstepper

bambax 4y ago
Interesting! The way I understand it, Web Audio API only lets one schedule audio source nodes with start and stop methods. As I needed to schedule something that was not audio related, I ended up creating silent oscillators of almost zero length and relying on the 'ended' event. But running a scheduled callback is better! It's not immediately clear to me how you do it. Can you maybe explain your code a little?
- errozero 4y ago
  Hey, sure I'll try... if you mean usage of the library: It runs your callback every 16th note, slightly before it is actually due to play, this can vary by a few milliseconds each time but that doesn't matter as the actual audio context start time for the note is passed in to the callback, and you use that to actually schedule the events for that 16th note, eg: osc.start(time). You can schedule 32nd notes etc too by using the stepLength property that is also passed in, time + (stepLength/2) would be a 32nd note. Hope that makes sense? I do need to write a better description on the github page of what it actually is. The inner workings of the library itself are mostly just as described in the article with a few tweaks.

flohofwoe 4y ago

This really needs a (2013) in the title (or at least a "pre-Spectre/Meltdown"). The "up-and-coming" (back in 2013) high-resolution timer performance.now() exists on all browsers for a long time by now, but is clamped to milliseconds on some browsers because of the Spectre/Meltdown mitigations. The web audio timer is also affected by this precision reduction, but I don't know what's the current situation across browsers (I was actually hoping that the article would talk about this). Anti-fingerprinting settings can also heavily reduce timer precision. In short: timing on the web platform is (currently) an incredible mess.

taf2 4y ago
Add to that energy saving features that suspend tabs and you can get variable length, setTimeout(() => { console.log('timers'); }, 1); ... ... ... ... timers see: https://developer.chrome.com/blog/timer-throttling-in-chrome-88/ https://developer.chrome.com/blog/timer-throttling-in-chrome...
pistachiopro 4y ago
I believe all the major browser vendors now support the notion of "secure context," where you need to use TLS, <Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp> headers, and maybe a couple more requirements. They say you regain access to high precision timers, as well as SharedArrayBuffers and anything else that got lost during the Spectre/Meltdown mitigations. It appears to work in my limited testing. This secure context makes it harder to support various random ad networks, but if you're deploying your app via the web and don't need to monetize it by ads, it's a way forward.
- flohofwoe 4y ago
  It also doesn't work on popular web hosters that don't allow to change the web server configuration (like Github Pages). (there is a workaround however by injecting the required response headers on the client side, but who knows how long that's going to work: http://stefnotch.github.io/web/COOP%20and%20COEP%20Service%20Worker/ http://stefnotch.github.io/web/COOP%20and%20COEP%20Service%2...)

kierank 4y ago

"The Web Audio API exposes access to the audio subsystem’s hardware clock." I can't speak for Windows and Mac but in ALSA this isn't possible.

eqvinox 4y ago
> in ALSA this isn't possible. The ALSA documentation seems to disagree with you? https://www.alsa-project.org/alsa-doc/alsa-lib/group___timer.html https://www.alsa-project.org/alsa-doc/alsa-lib/group___timer... (disclaimer: I have not used this API and don't know if it is missing some functionality — is it?)

aaaaaaaaaaab 4y ago

If it wasn’t clear from the article, the web is not ready for any kind of low-latency interactive audio yet.

flohofwoe 4y ago
The article is from 2013 and very outdated, in the meantime there are Audio Worklets which improve the situation somewhat: https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet https://developer.mozilla.org/en-US/docs/Web/API/AudioWorkle... ...doesn't change the fact that WebAudio is an incredibly overengineered contraption of course.
- CharlesW 4y ago
  For anyone else curious about browser support for the AudioWorklet API: https://caniuse.com/mdn-api_audioworklet https://caniuse.com/mdn-api_audioworklet

eqvinox 4y ago

> This clock is exposed on the AudioContext object through its .currentTime property, as a floating-point number of seconds since the AudioContext was created. [...] Since there are around 15 decimal digits of precision in a “double”, even if the audio clock has been running for days, it should still have plenty of bits left over to point to a specific sample even at a high sample rate. It is mindboggling to me that a floating-point representation was designed into this. The required precision is well known and won't change. It is also well understood that this precision is required "locally" at possibly large future values. Even having the value wrap is mostly not a problem. Floating point is the opposite of designing for these things… (and no other audio API I'm aware of does this.) (Okay, after doing some quick math, a double precision float should be OK for more than 100 years. The mismatch still irks me…)

bradknowles 4y ago
Just because you have lots of digits available doesn't mean you actually have that much precision or accuracy.
flohofwoe 4y ago
Javascript doesn't have integer types, only a 'number' type which is equivalent to a 'double', so even if the clock would use an integer representation internally it would need to be exposed as a floating point value in the WebAudio API.

iostream24 4y ago

This is somewhat insane. WHY is no one looking at how protocols explicitly invented to solve this problem work? PTP precision time protocol, albeit implemented on the wrong layer to be directly useful here, solves it by timestamping frames.

radiowave 4y ago
It's not clear to me that any of the challenges the article addresses could be improved by the use of a clock synchronization protocol, such as PTP - it is not within the gift of application software to modify the audio sample rate clock, nor the display refresh rate. Beyond what the article discusses, I can imagine long-running audio applications where it might be useful to build an estimate of the rate at which the system time drifts relative to the sample rate clock. Even so, we're still in the realm of the application software having to take these slightly divergent timing sources as given.
- kierank 4y ago
  Or just have a way for a chipset to expose it's clock instead of all the methods people use to guess using buffer fullness.
tomduncalf 4y ago
Isn’t timestamping frames (events) essentially what osc.start/stop (in this example) is doing, by allowing you to pass a timestamp in the future for an event which will then be internally scheduled on the audio thread? The issue really is that all your code is running on the UI thread with JavaScript, so you can’t accurately time stuff, whereas in C++ you’d write this stuff to execute on the audio thread callback. That said, this is now possible with AudioWorklets - but I don’t think you could interact in a sample accurate way with browser-supplied WebAudio nodes from there, so ultimately you still end up needing to schedule them for a future time if accuracy matters.
flohofwoe 4y ago
This would be way overengineered, all that's needed is a local monotonic time source with a good enough precision of somewhere between 1 to 10 microseconds. There's no need to synchronize the time with other sources, unless the goal is to control peripheral devices (which is out of scope for WebAudio anyway). The other problem of (traditional) WebAudio is that all work needs to happen on the browser thread which runs in time slices at a very low frequency (usually 60Hz or the display refresh rate). There's now a solution called audio worklets, but those aren't exactly trivial to work with unless all work can happen in the worklet thread (for instance you need to write your own ring buffer to push commands or sample data from the browser to the audio thread).