3 ms·

As for contributing to existing projects, I hear you — I've been highly active in Open Source for well over a decade. Conceptually, the closest thing to what I

As for contributing to existing projects, I hear you — I've been highly active in Open Source for well over a decade. Conceptually, the closest thing to what I want to do is Pure Data — and I have in fact contributed to it. But I'm also extremely motivated and willing to go all the way down and write the entire thing from scratch if I have to. Or to learn enough about every last step in the chain that I can actually control for latency — and it may be that it takes about the same amount of work. Finding numbers I can trust on latency seems hopeless. Everybody fudges rather than fails.

PaulDavisThe1st 6y ago
The word "latency" covers many things, many subtly different from each other. One of its meanings, unrelated to the one you're talking about, is the delay in signal flow caused by algorithms (many digital filters, for example). Often called "plugin delay compensation", or more generically "latency compensation". Let me just point out that it has taken 20 years and a guy whose PhD thesis was about latency compensation in a DAW to finally "fix" this sort of "latency" in Ardour. This is a massively harder problem from a design perspective than the scheduling latency issues you've referred to. [ EDIT: to be fair, the actual correct solution to latency compensation didn't take 20 years to implement, more like a year or so when taking place within the context of a large existing code base. ]
- rectang 6y ago
  The complexity of solving such problems generally is actually what drives me to start from scratch: rather than solve an intractable general problem, instead limit the scope of where the project must run and what it must do. My perception is that I am not capable of reliably tuning a general purpose OS for a total system latency of under 10 msec. I can't give the total system a hard number and believe that it will obey; instead, I need to perform a lot of esoteric tweakery of subsystems I probably don't understand. The system won't alert me reliably when it fails, but will instead either drop out or just give me more latency than I asked for — and there are innumerable factors outside my control that could cause it to fail. However, the composition tool I want to create has a pretty small set of requirements, if I accept that it only need serve my particular use case. So how about I ensure that my app is the only thing running on the hardware, via unikernel, or RTOS, or even bare metal? Implementing all of my compositional requirements is probably easier and certainly more rewarding than tweaking latency parameters without having confidence that my results will be enduring or predictable. If it turns out that the knowledge I gain during that exercise allows me to control latency well enough and I can return to mainline operating systems and contribute to existing projects, all the better. I don't really want to go down this path, but I'm not willing to accept a tool that maybe-kinda-sorta-sometimes meets my absolute requirements.
  - PaulDavisThe1st 6y ago
    You still won't be safe against several/most of the causes outlined in the page from the Ardour manual that I linked to. SMIs, random hardware that locks the bus for too long ... out of the kernel's control. If you really want 10msec finger->ear (a goal that I would say is reasonable, though given many live performer's and mixing engineer's normal physical setups, probably excessive) and you want to guarantee it, it's not just the OS but the hardware you'll have to change. You cannot guarantee the required scheduling on general purpose (intel/amd) motherboards unless you take great care with the selection, and even then ... I've heard that financial services/investment firms are the main reason you can still buy mobos without SMIs, because their "latency" requirements would be broken by these interrupts. On the other hand, the "not-guaranteed" case with a reasonable mobo, sensible bus connected devices, an RT kernel and properly written user space software is going to work almost all of the time. Just no guarantees.
    - rectang 6y ago
      Paul, Thank you very much — for your ongoing work in Open Source audio, for being willing to engage at length in this thread, and for being straightforward about what the system can deliver. > If you really want 10msec finger->ear (a goal that I would say is reasonable, though given many live performer's and mixing engineer's normal physical setups, probably excessive) I worked in a recording studio for 6 years, including two years as a mastering engineer. Most of the sonic adjustments I would make during mastering and mixing fell below the threshold of perception — but when added together they would produce something well above the threshold of perception. There's nothing magic about this. I don't have "golden ears" (although because I've trained I can more quickly identify certain patterns than people who haven't trained). The point is simply that an aggregation of imperceptible changes can sum to a perceptible result. It's akin to why you perform intermediate processing in both video and audio at a higher resolution than the final delivery medium: otherwise an accumulation of small, possibly imperceptable degradations will cause perceivable degradation of the finished product. And so, I dispute the idea that just because there are other sources of latency, we should resign ourselves and accept substantial contributors to latency which fall below perceptual threshold. The only number that matters is the final sum of all latencies. > You cannot guarantee the required scheduling on general purpose (intel/amd) motherboards unless you take great care with the selection, and even then ... I've heard that financial services/investment firms are the main reason you can still buy mobos without SMIs, because their "latency" requirements would be broken by these interrupts. With this in mind, I will set aside one possibity I'd considered: writing for general purpose CPUs (e.g. multicore x86_64) outside of mainstream operating systems. Instead, while I'll continue prototyping the project on mainstream operating systems, I'll probably look more deeply into dedicated outboard DSP boards. > *On the other hand, the "not-guaranteed" case with a reasonable mobo, sensible bus connected devices, an RT kernel and properly written user space software is going to work almost all of the time. Just no guarantees. I appreciate how hard you've worked to achieve that. My question, then, is how can I be confident that I'm actually meeting these "almost-all-of-time-time" latency requirements? In my experience, most systems recommended that you lower the latency until you hear clicks and pops. That convention leaves me... dissatisfied. A dropout is a detectable event, and the monitoring system should surface it. Just as boggling when you have exacting standards is when the system falls back and delivers something subtly degraded without telling you, like changing the latency without warning because the system would otherwise go down. I understand why systems are designed to prefer degradation over failure, but for my purposes I need to know when it happens. Expecting me to monitor continuously for an effect which is at the threshold of perception, such as subtly increased latency, is draining — and ultimately unreasonable. We have meters for noise floors and red lights indicating that clipping occurred. What facilities exist to help me understand when the latency behaviors of my rig are not meeting my requirements?
jcelerier 6y ago
If you want to check, I have a small (and not very good tbh) benchmarking chapter in my thesis where I detail the steps I have followed to get fairly low scheduler tick times of my software - for simple graphs the best I could do was to go to less than 100 microseconds with ~20 microseconds of jitter (p.187) for a tick: https://tel.archives-ouvertes.fr/tel-01947309/document https://tel.archives-ouvertes.fr/tel-01947309/document ; also see the intro of section 10.4 with a couple relevant references. Of course that only benchmarks the part that you as a developer can have a meaningful impact on, not the time spent in JACK / ALSA / ... To give an anecdote, with the hardware mentioned at the beginning of said chapter I can reliably put JACK in a 16 samples buffer at 44100 (~0.7 ms) and do a few things without crackles. not a lot though :p
- PaulDavisThe1st 6y ago
  What audio hardware are you using that allows 16 samples per buffer? Must be something ancient, I think?
  - jcelerier 6y ago
    it is my pci-express multiface II, that I will cherish until death - it has been rocking hard for the last 11 years so far :-) so I'm hopeful I can get a bit more out of it !

PaulDavisThe1st 6y ago

The word "latency" covers many things, many subtly different from each other. One of its meanings, unrelated to the one you're talking about, is the delay in signal flow caused by algorithms (many digital filters, for example). Often called "plugin delay compensation", or more generically "latency compensation". Let me just point out that it has taken 20 years and a guy whose PhD thesis was about latency compensation in a DAW to finally "fix" this sort of "latency" in Ardour. This is a massively harder problem from a design perspective than the scheduling latency issues you've referred to. [ EDIT: to be fair, the actual correct solution to latency compensation didn't take 20 years to implement, more like a year or so when taking place within the context of a large existing code base. ]

rectang 6y ago
The complexity of solving such problems generally is actually what drives me to start from scratch: rather than solve an intractable general problem, instead limit the scope of where the project must run and what it must do. My perception is that I am not capable of reliably tuning a general purpose OS for a total system latency of under 10 msec. I can't give the total system a hard number and believe that it will obey; instead, I need to perform a lot of esoteric tweakery of subsystems I probably don't understand. The system won't alert me reliably when it fails, but will instead either drop out or just give me more latency than I asked for — and there are innumerable factors outside my control that could cause it to fail. However, the composition tool I want to create has a pretty small set of requirements, if I accept that it only need serve my particular use case. So how about I ensure that my app is the only thing running on the hardware, via unikernel, or RTOS, or even bare metal? Implementing all of my compositional requirements is probably easier and certainly more rewarding than tweaking latency parameters without having confidence that my results will be enduring or predictable. If it turns out that the knowledge I gain during that exercise allows me to control latency well enough and I can return to mainline operating systems and contribute to existing projects, all the better. I don't really want to go down this path, but I'm not willing to accept a tool that maybe-kinda-sorta-sometimes meets my absolute requirements.
- PaulDavisThe1st 6y ago
  You still won't be safe against several/most of the causes outlined in the page from the Ardour manual that I linked to. SMIs, random hardware that locks the bus for too long ... out of the kernel's control. If you really want 10msec finger->ear (a goal that I would say is reasonable, though given many live performer's and mixing engineer's normal physical setups, probably excessive) and you want to guarantee it, it's not just the OS but the hardware you'll have to change. You cannot guarantee the required scheduling on general purpose (intel/amd) motherboards unless you take great care with the selection, and even then ... I've heard that financial services/investment firms are the main reason you can still buy mobos without SMIs, because their "latency" requirements would be broken by these interrupts. On the other hand, the "not-guaranteed" case with a reasonable mobo, sensible bus connected devices, an RT kernel and properly written user space software is going to work almost all of the time. Just no guarantees.
  - rectang 6y ago
    Paul, Thank you very much — for your ongoing work in Open Source audio, for being willing to engage at length in this thread, and for being straightforward about what the system can deliver. > If you really want 10msec finger->ear (a goal that I would say is reasonable, though given many live performer's and mixing engineer's normal physical setups, probably excessive) I worked in a recording studio for 6 years, including two years as a mastering engineer. Most of the sonic adjustments I would make during mastering and mixing fell below the threshold of perception — but when added together they would produce something well above the threshold of perception. There's nothing magic about this. I don't have "golden ears" (although because I've trained I can more quickly identify certain patterns than people who haven't trained). The point is simply that an aggregation of imperceptible changes can sum to a perceptible result. It's akin to why you perform intermediate processing in both video and audio at a higher resolution than the final delivery medium: otherwise an accumulation of small, possibly imperceptable degradations will cause perceivable degradation of the finished product. And so, I dispute the idea that just because there are other sources of latency, we should resign ourselves and accept substantial contributors to latency which fall below perceptual threshold. The only number that matters is the final sum of all latencies. > You cannot guarantee the required scheduling on general purpose (intel/amd) motherboards unless you take great care with the selection, and even then ... I've heard that financial services/investment firms are the main reason you can still buy mobos without SMIs, because their "latency" requirements would be broken by these interrupts. With this in mind, I will set aside one possibity I'd considered: writing for general purpose CPUs (e.g. multicore x86_64) outside of mainstream operating systems. Instead, while I'll continue prototyping the project on mainstream operating systems, I'll probably look more deeply into dedicated outboard DSP boards. > *On the other hand, the "not-guaranteed" case with a reasonable mobo, sensible bus connected devices, an RT kernel and properly written user space software is going to work almost all of the time. Just no guarantees. I appreciate how hard you've worked to achieve that. My question, then, is how can I be confident that I'm actually meeting these "almost-all-of-time-time" latency requirements? In my experience, most systems recommended that you lower the latency until you hear clicks and pops. That convention leaves me... dissatisfied. A dropout is a detectable event, and the monitoring system should surface it. Just as boggling when you have exacting standards is when the system falls back and delivers something subtly degraded without telling you, like changing the latency without warning because the system would otherwise go down. I understand why systems are designed to prefer degradation over failure, but for my purposes I need to know when it happens. Expecting me to monitor continuously for an effect which is at the threshold of perception, such as subtly increased latency, is draining — and ultimately unreasonable. We have meters for noise floors and red lights indicating that clipping occurred. What facilities exist to help me understand when the latency behaviors of my rig are not meeting my requirements?

jcelerier 6y ago

If you want to check, I have a small (and not very good tbh) benchmarking chapter in my thesis where I detail the steps I have followed to get fairly low scheduler tick times of my software - for simple graphs the best I could do was to go to less than 100 microseconds with ~20 microseconds of jitter (p.187) for a tick: https://tel.archives-ouvertes.fr/tel-01947309/document https://tel.archives-ouvertes.fr/tel-01947309/document ; also see the intro of section 10.4 with a couple relevant references. Of course that only benchmarks the part that you as a developer can have a meaningful impact on, not the time spent in JACK / ALSA / ... To give an anecdote, with the hardware mentioned at the beginning of said chapter I can reliably put JACK in a 16 samples buffer at 44100 (~0.7 ms) and do a few things without crackles. not a lot though :p

PaulDavisThe1st 6y ago
What audio hardware are you using that allows 16 samples per buffer? Must be something ancient, I think?
- jcelerier 6y ago
  it is my pci-express multiface II, that I will cherish until death - it has been rocking hard for the last 11 years so far :-) so I'm hopeful I can get a bit more out of it !