5 ms·
(reposting my comment from the original) There’s some potentially misleading information here. Background: I’ve spent the last 20+ years writing low-latency re
by PaulDavisThe1st 7y ago
(reposting my comment from the original)
There’s some potentially misleading information here. Background: I’ve spent the last 20+ years writing low-latency realtime audio applications, technically cross-platform but focused on Linux.
If you care about low latency on any general purpose OS, you need to use a realtime scheduling policy. The default scheduling on these OS’s is intended to maximise some combination of bandwidth and/or fairness. Low latency requires ditching both of those in favor of limiting the maximum scheduling delay of a thread that is otherwise ready to run.
Measuring how long synchronization primitives take without SCHED_FIFO is illustrative, but only of why, if you care about scheduling latency, you need SCHED_FIFO. There are several alternative schedulers for Linux – none of them remove the need for SCHED_FIFO if latency is important.
It is absolutely not the case that using SCHED_FIFO automatically starves non-SCHED_FIFO threads. Scheduling policy is set per-thread, and SCHED_FIFO will only cause issues if the threads that use it really do “burn the CPU” (e.g. by using spinlocks). If you combine SCHED_FIFO with spinlocks you need to be absolutely certain that the locks have low contention and/or are held for extremely short periods (preferably just a few instructions). If you use mutexes (which ultimately devolve to futexes at the kernel level), the kernel will take care of you a little better, unless your SCHED_FIFO thread doesn’t block – if it doesn’t do that, that’s entirely on you. Blocking means making some sort of system call that will cause the scheduler to put the thread to sleep – could be a wait on a futex, waiting for data, or an explicit sleep.
In particular, this: “This was known for a while simply because audio can stutter on Linux when all cores are busy (which doesn’t happen on Windows)” is NOT true. Linux actually has better audio performance than Windows or macOS, but only if the app developer understands a few basic principles. One of them is using SCHED_FIFO appropriately.
Pro-audio/music creation scheduling is MUCH more demanding than video, and a bit more demanding than games. Linux handles this stuff just fine – it just gives you enough rope to shoot yourself in the foot if you don’t fully understand what you’re doing.
- segfaultbuserr 7y agoFor the same reason, by default, PulseAudio uses rtkit to escalate its privileges and set its scheduler policy to SCHED_FIFO (then drop the privilege). Many people don't understand it and see it as another example of how Lennart's software is taking over the world (especially when you use a distribution like Arch that didn't already have rtkil configured for you), but it's essential for good audio latency (but the audio player needs SCHED_FIFO as well, there's nothing to help if the player itself is locking up). > SCHEDULING > high-priority= Renice the daemon after startup to become a high-priority process. This a good idea if you experience drop-outs during playback. However, this is a certain security issue, since it works when called SUID root only, or RLIMIT_NICE is used. root is dropped immediately after gaining the nice level on startup, thus it is presumably safe. See pulseaudio(1) for more information. > Takes a boolean argument, defaults to yes. The --high-priority command line option takes precedence. > realtime-scheduling= Try to acquire SCHED_FIFO scheduling for the IO threads. The same security concerns as mentioned above apply. However, if PA enters an endless loop, realtime scheduling causes a system lockup. Thus, realtime scheduling should only be enabled on trusted machines for now. Please note that only the IO threads of PulseAudio are made real-time. The controlling thread is left a normally scheduled thread. Thus enabling the high-priority option is orthogonal. See pulseaudio(1) for more information. > Takes a boolean argument, defaults to yes. The --realtime command line option takes precedence.
- PaulDavisThe1st 7y agoOn modern Linux systems, it is impossible for SCHED_FIFO to cause system lockups. A configurable percentage of CPU cycles is reserved by the (default) scheduler for non-SCHED_FIFO threads, so even a SCHED_FIFO thread that runs "while (true){}" will not be allowed to use 100% of a core's cycles. RLIMIT_NICE is of zero use with realtime audio. This documentation is based on (1) the state of Linux circa 2008 or so (2) a complete misunderstanding of the role of nice values for scheduling (hint; it's only useful for altering the behavior of non-RT threads, which means threads that do not have hard or semi-hard timing deadlines). It is a notable problem that there is so much out of date information about the use of nice with audio applications still circulating online. It was never true, and it's even less true today.
- segfaultbuserr 7y agoGlad to see someone with industrial experience on realtime processing to respond. > a complete misunderstanding of the role of nice values for scheduling (hint; it's only useful for altering the behavior of non-RT threads, which means threads that do not have hard or semi-hard timing deadlines). I suspect you're misreading the quoted documentation, I believe the quoted documentation is fine. To recite, "only the IO threads of PulseAudio are made real-time. The controlling thread is left a normally scheduled thread. Thus enabling the high-priority option is orthogonal (to SCHED_FIFO)". Nice is applied to the controller responsible for other tasks, SCHED_FIFO is applied to the I/O threads, independently. > On modern Linux systems, it is impossible for SCHED_FIFO to cause system lockups. [...] so even a SCHED_FIFO thread that runs "while (true){}" will not be allowed to use 100% of a core's cycles. Really? The rtkit documentation mentioned, > If processes that have real-time scheduling privileges enter a busy loop they can freeze the entire the system. To make sure such run-away processes cannot do this RLIMIT_RTTIME has been introduced. Being a per-process limit it is however easily cirumvented by combining a fork bomb with a busy loop. > RealtimeKit hands out RT scheduling to specific threads that ask for it -- but only to those and due to SCHED_RESET_ON_FORK it can be sure that this won't 'leak'. > In contrast to RLIMIT_RTPRIO, the RealtimeKit logic makes sure that only a certain number of threads can be made realtime, per user, per process and per time interval. So "while (true){}" won't lock your system up, but "while (true) { fork(); }" certainly will, so the rational of only using SCHED_FIFO on trusted processes is reasonable. I think it's still true. Is it?
- xenadu02 7y ago> Linux actually has better audio performance than [...] macOS That seems unlikely to me and doesn't match any experience I've ever had or known anyone to have. Can you explain what you mean by that?
- barrkel 7y agoWindows automatically boosts the prio of threads playing audio, basically treats them as realtime in its prio hierarchy. What I expect GP is talking about is minimum latency achievable, i.e. that with focused tuning, you can outperform Windows. Regular users of not so tuned apps probably experience this differently because Windows tries to do the tuning automatically and Linux doesn't. The Windows scheduler has other issues, like random prio boosting to escape from prio inversion.
- xenadu02 7y agoRight, I'm aware of the issues on Windows. I was asking about XNU-based platforms which are known for having very low latency and excellent realtime audio perf out of the box. (Granted, the out of box behavior and the achievable latency are two separate issues)
- jcelerier 7y ago> Right, I'm aware of the issues on Windows. I was asking about XNU-based platforms which are known for having very low latency and excellent realtime audio perf out of the box. I have never seen a mac box achieving reliable lower latency than Linux or Windows on the same hardware. Macs are nice for the plug'n'play experience - even though more often than not you still want to install your soundcard's driver. But when it comes to doing actual work... https://www.dawbench.com/win7-v-osx-6.htm https://www.dawbench.com/win7-v-osx-6.htm
- AdmiralBumbleB 7y agoI'm not sure that posting 9 year old benchmarks is particularly useful, particularly when those results have been heavily debated. If we're talking pure latency then UA, Focusrite, Apogee and Avid have lower RTL for the same buffer size in macOS. RME is identical on both platforms and I'm unsure about antelope. Regarding loads before under-runs that's a very tough thing to test, as it varies between products. GENERALLY: Comparing Digital Performer between platforms heavily favours macOS. Comparing Pro Tools favours macOS (currently). Comparing Reaper favours windows slightly. Cubase favours Windows... etc... There's also load type that factors in, as with macs you get a specific build with minimal leeway. Windows builds can be planned for a given build (highly serial, highly parallel, RAM hungry, Latency performant etc...). It's not too hard to build a test suite that favours macOS everywhere on the same machine, likewise for Windows. But to the original topic, latency, macOS is generally superior in this regard when looking at RTL specs for the same device.
- Intermernet 7y agoOff topic, but thanks to you and the JACK team! I think it's one of the most underrated pieces of software around. I have used it many times over the years and I'm constantly amazed at how good it is, even compared to industry standard proprietary stuff.
- jcelerier 7y ago> Blocking means making some sort of system call that will cause the scheduler to put the thread to sleep – could be a wait on a futex, waiting for data, or an explicit sleep. Speaking of which, do you know of a tool that would allow to ensure that e.g. your RT threads won't do bad things - short of putting a breakpoint on every syscall-like thing ? I wonder if at that point the audio community should not look into developing some clang-based static analyzer to enforce that.
- PaulDavisThe1st 7y agoThere have been several such tools over the years - I regret that off the top of my head I don't recall the names of any of them. Clang can be used for this but AFAIK you need to annotate functions/methods to indicate whether or not they are intended to be RT-safe. This gets complex if you have code that can be used in an RT context and in a non-RT context. The annotation part would be relatively simple in, say, a small audio player. In a DAW, it would be a substantial and complex task.
- richardwhiuk 7y agoI've done things before where you shim in a library (using LD_PRELOAD) between the executable and the C standard library, and blacklist function calls which are potentially blocking, (abort if they get called in the wrong context). It's usually sufficient to do this in testing, and then have as is, in production.
- temac 7y agoAlso, a yielded thread has absolutely no reason to be woken up with low latency, and this has nothing to do with the supposed scheduler quality. I'm not sure a kernel blindly prioritizing otherwise cpu-bound thread just because they yielded would give good results in the general case, esp out of microbenchmarks. If you want low wake up latency, you need a proper design, not mere reliance on magic that does not exist. Plus highly contended critical sections should really really be avoided. So I'm not sure the author's benchmark even make any sense: if all of your workload is serialized, use a single thread.