6 ms·

Reminds me of how Windows Vista's "Multimedia Class Scheduler Service" would put a low cap on network throughput if any sound was playing: http://www.zdnet.com

Reminds me of how Windows Vista's "Multimedia Class Scheduler Service" would put a low cap on network throughput if any sound was playing: http://www.zdnet.com/article/follow-up-playing-music-severely-degrades-network-transfer-performance-in-vista/ http://www.zdnet.com/article/follow-up-playing-music-severel... Mark Russinovich justified it by explaining that the network interrupt routine was just too expensive to be able to guarantee no glitches in media playback, so it was limited to 10 packets per millisecond when any media was playing: https://blogs.technet.microsoft.com/markrussinovich/2007/08/26/vista-multimedia-playback-and-network-throughput/ https://blogs.technet.microsoft.com/markrussinovich/2007/08/... but obviously this is a pretty crappy one-size-fits-all prioritization scheme for something marketed as a most-sophisticated best-ever OS at the time: https://blogs.technet.microsoft.com/markrussinovich/2008/02/04/inside-vista-sp1-file-copy-improvements/ https://blogs.technet.microsoft.com/markrussinovich/2008/02/... Many people had perfectly consistent mp3 playback when copying files over the network 10 times as fast in other OSes (including Win XP!) Often a company will have a "sophisticated best-ever algorithm" and then put in a hacky lazy work-around for some problem, and obviously don't tell anyone about it. Sometimes the simpler less-sophisticated solution just works better in practice.

ShabbosGoy 9y ago
Thanks for the links and reading material. It's amazing how important the scheduler is in an OS. I'm no expert on OS-level programming, but couldn't they have just used a simple round robin scheduler?
- zeta0134 9y ago
  In this case the scheduler isn't as much of a problem as the interrupt service routine, which by its very nature interrupts the normal thread-level scheduler to do some work with the hardware. Both the Network Interface Card (or wireless chip) and the Audio hardware generate interrupts for a variety of reasons. Network cards generate interrupts when data is finished sending or receiving. Audio hardware generally generates its own interrupts when its buffers have emptied out, to signal the OS to provide it more audio to play. In this case, it sounds like Vista's network interrupt was running for so long that it was taking priority over other interrupts (audio) and causing them to not be serviced quickly enough. This is definitely a problem, but the fix should not have been to limit the amount of work done per interrupt, but instead to delegate that work to some other process with lower priority. It's weird to think of audio as higher priority than networking traffic, but in this case that actually makes a ton of sense. Audio generation only requires ~88.2 KB of data per second, but latency is much lower and less tolerable by a human listener. Modern network connections are working with multiple MB per second, and are (relatively) more tolerant of slight delays. Same with mouse and keyboard input, it's why your mouse tends to move smoothly on most Windows machines even when the rest of the OS is obviously struggling with some heavy task.
  - tinus_hn 9y ago
    On Windows, like other modern operating systems, a driver is supposed to do only the minimum amount of work possible in the interrupt handler, to avoid this problem. The real work is supposed to happen later when it can be scheduled.
    - AndrewGaspar 9y ago
      While it's true that networking device drivers on Windows defer all meaningful work to post-ISR, it only moves the goal posts a little bit. On Windows, a lot of network traffic processing (e.g. tcp/ip acknowledgment, virtual switch packet forwarding) is performed in a DPC[1] queued in response to the device's interrupt. A DPC generally runs at dispatch level, which gives it more-or-less exclusive access to the CPU until completion. This preempts the thread scheduler and can result in user-visible stutters in audio and video when the network load outpaces the CPU's ability to quickly service the network traffic. [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/introduction-to-dpcs https://docs.microsoft.com/en-us/windows-hardware/drivers/ke...
      - tinus_hn 9y ago
        These are hardly complex tasks, they could be done by computers from the 70s. It's quite possible that Windows makes a mess of it anyway, of course. But the top comment was about how interrupts from the network card interfere with interrupts from the sound card so it can't keep its buffer filled. That shouldn't happen. Unless the CPU just isn't fast enough to do networking and decode or generate the audio at the same time, but then it's never going to work properly.
- radicalbyte 9y ago
  Just look at the process you have running, 90% of them are doing nothing. Round Robin is a waste for an OS.
  - DSMan195276 9y ago
    Processes that are waiting on some event (like polling some file descriptors) will never be scheduled until they're woken up. It doesn't matter what type of scheduler you are using. The algorithm just decides which to run within the set of programs actually running.
amluto 9y ago
I thought Windows had a fancy interrupt priority system that should, in principle, allow sound playback to preempt network interrupts? AMD64 added a fancy feature (CR8 access to the task priority register) just to accelerate interrupt prioritization. (This is all very vague memory. I know how this stuff works on Linux. Linux does not have interrupt priorities.)
- sharkbot 9y ago
  I could be wrong, but my understanding of some sound cards is that they have essentially a single memory buffer that they are reading from when instructed to play a sound. Most sound cards let the OS split the buffer into two halves and raise an interrupt when one half of the buffer completes playing. Interrupt prioritization doesn't help much, because the sound data is likely being generated from user mode, and the playback complete interrupt is being handled in kernel mode, which would require a transition back for further processing. When receiving data from a network card, no transfer back to user mode is required and will have implicit priority over sound generation. Therefore, network processing is likely to starve out sound generation barring choices like those described. (This is somewhat hearsay, corrections welcome :))
  - xorblurb 9y ago
    Needing to wake up usermode is not necessarily an insurmountable obstacle; under Linux you can use a PREEMPT config. Storming a network card having unwanted consequences can also be mitigated: under Linux you have some network device driver that use an hybrid interrupt/polling approach with the NAPI. Combining both should theoretically allow a system to generate sound smoothly in the described conditions.
    - amluto 9y ago
      IMO you shouldn't really need to wake user mode that often. Just let user code buffer the output samples well ahead of the hardware needing them and have the kernel do the copies when needed.
cm2187 9y ago
Reminds me of Apple's Jim Reekes "engineers are retarded". https://m.youtube.com/watch?v=C5d151lqJsA https://m.youtube.com/watch?v=C5d151lqJsA (The annecdote really starts at 2:30).
- Systemic33 9y ago
  Haha, what a great video. To some extent he's onto something. Sometimes you are in this zone when everything will work if you just do this one quick little hack, and then the hack ends up not getting replaced with a proper solution, and in the end you end up with a harddrive right next to a speaker >.<
  - btschaegg 9y ago
    There actually seem to be people whose default mode of operation seems to lie entirely in that zone though. I'm pretty sure everyone arrives at a similar conclusion if they realise that attempts to fix a certain person's code always end in a express trip to the 6th stage of debugging [1]... [1]: http://plasmasturm.org/log/6debug/ http://plasmasturm.org/log/6debug/
    - jzwinck 9y ago
      Definitely. And that "quick hack by default" person will often end up being the favorite of management because they get so many things "done." So it's important to figure out what kind of manager you have, and try not to be the slow guy on a team full of quick-hack people. You won't look good and nobody will be happy.
      - chii 9y ago
        > the slow guy on a team full of quick-hack people isn't that called 'cultural fit'?
        deleted 9y ago
        [deleted]
adsfqwop 9y ago
Yeah thanks Microsoft. They apparently never thought of someone playing mp3's from a network drive!! It drove me nuts for months, until I figured out exactly what you just posted.

ShabbosGoy 9y ago

Thanks for the links and reading material. It's amazing how important the scheduler is in an OS. I'm no expert on OS-level programming, but couldn't they have just used a simple round robin scheduler?

zeta0134 9y ago
In this case the scheduler isn't as much of a problem as the interrupt service routine, which by its very nature interrupts the normal thread-level scheduler to do some work with the hardware. Both the Network Interface Card (or wireless chip) and the Audio hardware generate interrupts for a variety of reasons. Network cards generate interrupts when data is finished sending or receiving. Audio hardware generally generates its own interrupts when its buffers have emptied out, to signal the OS to provide it more audio to play. In this case, it sounds like Vista's network interrupt was running for so long that it was taking priority over other interrupts (audio) and causing them to not be serviced quickly enough. This is definitely a problem, but the fix should not have been to limit the amount of work done per interrupt, but instead to delegate that work to some other process with lower priority. It's weird to think of audio as higher priority than networking traffic, but in this case that actually makes a ton of sense. Audio generation only requires ~88.2 KB of data per second, but latency is much lower and less tolerable by a human listener. Modern network connections are working with multiple MB per second, and are (relatively) more tolerant of slight delays. Same with mouse and keyboard input, it's why your mouse tends to move smoothly on most Windows machines even when the rest of the OS is obviously struggling with some heavy task.
- tinus_hn 9y ago
  On Windows, like other modern operating systems, a driver is supposed to do only the minimum amount of work possible in the interrupt handler, to avoid this problem. The real work is supposed to happen later when it can be scheduled.
  - AndrewGaspar 9y ago
    While it's true that networking device drivers on Windows defer all meaningful work to post-ISR, it only moves the goal posts a little bit. On Windows, a lot of network traffic processing (e.g. tcp/ip acknowledgment, virtual switch packet forwarding) is performed in a DPC[1] queued in response to the device's interrupt. A DPC generally runs at dispatch level, which gives it more-or-less exclusive access to the CPU until completion. This preempts the thread scheduler and can result in user-visible stutters in audio and video when the network load outpaces the CPU's ability to quickly service the network traffic. [1] https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/introduction-to-dpcs https://docs.microsoft.com/en-us/windows-hardware/drivers/ke...
    - tinus_hn 9y ago
      These are hardly complex tasks, they could be done by computers from the 70s. It's quite possible that Windows makes a mess of it anyway, of course. But the top comment was about how interrupts from the network card interfere with interrupts from the sound card so it can't keep its buffer filled. That shouldn't happen. Unless the CPU just isn't fast enough to do networking and decode or generate the audio at the same time, but then it's never going to work properly.
radicalbyte 9y ago
Just look at the process you have running, 90% of them are doing nothing. Round Robin is a waste for an OS.
- DSMan195276 9y ago
  Processes that are waiting on some event (like polling some file descriptors) will never be scheduled until they're woken up. It doesn't matter what type of scheduler you are using. The algorithm just decides which to run within the set of programs actually running.

amluto 9y ago

I thought Windows had a fancy interrupt priority system that should, in principle, allow sound playback to preempt network interrupts? AMD64 added a fancy feature (CR8 access to the task priority register) just to accelerate interrupt prioritization. (This is all very vague memory. I know how this stuff works on Linux. Linux does not have interrupt priorities.)

sharkbot 9y ago
I could be wrong, but my understanding of some sound cards is that they have essentially a single memory buffer that they are reading from when instructed to play a sound. Most sound cards let the OS split the buffer into two halves and raise an interrupt when one half of the buffer completes playing. Interrupt prioritization doesn't help much, because the sound data is likely being generated from user mode, and the playback complete interrupt is being handled in kernel mode, which would require a transition back for further processing. When receiving data from a network card, no transfer back to user mode is required and will have implicit priority over sound generation. Therefore, network processing is likely to starve out sound generation barring choices like those described. (This is somewhat hearsay, corrections welcome :))
- xorblurb 9y ago
  Needing to wake up usermode is not necessarily an insurmountable obstacle; under Linux you can use a PREEMPT config. Storming a network card having unwanted consequences can also be mitigated: under Linux you have some network device driver that use an hybrid interrupt/polling approach with the NAPI. Combining both should theoretically allow a system to generate sound smoothly in the described conditions.
  - amluto 9y ago
    IMO you shouldn't really need to wake user mode that often. Just let user code buffer the output samples well ahead of the hardware needing them and have the kernel do the copies when needed.

cm2187 9y ago

Reminds me of Apple's Jim Reekes "engineers are retarded". https://m.youtube.com/watch?v=C5d151lqJsA https://m.youtube.com/watch?v=C5d151lqJsA (The annecdote really starts at 2:30).

Systemic33 9y ago
Haha, what a great video. To some extent he's onto something. Sometimes you are in this zone when everything will work if you just do this one quick little hack, and then the hack ends up not getting replaced with a proper solution, and in the end you end up with a harddrive right next to a speaker >.<
- btschaegg 9y ago
  There actually seem to be people whose default mode of operation seems to lie entirely in that zone though. I'm pretty sure everyone arrives at a similar conclusion if they realise that attempts to fix a certain person's code always end in a express trip to the 6th stage of debugging [1]... [1]: http://plasmasturm.org/log/6debug/ http://plasmasturm.org/log/6debug/
  - jzwinck 9y ago
    Definitely. And that "quick hack by default" person will often end up being the favorite of management because they get so many things "done." So it's important to figure out what kind of manager you have, and try not to be the slow guy on a team full of quick-hack people. You won't look good and nobody will be happy.
    - chii 9y ago
      > the slow guy on a team full of quick-hack people isn't that called 'cultural fit'?
      - deleted 9y ago
        [deleted]

adsfqwop 9y ago

Yeah thanks Microsoft. They apparently never thought of someone playing mp3's from a network drive!! It drove me nuts for months, until I figured out exactly what you just posted.