8 ms·

This all sounds very good, and I grant that Linus knows what he's talking about, but didn't this whole thing start with a bunch of game developers that noticed

by OskarS 7y ago

This all sounds very good, and I grant that Linus knows what he's talking about, but didn't this whole thing start with a bunch of game developers that noticed massive stalls when porting games to Google Stadia? Stalls that, presumably, didn't happen on other platforms (Windows and PlayStation, using a variant of FreeBSD). If these kinds of spinlocks are so fundamentally misguided, why do they only cause problems on Linux?

baq 7y ago
Game developers running their software that requires a system tuned for interactive latencies get burned by systems optimized for throughput, proceed to blame the system. I’m more for polite passive aggressive suggestion of incompetence instead of in your face blaming, otherwise the response was quite on point. On the plus side the original author now has plenty to read about and us here get to learn interesting things by proxy.
- draw_down 7y ago
  > I’m more for polite passive aggressive suggestion of incompetence instead of in your face blaming Good grief.
- forrestthewoods 7y ago
  Moral of the story... don’t ship games on Linux?
  - doctorpangloss 7y ago
    The irony is that because of Android, another platform Google owns besides Stadia, Linux is already the world’s biggest gaming platform.
    - mrtksn 7y ago
      How much Linux is android though? Does Android feedback into Linux? This argument about Linux being the most popular consumer OS or Gaming platform thanks to Android sounds like Syria taking credit for the iPhone because of Steve Jobs being a Syrian by bloodline. "You know, although we are consumed by sectarian fights and iron-fisted leaders that will crush anyone having opinions, It is important to note that worlds one of the worlds most successful contemporary technology companies is actually Syrian".
      - hrydgard 7y ago
        From a game's perspective, Android is very much like Linux, actually.
    - pjmlp 7y ago
      iOS is the biggest gaming platform, on $$$ per user, including some exclusive hits, with several frameworks and tooling tailored to writing games. Android is such a pain to develop games for, that Google has started a Android Games initiative only in 2019, to appease the growing complaints from game studios. An NDK that feels like a 20% project, many game related APIs (e.g. image codecs, networking, files, permissions) are Java only and require JNI even if implemented in C++, frame rate setup, build infrastructure regarding NDK libraries, C only API for what are actually C++ classes forcing wrapping code for RAII and such, Vulkan feels like a code dump that everyone has to manually integrate into the project vs Metal tooling, graphics debugging and real time audio are outsourced to a GitHub projects not directly supported on the official Android site, network debugging is only supported for Java layer,.... Finally, the only Linux that Android has is the kernel, which Google could replace tomorrow and user space wouldn't even notice. Linux fans keep failing to realise how little Linux Android actually exposes.
      - th3h4mm3r 7y ago
        Oh no! An Apple fanboy!
        pjmlp 7y ago
        Rather someone that actually understands how meaningless is Android making use of the Linux kernel, beyond saving money paying for licenses.
        thenewnewguy 7y ago
        Paying to license what? I'm not aware of any other kernel suitable for mobile phone use you could buy with a license (i.e. iOS doesn't count as there's no way you're getting Apple to sell you even a license)
        pjmlp 7y ago
        Windows and plenty of embedded ones to chose from. After all, it was Series 30, Bada, Symbian, Blackberry, Psion and Windows CE that started mobile phones.
  - imtringued 7y ago
    Don't use userspace spinlocks on Linux.
  - soraminazuki 7y ago
    No, the moral of the story is don't use spinlocks in userspace and use established locking libraries that suite your needs. You can't expect any operating system to optimize for poorly written applications when you have thousands of them screwing up in all different ways.
- brigade 7y ago
  AKA the Linux scheduler isn't intended for desktop environments and applications. Certainly that same observation drove the development of BFS / MuQSS, which unsurprisingly behaves better on his artificial benchmark than the mainline scheduler.
  - Taek 7y ago
    Linux can be tuned for low latency, it's a kernel setting.
    - skrebbel 7y ago
      Either that's not sufficient or the Google Stadia people didn't even bother tuning their systems for gaming, which I doubt.
      - im3w1l 7y ago
        Maybe they did some tuning but missed this thing.
  - gpderetta 7y ago
    As shown in the original benchmark, using proper mutexes is near optimal and there is little need for kernel tuning (although still possible).
baybal2 7y ago
Ironically things went upside down with linux scheduling over the last 15 years. All major OSes had PhD CS people doing their schedulers "efficient" at around 2000, and be able to sustain 98%+ MP loads. Linux was an odd case of a very naive, and "inefficient" scheduler that left CPU 30-20% idle unless you really knew how to hack around that. Linux however came with near no stalls, and latency spikes. Things have changed dramatically now. Linux got attention of the entire industry on it, and years of holy wars about scheduling. You can now hit 95%+ under multithreaded load easily, but even on the desktop you can now see how a block in IO hog app can visibly freeze your system for few seconds. Now, the trend has reversed again, and now everybody is praising conservative scheduling that doesn't give out too much CPU :-/
- joosters 7y ago
  This could be due to the focus of the kernel developers involved. At some point in the 2000s, companies such as IBM invested heavily in Linux, working on making the kernel scale well on 'big iron' machines with tens of cores and NUMA-style architecture. As a result, the focus shifted away from 'normal' desktop boxes. But was this a good or bad thing? Even desktop machines can now have 16+ cores and NUMA. So, perhaps that focus has paid off for more users?
- atq2119 7y ago
  Is that a CPU scheduler issue though? Linux traditionally has a problem of overall system architecture in that userspace isn't set up to provide policy information such as "this thread is doing UI" vs. "this is a background task" by default. However, if I help the scheduler along by nice(1)-ing my big compute tasks, then the desktop remains perfectly smooth even while the CPU is 100% busy. It'd be much better to have this experience by default, but unfortunately it requires architectural improvements throughout the entire software stack. Trying to band-aid it in the kernel scheduler may allow you to heuristically improve some workloads, but it's ultimately a fool's errand. Where the desktop does tend to fall over and become totally unusable is when the system starts to swap...
  - blattimwind 7y ago
kbumsik 7y ago
The claims from the OP: > So this all started like this: I overheard somebody at work complaining about mysterious stalls while porting Rage 2 to Stadia. The only thing those mysterious stalls had in common was that they were all using spinlocks. If the OP's benchmark is invalid and if the root cause is just "overheard" by somebody without sharing detailed analysis, we don't even know if the problem is really about spinlocks. We are still unsure if it comes from Linux kernel or just Stadia platform-specific. Do we have similar issues with Android?
- OskarS 7y ago
  Fair enough, I suppose the original post was quite light on technical details.
asdfaoeu 7y ago
> So the code in question is pure garbage. You can't do spinlocks like that. Or rather, you very much can do them like that, and when you do that you are measuring random latencies and getting nonsensical values, because what you are measuring is "I have a lot of busywork, where all the processes are CPU-bound, and I'm measuring random points of how long the scheduler kept the process in place". > And then you write a blog-post blamings others, not understanding that it's your incorrect code that is garbage, and is giving random garbage values. > And then you test different schedulers, and you get different random values that you think are interesting, because you think they show something cool about the schedulers. Tldr: They got probably lucky on Windows. Conceivably similar code might have worked on Linux and then had issues on Windows.
- seba_dos1 7y ago
  ...and that would be extremely likely to happen if the Linux version was being written first and only then ported to Windows.
gpderetta 7y ago
> If these kinds of spinlocks are so fundamentally misguided, why do they only cause problems on Linux? It is possible that, as explained by Linus, the Windows scheduler just doesn't ootimize for NUMA as aggressively as linux (which attempts to keep threads on their own cores as much as possible). It is also as likely that, because on Windows games have been an important load for a while, the Windows scheduler likely bends over backward to support broken code either by penalising other use cases or by attempting to recognise this sort of spinlock behaviour and special casing it.
- joosters 7y ago
  Perhaps Linux should do more 'special-casing' of broken/older apps? In the email thread, Linus points out that GNU make has a bug in its concurrency code, which was only hit when newer kernels added some optimizations. As a result, they had to walk back on these improvements. Now, I'm sure that the bug has been fixed in newer versions of make, but 'make' is a program that many people are unlikely to keep up-to-date (most users of it will be only using functionality that has been in make for decades and wouldn't ever consider needing to have a brand new version of it) It would be a shame to hold back kernel optimizations if they trigger bugs in older user-space code, so in these kind of cases it would be great if Linux had some of the 'backwards-compatibility cruft' that Windows is famous for, allowing for new optimizations while detecting and working around known problems in common older code.
  - gpderetta 7y ago
    Sure! Remember Linus mantra of "never break userspace". But the code in point is not a legacy binary, it is a new program getting ported to linux. Still it might make sense to change linux scheduler behaviour to improve performance of unmodified Windows games under Wine if it turns out that using yield (or the Windows equivalent Sleep(0)) is common. And in fairness to the original post, the author had a latency issue, they invertigated it and correctly identified the problem with spinlock and yield and found out that using a proper mutex is the best solution. The only issue is that they thought that the scheduler was the root cause of the bad behaviour, not the yield itself.
- 7y ago

by OskarS 7y ago

baq 7y ago

Game developers running their software that requires a system tuned for interactive latencies get burned by systems optimized for throughput, proceed to blame the system. I’m more for polite passive aggressive suggestion of incompetence instead of in your face blaming, otherwise the response was quite on point. On the plus side the original author now has plenty to read about and us here get to learn interesting things by proxy.

draw_down 7y ago
> I’m more for polite passive aggressive suggestion of incompetence instead of in your face blaming Good grief.
forrestthewoods 7y ago
Moral of the story... don’t ship games on Linux?
- doctorpangloss 7y ago
  The irony is that because of Android, another platform Google owns besides Stadia, Linux is already the world’s biggest gaming platform.
  - mrtksn 7y ago
    How much Linux is android though? Does Android feedback into Linux? This argument about Linux being the most popular consumer OS or Gaming platform thanks to Android sounds like Syria taking credit for the iPhone because of Steve Jobs being a Syrian by bloodline. "You know, although we are consumed by sectarian fights and iron-fisted leaders that will crush anyone having opinions, It is important to note that worlds one of the worlds most successful contemporary technology companies is actually Syrian".
    - hrydgard 7y ago
      From a game's perspective, Android is very much like Linux, actually.
  - pjmlp 7y ago
    iOS is the biggest gaming platform, on $$$ per user, including some exclusive hits, with several frameworks and tooling tailored to writing games. Android is such a pain to develop games for, that Google has started a Android Games initiative only in 2019, to appease the growing complaints from game studios. An NDK that feels like a 20% project, many game related APIs (e.g. image codecs, networking, files, permissions) are Java only and require JNI even if implemented in C++, frame rate setup, build infrastructure regarding NDK libraries, C only API for what are actually C++ classes forcing wrapping code for RAII and such, Vulkan feels like a code dump that everyone has to manually integrate into the project vs Metal tooling, graphics debugging and real time audio are outsourced to a GitHub projects not directly supported on the official Android site, network debugging is only supported for Java layer,.... Finally, the only Linux that Android has is the kernel, which Google could replace tomorrow and user space wouldn't even notice. Linux fans keep failing to realise how little Linux Android actually exposes.
    - th3h4mm3r 7y ago
      Oh no! An Apple fanboy!
      - pjmlp 7y ago
        Rather someone that actually understands how meaningless is Android making use of the Linux kernel, beyond saving money paying for licenses.
        thenewnewguy 7y ago
        Paying to license what? I'm not aware of any other kernel suitable for mobile phone use you could buy with a license (i.e. iOS doesn't count as there's no way you're getting Apple to sell you even a license)
        pjmlp 7y ago
        Windows and plenty of embedded ones to chose from. After all, it was Series 30, Bada, Symbian, Blackberry, Psion and Windows CE that started mobile phones.
- imtringued 7y ago
  Don't use userspace spinlocks on Linux.
- soraminazuki 7y ago
  No, the moral of the story is don't use spinlocks in userspace and use established locking libraries that suite your needs. You can't expect any operating system to optimize for poorly written applications when you have thousands of them screwing up in all different ways.
brigade 7y ago
AKA the Linux scheduler isn't intended for desktop environments and applications. Certainly that same observation drove the development of BFS / MuQSS, which unsurprisingly behaves better on his artificial benchmark than the mainline scheduler.
- Taek 7y ago
  Linux can be tuned for low latency, it's a kernel setting.
  - skrebbel 7y ago
    Either that's not sufficient or the Google Stadia people didn't even bother tuning their systems for gaming, which I doubt.
    - im3w1l 7y ago
      Maybe they did some tuning but missed this thing.
- gpderetta 7y ago
  As shown in the original benchmark, using proper mutexes is near optimal and there is little need for kernel tuning (although still possible).

baybal2 7y ago

Ironically things went upside down with linux scheduling over the last 15 years. All major OSes had PhD CS people doing their schedulers "efficient" at around 2000, and be able to sustain 98%+ MP loads. Linux was an odd case of a very naive, and "inefficient" scheduler that left CPU 30-20% idle unless you really knew how to hack around that. Linux however came with near no stalls, and latency spikes. Things have changed dramatically now. Linux got attention of the entire industry on it, and years of holy wars about scheduling. You can now hit 95%+ under multithreaded load easily, but even on the desktop you can now see how a block in IO hog app can visibly freeze your system for few seconds. Now, the trend has reversed again, and now everybody is praising conservative scheduling that doesn't give out too much CPU :-/

joosters 7y ago
This could be due to the focus of the kernel developers involved. At some point in the 2000s, companies such as IBM invested heavily in Linux, working on making the kernel scale well on 'big iron' machines with tens of cores and NUMA-style architecture. As a result, the focus shifted away from 'normal' desktop boxes. But was this a good or bad thing? Even desktop machines can now have 16+ cores and NUMA. So, perhaps that focus has paid off for more users?
atq2119 7y ago
Is that a CPU scheduler issue though? Linux traditionally has a problem of overall system architecture in that userspace isn't set up to provide policy information such as "this thread is doing UI" vs. "this is a background task" by default. However, if I help the scheduler along by nice(1)-ing my big compute tasks, then the desktop remains perfectly smooth even while the CPU is 100% busy. It'd be much better to have this experience by default, but unfortunately it requires architectural improvements throughout the entire software stack. Trying to band-aid it in the kernel scheduler may allow you to heuristically improve some workloads, but it's ultimately a fool's errand. Where the desktop does tend to fall over and become totally unusable is when the system starts to swap...
- blattimwind 7y ago

kbumsik 7y ago

The claims from the OP: > So this all started like this: I overheard somebody at work complaining about mysterious stalls while porting Rage 2 to Stadia. The only thing those mysterious stalls had in common was that they were all using spinlocks. If the OP's benchmark is invalid and if the root cause is just "overheard" by somebody without sharing detailed analysis, we don't even know if the problem is really about spinlocks. We are still unsure if it comes from Linux kernel or just Stadia platform-specific. Do we have similar issues with Android?

OskarS 7y ago
Fair enough, I suppose the original post was quite light on technical details.

asdfaoeu 7y ago

> So the code in question is pure garbage. You can't do spinlocks like that. Or rather, you very much can do them like that, and when you do that you are measuring random latencies and getting nonsensical values, because what you are measuring is "I have a lot of busywork, where all the processes are CPU-bound, and I'm measuring random points of how long the scheduler kept the process in place". > And then you write a blog-post blamings others, not understanding that it's your incorrect code that is garbage, and is giving random garbage values. > And then you test different schedulers, and you get different random values that you think are interesting, because you think they show something cool about the schedulers. Tldr: They got probably lucky on Windows. Conceivably similar code might have worked on Linux and then had issues on Windows.

seba_dos1 7y ago
...and that would be extremely likely to happen if the Linux version was being written first and only then ported to Windows.

gpderetta 7y ago

> If these kinds of spinlocks are so fundamentally misguided, why do they only cause problems on Linux? It is possible that, as explained by Linus, the Windows scheduler just doesn't ootimize for NUMA as aggressively as linux (which attempts to keep threads on their own cores as much as possible). It is also as likely that, because on Windows games have been an important load for a while, the Windows scheduler likely bends over backward to support broken code either by penalising other use cases or by attempting to recognise this sort of spinlock behaviour and special casing it.

joosters 7y ago
Perhaps Linux should do more 'special-casing' of broken/older apps? In the email thread, Linus points out that GNU make has a bug in its concurrency code, which was only hit when newer kernels added some optimizations. As a result, they had to walk back on these improvements. Now, I'm sure that the bug has been fixed in newer versions of make, but 'make' is a program that many people are unlikely to keep up-to-date (most users of it will be only using functionality that has been in make for decades and wouldn't ever consider needing to have a brand new version of it) It would be a shame to hold back kernel optimizations if they trigger bugs in older user-space code, so in these kind of cases it would be great if Linux had some of the 'backwards-compatibility cruft' that Windows is famous for, allowing for new optimizations while detecting and working around known problems in common older code.
- gpderetta 7y ago
  Sure! Remember Linus mantra of "never break userspace". But the code in point is not a legacy binary, it is a new program getting ported to linux. Still it might make sense to change linux scheduler behaviour to improve performance of unmodified Windows games under Wine if it turns out that using yield (or the Windows equivalent Sleep(0)) is common. And in fairness to the original post, the author had a latency issue, they invertigated it and correctly identified the problem with spinlock and yield and found out that using a proper mutex is the best solution. The only issue is that they thought that the scheduler was the root cause of the bad behaviour, not the yield itself.
7y ago