4 ms·

Can you please explain that more?

the_why_of_y 9y ago
The paper has a clickbaity title, but the actual performance impact of the scheduler behaviors discussed therein is negligible on many workloads. YMMV.
awalton 9y ago
The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.
- greglindahl 9y ago
  Many HPC workloads run 1 process per core, and pin them. So they can't observe this bad behavior. The Linpack benchmark is an example of an HPC code that should run one process per core and be pinned.
- lobster_johnson 9y ago
  Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture. We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.
  - anarazel 9y ago
    > We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result. I'd bet, but not too much, that that was more due to a) postgres' internal locking implementation scaling horribly at that time b) zone_reclaim_mode leading to bad behaviour around IO.
    - lobster_johnson 9y ago
      It's possible, but we weren't testing 48 clients at that time, just our normal workload, which had much less parallelism than that. The person in charge of setting up the systems explained the performance issues as being due to NUMA.
  - velox_io 9y ago
    You may find this a fascinating read. It's regarding how SQL Server has it's own threading scheduler to bypass the Windows scheduler. When ran on a machine with HyperThreading enabled it will only use 50% of the CPU, I guess they're going after CPU Cache utilisation. It's really hard to find a decent article on the subject, here's two of the best I've found. https://www.mssqltips.com/sqlservertip/4403/understanding-sql-server-schedulers-workers-and-tasks/ https://www.mssqltips.com/sqlservertip/4403/understanding-sq... https://docs.microsoft.com/en-us/sql/relational-databases/thread-and-task-architecture-guide https://docs.microsoft.com/en-us/sql/relational-databases/th...
brendangregg 9y ago
I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this. The better question is: why has the Linux Foundation been silent about it? If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux, and due to the Linux scheduling maintainers and community failing to identify such egregious problems. The industry as a whole -- including every device and server that runs Linux -- would have lost many BILLIONS. It would be one of the most costly failures in technology EVER. And the Linux Foundation is silent about this? Seriously?
- bogomipz 9y ago
  >"I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this." Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.
  - brendangregg 9y ago
    I've already answered it elsewhere -- more than once -- and I'm trying not to waste any more time on it. I think the Linux Foundation is best positioned to coordinate a detailed written response. There's some extra research I did that I haven't shared yet, including, for example, when the bugs were introduced (still needs double checking): - Bug 1: Mar 2011, for Linux 2.6.38 - Bug 2: Apr 2012, for Linux 3.4 - Bug 3: Dec 2009, for Linux 2.6.32 - Bug 4: Feb 2015, for Linux 3.19 This paper was published in early 2016, with the title "A decade of wasted cores".
    - thechao 9y ago
      Ah, but they were very intense years, so they counted more than one!

The paper has a clickbaity title, but the actual performance impact of the scheduler behaviors discussed therein is negligible on many workloads. YMMV.

awalton 9y ago

The tl;dr is that unless you had an HPC workload with a NUMA box, you couldn't observe this bad behavior. Which means in reality, you could name approximately everyone that ran into this issue on a single list: top500.org.

greglindahl 9y ago
Many HPC workloads run 1 process per core, and pin them. So they can't observe this bad behavior. The Linpack benchmark is an example of an HPC code that should run one process per core and be pinned.
lobster_johnson 9y ago
Is NUMA that rare? Back in 2007-2008 or so, my company bought some rack servers fitted with 48-core AMD Opteron (Magny-Cours), which weren't particularly expensive, and had a NUMA architecture. We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result.
- anarazel 9y ago
  > We didn't have HPC workloads, just Postgres, which uses one OS process per connection, and performance was terrible as a result. I'd bet, but not too much, that that was more due to a) postgres' internal locking implementation scaling horribly at that time b) zone_reclaim_mode leading to bad behaviour around IO.
  - lobster_johnson 9y ago
    It's possible, but we weren't testing 48 clients at that time, just our normal workload, which had much less parallelism than that. The person in charge of setting up the systems explained the performance issues as being due to NUMA.
- velox_io 9y ago
  You may find this a fascinating read. It's regarding how SQL Server has it's own threading scheduler to bypass the Windows scheduler. When ran on a machine with HyperThreading enabled it will only use 50% of the CPU, I guess they're going after CPU Cache utilisation. It's really hard to find a decent article on the subject, here's two of the best I've found. https://www.mssqltips.com/sqlservertip/4403/understanding-sql-server-schedulers-workers-and-tasks/ https://www.mssqltips.com/sqlservertip/4403/understanding-sq... https://docs.microsoft.com/en-us/sql/relational-databases/thread-and-task-architecture-guide https://docs.microsoft.com/en-us/sql/relational-databases/th...

brendangregg 9y ago

I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this. The better question is: why has the Linux Foundation been silent about it? If this were true -- "a decade of wasted cores", with losses of "13-24% for typical Linux workloads" (for a decade, as the title suggests) -- then companies like Netflix would have lost many millions due to our choice of Linux, and due to the Linux scheduling maintainers and community failing to identify such egregious problems. The industry as a whole -- including every device and server that runs Linux -- would have lost many BILLIONS. It would be one of the most costly failures in technology EVER. And the Linux Foundation is silent about this? Seriously?

bogomipz 9y ago
>"I've commented several times -- it shouldn't be too hard to find -- and I'm trying not to waste any more time on this." Then why make a cryptic comment like you did? Cryptic comments kind of invite questions.
- brendangregg 9y ago
  I've already answered it elsewhere -- more than once -- and I'm trying not to waste any more time on it. I think the Linux Foundation is best positioned to coordinate a detailed written response. There's some extra research I did that I haven't shared yet, including, for example, when the bugs were introduced (still needs double checking): - Bug 1: Mar 2011, for Linux 2.6.38 - Bug 2: Apr 2012, for Linux 3.4 - Bug 3: Dec 2009, for Linux 2.6.32 - Bug 4: Feb 2015, for Linux 3.19 This paper was published in early 2016, with the title "A decade of wasted cores".
  - thechao 9y ago
    Ah, but they were very intense years, so they counted more than one!