4 ms·

Intel continues with more optimizations to the Linux kernel

126 points by isaacfrond 4y ago

hardware2win 4y ago
Wow, thats huge difference
chaorace 4y ago
Skipping to the good stuff from the patch notes: > Wangyang and Arjan reported a bottleneck in the networking code related to struct dst_entry::__refcnt. Performance tanks massively when concurrency on a dst_entry increases. > This happens when there are a large amount of connections to or from the same IP address. The memtier benchmark when run on the same host as memcached amplifies this massively. But even over real network connections this issue can be observed at an obviously smaller scale (due to the network bandwith limitations in my setup, i.e. 1Gb). Basically, processes that produce a lot of threads (i.e.: 100+) suffer performance degradation. This impacted network performance by up to 5% on 1 Gb/s connections and showed up to 3.2x performance degradation in synthetic benchmarks. Independent testing verifies these improvements in the Memcached benchmark. Simulating a more realistic workload using Cockroach DB shows a more modest performance improvement of ~5% in terms of ops/s.
- BenoitP 4y ago
  > struct dst_entry::__refcnt. Performance tanks massively when concurrency on a dst_entry increases Maybe we could just stop tracking references to these dst_entry. This way there would be no contention when tracking reference counts. So no performance loss. Some would no longer be used, but hey RAM is cheap. And to cleanup memory we'd scan the memory for these references from time to time. Probably beginning by the only way (transitive) references can occur from their roots: the thread stacks. We'd quickly pause threads, scan their stack, unpause them while ordering them to notify us if they would write new content to memory. We'd do that at the compiler level so this can be general to all memory management. This way we'd have a concurrent scanning. We could also copy memory as we scan it, removing the gaps and compacting it for better CPU cache usage. But that would require a memory ordering model, threading, and compiler behavior to go hand-in-hand. We'd call it an abstract machine, dare I say virtual? This system would help us automatically collect these memory objects that have become garbage...
  - secondcoming 4y ago
    You want a device driver written in Java?
    - Cthulhu_ 4y ago
      I'm 99% confident the post is intended to be sarcastically suggesting memory management is replaced with a garbage collector.
  - msla 4y ago
    I mean, congratulations on pointing out why garbage collection is not acceptable in most parts of an OS kernel, especially a general-purpose one as opposed to one only designed to run on specialized high-end hardware, but was that really something that needed to be said?
  - otabdeveloper4 4y ago
    > We'd quickly pause threads Yeah, no. Not a chance.
twic 4y ago
Why does Intel spend money on doing this sort of stuff? I'm certainly happy they do, just curious as to why. I can't see how Linux generically being a bit faster helps them as a company. Are they specifically looking at places where they have a disadvantage or a potential advantage relative to ARM etc?
- deleted 4y ago
  [deleted]
- zitterbewegung 4y ago
  The reason is that they want software on their hardware to be faster and they don’t care if it makes competitors faster . They still have sales in the data center regardless.
- jrochkind1 4y ago
  I wonder if there are public statistics on how many intel chips, what portion of production, end up running linux. Probably quite a lot, what with all the cloud fleets? So, yeah, ARM etc reminds them that there are alternatives. They don't need to focus on specific comparisons with ARM for it to be good business to keep Linux running as well as they can make it on Intel. Plus the marketing from being seen to do it. It seems pretty straightforward to me. Focusing only on spending money on very specific measurable outcomes on your bottom line, while it might be all the rage these days, can be pretty self-defeating in the long-term. Big companies generally know this, and can afford it.
- WJW 4y ago
  It's a basic business strategy called "commoditizing your complement". The better and faster software gets, the more people will want to use it and the more demand for CPUs will be created. Since Intel sells a good percentage of all server CPUs, that will also increase their own sales. The article at https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ is an excellent introduction if you want to know more.
  - Teever 4y ago
didgetmaster 4y ago
I have found that thread contention can be a real problem if you get a little lazy in minimizing the area of contention. My own data management project (https://didgets.com/ https://didgets.com/) is highly threaded. On high core count CPUs, threads can greatly speed up data processing; but multiple threads working on the same piece of data have to be very careful around critical sections. Two problems I have encountered are 1) Locking too big an area (i.e. holding a mutex over an entire function when you just have to protect a single line of code) and 2) spinning off too many threads. I check the hardware to know how many physical cores the CPU has before deciding how many threads to create. Spinning off hundreds of threads when the CPU only can run 8 at a time often does not help and can even be counter-productive.
- dragontamer 4y ago
  > Spinning off hundreds of threads when the CPU only can run 8 at a time often does not help and can even be counter-productive. Depends on how many of those threads become blocked by I/O. If you spin 200 threads, but 192 of them are blocked by I/O (waiting for disk and/or ethernet), then those 8-remaining threads effectively use the 8-cores of your CPU. Old-school programmers would remember the days of 1-core CPUs, where you'd need to use a threads (or processes actually) to effectively utilize hard drives or your ethernet connections.
  - didgetmaster 4y ago
    There are certainly instances where a large number of threads blocked waiting for I/O can be beneficial. I was merely pointing out that this is not always the case and you have to understand how adding another thread can help vs just getting in the way. Even with Disk I/O, too many threads trying to read/write to a HDD at once can lead to performance problems as the heads are thrashing trying to access different parts of the disk. Too many programmers just spin off threads without understanding what the implications really are.
celeritascelery 4y ago
I recently learned that Intel has been the top contributor to the Linux kernel for the last 10 years. Meaning it does more than enterprise distros like red hat and suse. Really cool to see that.
- xattt 4y ago
  Is this a long play to ensure viability for x86 under Linux to hedge the risk of Microsoft pivoting to ARM?
  - klooney 4y ago
    I mean, it's good for your products to work well, and with Linux you can do that directly instead of petitioning Microsoft
  - celeritascelery 4y ago
    10 years ago windows ARM was not really a thing, so I think this is a bigger than that. I think Intel is just invested in making software take full advantage of features in their CPU's. They can directly do that on Linux.
    - metadat 4y ago
      In fact, Windows ARM was a thing 10 years ago in 2013 with the remarkably popular-at-the-time Surface Pro.
- singhrac 4y ago
  People continue to say that Intel has lost its way, and maybe it has in hardware, but for a long time they have had the best numerical computing software teams outside of NVIDIA. You can look at mkl, their recent updates to numpy, etc. This is totally orthogonal to whether they are good actors in open source, to be clear.
segfaultedbard 4y ago
Oh goodie, now fix the userspace before we get a 9th way to configure ethernet devices.

126 points by isaacfrond 4y ago

Wow, thats huge difference

Skipping to the good stuff from the patch notes: > Wangyang and Arjan reported a bottleneck in the networking code related to struct dst_entry::__refcnt. Performance tanks massively when concurrency on a dst_entry increases. > This happens when there are a large amount of connections to or from the same IP address. The memtier benchmark when run on the same host as memcached amplifies this massively. But even over real network connections this issue can be observed at an obviously smaller scale (due to the network bandwith limitations in my setup, i.e. 1Gb). Basically, processes that produce a lot of threads (i.e.: 100+) suffer performance degradation. This impacted network performance by up to 5% on 1 Gb/s connections and showed up to 3.2x performance degradation in synthetic benchmarks. Independent testing verifies these improvements in the Memcached benchmark. Simulating a more realistic workload using Cockroach DB shows a more modest performance improvement of ~5% in terms of ops/s.

BenoitP 4y ago
> struct dst_entry::__refcnt. Performance tanks massively when concurrency on a dst_entry increases Maybe we could just stop tracking references to these dst_entry. This way there would be no contention when tracking reference counts. So no performance loss. Some would no longer be used, but hey RAM is cheap. And to cleanup memory we'd scan the memory for these references from time to time. Probably beginning by the only way (transitive) references can occur from their roots: the thread stacks. We'd quickly pause threads, scan their stack, unpause them while ordering them to notify us if they would write new content to memory. We'd do that at the compiler level so this can be general to all memory management. This way we'd have a concurrent scanning. We could also copy memory as we scan it, removing the gaps and compacting it for better CPU cache usage. But that would require a memory ordering model, threading, and compiler behavior to go hand-in-hand. We'd call it an abstract machine, dare I say virtual? This system would help us automatically collect these memory objects that have become garbage...
- secondcoming 4y ago
  You want a device driver written in Java?
  - Cthulhu_ 4y ago
    I'm 99% confident the post is intended to be sarcastically suggesting memory management is replaced with a garbage collector.
- msla 4y ago
  I mean, congratulations on pointing out why garbage collection is not acceptable in most parts of an OS kernel, especially a general-purpose one as opposed to one only designed to run on specialized high-end hardware, but was that really something that needed to be said?
- otabdeveloper4 4y ago
  > We'd quickly pause threads Yeah, no. Not a chance.

twic 4y ago

Why does Intel spend money on doing this sort of stuff? I'm certainly happy they do, just curious as to why. I can't see how Linux generically being a bit faster helps them as a company. Are they specifically looking at places where they have a disadvantage or a potential advantage relative to ARM etc?

deleted 4y ago
[deleted]
zitterbewegung 4y ago
The reason is that they want software on their hardware to be faster and they don’t care if it makes competitors faster . They still have sales in the data center regardless.
jrochkind1 4y ago
I wonder if there are public statistics on how many intel chips, what portion of production, end up running linux. Probably quite a lot, what with all the cloud fleets? So, yeah, ARM etc reminds them that there are alternatives. They don't need to focus on specific comparisons with ARM for it to be good business to keep Linux running as well as they can make it on Intel. Plus the marketing from being seen to do it. It seems pretty straightforward to me. Focusing only on spending money on very specific measurable outcomes on your bottom line, while it might be all the rage these days, can be pretty self-defeating in the long-term. Big companies generally know this, and can afford it.
WJW 4y ago
It's a basic business strategy called "commoditizing your complement". The better and faster software gets, the more people will want to use it and the more demand for CPUs will be created. Since Intel sells a good percentage of all server CPUs, that will also increase their own sales. The article at https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ is an excellent introduction if you want to know more.
- Teever 4y ago

didgetmaster 4y ago

I have found that thread contention can be a real problem if you get a little lazy in minimizing the area of contention. My own data management project (https://didgets.com/ https://didgets.com/) is highly threaded. On high core count CPUs, threads can greatly speed up data processing; but multiple threads working on the same piece of data have to be very careful around critical sections. Two problems I have encountered are 1) Locking too big an area (i.e. holding a mutex over an entire function when you just have to protect a single line of code) and 2) spinning off too many threads. I check the hardware to know how many physical cores the CPU has before deciding how many threads to create. Spinning off hundreds of threads when the CPU only can run 8 at a time often does not help and can even be counter-productive.

dragontamer 4y ago
> Spinning off hundreds of threads when the CPU only can run 8 at a time often does not help and can even be counter-productive. Depends on how many of those threads become blocked by I/O. If you spin 200 threads, but 192 of them are blocked by I/O (waiting for disk and/or ethernet), then those 8-remaining threads effectively use the 8-cores of your CPU. Old-school programmers would remember the days of 1-core CPUs, where you'd need to use a threads (or processes actually) to effectively utilize hard drives or your ethernet connections.
- didgetmaster 4y ago
  There are certainly instances where a large number of threads blocked waiting for I/O can be beneficial. I was merely pointing out that this is not always the case and you have to understand how adding another thread can help vs just getting in the way. Even with Disk I/O, too many threads trying to read/write to a HDD at once can lead to performance problems as the heads are thrashing trying to access different parts of the disk. Too many programmers just spin off threads without understanding what the implications really are.

celeritascelery 4y ago

I recently learned that Intel has been the top contributor to the Linux kernel for the last 10 years. Meaning it does more than enterprise distros like red hat and suse. Really cool to see that.

xattt 4y ago
Is this a long play to ensure viability for x86 under Linux to hedge the risk of Microsoft pivoting to ARM?
- klooney 4y ago
  I mean, it's good for your products to work well, and with Linux you can do that directly instead of petitioning Microsoft
- celeritascelery 4y ago
  10 years ago windows ARM was not really a thing, so I think this is a bigger than that. I think Intel is just invested in making software take full advantage of features in their CPU's. They can directly do that on Linux.
  - metadat 4y ago
    In fact, Windows ARM was a thing 10 years ago in 2013 with the remarkably popular-at-the-time Surface Pro.
singhrac 4y ago
People continue to say that Intel has lost its way, and maybe it has in hardware, but for a long time they have had the best numerical computing software teams outside of NVIDIA. You can look at mkl, their recent updates to numpy, etc. This is totally orthogonal to whether they are good actors in open source, to be clear.

segfaultedbard 4y ago

Oh goodie, now fix the userspace before we get a 9th way to configure ethernet devices.