4 ms·

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehementl

by Annatar 10y ago

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobody else is capable of designing a high performance TCP/IP stack, and indeed, I have been able to max out 1 Gbit connection running Solaris 10 on a measly DELL R910. If the network administrator hadn't come running to "turn it off! Turn the damn thing off!", I would have maxed out a trunked 40 Gbit link too. I sure taught that guy a lesson that day, never again have I heard a peep about "not ever seeing anyone being able to max out a 1 Gbit connection with a single machine". Only Solaris / illumos' FireEngine makes it possible.

jsnell 10y ago
Help me understand this. How is maxing out a 1Gbps connection with a 4U, 4 socket (so presumably 24-32 core) Xeon server supposed to be impressive?
- Annatar 10y ago
  Considering I did it seven years ago, I think it's impressive in that context.
peterwwillis 10y ago
I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput. And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate, which requires the minimum possible frame size. The maximum rate results in about 18x more packets per second. (This is for a typical MTU of 1500; let's not get into jumbo frames...) And you talk about 1Gbps. For a 10Gbps link, which is the most likely use case in the article, it results in 10x more packets per second, which requires a comparable scale in computing resources. But the stack is only designed to scale to the more likely use case of 1Gbps links; after 8 or so cores, performance can drop off precipitously. So to hit the maximum packets per second of a 10Gbps link, you would need to handle 180x more packets per second than the max throughput of a 1Gbps link. Most stacks are not designed or performance tested with that in mind.
- Annatar 10y ago
  Solaris 10 can drive a 10Gb link at 7.3Gbps (limited by PCI-X bandwidth) using 2x2.2Ghz opteron CPUs utilized at less than 50% https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-networking-the-magic-revealed/ https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne..., page nine. Now the PCIe is multiple times faster than the old PCI-X, and in addition to being several times faster then the old Opteron 939 and 940 series, modern intel based systems have 80 CPU's or more.
  - wmf 10y ago
    Let's call that 3.3 Gbps/GHz. Linux can now drive 25 Gbps with a single 4 GHz core, so that's over 6 Gbps/GHz. Given the number of years involved, it's hard to tell how much improvement is due to hardware and how much is software. But I'm not blown away.
  - peterwwillis 10y ago
    You are still quoting throughput when i'm quoting frame/packet rate. On top of this the line you have in italics doesn't show up on that page (it shows up in two random websites that provide no detail as to that claim), I have no idea what "page nine" on my resolution monitor is compared to yours, and you're trying to suggest something about the PCI bus being a factor (which it isn't; the limiting factor is cycles per packet). So I don't think you understand what's going on.
    - Annatar 10y ago
      My apologies, wrong link. This is hard to do on a mobile telephone. http://www.baylisa.org/library/slides/2005/august2005.pdf http://www.baylisa.org/library/slides/2005/august2005.pdf, page nine, and in more detail further in the document.

by Annatar 10y ago

jsnell 10y ago

Help me understand this. How is maxing out a 1Gbps connection with a 4U, 4 socket (so presumably 24-32 core) Xeon server supposed to be impressive?

Annatar 10y ago
Considering I did it seven years ago, I think it's impressive in that context.

peterwwillis 10y ago

I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput. And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate, which requires the minimum possible frame size. The maximum rate results in about 18x more packets per second. (This is for a typical MTU of 1500; let's not get into jumbo frames...) And you talk about 1Gbps. For a 10Gbps link, which is the most likely use case in the article, it results in 10x more packets per second, which requires a comparable scale in computing resources. But the stack is only designed to scale to the more likely use case of 1Gbps links; after 8 or so cores, performance can drop off precipitously. So to hit the maximum packets per second of a 10Gbps link, you would need to handle 180x more packets per second than the max throughput of a 1Gbps link. Most stacks are not designed or performance tested with that in mind.

Annatar 10y ago
Solaris 10 can drive a 10Gb link at 7.3Gbps (limited by PCI-X bandwidth) using 2x2.2Ghz opteron CPUs utilized at less than 50% https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-networking-the-magic-revealed/ https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne..., page nine. Now the PCIe is multiple times faster than the old PCI-X, and in addition to being several times faster then the old Opteron 939 and 940 series, modern intel based systems have 80 CPU's or more.
- wmf 10y ago
  Let's call that 3.3 Gbps/GHz. Linux can now drive 25 Gbps with a single 4 GHz core, so that's over 6 Gbps/GHz. Given the number of years involved, it's hard to tell how much improvement is due to hardware and how much is software. But I'm not blown away.
- peterwwillis 10y ago
  You are still quoting throughput when i'm quoting frame/packet rate. On top of this the line you have in italics doesn't show up on that page (it shows up in two random websites that provide no detail as to that claim), I have no idea what "page nine" on my resolution monitor is compared to yours, and you're trying to suggest something about the PCI bus being a factor (which it isn't; the limiting factor is cycles per packet). So I don't think you understand what's going on.
  - Annatar 10y ago
    My apologies, wrong link. This is hard to do on a mobile telephone. http://www.baylisa.org/library/slides/2005/august2005.pdf http://www.baylisa.org/library/slides/2005/august2005.pdf, page nine, and in more detail further in the document.