3 ms·

I don't understand where you're getting the increase in context switches from. The expectation should be the opposite, since the network stack is almost certain

by jsnell 11y ago

I don't understand where you're getting the increase in context switches from. The expectation should be the opposite, since the network stack is almost certainly going to be user-space and in the same process Does it make sense for every application? Unlikely. Does it make sense for an application that runs on 10k nodes, and where moving all IO to poll mode user-space doubles the number of requests you can serve per unit of time? Saving 5k machines worth of capex buys a lot of engineering complexity. And since we know that systems like this have been done, 2x should not be an unreasonable number. See for example https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-peter_simon.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14... and https://www.cs.cmu.edu/~hl/papers/mica-nsdi2014.pdf https://www.cs.cmu.edu/~hl/papers/mica-nsdi2014.pdf

notacoward 11y ago
It all depends how many cores you're willing to burn. If you're getting 2x performance but you only have 75% as many CPUs left for your application, you're going to need more than half as many nodes. At SiCortex we were selling into HPC. These people were thoroughly used to working on football-field-sized compute complexes, and they still bitched about how much CPU time was being burned in the I/O stack. No matter how many cycles they had available, they wanted those cycles used for their apps. As I also said before, this kind of thing does have its place. It all depends on how many cycles you're likely to spin before you actually find anything to do, and whether you have another need for those cycles. If it's not many cycles because you really are pushing a lot of I/O, that's great. If you don't need those cycles for something else because you're an appliance and this is your only job, that's great too. If it's a lot of cycles and you do need those cycles for other things - which is the most common case - then burning lots of cycles busy-waiting for events that haven't happened yet only decreases real hardware utilization and increases either capex or time to completion.
- jsnell 11y ago
  I'm not quite sure where you're getting that 25% reduction in cores from. Yes, it'd probably be a pretty bad tradeoff to dedicate 25% of the machine to busy-looping on IO just to reduce the IO overhead by a factor of 2. But that's not at all what I'm suggesting. I think there's a reasonable case to be made that application performance measured on two systems with exactly the same hardware would be 2x higher when the IO is moved into user-space.
  - notacoward 11y ago
    Kind of depends on the application, wouldn't you say? For some applications, absolutely not. For some applications, maybe yes, assuming the application programmer knows enough not to negate the advantage e.g. by creating lock contention or cache thrashing. And they know enough to use something like LKL (interestingly also from Intel) instead of reinventing their own filesystem-like layer on top of that raw storage. And they don't make huge security blunders. Because if they don't get all of those things right, it doesn't matter if their performance is 2x for one brief shining moment before everything goes to hell. That's not comparing apples to apples. You have to hold the functionality/quality bar constant or else it's meaningless.
- wmf 11y ago
  I would hope that DPDK apps only poll when they have no other work to do, but I don't know if it's actually set up this way.
  - jeff_marshall 11y ago
    The DPDK operates (or at least it did the last time I looked) in a run to completion model for packets received. once the current batch of received packets has been processed by a thread, the DPDK run time immediately looks for more packets to process. The number of polling threads is configurable, and often there are multiple threads per network interface (esp. on 10 gig or faster interfaces) with recieve flow steering to direct related packets to the same receiving thread to maintain cache coherency during packet processing. If you have long-lived work that shouldn't block processing more packets, it would be typical to offload that to separate thread/process from the one doing the packet processing (e.g., for control plane work).
  - notacoward 11y ago
    I haven't looked at DPDK specifically (I'd like to) but most such libraries I've seen poll on their own in their own thread. If the application has to tell it to poll, then it's cooperative multitasking; we all saw how much fun that was in Windows or (pre-OSX) MacOS. Polling introduces a classic latency/resource-consumption tradeoff. If you do it too much, you waste resources. If you don't do it enough, you get crappy latency. This is why interrupts were invented, and they work quite well when used correctly. Whether the storage or networking stacks in the Linux kernel uses them correctly is left as a point for the reader to ponder. ;)
    - wmf 11y ago
      IMO people should only try to use DPDK if they're willing to go all in on "1975 programming", so the event loop is a minor annoyance on the scale of things that you have to deal with. Besides, kids today love event loops and callbacks anyway.
      - mtanski 11y ago
- eloff 11y ago
  Your assuming this uses more cycles, but that's probably false. All the interrupts, context switches, and kernel processing related to IO is a lot of processing. You have control here how many cycles to burn busy waiting, it's the classic latency vs throughout tradeoff. Longer sleep times in between polling means less busy waiting and more latency.

by jsnell 11y ago

notacoward 11y ago

It all depends how many cores you're willing to burn. If you're getting 2x performance but you only have 75% as many CPUs left for your application, you're going to need more than half as many nodes. At SiCortex we were selling into HPC. These people were thoroughly used to working on football-field-sized compute complexes, and they still bitched about how much CPU time was being burned in the I/O stack. No matter how many cycles they had available, they wanted those cycles used for their apps. As I also said before, this kind of thing does have its place. It all depends on how many cycles you're likely to spin before you actually find anything to do, and whether you have another need for those cycles. If it's not many cycles because you really are pushing a lot of I/O, that's great. If you don't need those cycles for something else because you're an appliance and this is your only job, that's great too. If it's a lot of cycles and you do need those cycles for other things - which is the most common case - then burning lots of cycles busy-waiting for events that haven't happened yet only decreases real hardware utilization and increases either capex or time to completion.

jsnell 11y ago
I'm not quite sure where you're getting that 25% reduction in cores from. Yes, it'd probably be a pretty bad tradeoff to dedicate 25% of the machine to busy-looping on IO just to reduce the IO overhead by a factor of 2. But that's not at all what I'm suggesting. I think there's a reasonable case to be made that application performance measured on two systems with exactly the same hardware would be 2x higher when the IO is moved into user-space.
- notacoward 11y ago
  Kind of depends on the application, wouldn't you say? For some applications, absolutely not. For some applications, maybe yes, assuming the application programmer knows enough not to negate the advantage e.g. by creating lock contention or cache thrashing. And they know enough to use something like LKL (interestingly also from Intel) instead of reinventing their own filesystem-like layer on top of that raw storage. And they don't make huge security blunders. Because if they don't get all of those things right, it doesn't matter if their performance is 2x for one brief shining moment before everything goes to hell. That's not comparing apples to apples. You have to hold the functionality/quality bar constant or else it's meaningless.
wmf 11y ago
I would hope that DPDK apps only poll when they have no other work to do, but I don't know if it's actually set up this way.
- jeff_marshall 11y ago
  The DPDK operates (or at least it did the last time I looked) in a run to completion model for packets received. once the current batch of received packets has been processed by a thread, the DPDK run time immediately looks for more packets to process. The number of polling threads is configurable, and often there are multiple threads per network interface (esp. on 10 gig or faster interfaces) with recieve flow steering to direct related packets to the same receiving thread to maintain cache coherency during packet processing. If you have long-lived work that shouldn't block processing more packets, it would be typical to offload that to separate thread/process from the one doing the packet processing (e.g., for control plane work).
- notacoward 11y ago
  I haven't looked at DPDK specifically (I'd like to) but most such libraries I've seen poll on their own in their own thread. If the application has to tell it to poll, then it's cooperative multitasking; we all saw how much fun that was in Windows or (pre-OSX) MacOS. Polling introduces a classic latency/resource-consumption tradeoff. If you do it too much, you waste resources. If you don't do it enough, you get crappy latency. This is why interrupts were invented, and they work quite well when used correctly. Whether the storage or networking stacks in the Linux kernel uses them correctly is left as a point for the reader to ponder. ;)
  - wmf 11y ago
    IMO people should only try to use DPDK if they're willing to go all in on "1975 programming", so the event loop is a minor annoyance on the scale of things that you have to deal with. Besides, kids today love event loops and callbacks anyway.
    - mtanski 11y ago
eloff 11y ago
Your assuming this uses more cycles, but that's probably false. All the interrupts, context switches, and kernel processing related to IO is a lot of processing. You have control here how many cycles to burn busy waiting, it's the classic latency vs throughout tradeoff. Longer sleep times in between polling means less busy waiting and more latency.