7 ms·
"operating in a polled mode instead of relying on interrupts, which avoids kernel context switches and eliminates interrupt handling overhead" Yeah, that's gre
by notacoward 11y ago
"operating in a polled mode instead of relying on interrupts, which avoids kernel context switches and eliminates interrupt handling overhead"
Yeah, that's great if all you're running is a benchmark. As soon as you need to combine this with a network stack, also polling its own devices at fast as it can, it becomes a lot harder to avoid those context switches. If you're running an actual application it becomes even harder. Likewise if you have more devices than you have cores to spin waiting for them.
In an extremely latency-sensitive and resource-rich environment this kind of thing can yield great results, but otherwise it's almost a form of cheating. Yes, that's what I was accused of when I wrote network drivers at Dolphin and again at SiCortex that some people felt polled too much. Oddly enough, users felt that their CPUs should be running their applications, and were more interested in maximum throughput per cycle consumed. Hardware designers don't build interrupt-based interfaces just for fun, and it's worth remembering that Intel makes most of its money from selling CPUs. Think about it.
- drv 11y agoIt's certainly true that the polled-mode driver model doesn't interact well when the application needs to use other APIs that don't provide a polled mode. However, SPDK can be used in conjunction with a polled user-mode network stack so that a storage application can operate fully in user mode without any user-to-kernel context switches or hardware interrupts. It's definitely not a drop-in replacement for a kernel storage stack in the general case, but rather an optimization for specific applications (e.g. storage appliances) that can be structured to take advantage of the polled/no-interrupts model.
- notacoward 11y agoThank you. That's kind of the point I was trying to make. For a storage appliance - and that's the most common deployment model for the software I work on - it's great. I just don't want to see every "full stack" halfwit trying to use a specialized tool for general-purpose stuff. That often ends up wasting everyone's time - especially that of the people providing the tool.
- vetrom 11y agoThat said, there are cases where it would make sense in a multicore environment to pin one core to storage polling, and another to network polling. You will then, of course, have contention in transitioning data between the two, but there are existing and useful models for eliding much of the locking load there.
- jsnell 11y agoI don't understand where you're getting the increase in context switches from. The expectation should be the opposite, since the network stack is almost certainly going to be user-space and in the same process Does it make sense for every application? Unlikely. Does it make sense for an application that runs on 10k nodes, and where moving all IO to poll mode user-space doubles the number of requests you can serve per unit of time? Saving 5k machines worth of capex buys a lot of engineering complexity. And since we know that systems like this have been done, 2x should not be an unreasonable number. See for example https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-peter_simon.pdf https://www.usenix.org/system/files/conference/osdi14/osdi14... and https://www.cs.cmu.edu/~hl/papers/mica-nsdi2014.pdf https://www.cs.cmu.edu/~hl/papers/mica-nsdi2014.pdf
- notacoward 11y agoIt all depends how many cores you're willing to burn. If you're getting 2x performance but you only have 75% as many CPUs left for your application, you're going to need more than half as many nodes. At SiCortex we were selling into HPC. These people were thoroughly used to working on football-field-sized compute complexes, and they still bitched about how much CPU time was being burned in the I/O stack. No matter how many cycles they had available, they wanted those cycles used for their apps. As I also said before, this kind of thing does have its place. It all depends on how many cycles you're likely to spin before you actually find anything to do, and whether you have another need for those cycles. If it's not many cycles because you really are pushing a lot of I/O, that's great. If you don't need those cycles for something else because you're an appliance and this is your only job, that's great too. If it's a lot of cycles and you do need those cycles for other things - which is the most common case - then burning lots of cycles busy-waiting for events that haven't happened yet only decreases real hardware utilization and increases either capex or time to completion.
- jsnell 11y agoI'm not quite sure where you're getting that 25% reduction in cores from. Yes, it'd probably be a pretty bad tradeoff to dedicate 25% of the machine to busy-looping on IO just to reduce the IO overhead by a factor of 2. But that's not at all what I'm suggesting. I think there's a reasonable case to be made that application performance measured on two systems with exactly the same hardware would be 2x higher when the IO is moved into user-space.
- greglindahl 11y agoYou're experienced in an environment (HPC) where polling for communications works great for compute/network intensive programs. InfiniPath didn't use interrupts, basically, and its latest evolution as Intel's Omni-Path probably does the same thing. Even Mellanox InfiniBand doesn't use interrupts when doing HPC-ish things. This Intel library means that the network and storage can use the same event loop. It will integrate beautifully with Omni-Path's user-level library.
- notacoward 11y agoIt will be great if that actually happens. I'm not down on the idea as an addition to the arsenal; I'm just pointing out that it's not as generally applicable as people might think. User-space I/O is another one of those ideas that has come and gone many times over the last few decades. This time it will find some applications, to be sure, but there are also good reasons why most people will probably continue to be better off sticking with the in-kernel implementations.
- greglindahl 11y agoIt's easy to predict that most people will probably not use this. It's much more fun to talk about why this will be awesome for some.
- rbanffy 11y agoOne reason to use a polling model instead of interrupts is that storage on non-volatile memory is much more predictable than a hard disk.