3 ms·

The disparity between what the OS can do and what the hardware is capable of delivering is off by a few orders of magnitude right now. It's downright ridiculous

by personZ 12y ago

The disparity between what the OS can do and what the hardware is capable of delivering is off by a few orders of magnitude right now. It's downright ridiculous how much performance we're giving up for supposed "convenience" today. What are they talking about?

Goopplesoft 12y ago
Yea, Im not sure thats a good/accurate bashing. This is a highly optimized tool for a very specific task. OSes have very very wide use cases they have to support.
- fulafel 12y ago
  It's a good and accurate bashing, irrespective of what this tool can do. Even more so once you remember that the networking equivalent of Moore's law tapered off about 10 years ago.
justincormack 12y ago
The operating system APIs (sockets) basically cannot delivery to/from userspace anything like what the hardware can do. The classic example is 10Gb ethernet, where if you want to do something like packet switching, you cannot do anything like line rate, due to cost of context switches, memory copies, scheduling, cache misses in the stack layers and so on. Given that servers are getting 10Gb, 40Gb or more, this is becoming more and more of an issue. The hardware is capable of it, but the abstractions are getting in the way.
- personZ 12y ago
  There is an OS overhead for sure (although you can mitigate that with things like Zero Copy solutions), but it generally imposes a couple of percentage points of overhead. I was thrown off by the several orders of magnitude claim. Reading it again, it sounds more holistic -- that a simplified, monolithic application with a local, lightweight database is much higher performance than, for instance, an n-tier, distributed database type solution.
  - justincormack 12y ago
    It is not just a few percent for applications where userspace needs to look at each packet. There are some performance comparisons on the netmap page for example http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/
    - personZ 12y ago
      That comparison is of questionable merit. It compares bulk, unmetered packet generation (in an OS, worth noting, just as a kernel module) against netsend, which is intentionally a rate limited generator (using busy waits, as an aside, which means that the CPU probably was at 100%...doing nothing), with the significant overhead that entails. where userspace needs to look at each packet That benchmark is like those naive "look how fast my web server is when it returns just status code 200" comments. Even if we accepted that the overhead was anywhere close to the linked, which it isn't, the moment you actually do something with the packets those savings disappear into rounding errors.
      - lrizzo 12y ago
        netsend and other apps used in that comparison were not rate limited. Surely that test only measures system's overheads, but i put that disclaimer very clearly in all papers and talks (btw "which it isn't" suggests that you have different numbers so please let us know). Surely, sometimes these per-packet savings are irrelevant, but there are a number of use cases where this kind of savings matter a lot. This is true for netmap, dpdk, DNA and all other network-stack-bypass frameworks. On passing, people typically use the term 'os-bypass' but netmap relies on the OS for protection, synchronization, memory management etc -- all things that the OS does well and i find no reason to reinvent.
        personZ 12y ago
        Unless you used a specialized, customized version of netsend, yes it was rate limited -- it by default waits on the clock interval, and warns you if you try to call it in defiance of that. Further it is endlessly calculating the time and calling system functions to get the time. As a test of max throughput, it is a horrible test. I don't have the motivation to prove it, but I would be surprised if more than 5% of the CPU load actually went towards networking, the rest time calculations and interval tests.

by personZ 12y ago

Goopplesoft 12y ago

Yea, Im not sure thats a good/accurate bashing. This is a highly optimized tool for a very specific task. OSes have very very wide use cases they have to support.

fulafel 12y ago
It's a good and accurate bashing, irrespective of what this tool can do. Even more so once you remember that the networking equivalent of Moore's law tapered off about 10 years ago.

justincormack 12y ago

The operating system APIs (sockets) basically cannot delivery to/from userspace anything like what the hardware can do. The classic example is 10Gb ethernet, where if you want to do something like packet switching, you cannot do anything like line rate, due to cost of context switches, memory copies, scheduling, cache misses in the stack layers and so on. Given that servers are getting 10Gb, 40Gb or more, this is becoming more and more of an issue. The hardware is capable of it, but the abstractions are getting in the way.

personZ 12y ago
There is an OS overhead for sure (although you can mitigate that with things like Zero Copy solutions), but it generally imposes a couple of percentage points of overhead. I was thrown off by the several orders of magnitude claim. Reading it again, it sounds more holistic -- that a simplified, monolithic application with a local, lightweight database is much higher performance than, for instance, an n-tier, distributed database type solution.
- justincormack 12y ago
  It is not just a few percent for applications where userspace needs to look at each packet. There are some performance comparisons on the netmap page for example http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/
  - personZ 12y ago
    That comparison is of questionable merit. It compares bulk, unmetered packet generation (in an OS, worth noting, just as a kernel module) against netsend, which is intentionally a rate limited generator (using busy waits, as an aside, which means that the CPU probably was at 100%...doing nothing), with the significant overhead that entails. where userspace needs to look at each packet That benchmark is like those naive "look how fast my web server is when it returns just status code 200" comments. Even if we accepted that the overhead was anywhere close to the linked, which it isn't, the moment you actually do something with the packets those savings disappear into rounding errors.
    - lrizzo 12y ago
      netsend and other apps used in that comparison were not rate limited. Surely that test only measures system's overheads, but i put that disclaimer very clearly in all papers and talks (btw "which it isn't" suggests that you have different numbers so please let us know). Surely, sometimes these per-packet savings are irrelevant, but there are a number of use cases where this kind of savings matter a lot. This is true for netmap, dpdk, DNA and all other network-stack-bypass frameworks. On passing, people typically use the term 'os-bypass' but netmap relies on the OS for protection, synchronization, memory management etc -- all things that the OS does well and i find no reason to reinvent.
      - personZ 12y ago
        Unless you used a specialized, customized version of netsend, yes it was rate limited -- it by default waits on the clock interval, and warns you if you try to call it in defiance of that. Further it is endlessly calculating the time and calling system functions to get the time. As a test of max throughput, it is a horrible test. I don't have the motivation to prove it, but I would be surprised if more than 5% of the CPU load actually went towards networking, the rest time calculations and interval tests.