5 ms·

These two issues are actually perfect counterpoint to bypassing kernel caching and scheduling. There is no way to overcome first issue in userspace, second is t

by dfox 13y ago

These two issues are actually perfect counterpoint to bypassing kernel caching and scheduling. There is no way to overcome first issue in userspace, second is trigered by what amounts to too agressive caching in userspace, and third is something that you should not be doing anyway (I'm not exactly sure if there is something better that kernel can do in that case, except doing write-thru caching on writes which has it's own performance implications for other users of O_DIRECT).

jandrewrogers 13y ago
I think you aren't understanding what a bypass kernel does. It literally takes control of the physical resources to the extent kernel interfaces exist that allow it to do so and, at least in the case of Linux, the level of control possible is quite high. That means taking control of the CPU, physical memory, disk I/O, network I/O, etc so that they can be scheduled and managed from userspace. All at the same time starting with bypass kernel initialization. For obvious reasons, it is usually a bad idea to run other non-trivial processes on the same machine because they will tend to be resource starved. Once you have these resources, you can organize them and use them as you see fit. Because it is not going to the kernel for any resources or buffering or scheduling or memory etc, there is little opportunity for the OS to do the wrong thing with resources that already are tightly controlled by the runtime. However, this is also why it is an "all or nothing" kind of situation.
- twoodfin 13y ago
  Interesting. When I read your first comment, I didn't realize you meant "bypass kernel" in as wide a scope as you describe here. Usurping CPU scheduling, in particular, was something I didn't think was common practice. Can you name a (presumably commercial) DBMS that represents this kind of "self-manage everything" architecture?
  - jandrewrogers 13y ago
    Most of the big "enterprise" OLTP databases are designed this way; the less portable they are, the more likely they are doing deep bypass optimizations. DB2, SQL Server, and similar are bypass designs. Oracle used to be a weird hybrid, due to portability requirements, but since they took control of the hardware I would assume recent versions are mostly pure bypass. Most commercial analytical databases are not bypass, due in large part to the fact that most of them are based on Postgres, ironically.

by dfox 13y ago

jandrewrogers 13y ago

I think you aren't understanding what a bypass kernel does. It literally takes control of the physical resources to the extent kernel interfaces exist that allow it to do so and, at least in the case of Linux, the level of control possible is quite high. That means taking control of the CPU, physical memory, disk I/O, network I/O, etc so that they can be scheduled and managed from userspace. All at the same time starting with bypass kernel initialization. For obvious reasons, it is usually a bad idea to run other non-trivial processes on the same machine because they will tend to be resource starved. Once you have these resources, you can organize them and use them as you see fit. Because it is not going to the kernel for any resources or buffering or scheduling or memory etc, there is little opportunity for the OS to do the wrong thing with resources that already are tightly controlled by the runtime. However, this is also why it is an "all or nothing" kind of situation.

twoodfin 13y ago
Interesting. When I read your first comment, I didn't realize you meant "bypass kernel" in as wide a scope as you describe here. Usurping CPU scheduling, in particular, was something I didn't think was common practice. Can you name a (presumably commercial) DBMS that represents this kind of "self-manage everything" architecture?
- jandrewrogers 13y ago
  Most of the big "enterprise" OLTP databases are designed this way; the less portable they are, the more likely they are doing deep bypass optimizations. DB2, SQL Server, and similar are bypass designs. Oracle used to be a weird hybrid, due to portability requirements, but since they took control of the hardware I would assume recent versions are mostly pure bypass. Most commercial analytical databases are not bypass, due in large part to the fact that most of them are based on Postgres, ironically.