5 ms·
Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective o
by jandrewrogers 13y ago
Issues like the ones raised are why all sufficiently advanced database engine designs tend to evolve toward a kernel bypass architecture. From the perspective of a database engine, operating systems do a lot of "dumb" things with resource management and scheduling in ways that are essentially impossible to avoid that the database engine has enough context to do intelligently on its own. OS bypass in a database can have substantial performance benefits and add robustness for many edge cases. The obvious downside is that you basically end up rewriting the operating system in userspace, minus the device drivers, which is an enormous amount of work. It is not an incremental kind of design approach and it makes portability difficult. PostgreSQL actually does a really good job trying to be fast and robust without going to bypass internals.
I've always asserted that the reason we've never seen a proper kernel bypass database engine in open source is that the Minimum Viable Product is too complex. A bare, stripped-down, low-level database engine that does full bypass of the operating system is usually at least 100kLoC of low-level C/C++, and that is before you add all the features a database user will actually want. That is a big initial investment by some people with fairly rare software design skills.
- dfox 13y agoThese two issues are actually perfect counterpoint to bypassing kernel caching and scheduling. There is no way to overcome first issue in userspace, second is trigered by what amounts to too agressive caching in userspace, and third is something that you should not be doing anyway (I'm not exactly sure if there is something better that kernel can do in that case, except doing write-thru caching on writes which has it's own performance implications for other users of O_DIRECT).
- jandrewrogers 13y agoI think you aren't understanding what a bypass kernel does. It literally takes control of the physical resources to the extent kernel interfaces exist that allow it to do so and, at least in the case of Linux, the level of control possible is quite high. That means taking control of the CPU, physical memory, disk I/O, network I/O, etc so that they can be scheduled and managed from userspace. All at the same time starting with bypass kernel initialization. For obvious reasons, it is usually a bad idea to run other non-trivial processes on the same machine because they will tend to be resource starved. Once you have these resources, you can organize them and use them as you see fit. Because it is not going to the kernel for any resources or buffering or scheduling or memory etc, there is little opportunity for the OS to do the wrong thing with resources that already are tightly controlled by the runtime. However, this is also why it is an "all or nothing" kind of situation.
- twoodfin 13y agoInteresting. When I read your first comment, I didn't realize you meant "bypass kernel" in as wide a scope as you describe here. Usurping CPU scheduling, in particular, was something I didn't think was common practice. Can you name a (presumably commercial) DBMS that represents this kind of "self-manage everything" architecture?
- jandrewrogers 13y agoMost of the big "enterprise" OLTP databases are designed this way; the less portable they are, the more likely they are doing deep bypass optimizations. DB2, SQL Server, and similar are bypass designs. Oracle used to be a weird hybrid, due to portability requirements, but since they took control of the hardware I would assume recent versions are mostly pure bypass. Most commercial analytical databases are not bypass, due in large part to the fact that most of them are based on Postgres, ironically.
- mattzito 13y agoAnd one of the reasons that software like Oracle is so complicated and expensive. Oracle spent years getting the OS kernels out of the way, partially for platform consistency/supportability sake and partially for performance. They obviously still depend on the kernel for a variety of things, but memory management, networking, filesystems, etc. can all be done in Oracle-space at this point.
- beagle3 13y agoBut what it buys Oracle is not performance, but rather bragging rights for being a little faster at a moment in time. However 3-6 months later, you'll get comparable performance from improved kernel, CPU and disk speeds. Are those 20% in performance for 6 months worth the premium oracle is charging (which, in part, reflects their harder work)? For most customers most of the time the answer is no. If you depend on performance, you don't use Oracle in the first place - Vyahu, OneTick, kdb, Vertica are the speed demons (as well as TimesTen which was acquired by Oracle - but is distinct from their "standard" offering)
- drzaiusapelord 13y agoHow does this work? I don't see how its possible to fight the scheduler or how caching works via userspace. Maybe I don't understand this stuff, but maybe the kernel should have some bypass API for high performance applications, instead of coders finding curious ways to fight it.
- dap 13y agoYou can bypass the scheduler by binding processes to processors. Some runtimes do this -- and sometimes get much worse performance as a result. My colleagues and I have seen many programs that think they know better than the OS do much worse for trying.
- usefulcat 13y agoYou can bypass filesystem caching on Linux using O_DIRECT. The tradeoff (besides the obvious lack of caching) is that there are specific alignment restrictions on the length and address of the userspace buffers and file offsets, and these restrictions may vary by filesystem and by kernel.
- jandrewrogers 13y agoThe kernel does have bypass APIs, or APIs that can be used for that purpose. However, you can't use just a little; once you start down that path you need to bypass everything. To be clear, while the bypass APIs are simple to use you actually have to know what you are doing since you become responsible for doing things the OS used to do for you. I/O scheduling, disk caching, process scheduling, memory management, etc all have to be reimplemented in userspace. It is why I mentioned that the skill set required to do bypass kernels is fairly rarified. You can't just reimplement what the OS already does, you need to implement something that is different than the OS design but also better at providing functionality the OS provides for the use case. You are essentially writing a purpose-optimized OS without the device drivers.
- jeffdavis 13y agoI assert that open source databases don't do it because it's a bad idea. It's the kind of you do when you have a lot of spare engineering resources and not many innovative ideas. Not only is the initial devepment expensive, so is the maintenance burden. It makes every new idea cost more to implement. Postgres has been extraordinarily innovative; offering things like transactional DDL, advanced indexing, first-class extensibility, serializable snapshot isolation, per-transaction durability, sophisticated constraints (e.g. non-overlapping ranges), etc. These features are possible because postgres didn't get bogged down reimplementing and maintaining a filesystem. And what would all of that work gain, anyway? +25% single-node performance? Not a very strategic direction to go for databases. Better to improve JavaScript/json support and get a good logical replication solution in place. That being said, there are some low level features that are really worth doing. Robert Haas (the author) did some great work with lockless algorithms, which has achieved great concurrency with a manageable maintenance burden.
- jandrewrogers 13y agoYou both overestimate the engineering complexity and underestimate the benefits. I've both designed and worked on a couple different bypass kernels as well PostgreSQL internals over the years. You are correct that the initial development is steep. However, once the infrastructure is there it really is not much different than working with the operating system infrastructure and you gain a level of predictability and stability in terms of behavior that saves engineering time. Also, bypass implementations have almost no locking internally (either "lock-free" types or heavier types) which reduces complexity considerably. Some bypass kernel code bases allow you to compile with the bypass implementation disabled, using highly-optimized PostgreSQL-like internals. I've seen and run quite a few comparative benchmarks on the same design with and without bypass enabled, as well as absolute benchmarks against engines like PostgreSQL. We don't have to guess about single node performance. Broadly speaking, a properly designed bypass kernel buys you 2-3x the throughput of a highly optimized non-bypass kernel in my experience. If it was only 25% no one would bother. Furthermore, for massively parallel databases, you essentially require a bypass kernel to design a well-behaved system due to the adaptive operation scheduling requirements. I agree that it is a lot of work but it is also entirely worth it if you need to either (1) maximize throughput on a single node and (2) build a well-behaved massively parallel database kernel. The differences are not trivial.
- hcarvalhoalves 13y agoMaybe it's b possible to port a database to a model like OpenMirage's [1], building against all the runtime it requires but running on top of a hypervisor? [1] http://www.openmirage.org/ http://www.openmirage.org/
- ape4 13y agoHow about a special filesystem api for databases?
- pekk 13y agoAt this point why aren't you just building the database on top of a minimal OS? Then run that in a VM, say
- ori_b 13y ago> Then run that in a VM You think that VMs aren't subject to the host OS scheduler, caching, and memory allocation quirks?
- Dylan16807 13y ago>scheduler eh, it could pin threads at realtime priority >caching vm wouldn't use the OS cache >memory allocation it can easily allocate memory up front