4 ms·
Doesn't Oracle (and I think reaches waaaaay back in the memory banks DB2) do something similar in the sense that it is packaged up with its own VM/storage subsy
by kjs3 3y ago
Doesn't Oracle (and I think reaches waaaaay back in the memory banks DB2) do something similar in the sense that it is packaged up with its own VM/storage subsystems that bypasses the native OSes facilities and is hyper-optimized for the DB use case?
- cmrdporcupine 3y agoAll serious databases are doing their own low-level buffer pool management for relations&tuples, which bypass e.g. malloc, for a bunch of reasons. There are various techniques for this. The papers I link to in my comment refer to some recent ones. My point is that in the end what a DB needs to do at this level is translate tuple or btree etc. node references into physical memory references; either by a fault that pulls them from disk, by a fault that maps a virtual address into the kernel as physical memory, or by a live reference, etc. And it feels to me that a lot of the logic there mirrors what already happens in the OS's VM subsystem. (For various reason's the "automagic" facilities that do things like this in the OS -- file-backed mmap'd -- are a terrible way to implement a DB, BTW: https://db.cs.cmu.edu/mmap-cidr2022/ https://db.cs.cmu.edu/mmap-cidr2022/) That and of course at the persistent storage layer the data structures are tuned around physical device performance characteristics. Back in the days to deal with the mechanics of head & cylinders etc; these days, the interface and aspects of flash storage, write amplification, etc. Hence log structured merge trees, b-epsilon-trees, etc. etc. The persistent storage layer inside the DB looks very much like a filesystem. I mean, it kind of is one, but with a different concept of namespace (relations&tuples&versions etc instead of files and directories) So, I dunno, I'd maybe rather drink straight from the spring rather than get a bottle of bubbly water at the restaurant. Or something something weird analogy.
- miohtama 3y agoOracle did its own storage on block level devices (disk partition) as far back as 2000. I don’t know if it still does it.
- cmrdporcupine 3y agoI have never used it but I've heard people talk about that as an essentially failed path. Admins apparently hated it, and the performance likely wasn't all that. However, we are in a different era now. Virtualization and cloud infrastructure changes this whole situation quite a bit.
- emaginniss 3y agoAs I recall, Oracle on Solaris was able to go direct to the disks and experienced a lot of perf improvements over similar HPUX or Linux machines
- kjs3 3y agoThis is my recollection as well. I was at a big Solaris shop when I cared enough about DBs to dig in this deep, but there were many, many discussions with the ops folks about whether this bypass was a good thing, and in the end the performance metrics were judged 'better', for values of better I don't accurately recall.
- crabbone 3y agoThe database itself assumes the existence of a filesystem. The whole toolset from Oracle, also includes a filesystem that you can install and use with the database, but the database itself cannot just work with a bare block device.
- kjs3 3y agoI think that's exactly what I said.