3 ms·
Aha! I was thinking about trying this exact same approach recently, so I'm glad to see I wasn't insane! Very clever. I have been "in my spare time" (that is, n
by cmrdporcupine 4y ago
Aha! I was thinking about trying this exact same approach recently, so I'm glad to see I wasn't insane! Very clever.
I have been "in my spare time" (that is, never finishing) building out a page buffer mgmt system based on the one in the Umbra paper (see https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf) which has a kind of novel approach where they essentially allocate the same physical address at multiple virtual addresses but with different block/slab sizes so that they can reduce fragmentation and still support dynamic page sizes. Anyways, it seems this author is also on the same "page" (sorry) as me but focusing more on exploiting `io_uring`. Nifty.
I'm working in Rust, not Zig, but a lot of nice stuff to read through here.
I will have to read through this in more detail later when I have more brain-space available.
- gavinray 4y agoJinx, I'm also working on this, as a tutorial series! I had the pleasure of hearing Neumann present the Umbra architecture on the CMU DB series webinar recently. I'm slowly working through re-implementing the Coroutines async buffer pool paper. Blocked on trying to properly implement and understand "Morsel-Driven Parallelism": https://db.in.tum.de/~fent/papers/coroutines.pdf https://db.in.tum.de/~fent/papers/coroutines.pdf I have the async, io_uring buffer pool written in Kotlin using it's native Coroutines but the scheduling and morsel stuff I'm still trying to understand
- cmrdporcupine 4y agoI watched the same CMU lecture, very cool. Are you getting paid to work on this kind of thing, anywhere? Do you want to be?
- gavinray 4y ago> Are you getting paid to work on this kind of thing, anywhere? My dayjob is a fair bit higher level. I work on a sort of query engine on the JVM that translates an internal query IR/AST into dialect-specific SQL + other languages so you can query anything with the same frontend language: https://techcrunch.com/2022/06/28/hasura-now-lets-developers-turn-any-data-source-into-a-graphql-api/ https://techcrunch.com/2022/06/28/hasura-now-lets-developers... I think it'd be neato to work on a database, but my dayjob is also pretty interesting to me.
- gavinray 4y ago> I'm working in Rust, not Zig, but a lot of nice stuff to read through here. FYI, Datadog has a Rust library for scheduling things to run thread-per-core with io_uring It'd be really useful for DB use cases: https://github.com/DataDog/glommio https://github.com/DataDog/glommio
- cmrdporcupine 4y agoNifty. I started from the part of the Umbra presentation that initially struck me as most interesting -- the allocator portion (where he talks about repeatedly MMAP_ANONYMOUS on the same address range to reduce fragmentation, etc.) I stopped working when I started pondering the bits about pointer swizzling and how I'd make that work nicely in Rust, and never got as far as the pieces that would actually do the syncs to disk. I also came across another fellow who was inspired by the optimistic locking btree presented in LeanStore Umbra: https://github.com/NeowayLabs/bplustree https://github.com/NeowayLabs/bplustree