6 ms·

Umbra: A Disk-Based System with In-Memory Performance [pdf]

epistasis 2y ago
This is a Database System, if you're checking the comments to understand what type of system this is about. The paper appears in 10th Annual Conference on Innovative Data Systems Research, and appearing in that context makes it clear.
hinkley 2y ago
I still maintain that the existence of in memory databases has two main sources: scalability bottlenecks in GC, and storage latency falling behind network latency and staying there. If general purpose programming languages could store the data efficiently in main memory, the feature set of in memory databases is not so high that you can’t roll your own incrementally. But your GC times are going to go nuts, and you’ll go off the rails. If the speed of light governed data access, you’d collect your data locally and let the operating system decide which hot paths to keep in memory versus storage. The last time network was faster than disk was the 1980’s, and we got things like process migration systems (Sprite). Those evaporated once the pendulum swung back.
- slaymaker1907 2y ago
  It depends on what is meant by in-memory database. The most useful kind IMO is the one which actually saves everything to disk, but is not designed like a traditional RDMS in that it assumes everything, including indices, can be saved in memory. Therefore, you don't need a complicated buffer pool system and you don't need to touch disk at all after startup to service read queries. The most simple approach to such a database is just to MMAP a file. This kind of workload is probably the most common in all of software development for the past couple of decades given how plentiful RAM is as well as most applications having some need for storing persistent data.
  - hinkley 2y ago
    I always feel a little weird using memcached because it has never once crashed on us but when it goes down we have a bad time with circuit breakers. We only have problems with memcached when we create them ourselves. Disk backing store would soften that considerably.
    - lanstin 2y ago
      memcached type API with RocksDB backing store is pretty good. Honestly, at this point hasn't every one written some in memory DB with various methods to persist and various consistency models as a result? At this point the magic is in client routing to the appropriate shard without having to redeploy to change the shard configuration and to allow multi-remote callers to access the data and still get access without disk load or mutex/locking around the data. I have a thing where the shards each have sub-shards in process and 1 go-routine per sub shard; communication to/from the remote callers is via channels to a per-request go-routine (or whatever it is gRPC does) and the main subs hard go-routine has no locking on itself. Just a big hash map and a DLL to implement an LRU so I have a hard cap on memory usage, and no allocations for lookups or mutations (just creations).
  - anotherguy0 2y ago
    Since you mentioned MMAP: "Are You Sure You Want to Use MMAP in Your Database Management System?" https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
    - riku_iki 2y ago
      Dataset was 20x times larger than available RAM in that case, so it makes sense that OS cashing was useless and only induced overhead.. Another potential issue was that they compared their mmap code to fio O_DIRECT code, which kinda not clean experiment, fio could be just much more optimized itself..
      - slaymaker1907 2y ago
- avmich 2y ago
  I suspect John Ousterhout https://web.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf https://web.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf would have other ideas.
CurtHagenlocher 2y ago
You can see additional papers from the same group at https://umbra-db.com/#publications https://umbra-db.com/#publications
- VHRanger 2y ago
  But no github!
  - sakras 2y ago
    Unfortunately this is standard for TUM's database group. Their previous database, HyPer was similarly cutting-edge, but was closed source under a proprietary license. Last I heard it got sold to Tableau.
    - iamlucaswolf 2y ago
      Umbra was recently spun out as CedarDB [1]. And Hyper is alive and well at Salesforce/Tableau! The team working on it is still in large parts the original Hyper team from TUM. You can actually download Hyper (as a binary with language bindings) and play around with it [2] for non-commercial use cases. If you think Hyper/Umbra is cool, the TUM database group has lots of other very interesting projects going on at the moment. LingoDB [3] pushes the database-as-a-compiler idea to the extreme by implementing query optimization and compilation query compilation in MLIR. LingoDB is open-source. Also Viktor Leis, who stands behind (among many other things) Hyper's Morsel scheduling and ART indexes as well as Umbra's buffer management recently started a very interesting project [4] to heavily co-design the DBMS together with the OS in a unikernel approach. Really interesting stuff! Disclaimer: I work on Hyper. Views are my own. [1]: https://cedardb.com/ https://cedardb.com/ [2]: https://tableau.github.io/hyper-db/docs/ https://tableau.github.io/hyper-db/docs/ [3]: https://www.lingo-db.com/ https://www.lingo-db.com/ [4]: https://www.cs.cit.tum.de/dis/research/cumulus/ https://www.cs.cit.tum.de/dis/research/cumulus/
beoberha 2y ago
Obligatory link to Neumann’s presentation for the CMU DB lecture series https://m.youtube.com/watch?v=pS2_AJNIxzU https://m.youtube.com/watch?v=pS2_AJNIxzU
zX41ZdbW 2y ago
Umbra in ClickBench: https://github.com/ClickHouse/ClickBench/pull/161 https://github.com/ClickHouse/ClickBench/pull/161 The initial submission didn't reproduce successfully due to a segmentation fault in an attempt to restart it after data loading. But after some changes, it started to work and showed exceptionally good results.
- deleted 2y ago
  [deleted]
- threeseed 2y ago
  Benchmarks: https://benchmark.clickhouse.com https://benchmark.clickhouse.com So definitely compared against PostgreSQL, MariaDB it is significantly faster. On par with lower-end Snowflake.
m3kw9 2y ago
What black magic is this?
nextaccountic 2y ago
Does this use io_uring?
- lvogel 2y ago
  Yes, we do use io_uring
planckscnst 2y ago
> we can achieve comparable performance to an in-memory database system for the cached working set Should I keep reading, or is the title misleading? The abstract seems to say that the system provides memory-comparable performance for data that is... in-memory
- mattb314 2y ago
  The idea is that memory-only data systems like HyPer are able to make design decisions that make them significantly faster that disk-based systems (eg postgres), even when the working set fits entirely within cache for the disk-based system. Umbra attempts to act like an in-memory DBMS when the working set fits in memory while degrading gracefully as the working set grows beyond memory. Agree the title doesn’t have enough detail to see this though.
avinassh 2y ago
Can this beat Redis
- lvogel 2y ago
  Depends on your use case. Redis is a key-value store that is optimized for simpler workloads and is designed as a caching layer. Umbra, on the other hand, is a full fledged, ACID compliant relational database systems with all bells and whistles. However, Umbra can process multiple 100ks of transactions per second. If you're interested in more details, take a look at Figure 6 in this research paper: https://db.in.tum.de/~freitag/papers/p2797-freitag.pdf https://db.in.tum.de/~freitag/papers/p2797-freitag.pdf Disclosure: I work for CedarDB (https://cedardb.com https://cedardb.com), the commercial spinoff of Umbra.

This is a Database System, if you're checking the comments to understand what type of system this is about. The paper appears in 10th Annual Conference on Innovative Data Systems Research, and appearing in that context makes it clear.

hinkley 2y ago

I still maintain that the existence of in memory databases has two main sources: scalability bottlenecks in GC, and storage latency falling behind network latency and staying there. If general purpose programming languages could store the data efficiently in main memory, the feature set of in memory databases is not so high that you can’t roll your own incrementally. But your GC times are going to go nuts, and you’ll go off the rails. If the speed of light governed data access, you’d collect your data locally and let the operating system decide which hot paths to keep in memory versus storage. The last time network was faster than disk was the 1980’s, and we got things like process migration systems (Sprite). Those evaporated once the pendulum swung back.

slaymaker1907 2y ago
It depends on what is meant by in-memory database. The most useful kind IMO is the one which actually saves everything to disk, but is not designed like a traditional RDMS in that it assumes everything, including indices, can be saved in memory. Therefore, you don't need a complicated buffer pool system and you don't need to touch disk at all after startup to service read queries. The most simple approach to such a database is just to MMAP a file. This kind of workload is probably the most common in all of software development for the past couple of decades given how plentiful RAM is as well as most applications having some need for storing persistent data.
- hinkley 2y ago
  I always feel a little weird using memcached because it has never once crashed on us but when it goes down we have a bad time with circuit breakers. We only have problems with memcached when we create them ourselves. Disk backing store would soften that considerably.
  - lanstin 2y ago
    memcached type API with RocksDB backing store is pretty good. Honestly, at this point hasn't every one written some in memory DB with various methods to persist and various consistency models as a result? At this point the magic is in client routing to the appropriate shard without having to redeploy to change the shard configuration and to allow multi-remote callers to access the data and still get access without disk load or mutex/locking around the data. I have a thing where the shards each have sub-shards in process and 1 go-routine per sub shard; communication to/from the remote callers is via channels to a per-request go-routine (or whatever it is gRPC does) and the main subs hard go-routine has no locking on itself. Just a big hash map and a DLL to implement an LRU so I have a hard cap on memory usage, and no allocations for lookups or mutations (just creations).
- anotherguy0 2y ago
  Since you mentioned MMAP: "Are You Sure You Want to Use MMAP in Your Database Management System?" https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
  - riku_iki 2y ago
    Dataset was 20x times larger than available RAM in that case, so it makes sense that OS cashing was useless and only induced overhead.. Another potential issue was that they compared their mmap code to fio O_DIRECT code, which kinda not clean experiment, fio could be just much more optimized itself..
    - slaymaker1907 2y ago
avmich 2y ago
I suspect John Ousterhout https://web.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf https://web.stanford.edu/~ouster/cgi-bin/papers/ramcloud.pdf would have other ideas.

CurtHagenlocher 2y ago

You can see additional papers from the same group at https://umbra-db.com/#publications https://umbra-db.com/#publications

VHRanger 2y ago
But no github!
- sakras 2y ago
  Unfortunately this is standard for TUM's database group. Their previous database, HyPer was similarly cutting-edge, but was closed source under a proprietary license. Last I heard it got sold to Tableau.
  - iamlucaswolf 2y ago
    Umbra was recently spun out as CedarDB [1]. And Hyper is alive and well at Salesforce/Tableau! The team working on it is still in large parts the original Hyper team from TUM. You can actually download Hyper (as a binary with language bindings) and play around with it [2] for non-commercial use cases. If you think Hyper/Umbra is cool, the TUM database group has lots of other very interesting projects going on at the moment. LingoDB [3] pushes the database-as-a-compiler idea to the extreme by implementing query optimization and compilation query compilation in MLIR. LingoDB is open-source. Also Viktor Leis, who stands behind (among many other things) Hyper's Morsel scheduling and ART indexes as well as Umbra's buffer management recently started a very interesting project [4] to heavily co-design the DBMS together with the OS in a unikernel approach. Really interesting stuff! Disclaimer: I work on Hyper. Views are my own. [1]: https://cedardb.com/ https://cedardb.com/ [2]: https://tableau.github.io/hyper-db/docs/ https://tableau.github.io/hyper-db/docs/ [3]: https://www.lingo-db.com/ https://www.lingo-db.com/ [4]: https://www.cs.cit.tum.de/dis/research/cumulus/ https://www.cs.cit.tum.de/dis/research/cumulus/

beoberha 2y ago

Obligatory link to Neumann’s presentation for the CMU DB lecture series https://m.youtube.com/watch?v=pS2_AJNIxzU https://m.youtube.com/watch?v=pS2_AJNIxzU

zX41ZdbW 2y ago

Umbra in ClickBench: https://github.com/ClickHouse/ClickBench/pull/161 https://github.com/ClickHouse/ClickBench/pull/161 The initial submission didn't reproduce successfully due to a segmentation fault in an attempt to restart it after data loading. But after some changes, it started to work and showed exceptionally good results.

deleted 2y ago
[deleted]
threeseed 2y ago
Benchmarks: https://benchmark.clickhouse.com https://benchmark.clickhouse.com So definitely compared against PostgreSQL, MariaDB it is significantly faster. On par with lower-end Snowflake.

m3kw9 2y ago

What black magic is this?

nextaccountic 2y ago

Does this use io_uring?

lvogel 2y ago
Yes, we do use io_uring

planckscnst 2y ago

> we can achieve comparable performance to an in-memory database system for the cached working set Should I keep reading, or is the title misleading? The abstract seems to say that the system provides memory-comparable performance for data that is... in-memory

mattb314 2y ago
The idea is that memory-only data systems like HyPer are able to make design decisions that make them significantly faster that disk-based systems (eg postgres), even when the working set fits entirely within cache for the disk-based system. Umbra attempts to act like an in-memory DBMS when the working set fits in memory while degrading gracefully as the working set grows beyond memory. Agree the title doesn’t have enough detail to see this though.

avinassh 2y ago

Can this beat Redis

lvogel 2y ago
Depends on your use case. Redis is a key-value store that is optimized for simpler workloads and is designed as a caching layer. Umbra, on the other hand, is a full fledged, ACID compliant relational database systems with all bells and whistles. However, Umbra can process multiple 100ks of transactions per second. If you're interested in more details, take a look at Figure 6 in this research paper: https://db.in.tum.de/~freitag/papers/p2797-freitag.pdf https://db.in.tum.de/~freitag/papers/p2797-freitag.pdf Disclosure: I work for CedarDB (https://cedardb.com https://cedardb.com), the commercial spinoff of Umbra.