6 ms·
Efficient String Compression for Modern Database Systems
- mbfg 9mo agoI wonder how one does like queries.
- speed_spread 9mo agos, jst cmprss ll qrs b rmvng vyls!
- hcs 9mo agoAfter decompression, with the performance characteristics you'd expect. If it has to come off disk it's still a win or at least usually breaks even in their measurements. https://cedardb.com/blog/string_compression/#query-runtime https://cedardb.com/blog/string_compression/#query-runtime The paper suggests that you could rework string matching to work on the compressed data but they haven't done it.
- ForHackernews 9mo agoNever heard of CedarDB. Seems to be another commercial cloud-hosted thing offering a Postgres API? https://dbdb.io/db/cedardb https://dbdb.io/db/cedardb https://cedardb.com/blog/ode_to_postgres/ https://cedardb.com/blog/ode_to_postgres/
- switz 9mo agoI was evaluating it recently but it's not FOSS, so buyer beware. I'm totally fine with commercialization, but I hesitate to build on top of data stores with no escape hatches or maintenance plans–especially when they're venture backed. It is self-hostable, but not OSS.
- cmrdporcupine 9mo agoIt's a startup founded by -- and built with tech coming out of research by -- some well known people in the DB research community. Successor to Umbra, I believe. I know somebody (quite talented) working there. It's likely to kick ass in terms of performance. But it's hard to get people to pay for a DB these days.
- atombender 9mo agoIt's probably going to be acquired. The last effort to commercialize the TUM (Technical University of Munich) database group's work was acquired by Snowflake and disappeared into that stack. CedarDB is the commercialization of Umbra, the TUM group's in-memory database lead by professor Thomas Neumann. Umbra is a successor to HyPer, so this is the third generation of the system Neumann came up with. Umbra/CedarDB isn't a completely new way of doing database stuff, but basically a combination of several things that rearchitect the query engine from the ground up for modern systems: A query compiler that generates native code, a buffer pool manager optimized for multi core, push-based DAG execution that divides work into batches ("morsels"), and in-memory Adaptive Radix Tries (never used in a database before, I think). It also has an advanced query planner that embraces the latest theoretical advances in query optimization, especially some techniques to unnest complex multi-join query plans, especially with queries that have a ton of joins. The TUM group has published some great papers on this.
- senderista 9mo agoAre you thinking of Hyper being acquired by Tableau?
- atombender 9mo agoMy bad. HyPer was acquired by Tableau, which was acquired by Salesforce.
- senderista 9mo agoUmbra is not an in-memory database (Hyper was). TUM gave up on the feasibility of in-memory databases several years ago (when the price of RAM relative to storage stopped falling).
- cmrdporcupine 9mo agoYeah I think the way Umbra was pitched when I watched the talks and read the paper was as more as "hybrid" in the sense that it aimed for something close to in-memory performance while optimizing the page-in/page-out performance profile. The part of Umbra I found interesting was the buffer pool, so that's where focused most of my attention when reading though.
- crazygringo 9mo agoI'm genuinely surprised that there isn't column-level shared-dictionary string compression built into SQLite, MySQL/MariaDB or Postgres, like this post is describing. SQLite has no compression support, MySQL/MariaDB have page-level compression which doesn't work great and I've never seen anyone enable in production, and Postgres has per-value compression which is good for extremely long strings, but useless for short ones. There are just so many string columns where values and substrings get repeated so much, whether you're storing names, URL's, or just regular text. And I have databases I know would be reduced in size by at least half. Is it just really really hard to maintain a shared dictionary when constantly adding and deleting values? Is there just no established reference algorithm for it? It still seems like it would be worth it even if it were something you had to manually set. E.g. wait until your table has 100,000 values, build a dictionary from those, and the dictionary is set in stone and used for the next 10,000,000 rows too unless you rebuild it in the future (which would be an expensive operation).
- groundzeros2015 9mo agoHow do you layout all that variable length data in memory m?
- hinkley 9mo agoThere are some databases that can move an entire column into the index. But that's mostly going to work for schemas where the number of distinct values is <<< rowcount, so that you're effectively interning the rows.
- analyst74 9mo agocompression is not free, dictionary compression: 1, complicates and slows down update, which is typically more important in OLTP than OLAP 2, is generally bad for high cardinality columns, which requires tracking cardinality to make decisions, which further complicates things. lastly, additional operational complexity (like the table maintenance system you described in last paragraph) could reduce system reliability, and they might decide it's not worth the price or against their philosophy.
- ww520 9mo ago
- ayuhito 9mo agoDuckDB has one of my favourite articles on this topic if you want something a little more high level: https://duckdb.org/2022/10/28/lightweight-compression https://duckdb.org/2022/10/28/lightweight-compression
- vismit2000 9mo agoHonorary mention: https://github.com/pytries/marisa-trie https://github.com/pytries/marisa-trie
- lor_louis 9mo agoI've implemented a similar system based on the original 2020 paper, but we applied it to the text log to try to "extract" similar features from free-form text. It looked promising and even supported full regex search, but the work was ultimately abandoned when we got acquired.
- semiinfinitely 9mo agoits a lesser-know fact that LLMs are SOTA at lossless string compression
- boulos 9mo agoThis isn't strictly correct: you probably mean wrt compressed size. Compression is a tradeoff between size reduction and compression and decompression speed. So while things like Bellard's tz_zip (https://bellard.org/ts_zip/ https://bellard.org/ts_zip/) or nncp compress really well they are extremely slow compared to say zstd or the much faster compression scheme in the article. It's a totally different class of codec.
- semiinfinitely 9mo agoan LLM can be used to losslessly compress a string to a size equal to the number of bits of entropy of next token prediction loss over the string, by encoding the extra bits of entropy with arithmetic encoding. its sota compression for the distribution of string found on the internet an insightful video on the topic: https://www.youtube.com/watch?v=dO4TPJkeaaU https://www.youtube.com/watch?v=dO4TPJkeaaU
- vivzkestrel 9mo agopardon my stupid question, what does cedardb do that postgres, duckdb, cassandra and the 758452758421 databases that came before dont?
- cmrdporcupine 9mo agogo fast
- cespare 9mo agoThis scheme reminds me of smaz (https://github.com/antirez/smaz https://github.com/antirez/smaz) but with dynamically generated dictionaries.
- cryptonector 9mo ago^F intern -> not found. But string interning is what they're doing, isn't it? > Dictionary compression is a well-known and widely applied technique. The basic idea is that you store all the unique input values within a dictionary, and then you compress the input by substituting the input values with smaller fixed-size integer keys that act as offsets into the dictionary. Building a CedarDB dictionary on our input data and compressing the data would look like this: That's string interning!! Is interning just too old a concept now and it has to be rediscovered/reinvented and renamed?
- 0928374082 9mo ago> rediscovered/reinvented and renamed It's not old yet; it'll be old once it's been renamed two or three times...
- masklinn 9mo agoString interning only stores whole strings, dictionary compression stores substrings. Essentially string interning is a trivial subset of dictionary compression (because you don’t need a substringing scheme, which is the hard part).
- snackbroken 9mo agoDoesn't interning usually refer to when you only consider identical copies, as opposed to dictionary compression where you allow for concatenations? E.g. Interning: 1: "foo" 2: "bar" my_string = "foo" // stored as ref->1 my_other_string = "foobarbaz" // not found & too long to get interned, stored as "foobarbaz" Dictionary compression: 1: "foo" 2: "bar" my_string = "foo" // stored as ref->1 my_other_string = "foobarbaz" // stored as ref->1,ref->2,"baz" (or ref->1,ref->2,ref->3 and "baz" is added to the dict)
- latch 9mo agoMakes me think of an iolist in erlang/elixir.
- Someone 9mo agomy_string = "foo" // stored as ref->1 Only if you explicitly intern the string. Interning can be expensive because - the runtime has to check whether the string already is interned. - you’re permanently creating the string, so if it isn’t reused later, that means your memory usage needlessly goes up. Both get more expensive the more strings you intern. I think interning is a hack that very rarely, if ever should be used. It likely won’t work well for large strings, as these tend to be unique, and use cases where it helps for shirt strings often are better handled by using enums or symbols, or by using a custom set of strings. If you do the latter, you have more control over Emory usage; you can do such things such as removing the least recently used strings, or ditching the entire cache when you’re done needing it (prime example: parsing a large XML file with many repeated nodes)
- stephantul 9mo agoThe compression algorithm is very similar to a greedy subword tokenizer, which is used in BERT and other older language models, but has become less popular in favor of BPE.
- solatic 9mo agoReally don't understand. Their own cherry-picked example benchmarks show that there is frequently no benefit, after enabling FSST, to query runtimes and sometimes query runtimes are actually negatively affected. Despite hard empirical evidence against FSST, they decided to enable it across-the-board? Not even "hey, we built this cool thing which may or may not improve your workloads, here's how to opt-in", but pushing straight up performance regressions for their customers with no way of opting-out? Why would I ever trust my data to engineers who are more interested in their academically-interesting resume-driven-development than in actual customer outcomes?