11 ms·
KeyDB – A Multithreaded Fork of Redis
- colesantiago 3y agoThere is also DragonflyDB which claims to be faster than both Redis and KeyDB. https://www.dragonflydb.io/ https://www.dragonflydb.io/
- bionsystem 3y agoAre they linked in any way with DragonflyBSD ?
- fweimer 3y agoUnlikely, it's not even open source: https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.md https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.m...
- Buttons840 3y agoI've never seen this license. It looks like it will be open source under the Apache license after 5 years though. That is, I can save the code now, and in 5 years I can do whatever I want with it (under the Apache license terms). When open-source isn't an option, then this is the next best thing.
- solarkraft 3y agoHuh, interesting concept. It's definitely a big step above "source available". It even kind of allows maintaining a community version with outside patches, albeit slowly. It kind of rivals the KDE/Qt deal of "freely licensed when the company goes under" in its effects of the code eventually being community-maintainable once the company doesn't care for it anymore. 5 years is a bit much though.
- rollcat 3y agoIt is open source, you can browse the source all you like. It's not free software. BSL-style licenses seem to be a popular choice for databases, thanks to AWS.
- erk__ 3y agoWell "source available" would probably be better. It is not open source by the definitions most follow in this case the OSI definition. Eg it goes against §6 of their definition. https://opensource.org/osd/ https://opensource.org/osd/
- Un1corn 3y agoPlease don't try to delude people by changing the definition of open source. While sadly Open Source Initiative were not able to get the trademark for open source, the de-facto definition of open source is practically the same as free software. Dragonfly is source available which is a completely different thing.
- xyzzy_plugh 3y agoThat's just like, your opinion, man. I think the parallels to free software are markedly correct. They're just words after all. It will forever be used in ways incompatible with the OSI definition, showing up after every misuse to correct folks isn't helpful. You meant free as in beer, right?
- rollcat 3y ago> Please don't try to delude people by changing the definition of open source. Don't blame it on me, that ship has sailed over two decades ago. That's why RMS didn't like the term in the first place. Even if I disagree with RMS on most things, I have to admit I'm 100% with him on this one. It's almost as if the term was coined to create this kind of confusion. In my opinion, the mental gymnastics around the definition of "open source" led to abominations like CDDL, which was carefully and explicitly designed to make it impossible/impractical/illegal to properly integrate ZFS or DTrace with Linux. CDDL is perfectly "open source" by definition, but its primary purpose was to lock people out of actually using software licensed under it, unless they happen to be running Solaris. In all this mess, I actually think BSL is cool. It's a legally binding vow to actually make a particular release free (as in freedom) down the line. They could have kept it proprietary (which I think is totally fair), or made vague promises instead.
- Kiro 3y agoThis comment made me think Dragonfly is a much better choice: "We use keydb at work, and I absolutely do NOT recommend it due to its extreme instability, in fact we're currently in the process of switching to dragonfly precisely due to keydb's instability." https://news.ycombinator.com/item?id=35990897 https://news.ycombinator.com/item?id=35990897
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- lopkeny12ko 3y agoWe evaluated DragonflyDB for Memcache. It was repeatably orders of magnitude slower under default configurations than original Memcache, using their own benchmark setup. Either they didn't even test their own product, lied entirely about the performance, or got the marketing department to write the copy without any input from the development department.
- romange 3y agoI personally benchmarked Dragonfly vs Memcached. Are you calling me a liar? :) Do you think I also photoshopped this document? https://github.com/dragonflydb/dragonfly/blob/master/docs/memcached_benchmark.md https://github.com/dragonflydb/dragonfly/blob/master/docs/me...
- lopkeny12ko 3y agoI assume you are the lead developer or someone in an exeuctive position associated with Dragonfly. Your defensive, holier-than-thou attitude and tone here and elsewhere is another reason we decided not to adopt Dragonfly internally. Yes, your results are either inaccurate or deceptive at best. I challenge you run to memcached, under all default settings, and Dragonfly, under all default settings, and memtier_benchmark, under all default settings. Performance is reproducibly orders of magnitude slower, and Dragonfly is also much less efficient--consuming more than double the CPU usage for the same workload. We also created a test Dragonfly cluster mirroring a small percentage of production traffic in order to do a side-by-side comparison with Memcache. Dragonfly consumed 47% higher CPU usage and regressed P99 latency by 22%. Perhaps our workload is unique, but claiming Dragonfly outperforms Memcache the way you do in your marketing material is an outright lie.
- romange 3y agoI apologize. I must say that my tone, as you rightly wrote, was inappropriate. Indeed, I am the lead developer for Dragonfly. As such, I am deeply concerned with the performance aspects of our product. Dragonfly claims to be a drop-in, better performant replacement for Redis and Memcached. Every test & benchmark we've run on multiple cpus reinforced that. I've never faked or tweaked any of these benchmarks. That is, of course, not an excuse, and is why I opened by apologizing. I'd like to take this opportunity, if I can kindly ask so, to learn what made your results differ so much from ours. I'll personally try to reproduce what you described. If you could also reach out to me, I'd be happy to learn more about the environments in which you've conducted the aforementioned tests.
- andrewstuart 3y agoI’m surprised some Rust enthusiasts have not cloned Redis.
- revskill 3y agoBecause the original Redis is the fastest already ?
- andrewstuart 3y agoRedis is single threaded, in a world in which 16 cores/32 threads is affordable.
- eternalban 3y agoNot clear how a rust "clone" fits into this. I wonder if people have tried pinning instances to cores and treating the ensemble as a host device bounded distributed variant. (just musing, haven't looked at Redis for ages.)
- secondcoming 3y agoWe do this as a poor man's cluster... a python script starts a redis-server process on each core and the 'master' process has a key that lets clients know about the other processes running on the machine. It only really works well if the client can shard the redis command to the right process itself.
- dijit 3y agoIf your goal is to fully utilise the hardware you have then doing some deterministic sharding on top will likely be good enough. Honestly, redis makes pretty sane tradeoffs, it's not worth the added complexity to add multi-threading as it would almost certainly slow it down while it does locking, and redis isn't typically CPU bound (except potentially the LUA stuff, but that's up to the user), so being multi-threaded doesn't really help much. If you need multi-threading, there are other solutions, but of course they are slower w.r.t latency and throughput, since that's the trade-off.
- gizzlon 3y agohttps://github.com/Snapchat/KeyDB/issues?q=is%3Aopen+is%3Aissue+label%3A%22Priority+1%22+ https://github.com/Snapchat/KeyDB/issues?q=is%3Aopen+is%3Ais... :O
- yard2010 3y agoSo many open bugs and each one of them sounds worse than the other... Use after free.. I'll wait for the Rust clone ;)
- alphanullmeric 3y ago[flagged]
- henry700 3y agoI can't hear you over the literal backbone of the internet running in Rust inside Cloudflare right now
- alphanullmeric 3y ago[dead]
- cies 3y agoGood point. I remember Redis is designed as single thread as it makes the design simpler. Now with Rust we can actually manage complexity from multiple threads (if that's even still needed when using an async/evented/eventloop/io_uring-based architecture).
- misnome 3y ago[Crash]
- secondcoming 3y agoWe run into the following one quite regularly: https://github.com/Snapchat/KeyDB/issues/465 https://github.com/Snapchat/KeyDB/issues/465
- xerxes901 3y agoI winced when I saw the phrase "multithreaded fork" but then I realised this is the _other_ kind of fork, not the fork(2) syscall.
- mgaunard 3y agoWhenever I see someone arguing their code is better because it's multithreaded, I cringe. Most developers cannot do multithreading correctly, and unless you're particularly good about it it's just going to introduce not only lots of bugs but also performance problems. The only folks in that space that seem to do it well are ScyllaDB.
- ddorian43 3y agoScyllaDB doesn't do multithreading. It has 1 thread per core.
- akoncius 3y agonot do dismiss anything or what.. but how does this kind of multithreading (one thread per core) is better than proper multithreading (many threads per core)?
- mgaunard 3y agoDoing multiple threads per core is not "proper multithreading", since it's a well-known antipattern.
- kristiandupont 3y ago..it's not good because it's bad? Why is it an anti pattern, this is news to me?
- janhaa 3y agoMore threads on a single core equal more context switches, which reduces the effective amount of instructions you can process.
- kristiandupont 3y agoWell sure, but why would that make it "improper multithreading"? Is polymorphism based on vtables not "proper OOP"? We rely on many abstractions that aren't free in terms of CPU cycles because it makes development easier or less error prone. And setting up, say, one thread per HTTP request will likely be negligible because blocking I/O is where time is spent anyways..
- SillyUsername 3y agoI'm confused. Redis is a cache with persistent backing option. This is a database. Have I missed something?
- shawabawa3 3y agoRedis is a database It's often used as a cache because it does key-value storage in memory well
- SillyUsername 3y agoIt rebranded in 2015. It used to describe itself as a cache (earlier revisions also declare themselves to be a memory store) https://web.archive.org/web/20150304014343/http://redis.io/ https://web.archive.org/web/20150304014343/http://redis.io/ I consider it to be cache first, "db" second, with true definition of db first being something that can execute SQL or SQL like statements (such as Cassandra's CQL). It's the same reason I don't call Cassandra a cache, although it can achieve the same result.
- JohnBooty 3y agotrue definition of db first being something that can execute SQL or SQL like statements Pedantic note: the term "database" existed long before the relational model or SQL existed. Many of the dominant databases of the 80s and 90s (dBase, etc) would not fit your invented definition. Additionally, a lot of "toy" databases like Access can execute SQL statements, so the ability to execute SQL statements isn't necessarily a great way to tell what's a "real" database. In practical terms, I do agree with you -- if somebody in 2023 is referring to "the database" in their app they had darn well better be talking about something robust and ACID-compliant like Postgres or whatever.
- viralpraxis 3y agoYour definition does not make any sense. Imagine defining « prime numbers » as {2,3,5} set
- Zip3d 3y ago
- secondcoming 3y agoLooks like the person behind KeyDB has lost interest, but I was just made aware of dragonflydb.io.
- dig1 3y agoWithout the intention to undermine anyone's work (and I truly appreciate the work the KeyDB guys did), I would not use this (in production) unless I see Jepsen test results [1] either by Kyle or someone supervised by Kyle. But, unfortunately, databases are complex enough and distributed systems even more, and I've lost many sleepless nights debugging obscure alternatives that claimed to be faster in edge cases. And the worst thing is that these projects get abandoned by the original authors after a few years, leaving me with (to quote a movie) "a big bag of odor". I see databases as a programming languages - if they have a proven track record of frequent releases and responsive authors after 5-7 years, they are usable for wider adoption. [1] https://jepsen.io/analyses https://jepsen.io/analyses
- bhouston 3y agoIt is a Snapchat project though so it is likely at least somewhat decent compared to a random unfunded project. Maybe snap could pay Jensen to review it?
- VWWHFSfQ 3y agoI think I'll stay far away from this thing anyway. Numerous show-stopper bug reports open and there hasn't been a substantial commit on the main branch in at least a few weeks, and possibly months. I'll be surprised if Snap is actually paying anybody to work on this.
- berkle4455 3y ago[flagged]
- wcarron 3y agoThat's so ridiculously tone deaf, self-righteous, and demeaning. Lots of talented devs work in thousands of companies. Learn some humility and perspective.
- hotstickyballs 3y agoGiven how much of a typical code base does depend on databases I think that’s a good perspective.
- silverwind 3y agoKeyDB is useful for it's Multi-Master mode, but other than that, I can not recommend it because of how many serious bugs it has.
- avinassh 3y agohow does it handle conflicts in multi-master mode?
- flagrant_taco 3y agoI believe it includes A Thunderdome protocol. Two matters ever, one master leaves.
- joshxyz 3y agohow does this compare to dragonfly which is also redis alternative?
- byby 3y agoSide question about redis: Does redis do async operations? I'm not sure how that would work because it's known to be in memory. But I do know it persists to disk. So basically my question is: Does almost absolutely every operation on a redis database happen serially? Maybe not every single operation, but in general.
- drwiggly 3y agoEvery user/client operation is serial.
- byby 3y agoThis is mainly because it's single threaded and operates on in memory state. Async io doesn't apply to in memory so all operations have to be serial, is this characterization correct? Thanks.
- T0pH4t 3y agoSort of. To my knowledge, the original redis was literally signal threaded and performed io on the same threads as managing the DB (asynchronously of coarse). Then after either redis 5 or 6 they added IO threads to redis such that the DB stayed managed by a single thread and network/disk IO happened on the IO threads.
- byby 3y agoIt makes sense for disk persistence to be io but redis is primarily in memory. So I don't think async applies to that aspect of it. In theory that makes it so all operations on the database from the user perspective are serial. Not sure about this that's why I'm asking here for a definitive concrete answer about this.
- bdhcuidbebe 3y agoYou want databases to be boring. Stick to tried and true stuff so you can focus on your product rather than beta testing fortune-500 crap.
- notmypenguin 3y agoA) key-value is MUCH simpler to implement than an RDBMS, code wise. It’s arguably more boring and less theoretical than indices on PG or foreign key constraints B) there’s a lot of tricky stuff with indices on PG and you generally need a DB admin from day 1 C) your comment is probably more appropriate for either layered databases or new fangled stuff like time series or graph db’s etc
- bdhcuidbebe 3y agoIn this context I mean state of the art db:s like redis or memcached.
- ploppyploppy 3y ago> B) there’s a lot of tricky stuff with indices on PG and you generally need a DB admin from day 1 Sorry this is absolute nonsense. Any software engineer worth their salary should be comfortable working with RDBMS index concepts and interrogating their relational model to determine best practice and direction for table indexing.
- Patrickmi 3y agoHonestly I’ve haven’t seen “The Redis Alternative” there’re some out there but it feels like “oh this is meant for us but it’s open source”
- efficax 3y agoredis being single threaded is actually a feature since you have guarantees about consistency and serializability
- rafaelturk 3y agoLooks really cool, but What is the goal? Redis already something instantly fast, and single thread brings a lot of benefits. Also Redis is multithreaded in a few smart things (sync to disk is a separated thread)
- teaearlgraycold 3y agoAnyone using an obscure database in production is about to learn quite a few lessons they really need to learn.
- brunoluiz 3y agoOut of curiosity: has anyone found or know the list of trade-offs? There is not free lunch, and the single-threaded model gives certainly consistency guarantees… but what else?
- blablablub 3y agoFor people who like multitreaded forks, you can also have a look at tendis and kvrocks https://github.com/Tencent/Tendis https://github.com/Tencent/Tendis https://github.com/apache/incubator-kvrocks https://github.com/apache/incubator-kvrocks
- ivolimmen 3y agoSo how does this compare to: edis, dragonfly, vedis, iceFireDb, qdb and rlite?