14 ms·
Guys, I am the author of the project. Would love to answer any questions you have. Meanwhile will try to do it by replying comments below.
by romange 4y ago
Guys, I am the author of the project. Would love to answer any questions you have. Meanwhile will try to do it by replying comments below.
- ddorian43 4y agoDid you think about re-using redis open source and "just" changing storage/replication? (like yugabytedb does with postgresql). Why not reuse seastar framework? Can you describe your distributed log thing? Is it like facebook-logdevice or apache-bookeeper?
- romange 4y agoI honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc. With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++. My choice was using fibers - which provide more natural way of writing code. I have not designed the distributed long thingy. Will do it in the next 2 months.
- deleted 4y ago[deleted]
- treis 4y agoIs there any catch or gotcha for using this as a drop in replacement for Redis?
- tlarkworthy 4y agoI can see it's missing some important commands like WATCH, and maybe the WAL for high durability contexts.
- romange 4y agoIf you see that important command are missing, pks open an issue and we will prioritize their implementation.
- Strom 4y agoA lot less features. If you don't care about the missing features, then it might be a completely reasonable replacement.
- romange 4y agoI've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6. I will continue working on DF. Primary/Secondary replication is my next milestone.
- deleted 4y ago[deleted]
- jkmcf 4y agoIt's only at release 0.1 so I'd let them bake a little longer! Looks awesome so far, though!
- thayne 4y agoThe license is different. Redis core is BSD 3 clause, which is a permissive open source license. Dragonflydb uses the Business Source License, with some additional use grants. Depending on your situation, that could be an impediment.
- rkunde 4y agoDo you know how DFDB compares, architecturally and performance-wise, to Pelikan and its different engines?
- romange 4y agoNot really but I read pelikan posts by twitter team. One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.
- thatwasunusual 4y agoTwo things: Where's the benchmark compared to memcached? Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data). For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.
- soheil 4y agoI find the Redis benchmark suspect. Did you disable write to disk and live snapshotting? In production Redis shouldn’t be configured with write to disk.
- staticassertion 4y agoHave any thoughts on Anna? They take a similar kind of shared-nothing approach, with CALM communication primitives.
- ignoramous 4y agoAnna is a KV store, for those out of loop: https://muratbuffalo.blogspot.com/2022/04/anna-key-value-store-for-any-scale.html https://muratbuffalo.blogspot.com/2022/04/anna-key-value-sto...
- romange 4y agoI remember that I read about Anna. Very interesting paper. From what I remember that require that the operations will be conflict free. I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...
- tdullien 4y agoGood stuff :-) I would also love to see a comparison against Scylla and Aerospike, even if the use-case seems slightly different?
- romange 4y agoI do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. Disk I/O is relativle expensive in the cloud and requires either large instances or special storage optimized family types like i3 etc. Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.
- ddorian43 4y ago> that means they need to touch a disk upon each write or update They don't fsync the wal on every write, it's probably done as group commit every x seconds or every x MB.
- romange 4y agothey need at least perform fdatasync before replying to a client, imho. so I do not think it's every x seconds. I am not deeply familiar with scylla so maybe I am mistaken.
- ddorian43 4y agoI think you're wrong. It does that every x seconds. The disk writing is async and it's meant that you protect data-loss by keeping replicas in different availability zones.
- romange 4y agointeresting. i still need to learn a lot about how other systems provide durability guarantees. Based on what you are saying, Scylla guarantees H/A instead of full durability. I was thinking about WAL (AOF) for dragonfly and I kinda thought to start with "fdatasync once in every X seconds", but dragonfly is not a database. I would expect stronger guarantees from databases. Maybe it's inpractical to expect this in cloud environments...
- ignoramous 4y agoThanks for your work. Curious: Why BSL? Why not open core [0] (or xGPLv3) like what most other commercial OSS projects seem to be doing? [0] https://archive.is/nT9DF#selection-410.0-410.1 https://archive.is/nT9DF#selection-410.0-410.1
- OPoncz 4y agoGPL is copyleft and more restrictive in some elements. BSL 1.1 is actually quite popular nowdays.
- felixg3 4y agoThey probably don't want to end up being replaced by an AWS/GCP/Azure service. In my opinion, the BSL is a fair license model, especially if it is limited in duration (let's say 2-3 years BSL then automatically changing to Apache/BSD).
- ethbr0 4y agoA duration limit in the license, after which it becomes a permissive license, seems the critical point. Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it. Seems to align all interests pretty nicely. (I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)
- romange 4y agoExactly. Finding the right balance is the right thing to do. I am not angry at cloud providers , but i am angry at OSI and their blind idealism that leaves us out there alone. I think that not solving the problem by them, hurt first of all the oss community.
- ignoramous 4y ago> I think that not solving the problem by them, hurt first of all the oss community. F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies. See also: https://news.ycombinator.com/item?id=25896162 https://news.ycombinator.com/item?id=25896162
- oneepic 4y agoApologies if this was already asked -- I saw you guys already showed benchmarks in AWS itself, but would it make sense to have some benchmarks outside of AWS (or run benchmarks in a VM with Redis/DF/Keydb installed within)? I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/RestrictedCommands.html https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/... (edit: added "affect the speed" for clarity)
- romange 4y agoWe compared DF with Redis OSS and KeyDB OSS, we have not used any of managed services for the comparison. AWS was used as a platform for using EC2 and cloud environment.
- AtNightWeCode 4y agoFrom my experience, Redis performance seems to be all over the place depending on the circumstances. In what cases does this solution perform well and where does it fail? Most caches I worked with really love to have everything close by at low latency and works best when the consumers have about as much memory as the cache in the first place.
- baobob 4y agoBenchmarks done on only one metric are often misleading (and in commercial circumstances usually intentionally!). Would love to see visually what trade offs Dragonfly is making to achieve the numbers from the chart at the top of the README. If excellent technical work means there really is no trade off, that would also be a great reason to chart it visually. Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question
- YarickR2 4y agoTradeoff is inability to run on anything besides recent Linux kernels
- tyingq 4y agoI suppose because of io_uring? It does sound like extendible hashing might have downsides in some scenarios also.
- miohtama 4y agoDoes anyone run production Redis/memcached outside Linux any case?
- genewitch 4y agoProbably not many in production, but the "recent kernel" is going to be the more important part. I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all. Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.
- 10000truths 4y agoThe kernel has its own LTS releases, and running Linux in the cloud is a well tested use case. Any library not available in your distro of choice can be built from source and tracked manually, which usually works well enough.
- frankdejonge 4y agoAny intention to implement Redis Streams?
- thecleaner 4y agoWhat prompted you to write DragonflyDB ? How did you know that Redis would consume more memory ?
- romange 4y agoI've wrote in in the background section of Readme. Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch. At some point I decided to challenge the status quo, so I left the company and..and here we are.
- romange 4y agoSorry, missed your second question. Redis is using fork-based save for snapshotting and full replica sync. This means that memory pages are duplicated on write. That consumes more memory than DF implementation that implements algorithmic point-in-time snapshotting. In addition DF was designed to use less overhead memory in its keys table. As for the why, you can read about it here: https://github.com/dragonflydb/dragonfly#background https://github.com/dragonflydb/dragonfly#background
- kristianpaul 4y agoDo you have a saas cloud offering like redis cloud?
- romange 4y agonot yet. We are focused on building the community at this point.
- cultofmetatron 4y agoI gotta say, I looked through some files. I wish my code was this clean. its some of the prettiest c++ I've ever seen
- romange 4y agoomg, can I marry you?
- romange 4y agobtw, it's not just pretty - it has brains too. take it for a spin.
- cultofmetatron 4y agois there an elixir client library?
- ukd1 4y agoI presume you just use a memcache or redis client? So yes?
- pierrebai 4y agoI dunno... I took a look at a single file and found: static constexpr unsigned NUM_SLOTS = Policy::kSlotNum; static constexpr unsigned BUCKET_CNT = Policy::kBucketNum; static constexpr unsigned STASH_BUCKET_NUM = Policy::kStashBucketNum; NUM_, _CNT, _NUM, three different prefix/suffix for what seems to me like the same concept. That just tickled my inner nit-picker.
- simonw 4y agoIs this a project of https://en.wikipedia.org/wiki/Atos https://en.wikipedia.org/wiki/Atos ? How big is the team, how is it funded and suchlike?
- kodbraker 4y agohey, great work. I couldn't find the specifics of the benchmark. Is there, by any chance you compare 1 instance of single threaded redis running on 64vcore to a multithreaded key-value store? Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?
- romange 4y agoThats what i did. I did not try to cheat. Running N redises is a completely different product. You will have n listening ports and your loadtest program will need to connect to each one of them. The power of dragonfly is that it hides all this complexity. You are saying that comparing single threaded redis to df is unfair, and i say - comparing n redises to df is unfair because you compare different products.
- reilly3000 4y agoThis looks like awesome work! I appreciate you operationalizing of some of the best things to come from computer science in the past decade or so. Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.
- romange 4y agoYes, I found many different bottlenecks. I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor. In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem: With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled. As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.
- amluto 4y agoI'm curious: since you're targeting Graviton, and AFAICT Graviton 2 and up support ARM LSE, have you tried directly using LSE for metrics? ARM LSE offers STADD to add to a 64-bit memory location without reading the contents or ordering the access. (I think LDADD with XZR as the destination is identical, but I could be missing some subtlety. I'm not an ARM expert.) You might be able to get good performance by using a global counter and just STADDing to it. x86 will perform terribly if you use XADD because it has no corresponding optimized forms.
- nextaccountic 4y agoHave you thought about also offering dragonflydb as a library, like sqlite? This would avoid context switching when running everything on the same machine.
- romange 4y agoit could be done technically. not sure how widespread that usecase would be. dragonfly is a linking of a library and a main file dfly_main.cc so without this file you will have the lib. Redis is "Remote Dictionary Server". You gonna loose the remote part :)
- obert 4y agofor development and demos it would help a lot, son one just runs the project at hand, instead of managing multiple procs
- secondcoming 4y agoBeing able to use redis as an in-process L1 cache is something I’ve wanted to do. I don’t want the overhead of domain sockets.
- manigandham 4y agoThis is what in-memory data grids can do, look at Apache Ignite for example: https://ignite.apache.org/ https://ignite.apache.org/
- molbioguy 4y agoJust wanted to say 'thanks' for doing such a great job responding to questions and comments. Nice job.
- romange 4y agoThank you!
- beanjuiceII 4y agohow many people have told you to rewrite it in rust so far?
- romange 4y agoQuite a few :) I wish i was 20 again - to have time to learn the language and to implement df in rust :) i am an old with old habits. Btw, I spent a short time coding in rust last year and i understand why the fuzz.
- m2com 4y agoVery cool project! Are there any example projects using early stage Dragonflydb yet?
- efficax 4y agocurious about the consistency guarantees. redis is single threaded which seems old fashioned but guarantees that all transactions are serial, so there is no chance of conflict between reads, no issue with race conditions , a complicated problem to solve once concurrency is introduced . if the redis protocol is used how is this problem addressed?
- romange 4y agoYes this problem is complex and i guess this is why it have not been solved before. We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.
- romange 4y agobtw, please follow me for updates about dragonflydb at https://twitter.com/romanger https://twitter.com/romanger