6 ms·
Dragonfly Is Production Ready (and we raised $21M)
- ensignavenger 4y agoWhen I saw the title, I thought it was a post about DragonFly BSD. But apparently it is about a closed source in-memory database meant as a competitor to Redis.
- worldsoup 4y agonot closed source https://github.com/dragonflydb/dragonfly https://github.com/dragonflydb/dragonfly
- ilc 4y agoNot open either. :)
- omni 4y agoWhat's not open about it? I see the source, I see instructions to build from source, I see a license that seems to say I can copy and make derivative works out of it. Can you try to make a more substantive comment?
- dsrw 4y agoIt's a weird license that seems to say "you can use this as long as you don't compete with us", with an automatic switchover to Apache 2.0 in 5 years. Definitely better than closed source, but probably not Open Source by the OSI definition.
- romange 4y agoBSL license is more company-friendly than AGPL.
- madeofpalk 4y agoBeing company-friendly is a separate point from being open source.
- kakwa_ 4y agoRestricting Usage or Distribution makes it de-facto not OSS. It's actually a slightly different form of an old debate. I'm thinking in particular about the Crockford license (the MIT-like one with "The Software shall be used for Good, not Evil." bit). It was determined to be non-free quite a while back due to such restrictions. That being said, it hard to be a commercially successful software editor with an OSS model (RethinkDB comes to mind). I do understand why BSL exists, but it feels to me like an unsatisfactory compromise.
- geewee 4y agoThe license comes with restrictions on what you can use your derivative works for - e.g. not creating an in-memory datastore service. It's essentially an Apache 2 with a "Also AWS can't just steal it and sell it as a service when it gets huge" Is it open-source? Well, depending on your definition probably not. Is it a fair license? Yeah I'd think so.
- akvadrako 4y agoIt is open-source, no question. What you might mean is that it isn't free/libre as in FOSS.
- codetrotter 4y agoThat’s the Stallman kind of point of view. https://en.m.wikipedia.org/wiki/Open_source https://en.m.wikipedia.org/wiki/Open_source section "Open" versus "free" versus "free and open" But to a lot of us, “open source” has a specific meaning as well. > Open source doesn’t just mean access to the source code. https://opensource.org/osd/ https://opensource.org/osd/
- exelib 4y agoFrom my point of view, for example, GPL is not "open" at all. And yet it is on the list. In my opinion, BSL is even more "open" than GPL. Feel free to have a different view.
- deleted 4y ago[deleted]
- ksec 4y agoOpen Source as defined by OSI, which means BSL is not considered as Open Source. Open Source as defined by majority of programmers on Internet, which means either GPL, MIT, or Apache and their derivatives. Open Source as defined by HN, depending on which timeline you join HN, it could be MIT, BSD only all the way to AGPL only. Open Source as defined by layman, anything I can see its source is considered as Open. Open Source in its literal sense.
- romange 4y agoDragonflyDB cofounder here. I am not shy about our choice of license. Like with software design, everything is about trade-offs. Folks here voiced reasons why we chose BSL. I am sure you perfectly aware about all this. I do not know personally you but I noticed that you posted the link to the announcement. I am guessing you are passionate about the technology and innovation. Dragonfly is much more than the licensing choice we made. I wish HN discussions here were about how fibers work in Dragonfly and how SSD tiering is gonna be implemented and how we provide atomicity for lua scripts while running many of them in parallel etc. Btw, Dragonfly relies on an io-engine called helio (roughly equivalent to tokio) that has been developed by me and open sourced under Apache 2.0.
- ensignavenger 4y agoWhile technical discussion about those details would be interesting, HN is also a strong entrepreneurial community and your license choice has a big impact on whether or not a business would choose to depend on your product. I would not choose to build on a BSL licensed foundation for my product because of real business concerns with vendor lock-in. I would also not choose to contribute to a BSL licensed code base because the CLA means I am not free to use the code base on equal grounds with other contributors (DragonflyDB Ltd in this case). Because of these two things, I didn't spend a lot of time looking into the technical details and therefore can't really say much on them. By all means it sounds like a very interesting piece of technology- but the license makes it useless to me.
- exelib 4y ago
- ensignavenger 4y agoIt is not open to use as you please. For example, I cannot pay the supplier of my choosing to provide services using the code.
- exelib 4y agoTBH, I love BSL licenses. You can use it as you want for free, except being a competitor, and there is a high chance of a sustainable business model. (What benefit do you expect if the company goes bankrupt?) Despite the unpopular opinion, you have free and easy access to the code, so it is open source in the sense of words. Just not in the sense that you can steal their business like AWS does. Feel free to start a similar project, invest your time and money, and make it available under whatever license you want.
- ensignavenger 4y agoIt doesn't matter how much you love it. It is still not open source.
- exelib 4y agoIt is open for me to use as I please. And I don't want to destroy their business. I can understand your ideological drive, but in reality it doesn't matter until you behave unethically and steal their intellectual property. Do you want that?
- ensignavenger 4y agoIf you are sure that what you please is and always will be in line with what their company pleases, then I suppose so. Ideology has nothing to do with it. Nor does "unethically and steal their intellectual property" have anything to do with it- they can choose whatever license they want for their intellectual property and I can ethically choose not to use it. Unless your idea of "ethics" is for force me to use their intellectual property and agree to their license?
- Pet_Ant 4y ago> It is open for me to use as I please. Sure, but that's not what "open source" means. I mean I can go around calling C a functional programming language because it functions and I can build functioning programs with it... but that's not what the word means in anyone's discourse besides yours. I mean I support your freedom of speech, but also mine in saying your usage is disingenuous.
- ensignavenger 4y agoThe license is a closed-source, proprietary, source-available license. It is not open to use as you please.
- bacchusracine 4y ago>DragonFly BSD You too, huh?
- codetrotter 4y agoThere are dozens of us. Dozens!
- mnd999 4y agoI think DragonflyBSD has been production ready for some time.
- FlyingSnake 4y agoI seriously thought a bunch of BSD experts got together and raised funds for making 2023 the "Year of the BSD". On a different note, I wish new projects respect the history of other projects while naming their project. I am sure they would've found a better name for this DB.
- romange 4y agoI made an honest mistake. While I saw DragonflyBSD, I didn't realize its prominence within the BSD community. I assumed that since our project only focuses on Linux, having a similar name in a different "namespace" wasn't a significant issue.
- FlyingSnake 4y agoAssuming you are the project lead, I would implore you to change the name. There are noble precedents in our industry E.g. Firebird -> Firefox. 1: https://en.wikipedia.org/wiki/Firefox_early_version_history#Phoenix_and_Firebird https://en.wikipedia.org/wiki/Firefox_early_version_history#...
- miah_ 4y agoI honestly would rather see a BSD get 21m in funding versus another database startup that won't exist in 4 years.
- romange 4y agoIt does not have to be instead of our startup, really.
- kakwa_ 4y agoHonestly, I don't think giving 21m to DragonflyBSD would be a good idea, not sure how the project could adapt back to a shoestring budget after the money runs out. I would not be against giving them a few 100Ks so that they could get one or two full time developers however. (joke (but also food for thoughts): alternatively, just "safe" invest the 21M. At 3% interest rate, that 630k per year, enough for ~2 to 10 developers depending on the geo. An OSS project can do a lot of things with this kind of budget).
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- O__________O 4y agoTLDR: Dragonfly throughput is 25X faster than Redis for both GET and SET operations. https://dragonflydb.io/blog/scaling-performance-redis-vs-dragonfly https://dragonflydb.io/blog/scaling-performance-redis-vs-dra...
- donatj 4y ago> Dragonfly throughput is 25X higher than Redis for both GET and SET operations. That's unbelievable. In both senses of that phrase.
- fernandopj 4y agoPutting aside what seems to be an amazing new product, but the blog post showing the results are also downplaying what (to me at least) seems to be a 25% or more increase in latency.
- worldsoup 4y agoDragonfly employee here...where are you seeing the 25% increase in latency? that's not the case so want to make sure the post is not confusing.
- reconditerose 4y agoI wouldn't take their latency numbers too seriously since their measure isn't relative to the throughput. It's not always obvious, but the latency of high performance is tightly coupled with throughput. The latency of high performance systems is the server side execution time (for Redis that is a couple of microseconds), the network hop (probably a couple of hundred microseconds), and the congestion on the server (this is the time it takes the server to actually getting around to processing your request, since the server probably needs to handle other requests first before getting around to yours). The congestion, is directly tied to the throughput. The most useful measurement I've seen is you pick a latency target, and evaluate how many QPS you can send to the server that meets that latency target. That gives a fairly simple dimension to compare to.
- bullfightonmars 4y ago> it's worth noting that if we were to reduce Dragonfly's throughput to match that of Redis, Dragonfly would have much lower P99 latency than Redis It sounds like this is a function of higher throughput in the benchmark for Dragonfly.
- akvadrako 4y ago
- skimdesk 4y agoSeems to be Redis/Memcached API-compatible. Can I use it as a drop-in replacement?
- rmurri 4y agoFrom the web page: Dragonfly is an in-memory data store built for modern application workloads. It is fully compatible with the Redis and Memcached APIs, required no code changes to adopt
- romange 4y agoYes, you can - in fact it supports out of the box existing frameworks and clients like redisson, sidekiq and many others
- throwawaymaroon 4y ago[dead]
- andrenotgiant 4y agoI think maybe I'm just small-potatoes, but the only limitations or constraints I've ever run into with Redis are: 1. memory utilization 2. deployment/orchestration 3. bugs I created for myself related to using caches What are the use cases that max out Redis speed/throughput?
- romange 4y agoDragonfly is better with 1. Memory utilization (read the announcement, they mention it) 2. Deployment/orchestration - your initial threshold that forces you to scale horizontally just went up by order of magnitude. In fact for many use-cases you will never need to go horizontally with Dragonfly. 3. Dragonfly also provides a better experience when working with the system. Just a week ago one of the community contributors submitted a PR that introduces automatic recognition of the hot keys: https://github.com/dragonflydb/dragonfly/pull/951 https://github.com/dragonflydb/dragonfly/pull/951 (this feature is not ready for production use yet but we will get there). It also has a built-in open-metrics support, built in cpu-profiler support, fully asynchronous I/O that allows answering INFO commands even under load etc.
- worldsoup 4y agoIt really depends on the type of application but a common one is getting a large spike in traffic beyond the norm (front page on HN, flash sale, etc.) I do think #1 and #2 that you mentioned are more common constraints and are ones that also both addressed in Dragonfly (much more efficient memory utilization and ability to scale vertically which negates the need for complex orchestration)
- mxuribe 4y ago> ...the most performant in-memory datastore for cloud workloads... Oh, ok, so def. not about Dragonfly BSD.
- henry700 4y agoCool initiative. Just watch out for bullshit. Redis reply from some months ago to those benchmarks: https://redis.com/blog/redis-architecture-13-years-later/ https://redis.com/blog/redis-architecture-13-years-later/ (Originally posted 3 months ago on https://news.ycombinator.com/item?id=34231033 https://news.ycombinator.com/item?id=34231033)
- berkle4455 4y agoYep. Dragonfly compares a single-threaded Redis with a multi-threaded Dragonfly. It’s an extremely misleading benchmark.
- dljsjr 4y agoHow is it misleading when the whole point is that Redis can only be single threaded†? That's why Dragonfly (claims) to scale better. If anything, it's the Redis rebuttal that comes across as misleading; the posted announcement is very up front that Dragonfly's value proposition is that you get vertical scaling for free without having the additional ops overhead of a Redis cluster, which is very much not free in terms of maintenance and opportunity cost. †: Redis 6 added threads, but AFAIK this is only for handling connection I/O. Actual database access is still single threaded. The only way I'm aware of to scale Redis is via clustering.
- berkle4455 4y ago> How is it misleading It's misleading because the comparison would be redis cluster vs dragonfly. There's no speed-up if the Redis user isn't fully saturating a single core. The real question is why is it only 25x faster on a 64-vCPU machine? Why isn't it 64x? Does this mean it's 60% slower when the request volume is below the needs of a single-threaded redis? > Dragonfly's value proposition Dragonfly has zero value proposition other than a ticking-time-bomb of pricing fuckery when they're forced to yield a return on that $21M investment.
- romange 4y agoIt compares a process with a listening port with another process with a listening port. To give another example - nobody compares minio with bunch of disks to which you can write separately, and probably more efficiently.
- mmaia 4y agoDoes it support modules from the Redis Stack, like RediSearch? https://redis.io/docs/stack/ https://redis.io/docs/stack/
- Pet_Ant 4y agoLicense: https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.md https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.m... It is source available. Generally can't use it to create a competing product... but also means you cannot combine it with any of the popular open source licenses. Decide for yourself what that means to you.
- tmikaeld 4y agoCongrats on the funding and getting production ready, it's good that KeyDB (and Redis) get some competition. https://docs.keydb.dev/ https://docs.keydb.dev/ Open question, how does Dragonfly differ from KeyDB?
- OPoncz 4y agoKeyDB is a fork of Redis, whereas Dragonfly introduces a brand-new architecture, crafted from the ground up utilizing a share nothing, multi-threaded design. It implements both Redis and memcached APIs
- romange 4y agoKeyDB implements multiple threading with spin-locks that protect a global shared data structure. Dragonfly is built upon shared-nothing architecture where each thread manages its own slice of data, hence no need for classical locks and no contention under high load. It still provides atomicity guarantees but allows multiple transactions to progress independently as long as they do not need exlusive access to the same keys. So basically different approaches to the same promise - scale. Also different trade-offs. Shared-nothing approach has less contention and more flexible transaction framework but inhibits a slightly higher 50%th percentile latency (order of 30usec).