9 ms·

Dragonfly – Performant in-memory data store

77 points by jlokier 3y ago

pella 3y ago
The v1.3.0 version was released 3 days ago. https://github.com/dragonflydb/dragonfly/releases/tag/v1.3.0 https://github.com/dragonflydb/dragonfly/releases/tag/v1.3.0 If you want to integrate it, you also need to check the Software License. https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.md https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.m... Dragonfly Business Source License 1.1 License: BSL 1.1 Licensor: DragonflyDB, Ltd. Licensed Work: Dragonfly including the software components, or any portion of them, and any modification. Change Date: March 15, 2028 Change License: Apache License, Version 2.0, as published by the Apache Foundation. Additional Use Grant: You may make use of the Licensed Work (i) only as part of your own product or service, provided it is not an in-memory data store product or service; and (ii) provided that you do not use, provide, distribute, or make available the Licensed Work as a Service. A “Service” is a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work to third parties as a software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other similar services that compete with Licensor products or services. ...
- dvh 3y ago
  And people are still mocking Stallman.
  - viraptor 3y ago
    What do you mean specifically? This sounds like a pretty cool way to balance the open/commercial aspect to me.
    - rakoo 3y ago
      It's not open at all. I can't offer this as a service with my non profit org, because the non profit still needs a 10€ annual fee to exist. I can't make improvements that truly belong to the community. This is an attempt to fight off capitalism with private property, which means it only tries to fight off capitalism that doesn't benefit them. If you're going that way, might as well close source it. AGPL allows all of that, allows you to run it as a service, allows you to earn money, and if amazon ever reuses it you have access to their differentiating sauce so the playing field is leveled profiting everyone, including you. Most likely though amazon won't use it, which is also the original goal.
      - viraptor 3y ago
        > I can't offer this as a service with my non profit org You mean make a hosted dragonfly a service you provide? (I'm not sure why the €10 matters - maybe I'm missing something) Yeah, that's exactly the point of that licence. You can do it in 5 years if you want.
        rakoo 3y ago
        > You mean make a hosted dragonfly a service you provide? Yes, why not ? > (I'm not sure why the €10 matters - maybe I'm missing something) That's a standard fee members pay to cover expenses the org may have > Yeah, that's exactly the point of that licence. You can do it in 5 years if you want. Good! At least there's a recognition of not being of public service and not doing this for everyone but only for themselves. I wish they chose another license, but it's a start
        viraptor 3y ago
        > Yes, why not ? That's one specific case out of all possible uses that's not allowed with this licence. It honestly doesn't sound like much of a limitation. It would also be an extremely weird service to run as a non-profit. > That's a standard fee members pay to cover expenses the org may have Yes, but why did you mention it? It seems irrelevant to the question of what service you can build.
- tebbers 3y ago
  Sounds fair enough to me. Look what AWS did to Elasticsearch (amongst many others).
- nly 3y ago
  Does it really stop anything? Amazon etc can just "take inspiration" from it and develop something from scratch that's almost the same
  - viraptor 3y ago
    It stops companies from taking it for free. Sure, you could find senior developers experienced in building performant and reliable databases to build one for you in many months. But you need to commit the resources to that and you'll be playing catch-up to what dragonflydb achieved in the meantime.
pearjuice 3y ago
I don't know how well the results hold up after a year, but according to Redis the Dragonfly performance test is biased and with Redis configured properly it reached higher throughput than Dragonfly. YMMV but just putting this up here. Personally I never used Dragonfly so I wonder if the "marketing metrics" actually hold up in production. https://redis.com/blog/redis-architecture-13-years-later/ https://redis.com/blog/redis-architecture-13-years-later/
- youngtaff 3y ago
  Yeh but isn’t the Redis one just a biased? What might have been interesting would be to test on a range of cores / clusters, and consider the overhead of managing 1VM vs 64VMs etc.
  - arp242 3y ago
    The Dragonfly benchmark runs one Redis instance on a 64-CPU machine and compares it with one Dragonfly instance on the same machine. But there is nothing stopping you from running 64 Redis instances on one machine if it has 64 cores, which is what Redis did (actually, they ran just 40). That actually seems like a nicer design overall, as it scales "naturally" to multiple machines without any extra effort/code, it keeps the code simpler, you can also have one of these Redis instances segfault without bringing your entire cache down. Other than that, they seem to have run the same benchmark. YMMV for other types of workloads of course, and perhaps Dragonfly could be configured better in some way. Either way: it seems the Dragonfly benchmark is not just biased, but highly misleading. And while the Redis benchmark may be biased, it certainly doesn't seem highly misleading.
    - ec109685 3y ago
      The should have chosen a 1024 core box and really shocked the world.
    - PlutoIsAPlanet 3y ago
      To me, spinning up multiple copies of the database is cheating. You're comparing a box of Apples to a single apple. Yes, using a Redis cluster is the only way to get Redis to actually use system resources effectively, but its a relatively complex thing to create and manage compared to just running 1 server.
pella 3y ago
other related thread in the HN (today) "KeyDB – A Multithreaded Fork of Redis (keydb.dev)" https://news.ycombinator.com/item?id=36018149 https://news.ycombinator.com/item?id=36018149
monlockandkey 3y ago
How does performance look like on a vCPU machine Vs Redis? Cause this looks interesting. I'm most interested in latency to store the coordinates of thousands of players in real time. Well definitely check this out
electic 3y ago
There was a great HN post of DragonFly vs. some other databases: https://news.ycombinator.com/item?id=31796311 https://news.ycombinator.com/item?id=31796311. I am not sure if the latest version improves performance but those are some bold claims of Dragonfly.
chocolatkey 3y ago
I apologize for the somewhat off-topic nature off this comment, but it really irritates me when I see the website for a product with competition, go to their sitemap, and see this blatant SEO gaming. For example, I went to their FAQ (https://www.dragonflydb.io/faq https://www.dragonflydb.io/faq) expecting things about their database, instead I get a bunch of copycat answers to weird questions intended to drive traffic to their site when people ask questions about other databases. I see many application sites doing this nowadays, and it makes me not want to use them. I imagine with GPT this is very easy to generate now
- Veen 3y ago
  > I imagine with GPT this is very easy to generate now That's exactly what they are doing, but if they had any brains, they'd have a human read the answers before they publish: https://www.dragonflydb.io/faq/is-memcached-good https://www.dragonflydb.io/faq/is-memcached-good > As an AI language model, I am unable to make subjective judgments on whether Memcached is "good" or not. However, I can provide information about the features, benefits, and limitations of Memcached.
  - boundlessdreamz 3y ago
    This is tragically funny. If the dragon fly devs are reading this, stop this nonsense seo gaming. This makes me trust your db less and cheapens everything that you stand for.
    - romange 3y ago
      We are reading this and we will fix it shortly.
      - dangoodmanUT 3y ago
        Seems to be fixed
        viraptor 3y ago
warkanlock 3y ago
Weirdly, in-memory data storage is barely the bottleneck most of the time. Why do we need faster storage? We need better models & domains that improve our developer experience. Not something faster. We don't need cars that go 650km/h. We need better roads and superior security controls Blazingly fast wasn't enough; now we add "on Earth" to be impactful enough to capture our attention
- quags 3y ago
  I use redis with rspamd and have systems that scan millions of mails per day and redis is a bottle neck for some of the modules. I’ve been impressed with dragonfly for the reputation set which has stopped the timeouts I was seeing and was able to replace a multi redis set up using rspamd internal hash/sharding to a single dragonflydb. I plan to replace more over but need stream support which is coming. The dev is also responsive on their discord channel.
  - dangoodmanUT 3y ago
    1M req/day is 12 req/s. If you can't pull event 1k req/s out of it I'd assume it's being used incorrectly.
    - viraptor 3y ago
      A single email doesn't have to mean a single redis query.
      - qeternity 3y ago
        Presume the upper bound of single digit millions (9) for just over 100 emails/second. Also presume a rather pedestrian single instance (single core) Redis limit of 100k/ops/sec. Are people really running software that is making 1k Redis calls/message?
        quags 3y ago
        First let me say redis has served me well for years. Before rspamd I used spamasassassin - another perfectly fine piece of software. Having more options is a good thing. For redis I started with single instances and moved to master (for writes), slaves (for reads), or sharding/hashing depending on what rspamd uses. I moved one section so far to dragonflydb called reputation module. This is performing better for my usage. As for email, if email was broken down and nicely distributed over the same period I likely would have no issues. However, there are times where there is much more scanning going on, and a good 4 hours of peak time. This is where issues can crop up. If rspamd scan can't get a return fast enough, that test will be skipped. It still works, but I'd rather have all tests go through. So lets then take a single email and where redis is used *Ratelimit checking * Replies plugin * Some multi map are stored in redis *mx check module * neural module (a bit like bayes but automatic and short term) * fuzzy module used from spam traps * history module - for redis admin, stats So a single email is doing more than a single redis query for each email. The mail server (for smtp), which is not rspamd, is also using redis on its own. Now imagine a spike of emails, perhaps a large spamming activity that is being stopped. Needing to scan 100 emails in a second isn't impossible at peak. The scanning is also for incoming and outgoing email.
isubasinghe 3y ago
That is a huge claim, almost screams red flag.
dangoodmanUT 3y ago
I never trust metrics from a company anymore, see the bloodbath comment thread on the recent Kafka vs Redpanda for proof.
- avinassh 3y ago
  Their blog post goes into detail on how they are benchmarking and they use the benchmarking tool developed by Redis [1] > We ran our tests on AWS Graviton2 EC2 instances, which are network-optimized and provide the best performance for web applications. We used memtier_benchmark — a widely used benchmarking tool developed by Redis Ltd. — to test throughput and latency, and Prometheus to monitor and visualize memory usage. It is easy to reproduce and check the claims. Redis also countered with a blog post [2] [1] - https://www.dragonflydb.io/blog/scaling-performance-redis-vs-dragonfly https://www.dragonflydb.io/blog/scaling-performance-redis-vs... [2] - https://redis.com/blog/redis-architecture-13-years-later/ https://redis.com/blog/redis-architecture-13-years-later/
jgalt212 3y ago
Once one moves to dragonfly cloud, I guess the "performant" part of the value prop falls by the wayside. i.e. you need your in-memory data store on the same machine (or on the same rack) otherwise really un-peformant solutions can out-perform dragonfly cloud.
sgt 3y ago
> high-performance, low-complexity, and built for scale. So, exactly like redis?
rickstanley 3y ago
Sorry, but I find "Redis API compatible" checked for Redis itself funny. :)
FrostKiwi 3y ago
I instantly confused this with DragonFlyBSD.

77 points by jlokier 3y ago

The v1.3.0 version was released 3 days ago. https://github.com/dragonflydb/dragonfly/releases/tag/v1.3.0 https://github.com/dragonflydb/dragonfly/releases/tag/v1.3.0 If you want to integrate it, you also need to check the Software License. https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.md https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.m... Dragonfly Business Source License 1.1 License: BSL 1.1 Licensor: DragonflyDB, Ltd. Licensed Work: Dragonfly including the software components, or any portion of them, and any modification. Change Date: March 15, 2028 Change License: Apache License, Version 2.0, as published by the Apache Foundation. Additional Use Grant: You may make use of the Licensed Work (i) only as part of your own product or service, provided it is not an in-memory data store product or service; and (ii) provided that you do not use, provide, distribute, or make available the Licensed Work as a Service. A “Service” is a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work to third parties as a software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other similar services that compete with Licensor products or services. ...

dvh 3y ago
And people are still mocking Stallman.
- viraptor 3y ago
  What do you mean specifically? This sounds like a pretty cool way to balance the open/commercial aspect to me.
  - rakoo 3y ago
    It's not open at all. I can't offer this as a service with my non profit org, because the non profit still needs a 10€ annual fee to exist. I can't make improvements that truly belong to the community. This is an attempt to fight off capitalism with private property, which means it only tries to fight off capitalism that doesn't benefit them. If you're going that way, might as well close source it. AGPL allows all of that, allows you to run it as a service, allows you to earn money, and if amazon ever reuses it you have access to their differentiating sauce so the playing field is leveled profiting everyone, including you. Most likely though amazon won't use it, which is also the original goal.
    - viraptor 3y ago
      > I can't offer this as a service with my non profit org You mean make a hosted dragonfly a service you provide? (I'm not sure why the €10 matters - maybe I'm missing something) Yeah, that's exactly the point of that licence. You can do it in 5 years if you want.
      - rakoo 3y ago
        > You mean make a hosted dragonfly a service you provide? Yes, why not ? > (I'm not sure why the €10 matters - maybe I'm missing something) That's a standard fee members pay to cover expenses the org may have > Yeah, that's exactly the point of that licence. You can do it in 5 years if you want. Good! At least there's a recognition of not being of public service and not doing this for everyone but only for themselves. I wish they chose another license, but it's a start
        viraptor 3y ago
        > Yes, why not ? That's one specific case out of all possible uses that's not allowed with this licence. It honestly doesn't sound like much of a limitation. It would also be an extremely weird service to run as a non-profit. > That's a standard fee members pay to cover expenses the org may have Yes, but why did you mention it? It seems irrelevant to the question of what service you can build.
tebbers 3y ago
Sounds fair enough to me. Look what AWS did to Elasticsearch (amongst many others).
nly 3y ago
Does it really stop anything? Amazon etc can just "take inspiration" from it and develop something from scratch that's almost the same
- viraptor 3y ago
  It stops companies from taking it for free. Sure, you could find senior developers experienced in building performant and reliable databases to build one for you in many months. But you need to commit the resources to that and you'll be playing catch-up to what dragonflydb achieved in the meantime.

pearjuice 3y ago

I don't know how well the results hold up after a year, but according to Redis the Dragonfly performance test is biased and with Redis configured properly it reached higher throughput than Dragonfly. YMMV but just putting this up here. Personally I never used Dragonfly so I wonder if the "marketing metrics" actually hold up in production. https://redis.com/blog/redis-architecture-13-years-later/ https://redis.com/blog/redis-architecture-13-years-later/

youngtaff 3y ago
Yeh but isn’t the Redis one just a biased? What might have been interesting would be to test on a range of cores / clusters, and consider the overhead of managing 1VM vs 64VMs etc.
- arp242 3y ago
  The Dragonfly benchmark runs one Redis instance on a 64-CPU machine and compares it with one Dragonfly instance on the same machine. But there is nothing stopping you from running 64 Redis instances on one machine if it has 64 cores, which is what Redis did (actually, they ran just 40). That actually seems like a nicer design overall, as it scales "naturally" to multiple machines without any extra effort/code, it keeps the code simpler, you can also have one of these Redis instances segfault without bringing your entire cache down. Other than that, they seem to have run the same benchmark. YMMV for other types of workloads of course, and perhaps Dragonfly could be configured better in some way. Either way: it seems the Dragonfly benchmark is not just biased, but highly misleading. And while the Redis benchmark may be biased, it certainly doesn't seem highly misleading.
  - ec109685 3y ago
    The should have chosen a 1024 core box and really shocked the world.
  - PlutoIsAPlanet 3y ago
    To me, spinning up multiple copies of the database is cheating. You're comparing a box of Apples to a single apple. Yes, using a Redis cluster is the only way to get Redis to actually use system resources effectively, but its a relatively complex thing to create and manage compared to just running 1 server.

pella 3y ago

other related thread in the HN (today) "KeyDB – A Multithreaded Fork of Redis (keydb.dev)" https://news.ycombinator.com/item?id=36018149 https://news.ycombinator.com/item?id=36018149

monlockandkey 3y ago

How does performance look like on a vCPU machine Vs Redis? Cause this looks interesting. I'm most interested in latency to store the coordinates of thousands of players in real time. Well definitely check this out

electic 3y ago

There was a great HN post of DragonFly vs. some other databases: https://news.ycombinator.com/item?id=31796311 https://news.ycombinator.com/item?id=31796311. I am not sure if the latest version improves performance but those are some bold claims of Dragonfly.

chocolatkey 3y ago

I apologize for the somewhat off-topic nature off this comment, but it really irritates me when I see the website for a product with competition, go to their sitemap, and see this blatant SEO gaming. For example, I went to their FAQ (https://www.dragonflydb.io/faq https://www.dragonflydb.io/faq) expecting things about their database, instead I get a bunch of copycat answers to weird questions intended to drive traffic to their site when people ask questions about other databases. I see many application sites doing this nowadays, and it makes me not want to use them. I imagine with GPT this is very easy to generate now

Veen 3y ago
> I imagine with GPT this is very easy to generate now That's exactly what they are doing, but if they had any brains, they'd have a human read the answers before they publish: https://www.dragonflydb.io/faq/is-memcached-good https://www.dragonflydb.io/faq/is-memcached-good > As an AI language model, I am unable to make subjective judgments on whether Memcached is "good" or not. However, I can provide information about the features, benefits, and limitations of Memcached.
- boundlessdreamz 3y ago
  This is tragically funny. If the dragon fly devs are reading this, stop this nonsense seo gaming. This makes me trust your db less and cheapens everything that you stand for.
  - romange 3y ago
    We are reading this and we will fix it shortly.
    - dangoodmanUT 3y ago
      Seems to be fixed
      - viraptor 3y ago

warkanlock 3y ago

Weirdly, in-memory data storage is barely the bottleneck most of the time. Why do we need faster storage? We need better models & domains that improve our developer experience. Not something faster. We don't need cars that go 650km/h. We need better roads and superior security controls Blazingly fast wasn't enough; now we add "on Earth" to be impactful enough to capture our attention

quags 3y ago
I use redis with rspamd and have systems that scan millions of mails per day and redis is a bottle neck for some of the modules. I’ve been impressed with dragonfly for the reputation set which has stopped the timeouts I was seeing and was able to replace a multi redis set up using rspamd internal hash/sharding to a single dragonflydb. I plan to replace more over but need stream support which is coming. The dev is also responsive on their discord channel.
- dangoodmanUT 3y ago
  1M req/day is 12 req/s. If you can't pull event 1k req/s out of it I'd assume it's being used incorrectly.
  - viraptor 3y ago
    A single email doesn't have to mean a single redis query.
    - qeternity 3y ago
      Presume the upper bound of single digit millions (9) for just over 100 emails/second. Also presume a rather pedestrian single instance (single core) Redis limit of 100k/ops/sec. Are people really running software that is making 1k Redis calls/message?
      - quags 3y ago
        First let me say redis has served me well for years. Before rspamd I used spamasassassin - another perfectly fine piece of software. Having more options is a good thing. For redis I started with single instances and moved to master (for writes), slaves (for reads), or sharding/hashing depending on what rspamd uses. I moved one section so far to dragonflydb called reputation module. This is performing better for my usage. As for email, if email was broken down and nicely distributed over the same period I likely would have no issues. However, there are times where there is much more scanning going on, and a good 4 hours of peak time. This is where issues can crop up. If rspamd scan can't get a return fast enough, that test will be skipped. It still works, but I'd rather have all tests go through. So lets then take a single email and where redis is used *Ratelimit checking * Replies plugin * Some multi map are stored in redis *mx check module * neural module (a bit like bayes but automatic and short term) * fuzzy module used from spam traps * history module - for redis admin, stats So a single email is doing more than a single redis query for each email. The mail server (for smtp), which is not rspamd, is also using redis on its own. Now imagine a spike of emails, perhaps a large spamming activity that is being stopped. Needing to scan 100 emails in a second isn't impossible at peak. The scanning is also for incoming and outgoing email.

isubasinghe 3y ago

That is a huge claim, almost screams red flag.

dangoodmanUT 3y ago

I never trust metrics from a company anymore, see the bloodbath comment thread on the recent Kafka vs Redpanda for proof.

avinassh 3y ago
Their blog post goes into detail on how they are benchmarking and they use the benchmarking tool developed by Redis [1] > We ran our tests on AWS Graviton2 EC2 instances, which are network-optimized and provide the best performance for web applications. We used memtier_benchmark — a widely used benchmarking tool developed by Redis Ltd. — to test throughput and latency, and Prometheus to monitor and visualize memory usage. It is easy to reproduce and check the claims. Redis also countered with a blog post [2] [1] - https://www.dragonflydb.io/blog/scaling-performance-redis-vs-dragonfly https://www.dragonflydb.io/blog/scaling-performance-redis-vs... [2] - https://redis.com/blog/redis-architecture-13-years-later/ https://redis.com/blog/redis-architecture-13-years-later/

jgalt212 3y ago

Once one moves to dragonfly cloud, I guess the "performant" part of the value prop falls by the wayside. i.e. you need your in-memory data store on the same machine (or on the same rack) otherwise really un-peformant solutions can out-perform dragonfly cloud.

sgt 3y ago

> high-performance, low-complexity, and built for scale. So, exactly like redis?

rickstanley 3y ago

Sorry, but I find "Redis API compatible" checked for Redis itself funny. :)

FrostKiwi 3y ago

I instantly confused this with DragonFlyBSD.