romange

15 ms·

91.
by romange 4y ago
It's exactly that. It gives us a little chance to fight against the Giants. In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda D
92.
by romange 4y ago
If I would choose another language it would be Rust. Why I did not choose Rust? 1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with i
93.
by romange 4y ago
Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you. For intra-process framework it's not an issue (as long as we do not have
94.
by romange 4y ago
And no, it's not just because of io_uring it is faster. It's also because it's multi-threaded, has absolutely different hashtable design, uses a different memory allocator and many other reasons (i.e design decisions we took
95.
by romange 4y ago
we use io_uring for everything: network and disk. Each thread maintains its own polling loop that dispatches completions for I/O events, schedules fibers etc. Everything is done via io_uring API. All socket writes are done via ring buf
96.
by romange 4y ago
Yes, we use io_uring for networking and for disk. io_uring provides a unified interface on linux to poll for all I/O events. Re disk - we use it for storing snapshots. We will use it for writing WALs. And we have more plans for using i
97.
by romange 4y ago
I do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. D
98.
by romange 4y ago
I've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6. I will continue working on DF. Primary/Secondary replication is my n
99.
by romange 4y ago
The fact that we based our core hashtable implementation on paper from 2020 does not justify it?
100.
by romange 4y ago
If I would choose another language it would be Rust. Why I did not choose Rust? 1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in
101.
by romange 4y ago
I do not even know how to to that comparison. Redis and DF share the same protocol and the same API so it's each to compare with memtier_benchmark.
102.
by romange 4y ago
The reason for this is the networking limitations of each instance type. DF consistently reaches the networking limits for each instance type in the cloud. On r6g it's 1.4M qps and then it's saturated on interrupts due to ping pon
103.
by romange 4y ago
You are correct - GET/SET do not require any locking as long as they do not contend and they do not in those benchmarks. You are right that for MSET/MGET you will see lower numbers. But still it will be much higher than with REDIS
104.
by romange 4y ago
Of course, I disabled. Why do you think it's suspicious? P.S. You can easily reproduce it - I wrote the command line and the instance type.
105.
by romange 4y ago
If you look below the throughput table you will see the 99th percentile latency of Dragonfly in the table. Now, please take into account that DF maintains 99th percentile of 1ms at 3M! qps and not at 200K.
106.
by romange 4y ago
True. Keydb tackled the same problems as us. But we chose differrent paths. We decided to go for a complete redesign, feeling that there is a critical mass of innovation out there that can be applied for inmemory store. KeyDb wanted to stay
107.
by romange 4y ago
memcached was born in 2003. Redis was born in 2008. Both have strength and weaknesses. DF tries to unite them into a single backend and keep the strengths of each one of them.
108.
by romange 4y ago
Thank you. And if you are curious to learn more - we would love to share! And we will.
109.
by romange 4y ago
Yes, helio is the library that allows you to build c++ backends easily similar to Seastar. Unlike Seastar that is designed as futures and continuations library, helio uses fibers which I think simpler to use and reason about. I've wrot
110.
by romange 4y ago
Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a sing
111.
by romange 4y ago
Dude, open an issue :) I was waiting for someone to ask for that because I do not know anyone who uses it :)
112.
by romange 4y ago
Yes, I can confirm that Memcached can reach similar performance as DF. However, one of the goals of DF was to combine the performance of Memcached with versatility of Redis. I implemented an engine that provides atomicity guarantees for all
113.
by romange 4y ago
I honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc. With multi-threading you n
114.
by romange 4y ago
It's really good questions. I invite you to try it yourself using memtier_benchmark :) if you pass `--key-maximum=0` you will get a single key contention when doing the loadtest. Spoiler alert - it's still much faster than Redis.
115.
by romange 4y ago
It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem
116.
by romange 4y ago
ScyllaDB and Seastar was my insipration when I thought about the architecture for Dragonfly
117.
by romange 4y ago
Guys, I am the author of the project. Would love to answer any questions you have. Meanwhile will try to do it by replying comments below.
118.
by romange 7y ago
Includes a highly performant RPC engine. Implements a MapReduce framework for single-node, multicore applications. The MR framework works natively with Google Cloud. It's a first open sourced framework that I know of that combines Boos
119.
Modern C++ framework for cloud development(github.com)
1 points by romange 7y ago|1 comments