4 ms·
Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of conne
by jdw64 2mo ago
Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy.
But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be:
'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'
But I think using Redis is much more elegant. It's easier to scale. I'd even argue that something like Saga would be a better approach. Of course, we might just have different opinions. But in my experience, reducing layers always ends up making things more complicated in the long run.
p.s. We have different views, but I do think some of your points are valid, so I upvoted your comment
- codedokode 2mo agoThe fsync policy equivalent to SQL database would be "fsync on each change before reporting successful update to the app". Redis (as many NoSQL databases) also doesn't have SQL and is a pain to view the data, you need to write extra tools when investigating the problems. RDB snapshots can cause multiple page faults due to use of fork() and CoW. > It's easier to scale The company in question manages online stores and they could easily scale by allocating a separate database for each store (sharding). > But I think using Redis is much more elegant. I cannot agree because I think using a single database for all the data is more elegant, than multiple different databases and there are less problems to deal with. I dislike microservice-style architecture strongly and believe it is mostly good for wasting company's money. > 'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!' I read it as "do not create unnecessary work by using a single database".
- jdw64 2mo agoIt's interesting that our views differ. I think it's because of our different experiences. I believe MSA is the right approach. But this doesn't seem like a debate that can be resolved through discussion. It's something that needs to be implemented and tested. Still, I respect your perspective and your experience. We clearly have different values, but I think you have a mature engineering mindset. Ultimately, I think only real measurements can settle this. Have a great day.
- codedokode 2mo agoI became curious how fast fdatasync (a better version of fsync) is, and asked LLM to generate a microbenchmark for me. The benchamrk accepts records size, number of records, opens a file, and appends records one by one, making fdatasync after each one. On my SSD, the throughput is 120 fsyncs/second, or ~0.5 MB/s. So if I had Redis to flush the log after each operation, it wouldn't be able to serve those thousands of connections. That's why Redis is either used without AOF, or they flush the data once per second. And that's why SQL databases might look "slow". Of course one could optimize this - for example, while fsync is being executed, we could accept the queries from other clients and execute them, and once previous flush finishes, flush multiple transactions at once. However, I am not sure if Redis can do this due to being single-thread. And obviously maybe there are problems with drivers, or with my consumer-level SSD and maybe "professional" SSDs can do more flushes per second. Writing the code took less than a minute. I often do microbenchmarks now because it is so easy.
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- jdw64 2mo agoConcurrent connection count and durable write throughput are different things. And AOF doesn't necessarily mean one fsync per record. The benchmark is taking the worst-case speed of Redis and comparing it to a best-case SQL benchmark. Redis AOF isn't one fsync per record—Redis can use group commit to share fsync across multiple commands. In other words, that could mean a difference of tens of times in your benchmark. I see it differently.
- CoolCold 2mo agomy common dumb test approach with fio fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=220m --bs=2300 --name=mytest Consumer grade SSDs 100-500 iops, datacenter SSDs in thousands (like ~ 5 year old samsung model on Hetzher AFAIR had ~ 3000 iops in that test). Cheap VPSes - easily in 5-30 iops of this sort more often then not, fsync/fdatasync is overlooked by programmers