5 ms·

On Redis, Memcached, Speed, Benchmarks and The Toilet

65 points by julien 16y ago

njharman 16y ago
Really don't buy any of OA's weak arguments for speed over scalability. But I do appreciate people taking shots at assumptions. Like mine that "memcached is the fastest."
- antirez 16y ago
  Not over, we need systems that are both fast and scalable. But I suspect than in five years since now most people will consider a much more reasonable thing to shard client side, by hand, and in the most uncool ways a system that can deliver 100k requests/second than an automatically sharding system with 10k requests/second. My guess is that currently the queries/watt figure is not taken seriously enough. With Redis we will shortly start working full time at "Redis cluster" so it's not that we don't care :) Just I think the first goal should be to have a very efficient "single" node, and then going the step forward trying to retain the same speed if possible.
crad 16y ago
memcached's ASCII protocol is known to be slower and less effective than the binary protocol and using TCP for memcached is slower than using UDP. It's not a fair assertion that redis is faster than memcached when you're picking the slowest way to communicate with memcached. Also, one should not expect to see parity in get/set ratios or you're doing it wrong. Cache get with a good hit ratio/high effectiveness is the point of memcached. While memcached gets effectively plateaued in his low concurrency example, redis get performance degraded as you added more clients/concurrency. In an unoptimized use of memcached, high transaction rate environment, I see sets at 10% of gets. Redis may very well be faster, but an effective benchmark should be closer to real world use case and shouldn't hold one party back for parity in protocol complexity.
- antirez 16y ago
  Crad: I compared the same protocols, both TCP, both ASCII protocol. This is a fair comparison for sure with TCP, as with UDP you have many tradeoffs, so this is specifically a comparison of the TCP (reliable) protocol you can use with Redis. About the binary protocol: I think this is currently not the widely deployed one, but the point here is that if the difference is about the protocol, even Redis could switch to a faster protocol using a binary one. Using almost the same protocol in both servers we are actually measuring the internals. Anyway your reasoning can be applied the other way around. Many memcached systems are currently using CAS extensively while you can perform just one query with Redis atomic operations in many of the same real world scenarios. So should I double the Redis numbers by two because there is a faster way to do something in a Redis specific way? About SET/GET ratio, I measured both SET and GET performances in an independent way, so I don't get it.
  - crad 16y ago
    I get your reasoning about ASCII/TCP and it makes sense... apples to apples. When asking for a performance benchmark, specifically about systems that are very dependent on communication protocol for performance, I think that it's more valuable to know how the best performing protocol of either side would perform. As far as atomic vs CAS, I'd be more interested in the "right way" than the same way, so for my own interest, yes. Re: Ratio, from your graph it appeared they were co-mingled. If each was done on their own then my point is moot.
  - saurik 16y ago
    This assumes the people developing memcached decided to spend a lof of time optimizing their implementation of the ASCII protocol, despite knowing that performance conscious users wouldn't use it anyway; while it may very well be true (I don't remember), it doesn't seem like something that is safe to take for granted.
pvg 16y ago
Ooof, antirez, get someone to help you proofread these things because in some places they nearly make no sense. "Stressing scalability so much put somewhat in shadow what in the 80's was one of the most interesting meters of software coolness [...]" "Unfortunately I also think that our dude is part of the problem, since running crappy benchmarks is not going to help our pop culture." "Apple to Apple, with the same mouth"
- antirez 16y ago
  Sorry I'm trying to improve my English but progresses are really logarithmic at this point... Well also the fact that I try to write this blog posts in very little time (it took me two hours at max, probably even a mother tongue should alloc more time to end with an high quality blog post) does not help either.
  - cullenking 16y ago
    I wouldn't stress about your english skills; I would bet the vast majority of us have no problem following your ideas.
  - nkurz 16y ago
    Your English is great. I'm a native speaker, and reading your post, I thought you were too. A slightly sloppy one with a couple odd turns of phrase, but perfectly clear and comprehensible. I understood the quoted parts just fine. That said, please don't make excuses about how little time you spent on it. It might be true, but it shows lack of respect for your audience. Like code, if it's worth writing at all, it's probably worth writing well. Keep writing, keep coding!
    - antirez 16y ago
      Thank you nkurz, sorry about the excuses about little time, I did not argumented it enough: the fact is that I really love to share what I'm doing using blog posts, so I tend to work hard to achieve something that's worth sharing, but then I'm so excited about sharing it that I write the post as fast as possible! I should try to resist some more hours in order to obtain a post that is more consistent and easy to follow... instead I tend to push the "publish" button and even when the post is already live I continue fixing it... Thanks for the comment!
weixiyen 16y ago
I'm Redis using it for: - session management - message queue (previously using 0mq pub sub) - memory cache (LRU argument = weak) That's previously 3 technologies I had to manage, now only having to install 1. That means only having to worry about 1 driver, and only dealing with 1 API. Less technologies in the stack is a good thing, esp if you get a performance boost as well. The current bottleneck is of course an easy way to scale, and 2.2 comes with Redis Cluster. The other problem some have indicated is the LRU implementation, which imo is a non-issue in almost all use cases.
- robhu 16y ago
  Are you using it as a LRU cache? I'd really like to use Redis for this -- are there any examples out there of how to do it / could you share how you've done it?
  - weixiyen 16y ago
    set max-memory inside the config, it'll automatically do it for you as long as you are using expire on the keys. The implementation according to antirez is that they sample 3 random keys, and expire the soonest one, so put your high priority data with a longer expire and you'll be fine.
    - robhu 16y ago
      Thank you ^_^

65 points by julien 16y ago

Really don't buy any of OA's weak arguments for speed over scalability. But I do appreciate people taking shots at assumptions. Like mine that "memcached is the fastest."

antirez 16y ago
Not over, we need systems that are both fast and scalable. But I suspect than in five years since now most people will consider a much more reasonable thing to shard client side, by hand, and in the most uncool ways a system that can deliver 100k requests/second than an automatically sharding system with 10k requests/second. My guess is that currently the queries/watt figure is not taken seriously enough. With Redis we will shortly start working full time at "Redis cluster" so it's not that we don't care :) Just I think the first goal should be to have a very efficient "single" node, and then going the step forward trying to retain the same speed if possible.

crad 16y ago

memcached's ASCII protocol is known to be slower and less effective than the binary protocol and using TCP for memcached is slower than using UDP. It's not a fair assertion that redis is faster than memcached when you're picking the slowest way to communicate with memcached. Also, one should not expect to see parity in get/set ratios or you're doing it wrong. Cache get with a good hit ratio/high effectiveness is the point of memcached. While memcached gets effectively plateaued in his low concurrency example, redis get performance degraded as you added more clients/concurrency. In an unoptimized use of memcached, high transaction rate environment, I see sets at 10% of gets. Redis may very well be faster, but an effective benchmark should be closer to real world use case and shouldn't hold one party back for parity in protocol complexity.

antirez 16y ago
Crad: I compared the same protocols, both TCP, both ASCII protocol. This is a fair comparison for sure with TCP, as with UDP you have many tradeoffs, so this is specifically a comparison of the TCP (reliable) protocol you can use with Redis. About the binary protocol: I think this is currently not the widely deployed one, but the point here is that if the difference is about the protocol, even Redis could switch to a faster protocol using a binary one. Using almost the same protocol in both servers we are actually measuring the internals. Anyway your reasoning can be applied the other way around. Many memcached systems are currently using CAS extensively while you can perform just one query with Redis atomic operations in many of the same real world scenarios. So should I double the Redis numbers by two because there is a faster way to do something in a Redis specific way? About SET/GET ratio, I measured both SET and GET performances in an independent way, so I don't get it.
- crad 16y ago
  I get your reasoning about ASCII/TCP and it makes sense... apples to apples. When asking for a performance benchmark, specifically about systems that are very dependent on communication protocol for performance, I think that it's more valuable to know how the best performing protocol of either side would perform. As far as atomic vs CAS, I'd be more interested in the "right way" than the same way, so for my own interest, yes. Re: Ratio, from your graph it appeared they were co-mingled. If each was done on their own then my point is moot.
- saurik 16y ago
  This assumes the people developing memcached decided to spend a lof of time optimizing their implementation of the ASCII protocol, despite knowing that performance conscious users wouldn't use it anyway; while it may very well be true (I don't remember), it doesn't seem like something that is safe to take for granted.

pvg 16y ago

Ooof, antirez, get someone to help you proofread these things because in some places they nearly make no sense. "Stressing scalability so much put somewhat in shadow what in the 80's was one of the most interesting meters of software coolness [...]" "Unfortunately I also think that our dude is part of the problem, since running crappy benchmarks is not going to help our pop culture." "Apple to Apple, with the same mouth"

antirez 16y ago
Sorry I'm trying to improve my English but progresses are really logarithmic at this point... Well also the fact that I try to write this blog posts in very little time (it took me two hours at max, probably even a mother tongue should alloc more time to end with an high quality blog post) does not help either.
- cullenking 16y ago
  I wouldn't stress about your english skills; I would bet the vast majority of us have no problem following your ideas.
- nkurz 16y ago
  Your English is great. I'm a native speaker, and reading your post, I thought you were too. A slightly sloppy one with a couple odd turns of phrase, but perfectly clear and comprehensible. I understood the quoted parts just fine. That said, please don't make excuses about how little time you spent on it. It might be true, but it shows lack of respect for your audience. Like code, if it's worth writing at all, it's probably worth writing well. Keep writing, keep coding!
  - antirez 16y ago
    Thank you nkurz, sorry about the excuses about little time, I did not argumented it enough: the fact is that I really love to share what I'm doing using blog posts, so I tend to work hard to achieve something that's worth sharing, but then I'm so excited about sharing it that I write the post as fast as possible! I should try to resist some more hours in order to obtain a post that is more consistent and easy to follow... instead I tend to push the "publish" button and even when the post is already live I continue fixing it... Thanks for the comment!

weixiyen 16y ago

I'm Redis using it for: - session management - message queue (previously using 0mq pub sub) - memory cache (LRU argument = weak) That's previously 3 technologies I had to manage, now only having to install 1. That means only having to worry about 1 driver, and only dealing with 1 API. Less technologies in the stack is a good thing, esp if you get a performance boost as well. The current bottleneck is of course an easy way to scale, and 2.2 comes with Redis Cluster. The other problem some have indicated is the LRU implementation, which imo is a non-issue in almost all use cases.

robhu 16y ago
Are you using it as a LRU cache? I'd really like to use Redis for this -- are there any examples out there of how to do it / could you share how you've done it?
- weixiyen 16y ago
  set max-memory inside the config, it'll automatically do it for you as long as you are using expire on the keys. The implementation according to antirez is that they sample 3 random keys, and expire the soonest one, so put your high priority data with a longer expire and you'll be fine.
  - robhu 16y ago
    Thank you ^_^