8 ms·

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis.

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-usage CPUs on the app server fleet and mouth "But... why?" Replacing Redis with "something faster" is a bit like removing the doors on a car because "lighter means faster!". It might look good on a racetrack, but it's about as pragmatic as climbing through a window every morning before setting off for work.

jdsully 7y ago
If the perf isn’t needed we also have Active Replication, subkey expiries, and S3 integration. Multithreading is how we got our start and the reason most people use us though.
manigandham 7y ago
True, it's rarely just raw performance. KeyDB has other advantages like multi-threading and disk-based persistence (instead of being limited by RAM) that makes it better at utilizing your server resources and handling larger scales.
- azhng 7y ago
  But isn't the point of Redis is to be used as a caching layer instead of persistent layer
  - t0mbstone 7y ago
    But what if you could have the speed of RAM-based storage, and still have persistence? Sounds pretty appealing to me, personally.
    - penagwin 7y ago
      You already get this with either Redis or Keydb. The problem is that to avoid the disk being the bottleneck you must write to RAM, then later to the disk (you can't wait for confirmation that it was written) which means that even though a key may successfully be written to the DB, it hasn't been confirmed to persist. Basically redis can't necessarily guarantee data safety - which is perfectly fine for it's normal use case as a caching layer and data you don't necessarily care if you lose (for example rate limiting relies on tracking the number of requests/s within say an hour - if you lose the data you don't really care). User sessions too, worst case people have to log in again, meh whatever.
      - manigandham 7y ago
        The major bottleneck for most applications using Redis is the RAM capacity, not the write performance. Disk persistence can be more than fast enough as proven by dozens of key/value stores out there.
    - lllr_finger 7y ago
      Sounds like an embedded key value store, ie. RocksDB. I've used it with great success in constrained environments and also when pushing cache to the edge for stable, low-latency, high volume APIs.
  - nine_k 7y ago
    You can use redis as a write-back cache that serves spikes well and streams the state to disk (or an RDBMS) in a rate-limited manner in the background.
  - manigandham 7y ago
    Redis is a data structure server rather than a simple key/value store. It's very useful in lots of scenarios, even if the persistence means it doesn't have ACID semantics. It would be great to remove limitations like RAM-only capacity in exchange for a slight performance hit, and while also gaining better core utilization. We used ScyllaDB (a very fast cassandra clone) in the past for the cpu/disk scalability but always felt Redis offered better APIs. Now it's a real option.
- jrochkind1 7y ago
  > ...it's rarely just raw performance. KeyDB has other advantages like multi-threading... What are the advantages of "multi-threading" other than for performance?
  - manigandham 7y ago
    Taking advantage of all your cores. Redis is awkward at high capacity since most servers scale CPU with RAM and you'll end up with most cores doing nothing. You can manually run multiple instances on the same server but now they're separate databases with operations and sharding overhead. Multithreading IO also reduces the latency hit from disk-persistence and provides more concurrency and throughput, which is a great tradeoff when you don't really need sub-millisecond performance but want the same API across a larger dataset limited by disk instead of RAM space.
    - fwip 7y ago
      That's just performance spelled differently.
      - manigandham 7y ago
        Ok fair. Disk being primary storage is still a ops and scalability benefit.
    - rumanator 7y ago
      > Taking advantage of all your cores. That means nothing at all, if performance is not your concern.
    - jrochkind1 7y ago
      >> What are the advantages of "multi-threading" other than for performance? > Taking advantage of all your cores. Uh, is there any reason you'd want to "take advantage of all your cores" other than performance? The question/proposal/dialog we started from, don't forget, was: 1. Redis is already as performant as I want, it's not even close to being a bottleneck, so I have no need for multi-threading, and this is a _very very common_ case, as redis is performant enough for a lot. 2. There are other advantages of multi-threading than performance. (Ie, other reasons you'd want it despite 1). You seem to just be going around in circles. If 1, why would you care about "taking advantage of all your cores"?
BubRoss 7y ago
If redis isn't multi-threaded the more cores and threads you have, the less CPU usage the machine would show, even if Redis is being stressed.
- bnegreve 7y ago
  Except if you monitor per CPU usage, as you should.
  - jiggawatts 7y ago
    Except that you can't, because most operating systems will not use the same CPU core consistently for a single thread. This is done for a variety of reasons, including spreading thermal loads to ensure that turbo boost can consistently kick in.
    - bnegreve 7y ago
      You're right but still, I don't think it is difficult to realize that a particular process is only using one core. If you think it is, you should probably consider another application to monitor system usage.
  - haimez 7y ago
    If you use cloudwatch, the “should” part is tricky. “Should” you be able to and “should” you do it if metrics were reasonably priced based on their basic required storage / computational footprint? Yes. However, “should” you in a world where you only have cloudwatch because you don’t want to roll your own monitoring or bring in a third party vendor? Probably not. It’s not a good thing, but it’s a reality. 1/n approximation is your go to here.
vimota 7y ago
The ability to scale vertically instead of sharding is a very nice sysadmin feature that although isn't solving "must faster than redis" is related to it due to multi-threading.
segmondy 7y ago
Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.
- ignoramous 7y ago
  > Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982 https://news.ycombinator.com/item?id=21181982
  - jacquesm 7y ago
    You should post that.
  - IanCal 7y ago
    Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
    - jgalt212 7y ago
      It's mostly telemetry data. /s or not /s not sure.
      - javiermaestro 7y ago
        Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well). Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe. (disclaimer: I work in Scribe) Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-... Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin...
xfalcox 7y ago
For some apps, where you can easily scale app servers horizontally and be stuck in a single redis (because redis cluster mode is not well supported by drivers, or because it can't access some atomic LUA operations) you can get redis to be your number 1 bottleneck. Specially because you can throw more cores at the PostgreSQL so Redis quickly becomes the only thing that doesn't scale.
takeda 7y ago
The idle CPU might very well be due to redis being single threaded. It might be hitting is limits and still show low CPU utilization. Also, you didn't mention that, but I've seen this happening often, you should never use redis as a primary data store if your data is important.
- cogman10 7y ago
  You'd expect a 1/n where n is number of cores available utilization in the case of redis being CPU bound. I guess I don't know how redis deals with the network, but I'd assume that it handles concurrent requests. If not, then I guess that's the case where you'd see less than 1/n cpu utilization and it could still be CPU bound.
  - takeda 7y ago
    Redis handle all request serially, because of that it is much simpler to be correct and doesn't have to worry as much about distributed gotchas. Because redis handles all requests serially you need to be very careful if the redis cluster is shared between different apps. You want to share the cluster among apps with similar usage patterns.
sethhochberg 7y ago
I've personally written a (somewhat hacky but simple) setup for multiplexing high-throughput job queues for a Rails app across multiple Redis instances because our chosen job framework at the time relied a lot on Lua operations on large sets in Redis, and these could easily result in 100% CPU utilization for the Redis process. We picked the job queue framework long before we started moving tens of thousands of jobs a second through it with a later feature, and probably exceeded its design constraints - in that instance we effectively were using Redis + jobs as a pauseable and throttleable write buffer for MySQL. With major tooling changes we likely could have come up with something a lot more elegant, I'm not longer with that company, but our plan was always to get rid of the need for that system to hit MySQL at all, which would eliminate a lot of our need to control throughput with our Redis buffer. Of course, doing that would have taken a lot more time and effort, and in our case doing "the simplest possible thing that would work reliably and serve our customers" meant that we probably could have used a faster Redis instead of multiplexing requests and workers across multiple Redis instances on the same hardware. Sometimes an "improved" version of a tool you're already using can be really useful, if you find yourself in a bind and the alternative is major architectural changes.
wakatime 7y ago
We're using a Redis alternative called SSDB [1] in production because the entire data set doesn't need to fit in RAM. This saves us thousands of dollars per month in server costs. Like you say, latency gains is nominal. The main reason I see alternative databases used is one missing key feature that's custom to a specific use case. [1]: https://github.com/ideawu/ssdb https://github.com/ideawu/ssdb
- sriku 7y ago
  Did you test out ARDB [1] when you chose SSDB? I've been considering these for a while since I know I'm going to hit that choice point though that's not happened yet. [1] https://github.com/yinqiwen/ardb https://github.com/yinqiwen/ardb
  - wakatime 7y ago
    Looked at it, but setup for SSDB was easier and already was using SSDB in production. Hesitant to change something that's working so well. Do have periodic connection issues with SSDB but so infrequent it's easy to work around client-side. Using the python redis [1] client with Flask. [1] https://pypi.org/project/redis/ https://pypi.org/project/redis/
katzgrau 7y ago
Not only that, assuming we're not trying to hyperoptimize before code ever hits production, typically as a dev I look for performance gains when a bottleneck/slowdown seems to occur. Redis is a possibility I guess, but it's also not in the top 20 places I'm going to look first. It's plenty fast for nearly every use case I've come across and while I'm sure someone out there needs something faster, I agree - I can't imagine there's broad market appeal for it.
mperham 7y ago
As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better. I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when licensing a project.
- foota 7y ago
  Ok, what is a job/second?
  - nitrogen 7y ago
    Look up what Sidekiq is. It is awesome.
    - foota 7y ago
      Aw, thanks! I was mis-parsing this as '5000 Redis jobs per second' instead of the intended 'needed to use sharding when sidekiq is scheduling more than 5000 jobs/second' Thanks!
  - todd3834 7y ago
    Jobs per second
- ekianjo 7y ago
  May have been better to use GPL2?
  - lilyball 7y ago
    Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.
    - pjmlp 7y ago
cobookman 7y ago
Happens more often than you think with larger tech companies. Especially if one has hot keys. However yeah, 99% of all deployments could get by with SQLite/MySQL/Postgres performance amounts.
deleted 7y ago
[deleted]

jdsully 7y ago

If the perf isn’t needed we also have Active Replication, subkey expiries, and S3 integration. Multithreading is how we got our start and the reason most people use us though.

manigandham 7y ago

True, it's rarely just raw performance. KeyDB has other advantages like multi-threading and disk-based persistence (instead of being limited by RAM) that makes it better at utilizing your server resources and handling larger scales.

azhng 7y ago
But isn't the point of Redis is to be used as a caching layer instead of persistent layer
- t0mbstone 7y ago
  But what if you could have the speed of RAM-based storage, and still have persistence? Sounds pretty appealing to me, personally.
  - penagwin 7y ago
    You already get this with either Redis or Keydb. The problem is that to avoid the disk being the bottleneck you must write to RAM, then later to the disk (you can't wait for confirmation that it was written) which means that even though a key may successfully be written to the DB, it hasn't been confirmed to persist. Basically redis can't necessarily guarantee data safety - which is perfectly fine for it's normal use case as a caching layer and data you don't necessarily care if you lose (for example rate limiting relies on tracking the number of requests/s within say an hour - if you lose the data you don't really care). User sessions too, worst case people have to log in again, meh whatever.
    - manigandham 7y ago
      The major bottleneck for most applications using Redis is the RAM capacity, not the write performance. Disk persistence can be more than fast enough as proven by dozens of key/value stores out there.
  - lllr_finger 7y ago
    Sounds like an embedded key value store, ie. RocksDB. I've used it with great success in constrained environments and also when pushing cache to the edge for stable, low-latency, high volume APIs.
- nine_k 7y ago
  You can use redis as a write-back cache that serves spikes well and streams the state to disk (or an RDBMS) in a rate-limited manner in the background.
- manigandham 7y ago
  Redis is a data structure server rather than a simple key/value store. It's very useful in lots of scenarios, even if the persistence means it doesn't have ACID semantics. It would be great to remove limitations like RAM-only capacity in exchange for a slight performance hit, and while also gaining better core utilization. We used ScyllaDB (a very fast cassandra clone) in the past for the cpu/disk scalability but always felt Redis offered better APIs. Now it's a real option.
jrochkind1 7y ago
> ...it's rarely just raw performance. KeyDB has other advantages like multi-threading... What are the advantages of "multi-threading" other than for performance?
- manigandham 7y ago
  Taking advantage of all your cores. Redis is awkward at high capacity since most servers scale CPU with RAM and you'll end up with most cores doing nothing. You can manually run multiple instances on the same server but now they're separate databases with operations and sharding overhead. Multithreading IO also reduces the latency hit from disk-persistence and provides more concurrency and throughput, which is a great tradeoff when you don't really need sub-millisecond performance but want the same API across a larger dataset limited by disk instead of RAM space.
  - fwip 7y ago
    That's just performance spelled differently.
    - manigandham 7y ago
      Ok fair. Disk being primary storage is still a ops and scalability benefit.
  - rumanator 7y ago
    > Taking advantage of all your cores. That means nothing at all, if performance is not your concern.
  - jrochkind1 7y ago
    >> What are the advantages of "multi-threading" other than for performance? > Taking advantage of all your cores. Uh, is there any reason you'd want to "take advantage of all your cores" other than performance? The question/proposal/dialog we started from, don't forget, was: 1. Redis is already as performant as I want, it's not even close to being a bottleneck, so I have no need for multi-threading, and this is a _very very common_ case, as redis is performant enough for a lot. 2. There are other advantages of multi-threading than performance. (Ie, other reasons you'd want it despite 1). You seem to just be going around in circles. If 1, why would you care about "taking advantage of all your cores"?

BubRoss 7y ago

If redis isn't multi-threaded the more cores and threads you have, the less CPU usage the machine would show, even if Redis is being stressed.

bnegreve 7y ago
Except if you monitor per CPU usage, as you should.
- jiggawatts 7y ago
  Except that you can't, because most operating systems will not use the same CPU core consistently for a single thread. This is done for a variety of reasons, including spreading thermal loads to ensure that turbo boost can consistently kick in.
  - bnegreve 7y ago
    You're right but still, I don't think it is difficult to realize that a particular process is only using one core. If you think it is, you should probably consider another application to monitor system usage.
- haimez 7y ago
  If you use cloudwatch, the “should” part is tricky. “Should” you be able to and “should” you do it if metrics were reasonably priced based on their basic required storage / computational footprint? Yes. However, “should” you in a world where you only have cloudwatch because you don’t want to roll your own monitoring or bring in a third party vendor? Probably not. It’s not a good thing, but it’s a reality. 1/n approximation is your go to here.

vimota 7y ago

The ability to scale vertically instead of sharding is a very nice sysadmin feature that although isn't solving "must faster than redis" is related to it due to multi-threading.

segmondy 7y ago

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.

ignoramous 7y ago
> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982 https://news.ycombinator.com/item?id=21181982
- jacquesm 7y ago
  You should post that.
- IanCal 7y ago
  Absolutely astounding to me, petabytes an hour? That's in the region of a meg to several megs per user per hour looking at their monthly active user figures.
  - jgalt212 7y ago
    It's mostly telemetry data. /s or not /s not sure.
    - javiermaestro 7y ago
      Mmm it's not only "telemetry data". It's that (e.g. Scuba) and other types of logs, and not only Facebook (e.g. Instagram as well). Basically, everything that needs logging and post-processing by both real-time systems (e.g. Puma) and batch processing (e.g. all of the data that's ingested and sent to the data warehouse) goes through Scribe. (disclaimer: I work in Scribe) Scuba: https://research.fb.com/publications/scuba-diving-into-data-at-facebook/ https://research.fb.com/publications/scuba-diving-into-data-... Puma: https://research.fb.com/publications/realtime-data-processing-at-facebook/ https://research.fb.com/publications/realtime-data-processin...

xfalcox 7y ago

For some apps, where you can easily scale app servers horizontally and be stuck in a single redis (because redis cluster mode is not well supported by drivers, or because it can't access some atomic LUA operations) you can get redis to be your number 1 bottleneck. Specially because you can throw more cores at the PostgreSQL so Redis quickly becomes the only thing that doesn't scale.

takeda 7y ago

The idle CPU might very well be due to redis being single threaded. It might be hitting is limits and still show low CPU utilization. Also, you didn't mention that, but I've seen this happening often, you should never use redis as a primary data store if your data is important.

cogman10 7y ago
You'd expect a 1/n where n is number of cores available utilization in the case of redis being CPU bound. I guess I don't know how redis deals with the network, but I'd assume that it handles concurrent requests. If not, then I guess that's the case where you'd see less than 1/n cpu utilization and it could still be CPU bound.
- takeda 7y ago
  Redis handle all request serially, because of that it is much simpler to be correct and doesn't have to worry as much about distributed gotchas. Because redis handles all requests serially you need to be very careful if the redis cluster is shared between different apps. You want to share the cluster among apps with similar usage patterns.

sethhochberg 7y ago

I've personally written a (somewhat hacky but simple) setup for multiplexing high-throughput job queues for a Rails app across multiple Redis instances because our chosen job framework at the time relied a lot on Lua operations on large sets in Redis, and these could easily result in 100% CPU utilization for the Redis process. We picked the job queue framework long before we started moving tens of thousands of jobs a second through it with a later feature, and probably exceeded its design constraints - in that instance we effectively were using Redis + jobs as a pauseable and throttleable write buffer for MySQL. With major tooling changes we likely could have come up with something a lot more elegant, I'm not longer with that company, but our plan was always to get rid of the need for that system to hit MySQL at all, which would eliminate a lot of our need to control throughput with our Redis buffer. Of course, doing that would have taken a lot more time and effort, and in our case doing "the simplest possible thing that would work reliably and serve our customers" meant that we probably could have used a faster Redis instead of multiplexing requests and workers across multiple Redis instances on the same hardware. Sometimes an "improved" version of a tool you're already using can be really useful, if you find yourself in a bind and the alternative is major architectural changes.

wakatime 7y ago

We're using a Redis alternative called SSDB [1] in production because the entire data set doesn't need to fit in RAM. This saves us thousands of dollars per month in server costs. Like you say, latency gains is nominal. The main reason I see alternative databases used is one missing key feature that's custom to a specific use case. [1]: https://github.com/ideawu/ssdb https://github.com/ideawu/ssdb

sriku 7y ago
Did you test out ARDB [1] when you chose SSDB? I've been considering these for a while since I know I'm going to hit that choice point though that's not happened yet. [1] https://github.com/yinqiwen/ardb https://github.com/yinqiwen/ardb
- wakatime 7y ago
  Looked at it, but setup for SSDB was easier and already was using SSDB in production. Hesitant to change something that's working so well. Do have periodic connection issues with SSDB but so infrequent it's easy to work around client-side. Using the python redis [1] client with Flask. [1] https://pypi.org/project/redis/ https://pypi.org/project/redis/

katzgrau 7y ago

Not only that, assuming we're not trying to hyperoptimize before code ever hits production, typically as a dev I look for performance gains when a bottleneck/slowdown seems to occur. Redis is a possibility I guess, but it's also not in the top 20 places I'm going to look first. It's plenty fast for nearly every use case I've come across and while I'm sure someone out there needs something faster, I agree - I can't imagine there's broad market appeal for it.

mperham 7y ago

As the Sidekiq maintainer, I’ve seen many customers need to shard Redis around 5000-10000 jobs/sec. Sharding is a major operational headache so this could be very useful to heavy job processors if it does prove to scale better. I also find it interesting that the BSD license enables this 3rd party company to fork Redis and build closed source commercial software on top of it. One of the trade offs to consider when licensing a project.

foota 7y ago
Ok, what is a job/second?
- nitrogen 7y ago
  Look up what Sidekiq is. It is awesome.
  - foota 7y ago
    Aw, thanks! I was mis-parsing this as '5000 Redis jobs per second' instead of the intended 'needed to use sharding when sidekiq is scheduling more than 5000 jobs/second' Thanks!
- todd3834 7y ago
  Jobs per second
ekianjo 7y ago
May have been better to use GPL2?
- lilyball 7y ago
  Better how? If using GPLv2 means companies stay away from the project instead of building on top of it, that's not a win for anyone.
  - pjmlp 7y ago

cobookman 7y ago

Happens more often than you think with larger tech companies. Especially if one has hot keys. However yeah, 99% of all deployments could get by with SQLite/MySQL/Postgres performance amounts.

deleted 7y ago

[deleted]