3 ms·

And do not use Redis as a broker under a high load. It will random crash or refuse to set workers to work, while queue is filling up with tasks.

mooted1 11y ago
At Uber we use redis as a broker for 10k+ requests per second (pushing 50k one some days) with up to several million items backed up. This is a single redis instance with replication turned on and RDB snapshots for durability. Curious how our use patterns are different that causes what you're seeing.
- bliti 11y ago
  Tell me about the hardware specs where your Redis runs.
- andy_ppp 11y ago
  Is that with Celery or are you using your own queue management?
taylorhughes 11y ago
Good point. We've run into some issues with dropping tasks when the redis host is underwater; can imagine that becoming problematic.
- sandGorgon 11y ago
  We recently built out a small background task runner - we were told to use RQ+Redis rather than Celery because (and I'm repeating hearsay here) - 1. Celery's code is very messy and 2. because of the way it keeps jobs in memory, it is very prone to memory leaks. Is there any truth to this ?
  - caio1982 11y ago
    I don't know how much of it is truth but I've definitely had problems with Celery and memory leaks in the past myself. We can add this up to the hearsay pile though.
    - gsharma 11y ago
      What do you use as a broker? We have seen these issues too and use Redis as the broker.
  - phunge 11y ago
    Celery works, that's been my experience. It's buggy, complicated, and we've had unresolved issues, but it more or less does the job. The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
    - sandGorgon 11y ago
      have you looked at RQ - would love to get a comparitive opinion.
  - mpdehaan2 11y ago
    My experience is that Redis slows down considerably with a large amount of data in the queue, and also consumes large amounts of RAM. I'm not sure I'd look to something that explicitly uses Redis as an alternative unless I could be sure the queue would be reasonably small. Which, if the jobs are not going to be overwhelmed with backed up jobs, maybe that's fine. Also, maybe we were doing it wrong.

mooted1 11y ago

At Uber we use redis as a broker for 10k+ requests per second (pushing 50k one some days) with up to several million items backed up. This is a single redis instance with replication turned on and RDB snapshots for durability. Curious how our use patterns are different that causes what you're seeing.

bliti 11y ago
Tell me about the hardware specs where your Redis runs.
andy_ppp 11y ago
Is that with Celery or are you using your own queue management?

taylorhughes 11y ago

Good point. We've run into some issues with dropping tasks when the redis host is underwater; can imagine that becoming problematic.

sandGorgon 11y ago
We recently built out a small background task runner - we were told to use RQ+Redis rather than Celery because (and I'm repeating hearsay here) - 1. Celery's code is very messy and 2. because of the way it keeps jobs in memory, it is very prone to memory leaks. Is there any truth to this ?
- caio1982 11y ago
  I don't know how much of it is truth but I've definitely had problems with Celery and memory leaks in the past myself. We can add this up to the hearsay pile though.
  - gsharma 11y ago
    What do you use as a broker? We have seen these issues too and use Redis as the broker.
- phunge 11y ago
  Celery works, that's been my experience. It's buggy, complicated, and we've had unresolved issues, but it more or less does the job. The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
  - sandGorgon 11y ago
    have you looked at RQ - would love to get a comparitive opinion.
- mpdehaan2 11y ago
  My experience is that Redis slows down considerably with a large amount of data in the queue, and also consumes large amounts of RAM. I'm not sure I'd look to something that explicitly uses Redis as an alternative unless I could be sure the queue would be reasonably small. Which, if the jobs are not going to be overwhelmed with backed up jobs, maybe that's fine. Also, maybe we were doing it wrong.