5 ms·

Resque Bus

56 points by bleonard 13y ago

deleted 13y ago
[deleted]
est 13y ago
One thing I really hope with Redis's PUB/SUB is that it could support a time parameter, so we could develop a cronjob replacement, could be add/delete dynamically, and broadcast, and managable with API or Web UI. Currently all redis-based background job implementations does not support scheduled jobs. Redis is not only a memcache replacement, but also a good data structure server, as central hub I hope it add more time-sturcture functions.
- bleonard 13y ago
  I preferred the approach described here over pure Redis pub/sub because of the ability to disconnect or lose connections and still end up with the data eventually. Specifically about "cron" capability, resque-scheduler does a pretty good job for Resque in general. Resque Bus uses it for exponential-backoff retry. It uses the scheduling capability for the "Heartbeat" described in the README. I'll write more about that later if people are interested. Quickly, it's an automated way to add events every minute. We use this like a global (shared across applications and servers) cron clock.
- jordanthoms 13y ago
  Sidekiq supports scheduled jobs: https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
- aaronblohowiak 13y ago
  >Currently all redis-based background job implementations does not support scheduled jobs. That has been false for 4 years. https://github.com/resque/resque-scheduler https://github.com/resque/resque-scheduler
  - est 13y ago
    It's polling based approach. https://github.com/resque/resque-scheduler/blob/master/lib/resque/scheduler.rb#L36 https://github.com/resque/resque-scheduler/blob/master/lib/r... https://github.com/resque/resque-scheduler/blob/master/lib/resque_scheduler.rb#L225 https://github.com/resque/resque-scheduler/blob/master/lib/r... Resque will scan for set value with key prefix like "timestamps:" or "delayed:", for every five seconds, if there's any task queued it will be fired. There's many pros/cons with this approach, e.g. in Redis SADD, SMEMBERS are O(N) operations while in theory dedicated cron algorithm could make O(log(n)). https://en.wikipedia.org/wiki/Cron#Multi-user_capability https://en.wikipedia.org/wiki/Cron#Multi-user_capability This is is important if you have many short lived, timestamp based events need to handle. My use case is design item cool down for a mobile/web game using server side push, with millions of users. While resque could in theory solve this problem, but it's no different from a per 5 minute crontab command. So anyway, what I hope is a native clock in Redis handling all this.
- 616c 13y ago
  I know it is not quite what you are looking for, but if you looking at that type of cronjob replacement, did you not see a HN link to AirBNB guys talking about their tool Chronos? You might like that. http://nerds.airbnb.com/introducing-chronos/ http://nerds.airbnb.com/introducing-chronos/
- Argorak 13y ago
  If you don't mind running development versions: Notifications can be a part of building that. Problem: you (currently) have to be connected when it happens. http://redis.io/topics/notifications http://redis.io/topics/notifications
sjtgraham 13y ago
Why not use AMQP?
- bleonard 13y ago
  Edited a comment from the blog comments a bit... I tried my best to explain some of the reasons we went this way instead of using things out there. I'm not claiming to have invented anything particularly novel here because it's a simple problem to move things from one list to N lists based on keys. We found value in doing it with technology that was already part of our stack and very well known amongst the team and community. I did an implementation in RabbitMQ at the outset. I found that the simplicity of just moving the data around, clarity of persistence, and control over subscriptions was worth the relatively short time it took to implement this gem. For example, we have found value in matching on any of the or multiple published keys (specific values, regex, blank, present, not present, etc) as part of the subscription model. I could not figure this out using the routing_key and/or topic kind of stuff in RabbitMQ.
bradhe 13y ago
This is truly awesome. I have been wanting this semanticto supplement my queues for a long time and was dressing adopting an operationally more complicated tool like Kafka, not to mention all the effort that would be needed to port code.
agopaul 13y ago
How do you handle exceptions and errors who would require a rework of the job? That is interesting, I was looking for something similar to an EBS for PHP time ago, but eventually I wrote myself an ad hoc solution which uses MySQL (and row locks) plus a gearman worker and cron.
- bleonard 13y ago
  Resque Bus uses resque-retry and resque-scheduler to do exponential backoff. That is, it will retry immediately, 2 seconds later, 4 seconds later, and so on up to several hours. It was a little tricky to make sure it ended back in the right application queue, but it worked. If it was put in another one, it would crash because the code to process it would not line up.
agerlic 13y ago
Good example of resque/redis ecosystem usage. It could be great to have more info about your failover and monitoring strategies.
hcm 13y ago
Check out Hutch (https://github.com/gocardless/hutch https://github.com/gocardless/hutch) - a similar system built on top of RabbitMQ
- bleonard 13y ago
  That is close. Thanks for the link. The Consumer mixin is almost the same as the Subscriber one that I came up with. Is there a way to subscribe on more than just the topic type? For example, to anything published that has { action: 'received' } That's something that really enables some great use cases for us.

56 points by bleonard 13y ago

deleted 13y ago

[deleted]

est 13y ago

One thing I really hope with Redis's PUB/SUB is that it could support a time parameter, so we could develop a cronjob replacement, could be add/delete dynamically, and broadcast, and managable with API or Web UI. Currently all redis-based background job implementations does not support scheduled jobs. Redis is not only a memcache replacement, but also a good data structure server, as central hub I hope it add more time-sturcture functions.

bleonard 13y ago
I preferred the approach described here over pure Redis pub/sub because of the ability to disconnect or lose connections and still end up with the data eventually. Specifically about "cron" capability, resque-scheduler does a pretty good job for Resque in general. Resque Bus uses it for exponential-backoff retry. It uses the scheduling capability for the "Heartbeat" described in the README. I'll write more about that later if people are interested. Quickly, it's an automated way to add events every minute. We use this like a global (shared across applications and servers) cron clock.
jordanthoms 13y ago
Sidekiq supports scheduled jobs: https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
aaronblohowiak 13y ago
>Currently all redis-based background job implementations does not support scheduled jobs. That has been false for 4 years. https://github.com/resque/resque-scheduler https://github.com/resque/resque-scheduler
- est 13y ago
  It's polling based approach. https://github.com/resque/resque-scheduler/blob/master/lib/resque/scheduler.rb#L36 https://github.com/resque/resque-scheduler/blob/master/lib/r... https://github.com/resque/resque-scheduler/blob/master/lib/resque_scheduler.rb#L225 https://github.com/resque/resque-scheduler/blob/master/lib/r... Resque will scan for set value with key prefix like "timestamps:" or "delayed:", for every five seconds, if there's any task queued it will be fired. There's many pros/cons with this approach, e.g. in Redis SADD, SMEMBERS are O(N) operations while in theory dedicated cron algorithm could make O(log(n)). https://en.wikipedia.org/wiki/Cron#Multi-user_capability https://en.wikipedia.org/wiki/Cron#Multi-user_capability This is is important if you have many short lived, timestamp based events need to handle. My use case is design item cool down for a mobile/web game using server side push, with millions of users. While resque could in theory solve this problem, but it's no different from a per 5 minute crontab command. So anyway, what I hope is a native clock in Redis handling all this.
616c 13y ago
I know it is not quite what you are looking for, but if you looking at that type of cronjob replacement, did you not see a HN link to AirBNB guys talking about their tool Chronos? You might like that. http://nerds.airbnb.com/introducing-chronos/ http://nerds.airbnb.com/introducing-chronos/
Argorak 13y ago
If you don't mind running development versions: Notifications can be a part of building that. Problem: you (currently) have to be connected when it happens. http://redis.io/topics/notifications http://redis.io/topics/notifications

sjtgraham 13y ago

Why not use AMQP?

bleonard 13y ago
Edited a comment from the blog comments a bit... I tried my best to explain some of the reasons we went this way instead of using things out there. I'm not claiming to have invented anything particularly novel here because it's a simple problem to move things from one list to N lists based on keys. We found value in doing it with technology that was already part of our stack and very well known amongst the team and community. I did an implementation in RabbitMQ at the outset. I found that the simplicity of just moving the data around, clarity of persistence, and control over subscriptions was worth the relatively short time it took to implement this gem. For example, we have found value in matching on any of the or multiple published keys (specific values, regex, blank, present, not present, etc) as part of the subscription model. I could not figure this out using the routing_key and/or topic kind of stuff in RabbitMQ.

bradhe 13y ago

This is truly awesome. I have been wanting this semanticto supplement my queues for a long time and was dressing adopting an operationally more complicated tool like Kafka, not to mention all the effort that would be needed to port code.

agopaul 13y ago

How do you handle exceptions and errors who would require a rework of the job? That is interesting, I was looking for something similar to an EBS for PHP time ago, but eventually I wrote myself an ad hoc solution which uses MySQL (and row locks) plus a gearman worker and cron.

bleonard 13y ago
Resque Bus uses resque-retry and resque-scheduler to do exponential backoff. That is, it will retry immediately, 2 seconds later, 4 seconds later, and so on up to several hours. It was a little tricky to make sure it ended back in the right application queue, but it worked. If it was put in another one, it would crash because the code to process it would not line up.

agerlic 13y ago

Good example of resque/redis ecosystem usage. It could be great to have more info about your failover and monitoring strategies.

hcm 13y ago

Check out Hutch (https://github.com/gocardless/hutch https://github.com/gocardless/hutch) - a similar system built on top of RabbitMQ

bleonard 13y ago
That is close. Thanks for the link. The Consumer mixin is almost the same as the Subscriber one that I came up with. Is there a way to subscribe on more than just the topic type? For example, to anything published that has { action: 'received' } That's something that really enables some great use cases for us.