4 ms·

Is Centrifuge going to be open source?

achille-roussel 8y ago
We do want to open source the code. In order to get to production we took some shortcuts which tightly integrated some components of this system with Segment’s infrastructure and would make it hard for anyone to deploy. We discussed open sourcing the code along with this blog post but we felt like it would have more value once we’ve put a bit more work into it to make it easier to work with, and add a couple of features we have in the pipeline.
no1youknowz 8y ago
Would like to know this as well. I have been on the lookout for a similar system to SideKiq. Unfortunately for Go, there isn't anything that matches it 100%. I have looked into: https://github.com/celrenheit/sandglass https://github.com/celrenheit/sandglass https://github.com/RichardKnop/machinery https://github.com/RichardKnop/machinery https://github.com/contribsys/faktory https://github.com/contribsys/faktory In the end I went with machinery due to supporting: - Batches (or Groups, tasks executed at the same time) - Chains (tasks executed one by one) - Callbacks (a task executed, on completion of a Batch) - Cron Jobs - Rescheduling failed jobs - Long-runnng Jobs - Distributed/Fault Tolerant DB (for Jobs) - Distributed/Fault Tolerant Workers + other things I cant remember now. It would be very interesting to know if this is going to be opened sourced and whether it supports the list above.
- mnutt 8y ago
  I don't know if it exactly fits, but maybe check out some of the stuff Customer.io has been doing, such as https://github.com/customerio/fairway https://github.com/customerio/fairway? It seems like they're solving similar problems and contribute a lot of Go projects.
  - no1youknowz 8y ago
    Unfortunately it's not. From their own ReadMe: > Fairway isn't meant to be a robust system for processing queued messages/jobs. To more reliably process queued messages, we've integrated with Sidekiq.

We do want to open source the code. In order to get to production we took some shortcuts which tightly integrated some components of this system with Segment’s infrastructure and would make it hard for anyone to deploy. We discussed open sourcing the code along with this blog post but we felt like it would have more value once we’ve put a bit more work into it to make it easier to work with, and add a couple of features we have in the pipeline.

no1youknowz 8y ago

Would like to know this as well. I have been on the lookout for a similar system to SideKiq. Unfortunately for Go, there isn't anything that matches it 100%. I have looked into: https://github.com/celrenheit/sandglass https://github.com/celrenheit/sandglass https://github.com/RichardKnop/machinery https://github.com/RichardKnop/machinery https://github.com/contribsys/faktory https://github.com/contribsys/faktory In the end I went with machinery due to supporting: - Batches (or Groups, tasks executed at the same time) - Chains (tasks executed one by one) - Callbacks (a task executed, on completion of a Batch) - Cron Jobs - Rescheduling failed jobs - Long-runnng Jobs - Distributed/Fault Tolerant DB (for Jobs) - Distributed/Fault Tolerant Workers + other things I cant remember now. It would be very interesting to know if this is going to be opened sourced and whether it supports the list above.

mnutt 8y ago
I don't know if it exactly fits, but maybe check out some of the stuff Customer.io has been doing, such as https://github.com/customerio/fairway https://github.com/customerio/fairway? It seems like they're solving similar problems and contribute a lot of Go projects.
- no1youknowz 8y ago
  Unfortunately it's not. From their own ReadMe: > Fairway isn't meant to be a robust system for processing queued messages/jobs. To more reliably process queued messages, we've integrated with Sidekiq.