4 ms·

The one thing I genuinely don't understand is why there is no single task queue that works across ruby and python. I get that at some point people just started

The one thing I genuinely don't understand is why there is no single task queue that works across ruby and python. I get that at some point people just started making http based microservices to pass information around, but at the end of the day a simple task queue that has a unified storage format across both is a better way to connect ruby(rails) based with the ml stack. There are probably thousands of custom rabbitmq or redis based private company solutions out there.

byroot 3y ago
Mike Perham (the sidekiq maintainer) also maintains the less well known faktory[0] which is language agnostic and has runners for both Ruby and Python [0] https://github.com/contribsys/faktory https://github.com/contribsys/faktory
- rjzzleep 3y ago
  That's awesome. Any idea why he doesn't just supersede Sidekiq with that? I spent quite some time hacking my own solution. I just looked at the source, I guess Mike has been mostly working on both projects on his own for the last 4 years, so Faktory has a lot of features that require the enterprise license. I wonder if he could change the situation it he markets a bit more to the python and more specifically Django community.
  - f6v 3y ago
    Because people running sidekiq with their Ruby app on production don’t care about cross-language queue. If you pay for Pro or Enterprise you don’t want any major changes that are potentially breaking.
    - cactusplant7374 3y ago
      The only reason we pay is because the pro version doesn't lose jobs if a worker crashes. You would think that would be a core feature.
      - ckolkey 3y ago
        Sounds like Mike found a good feature that would encourage companies to purchase a license. I have a tremendous admiration for the business he's built.
        cactusplant7374 3y ago
        You would think not losing jobs would be a required feature in any job queue software product.
        f6v 3y ago
        > You would think not losing jobs would be a required feature in any job queue software product. A free software doesn’t have to be anything because it’s free.
        cactusplant7374 3y ago
        Free software doesn't even have to work. It can be an empty git repo. Your comment doesn't move forward the conversation.
      - deleted 3y ago
        [deleted]
      - aaronbrethorst 3y ago
        A man’s gotta eat.
  - ignurant 3y ago
    It is pretty cool. I’ve been using it for 3-4 years as a queue between Ruby and Node for scraping tasks. Ruby queues the work, a node worker does the downloading, and a Ruby process parses and loads data. It works incredibly well in my use case and has saved me from having to shift everything to one language or the other. It’s been very reliable.
- keithalewis 3y ago
  This relies on the Go runtime scheduler. Sometimes that is not good enough.
akvadrako 3y ago
Because it's a rare need and easy to write your own based on postgres or redis. External systems come with a cost, especially if it's more than a library.
pmontra 3y ago
My customers are using Celery for Python and Sidekiq for Ruby. Those are parts of Django and Rails web apps. Those customers don't mix languages so they don't need workers able to run code in multiple languages. One of them is also using SQS though so we could receive a JSON in a server written in any language, do some processing and return the result. However the database of that app has been "destroyed by design" by using Django's ORM inheritance (my suggestion: never use it) so nothing can interact effectively with it except Python code using the same models. By the way, celery was born as a protocol spec to support multiple languages but never moved past Python. I can't google a quote for that, I remember I saw it years ago in the documentation somewhere.
- rjzzleep 3y ago
  I can see that it was born that way, but nowadays most job queues say, don't touch the wire protocol, it's not intended to be used directly. I still think Rails has the best ORM design I've ever seen, iterated with practical applications. Django's ORM and migrations are, for lack of a better word, odd. I'm a bit surprised that people here argue that no one using Rails would ever want to interface with other languages. Most big companies do. How can you not interface with python these days. Looks to me like celery might just be the only job queue left like that. Might be worth writing a current ruby en-queuing library for it. Retracting my previous statement about ActiveJob since it would probably be too much effort to execute anything bidirectionally. https://docs.celeryq.dev/en/stable/internals/protocol.html# https://docs.celeryq.dev/en/stable/internals/protocol.html#
- riffraff 3y ago
  What's wrong with Django's orm? Does it have something particularly bad compared to others?
  - cdcarter 3y ago
    I think the poster is specifically referring to using inheritance in Django ORM, where if you had e.g. a model Book and then a model Novel that inherits from it. In python these are modeled as a class inheritance hierarchy, and Django (at least, by default) creates a database table per class in the hierarchy. If you have 3-4 levels of inheritance, that's 3-4 extra joins per query.
emmelaich 3y ago
There is Perl Directory::Queue / Python dirq which also has implementations in Go, Java, and C. A Ruby implementation would probably not be hard. I don't how much people use this in serious or high performance work but it might be an option.
petepete 3y ago
There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner. https://beanstalkd.github.io/ https://beanstalkd.github.io/
jsjohnst 3y ago
> The one thing I genuinely don't understand is why there is no single task queue that works across ruby and python. Not sure I can recommend it, but Gearman does fit the bill. There’s client and worker libraries for a dozen languages, including Python3 and Ruby.

byroot 3y ago

Mike Perham (the sidekiq maintainer) also maintains the less well known faktory[0] which is language agnostic and has runners for both Ruby and Python [0] https://github.com/contribsys/faktory https://github.com/contribsys/faktory

rjzzleep 3y ago
That's awesome. Any idea why he doesn't just supersede Sidekiq with that? I spent quite some time hacking my own solution. I just looked at the source, I guess Mike has been mostly working on both projects on his own for the last 4 years, so Faktory has a lot of features that require the enterprise license. I wonder if he could change the situation it he markets a bit more to the python and more specifically Django community.
- f6v 3y ago
  Because people running sidekiq with their Ruby app on production don’t care about cross-language queue. If you pay for Pro or Enterprise you don’t want any major changes that are potentially breaking.
  - cactusplant7374 3y ago
    The only reason we pay is because the pro version doesn't lose jobs if a worker crashes. You would think that would be a core feature.
    - ckolkey 3y ago
      Sounds like Mike found a good feature that would encourage companies to purchase a license. I have a tremendous admiration for the business he's built.
      - cactusplant7374 3y ago
        You would think not losing jobs would be a required feature in any job queue software product.
        f6v 3y ago
        > You would think not losing jobs would be a required feature in any job queue software product. A free software doesn’t have to be anything because it’s free.
        cactusplant7374 3y ago
        Free software doesn't even have to work. It can be an empty git repo. Your comment doesn't move forward the conversation.
    - deleted 3y ago
      [deleted]
    - aaronbrethorst 3y ago
      A man’s gotta eat.
- ignurant 3y ago
  It is pretty cool. I’ve been using it for 3-4 years as a queue between Ruby and Node for scraping tasks. Ruby queues the work, a node worker does the downloading, and a Ruby process parses and loads data. It works incredibly well in my use case and has saved me from having to shift everything to one language or the other. It’s been very reliable.
keithalewis 3y ago
This relies on the Go runtime scheduler. Sometimes that is not good enough.

akvadrako 3y ago

Because it's a rare need and easy to write your own based on postgres or redis. External systems come with a cost, especially if it's more than a library.

pmontra 3y ago

My customers are using Celery for Python and Sidekiq for Ruby. Those are parts of Django and Rails web apps. Those customers don't mix languages so they don't need workers able to run code in multiple languages. One of them is also using SQS though so we could receive a JSON in a server written in any language, do some processing and return the result. However the database of that app has been "destroyed by design" by using Django's ORM inheritance (my suggestion: never use it) so nothing can interact effectively with it except Python code using the same models. By the way, celery was born as a protocol spec to support multiple languages but never moved past Python. I can't google a quote for that, I remember I saw it years ago in the documentation somewhere.

rjzzleep 3y ago
I can see that it was born that way, but nowadays most job queues say, don't touch the wire protocol, it's not intended to be used directly. I still think Rails has the best ORM design I've ever seen, iterated with practical applications. Django's ORM and migrations are, for lack of a better word, odd. I'm a bit surprised that people here argue that no one using Rails would ever want to interface with other languages. Most big companies do. How can you not interface with python these days. Looks to me like celery might just be the only job queue left like that. Might be worth writing a current ruby en-queuing library for it. Retracting my previous statement about ActiveJob since it would probably be too much effort to execute anything bidirectionally. https://docs.celeryq.dev/en/stable/internals/protocol.html# https://docs.celeryq.dev/en/stable/internals/protocol.html#
riffraff 3y ago
What's wrong with Django's orm? Does it have something particularly bad compared to others?
- cdcarter 3y ago
  I think the poster is specifically referring to using inheritance in Django ORM, where if you had e.g. a model Book and then a model Novel that inherits from it. In python these are modeled as a class inheritance hierarchy, and Django (at least, by default) creates a database table per class in the hierarchy. If you have 3-4 levels of inheritance, that's 3-4 extra joins per query.

emmelaich 3y ago

There is Perl Directory::Queue / Python dirq which also has implementations in Go, Java, and C. A Ruby implementation would probably not be hard. I don't how much people use this in serious or high performance work but it might be an option.

petepete 3y ago

There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner. https://beanstalkd.github.io/ https://beanstalkd.github.io/

jsjohnst 3y ago

> The one thing I genuinely don't understand is why there is no single task queue that works across ruby and python. Not sure I can recommend it, but Gearman does fit the bill. There’s client and worker libraries for a dozen languages, including Python3 and Ruby.