3 ms·

I’m wondering if there are language agnostic queues where the queue consumers and publishers could be written in different languages?

rco8786 2y ago
That's exactly what this is. You write your own consumer/publisher code however you want, and interact with the queues via SQL queries.
SideburnsOfDoom 2y ago
That's normal, yes. Name a queuing system, and with very few exceptions it will have clients for a variety of languages. It is also normal to exchange messages with contant as json, protobuf or similar format, which again can be processed by any language that aims to be widely used. In fact, are there any queues that aren't language-agnostic? I had the idea that ZeroMQ was a C/C++ only thing, but I checked the docs and it's got the usual plethora of language bindings https://zeromq.org/get-started/ https://zeromq.org/get-started/ So right now I can't name any queue systems that are single language. They're all aimed at interop.
- thangngoc89 2y ago
  Interesting. I've been looking at a much more simpler system than Celery queue to publish jobs from Golang and consume jobs from Python side (AI/ML stuff). This threads gave a lot of names and links for further investigation.
  - anamexis 2y ago
    As mentioned, there are a plethora of queuing systems that have cross platform clients. If you’re interested specifically in a background job system, you may want to check out Faktory. It’s Mike Perham’s language-agnostic follow-up to Sidekiq. https://github.com/contribsys/faktory https://github.com/contribsys/faktory
    - thangngoc89 2y ago
      Thanks for the recommendation. I've checked this out and looks like a good alternative with much simpler configuration and less foot-gun than Celery.
- justinclift 2y ago
  > Name a queuing system, and with very few exceptions it will have clients for a variety of languages. RQ seems to be one of those exceptions, as even after many years it looks to be Python only. :( https://python-rq.org https://python-rq.org
  - thangngoc89 2y ago
    RQ uses Python's Pickle and it doesn't have any serialization protocol so it stays Python only.
    - justinclift 2y ago
      Yeah, it's a right pain. :(
    - SideburnsOfDoom 2y ago
      Most queuing systems that I have used, you can send text or bytes, so the most common denominator is DTOs encoded as json, encoded as UTF-8 text. You could do likewise, (i.e. send text containing a representation of what you want) and this would allow other languages to consume? But that's an edge case (sending text) of an edge case (RQ).
      - thangngoc89 2y ago
        RQ never publishes a real protocol for sending and receiving jobs. Also the `data` field could be anything because it's pickled so you could send a whole Python's object there. See [1] for efforts to make RQ cross language compatible [1] https://github.com/rq/rq/issues/1808 https://github.com/rq/rq/issues/1808
        SideburnsOfDoom 2y ago
        > RQ never publishes a real protocol for sending and receiving jobs. Then, RQ sounds substandard. > Also the `data` field could be anything "the sender could send anything" is true in theory, but in practice it is both useless and false. If you want to make it work, sender and receiver need to agree on a lot. JSON could take any forms too, but the sender and receiver have to typically agree, implicitly or explicitly, on many of the details of the message. Like, what fields are needed, what the allowable values are, etc.

rco8786 2y ago

That's exactly what this is. You write your own consumer/publisher code however you want, and interact with the queues via SQL queries.

SideburnsOfDoom 2y ago

That's normal, yes. Name a queuing system, and with very few exceptions it will have clients for a variety of languages. It is also normal to exchange messages with contant as json, protobuf or similar format, which again can be processed by any language that aims to be widely used. In fact, are there any queues that aren't language-agnostic? I had the idea that ZeroMQ was a C/C++ only thing, but I checked the docs and it's got the usual plethora of language bindings https://zeromq.org/get-started/ https://zeromq.org/get-started/ So right now I can't name any queue systems that are single language. They're all aimed at interop.

thangngoc89 2y ago
Interesting. I've been looking at a much more simpler system than Celery queue to publish jobs from Golang and consume jobs from Python side (AI/ML stuff). This threads gave a lot of names and links for further investigation.
- anamexis 2y ago
  As mentioned, there are a plethora of queuing systems that have cross platform clients. If you’re interested specifically in a background job system, you may want to check out Faktory. It’s Mike Perham’s language-agnostic follow-up to Sidekiq. https://github.com/contribsys/faktory https://github.com/contribsys/faktory
  - thangngoc89 2y ago
    Thanks for the recommendation. I've checked this out and looks like a good alternative with much simpler configuration and less foot-gun than Celery.
justinclift 2y ago
> Name a queuing system, and with very few exceptions it will have clients for a variety of languages. RQ seems to be one of those exceptions, as even after many years it looks to be Python only. :( https://python-rq.org https://python-rq.org
- thangngoc89 2y ago
  RQ uses Python's Pickle and it doesn't have any serialization protocol so it stays Python only.
  - justinclift 2y ago
    Yeah, it's a right pain. :(
  - SideburnsOfDoom 2y ago
    Most queuing systems that I have used, you can send text or bytes, so the most common denominator is DTOs encoded as json, encoded as UTF-8 text. You could do likewise, (i.e. send text containing a representation of what you want) and this would allow other languages to consume? But that's an edge case (sending text) of an edge case (RQ).
    - thangngoc89 2y ago
      RQ never publishes a real protocol for sending and receiving jobs. Also the `data` field could be anything because it's pickled so you could send a whole Python's object there. See [1] for efforts to make RQ cross language compatible [1] https://github.com/rq/rq/issues/1808 https://github.com/rq/rq/issues/1808
      - SideburnsOfDoom 2y ago
        > RQ never publishes a real protocol for sending and receiving jobs. Then, RQ sounds substandard. > Also the `data` field could be anything "the sender could send anything" is true in theory, but in practice it is both useless and false. If you want to make it work, sender and receiver need to agree on a lot. JSON could take any forms too, but the sender and receiver have to typically agree, implicitly or explicitly, on many of the details of the message. Like, what fields are needed, what the allowable values are, etc.