6 ms·

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the s

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex than customers deem acceptable. I know some people use Akka with Persistence module, but I would welcome other alternatives.

notduncansmith 7y ago
Redis fits this niche for me most of the time, but I always start in-process for simplicity. If durability is all you need, SQLite is a great reliable option.
wmfiv 7y ago
Depending on the project a database (Postgres) can make a nice database, queue and cache. You get transactions among all three which which makes life much simpler in the beginning and it will happily march along processing thousands or tens of thousands of TPS. You can scale by adding read replicas and eventually moving to separate databases for each purpose. If you're on the JVM then ActiveMQ (and other Java queues) will usually run embedded and have options to use a database for persistence.
gizzlon 7y ago
Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/
- eropple 7y ago
  I am curious what definition of "queue" fits NSQ. It's a distributed messaging platform.
  - gizzlon 7y ago
    The normal one. It puts stuff in a queue. "nsqd is the daemon that receives, queues, and delivers messages to clients."
- polskibus 7y ago
  Do they offer at-least-once or better delivery guarantees?
- polskibus 7y ago
  Seems like NATS streaming would fit my case - have you heard of any real world deployments that use it ? Are there any larger issues that don't make it a good choice ?
  - manigandham 7y ago
    NATS Streaming is not as well tested and has some design issues that make scaling hard. NATS itself has a new version 2 that has a protocol update and NATS Streaming should follow with a new design as well, but I would recommend other options if you want persistence.
    - tuxychandru 7y ago
      Can you share some details or point to articles describing the design issues with NATS streaming?
      - manigandham 7y ago
        See the other comment but this is a good post/series: https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/ https://bravenewgeek.com/building-a-distributed-log-from-scr...
    - polskibus 7y ago
      What are the design flaws that you have in mind? Is it ok for a couple of nodes or even then it would have trouble to keep up with a medium load? Or maybe the design flaws are to do with providing durability and other guarantees? What other options would you recommend, that can provide at least once delivery and are lighweight enough not to require zookeeper etc?
      - manigandham 7y ago
        Nats streaming isn't just a persistence layer to NATS. It's an entirely different system that basically acts as a client to NATS and then records messages it sees. Basically think of how you would design a persistent queue on top of the ephemeral NATS pub/sub and that's what NATS streaming is. Here's a good post (and series) about distributed logs and NATS design issues: https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/ https://bravenewgeek.com/building-a-distributed-log-from-scr...
eropple 7y ago
"Low overhead"? Redis. Reliable and durable? Emphatically not Redis, and at that point you probably want to just start looking at SQS. (I've had good luck shipping products as docker-compose files, these days. Even to fairly backwards clients.)
- polskibus 7y ago
  I guess I didn't make it clear. Solution has to work in-house / in private cloud.
  - deleted 7y ago
    [deleted]
- tjanks 7y ago
  Can you explain why Redis is not durable? Looking into it for a project but this comment worries me.
  - chucksmash 7y ago
    Intentionally so. It's not a deficiency or a footgun, it's a design decision to be aware of. Redis is an in-memory database first. You can configure Redis for durability. The docs[1] page for persistence has a good examination of the pros and cons. [1]: https://redis.io/topics/persistence https://redis.io/topics/persistence
    - cookiecaper 7y ago
      antirez put a lot of work in around Redis 3.0 (iirc) to make the persistence reliable and strong. As long as your server is configured and used correctly (obviously a large caveat, but you can only hold someone's hand so much), I don't think there is any reason to doubt Redis's persistence anymore. It's important to make this distinction because there are commonly-used systems that offer a best-effort style persistence that usually works fine, but explicitly warn developers not to trust it, and developers rarely understand that. We badly need to get better at distinguishing between true, production-level data integrity and "probably fine".

notduncansmith 7y ago

Redis fits this niche for me most of the time, but I always start in-process for simplicity. If durability is all you need, SQLite is a great reliable option.

wmfiv 7y ago

Depending on the project a database (Postgres) can make a nice database, queue and cache. You get transactions among all three which which makes life much simpler in the beginning and it will happily march along processing thousands or tens of thousands of TPS. You can scale by adding read replicas and eventually moving to separate databases for each purpose. If you're on the JVM then ActiveMQ (and other Java queues) will usually run embedded and have options to use a database for persistence.

gizzlon 7y ago

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

eropple 7y ago
I am curious what definition of "queue" fits NSQ. It's a distributed messaging platform.
- gizzlon 7y ago
  The normal one. It puts stuff in a queue. "nsqd is the daemon that receives, queues, and delivers messages to clients."
polskibus 7y ago
Do they offer at-least-once or better delivery guarantees?
polskibus 7y ago
Seems like NATS streaming would fit my case - have you heard of any real world deployments that use it ? Are there any larger issues that don't make it a good choice ?
- manigandham 7y ago
  NATS Streaming is not as well tested and has some design issues that make scaling hard. NATS itself has a new version 2 that has a protocol update and NATS Streaming should follow with a new design as well, but I would recommend other options if you want persistence.
  - tuxychandru 7y ago
    Can you share some details or point to articles describing the design issues with NATS streaming?
    - manigandham 7y ago
      See the other comment but this is a good post/series: https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/ https://bravenewgeek.com/building-a-distributed-log-from-scr...
  - polskibus 7y ago
    What are the design flaws that you have in mind? Is it ok for a couple of nodes or even then it would have trouble to keep up with a medium load? Or maybe the design flaws are to do with providing durability and other guarantees? What other options would you recommend, that can provide at least once delivery and are lighweight enough not to require zookeeper etc?
    - manigandham 7y ago
      Nats streaming isn't just a persistence layer to NATS. It's an entirely different system that basically acts as a client to NATS and then records messages it sees. Basically think of how you would design a persistent queue on top of the ephemeral NATS pub/sub and that's what NATS streaming is. Here's a good post (and series) about distributed logs and NATS design issues: https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system/ https://bravenewgeek.com/building-a-distributed-log-from-scr...

eropple 7y ago

"Low overhead"? Redis. Reliable and durable? Emphatically not Redis, and at that point you probably want to just start looking at SQS. (I've had good luck shipping products as docker-compose files, these days. Even to fairly backwards clients.)

polskibus 7y ago
I guess I didn't make it clear. Solution has to work in-house / in private cloud.
- deleted 7y ago
  [deleted]
tjanks 7y ago
Can you explain why Redis is not durable? Looking into it for a project but this comment worries me.
- chucksmash 7y ago
  Intentionally so. It's not a deficiency or a footgun, it's a design decision to be aware of. Redis is an in-memory database first. You can configure Redis for durability. The docs[1] page for persistence has a good examination of the pros and cons. [1]: https://redis.io/topics/persistence https://redis.io/topics/persistence
  - cookiecaper 7y ago
    antirez put a lot of work in around Redis 3.0 (iirc) to make the persistence reliable and strong. As long as your server is configured and used correctly (obviously a large caveat, but you can only hold someone's hand so much), I don't think there is any reason to doubt Redis's persistence anymore. It's important to make this distinction because there are commonly-used systems that offer a best-effort style persistence that usually works fine, but explicitly warn developers not to trust it, and developers rarely understand that. We badly need to get better at distinguishing between true, production-level data integrity and "probably fine".