6 ms·

I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq https://github.com/pou

by memset 2y ago

I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq https://github.com/poundifdef/smoothmq It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)

GordonS 2y ago
Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)
- memset 2y ago
  Not today. It's a work in progress! There are several iterations that I'm working on: 1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently. Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)
justinclift 2y ago
Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
- memset 2y ago
  It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull. The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.
  - agrothberg 2y ago
    SQS does offer long polling, which looks closer to "push" semantics.
    - memset 2y ago
      Fair enough. I do implement long polling!

by memset 2y ago

GordonS 2y ago

Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)

memset 2y ago
Not today. It's a work in progress! There are several iterations that I'm working on: 1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently. Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)

justinclift 2y ago

Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?

memset 2y ago
It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull. The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.
- agrothberg 2y ago
  SQS does offer long polling, which looks closer to "push" semantics.
  - memset 2y ago
    Fair enough. I do implement long polling!