3 ms·

We collect messages from tens of thousands of devices and use RabbitMQ specifically because it is uncoupled from the Postgres databases. If the shit hits the fa

We collect messages from tens of thousands of devices and use RabbitMQ specifically because it is uncoupled from the Postgres databases. If the shit hits the fan and a database needs to be taken offline the messages can pool up in RabbitMQ until we are in a state where things can be processed again.

hn_throwaway_99 4y ago
Still trivial to get that benefit with just a separate postgres instance for your queue, then you have the (very large IMO) benefit of simplifying your overall tech stack and having fewer separate components you have to have knowledge for, keep track of version updates for, etc.
wvenable 4y ago
You may well be the 1-5% of projects that need it.
- Spivak 4y ago
  Everyone should use this pattern unless there's a good reason not too though. Turning what would otherwise be outages into queue backups is a godsend. It becomes impossible to ever lose in-flight data. The moment you persist to your queue you can ack back to the client.
  - mordae 4y ago
    In my experience, persistent message queue is just a poor secondary database. If anything, I prefer to use ZeroMQ and make sure everything can recover from an outage and settle eventually. To ingest large inputs, I would just use short append only files and maybe send them over to the other node over ZeroMQ to get a little bit more reliability, but rarely are such high volume data that critical. There is nothing like free lunch when talking distributed fault tolerant systems and simplicity usually fares rather well.
  - colonwqbang 4y ago
    > The moment you persist to your queue you can ack back to the client. Relational databases also have this feature.
    - Spivak 4y ago
      And if you store your work inbox in a relational db then you invented a queueing system. The point is that queues can ingest messages much much faster and cheaper than a db, route messages based on priority and globally tune the speed of workers to keep your db from getting overwhelmed or use idle time.
  - jakearmitage 4y ago
    > The moment you persist to your queue you can ack back to the client You mean like... ACID transactions? https://www.postgresql.org/docs/current/tutorial-transactions.html https://www.postgresql.org/docs/current/tutorial-transaction...
    - Spivak 4y ago
      You act like the "I can persist data" is the part that matters. It's the fact that I can from my pool of app servers post a unit of work to be done and be sure it will happen even if the app server gets struck down. It's the architecture of offloading work from your frontend whenever possible to work that can be done at your leisure. Use whatever you like to actually implement the queue, Postgres wouldn't be my first or second choice but it's fine, I've used it for small one-off projects.
  - 0cf8612b2e1e 4y ago
    What if the queue goes down? Without insane engineering, there is always going to be a single point of failure somewhere.
    - Spivak 4y ago
      It doesn't, it's two clusters in replication. It's like the least insane engineering? Are y'all not running your stuff highly available?
      - wvenable 4y ago
        Databases run as clusters in replication...
        Spivak 4y ago
        Yes? I don't understand your point. Queueing systems are a kind of database with different semantics and performance characteristics. The answer to what happens if the queue goes down is that it runs highly available.
        wvenable 4y ago
        Most people don't need another kind of database. A few do but not enough to justify an "everyone should do it this way by default" opinion.
        Spivak 4y ago
        Sure, if you're at the "it's not worth maintaining two dbs or paying for any managed offering" stage then you should still use the pattern but implement the work queue in your primary db. It'll be slower and you'll have to diy a little if there aren't ready made libs to do it but you still get all the major benefits.
  - marcosdumay 4y ago
    You only start planning staying online when your database is down if you have some very rare and impactful requirements. For way more than 99% of the people, all of that should be irrelevant.
    - Spivak 4y ago
      It's not the database I'm worried about. If you push a change to your frontend server that does all the work itself and it breaks you have a service interruption. That same system based on queues it's a queue backup and you have breathing room. If you're doing a longer operation and your app server crashes that work is lost and the client has sit there and wait, in a queue system you can return almost immediately and if there's a crash it'll get picked up by another worker. Having a return queue and websockets let you give feedback to the client js in a way that is impervious to network interruptions and refreshes, once you reconnect it can catch up. These aren't theoretical, this architecture has saved my ass on more occasions than I can count.
      - Rapzid 4y ago
        I believe there are a lot of assumptions baked into your argument that are going undiscussed. Foremost being that queuing all requests is even desirable. For certain workloads that may be the case, like ingesting event data, but for others queuing the requests doesn't make the service any more available than not queuing the requests.

hn_throwaway_99 4y ago

Still trivial to get that benefit with just a separate postgres instance for your queue, then you have the (very large IMO) benefit of simplifying your overall tech stack and having fewer separate components you have to have knowledge for, keep track of version updates for, etc.

wvenable 4y ago

You may well be the 1-5% of projects that need it.

Spivak 4y ago
Everyone should use this pattern unless there's a good reason not too though. Turning what would otherwise be outages into queue backups is a godsend. It becomes impossible to ever lose in-flight data. The moment you persist to your queue you can ack back to the client.
- mordae 4y ago
  In my experience, persistent message queue is just a poor secondary database. If anything, I prefer to use ZeroMQ and make sure everything can recover from an outage and settle eventually. To ingest large inputs, I would just use short append only files and maybe send them over to the other node over ZeroMQ to get a little bit more reliability, but rarely are such high volume data that critical. There is nothing like free lunch when talking distributed fault tolerant systems and simplicity usually fares rather well.
- colonwqbang 4y ago
  > The moment you persist to your queue you can ack back to the client. Relational databases also have this feature.
  - Spivak 4y ago
    And if you store your work inbox in a relational db then you invented a queueing system. The point is that queues can ingest messages much much faster and cheaper than a db, route messages based on priority and globally tune the speed of workers to keep your db from getting overwhelmed or use idle time.
- jakearmitage 4y ago
  > The moment you persist to your queue you can ack back to the client You mean like... ACID transactions? https://www.postgresql.org/docs/current/tutorial-transactions.html https://www.postgresql.org/docs/current/tutorial-transaction...
  - Spivak 4y ago
    You act like the "I can persist data" is the part that matters. It's the fact that I can from my pool of app servers post a unit of work to be done and be sure it will happen even if the app server gets struck down. It's the architecture of offloading work from your frontend whenever possible to work that can be done at your leisure. Use whatever you like to actually implement the queue, Postgres wouldn't be my first or second choice but it's fine, I've used it for small one-off projects.
- 0cf8612b2e1e 4y ago
  What if the queue goes down? Without insane engineering, there is always going to be a single point of failure somewhere.
  - Spivak 4y ago
    It doesn't, it's two clusters in replication. It's like the least insane engineering? Are y'all not running your stuff highly available?
    - wvenable 4y ago
      Databases run as clusters in replication...
      - Spivak 4y ago
        Yes? I don't understand your point. Queueing systems are a kind of database with different semantics and performance characteristics. The answer to what happens if the queue goes down is that it runs highly available.
        wvenable 4y ago
        Most people don't need another kind of database. A few do but not enough to justify an "everyone should do it this way by default" opinion.
        Spivak 4y ago
        Sure, if you're at the "it's not worth maintaining two dbs or paying for any managed offering" stage then you should still use the pattern but implement the work queue in your primary db. It'll be slower and you'll have to diy a little if there aren't ready made libs to do it but you still get all the major benefits.
- marcosdumay 4y ago
  You only start planning staying online when your database is down if you have some very rare and impactful requirements. For way more than 99% of the people, all of that should be irrelevant.
  - Spivak 4y ago
    It's not the database I'm worried about. If you push a change to your frontend server that does all the work itself and it breaks you have a service interruption. That same system based on queues it's a queue backup and you have breathing room. If you're doing a longer operation and your app server crashes that work is lost and the client has sit there and wait, in a queue system you can return almost immediately and if there's a crash it'll get picked up by another worker. Having a return queue and websockets let you give feedback to the client js in a way that is impervious to network interruptions and refreshes, once you reconnect it can catch up. These aren't theoretical, this architecture has saved my ass on more occasions than I can count.
    - Rapzid 4y ago
      I believe there are a lot of assumptions baked into your argument that are going undiscussed. Foremost being that queuing all requests is even desirable. For certain workloads that may be the case, like ingesting event data, but for others queuing the requests doesn't make the service any more available than not queuing the requests.