4 ms·
If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messag
by onetimeuse92304 3y ago
If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring.
The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information.
Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches.
So what you can do in practice?
1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first".
2) Serialise the batch (for example, if it makes sense, create a compressed JSON file)
3) Upload the file to some high throughput, scalable, cheap storage (for example S3)
4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object.
This usually can be modified depending on specific project needs.
Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.
- usrusr 3y agoIf your use case can stomach the added failure modes this implies, yeah, that's what you can do.
- onetimeuse92304 3y agoEverything has some cons, some failure modes. All engineering is about knowing, understanding and making tradeoffs. In my practice, I am happy if I can get rid of hard problems (my messaging platform being unable to process X messages per second) and replace them with relatively easier problems (my persistence might sometimes fail and then I can't send a message). I would argue that distributed persistence solutions are usually more reliable than messaging platforms and also what is a very large throughput for a messsaging solution is usually nothing much for monsters that are engineered to take much larger volumes of data. And so, in my experience, reducing load to messaging and increasing load to persistence is net positive for the overall reliability.
- Beached 3y ago1) some designs can't tolerate the producer sending messages at such a delay. 3) s3 is not cheap storage, it is significantly higher cost than most on prem solutions when talking about large scale storage. (pedabytes scale)
- onetimeuse92304 3y ago1) If you can't accept and process the the load of messages with your messaging the discussion of whether 100ms is or is not acceptable delay is very much pointless. Messaging middleware is by design not suited well for architecting real time systems. If you require real time guarantees you would benefit from some other communication channel. 2) 3) S3 is orders of magnitude cheaper compared to messaging platforms like RabbitMQ or Kafka. If you take load off of your RabitMQ or Kafka and put it on S3 you should see a significant reduction in cost. S3 might be more expensive than other persistence solutions, true. Just choose whatever else you have. I used S3 as an example because it is extremely easy to implement and get going. Again, all engineering is about tradeoffs. You compare two solutions, they will always have some cons. You just decide which cons you can live with. If your platform can't process messages at all and you don't know how to scale it up to do so that's pretty large problem in my book.