4 ms·
What sensible scenarios are there for sending 1MB messages over a message bus?
by calpaterson 11y ago
What sensible scenarios are there for sending 1MB messages over a message bus?
- deleted 11y ago[deleted]
- justinsaccount 11y agoWhat is your idea of the arbitrary threshold at which point one should no longer use a message queue?
- brobinson 11y agoI don't know of a good threshold, but you should generally favor small messages -> big computations. If a 1mb message isn't handled properly, it could be retransmitted, etc. In the case of a 1mb message, it sounds like that data should be serialized into a database and the id or whatever is needed to access it is passed along in the message instead. It's like emailing a 10mb powerpoint instead of throwing it on Dropbox and sending a link to it.
- calpaterson 11y agoI just haven't worked on any messaging system that ever sent messages that large. Normally instead of putting data directly on a queue you put in a database and then send some kind of reference over the queue. That's mostly to avoid data races but as a side effect it makes it hard to get large messages like 1MB.
- revelation 11y agoA common use case for these high performance message passing libraries (not sure if that applies to the ones tested here) is in something like autonomous cars, where you have many sensors and many systems interconnected through a bus, and ethernet comes in handy here. It's natural to broadcast grayscale images and larger stuff to the bus. If you have slow ass VMs passing data from Zurich to NYC, yeah, maybe don't send 1MB messages.
- kasey_junk 11y agoKafka is often lumped in with message queue systems as it has similar semantics and you can build a message queue on top of it, but it is generally cleaner to think of it as a distributed data store. In that context it makes sense that you can run into scenarios where you want 1MB records for atomicity (though you probably shouldn't).
- jack9 11y agoLog aggregations. As you near the top of a tiered processing system, the data can get very big. I'm surprised he didn't benchmark 0mq or nano since they are the extreme lean case, not redis.