My general problem is that it's really hard to figure out which architecture is right for which system. There's a different architecture for: * one queue with

My general problem is that it's really hard to figure out which architecture is right for which system. There's a different architecture for: * one queue with billions of messages * a millions of queues with small numbers of messages per queue * many queues with many messages per queue There are also different topologies: * Anyone can send a message to anyone (O(n^2) queues) * One publisher with millions of subscribers * One subscribed with millions of publishers * Complex processing networks, where messages get routed in complex ways between processing nodes. There are differences in timing: * More-or-less instant push notifications * Jobs which run within e.g. 5 minutes with polling * Jobs which run in hours/days, with a cron-style architecture And in reliability: * Messages get delivered 100% of the time, and archived once delivered * Messages get delivered 99.999% of the time, but might be dropped on a system outage * ... all the way down to ephemeral pub-subs ... and so on. I'd give my VP's right eye to get a nice chart of what supports what. For the most part, I've found build to be cheaper than buy due to lack of benchmarks and documentation for my use cases. Otherwise, you build. You benchmark. You optimize. And things melt down. My use case right now requires a large number of queues (eventually millions). I'd like to have an archival record of messages. Peak volume is moderate (several messages per second per queue), but usage patterns are sporadic (most queues are idle most of the time). Routing is slightly complex but not supper-complex (typically, about 30 sources per sink, at most 200; most sources only go to one sink, but might go to 2-3). Messages are relatively small (typically, around 1k), but isolated messages might be much bigger (still <1MB, but not small). My experience has been that when I throw something like that into pick-your-queue/pub-sub, things melt down at some point, and building representative benchmarks is a ton of work.