4 ms·

Thanks, that's a very helpful explanation. (A partition key is optional in Kafka - the DefaultPartitioner just computes a random partition regardless of key.)

Thanks, that's a very helpful explanation. (A partition key is optional in Kafka - the DefaultPartitioner just computes a random partition regardless of key.) I am having trouble visualizing this in NSQ: > [A server] might consume and acknowledge multiple messages to produce one new message I can imagine a server consuming multiple messages and performing a single write to e.g. ElasticSearch or Cassandra - in this case it doesn't matter that the batched write is a random subset of messages. But I can't imagine consuming multiple messages and emitting another message - at least not without being able to reason about what partition of data was present on that server. For example, I can't detect an abandoned shopping cart because a single shopper's events will be scattered across all servers. Maybe I'm thinking of this wrong - can you give a live use case for consuming multiple events and emitting a single message in NSQ?

ploxiln 12y ago
You're right, it can't do some calculation across all data for a single user or session or anything like that. I suppose I was thinking that it might reduce "just a little", just adding stuff together as appropriate, but that wouldn't really be useful. In practice, the way multiple events turn into a single event is that some are just thrown out, according to some static filter rule, or after checking a datastore. The places where NSQ are typically used is where new data never stops flowing. So sometimes for this sort of thing, it updates a value in some datastore, and creates a new message for the next stage with the latest value, perhaps only if the latest value reaches some threshold, in a scheme where the value decays.