4 ms·

It looks like there isn't any kind of user-definable shard/partition key available within the topics - a message within a topic could go to any client subscribi

• ploxiln 12y ago
  For the applications I've dealt with, it's an upside that a workers distributed across a number of servers can process messages from producers distributed across a number of servers, with no user-defined partitioning. Kafka, I think, requires you to define partitions to have multiple workers consume a stream. In the NSQ case, if you need more throughput, just spin it up. Somewhat on a tangent, one of the goals of the design was to avoid any servers in the middle of the "workers" (consumers) and producers, and making it such that any consuming or producing server dieing or experiencing a network partition doesn't slow down any producers or consumers who can still reach each other. You're right, this isn't really focused on replacing map-reduce thing like some other message streaming systems are. For one, I can't imagine a "sorting" step using nsq. It's still used for somewhat hefty data-processing tasks, in multiple stages. Each stage has as many servers spun up as needed, consuming messages directly from the previous stage, and publishing to nsqd on localhost. They might consume and acknowledge multiple messages to produce one new message. For reduced inter-server hops before hitting the database, you can run workers on the same host as producers. So you might have a X hosts producing messages (and running their own nsqd locally, of course), and have Y consumers on each host, configured to consume messages from localhost instead of using nsqlookupd to find all sources, and those consumers would make direct requests to the appropriate database host. With this setup you can still use nsqadmin to monitor the queue levels on all producer hosts, and you can still be archiving messages to disk on some other remote hosts.
  • alexatkeplar 12y ago
    Thanks, that's a very helpful explanation. (A partition key is optional in Kafka - the DefaultPartitioner just computes a random partition regardless of key.) I am having trouble visualizing this in NSQ: > [A server] might consume and acknowledge multiple messages to produce one new message I can imagine a server consuming multiple messages and performing a single write to e.g. ElasticSearch or Cassandra - in this case it doesn't matter that the batched write is a random subset of messages. But I can't imagine consuming multiple messages and emitting another message - at least not without being able to reason about what partition of data was present on that server. For example, I can't detect an abandoned shopping cart because a single shopper's events will be scattered across all servers. Maybe I'm thinking of this wrong - can you give a live use case for consuming multiple events and emitting a single message in NSQ?
    • ploxiln 12y ago
      You're right, it can't do some calculation across all data for a single user or session or anything like that. I suppose I was thinking that it might reduce "just a little", just adding stuff together as appropriate, but that wouldn't really be useful. In practice, the way multiple events turn into a single event is that some are just thrown out, according to some static filter rule, or after checking a datastore. The places where NSQ are typically used is where new data never stops flowing. So sometimes for this sort of thing, it updates a value in some datastore, and creates a new message for the next stage with the latest value, perhaps only if the latest value reaches some threshold, in a scheme where the value decays.