4 ms·

OP here... I would be happy to discuss any comments you have.

papercruncher 12y ago
I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?
- itaifrenkel 12y ago
  The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)
  - doublerebel 12y ago
    I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).
    - itaifrenkel 12y ago
      We are currently not using nodejs streams. The nodejs components are the event dispatchers placing the events in a queue, they do not include much even processing. Most of the event processing is done in Apache Storm. We have also contributed the Storm-Nodejs integration, but the nodejs Storm components are not using streams either. I have tried using streams to compress the information between the Java parent process in Storm, and the nodejs child process, but compression wasn't a good tradeoff, so even that little bit didn't get into production.
pan69 12y ago
For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?
- ekimekim 12y ago
  Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue: http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basic http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi... > The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels. Depending on your client, it may be difficult to prioritize consumption of one queue over another, so this solution could be preferred.
  - itaifrenkel 12y ago
    Thanks! I'll update the post. A quick search found this RabbitMQ plugin https://github.com/rabbitmq/rabbitmq-priority-queue https://github.com/rabbitmq/rabbitmq-priority-queue , which specifies: "In contrast to the AMQP spec, RabbitMQ queues by default do not support priorities. When creating priority queues using this plugin, you can specify as many priority levels as you like."
- itaifrenkel 12y ago
  Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.
  - pan69 12y ago
    OK. Fair enough. But that's merely a configuration choice and not really a shortcoming of RabbitMQ and even without a cluster, persistent queues are at your disposal, it's just that a HA cluster would be able to give you more of a guarantee. Really enjoyed reading your post by the way...
chwahoo 12y ago
Out of curiosity, what tool did you use to sketch the various architectures for your post?
- itaifrenkel 12y ago
  I used Lucid Charts. It felt easy to work with. https://www.lucidchart.com/ https://www.lucidchart.com/ http://aws.amazon.com/architecture/icons/ http://aws.amazon.com/architecture/icons/
halayli 12y ago
Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other. When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of course this only works if latency is not a big deal.
- itaifrenkel 12y ago
  Our first architecture was indeed peer-to-peer where all components discovered each other. It is difficult to isolate failures on this kind of architecture. A reliable queue makes a big difference in zooming in on the root cause of the problem (a queue producer or a queue consumer problem). Regarding latency - we are aiming for a few ms tops. preferably less. So bulk events using S3 was not considered
  - halayli 12y ago
    I'd argue that queues can introduce a lot of similar problems. Especially when there's slowness (backing up, slow disk etc...). Messages over tcp sent directly to the processors are not that hard to design for failures and high availability and probably your'll reduce the latency significantly. You can also get rid of auto discovery and use config files.

I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?

itaifrenkel 12y ago
The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)
- doublerebel 12y ago
  I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).
  - itaifrenkel 12y ago
    We are currently not using nodejs streams. The nodejs components are the event dispatchers placing the events in a queue, they do not include much even processing. Most of the event processing is done in Apache Storm. We have also contributed the Storm-Nodejs integration, but the nodejs Storm components are not using streams either. I have tried using streams to compress the information between the Java parent process in Storm, and the nodejs child process, but compression wasn't a good tradeoff, so even that little bit didn't get into production.

pan69 12y ago

For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?

ekimekim 12y ago
Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue: http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basic http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi... > The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels. Depending on your client, it may be difficult to prioritize consumption of one queue over another, so this solution could be preferred.
- itaifrenkel 12y ago
  Thanks! I'll update the post. A quick search found this RabbitMQ plugin https://github.com/rabbitmq/rabbitmq-priority-queue https://github.com/rabbitmq/rabbitmq-priority-queue , which specifies: "In contrast to the AMQP spec, RabbitMQ queues by default do not support priorities. When creating priority queues using this plugin, you can specify as many priority levels as you like."
itaifrenkel 12y ago
Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.
- pan69 12y ago
  OK. Fair enough. But that's merely a configuration choice and not really a shortcoming of RabbitMQ and even without a cluster, persistent queues are at your disposal, it's just that a HA cluster would be able to give you more of a guarantee. Really enjoyed reading your post by the way...

chwahoo 12y ago

Out of curiosity, what tool did you use to sketch the various architectures for your post?

itaifrenkel 12y ago
I used Lucid Charts. It felt easy to work with. https://www.lucidchart.com/ https://www.lucidchart.com/ http://aws.amazon.com/architecture/icons/ http://aws.amazon.com/architecture/icons/

halayli 12y ago

Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other. When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of course this only works if latency is not a big deal.

itaifrenkel 12y ago
Our first architecture was indeed peer-to-peer where all components discovered each other. It is difficult to isolate failures on this kind of architecture. A reliable queue makes a big difference in zooming in on the root cause of the problem (a queue producer or a queue consumer problem). Regarding latency - we are aiming for a few ms tops. preferably less. So bulk events using S3 was not considered
- halayli 12y ago
  I'd argue that queues can introduce a lot of similar problems. Especially when there's slowness (backing up, slow disk etc...). Messages over tcp sent directly to the processors are not that hard to design for failures and high availability and probably your'll reduce the latency significantly. You can also get rid of auto discovery and use config files.