3 ms·

I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in?

I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?

itaifrenkel 12y ago
The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)
- doublerebel 12y ago
  I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).
  - itaifrenkel 12y ago
    We are currently not using nodejs streams. The nodejs components are the event dispatchers placing the events in a queue, they do not include much even processing. Most of the event processing is done in Apache Storm. We have also contributed the Storm-Nodejs integration, but the nodejs Storm components are not using streams either. I have tried using streams to compress the information between the Java parent process in Storm, and the nodejs child process, but compression wasn't a good tradeoff, so even that little bit didn't get into production.

itaifrenkel 12y ago

The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)

doublerebel 12y ago
I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).
- itaifrenkel 12y ago
  We are currently not using nodejs streams. The nodejs components are the event dispatchers placing the events in a queue, they do not include much even processing. Most of the event processing is done in Apache Storm. We have also contributed the Storm-Nodejs integration, but the nodejs Storm components are not using streams either. I have tried using streams to compress the information between the Java parent process in Storm, and the nodejs child process, but compression wasn't a good tradeoff, so even that little bit didn't get into production.