4 ms·

Much appreciated! This actually has me rethinking the architecture on a project I'm working on right now. It looks like Kinesis would be a little bit cheaper a

Much appreciated! This actually has me rethinking the architecture on a project I'm working on right now. It looks like Kinesis would be a little bit cheaper at the volume of data I'm looking at... But the SNS/SQS method will let me sidestep some potential future scale-up concerns I had with the 5 reads per second limit without making a Rube Goldberg machine of Kinesis Analytics feeding into additional Kinesis Streams, which would drive the cost up higher than an SNS/SQS fanout.

scarface74 8y ago
5 reads per second - yeah that’s kind of low. At maximum scale up. I am processing 80 messages simultaneously on 8 instances - each instance is running 10 (I/O bound) asynchronous threads. I could push it higher but the database starts screaming. But with Kinesis, it’s true that you can do only 5 reads per second, but each read can have up to 10,000 records. With SQS, you can only get 10 records per call. I would think you could get much higher throughout with Kinesis, you would just have to handle storing your iterator/sequence number per shard somewhere to know where you left off in case of crash. Kinesis is much better for higher throughput and you can always scale up instead of out if you need consistent throughout. It depends on your use case.
- cthalupa 8y ago
  Yeah, my primary concern is that there were some other teams expressing interest in hooking up to the stream for working with the data in real time. I know of a few workarounds - if the data is also being sent to S3, having some services working off of S3 events instead of the stream directly, or using Kinesis Analytics to fan out to additional Kinesis streams. I might also just hook up SNS to the stream via Lambda and have them fan out from SNS. Hmm.
  - nostrebored 8y ago
    Keep in mind -- SNS is a push source for Lambda. When doing an approach like SNS->Async Lambda->...->Async Lambda it can become easy to saturate the event bus for your account.
    - scarface74 8y ago
      Exactly. That’s why I suggested Event source -> SNS -> SQS -> lambda and set the concurrency limit of the lambda.

scarface74 8y ago

5 reads per second - yeah that’s kind of low. At maximum scale up. I am processing 80 messages simultaneously on 8 instances - each instance is running 10 (I/O bound) asynchronous threads. I could push it higher but the database starts screaming. But with Kinesis, it’s true that you can do only 5 reads per second, but each read can have up to 10,000 records. With SQS, you can only get 10 records per call. I would think you could get much higher throughout with Kinesis, you would just have to handle storing your iterator/sequence number per shard somewhere to know where you left off in case of crash. Kinesis is much better for higher throughput and you can always scale up instead of out if you need consistent throughout. It depends on your use case.

cthalupa 8y ago
Yeah, my primary concern is that there were some other teams expressing interest in hooking up to the stream for working with the data in real time. I know of a few workarounds - if the data is also being sent to S3, having some services working off of S3 events instead of the stream directly, or using Kinesis Analytics to fan out to additional Kinesis streams. I might also just hook up SNS to the stream via Lambda and have them fan out from SNS. Hmm.
- nostrebored 8y ago
  Keep in mind -- SNS is a push source for Lambda. When doing an approach like SNS->Async Lambda->...->Async Lambda it can become easy to saturate the event bus for your account.
  - scarface74 8y ago
    Exactly. That’s why I suggested Event source -> SNS -> SQS -> lambda and set the concurrency limit of the lambda.