3 ms·

The first question I have is why? SQS seems like such a simple thing to keep hosted.

jjeaff 7y ago
They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.
- rumanator 7y ago
  Even if you were off by an order of magnitude, that expenditure level is not justified to run a message broker service.
  - jhh 7y ago
    5k a month would be an amazing deal.
- deanCommie 7y ago
  I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput. No way they would actually hit that sustained throughput for the entire month. Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.
  - nostrebored 7y ago
    Why is this hard to believe? Maybe they have chatty IoT devices. Maybe a ton of sensors monitoring manufacturing plants for multiple different metrics in real-time. Maybe they just have large scale.
  - manigandham 7y ago
    We do 100k msgs/sec minimum in our streaming platform. This is on the low end for adtech and other industries.
addisonj 7y ago
Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place
- tybit 7y ago
  When we ran into a similar use case we went with writing to S3 and using S3 SNS notifications and consumers could subscribe their SQS queues to that. Having said that, assuming Pulsar has a similar feature to Kafkas log compaction I can definitely see the appeal! Also the fact that SQS Fifo doesn’t integrate with this setup is super annoying. Edit: Log compaction not key compaction

They said they are currently doing 50k messages a second and they aren't even done migrating everything over. 50k messages a second would cost you around $50k a month for AWS sqs, (math could be wrong, didn't double check). Plus, with sqs, you get what they have. No customizations.

rumanator 7y ago
Even if you were off by an order of magnitude, that expenditure level is not justified to run a message broker service.
- jhh 7y ago
  5k a month would be an amazing deal.
deanCommie 7y ago
I sincerely doubt they are sustaining 50k msgs/second. Likely that's the MAXIMUM throughput. No way they would actually hit that sustained throughput for the entire month. Even the other justifications about wanting to reference messages after delivery do not to me justify migrating off SQS/Kinesis, especially not at cost of 5 months development effort.
- nostrebored 7y ago
  Why is this hard to believe? Maybe they have chatty IoT devices. Maybe a ton of sensors monitoring manufacturing plants for multiple different metrics in real-time. Maybe they just have large scale.
- manigandham 7y ago
  We do 100k msgs/sec minimum in our streaming platform. This is on the low end for adtech and other industries.

addisonj 7y ago

Biggest thing is that we find ourselves needing to retain this data for more than initial delivery and also for use cases where we want to use the data more like a log and need ordering guarantees. It isn't just our current SQS use cases, it is being able to have one tech that does SQS like stuff and Kinesis like stuff in one place

tybit 7y ago
When we ran into a similar use case we went with writing to S3 and using S3 SNS notifications and consumers could subscribe their SQS queues to that. Having said that, assuming Pulsar has a similar feature to Kafkas log compaction I can definitely see the appeal! Also the fact that SQS Fifo doesn’t integrate with this setup is super annoying. Edit: Log compaction not key compaction