3 ms·
It depends. What is your use case? If you mean this project specifically, part one says "you will probably never need to build something like this yourself (nor
by TheCowboy 9y ago
It depends. What is your use case? If you mean this project specifically, part one says "you will probably never need to build something like this yourself (nor should you)".
Dumping everything into an S3 bucket may be perfectly fine and even efficient for some projects. I personally don't see myself wanting to use S3 buckets for transmitting messages between multiple different systems with different purposes.
If you have a lot of different services producing data that multiple services will consume, then a pub/sub system can simplify things. Some systems, like Kafka, also have added redundancy/scaling benefits.
- AndyNemmity 9y agoUse case is a Cloud, taking all audit/log data, wrapping in metadata, and then placing it in a location which then needs analytics available on. Currently this is done for the control plane by just loading them directly into Elasticsearch. Increasing the availability for all users, the consideration is, does a pub/sub add anything that just throwing the data directly into an s3 like bucket (swift in our case). I just assumed it did. Everyone does it. Scale, decoupling the sender from the receiver... But... I mean, why? What's the point of an additional hop, and another thing to manage. If your source is already wrapping metadata, and perhaps putting it into a structure, what is the value of a pub/sub there? If you can load it to the pub/sub, you can load it to the s3 bucket in the same manner.
- amarkov 9y agoPartitioning the S3 bucket scheme is terribly complicated. The senders have to know exactly how every receiver is partitioned, so they can adopt a compatible partitioning scheme when writing to S3. And the receivers have to know exactly how the senders are partitioned, so they can divvy up incoming data appropriately.