3 ms·
I love this. This is a property already baked into another messaging queuing system called Pulsar and I've thought about writing my own for internal use (maybe
by jonathanoliver 7y ago
I love this. This is a property already baked into another messaging queuing system called Pulsar and I've thought about writing my own for internal use (maybe to be open-sourced). The only quirk that I see is this link in the project description readme:
> it is guaranteed that each message will be saved in exactly one S3 file
This can get expensive. Specifically a 1-to-1 write per message with S3 (or any cloud storage provider) gets expensive very, very quickly. Further, when reading from cloud storage, you've got to read one message (object) at a time which adds up as well. I'm not talking about egress bandwidth either, I'm talking about S3 PUT/GET operations. I'd love to see some kind of batching operation that takes groups of messages (perhaps configurable time/size limits?) and writes them as a blog to cloud storage.
- onefuncman 7y agoIt's not a 1:1 write per message, it relies on Kafka offset guarantees to batch messages into S3 at a configurable threshold, 256MB per S3 object is common.
- jonathanoliver 7y agoOh! That's awesome. The way that sentence is written is a little tricky to parse, but the behavior of a batching lots of messages (configurable 256MB+ chunks) is exactly what I'm hoping for.
- justinsaccount 7y agoWhat they said: > it is guaranteed that each message will be saved in exactly one S3 file What you read: > it is guaranteed that each S3 file will contain exactly one message
- otterley 7y agoI think we can all agree that the verbiage could be more elucidating and less confusing.