3 ms·
Small objects are very inefficient in s3. Aggregate them together and form bigger log objects is critical to go from a small system log to a real environment.
by Mave83 2y ago
Small objects are very inefficient in s3. Aggregate them together and form bigger log objects is critical to go from a small system log to a real environment.
- MadsRC 2y agoThis is why systems such as WarpStream regularly runs compaction jobs to more efficiently store objects and cut down on API calls.
- avinassh 2y ago(author here) definitely! I plan to add a batch write API. Also, an API where it buffers till it reaches certain size or a timeout to write to S3 tracking the batch write here: https://github.com/avinassh/s3-log/issues/3 https://github.com/avinassh/s3-log/issues/3
- vitaliyf 2y agoThe company I work for open-sourced a straightforward library that does exactly that: https://github.com/embrace-io/s3-batch-object-store https://github.com/embrace-io/s3-batch-object-store