56 ms·
S3 is "eventually consistent", so I don't think indexing in the background would be such a big deal. But yeah, like I said this would only work for no-compressi
by remram 5y ago
S3 is "eventually consistent", so I don't think indexing in the background would be such a big deal. But yeah, like I said this would only work for no-compression or those schemes that are seekable (not gzip).
In any case it is definitely a lot more work than ZIP.
- klauspost 5y agoNo, S3, as MinIO, has a read-after-write consistency. So indexing would block on either writes or reads until it is done. We block when doing the zip indexing, but that is much more lightweight - and we limit to 100MB ZIP directory. That way we don't risk long-blocking index operations.
- remram 5y agoI see. Indeed that is a potentially long time to block.