I implemented a tool in Go based on the "Bag It" format spec for open data repositories. I remember investigating if it's possible to validate the contents of a

I implemented a tool in Go based on the "Bag It" format spec for open data repositories. I remember investigating if it's possible to validate the contents of a very large ZIP file in an API without uploading multiple gigabytes, that is, drop the upload if the central index does not contain required entries. I learned that ZIP is an indexed format and that the index is (usually) at the end, while TAR is a streaming format (because it wrote to non-random access tape). You can inspect the index on the fly, but you will have already received almost all of the data at that point. The ZIP format (and derivatives) makes sense for the purpose it was likely designed for, which is not useful for use cases where a streaming or incremental approach would be a better solution.