5 ms·

Small File Archives in MinIO

57 points by edogrider 5y ago

toomuchtodo 5y ago
I have wanted this forever from S3 (cherry-picking from stored object archives) without having to implement reading/caching the zip index and doing a range request. This is awesome, solid feature.
- HyperRational 5y ago
- laurencerowe 5y ago
  While not zip S3 does now support SELECT into a number of datatypes. https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...
  - klauspost 5y ago
    Correct, but it pretty much forces a scan through the file. For AWS, you pay for scanning the file, even if only a few results are returned. Currently this is $0.002 per scanned GB, plus request and transfer fee. For MinIO you "pay" in CPU time and disk iops to scan the file. So for retrieving single values it is a very inefficient/expensive method.
- klauspost 5y ago
  (dev here) Thanks! The initial idea was a customer request was to specify ranges as key=x->y;key2=z->w when uploading a file via a header and then specifying the key as a substitute for Range on get. We chewed a bit on that and found that the zip method was much more flexible and easy to use, where the file name effectively is the key. Either would require client-side changes, so we might as well make it as easy to use as possible.
gopalv 5y ago
So this looks a lot like what Hadoop did with .har files[1] on HDFS (like storing GPS tiles on HDFS without blowing through the 1 M files per-dir limit). > It is not possible to update individual files inside the ZIP file. Therefore this should only be used for data that isn’t expected to change. I've actually done file-replaces on .zip files on HDFS, because .ZIP files are actually written with a directory in a footer, you can go to the end and append new data without having to "modify" existing files. This doesn't conflict with the block level immutability, though the entire write has to be a single commit to avoid leaving the file in a bad way. I'd say that the best case use-case for this is the storage of log files (like if you had fluentd writing .zip files by appending to it rather than a diff object for each 5 minute window). When it comes to stuff which compresses well but full of small objects, ZIP is pretty bad because each file in the zip independently contains a dictionary (look at the .xlsx file inside to know how MSFT solved that, but in way which makes you hate it - a strings directory for shared strings across all files). [1] - https://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html#How+to+Create+an+Archive https://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html#H...
- klauspost 5y ago
  > When it comes to stuff which compresses well but full of small objects, ZIP is pretty bad Correct, but it also allows you to independently access files, which is a win for this use-case. The goal isn't the compression itself, but reducing the number of objects, which in itself reduces the file system block overhead. Double checking, it seems like files compressed with zstandard method aren't supported. This will (when enabled) give both better compression and faster decompression. That should be added shortly.
heipei 5y ago
Backblaze, S3 and R2 listen up: This is a great feature and one which I would immediately use to store related data in one zip file. Not only does this cut down on the number of objects in your storage, it also makes deletion atomic since you don't have to ensure that all types of objects for the same ID have been deleted.
speedgoose 5y ago
Would it work in gateway mode ?
- y4m4b4 5y ago
  No, gateway mode is EOL'ed and it is not taking any new features
  - speedgoose 5y ago
    Oh I see. I checked the repository and it's being stopped very soon: https://github.com/minio/minio/issues/14331 https://github.com/minio/minio/issues/14331
jaytaylor 5y ago
Would this approach also work for Tar archives? Transparent support for sub-files from a .tar would be badass.
- rmgraham 5y ago
  Tar doesn't use any sort of index like zip does, so to extract the specified file the server side would need to parse through possibly the entire file just to see if the requested file is there, and then start streaming it. Requests for files that aren't in the tar archive would be prohibitively expensive. There are definitely ways to do it without those problems, though. They just wouldn't be quite as simple as the approach done for supporting zip.
  - remram 5y ago
    You could pre-index them I suppose. Though even that would only work with a subset of compression methods or no compression.
    - deleted 5y ago
      [deleted]
    - blacha 5y ago
      This is basically exactly what we do we have created a cloud optimised tar (cotar)[1] by creating a hash index of the files inside the tar. I work with serving tiled geospatial data [2] (Mapbox vector tiles) to our users as slippy maps where we serve millions of small (mostly <100KB) files to our users, our data only changes weekly so we precompute all the tiles and store them in a tar file in s3. We compute a index for the tar file then use s3 range requests to serve the tiles to our users, this means we can generally fetch a tile from s3 with 2 (or 1 if the index is cached) requests to s3 (generally ~20-50ms). To get full coverage of the world with map box vector tiles it is around 270M tiles and a ~90GB tar file which can be computed from open street map data [3] > Though even that would only work with a subset of compression methods or no compression. We compress the individual files as a work around, there are options for indexing a compressed (gzip) tar file but the benefits of a compressed tar vs compressed files are small for our use case [1] https://github.com/linz/cotar https://github.com/linz/cotar (or wip rust version https://github.com/blacha/cotar-rs https://github.com/blacha/cotar-rs) [2] https://github.com/linz/basemaps https://github.com/linz/basemaps or https://basemaps.linz.govt.nz https://basemaps.linz.govt.nz [3] https://github.com/onthegomap/planetiler https://github.com/onthegomap/planetiler

57 points by edogrider 5y ago

I have wanted this forever from S3 (cherry-picking from stored object archives) without having to implement reading/caching the zip index and doing a range request. This is awesome, solid feature.

HyperRational 5y ago
laurencerowe 5y ago
While not zip S3 does now support SELECT into a number of datatypes. https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...
- klauspost 5y ago
  Correct, but it pretty much forces a scan through the file. For AWS, you pay for scanning the file, even if only a few results are returned. Currently this is $0.002 per scanned GB, plus request and transfer fee. For MinIO you "pay" in CPU time and disk iops to scan the file. So for retrieving single values it is a very inefficient/expensive method.
klauspost 5y ago
(dev here) Thanks! The initial idea was a customer request was to specify ranges as key=x->y;key2=z->w when uploading a file via a header and then specifying the key as a substitute for Range on get. We chewed a bit on that and found that the zip method was much more flexible and easy to use, where the file name effectively is the key. Either would require client-side changes, so we might as well make it as easy to use as possible.

gopalv 5y ago

So this looks a lot like what Hadoop did with .har files[1] on HDFS (like storing GPS tiles on HDFS without blowing through the 1 M files per-dir limit). > It is not possible to update individual files inside the ZIP file. Therefore this should only be used for data that isn’t expected to change. I've actually done file-replaces on .zip files on HDFS, because .ZIP files are actually written with a directory in a footer, you can go to the end and append new data without having to "modify" existing files. This doesn't conflict with the block level immutability, though the entire write has to be a single commit to avoid leaving the file in a bad way. I'd say that the best case use-case for this is the storage of log files (like if you had fluentd writing .zip files by appending to it rather than a diff object for each 5 minute window). When it comes to stuff which compresses well but full of small objects, ZIP is pretty bad because each file in the zip independently contains a dictionary (look at the .xlsx file inside to know how MSFT solved that, but in way which makes you hate it - a strings directory for shared strings across all files). [1] - https://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html#How+to+Create+an+Archive https://hadoop.apache.org/docs/r1.2.1/hadoop_archives.html#H...

klauspost 5y ago
> When it comes to stuff which compresses well but full of small objects, ZIP is pretty bad Correct, but it also allows you to independently access files, which is a win for this use-case. The goal isn't the compression itself, but reducing the number of objects, which in itself reduces the file system block overhead. Double checking, it seems like files compressed with zstandard method aren't supported. This will (when enabled) give both better compression and faster decompression. That should be added shortly.

heipei 5y ago

Backblaze, S3 and R2 listen up: This is a great feature and one which I would immediately use to store related data in one zip file. Not only does this cut down on the number of objects in your storage, it also makes deletion atomic since you don't have to ensure that all types of objects for the same ID have been deleted.

speedgoose 5y ago

Would it work in gateway mode ?

y4m4b4 5y ago
No, gateway mode is EOL'ed and it is not taking any new features
- speedgoose 5y ago
  Oh I see. I checked the repository and it's being stopped very soon: https://github.com/minio/minio/issues/14331 https://github.com/minio/minio/issues/14331

jaytaylor 5y ago

Would this approach also work for Tar archives? Transparent support for sub-files from a .tar would be badass.

rmgraham 5y ago
Tar doesn't use any sort of index like zip does, so to extract the specified file the server side would need to parse through possibly the entire file just to see if the requested file is there, and then start streaming it. Requests for files that aren't in the tar archive would be prohibitively expensive. There are definitely ways to do it without those problems, though. They just wouldn't be quite as simple as the approach done for supporting zip.
- remram 5y ago
  You could pre-index them I suppose. Though even that would only work with a subset of compression methods or no compression.
  - deleted 5y ago
    [deleted]
  - blacha 5y ago
    This is basically exactly what we do we have created a cloud optimised tar (cotar)[1] by creating a hash index of the files inside the tar. I work with serving tiled geospatial data [2] (Mapbox vector tiles) to our users as slippy maps where we serve millions of small (mostly <100KB) files to our users, our data only changes weekly so we precompute all the tiles and store them in a tar file in s3. We compute a index for the tar file then use s3 range requests to serve the tiles to our users, this means we can generally fetch a tile from s3 with 2 (or 1 if the index is cached) requests to s3 (generally ~20-50ms). To get full coverage of the world with map box vector tiles it is around 270M tiles and a ~90GB tar file which can be computed from open street map data [3] > Though even that would only work with a subset of compression methods or no compression. We compress the individual files as a work around, there are options for indexing a compressed (gzip) tar file but the benefits of a compressed tar vs compressed files are small for our use case [1] https://github.com/linz/cotar https://github.com/linz/cotar (or wip rust version https://github.com/blacha/cotar-rs https://github.com/blacha/cotar-rs) [2] https://github.com/linz/basemaps https://github.com/linz/basemaps or https://basemaps.linz.govt.nz https://basemaps.linz.govt.nz [3] https://github.com/onthegomap/planetiler https://github.com/onthegomap/planetiler