5 ms·

Other example: raster map tiles (basically up to millions of tiny square pictures) Zip vs tar vs filesystem vs sqlite. Tested all these scenarios, and sqlite w

Other example: raster map tiles (basically up to millions of tiny square pictures) Zip vs tar vs filesystem vs sqlite. Tested all these scenarios, and sqlite was the fastest and the smallest, even beating plain archives with no overhead

vetinari 3y ago
Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is faster.
- m4rtink 3y ago
  Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.
  - vetinari 3y ago
    It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs. It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.
    - 5e92cb50239222b 3y ago
      It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory. btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower). ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like ├── aa │ ├── aa │ │ └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709 │ ├── ab │ └── ac └── ab ├── aa ├── ab └── ac
      - deleted 3y ago
        [deleted]
  - Tigress8780 3y ago
    Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great. We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.
    - nxobject 3y ago
      This is an extremely low quality comment, and I accept any downvotes, but I can't resist: would you say that MongoDB was web scale?
      - Tigress8780 3y ago
        The way our "MongoFS" was organized is actually also low quality and probably would fall apart pretty quickly if used as a serious web service. However, it works if you try to quickly deploy and serve millions of files to a small number of clients. Our use case was a Maven mirror for disconnected environment that only contains metadata (i.e. lots of small XML files, without the actual jar). We already had a MongoDB service for some other JSON data, so here we are.
liuliu 3y ago
If SQLite is faster, the problem is the zip library you use. SQLite has a major draw back (and yes, I love SQLite and built a lot of things around it over the years): the blob you get from the DB cannot be mmap and you have to copy it to somewhere else. For zip files, as long as the file is not compressed, you can mmap it (or it is compressed using some exotic encoding such as PVRTC) just fine.

vetinari 3y ago

Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is faster.

m4rtink 3y ago
Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.
- vetinari 3y ago
  It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs. It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.
  - 5e92cb50239222b 3y ago
    It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory. btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower). ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like ├── aa │ ├── aa │ │ └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709 │ ├── ab │ └── ac └── ab ├── aa ├── ab └── ac
    - deleted 3y ago
      [deleted]
- Tigress8780 3y ago
  Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great. We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.
  - nxobject 3y ago
    This is an extremely low quality comment, and I accept any downvotes, but I can't resist: would you say that MongoDB was web scale?
    - Tigress8780 3y ago
      The way our "MongoFS" was organized is actually also low quality and probably would fall apart pretty quickly if used as a serious web service. However, it works if you try to quickly deploy and serve millions of files to a small number of clients. Our use case was a Maven mirror for disconnected environment that only contains metadata (i.e. lots of small XML files, without the actual jar). We already had a MongoDB service for some other JSON data, so here we are.

liuliu 3y ago

If SQLite is faster, the problem is the zip library you use. SQLite has a major draw back (and yes, I love SQLite and built a lot of things around it over the years): the blob you get from the DB cannot be mmap and you have to copy it to somewhere else. For zip files, as long as the file is not compressed, you can mmap it (or it is compressed using some exotic encoding such as PVRTC) just fine.