4 ms·

Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is

Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is faster.

m4rtink 3y ago
Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.
- vetinari 3y ago
  It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs. It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.
  - 5e92cb50239222b 3y ago
    It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory. btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower). ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like ├── aa │ ├── aa │ │ └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709 │ ├── ab │ └── ac └── ab ├── aa ├── ab └── ac
    - deleted 3y ago
      [deleted]
- Tigress8780 3y ago
  Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great. We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.
  - nxobject 3y ago
    This is an extremely low quality comment, and I accept any downvotes, but I can't resist: would you say that MongoDB was web scale?
    - Tigress8780 3y ago
      The way our "MongoFS" was organized is actually also low quality and probably would fall apart pretty quickly if used as a serious web service. However, it works if you try to quickly deploy and serve millions of files to a small number of clients. Our use case was a Maven mirror for disconnected environment that only contains metadata (i.e. lots of small XML files, without the actual jar). We already had a MongoDB service for some other JSON data, so here we are.

Many filesystems have an issue with tens of thousands or more files in a single directory, which is exactly what you can get with map tiles. No wonder sqlite is faster.

m4rtink 3y ago

Yeah, that's why sqlite was adopted for this back then - many devices still used FAT32 on the storage volumes where tiles we often stored/cached and that had horrendous small file performance - a plain white 130 Byte PNG tile could result in 64 kB being used.

vetinari 3y ago
It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs. It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.
- 5e92cb50239222b 3y ago
  It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory. btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower). ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like ├── aa │ ├── aa │ │ └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709 │ ├── ab │ └── ac └── ab ├── aa ├── ab └── ac
  - deleted 3y ago
    [deleted]
Tigress8780 3y ago
Once we had to ship millions of extremely small files to our customer, we ended up throwing them into a MongoDB and serve them with a web server. It worked great. We tried to use an image of traditional filesystems (ext4 and fat32), but with most files being under 1 KiB, it was super wasteful.
- nxobject 3y ago
  This is an extremely low quality comment, and I accept any downvotes, but I can't resist: would you say that MongoDB was web scale?
  - Tigress8780 3y ago
    The way our "MongoFS" was organized is actually also low quality and probably would fall apart pretty quickly if used as a serious web service. However, it works if you try to quickly deploy and serve millions of files to a small number of clients. Our use case was a Maven mirror for disconnected environment that only contains metadata (i.e. lots of small XML files, without the actual jar). We already had a MongoDB service for some other JSON data, so here we are.