4 ms·

My experience (for an application which had a working set of under 1 GB of files in the 50kb to N MB range, and approximately 50 GB persisted at any given time)

My experience (for an application which had a working set of under 1 GB of files in the 50kb to N MB range, and approximately 50 GB persisted at any given time) was that preserving access to the toolchain which operates trivially with files was worth the additional performance overhead of working with the files and, separately, occasionally having to retool things to e.g. not have 10e7 files in a single folder, which is something that Linux has some opinions (none of them good) about. Trivial example: it's easy to delete any file most recently accessed more than N months ago [+] with a trivial line in cron (the exact line escapes me -- it involves find), but doing that with a database requires that you roll your own access tracking logic. Incremental backups of a directory structure are easy (rsync or tarsnap); incremental backups of a database appeared to me to be highly non-trivial. [+] Since we could re-generate PDFs or gifs from our source of truth at will (with 5~10 seconds of added latency), we deleted anything not accessed in N months to limit our hard disk usage.

Walf 9y ago
Just don't let anyone mount with noatime, for performance. I found that little gem on a drive where paring down to used content would have been very helpful.
- cat199 9y ago
  Or, consider your use case for this at least. Some user-level backups for example, will make the atime useless.
troutaway123 9y ago
¿Porque no los dos? Store the files in a database but expose them via FUSE.
- pjc50 9y ago
  That gives you the time cost of a filesystem plus the time cost of a database plus an extra round-trip in and out of kernel space.
  - troutaway123 9y ago
    > preserving access to the toolchain which operates trivially with files was worth the additional performance overhead of working with the files Sounds like they're OK with that. In any case, couldn't you avoid the kernel trips with some dylib foolery (assuming the toolchain is dynamically linked)?
    - firethief 9y ago
      So: store the files in a database, expose them via FUSE, bypass FUSE and access the database?
      - troutaway123 9y ago
        Yep ;D
- octetta 9y ago
  ...or v9fs... but the overhead might make the whole exercise pointless.
Twirrim 9y ago
> e.g. not have 10e7 files in a single folder, which is something that Linux has some opinions (none of them good) about. I keep running in to this as a common anti-pattern in software, even with software developers that should know better (having fallen afoul of it before). I ran in to it with some semi-bespoke software once, used by a remarkable number of state departments across the country. It was one of many, many things wrong with that godawful thing. Worse, not only were they dumping files in a single directory, almost any time they went to do something with the directory they'd request a directory listing too. Like many of the bizarre things the app did, it never actually used the information it acquired that way.
- cm2187 9y ago
  I never understood why it mattered that files would be in a single directory or split over multiple directories. At the end of the day it is the same file system with the same number of files. The directories are just a logical structure. Why would there be a difference?
  - jamie_ca 9y ago
    Spinning disks. If you want raw file access, your ideal filesystem is a giant key-value store that keys on the full path to the file. This choice means doing a directory listing will involve a lot of random access reads across the disk, and in the days before SSDs this would be a performance killer. So instead, a directory would have a little section of disk to write its file entries to. These would be fairly small, because most directories are small, and if there were more files added than there was room, another block would be allocated. Doing a file listing would be reasonably quick as you're just iterating over a block and following the link to the next block, but random file access would get slower the more files are added. It's probably a solvable problem in filesystem design, but because current filesystems fall down hard on the million-files-in-a-directory use case, nobody actually does that, so there's no drive to make a filesystem performant in that case.
    - the8472 9y ago
      ext4 uses btrees for the directory index, so accessing files by name is just as fast as manually splitting it into a prefix-based directory tree. Searching by filename on the other hand can be much faster with a prefix tree because directories only allow linear scans, not range-based ones.
ccommsxx 9y ago
50GB of files in the 50KB-inf range can be at most 10e5 files, not 10e7. (10e7 would be 5TB of 50KB files)
stuaxo 9y ago
Even with way fewer files than that you can't use wildcards in bash because it will expands to longer than the commandline allows. This makes any work in the shell painful.

Walf 9y ago

Just don't let anyone mount with noatime, for performance. I found that little gem on a drive where paring down to used content would have been very helpful.

cat199 9y ago
Or, consider your use case for this at least. Some user-level backups for example, will make the atime useless.

troutaway123 9y ago

¿Porque no los dos? Store the files in a database but expose them via FUSE.

pjc50 9y ago
That gives you the time cost of a filesystem plus the time cost of a database plus an extra round-trip in and out of kernel space.
- troutaway123 9y ago
  > preserving access to the toolchain which operates trivially with files was worth the additional performance overhead of working with the files Sounds like they're OK with that. In any case, couldn't you avoid the kernel trips with some dylib foolery (assuming the toolchain is dynamically linked)?
  - firethief 9y ago
    So: store the files in a database, expose them via FUSE, bypass FUSE and access the database?
    - troutaway123 9y ago
      Yep ;D
octetta 9y ago
...or v9fs... but the overhead might make the whole exercise pointless.

Twirrim 9y ago

> e.g. not have 10e7 files in a single folder, which is something that Linux has some opinions (none of them good) about. I keep running in to this as a common anti-pattern in software, even with software developers that should know better (having fallen afoul of it before). I ran in to it with some semi-bespoke software once, used by a remarkable number of state departments across the country. It was one of many, many things wrong with that godawful thing. Worse, not only were they dumping files in a single directory, almost any time they went to do something with the directory they'd request a directory listing too. Like many of the bizarre things the app did, it never actually used the information it acquired that way.

cm2187 9y ago
I never understood why it mattered that files would be in a single directory or split over multiple directories. At the end of the day it is the same file system with the same number of files. The directories are just a logical structure. Why would there be a difference?
- jamie_ca 9y ago
  Spinning disks. If you want raw file access, your ideal filesystem is a giant key-value store that keys on the full path to the file. This choice means doing a directory listing will involve a lot of random access reads across the disk, and in the days before SSDs this would be a performance killer. So instead, a directory would have a little section of disk to write its file entries to. These would be fairly small, because most directories are small, and if there were more files added than there was room, another block would be allocated. Doing a file listing would be reasonably quick as you're just iterating over a block and following the link to the next block, but random file access would get slower the more files are added. It's probably a solvable problem in filesystem design, but because current filesystems fall down hard on the million-files-in-a-directory use case, nobody actually does that, so there's no drive to make a filesystem performant in that case.
  - the8472 9y ago
    ext4 uses btrees for the directory index, so accessing files by name is just as fast as manually splitting it into a prefix-based directory tree. Searching by filename on the other hand can be much faster with a prefix tree because directories only allow linear scans, not range-based ones.

ccommsxx 9y ago

50GB of files in the 50KB-inf range can be at most 10e5 files, not 10e7. (10e7 would be 5TB of 50KB files)

stuaxo 9y ago

Even with way fewer files than that you can't use wildcards in bash because it will expands to longer than the commandline allows. This makes any work in the shell painful.