4 ms·

> I assume by better you mean more features? I would guess better on current Windows than on Windows 95. I don't know about faster, but NTFS is most probably m

by jraph 2y ago

> I assume by better you mean more features? I would guess better on current Windows than on Windows 95. I don't know about faster, but NTFS is most probably more reliable than FAT32. And also more features of course, and fewer limitations. At least the file size limit (4 Gb) and ownership / rights metadata (ACL).

PaulHoule 2y ago
Depends how you use it. If you are handling a stupendous number of small files (say doing an npm build) then metadata operations are terribly expensive on Windows because it is expensive to look up security credentials. Maven is not too different from npm in how it works except instead of installing 70,000 small files it installs 70 moderate sized JAR files that are really ZIP files that encase the little files. It works better in Windows than npm. Npm got popular and they had to face down the problem that people would try building the Linux kernel under WSL and get awful times. Microsoft knows it has to win the hearts and minds of developers and they believe in JS, TS and Visual Studio code so they’ve tried all sorts of half-baked things to speed up file I/O for developers.
- akx 2y ago
  No filesystem is great in the lots-of-small-files case, partly simply due to syscall overhead. There's a reason https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html , SquashFS, etc. are a thing, or why even Europe's fastestest supercomputer's admins admonish against lots of small files. https://docs.lumi-supercomputer.eu/storage/#about-the-number-of-files-quota https://docs.lumi-supercomputer.eu/storage/#about-the-number...
  - red_admiral 2y ago
    I guess that's where the "sqlite competes with fopen" part might help.
  - whoknw 2y ago
    The reason LUMI is advising against many files is that it uses the Lustre parallel filesystem, which is notoriously bad with small files. See here: https://www.lanl.gov/projects/national-security-education-center/information-science-technology/_assets/docs/2021-si-docs/booher-kulkarni-lustre-DNE.pdf https://www.lanl.gov/projects/national-security-education-ce... .
  - mech422 2y ago
    That used to be ReiserFS's claim to fame - tail packing small files. Doesn't seem like any FS has really optimized small file handling since then.
    - PaulHoule 2y ago
      It is not so much the wasted space that bugs me as all the metadata and processing of that metadata. For instance, the 70,000 files in your node_modules do not need separate owner and group fields (they all belong to the same user under normal conditions) and are all +rw for the files and +rwx for the directories. If you have access to one you have access to all and if your OS is access checking each file you are paying in terms of battery life, time, etc. On Windows it is the same story except the access control system is much more complex and is slower. (I hope that Linux is not wasting a lot of time processing the POSIX ACLs you aren’t using!)
  - account42 2y ago
    SquashFS is a (read only) filesystem designed for small files. Which shows that even if you don't want to call any filesystem great here, they differ vastly in just how bad they handle small files. Windows' filesystems (and more importantly it's virtual filesystem layer including filters) are on the extremely slow end of this spectrum.
EasyMark 2y ago
As someone who has done some fairly extensive reboot testing of “power failures” for fat32 and ntfs I can guarantee you that NTFS is far more robust but it’s not bullet proof

by jraph 2y ago

PaulHoule 2y ago

Depends how you use it. If you are handling a stupendous number of small files (say doing an npm build) then metadata operations are terribly expensive on Windows because it is expensive to look up security credentials. Maven is not too different from npm in how it works except instead of installing 70,000 small files it installs 70 moderate sized JAR files that are really ZIP files that encase the little files. It works better in Windows than npm. Npm got popular and they had to face down the problem that people would try building the Linux kernel under WSL and get awful times. Microsoft knows it has to win the hearts and minds of developers and they believe in JS, TS and Visual Studio code so they’ve tried all sorts of half-baked things to speed up file I/O for developers.

akx 2y ago
No filesystem is great in the lots-of-small-files case, partly simply due to syscall overhead. There's a reason https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html , SquashFS, etc. are a thing, or why even Europe's fastestest supercomputer's admins admonish against lots of small files. https://docs.lumi-supercomputer.eu/storage/#about-the-number-of-files-quota https://docs.lumi-supercomputer.eu/storage/#about-the-number...
- red_admiral 2y ago
  I guess that's where the "sqlite competes with fopen" part might help.
- whoknw 2y ago
  The reason LUMI is advising against many files is that it uses the Lustre parallel filesystem, which is notoriously bad with small files. See here: https://www.lanl.gov/projects/national-security-education-center/information-science-technology/_assets/docs/2021-si-docs/booher-kulkarni-lustre-DNE.pdf https://www.lanl.gov/projects/national-security-education-ce... .
- mech422 2y ago
  That used to be ReiserFS's claim to fame - tail packing small files. Doesn't seem like any FS has really optimized small file handling since then.
  - PaulHoule 2y ago
    It is not so much the wasted space that bugs me as all the metadata and processing of that metadata. For instance, the 70,000 files in your node_modules do not need separate owner and group fields (they all belong to the same user under normal conditions) and are all +rw for the files and +rwx for the directories. If you have access to one you have access to all and if your OS is access checking each file you are paying in terms of battery life, time, etc. On Windows it is the same story except the access control system is much more complex and is slower. (I hope that Linux is not wasting a lot of time processing the POSIX ACLs you aren’t using!)
- account42 2y ago
  SquashFS is a (read only) filesystem designed for small files. Which shows that even if you don't want to call any filesystem great here, they differ vastly in just how bad they handle small files. Windows' filesystems (and more importantly it's virtual filesystem layer including filters) are on the extremely slow end of this spectrum.

EasyMark 2y ago

As someone who has done some fairly extensive reboot testing of “power failures” for fat32 and ntfs I can guarantee you that NTFS is far more robust but it’s not bullet proof