4 ms·

This is true. I'm not sure what the obstacle is, and I know a lot of smart people have worked on it, but opening a ton of files and traversing a bunch of them i

by adzm 3y ago

This is true. I'm not sure what the obstacle is, and I know a lot of smart people have worked on it, but opening a ton of files and traversing a bunch of them in directories is very slow. Once the file is open, IO is just as fast though.

amlib 3y ago
Unless you have antivirus software and god knows what analyzing the files, then IO is actually slower...
martinald 3y ago
My guess is ACLs, which are enabled by default on NTFS but not (afiak) on most linux distros using ext4.
- e40 3y ago
  Extended attributes are definitely turned on in my ext4 filesystems and to my knowledge I didn't do anything to turn them on.
  - martinald 3y ago
    Hmm that's true but I don't think much makes use of them? As in they are not set with anything. On NTFS they are used for everything.
WorldMaker 3y ago
Filesystems are a database and have to deal with CAP Theorem trade-offs like everything else. Windows and NTFS both took a heavy focus on Consistency/(lack of) Partitions over Availability. Most POSIX operating systems and their filesystems took a heavy focus on Availability at the expense of Eventual Consistency and Sometimes Partitions. Neither approach is wrong, they are just very different approaches with very different performance trade-offs. Also Windows' filesystem supports an entire plugin stack including user-space plugins, to support things like anti-virus scanners and virtual filesystems and all sorts of other things. Not all of Windows' "slow" filesystem is first-party problems, a lot of it can be third-party drivers and tools that are installed.
- blibble 3y ago
  a local filesystem that can only be mounted once and of which state is 100% controlled by one entity (the local kernel) is not a distributed system the CAP theorem simply does not apply Windows IO subsystem was simply designed for extensibility over performance
  - WorldMaker 3y ago
    A) It's a useful analogy whether or not you think it technically applies or is a perfect analogy. B) The Windows filesystem (and to an extent the POSIX) isn't just "local", it also includes transparent and semi-transparent network file storage. C) Windows and POSIX are both multi-user and multi-process. They operate over multiple cores and multiple I/O buses. Even if it just one system API centralized in charge of all that, it still needs to be built on top of complex distributed dance of mutexes/locks/semaphores/other distributed control structures. Because of the nature of I/O control there are complex caches involved and transaction semantics of when data is actually pulled from/flushed to low level data stores. The transaction model in turn is reflected in what the files look like to other users or processes running at the same time. Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). Those two different transaction models are most easily explained in analogy to CAP Theorem. They are very different transaction models with different trade-offs. Whether or not you see a single "local machine" or you see a distributed system of cores, processes, users, diverse I/O buses is partly a matter of perspective and will reflect how well you personally think CAP Theorem is a "perfect" analogy for a filesystem.
    - blibble 3y ago
      an NTFS volume can only be mounted by one entity at a time, the same as a zfs volume, ext4 volume, btrfs volume with 100% of the state managed by a single entity it is almost the definition of "not a distributed system" the fact there might be nfs/smb/ceph/... volumes bolted into the same namespace that happens to include it does not make it one (and neither does requiring transactions) > Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). this is literally is not true, they are different abstractions and I suggest you observe the size of a large file being copied if you want to see how "strong" NTFS "highly consistent views" are https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...

by adzm 3y ago

amlib 3y ago

Unless you have antivirus software and god knows what analyzing the files, then IO is actually slower...

martinald 3y ago

My guess is ACLs, which are enabled by default on NTFS but not (afiak) on most linux distros using ext4.

e40 3y ago
Extended attributes are definitely turned on in my ext4 filesystems and to my knowledge I didn't do anything to turn them on.
- martinald 3y ago
  Hmm that's true but I don't think much makes use of them? As in they are not set with anything. On NTFS they are used for everything.

WorldMaker 3y ago

Filesystems are a database and have to deal with CAP Theorem trade-offs like everything else. Windows and NTFS both took a heavy focus on Consistency/(lack of) Partitions over Availability. Most POSIX operating systems and their filesystems took a heavy focus on Availability at the expense of Eventual Consistency and Sometimes Partitions. Neither approach is wrong, they are just very different approaches with very different performance trade-offs. Also Windows' filesystem supports an entire plugin stack including user-space plugins, to support things like anti-virus scanners and virtual filesystems and all sorts of other things. Not all of Windows' "slow" filesystem is first-party problems, a lot of it can be third-party drivers and tools that are installed.

blibble 3y ago
a local filesystem that can only be mounted once and of which state is 100% controlled by one entity (the local kernel) is not a distributed system the CAP theorem simply does not apply Windows IO subsystem was simply designed for extensibility over performance
- WorldMaker 3y ago
  A) It's a useful analogy whether or not you think it technically applies or is a perfect analogy. B) The Windows filesystem (and to an extent the POSIX) isn't just "local", it also includes transparent and semi-transparent network file storage. C) Windows and POSIX are both multi-user and multi-process. They operate over multiple cores and multiple I/O buses. Even if it just one system API centralized in charge of all that, it still needs to be built on top of complex distributed dance of mutexes/locks/semaphores/other distributed control structures. Because of the nature of I/O control there are complex caches involved and transaction semantics of when data is actually pulled from/flushed to low level data stores. The transaction model in turn is reflected in what the files look like to other users or processes running at the same time. Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). Those two different transaction models are most easily explained in analogy to CAP Theorem. They are very different transaction models with different trade-offs. Whether or not you see a single "local machine" or you see a distributed system of cores, processes, users, diverse I/O buses is partly a matter of perspective and will reflect how well you personally think CAP Theorem is a "perfect" analogy for a filesystem.
  - blibble 3y ago
    an NTFS volume can only be mounted by one entity at a time, the same as a zfs volume, ext4 volume, btrfs volume with 100% of the state managed by a single entity it is almost the definition of "not a distributed system" the fact there might be nfs/smb/ceph/... volumes bolted into the same namespace that happens to include it does not make it one (and neither does requiring transactions) > Windows and NTFS combined have strong transaction guarantees that other users and processes must see a highly consistent view of the same files. It makes heavy uses of locks by default even for cached data. POSIX favors the "inode" approach that favors high availability and fewer locks at the cost of eventual consistency and the occasional partition (the "same" file can and will sometimes have multiple "inodes" between different processes/users, many common Linux tools rely heavily on that). this is literally is not true, they are different abstractions and I suggest you observe the size of a large file being copied if you want to see how "strong" NTFS "highly consistent views" are https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=8813 https://devblogs.microsoft.com/oldnewthing/20111226-00/?p=88...