5 ms·

> What I actually care most about a file manager is how they perform on mounts with low IOPS Purely technical questions (let's put `being very snarky` and `dis

by apjana 8y ago

> What I actually care most about a file manager is how they perform on mounts with low IOPS Purely technical questions (let's put `being very snarky` and `dishonesty` and my other irrelevant personal traits aside): - How does "low IOPS" affect readdir()/scandir() and lstat64() in two C utilities differently? - What else would be affected?

laumars 8y ago
It’s about how and when they get used. Eg if you’re running on a lower performing mount then you might want to rely on caches more than frequent rescans. What I would often do on performance critical routines running against network mounts was store a tree of inodes and time stamps and only rescan a directory if the parents time stamp changed. It meant I’d miss any subsequent metadata changes to individual files (mtime, size, etc) but I would capture any new files being created. Which is the main thing I cared about so that was the trade off I was willing to make. There’s no right or wrong answer here though. Just different designs for different problems. Which was also the point the other developer was making when he was talking about his memory usage.
- apjana 8y ago
  > There’s no right or wrong answer here though. The processor cache plays an important role which you are ignoring. External storage devices: most of the time they are write-back and even equipped with read-ahead. Yes, I know there are some exceptions but if you are write-through non-read-ahead you _chose_ to be slow in your feedback already and this discussion doesn't even apply. Network mounts: cache coherency rules apply to CIFS as well. And again, if you _choose_ to ignore/disable, you are OK to be slow and this discussion does not apply. If `nnn` take n secs the first time, another utility will take around the same time on the first startup (from a cold boot). Now the next scans where you go into subdirs would be much faster even in `nnn` due to locality of caching of the information about the files (try it out). The CPU cache already does an excellent job here. And if you go up, both `nnn` and the other utility would rescan. > point the other developer was making Yes, he was saying - my memory usage may be 15 times higher because of storing all filenames (in a static snapshot!!!) but you are dishonest if you show the numbers from `top` output without reading my code first for an education of my utility.
  - laumars 8y ago
    I’m not sure what you mean by processor cache here. The processor wouldn’t cache file system meta data. Kernels will, but that is largely dependant on the respective file system driver (eg none of the hobby projects I’ve written in FUSE had any caching). Different write modes on external hardware also confuses the issue because you still have the slower bus speeds (eg a USB2 for an older memory stick) to and from the external device than you might have with dedicated internal hardware.
    - apjana 8y ago
      > The processor wouldn’t cache file system meta data What _file system meta data_? The processor doesn't care what data it is! I am talking of the hw control plane and you are still lurching at pure software constructs like the kernel and its drivers. All the CPU cares is the latest data fetched by a program. The CPU cache exists to store program instructions and _data_ (no matter where it comes from) used repeatedly in the operation of programs or information that the CPU is likely to need next. If the data is still available in the cacheline and isn't invalidated, CPU won't fetch it from an external unit (so bus request is not even made). _Any_ data coming to CPU sits in any Ln cache, source notwithstanding. The external memory is accessed in case of cache misses. However, the metadata these utilities fetch is very very less and the probability is greatly reduced. Moreover, your hypothetical utility also banks on the assumption that this data won't change and it wouldn't have to issue too many rescans to remain performant. It's the same thing you see when you copy a 2 GB file from a slow storage to SSD and the first time it's slow but the next time it's way faster. You can see it for yourself. Run `nnn` on any external disk you are having (with a lot of data preferably), navigate to the mounpoint, press `^J` (1. notice the time taken), move to a subdir, come back to mountpoint again (2. notice the time taken). You would see what I mean. > none of the hobby projects I’ve written in FUSE had any caching On a side note (and though not much relevant here), all serious drivers (e.g. those from Tuxera, btrfs) maintain buffer cache (https://www.tldp.org/LDP/sag/html/buffer-cache.html https://www.tldp.org/LDP/sag/html/buffer-cache.html). They always boost performance. If our Ln misses, this is where we would get the metadata from and _hopefully_ not from the disk which is the worst case.
- deleted 8y ago
  [deleted]

by apjana 8y ago

laumars 8y ago

It’s about how and when they get used. Eg if you’re running on a lower performing mount then you might want to rely on caches more than frequent rescans. What I would often do on performance critical routines running against network mounts was store a tree of inodes and time stamps and only rescan a directory if the parents time stamp changed. It meant I’d miss any subsequent metadata changes to individual files (mtime, size, etc) but I would capture any new files being created. Which is the main thing I cared about so that was the trade off I was willing to make. There’s no right or wrong answer here though. Just different designs for different problems. Which was also the point the other developer was making when he was talking about his memory usage.

apjana 8y ago
> There’s no right or wrong answer here though. The processor cache plays an important role which you are ignoring. External storage devices: most of the time they are write-back and even equipped with read-ahead. Yes, I know there are some exceptions but if you are write-through non-read-ahead you _chose_ to be slow in your feedback already and this discussion doesn't even apply. Network mounts: cache coherency rules apply to CIFS as well. And again, if you _choose_ to ignore/disable, you are OK to be slow and this discussion does not apply. If `nnn` take n secs the first time, another utility will take around the same time on the first startup (from a cold boot). Now the next scans where you go into subdirs would be much faster even in `nnn` due to locality of caching of the information about the files (try it out). The CPU cache already does an excellent job here. And if you go up, both `nnn` and the other utility would rescan. > point the other developer was making Yes, he was saying - my memory usage may be 15 times higher because of storing all filenames (in a static snapshot!!!) but you are dishonest if you show the numbers from `top` output without reading my code first for an education of my utility.
- laumars 8y ago
  I’m not sure what you mean by processor cache here. The processor wouldn’t cache file system meta data. Kernels will, but that is largely dependant on the respective file system driver (eg none of the hobby projects I’ve written in FUSE had any caching). Different write modes on external hardware also confuses the issue because you still have the slower bus speeds (eg a USB2 for an older memory stick) to and from the external device than you might have with dedicated internal hardware.
  - apjana 8y ago
    > The processor wouldn’t cache file system meta data What _file system meta data_? The processor doesn't care what data it is! I am talking of the hw control plane and you are still lurching at pure software constructs like the kernel and its drivers. All the CPU cares is the latest data fetched by a program. The CPU cache exists to store program instructions and _data_ (no matter where it comes from) used repeatedly in the operation of programs or information that the CPU is likely to need next. If the data is still available in the cacheline and isn't invalidated, CPU won't fetch it from an external unit (so bus request is not even made). _Any_ data coming to CPU sits in any Ln cache, source notwithstanding. The external memory is accessed in case of cache misses. However, the metadata these utilities fetch is very very less and the probability is greatly reduced. Moreover, your hypothetical utility also banks on the assumption that this data won't change and it wouldn't have to issue too many rescans to remain performant. It's the same thing you see when you copy a 2 GB file from a slow storage to SSD and the first time it's slow but the next time it's way faster. You can see it for yourself. Run `nnn` on any external disk you are having (with a lot of data preferably), navigate to the mounpoint, press `^J` (1. notice the time taken), move to a subdir, come back to mountpoint again (2. notice the time taken). You would see what I mean. > none of the hobby projects I’ve written in FUSE had any caching On a side note (and though not much relevant here), all serious drivers (e.g. those from Tuxera, btrfs) maintain buffer cache (https://www.tldp.org/LDP/sag/html/buffer-cache.html https://www.tldp.org/LDP/sag/html/buffer-cache.html). They always boost performance. If our Ln misses, this is where we would get the metadata from and _hopefully_ not from the disk which is the worst case.
deleted 8y ago
[deleted]