3 ms·

I'm sure I am just showing my ignorance, but in the man page it says: "-1 list one file per line." and "-f do not sort, enable -aU, disable -ls --color" It

by obloid 5y ago

I'm sure I am just showing my ignorance, but in the man page it says: "-1 list one file per line." and "-f do not sort, enable -aU, disable -ls --color" It's not obvious to me that this would bypass the buffer and fix the problem of being able to list a large number of files. Just that I would get the files unsorted, one line at a time.

p_l 5y ago
Because the buffer is not the problem. ls is slow because it will first load everything to ram, then do a bunch of operations (and it's not really optimized on those, it seems, at least not for speed) and if you have certain options enabled it will also need to stat() each file first. Been there, done that, learnt not to run ls without -1f in a directory that would result in half-hour run.
- obloid 5y ago
  In the linked article it seems his conclusion was the issue was the small buffer size ls uses. But it's not the buffer itself so much as the things done to the data in the buffer that slow things down?
  - p_l 5y ago
    The transfer buffer is not a problem (if you get slowdown in actually reading the directory, the reason is probably the amount of data is huge and you have slow i/o). The problem is when ls then tries to do all kinds of things after reading that data in order to sort, columnize, colorize, etc. For example, just figuring out if you should add / after filename to indicate directory can, in worst case, require stat() call on every returned file - because there's no guarantee that file type will be filled in response from readdir()/getdents(). Sorting and displaying in columns is particularly slow. All sorts of things that are nice to have, cheap quality of life benefits, breakdown and become huge slowdowns with such insane directories.
lmm 5y ago
As a programmer you should know that sorting generally requires everything to be buffered and therefore affects performance / stops streaming from working. Agreed that the stuff about knowing how many columns etc. is not obvious though.
- obloid 5y ago
  I'm not a programmer ;)

by obloid 5y ago

p_l 5y ago

Because the buffer is not the problem. ls is slow because it will first load everything to ram, then do a bunch of operations (and it's not really optimized on those, it seems, at least not for speed) and if you have certain options enabled it will also need to stat() each file first. Been there, done that, learnt not to run ls without -1f in a directory that would result in half-hour run.

obloid 5y ago
In the linked article it seems his conclusion was the issue was the small buffer size ls uses. But it's not the buffer itself so much as the things done to the data in the buffer that slow things down?
- p_l 5y ago
  The transfer buffer is not a problem (if you get slowdown in actually reading the directory, the reason is probably the amount of data is huge and you have slow i/o). The problem is when ls then tries to do all kinds of things after reading that data in order to sort, columnize, colorize, etc. For example, just figuring out if you should add / after filename to indicate directory can, in worst case, require stat() call on every returned file - because there's no guarantee that file type will be filled in response from readdir()/getdents(). Sorting and displaying in columns is particularly slow. All sorts of things that are nice to have, cheap quality of life benefits, breakdown and become huge slowdowns with such insane directories.

lmm 5y ago

As a programmer you should know that sorting generally requires everything to be buffered and therefore affects performance / stops streaming from working. Agreed that the stuff about knowing how many columns etc. is not obvious though.

obloid 5y ago
I'm not a programmer ;)