5 ms·

There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next p

There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next pass. If you do this before that data gets removed from the cache it will be much faster. imac:~ $ time find . | wc -l 419191 real 0m37.865s user 0m0.385s sys 0m2.611s imac:~ $ time find . | wc -l 419195 real 0m10.503s user 0m0.330s sys 0m1.430s

ck2 13y ago
This is a nasty hack where it is counting the output from find. What do you do when find cannot handle the sheer number of files, argument too long from the linux hardcoded MAX_ARG_PAGES We need a dedicated utility that counts inodes.
- deleted 13y ago
  [deleted]
- clarry 13y ago
  In this invocation find and wc both have two fixed arguments, you're never going to hit MAX_ARG_PAGES. The output from find is passed to wc over a pipe, and to my knowledge there's no upper limit on how much data you can transfer over a pipe. I've piped huge disk images.
- teddyh 13y ago
  I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l". Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?
  - ck2 13y ago
    I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count. All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.
    - res0nat0r 13y ago
      You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself. https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/ext3 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....
    - wiredfool 13y ago
      OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds. That's "time find . | wc -l" It's going to be disk limited unless you're in cache. Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.
      - jbarber 13y ago
        There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.
- inyourtenement 13y ago
  The only argument to his find command is '.', so where is the concern about argument list?
- js2 13y ago
  That is not at all correct. The output from find is a list of filenames separated by newlines. The wc command is reading from stdin and counting those newlines[1] -- wc itself has just two arguments in its argv array here ("wc", "-l"). You will not exceed its argument list. Further, it is not at all a hack to count the number of files this way; rather using pipes to compose functionality is the Unix way[2]. But fwiw, if you want a count of the inodes in use in the entire filesystem, you can get that directly from "df -i". In any case, as an exercise to the reader, I encourage you to grab the source to find and add the -count output option you desire. [1] Technically a filename can contain newlines, so this would throw off the count. The fix for that would be to use find's -print0 output option and then pipe to something which can count on the nulls. In practice, you're unlikely to have such filenames. [2] cf. http://harmful.cat-v.org/cat-v/unix_prog_design.pdf http://harmful.cat-v.org/cat-v/unix_prog_design.pdf

ck2 13y ago

This is a nasty hack where it is counting the output from find. What do you do when find cannot handle the sheer number of files, argument too long from the linux hardcoded MAX_ARG_PAGES We need a dedicated utility that counts inodes.

deleted 13y ago
[deleted]
clarry 13y ago
In this invocation find and wc both have two fixed arguments, you're never going to hit MAX_ARG_PAGES. The output from find is passed to wc over a pipe, and to my knowledge there's no upper limit on how much data you can transfer over a pipe. I've piped huge disk images.
teddyh 13y ago
I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l". Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?
- ck2 13y ago
  I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count. All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.
  - res0nat0r 13y ago
    You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself. https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/ext3 https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....
  - wiredfool 13y ago
    OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds. That's "time find . | wc -l" It's going to be disk limited unless you're in cache. Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.
    - jbarber 13y ago
      There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.
inyourtenement 13y ago
The only argument to his find command is '.', so where is the concern about argument list?
js2 13y ago
That is not at all correct. The output from find is a list of filenames separated by newlines. The wc command is reading from stdin and counting those newlines[1] -- wc itself has just two arguments in its argv array here ("wc", "-l"). You will not exceed its argument list. Further, it is not at all a hack to count the number of files this way; rather using pipes to compose functionality is the Unix way[2]. But fwiw, if you want a count of the inodes in use in the entire filesystem, you can get that directly from "df -i". In any case, as an exercise to the reader, I encourage you to grab the source to find and add the -count output option you desire. [1] Technically a filename can contain newlines, so this would throw off the count. The fix for that would be to use find's -print0 output option and then pipe to something which can count on the nulls. In practice, you're unlikely to have such filenames. [2] cf. http://harmful.cat-v.org/cat-v/unix_prog_design.pdf http://harmful.cat-v.org/cat-v/unix_prog_design.pdf