4 ms·

One of my commonly used Unix one-liners, using awk, is to get the sum of the file sizes for the files listed by the ls command (with the -R for recursive option

by vram22 10y ago

One of my commonly used Unix one-liners, using awk, is to get the sum of the file sizes for the files listed by the ls command (with the -R for recursive option if wanted): ls -lR /path/to/dir | awk ' { s += $5 } END { print s / 1024 " K" } ' $5 is the 5th field of the output, which is the file size field in the case of ls output. The code inside the first set of braces runs once for every line of input (which comes from standard input, so from the ls command, in this case), and the code inside the second set of braces runs at the end of the input, calculating and printing the desired result of the total of all file sizes for files found by ls, in kilobytes. It can easily be changed to output the total in bytes or megabytes by dropping the '/ 1024' or adding another one after the first. Variable s is initialized to 0 by default at the start. You can get similar info with "du -hs /path/to/dir" but the ls plus awk pipeline lends itself to more customization, such as adding conditions for the type or owner of the file, etc.

Hello71 10y ago
and your way breaks if a filename contains a new line.
- bdowling 10y ago
  You can fix that by adding -B (or -b) to the ls command.
  - mnaydin 10y ago
    Neither -b nor -B option is defined in the POSIX ls utility. The -b option is a GNU extension to print C-style escapes for nongraphic characters, and it is useful for this case. The -B option is also GNU extension not to list implied entries ending with ~, and I don't understand how it is relevant to this case.
    - bdowling 10y ago
      On OS X and BSD (at least, and possibly others) the -B option forces octal-style \xxx output for non-printable characters, while -b uses C-style escapes where possible. You are correct regarding the GNU option -B, which would not be relevant here, and regarding the fact that neither of these options are in POSIX. Thanks for pointing out those limitations.
mnaydin 10y ago
I'd use the find command with the -printf option (GNU find has this option but POSIX find doesn't define it) instead of ls. For instance: find /path/to/dir -type f -printf "%s\n" | awk ' { s += $0 } END { print s " bytes" } ' The find command has much powerful file filtering capabilities than that of the ls command and works better with weird characters in filenames.
- vram22 10y ago
  Thanks. Yes, I'm aware that in general find is a better option (long time Unix guy) than even a recursive ls command (ls -R) for finding files under a directory and processing them in some way (often together with xargs, to get around the args length limit). But mine was just a quick example, so I didn't use find. Actually, find is also better for this example, because with it, you do not have to deal with per-dir header lines like "dirname:" and "total n" (n blocks) that ls outputs. (The headers may not matter for my example, because I only process field 5, but they can matter for other kinds of processing of the output.) There is also the -print0 option to find to handle filenames with newlines in them. -print0 may be non-POSIX and a GNU extension. POSIX has -print, but interestingly, in some Unixes I have seen that not using -print still prints the filenames found, by default.
  - mnaydin 10y ago
    That's the expected behaviour. Quoting from spec: If no expression is present, -print shall be used as the expression. Otherwise, if the given expression does not contain any of the primaries -exec, -ok, or -print, the given expression shall be effectively replaced by: ( given_expression ) -print http://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/fi...
    - vram22 10y ago
      Yes, I wasn't implying the behavior is wrong. Was just mentioning it. Anyway, thanks for that link, which explains why. That Open Group info on POSIX utilities is a great resource for when you want to know the comprehensive, well-specified behavior of the commands.

by vram22 10y ago

Hello71 10y ago

and your way breaks if a filename contains a new line.

bdowling 10y ago
You can fix that by adding -B (or -b) to the ls command.
- mnaydin 10y ago
  Neither -b nor -B option is defined in the POSIX ls utility. The -b option is a GNU extension to print C-style escapes for nongraphic characters, and it is useful for this case. The -B option is also GNU extension not to list implied entries ending with ~, and I don't understand how it is relevant to this case.
  - bdowling 10y ago
    On OS X and BSD (at least, and possibly others) the -B option forces octal-style \xxx output for non-printable characters, while -b uses C-style escapes where possible. You are correct regarding the GNU option -B, which would not be relevant here, and regarding the fact that neither of these options are in POSIX. Thanks for pointing out those limitations.

mnaydin 10y ago

I'd use the find command with the -printf option (GNU find has this option but POSIX find doesn't define it) instead of ls. For instance: find /path/to/dir -type f -printf "%s\n" | awk ' { s += $0 } END { print s " bytes" } ' The find command has much powerful file filtering capabilities than that of the ls command and works better with weird characters in filenames.

vram22 10y ago
Thanks. Yes, I'm aware that in general find is a better option (long time Unix guy) than even a recursive ls command (ls -R) for finding files under a directory and processing them in some way (often together with xargs, to get around the args length limit). But mine was just a quick example, so I didn't use find. Actually, find is also better for this example, because with it, you do not have to deal with per-dir header lines like "dirname:" and "total n" (n blocks) that ls outputs. (The headers may not matter for my example, because I only process field 5, but they can matter for other kinds of processing of the output.) There is also the -print0 option to find to handle filenames with newlines in them. -print0 may be non-POSIX and a GNU extension. POSIX has -print, but interestingly, in some Unixes I have seen that not using -print still prints the filenames found, by default.
- mnaydin 10y ago
  That's the expected behaviour. Quoting from spec: If no expression is present, -print shall be used as the expression. Otherwise, if the given expression does not contain any of the primaries -exec, -ok, or -print, the given expression shall be effectively replaced by: ( given_expression ) -print http://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/fi...
  - vram22 10y ago
    Yes, I wasn't implying the behavior is wrong. Was just mentioning it. Anyway, thanks for that link, which explains why. That Open Group info on POSIX utilities is a great resource for when you want to know the comprehensive, well-specified behavior of the commands.