6 ms·

I think there's a middle point where you want to do something that's complex enough that a glob won't cut it but simple enough that switching languages is not w

I think there's a middle point where you want to do something that's complex enough that a glob won't cut it but simple enough that switching languages is not worth it. I think the example of "exclude these two types of files" is a good case. I often have to write stuff like `ls P* | grep -Ev "wav|draft"` which doesn't solve a problem I don't have (such as filenames with newlines in them) but does solve the one I do (keeping a subset of files that would be tricky to glob properly). In my experience 95% of those scripts are going to be discarded in a week, and bringing Python into it means I need to deal with `os.path` and `subprocess.run`. My rule of thumb: if it's not going to be version controlled then Bash is fine.

deleted 2y ago
[deleted]
OrderlyTiamat 2y ago
You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"` This uses regex to match files ending in .wav or .draft (which is what I interpreted you to want). Xargs then processes the file. You could use flags to have xargs pass the file names in a specific place in the command, which can even be a one liner shell call or some script. So the "find <regex> - xarg <command>" pattern is almost fully generally applicable to any problem where you want to execute a oneliner on a number of files with regular names. (I think gnu find has no extended regex, which is just as well- thats not a "regular expression" at that point)
- bheadmaster 2y ago
  > You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"` Find can even execute commands itself without using `xargs`: find -maxdepth 1 -iregex '.\.$wav\|draft$' -exec echo "found file:" {} \;
  - Izkata 2y ago
    Definitely do it this way if you want to stick to the pre-filtered version (I recommend the cousin comment, filter inside the loop). GP's version is buggy in the same way as the post misunderstands, particularly with files that somehow got newlines in the filename (xargs is newline-delimited by default). If for some reason you do need the "find | xargs" combo (maybe for concurrency), you can get it to work with "find -print0" and "xargs -0". Nulls can't be in filenames so a null-delimited list should work.
    - ykonstant 2y ago
      As an addendum, note that `-print0` and `-0` for find and xargs respectively are now in the latest POSIX standard, so their use is compliant.
      - genrilz 2y ago
        The latest standard I know of is SuS 2018, which I have the docs for, and does not include either switch. I searched around a bit and it doesn't seem like there is a new one. Are you referring to some draft? I sure wish this was true. That being said, I would interpret "-exec printf '%s\0' {} +" as being a posix compliant way for find to output null delimited files. I say this since the docs for the octal escape for printf allows zero digits. However, most posix tools operate on "text" input "files", which are defined as not having null characters. Thus I don't think outputting nulls could be easily used in a posix complaint way. In practice, I would expect many posix implementations to also not handle nulls well because C uses null to mean end of string, so lots of C library calls for dealing with strings will not correctly deal with null characters.
        ykonstant 2y ago
        The 2024 standard is out, but behind a paywall; I don't know when they will update the Open Group website.
        2y ago
- arp242 2y ago
  in zsh you can use: P*~*wav*~*draft* This looks a bit obscure due to lack of spaces, but it's simpler than it seems; the pattern is: [glob] ~ [exclude glob] P* ~ *wav* ~ *draft* The ~ being the negate operator, which can be added more than once. It's essentially the same as "grep -Ev" or "find -iregex". It's a lot less typing than find, and also something you're likely to use interactively once you're used to it, so it feels very natural.
bheadmaster 2y ago
It's not necessary to bring Python into it, Bash can handle filenames with weird characters properly if you know how to use it. E.g. instead of `ls | grep -Ev 'wav|draft'`, you'd have to do something like for filename in *; do if grep -E 'wav|draft' >/dev/null <<< "$filename" then : # ... fi done Of course, it's more convoluted, but when you're writing scripts that might be used for a long time and by many people, it helps to know that it is possible to write robust things. Tools like shellcheck certainly help.
- PeterWhittaker 2y ago
  grep -q and you won’t need the redirect of stdout.
- ykonstant 2y ago
  The above is perfectly fine for small directories, but in general the preferred way to loop over files is with find: find . ! -name . -prune \ -exec grep -qE 'wav|draft' {} \; \ -exec "${action}" \; ; Edit: I missed the herestring in the original code, so the above is wrong as mentioned in the comments; if your find has regex, you can use it to save one grep: find . ! -name . -prune \ -regex '.*wav.*\|.*draft.*' \ -exec "${action}" \; ; Otherwise you can call sh to printf the filename into a grep. However, the point of my post is that find can perform seek, filter and execute, and should be used for all three unless it is really impossible (which is unlikely).
  - tux1968 2y ago
    Your example is grepping the file contents, where GP is using grep to select the filenames.
    - ykonstant 2y ago
      D'oh!
- some_random 2y ago
some_random 2y ago
Before you write anything, you need to think about the cost of it breaking and the chance of it breaking, and Bash scripts in VC tend to maximize both. I like that heuristic a lot.

deleted 2y ago

[deleted]

OrderlyTiamat 2y ago

You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"` This uses regex to match files ending in .wav or .draft (which is what I interpreted you to want). Xargs then processes the file. You could use flags to have xargs pass the file names in a specific place in the command, which can even be a one liner shell call or some script. So the "find <regex> - xarg <command>" pattern is almost fully generally applicable to any problem where you want to execute a oneliner on a number of files with regular names. (I think gnu find has no extended regex, which is just as well- thats not a "regular expression" at that point)

bheadmaster 2y ago
> You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"` Find can even execute commands itself without using `xargs`: find -maxdepth 1 -iregex '.\.$wav\|draft$' -exec echo "found file:" {} \;
- Izkata 2y ago
  Definitely do it this way if you want to stick to the pre-filtered version (I recommend the cousin comment, filter inside the loop). GP's version is buggy in the same way as the post misunderstands, particularly with files that somehow got newlines in the filename (xargs is newline-delimited by default). If for some reason you do need the "find | xargs" combo (maybe for concurrency), you can get it to work with "find -print0" and "xargs -0". Nulls can't be in filenames so a null-delimited list should work.
  - ykonstant 2y ago
    As an addendum, note that `-print0` and `-0` for find and xargs respectively are now in the latest POSIX standard, so their use is compliant.
    - genrilz 2y ago
      The latest standard I know of is SuS 2018, which I have the docs for, and does not include either switch. I searched around a bit and it doesn't seem like there is a new one. Are you referring to some draft? I sure wish this was true. That being said, I would interpret "-exec printf '%s\0' {} +" as being a posix compliant way for find to output null delimited files. I say this since the docs for the octal escape for printf allows zero digits. However, most posix tools operate on "text" input "files", which are defined as not having null characters. Thus I don't think outputting nulls could be easily used in a posix complaint way. In practice, I would expect many posix implementations to also not handle nulls well because C uses null to mean end of string, so lots of C library calls for dealing with strings will not correctly deal with null characters.
      - ykonstant 2y ago
        The 2024 standard is out, but behind a paywall; I don't know when they will update the Open Group website.
        2y ago
arp242 2y ago
in zsh you can use: P*~*wav*~*draft* This looks a bit obscure due to lack of spaces, but it's simpler than it seems; the pattern is: [glob] ~ [exclude glob] P* ~ *wav* ~ *draft* The ~ being the negate operator, which can be added more than once. It's essentially the same as "grep -Ev" or "find -iregex". It's a lot less typing than find, and also something you're likely to use interactively once you're used to it, so it feels very natural.

bheadmaster 2y ago

It's not necessary to bring Python into it, Bash can handle filenames with weird characters properly if you know how to use it. E.g. instead of `ls | grep -Ev 'wav|draft'`, you'd have to do something like for filename in *; do if grep -E 'wav|draft' >/dev/null <<< "$filename" then : # ... fi done Of course, it's more convoluted, but when you're writing scripts that might be used for a long time and by many people, it helps to know that it is possible to write robust things. Tools like shellcheck certainly help.

PeterWhittaker 2y ago
grep -q and you won’t need the redirect of stdout.
ykonstant 2y ago
The above is perfectly fine for small directories, but in general the preferred way to loop over files is with find: find . ! -name . -prune \ -exec grep -qE 'wav|draft' {} \; \ -exec "${action}" \; ; Edit: I missed the herestring in the original code, so the above is wrong as mentioned in the comments; if your find has regex, you can use it to save one grep: find . ! -name . -prune \ -regex '.*wav.*\|.*draft.*' \ -exec "${action}" \; ; Otherwise you can call sh to printf the filename into a grep. However, the point of my post is that find can perform seek, filter and execute, and should be used for all three unless it is really impossible (which is unlikely).
- tux1968 2y ago
  Your example is grepping the file contents, where GP is using grep to select the filenames.
  - ykonstant 2y ago
    D'oh!
some_random 2y ago

some_random 2y ago

Before you write anything, you need to think about the cost of it breaking and the chance of it breaking, and Bash scripts in VC tend to maximize both. I like that heuristic a lot.