3 ms·

Why is find | grep an anti-pattern? Is eschewing memorizing the flags of one particular command in favor of using a pipeline which is much more broadly applica

Why is find | grep an anti-pattern? Is eschewing memorizing the flags of one particular command in favor of using a pipeline which is much more broadly applicable really an anti-pattern? Heck, I’m at the point now where I pipe tar into gzip.

Xophmeister 7y ago
I assume you mean “find | xargs”, rather than “find | grep”; the latter would grep against the paths that find matches, rather than their contents. Anyway, the reason “find | xargs” is considered an antipattern is because xargs splits its input on newlines and this would break on any paths that contained a newline. Arguably, you can get around this by using find’s -print0 and xargs’s -0 arguments, respectively, as paths cannot contain null bytes, but you might as well have used -exec at this point!
jrockway 7y ago
I think there is a cottage industry around telling people that they're using the interactive shell wrong. This all started with "useless use of cat", which people still bitch about on Reddit and Twitter on a regular basis, and so whenever someone finds something a little weird about UNIX, they are quick to call a person asking a question about it dumb. I think it might be stockholm syndrome, or a symptom of one's greatest achievement in life thusfar being reading the man page for find. Or maybe they genuinely think they're helping. I dunno. Ultimately, it's a cascading failure of bad design. Good design is that find and xargs each do one thing and do it well. Find can output files to anything! Xargs can xargify input from anything! Bad design is that find uses \n as the output record separator and that xargs uses \n as the input record separator, but \n can appear in filenames! This design can never work and will always lead to difficult-to-debug problems. So instead of fixing it, we blame the user for not knowing "oh, well find has xargs built in. sort of. a lot of the features are there. not all of them. but some. so never use find | args, smile." or "well, we designed the unix shell to be easy to use... but there is this corner case that we knew about and could have protected you from, but we chose not to, so 1% of the time you'll rm -rf / with your xargs command, UNLESS YOU REMEMBER THIS ONE WEIRD TRICK which is to use -0 to use an out-of-band symbol to separate records which we could have just made happen by default but chose not to, just to mess you up!" I appreciate the efforts of people that are re-imagining the interactive shell, like PowerShell. I am not very productive with PowerShell (to get a listing of files in your directory, just type "Could-We-Make-It-Even-More-Verbose -PerhapsInTheNextVersionWeWill!") but it's a good prototype of a good shell. I have personally stepped on enough UNIX landmines to be generally OK with the compromises, or more typically don't write shell scripts for anything that will be run more than twice. It's bad though, and I don't think the users are to blame.
- thaumasiotes 7y ago
  > Bad design is that find uses \n as the output record separator and that xargs uses \n as the input record separator, but \n can appear in filenames! This design can never work and will always lead to difficult-to-debug problems. So instead of fixing it Perhaps, but it also can't be avoided. Everything can appear in filenames. We want filenames to be able to contain any character. The most high-profile attempt to "fix" the "problem" you identify here is of course the ASCII standard, which defines several separators (0x1C - 0x1F) just for the purpose of delimiting fields with bytes that can't appear in data. Except of course nobody cares -- the solution was stillborn -- because data can contain any bytes that anyone wants it to. > I am not very productive with PowerShell (to get a listing of files in your directory, just type "Could-We-Make-It-Even-More-Verbose -PerhapsInTheNextVersionWeWill!") Well, you could do this by typing Get-ChildItem, but it would be both easier and more standard to type one of "gci" (what Microsoft would like you to use), "ls" (meant for those used to unix) or "dir" (for those used to dos).
  - unilynx 7y ago
    > Perhaps, but it also can't be avoided. Everything can appear in filenames. We want filenames to be able to contain any character. Who is "we" ? I wouldn't mind some 'sane' limits, eg UTF8, no control characters and no names starting with a dash. Still hoping for the day that some distribution sets a mount option to enforce sane filenames by default and starts weeding out the application bugs it'll trigger.. https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html
  - jrockway 7y ago
    There are two options. \0 can't appear in filenames, which is certainly questionable (why is that byte special?), so using \0 as the separator works. A better way of solving the problem is to just prefix the record with a field that represents the length. People are allergic to the length prefix because, say you set it to 2 bytes, now your max file length is 512 characters AND you have two bytes of overhead for every filename. Whereas using \0 means that you have only one byte of overhead and can have infinitely long filenames. Interestingly, I guess nobody has ever died because of a buffer overflow. Therac-25 was integer overflow and bad synchronization (and an open loop control system, which people still love). So I guess it doesn't matter.

Xophmeister 7y ago

I assume you mean “find | xargs”, rather than “find | grep”; the latter would grep against the paths that find matches, rather than their contents. Anyway, the reason “find | xargs” is considered an antipattern is because xargs splits its input on newlines and this would break on any paths that contained a newline. Arguably, you can get around this by using find’s -print0 and xargs’s -0 arguments, respectively, as paths cannot contain null bytes, but you might as well have used -exec at this point!

jrockway 7y ago

I think there is a cottage industry around telling people that they're using the interactive shell wrong. This all started with "useless use of cat", which people still bitch about on Reddit and Twitter on a regular basis, and so whenever someone finds something a little weird about UNIX, they are quick to call a person asking a question about it dumb. I think it might be stockholm syndrome, or a symptom of one's greatest achievement in life thusfar being reading the man page for find. Or maybe they genuinely think they're helping. I dunno. Ultimately, it's a cascading failure of bad design. Good design is that find and xargs each do one thing and do it well. Find can output files to anything! Xargs can xargify input from anything! Bad design is that find uses \n as the output record separator and that xargs uses \n as the input record separator, but \n can appear in filenames! This design can never work and will always lead to difficult-to-debug problems. So instead of fixing it, we blame the user for not knowing "oh, well find has xargs built in. sort of. a lot of the features are there. not all of them. but some. so never use find | args, smile." or "well, we designed the unix shell to be easy to use... but there is this corner case that we knew about and could have protected you from, but we chose not to, so 1% of the time you'll rm -rf / with your xargs command, UNLESS YOU REMEMBER THIS ONE WEIRD TRICK which is to use -0 to use an out-of-band symbol to separate records which we could have just made happen by default but chose not to, just to mess you up!" I appreciate the efforts of people that are re-imagining the interactive shell, like PowerShell. I am not very productive with PowerShell (to get a listing of files in your directory, just type "Could-We-Make-It-Even-More-Verbose -PerhapsInTheNextVersionWeWill!") but it's a good prototype of a good shell. I have personally stepped on enough UNIX landmines to be generally OK with the compromises, or more typically don't write shell scripts for anything that will be run more than twice. It's bad though, and I don't think the users are to blame.

thaumasiotes 7y ago
> Bad design is that find uses \n as the output record separator and that xargs uses \n as the input record separator, but \n can appear in filenames! This design can never work and will always lead to difficult-to-debug problems. So instead of fixing it Perhaps, but it also can't be avoided. Everything can appear in filenames. We want filenames to be able to contain any character. The most high-profile attempt to "fix" the "problem" you identify here is of course the ASCII standard, which defines several separators (0x1C - 0x1F) just for the purpose of delimiting fields with bytes that can't appear in data. Except of course nobody cares -- the solution was stillborn -- because data can contain any bytes that anyone wants it to. > I am not very productive with PowerShell (to get a listing of files in your directory, just type "Could-We-Make-It-Even-More-Verbose -PerhapsInTheNextVersionWeWill!") Well, you could do this by typing Get-ChildItem, but it would be both easier and more standard to type one of "gci" (what Microsoft would like you to use), "ls" (meant for those used to unix) or "dir" (for those used to dos).
- unilynx 7y ago
  > Perhaps, but it also can't be avoided. Everything can appear in filenames. We want filenames to be able to contain any character. Who is "we" ? I wouldn't mind some 'sane' limits, eg UTF8, no control characters and no names starting with a dash. Still hoping for the day that some distribution sets a mount option to enforce sane filenames by default and starts weeding out the application bugs it'll trigger.. https://dwheeler.com/essays/fixing-unix-linux-filenames.html https://dwheeler.com/essays/fixing-unix-linux-filenames.html
- jrockway 7y ago
  There are two options. \0 can't appear in filenames, which is certainly questionable (why is that byte special?), so using \0 as the separator works. A better way of solving the problem is to just prefix the record with a field that represents the length. People are allergic to the length prefix because, say you set it to 2 bytes, now your max file length is 512 characters AND you have two bytes of overhead for every filename. Whereas using \0 means that you have only one byte of overhead and can have infinitely long filenames. Interestingly, I guess nobody has ever died because of a buffer overflow. Therac-25 was integer overflow and bad synchronization (and an open loop control system, which people still love). So I guess it doesn't matter.