8 ms·
What happens when you grep the file you've redirected grep to?
- repsilat 9y agoA really fun one is having `bash` execute its output using a FIFO. It worked when I tried it last, but it'd obviously be more than crazy to rely on it (for more than one reason.)
- 13of40 9y agoReminds me of this old chestnut: type foo.bat >> foo.bat
- userbinator 9y agoOr for even more fun, self-modifying shell scripts: https://stackoverflow.com/questions/3398258/edit-shell-script-while-its-running https://stackoverflow.com/questions/3398258/edit-shell-scrip... (curiously, the accepted answer is incorrect and downvoted highly) ...and Windows batch files: https://stackoverflow.com/questions/906586/changing-a-batch-file-when-its-running https://stackoverflow.com/questions/906586/changing-a-batch-...
- jwbensley 9y agoSelf modifying code is wandering into the (very interesting IMO) realms of metamorphic and polymorphic code. I started an SO questionable here: https://stackoverflow.com/questions/10113254/metamorphic-code-examples https://stackoverflow.com/questions/10113254/metamorphic-cod... I'll try to find some time soon to port that C+inline assembly example to Linux, I didn't know much C or ASM back when I originally asked the question.
- draw_down 9y agoCan’t really imagine why I would do that though...
- ScottBurson 9y agoA variation you can run into with GNU grep: grep -r [something] . >foo I've done this by accident once or twice :-)
- AnimalMuppet 9y agoThe way I've done it: grep -r [something] * > foo What I now do instead: grep -r [something] * > .foo mv .foo foo Why it works: At least on bash, "*" does not match hidden files (files that begin with "." such as ".foo").
- asicsp 9y agogood one.. when I checked on Linux, both `cat` and `grep` give error when input file name is same as output... but not `sed/awk/head/tail/sort/etc`..
- goialoq 9y agoIf you redirect output, how can the command know the name of the destination?
- aaronmdjones 9y agofstat(2) on the output file descriptor, stat(2) on the input file name, see if they have the same block device and inode numbers.
- xxs 9y agoRedirecting (in simple terms) is setting a file descriptor. So the file descriptor can be tested if it partakes in the input. A quote for you [0]: Under normal circumstances every UNIX program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user's terminal (see tty(4) but might instead refer to files or other devices, depending on what the parent process chose to set up. (See also the "Redirection" section of sh(1).) [0]: https://linux.die.net/man/3/stdout https://linux.die.net/man/3/stdout
- Karliss 9y agoSomewhat more common use of this is to check if output goes to terminal so that terminal color codes don't get printed to files.
- rocqua 9y agoWhich is really annoying when you want to pipe grep to less, and need to add --color=always to get grep to understand less takes color codes (when using less -R).
- raimue 9y ago
- dima55 9y ago"apt install moreutils". Then look up the "sponge" tool
- NiceGuy_Ty 9y agoI did this before when using ripgrep, and accidentally created a several hundred GB file. Now ripgrep checks if this is the case and handles it gracefully: https://github.com/BurntSushi/ripgrep/pull/310 https://github.com/BurntSushi/ripgrep/pull/310
- userbinator 9y agowhich means that when we do the write in that for-loop, we’re guaranteed that the read which happens after in the while-loop must see the result of that write. So: we read an “a”, write an “a”, and when we get back to the while-loop we’re guaranteed that the next read will see that “a” we just wrote. In other words, it's behaving like a FIFO that persists a history of everything that's been written to it, which also suggests that there are two pointers, one for the read position and another for the write position; all becomes clear when we realise that it's not reading and writing with the same file descriptor, but rather two file descriptors of one file, opened twice. The tl;dr of why small sizes terminate and large sizes loop infinitely is this: when the size of data output is not enough to fill the write buffer, the read can reach the end of file (0 bytes read), and then it flushes the write buffer before terminating; but when the output size fills the write buffer, it gets flushed and ends up back in the input, causing the infinite loop. The C stdio buffering layer doesn't guarantee that data written with a fwrite() will show up in a fread() from a different file handle that references the same file (and now that I think about it, that would seem to be quite difficult to guarantee.)
- tankenmate 9y agoIt's not overly difficult to guarantee, the UNIX/Linux kernel does it just fine. It does require more work in that the libc file cache will need to invalidate itself if the source file / socket changes state (either via atime if the filesystem supports it or via inotify/fnotify). Obviously invalidating the cache will lead to lower performance but more correct behaviour.
- starpilot 9y agoSome things are not meant to be questioned.
- chickenthief 9y agoAgreed, this analysis helps nobody.
- vijaybritto 9y agoBrilliant write up. Was very easy to grasp!
- JepZ 9y agoReminds me of how easy it is to destroy a file with sed: sed 's/a/b/' test.txt > test.txt After that command the file is empty no matter what was in there before. Instead you have to use the -i flag like: sed 's/a/b/' -i test.txt And different operating systems seem to behave differently: https://stackoverflow.com/questions/5171901/sed-command-find-and-replace-in-file-and-overwrite-file-doesnt-work-it-empties https://stackoverflow.com/questions/5171901/sed-command-find...
- goialoq 9y agonoclobber https://en.wikipedia.org/wiki/Clobbering https://en.wikipedia.org/wiki/Clobbering
- jle17 9y agoMaybe this is just me but when I enable that king of options I end up getting the automatism of bypassing them (like typing rm -f everytime if rm has been aliased to rm -i). I prefer having a way of reverting changes rather than something that gets in the way.
- Ao7bei3s 9y agosed -i also has the less obvious advantage that it is atomic. (It writes to a temporary file, and then atomically renames it to the final location. Whereas if sed expr src > dst is killed, dst may be corrupt. You'd need to handle temporary file management yourself if your script needs to be reliable.)
- contingencies 9y agoMy memory is to sed -i -e 's/pattern/replacement/' file, the addition of -e was perhaps subconsciously to help protect against this eventuality by mentally segregating my standard pipeline invocation which takes neither argument. Anyway sed is so fast you should definitely check the output before overwriting stuff. Same goes for anything on the command line. Get ZFS. Do snapshots.
- ramshorns 9y ago
- stormbrew 9y agoThe section about the guarantees of read/write doesn't seem entirely correct to me. For sure it's relevant to why the data is there, but the reason it doesn't terminate is just because read doesn't tell you there's an eof until you try to read again past the end. It would be entirely possible to construct a version of this loop that would terminate. Though it would be awkward.
- IgorPartola 9y agoStupid question: in the cases where the input is a regular file, why doesn’t grep check the size of the file at the beginning? Is t actually desired behavior to search through stuff added to the file while we are processing the beginning of it?
- deleted 9y ago[deleted]
- tinus_hn 9y agoThe rules are that you open the file for reading until you get end-of-file. If you don’t follow the rules unexpected things go wrong. For instance it may not even be possible to get the file size. That is not suitable for a program like grep.
- rocqua 9y agoIn Unix we have the idiom of "Everything Is A File". As such, when grep gets a file descriptor, it cannot always get the file-size, the file-size might be infinite, or might vary over time. Instead, like other have said, the only way to know you've read 'until the end' is to read reports you've reached the end of the file. Consider for example what should happen when grepping /dev/null/? Or, a more sensible case, piping the output of some command to grep. Grep will read from 'standard in' which is "Just A File" so it just calls read until it reports end of file.
- jameshart 9y agoBut the article shows GNU grep actually looks ‘behind the curtain’ and identifies that input and output are the same file, then refuses to play. So everything is a file, but some are more filey than others.
- IgorPartola 9y ago> Consider for example what should happen when grepping /dev/null/ In the solution I gave, same thing as before because I said regular file. That is if you can open /dev/null for reading, which I thought you couldn't. I think I'd be fine with saying that the infinite loop is the behavior that happens. GNU's grep obviously is doing some janky check which it shouldn't be. Consider what it will do when you get just slightly more clever and it can't determine the input/output types. The one on macOS has arbitrary file size-dependent behavior, which is problematic. Doesn't seem like either of them does a consistent thing.
- raldi 9y ago> And about 15 up-arrow+enters later Gonna blow your mind here: Press up once and then Ctrl-O.
- ourmandave 9y agoUgh. Flash backs of my older brother's torment when he wanted to use the computer and I wouldn't get out of the way fast enough. "Quit grepping yourself! Quit grepping yourself!" #IKnowYourAreButWhoAmI?
- taneq 9y agoAh, the oldschool version of "quit googling yourself."
- deleted 9y ago[deleted]
- emilfihlman 9y ago>echo "a" > test.txt test.txt will contain "a\n" not just "a" -n to disable adding \n
- davebock 9y agoMALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH! MALCOVICH!
- JdeBP 9y agoI did enjoy how about halfway down the author bemoaned the fact that the C language does not have line numbers. (-:
- jwilk 9y agoIt was phrased poorly, but I think they meant the Apple's source browser doesn't show line numbers.