3 ms·

My biggest gripe with UNIX is that a command like: (sleep 5 && echo hello) > test.txt Causes a race condition where test.txt is created empty, then 5 seconds

My biggest gripe with UNIX is that a command like: (sleep 5 && echo hello) > test.txt Causes a race condition where test.txt is created empty, then 5 seconds later written with "hello". Try it without parentheses to see it wait. UNIX doesn't use our intuitive notion of order of operations on its piping, it just runs everything simultaneously. This allowed for tremendous efficiency and concurrency but it's hard to fathom how much this has cost us in bugs and lost development time. I feel like this was a lost opportunity because it prevented the Actor model (as seen in Erlang and Go) from really taking off decades ago. Perhaps this bug/feature was one of the motivations for commands like "parallel". Does anyone have a general workaround for this problem? Some command that we could insert in the chain to force a wait, without having to install any external tools? Thanx! Edit: I'm having a hard time explaining how insidious this race condition is for people who haven't encountered it yet. The gist of it is that file descriptors aren't opened when a pipe sends its first byte, they are opened when the shell command is interpreted. I'm also having a hard time finding examples, here is one I think, though there are many, many others: https://unix.stackexchange.com/questions/174788/am-i-hitting-a-race-condition-in-bash https://unix.stackexchange.com/questions/174788/am-i-hitting...

adamkruszewski 9y ago
I think it does, try it as: sleep 5 && (echo hello > test.txt) and you'll see it creates the file only after sleep ends.
chubot 9y ago
I don't understand what you're trying to say. I would expect these two commands to behave differently: (sleep 5 && echo hello) > test.txt sleep 5 && echo hello > test.txt "our intuitive notion of order of operations on its piping" -- I don't know what your intuitive notion is, and there is no piping in your example. Piping would be: sleep 5 | echo hello (this is an odd example, but it works) I also don't see any race condition in your example. A race would be two processes accessing the same data. They are redirects, and the way redirects like > and < work is to: 1. open a file 2. change the global process state 3. run arbitary commands, either builtin (echo) or external (ls). External commands require a new process and inherit their process state (descriptor table, where stdout and stdin are conected to) from the parent shell. 4. restore the global process state 5. close the file If you understand that, it might be clear why the file test.txt is opened before sleep. There's really no other way it can work, given the fact that you can execute arbitrary commands inside the redirect. I address some related things about redirects here: "Avoid Directly Manipulating File Descriptors in Shell" http://www.oilshell.org/blog/2017/08/12.html http://www.oilshell.org/blog/2017/08/12.html ----- The more unintuitive thing about shell is that what you wrote is technically better as: { sleep 5 && echo hello; } > test.txt () doesn't mean "grouping" like in other languages, it means subshell. {} means grouping, and it has an odd way of parsing, where ; is required before }, but it isn't required before ). This is because () are operators and {} aren't. I wrote about this here, "the subshell construct () is confused with the grouping construct { }": http://www.oilshell.org/blog/2017/02/06.html http://www.oilshell.org/blog/2017/02/06.html ----- EDIT: I will say that I was very confused by the syntax of > and < when I started learning shell. It does seem like ">" means "this goes into that". In fact I think I internalized that from MS-DOS. But that doesn't explain what these mean, and why they are equivalent: tac >out.txt <in.txt tac <in.txt >out.txt They are best read as prefix operators that modify the state of the process invocation. What's even more confusing is that you can do: tac 1>out.txt 0<in.txt tac 0<in.txt 1>out.txt Then they look like binary operators again. But the descriptor is "stuck on", it's actually part of the operator lexically. Valid: tac 1>out.txt 0< in.txt tac 1> out.txt 0< in.txt Invalid: tac 1 >out.txt 0 <in.txt tac 1 > out.txt 0 < in.txt In other words, there can't be any space between the LHS file descriptor and the > or <. The syntax of shell is horrible, and that's something I would like to fix with Oil shell. But I don't see the problem you are pointing out.
debacle 9y ago
(sleep 5 && echo hello) | > test.txt?
SamJson 9y ago
Workaround: (rm file; tac > file) < file

adamkruszewski 9y ago

I think it does, try it as: sleep 5 && (echo hello > test.txt) and you'll see it creates the file only after sleep ends.

chubot 9y ago

I don't understand what you're trying to say. I would expect these two commands to behave differently: (sleep 5 && echo hello) > test.txt sleep 5 && echo hello > test.txt "our intuitive notion of order of operations on its piping" -- I don't know what your intuitive notion is, and there is no piping in your example. Piping would be: sleep 5 | echo hello (this is an odd example, but it works) I also don't see any race condition in your example. A race would be two processes accessing the same data. They are redirects, and the way redirects like > and < work is to: 1. open a file 2. change the global process state 3. run arbitary commands, either builtin (echo) or external (ls). External commands require a new process and inherit their process state (descriptor table, where stdout and stdin are conected to) from the parent shell. 4. restore the global process state 5. close the file If you understand that, it might be clear why the file test.txt is opened before sleep. There's really no other way it can work, given the fact that you can execute arbitrary commands inside the redirect. I address some related things about redirects here: "Avoid Directly Manipulating File Descriptors in Shell" http://www.oilshell.org/blog/2017/08/12.html http://www.oilshell.org/blog/2017/08/12.html ----- The more unintuitive thing about shell is that what you wrote is technically better as: { sleep 5 && echo hello; } > test.txt () doesn't mean "grouping" like in other languages, it means subshell. {} means grouping, and it has an odd way of parsing, where ; is required before }, but it isn't required before ). This is because () are operators and {} aren't. I wrote about this here, "the subshell construct () is confused with the grouping construct { }": http://www.oilshell.org/blog/2017/02/06.html http://www.oilshell.org/blog/2017/02/06.html ----- EDIT: I will say that I was very confused by the syntax of > and < when I started learning shell. It does seem like ">" means "this goes into that". In fact I think I internalized that from MS-DOS. But that doesn't explain what these mean, and why they are equivalent: tac >out.txt <in.txt tac <in.txt >out.txt They are best read as prefix operators that modify the state of the process invocation. What's even more confusing is that you can do: tac 1>out.txt 0<in.txt tac 0<in.txt 1>out.txt Then they look like binary operators again. But the descriptor is "stuck on", it's actually part of the operator lexically. Valid: tac 1>out.txt 0< in.txt tac 1> out.txt 0< in.txt Invalid: tac 1 >out.txt 0 <in.txt tac 1 > out.txt 0 < in.txt In other words, there can't be any space between the LHS file descriptor and the > or <. The syntax of shell is horrible, and that's something I would like to fix with Oil shell. But I don't see the problem you are pointing out.

debacle 9y ago

(sleep 5 && echo hello) | > test.txt?

SamJson 9y ago

Workaround: (rm file; tac > file) < file