3 ms·

> commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. > I don't want to commit something half-generated. Maybe comm

> commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. > I don't want to commit something half-generated. Maybe commit_to_database replaces the current set of data with the updated version, serving bad data to users. Thanks, it's clear now. I (and I assume others recommending pipefail) was puzzled because idiomatic way to do your sorting example in a regular filesystem is: set -o pipefail whatever | false | tee output.tmp if [ $? = 0 ]; then echo Commit mv output.tmp output.sorted else echo Abort rm output.tmp exit 1 fi ...which gsutil command can't because it does the commit step implicitly, with no way to do it explicitly, I gather? But at that point I still don't understand how can you avoid the race condition (regardless of the language) and kill gsutil before it does the implicit commit step. > that kills the parent process, not the downstream process It kills all child processes, including commands started in a pipe.

thomashabets2 4y ago
Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. Sometimes the intermediate result doesn't even fit on disk.
- kasabali 4y ago
  > Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. It was an example, not a recipe. And why wouldn't it be a pipe? It is a pipe, with an explicit commit step. That's what you asked. It isn't an "intermediate" file, it's a journal/log file before the commit, if you will. While I've never used gsutil, I guess a similar construct would be "gsutil cp - gs://example/test.data.sorted.tmp" followed by "gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted". Updated example: set -o pipefail gsutil cat gs://example/test.data \ | sort -S300M \ | gsutil cp - gs://example/test.data.sorted.tmp if [ $? = 0 ]; then echo Commit gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted else echo Abort gsutil rm gs://example/test.data.sorted.tmp exit 1 fi Yes, I know it doesn't abort it early. I'm sure it can be done, I'm not sure it can be done neatly (maybe with coproc?). And what I still don't understand about your example is how solely depending on killing the gsutil cp command early enough isn't racy. There's gotta be an explicit step, no way around that.

thomashabets2 4y ago

Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. Sometimes the intermediate result doesn't even fit on disk.

kasabali 4y ago
> Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. It was an example, not a recipe. And why wouldn't it be a pipe? It is a pipe, with an explicit commit step. That's what you asked. It isn't an "intermediate" file, it's a journal/log file before the commit, if you will. While I've never used gsutil, I guess a similar construct would be "gsutil cp - gs://example/test.data.sorted.tmp" followed by "gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted". Updated example: set -o pipefail gsutil cat gs://example/test.data \ | sort -S300M \ | gsutil cp - gs://example/test.data.sorted.tmp if [ $? = 0 ]; then echo Commit gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted else echo Abort gsutil rm gs://example/test.data.sorted.tmp exit 1 fi Yes, I know it doesn't abort it early. I'm sure it can be done, I'm not sure it can be done neatly (maybe with coproc?). And what I still don't understand about your example is how solely depending on killing the gsutil cp command early enough isn't racy. There's gotta be an explicit step, no way around that.