3 ms·

Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. Sometimes the intermediate result doesn't even fit on disk.

• kasabali 4y ago
  > Yes, you can dump all intermediate results into a file. But then it's no longer a pipeline. It was an example, not a recipe. And why wouldn't it be a pipe? It is a pipe, with an explicit commit step. That's what you asked. It isn't an "intermediate" file, it's a journal/log file before the commit, if you will. While I've never used gsutil, I guess a similar construct would be "gsutil cp - gs://example/test.data.sorted.tmp" followed by "gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted". Updated example: set -o pipefail gsutil cat gs://example/test.data \ | sort -S300M \ | gsutil cp - gs://example/test.data.sorted.tmp if [ $? = 0 ]; then echo Commit gsutil mv gs://example/test.data.sorted.tmp gs://example/test.data.sorted else echo Abort gsutil rm gs://example/test.data.sorted.tmp exit 1 fi Yes, I know it doesn't abort it early. I'm sure it can be done, I'm not sure it can be done neatly (maybe with coproc?). And what I still don't understand about your example is how solely depending on killing the gsutil cp command early enough isn't racy. There's gotta be an explicit step, no way around that.