4 ms·

Except that if you read the problem statement, you'll see that tee doesn't meet the requirement of 'taking advantage of all available I/O parallelism'. Tee writ

Except that if you read the problem statement, you'll see that tee doesn't meet the requirement of 'taking advantage of all available I/O parallelism'. Tee writes to each output file sequentially. Given that their core business is data-stores that take full advantage of emerging storage technologies (such as flash), this seems like pretty relevant question.

deleted 15y ago
[deleted]
joshcorbin 15y ago
Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".
a3_nm 15y ago
I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.
- dexen 15y ago
  Tee writes in one-thread-per-file level of parallelism; simple experiment: $ tee foo.bin bar.bin baz.bin < /dev/zero writes to all three files `at once' (in small time-slices, actually). Used GNU tee from coreutils-8.10. Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?
  - deleted 15y ago
    [deleted]
  - mvgoogler 15y ago
    tee has a simple main loop: while not EOF: (synchrounously) read up to BUFSIZE bytes from stdin into buf for each output file: (synchrounously) write buf to file (source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee.c http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....) All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.
  - mvgoogler 15y ago
    tee has a simple main loop: while not EOF: (synchrounously) read up to BUFSIZE bytes from stdin into buf for each output file: (synchrounously) write buf to file (source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee.... http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....) All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.

deleted 15y ago

[deleted]

joshcorbin 15y ago

Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".

a3_nm 15y ago

I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.

dexen 15y ago
Tee writes in one-thread-per-file level of parallelism; simple experiment: $ tee foo.bin bar.bin baz.bin < /dev/zero writes to all three files `at once' (in small time-slices, actually). Used GNU tee from coreutils-8.10. Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?
- deleted 15y ago
  [deleted]
- mvgoogler 15y ago
  tee has a simple main loop: while not EOF: (synchrounously) read up to BUFSIZE bytes from stdin into buf for each output file: (synchrounously) write buf to file (source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee.c http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....) All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.
- mvgoogler 15y ago
  tee has a simple main loop: while not EOF: (synchrounously) read up to BUFSIZE bytes from stdin into buf for each output file: (synchrounously) write buf to file (source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee.... http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....) All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.