4 ms·

You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.

• AstralStorm 9y ago
  Still gets piped and hits same performance.
  • mkj 9y ago
    It's 2x times faster for a quick test here. Only copying on the read side?
    • readittwice 9y ago
      Out of interest: Could you please post the code?
      • fredmorcos 9y ago
        Don't think it's possible. The test was so quick it happened in his mind, not as actual proof on his computer.
      • mkj 9y ago
        https://pastebin.com/jrcJbjU4 https://pastebin.com/jrcJbjU4 I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said. core2duo e8400, 4.8 kernel, coreutils 8.25 > yes | dd bs=1M count=10000 of=/dev/null iflag=fullblock 10485760000 bytes (10 GB, 9.8 GiB) copied, 4.1901 s, 2.5 GB/s > ./zcyes | dd bs=1M count=10000 of=/dev/null iflag=fullblock 10485760000 bytes (10 GB, 9.8 GiB) copied, 1.8542 s, 5.7 GB/s
• mrb 9y ago
  Actually yes should use vmsplice(). And pv on the other side of the pipe should use splice(). Now that would be a complete zero-copy I/O path, purely limited by the CPU, not by memory bandwidth. It would benchmark at hundreds of GB/s :)
  • qb45 9y ago
    > And pv on the other side of the pipe should use splice(). Good catch, I just noticed that it does.
  • gpderetta 9y ago
    Someone posted an implementation which uses vmsplice on the reddit thread: https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/diu73y0/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
    • mrb 9y ago
      His code can further be optimized by having only 1 large element in the iovec (vs. many 2-byte "y\n" elements).
      • nyanscat 9y ago
        from what i can see it's already in a large iovec buffer
        • mrb 9y ago
          Correct, I misread the code. However bumping the pipe size buffer bumps the speed from 28 to 74 GB/s on my Skylake CPU (i5-6500). Edit: I optimized further and now get 123 GB/s See https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes_so_fast/diua761/ https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
          • gpderetta 9y ago
            next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).
• masklinn 9y ago
  On the other hand the current code is perfectly portable, tee(2) is a linux syscall.