4 ms·
The majority of this overhead (and the slow transfers) naively seem to be in the scripts/systems using the pipes. I was worried when I saw zfs send/receive use
by lazide 4y ago
The majority of this overhead (and the slow transfers) naively seem to be in the scripts/systems using the pipes.
I was worried when I saw zfs send/receive used pipes for instance because of performance worries - but using it in reality I had no problems pushing 800MB/s+. It seemed limited by iop/s on my local disk arrays, not any limits in pipe performance.
- Matthias247 4y agoRight. I’m actually surprised the test with 256kB transfers gives reasonable results, and would rather have tested with > 1GB instead. For such a small transfer it seemed likely that the overhead of spawning the process and loading libraries by far dominates the amount of actual work. I’m also surprised this didn’t show up in profiles. But if obviously depends on where the measurement start and end points are
- azornathogron 4y agoPerhaps I've misunderstood what you're referring to, but the test in the article is measuring speed transferring 10 GiB. 256 KiB is just the buffer size.
- Matthias247 4y agoThe first C program in the blog post allocates a 256kB buffer and writes that one exactly once to stdout. I don't see another loop which writes it multiple times.
- azornathogron 4y agoThere's an outer while(true){} loop - the write side just writes continuously. More generally though, sidenote 5 says that the code in the article itself is incomplete and the real test code is available in the github repo: https://github.com/bitonic/pipes-speed-test https://github.com/bitonic/pipes-speed-test