4 ms·

It is NOT limited to the external RAM speed and the best proof is that it actually uses over 40GB/s of memory bandwidth. For each byte of data passing through

by qb45 9y ago

It is NOT limited to the external RAM speed and the best proof is that it actually uses over 40GB/s of memory bandwidth. For each byte of data passing through pv: 1. the byte is read from yes memory to CPU registers by the write syscall handler in the kernel 2. the byte is written to kernel's internal buffer associated with the pipe 3. the byte is read back in the read syscall called by pv 4. the byte is written to a buffer in pv memory 5. and thas's the end because write syscall executed by pv on /dev/null very likely doesn't actually bother reading the submitted buffer at all edit: Actually it might only be 20GB/s because on Linux pv seems to use the splice syscall to transfer data from stdin to stdout without copying to userspace. This is also the reason why further "optimization" of this program in assembly was a fool's errand: the bulk of CPU load is in the kernel.

joosters 9y ago
All good valid points, but I'm still a bit surprised that the limit is not higher, I thought that the L2 cache was over an order of magnitude faster than main memory (plus, as someone pointed out in the reddit thread, the peak memory performance should really be double the quoted 12GB/s due to dual channel memory). The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple. (A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)
- maxhou 9y ago
  yes and pv processes are not scheduled on the same CPU core, so different L2 cache.
  - daveguy 9y ago
    I wonder if `taskset -c1 yes | taskset -c1 pv > /dev/null` would significantly change the throughput.
    - joosters 9y ago
      $ yes |pv > /dev/null 46.6GiB 0:00:05 [9.33GiB/s] $ taskset 1 yes |taskset 1 pv > /dev/null 32.9GiB 0:00:05 [6.58GiB/s] $ taskset 1 yes |taskset 2 pv > /dev/null 45.7GiB 0:00:05 [9.13GiB/s] $ taskset 1 yes |taskset 4 pv > /dev/null 45.7GiB 0:00:05 [9.18GiB/s] Very rough numbers - the 9.13/9.33 difference flip-flopped when I ran the commands again. Binding both processes to the same core is definitely a performance hit though. There might be some gain through a shared cache, but it's lost more through lack of parallelism. I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.
      - maxhou 9y ago
        How do you know that the dataset fits in L2 ? Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null. If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.
        joosters 9y ago
        I only learned of pv from this article so I can't speak much about its buffering. I would guess that the kernel would try to re-use recent freed pages to minimise cache thrashing. But anyway, on the 'yes' side, the program isn't re-allocating its 8kb buffer after every write(), so there's a lot of data being re-read from the same memory location.
- pjc50 9y ago
  L2 cache and copying "y" bytes have very little to do with this; I suspect if you could produce high-granularity timings it would almost all be in the syscall overhead. See eg. https://stackoverflow.com/questions/23599074/system-calls-overhead https://stackoverflow.com/questions/23599074/system-calls-ov... who benchmarked it at ~638ns per "read" call. (Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)
  - maxhou 9y ago
    Yes they do. A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot) What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode. You make that overhead negligible by calling read() with a large size.
  - joosters 9y ago
    (Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.) Snap! IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...

by qb45 9y ago

joosters 9y ago

All good valid points, but I'm still a bit surprised that the limit is not higher, I thought that the L2 cache was over an order of magnitude faster than main memory (plus, as someone pointed out in the reddit thread, the peak memory performance should really be double the quoted 12GB/s due to dual channel memory). The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple. (A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)

maxhou 9y ago
yes and pv processes are not scheduled on the same CPU core, so different L2 cache.
- daveguy 9y ago
  I wonder if `taskset -c1 yes | taskset -c1 pv > /dev/null` would significantly change the throughput.
  - joosters 9y ago
    $ yes |pv > /dev/null 46.6GiB 0:00:05 [9.33GiB/s] $ taskset 1 yes |taskset 1 pv > /dev/null 32.9GiB 0:00:05 [6.58GiB/s] $ taskset 1 yes |taskset 2 pv > /dev/null 45.7GiB 0:00:05 [9.13GiB/s] $ taskset 1 yes |taskset 4 pv > /dev/null 45.7GiB 0:00:05 [9.18GiB/s] Very rough numbers - the 9.13/9.33 difference flip-flopped when I ran the commands again. Binding both processes to the same core is definitely a performance hit though. There might be some gain through a shared cache, but it's lost more through lack of parallelism. I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.
    - maxhou 9y ago
      How do you know that the dataset fits in L2 ? Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null. If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.
      - joosters 9y ago
        I only learned of pv from this article so I can't speak much about its buffering. I would guess that the kernel would try to re-use recent freed pages to minimise cache thrashing. But anyway, on the 'yes' side, the program isn't re-allocating its 8kb buffer after every write(), so there's a lot of data being re-read from the same memory location.
pjc50 9y ago
L2 cache and copying "y" bytes have very little to do with this; I suspect if you could produce high-granularity timings it would almost all be in the syscall overhead. See eg. https://stackoverflow.com/questions/23599074/system-calls-overhead https://stackoverflow.com/questions/23599074/system-calls-ov... who benchmarked it at ~638ns per "read" call. (Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)
- maxhou 9y ago
  Yes they do. A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot) What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode. You make that overhead negligible by calling read() with a large size.
- joosters 9y ago
  (Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.) Snap! IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...