4 ms·
I wanted to test this and found a surprising result, although with shuf instead of tail... ( head -n40000 filelist_h.txt | grep 'v\|$' >> /dev/null; )
by PennRobotics 3y ago
I wanted to test this and found a surprising result, although with shuf instead of tail...
( head -n40000 filelist_h.txt | grep 'v\|$' >> /dev/null; ) 0.00s user 0.02s system 43% cpu 0.045 total
( tail -n40000 filelist_h.txt | grep 'v\|$' >> /dev/null; ) 0.00s user 0.01s system 22% cpu 0.066 total
( shuf -n40000 filelist_h.txt | grep 'v\|$' >> /dev/null; ) 0.71s user 1.37s system 12% cpu 16.874 total
( cat filelist_h.txt | head -n40000 | grep 'v\|$' >> /dev/null; 0.00s user 0.01s system 101% cpu 0.017 total
( cat filelist_h.txt | tail -n40000 | grep 'v\|$' >> /dev/null; 0.05s user 0.45s system 54% cpu 0.930 total
( cat filelist_h.txt | shuf -n40000 | grep 'v\|$' >> /dev/null; 0.60s user 0.91s system 96% cpu 1.565 total
The results are fairly repeatable:
- as expected, head and tail alone are equally quick alone
- shuf is ridiculously slow due to overhead compared to cat | shuf
- cat | shuf is just a bit slower than cat | tail.
- cat | tail is slower than tail.
- cat | head is faster than head (but only because of overhead)
Caveats are that this is WSL2 and the file is 480 MB (5 million lines) in a mounted Windows directory, although that helps magnify that slow I/O can influence how you pipe commands.
- karmakaze 3y agoWhy would "shuf -n40000 filelist_h.txt" be slower than "cat filelist_h.txt | shuf -n40000"? Seems like there's a random-access optimization that's severely impacting I/O performance.
- PennRobotics 3y agoI have no earthly idea, but WSL2 is known[1] for its snail-like cross-platform file transfers. Some of my colleagues install MSYS2 in parallel with WSL2 so they can sync our nightly much faster but still not have to jump through all the hoops of getting our proxy, certificates, network shares, etc working in MSYS2. I use the extra time waiting for WSL2 to get a coffee. [1] https://github.com/microsoft/WSL/issues/4197 https://github.com/microsoft/WSL/issues/4197