4 ms·
And that's why sane people will tell you to not use something like st_blksize as the IO size, because that's far to small. Many tools use 32 KiB, but that can b
by dom0 10y ago
And that's why sane people will tell you to not use something like st_blksize as the IO size, because that's far to small. Many tools use 32 KiB, but that can be limiting even on Linux and needlessly uses more CPU (especially if FUSE is involved -- reads are not coalesced at the VFS layer! Read and write merging is done at the block device layer by the IO scheduler there!). Something like 512k-1M is a sane default these days.
- gens 10y agoMaybe i'm misunderstanding you, but.. Before every one a "echo 3 > /proc/sys/vm/drop_caches" is executed. Processor is in "performance" mode. dd if=./big_file.file of=/dev/null bs=64K 535020540 bytes (535 MB, 510 MiB) copied, 8.70128 s, 61.5 MB/s dd if=./big_file.file of=/dev/null bs=4K 535020540 bytes (535 MB, 510 MiB) copied, 8.36283 s, 64.0 MB/s dd if=./big_file.file of=/dev/null bs=1K 535020540 bytes (535 MB, 510 MiB) copied, 8.82508 s, 60.6 MB/s Only thing gone up is the cpu usage (same is negligible at ~60MB/s). dd if=/dev/zero of=/dev/null bs=1K count=1024K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.06113 s, 1.0 GB/s dd if=/dev/zero of=/dev/null bs=4K count=256K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.343595 s, 3.1 GB/s dd if=/dev/zero of=/dev/null bs=32K count=32K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.160191 s, 6.7 GB/s
- brobinson 10y agoDon't forget "oflag=direct"
- dom0 10y agoUhm, does dd contain all the logic to work with that or is that literally just O_DIRECT |ed into the flags?
- dom0 10y agoAt just 60 MB/s it doesn't matter much (even 512 byte blocks would just be a little more than 100k syscalls / second). At a couple hundred MB/s it begins to matter, and with faster NVMe SSDs it's a legit issue. This becomes a bigger issue if it's not a kernel-based, local FS, but something that does more work, I mentioned FUSE above. Since no read merging happens, small reads like 4k or 32k start to hurt earlier, especially if the FS is written in an interpreter. Some of these also mishandled small reads, incurring an additional penalty (I know because I fixed that in an implementation).
- gens 10y agoNo point in speculating. I showed you how. Go test it. Even the fastest SSD is not even near the speed of /dev/zero. Easy to extrapolate any speed from those two points (minus the filesystem overhead (the thing we are talking about is the block layer)). I'd be interested in a test of a FUSE based fs like ntfs. So if you have one, just run some tests. (don't forget to flush the cache)
- cbsmith 10y agoThe whole thing is kind of odd. Normally I'd want to leave filling the file to filesystem...