5 ms·
> But who cares? Why not just let the command figure out the right buffer size automatically? Well, because it doesn't. At least on the Linux version I used in
by _abox 4y ago
> But who cares? Why not just let the command figure out the right buffer size automatically?
Well, because it doesn't. At least on the Linux version I used in the past, it defaulted to 512 byte blocks or something similarly small and it commits every block individually, leading to really slow performance with USB sticks with controllers not smart enough to bundle writes together. I wouldn't be surprised if that incurs some heavy write amplification at flash level too. Perhaps it's smart enough to figure a better block size now but this is where that habit comes from.
Another thing, creating sparse files with the seek option (simply put files containing zeros that are not actually written to the disk nor taking space, but do turn up all these zeroes when you read the file). Also something not duplicated with cat or head.
What I like about dd is that it can do pretty much all disk operations in one simple tool. Definitely worth learning all of it IMO.
- tinus_hn 4y agoAnother one is the conv=noerror read and fill bad blocks with zeroes option.
- smegsicle 4y ago>> But who cares? Why not just let the command figure out the right buffer size automatically? i think he's talking about cat being the command that figures it out automatically
- GekkePrutser 4y agoI don't read it that way. He calls out that 'weird' command specifically. But indeed he doesn't specify. I wonder what cat does in terms of buffers, I kinda doubt it has any special optimisations though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. Edit: Nope, I'm wrong there!! Also that command does more than just specify the memory buffer like he says. That's my point, it's useful for tuning which can be super helpful with huge images. It can also lead to some dangerous gotchas as well when working with files. But with full disk images these don't apply generally.
- LukeShu 4y ago> though I would guess the shell redirect might have. As that's really the thing doing the work there, not cat. No? The shell redirection is just int tmp_fd = open("/dev/sdb", O_WRONLY|O_TRUNC, 0666); dup2(tmp_fd, STDOUT_FILENO); close(tmp_fd); (plus error handling and whatnot) `cat` ends up with a file descriptor directly to the block device, same as `dd` does; the only difference is whether the `open()` call comes before or after the `execve()` call.
- GekkePrutser 4y agoAh ok good point! I stand corrected. I really doubt cat is smart enough to figure out a suitable block size though. At least with dd you can specify one.
- bhaney 4y ago> I really doubt cat is smart enough to figure out a suitable block size though It seems to try, at least https://github.com/coreutils/coreutils/blob/master/src/cat.c#L648-L649 https://github.com/coreutils/coreutils/blob/master/src/cat.c...
- LukeShu 4y agoGoing through the historical versions (copying from my other comment): - >=9.1 (2022-04-15) : Use the `copy_file_range()` syscall and let the kernel figure it out - >=8.23 (2014-07-18) : max(128KiB, st_blksize(infile), st_blksize(outfile)) - >=8.17 (2012-05-10) : max(64KiB, st_blksize(infile), st_blksize(outfile)) - >=7.2 (2009-03-31) : max(32KiB, st_blksize(infile), st_blksize(outfile)) - at least as far back as 1996 : max(st_blksize(infile), st_blksize(outfile)) (In my psuedo-code, `st_blksize(fd)` is the `ST_BLKSIZE(buf)` of the result of `fstat(fd, buf)`.)
- andrewflnr 4y agoI can confirm I've seen dramatic speed differences with block sizes in dd. I haven't tried comparing with cat, though.
- jrumbut 4y agoI don't have data at hand but if you choose the right value it's meaningfully better and can also lead to more efficient patterns in bash scripts that are more complicated than `cat in > out` Unfortunately dd has not just footguns but foot cannons that are amplified by the mistakes people often make with string escaping, odd file names, loops, null checking, and conditionals in bash.
- jcynix 4y ago> commits every block individually It uses a syscall per block write (which would slow it down if you use 512 byte blocks instead of 8M for example), but the OS does the file buffering and final writing to the device, unless you pass the fsync or fdatasync options to dd. Edit: here's writing to a old and slow stick and you can see that dd is fast and then the OS has to transfer the buffered data onto the stick: dd if=some-2GB-data status=progress of=/dev/sdX bs=8M 1801256960 bytes (1,8 GB, 1,7 GiB) copied, 7 s, 257 MB/s 0+61631 records in 0+61631 records out 2019556352 bytes (2,0 GB, 1,9 GiB) copied, 378,59 s, 5,3 MB/s And the stick is placed in a USB 3.2. port on a fast machine ;-0
- noir_lord 4y agoCan also do `oflag=direct` and it'll just skip as much of the caching as it can.
- jcynix 4y ago> Can also do `oflag=direct` and it'll just skip as much of the caching as it can. Correct, and that's one more point for dd compared to head/tail (which are fine commands by themselves). But wouldn't help much in my example, where I used an (very) old "high speed usb 2.0 stick" with 4 MByte/s write speed to demonstrate the difference between buffering and actual writing.
- ltbarcly3 4y agoThats when it matters, with cat it would cache the entire write. If you don't want it to cache everything and have no idea how much time is left or how fast it is writing, and then wait an hour for it to unmount instead then cat is fine.
- tomrod 4y agoLiterally my thoughts too! "Who cares?" --> people that care are people that want to have control over that option, as not every tool is written intelligently.
- guenthert 4y agoIndeed. dd can also serve also as an ad-hoc performance test tool (it gets a bit tricky when you need multiple threads/streams though).