4 ms·
"This is a strange program of obscure provenance that somehow, still manages to survive in the 21st century." -> links to wikipedia page with direct discriptio
by cat199 10y ago
"This is a strange program of obscure provenance that somehow, still manages to survive in the 21st century."
-> links to wikipedia page with direct discription of lineage back to 5th ed research unix
"That weird bs=4M argument in the dd version isn’t actually doing anything special—all it’s doing is instructing the dd command to use a 4 MB buffer size while copying. But who cares? Why not just let the command figure out the right buffer size automatically?"
Um -
a) it is 'doing the special thing' of changing the block size (not buffer size)
b) Because the command probably doesn't figure out the right size automatically, much like your 'cat' example above which also doesn't
c) And this can mean massive performance differences between invocations
> Another reason to prefer the cat variant is that it lets you actually string together a normal shell pipeline. For instance, if you want progress information with cat you can combine it with the pv command
Umm:
dd if=file bs=some-optimal-block-size | rest-of-pipeline
that was hard.
>If you want to create a file of a certain size, you can do so using other standard programs like head. For instance, here are two ways to create a 100 MB file containing all zeroes:
$ uname -sr
OpenBSD 6.0
$ head -c 10MB /dev/zero
head: unknown option -- c
usage: head [-count | -n count] [file ...]
well.. guess that wasn't so 'standard' after all..
I must be using some nonstandard version...
$ man head |sed -ne 47,51p
HISTORY
The head utility first appeared in 1BSD.
AUTHORS
Bill Joy, August 24, 1977.
$ sed -ne 4p /usr/src/usr.bin/head/head.c
* Copyright (c) 1980, 1987 Regents of the University of California.
Hmm..
> So if you find yourself doing that a lot, I won’t blame you for reaching for dd. But otherwise, try to stick to more standard Unix tools.
Like 'pv'?
edit: added formatting, sector size note, head manpage/head.c stuffs.. apologies.
- hisham_hm 10y agoThank you for saving me the effort. I cringe upon seeing people throwing the word "standard" like that.
- miles 10y ago>> if you want progress information with cat you can combine it with the pv command Since coreutils-8.24[1], dd "accepts a new status=progress level to print data transfer statistics on stderr approximately every second." > Because the command probably doesn't figure out the right size automatically . . . this can mean massive performance differences between invocations For anyone who's wondering, here are two good threads on determining optimal block size: https://superuser.com/questions/234199/good-block-size-for-disk-cloning-with-diskdump-dd https://superuser.com/questions/234199/good-block-size-for-d... http://stackoverflow.com/questions/6161823/dd-how-to-calculate-optimal-blocksize http://stackoverflow.com/questions/6161823/dd-how-to-calcula... [1] http://savannah.gnu.org/forum/forum.php?forum_id=8309 http://savannah.gnu.org/forum/forum.php?forum_id=8309
- derefr 10y agoOr, before that, just send the dd process SIGINFO to get the same output printed once. `while :; do kill -INFO %1; sleep 1; done` if you want a "progress bar" of sorts. As an aside: On BSDs (incl. macOS), SIGINFO is also able to be sent interactively by the line driver when you type ^T (like ^C sends SIGINT.) Kind of lame that Linux doesn't follow suit [or even have SIGINFO], or we'd see a lot more programs that build in useful "prod me for an update" hooks, the way they already have "prod me to reload my config" SIGHUP hooks.
- dozzie 10y agoErm... It's the first time I see SIGINFO signal. I think you meant SIGUSR1.
- farhaven 10y agoOn Unix, there's SIGINFO, which doesn't exist on Linux systems. That's why coreutils' dd uses SIGUSR1 instead.
- dozzie 10y agoOK, I take I didn't know of BSD-specific signal. But please don't claim it's on Unix, because it's not. Unless you can point me where in SUS (or any other specification) it is defined, as I couldn't find it here: http://pubs.opengroup.org/onlinepubs/009695399/basedefs/signal.h.html http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sign...
- cat199 10y agoBSD is Unix. Don't believe the lies.
- yjftsjthsd-h 10y agoNo, I think GP means SIGINFO, which the BSD family uses. On GNU/Linux SIGUSR1 is a passable substitute. See http://www.unix.com/man-page/FreeBSD/3/siginfo/ http://www.unix.com/man-page/FreeBSD/3/siginfo/ and https://unix.stackexchange.com/questions/179481/siginfo-on-gnu-linux-arch-linux-missing https://unix.stackexchange.com/questions/179481/siginfo-on-g... for more about it.
- gkya 10y ago> pv (You probably know this as you use OpenBSD, but) something I really like about BSDs is that nost of the core commands respond to ^T with progress info ofsome kin, dd included.
- arjie 10y agoWoah. That's neat. Thank you!
- ksherlock 10y agocool - by default, ^T generates the non-standard SIGINFO. That seems to be true of other BSDs as well, including OS X.
- gkya 10y agoAlso, Linux dd responds to SIGUSR1, writing progress info on stdout. Sth. to be vary of is that the same signal kills BSD dd.
- tankenmate 10y agoThe other reason for dd existence is that back in the day if you wrote to a block device with a write(2) size of anything other than a multiple of the devices actual block size you'd get a EIO or a EINVAL. So the program you used to make sure the write(2) size was always correct was dd. In the old days some truly weird block devices would accept a write(2) that used only the exact block size, i.e. you could only read and/or write one block at once (if you tried to read more than one block of data you'd get back exactly one block regardless of how big your read buffer was). Old raw CD and WORM drives come to mind. Audio CDs for example had a 2,352 byte block size (after removing the CRC). EDIT: fixed man section references
- mkup 10y ago> back in the day if you wrote to a block device with a write(2) size of anything other than a multiple of the devices actual block size you'd get a EIO or a EINVAL It's not back in the day, it's still true. In Linux, block devices are kernel-cached by default, unless opened with O_DIRECT flag. In general UNIX case (for example, FreeBSD), they aren't: https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-block.html https://www.freebsd.org/doc/en/books/arch-handbook/driverbas... So, in FreeBSD "dd bs=1" will fail if it involves any disk device: disk driver will return EINVAL from read(2) or write(2) because I/O size is not divisible by physical sector size. "cat" with buffer size X (which depends on implementation) will work or not depending on divisibility of X by physical sector size, and other random factors, like short file I/O caused by delivery of the signals. Summary: dd(1) still has its place and author of original article is getting it wrong.
- tankenmate 10y agoYes you are right, there are some UNIXes that ship today without a buffer (block) cache and/or don't default to using the block cache when open(2)ing a block device. For those wondering Linux uses a unified buffer / page cache so there isn't a coherency issue. The buffer cache entries typically point to the corresponding entry in the page cache if it exists. The biggest reason the two are separate but correlated is that the block size isn't always the same as the page size.
- 10y ago