7 ms·

Yes "D" is not for disk/drive/device/... It comes from the DD (data definition) statement of OS/360 JCL, and hence why dd has the unusual option syntax compare

Yes "D" is not for disk/drive/device/... It comes from the DD (data definition) statement of OS/360 JCL, and hence why dd has the unusual option syntax compared to other unix utils BTW if you are using dd to write usb drives etc. it's useful to bypass the Linux VM as much as possible to avoid systems stalls, especially with slow devices. You can do that with O_DIRECT. Also dd recently got a progress option, so... dd bs=2M if=disk.img of=/dev/sda... status=progress iflag=direct oflag=direct Note dd is a lower level tool, which is why there are some gotchas when using for higher level operations. I've noted a few at: http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd

the_mitsuhiko 10y ago
I thought it stands for copy and convert but cc was taken bu the c compiler.
- gonzo 10y ago
  Ask a graybeard.
lloeki 10y ago
On OS X (and BSD?) be sure to use /dev/rdisk[0-9]+ instead of /dev/disk[0-9]+. Details as to exactly why it's faster are welcome. (I just know it bypasses stuff). EDIT: someone mentioned this below http://superuser.com/questions/631592/why-is-dev-rdisk-about-20-times-faster-than-dev-disk-in-mac-os-x http://superuser.com/questions/631592/why-is-dev-rdisk-about...
- djent 10y ago
  Ah wish I knew this last week. Writing Xubuntu to my USB took something like 2900 seconds from a Mac.
  - josteink 10y ago
    Usually just specifying a reasonable blocksize works for me. bs=1m or so. Without that it does literally take hours. I suspect the default blocksize is really small (1?) and combined with uncached/unbuffered writes to slower devices, it just kills all performance outright. Edit: answered! https://news.ycombinator.com/item?id=13350002 https://news.ycombinator.com/item?id=13350002
  - muterad_murilax 10y ago
    In other words, about 48 minutes for a ~1.2 GB file?
    - isostatic 10y ago
      About 3mbit/second, or 400kbytes a second. I'd expect something 50-100 times faster.
  - NamTaf 10y ago
    Per the sibling comments, you just need to specify a sane block size. dd's default is really low and if you experiment a bit with 2M or around that you'll get near-theoretical throughput. NB: Remember the units! Without the units you specify it as bytes or something insanely small like that. I've made that mistake more than once!
- mkup 10y ago
  In FreeBSD, cached/block disk devices are long gone: https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-block.html https://www.freebsd.org/doc/en/books/arch-handbook/driverbas... so all disk devices in /dev are implicitly O_DIRECT. Though, read cache can be enabled manually by creating separate device via gcache(8). This is usually not required, because caching is done at the filesystem layer. It's important to specify block size for uncached devices, of course. dd(1) with bs= option will surely work, and with cp(1) your mileage may vary, depending on whether underlying disk driver supports I/O with partial sector size or not.
Esau 10y ago
Thank you for the historical information.
SSLy 10y ago
>it's useful to bypass the Linux VM as much as possible to avoid systems stalls Oh man, I didn't even know that was the cause of these problems.
drvdevd 10y ago
Nice! Thanks. Built-in status and that Linux bypass trick are beautiful.
Vosporos 10y ago
Oh? I had read that its name was originally "copy and convert" but `cc` was already taken by the compiler
- realfinkployd 10y ago
  Yeah, this article is wrong. Have you ever noticed the syntax for DD is unusual? It is set up more like a JCL syntax.
pmorici 10y ago
Hasn't the ability to check progress be around forever? You could always send it SIGUSR1 and get back a progress report on stderr.
- _joel 10y ago
  Sure it's been around for a while, GNU version on linux at least. Personally found pipe viewer (pv) quite handy too https://www.ivarch.com/programs/pv.shtml https://www.ivarch.com/programs/pv.shtml - available in most distros
- groovy2shoes 10y ago
  Yup. Just don't do what I did earlier and `pkill -USR1 -f dd` if your desktop session is currently being provided to you courtesy of `sddm` … As a programmer, it usually pays off to be on the lazy side, but every once in a while it comes back and bites me in the arse ;)
carussell 10y ago
You don't have to wait for updates to newer versions to get dd to report its progress. The status line that dd prints when it finishes can also be forced at any point during dd's operation by sending the USR1 or INFO signals to the process. E.g.: ps a | grep "\<dd" # [...] kill -USR1 $YOUR_DD_PID or pkill -USR1 ^dd It also doesn't require you to get everything nailed down at the beginning. You've just spent the last 20 seconds waiting and realize you want a status update, but you didn't think to specify the option ahead of time? No problem. I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway. To figure out the ideal interaction, try modeling it after the conversation you'd have if you were talking to a person instead of your shell: "Hey, how much longer is it going to take to flash that image?" The way dd works is close to this scenario.
- hueving 10y ago
  >I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway. Progress bars by default are also garbage if you are scripting and want to just log results. ffmpeg is terrible for this.
  - jff 10y ago
    > Persistent progress indicators are known to cause performance degradation unless implemented carefully. Are you referring to that npm progress bar thing a few months back? I'm pretty sure the reason for that can be summed up as "javascript, and web developers". Anyway, he's not proposing progress bars by default, he's proposing a method by which you can query a process to see how far it's come. I think there's even a key combination to do this on FreeBSD. Or, for example, you could write a small program that sends a USR1 signal every 5 seconds, splitting out the responsibility of managing a progress bar: % progress cp bigfile /tmp/ And then the 'progress' program would draw you a text progress bar, or even pop up an X window with a progress bar.
rsync 10y ago
"Yes "D" is not for disk/drive/device/..." But that's the very beauty of unix! If you can find a way to use 'dd' for disk/drive/device you can use it in interesting new manners (pipelines, etc.) and have very good confidence that it won't break in weird ways. It will do the small, simple thing it is supposed to do even if you are abusing it horribly. Like this, for instance: pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"
- takeda 10y ago
  Is there a benefit to use dd over cat in this case?
  - circuit_breaker 10y ago
    You could use it to rate limit... or arbitrarily set block sizes per use case. I've used it for the former when doing 'over the wire' backups through ssh
abrowne 10y ago
Thanks for the tips. Clueless noob here . . . most guides I've seen use bs=1M for writing e.g. a Linux installer to a USB drive. Does 1 MB vs 2 MB change anything?
- blr246 10y ago
  The setting controls the block size. When writing to block devices, you can maximize throughput by tuning the block size for the filesystem, architecture, and specific disk drive in use. You can tune it by benchmarks and searching over various multiples of 512K block sizes. For most modern systems, 1MB is a reasonable place to start. Even as high as 4MB can work well. The block size can make a major difference in terms of sustained write speed due to reduced overhead in system calls and saturation of the disk interface. A similar thing happens when writing to sockets where lots of small messages kill throughput, but they can decrease latency for a system that passes a high volume of small control messages.

the_mitsuhiko 10y ago

I thought it stands for copy and convert but cc was taken bu the c compiler.

gonzo 10y ago
Ask a graybeard.

lloeki 10y ago

On OS X (and BSD?) be sure to use /dev/rdisk[0-9]+ instead of /dev/disk[0-9]+. Details as to exactly why it's faster are welcome. (I just know it bypasses stuff). EDIT: someone mentioned this below http://superuser.com/questions/631592/why-is-dev-rdisk-about-20-times-faster-than-dev-disk-in-mac-os-x http://superuser.com/questions/631592/why-is-dev-rdisk-about...

djent 10y ago
Ah wish I knew this last week. Writing Xubuntu to my USB took something like 2900 seconds from a Mac.
- josteink 10y ago
  Usually just specifying a reasonable blocksize works for me. bs=1m or so. Without that it does literally take hours. I suspect the default blocksize is really small (1?) and combined with uncached/unbuffered writes to slower devices, it just kills all performance outright. Edit: answered! https://news.ycombinator.com/item?id=13350002 https://news.ycombinator.com/item?id=13350002
- muterad_murilax 10y ago
  In other words, about 48 minutes for a ~1.2 GB file?
  - isostatic 10y ago
    About 3mbit/second, or 400kbytes a second. I'd expect something 50-100 times faster.
- NamTaf 10y ago
  Per the sibling comments, you just need to specify a sane block size. dd's default is really low and if you experiment a bit with 2M or around that you'll get near-theoretical throughput. NB: Remember the units! Without the units you specify it as bytes or something insanely small like that. I've made that mistake more than once!
mkup 10y ago
In FreeBSD, cached/block disk devices are long gone: https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-block.html https://www.freebsd.org/doc/en/books/arch-handbook/driverbas... so all disk devices in /dev are implicitly O_DIRECT. Though, read cache can be enabled manually by creating separate device via gcache(8). This is usually not required, because caching is done at the filesystem layer. It's important to specify block size for uncached devices, of course. dd(1) with bs= option will surely work, and with cp(1) your mileage may vary, depending on whether underlying disk driver supports I/O with partial sector size or not.

Esau 10y ago

Thank you for the historical information.

SSLy 10y ago

>it's useful to bypass the Linux VM as much as possible to avoid systems stalls Oh man, I didn't even know that was the cause of these problems.

drvdevd 10y ago

Nice! Thanks. Built-in status and that Linux bypass trick are beautiful.

Vosporos 10y ago

Oh? I had read that its name was originally "copy and convert" but `cc` was already taken by the compiler

realfinkployd 10y ago
Yeah, this article is wrong. Have you ever noticed the syntax for DD is unusual? It is set up more like a JCL syntax.

pmorici 10y ago

Hasn't the ability to check progress be around forever? You could always send it SIGUSR1 and get back a progress report on stderr.

_joel 10y ago
Sure it's been around for a while, GNU version on linux at least. Personally found pipe viewer (pv) quite handy too https://www.ivarch.com/programs/pv.shtml https://www.ivarch.com/programs/pv.shtml - available in most distros
groovy2shoes 10y ago
Yup. Just don't do what I did earlier and `pkill -USR1 -f dd` if your desktop session is currently being provided to you courtesy of `sddm` … As a programmer, it usually pays off to be on the lazy side, but every once in a while it comes back and bites me in the arse ;)

carussell 10y ago

You don't have to wait for updates to newer versions to get dd to report its progress. The status line that dd prints when it finishes can also be forced at any point during dd's operation by sending the USR1 or INFO signals to the process. E.g.: ps a | grep "\<dd" # [...] kill -USR1 $YOUR_DD_PID or pkill -USR1 ^dd It also doesn't require you to get everything nailed down at the beginning. You've just spent the last 20 seconds waiting and realize you want a status update, but you didn't think to specify the option ahead of time? No problem. I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway. To figure out the ideal interaction, try modeling it after the conversation you'd have if you were talking to a person instead of your shell: "Hey, how much longer is it going to take to flash that image?" The way dd works is close to this scenario.

hueving 10y ago
>I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway. Progress bars by default are also garbage if you are scripting and want to just log results. ffmpeg is terrible for this.
- jff 10y ago
  > Persistent progress indicators are known to cause performance degradation unless implemented carefully. Are you referring to that npm progress bar thing a few months back? I'm pretty sure the reason for that can be summed up as "javascript, and web developers". Anyway, he's not proposing progress bars by default, he's proposing a method by which you can query a process to see how far it's come. I think there's even a key combination to do this on FreeBSD. Or, for example, you could write a small program that sends a USR1 signal every 5 seconds, splitting out the responsibility of managing a progress bar: % progress cp bigfile /tmp/ And then the 'progress' program would draw you a text progress bar, or even pop up an X window with a progress bar.

rsync 10y ago

"Yes "D" is not for disk/drive/device/..." But that's the very beauty of unix! If you can find a way to use 'dd' for disk/drive/device you can use it in interesting new manners (pipelines, etc.) and have very good confidence that it won't break in weird ways. It will do the small, simple thing it is supposed to do even if you are abusing it horribly. Like this, for instance: pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"

takeda 10y ago
Is there a benefit to use dd over cat in this case?
- circuit_breaker 10y ago
  You could use it to rate limit... or arbitrarily set block sizes per use case. I've used it for the former when doing 'over the wire' backups through ssh

abrowne 10y ago

Thanks for the tips. Clueless noob here . . . most guides I've seen use bs=1M for writing e.g. a Linux installer to a USB drive. Does 1 MB vs 2 MB change anything?

blr246 10y ago
The setting controls the block size. When writing to block devices, you can maximize throughput by tuning the block size for the filesystem, architecture, and specific disk drive in use. You can tune it by benchmarks and searching over various multiples of 512K block sizes. For most modern systems, 1MB is a reasonable place to start. Even as high as 4MB can work well. The block size can make a major difference in terms of sustained write speed due to reduced overhead in system calls and saturation of the disk interface. A similar thing happens when writing to sockets where lots of small messages kill throughput, but they can decrease latency for a system that passes a high volume of small control messages.