14 ms·

Is it better to use cat, dd, pv or another procedure to copy a CD/DVD? (2015)

134 points by polm23 6y ago

taneliv 6y ago
Interesting that dvdisaster is not mentioned in the answers.
- redis_mlc 6y ago
  Thanks for mentioning dvdisaster. Looks like it was originally written for optical disks, but I suppose it would work for other media: https://en.wikipedia.org/wiki/Dvdisaster https://en.wikipedia.org/wiki/Dvdisaster
AdmiralAsshat 6y ago
The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb. I don't know why dd remains the standard for so many tutorials when there are competent GUI tools that can take all the guesswork out of it. It's like a perpetual hazing ritual for new Linux users.
- extraduder_ire 6y ago
  dd can be faster than alternatives, if you set the block size right, and is always available, even without a GUI. I've often used whatever disk utility came with my distro. Or, rufus (rufus.ie) if I'm on windows.
  - sjburt 6y ago
    You can also just use cp, and then you don't need to worry about the block size. Really only need dd if you have to limit how much data is read.
- sitzkrieg 6y ago
  its simple, already installed and has huge existing tow. at least with newer classes of device storage drivers like nvme you end up with completely different names for system drives!
- inetknght 6y ago
  > I don't know why dd remains the standard for so many tutorials when there are competent GUI tools You miss the point entirely then. The point is to use a command line because what you think is competent for a GUI tool isn't nearly as intuitive as you think especially when a GUI isn't available.
- tinus_hn 6y ago
  Why do you think dd is special in that it can overwrite devices? Do you think ‘cat’ can’t? The root user is special in that it can overwrite devices. Look at what you’re typing before using its powers.
- nemosaltat 6y ago
  The “simple GUI” tool I keep seeing pushed Maker forums is Etcher. I needed to flash something from my work PC so I gave it a shot- 90MB of Electron bloat, while offering no extra functionality over dd . Plus, dd doesn’t advertise to me during the flash, and certainly doesn’t phone home to balena. For folks just getting started or more comfortable with a GUI, I’d recommend giving USBimager[0] a look. It does exactly what’d you’d expect based on the name, it performant, and they have native apps. No affiliation, just a fan of a KISS app done right. [0] https://gitlab.com/bztsrc/usbimager https://gitlab.com/bztsrc/usbimager
  - trinix912 6y ago
    There's also Rufus, sadly it's Windows-only.
    - ccmcarey 6y ago
      Woeusb is pretty decent on linux, and does the same uefi-ntfs magic that Rufus does, and that you'll need if you're burning a large ISO like recent Windows ones. Though if it's just for ISOs, ventoy is fantastic, just drag and drop the file and no burning at all :)
      - jorvi 6y ago
        > that you'll need if you're burning a large ISO like recent Windows ones. I ran into this very issue when trying to make a bootable Windows 10 USB on macOS. No amount of fiddling with dd, unetbootin or Etcher resulted in a bootable USB. Despite being principled about it, I had to admit defeat and just pulled an old <4GB Windows 10 iso and flashed that to the stick. I know I could have installed Linux through a virtual machine and got it done that way, but that seemed horrible overkill. Oh well.
        moondev 6y ago
        Double click iso to mount it, format usb as fat32 or exfat, copy and paste contents of mounted iso to usb drive This has always worked for me to boot UEFI installers
- zenlot 6y ago
  > and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb One trick to be sure that you've typed correctly, start with 'echo': # echo dd if=xxx of=yyy [enter] This way you'll be able to check that command will be executed with the parameters you want. Especially helpful if you're running loops, e.g: # for i in xxx; do echo yyy; done I would never use GUI for that, as you can't be sure what it will execute, no matter what it shows.
  - Kwpolska 6y ago
    How would `echo dd if=linux.iso of=/dev/sda` protect you from nuking the wrong drive? To be sure you aren’t overwriting the wrong drive, you need to read the command before executing it — and you don’t need echo for that, just pause for a second before hitting Enter — and you need to know which drive is which, and that requires running a different command, or using eg. /dev/disk/* instead of /dev/sd??.
    - deleted 6y ago
      [deleted]
- skissane 6y ago
  > The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb. Wouldn't one (at least partial) solution here be, that the kernel should refuse to let you write directly to a block device in use by a mounted filesystem? (Maybe combined with some special ioctl/whatever to bypass that restriction if you ever really need to.) Then, if you are running dd from an OS running on your main drive, the kernel will refuse to let dd overwrite the main drive, but will let it overwrite the flash drive (which presumably is not mounted, and anyway shouldn't be if you are about to overwrite it)
  - Hello71 6y ago
    There are two major problems with this: partitions/device-mapper and backwards compatibility. First, partitions inherently overlap the parent block device. You would need to carefully track exactly which portions of the device are in use, or your solution is useless (it doesn't protect /dev/sda when /dev/sda1 is in use) or blocks the vast majority of use cases (/dev/sda2 cannot be used when /dev/sda1 is in use). The same applies for device-mapper, but worse. Secondly, the Linux kernel has a very strong backwards compatibility guarantee. Any change that would break valid (or even invalid) uses will be loudly rejected by Linus. With your idea, many disk management programs will be broken.
    - TacticalCoder 6y ago
      You could though probably make a wrapper that only kicks in when dd is called from the command line (as opposed to, say, from a script or some GUI program) and asking a confirmation "Do you really want to overwrite your main drive?", without breaking backward compatibility? I have written a few wrappers like that on my own system preventing me from making a few common mistakes of mine (like scp'ing a file locally to a filename ressembling an IP address instead of on a remote server ;)
      - Hello71 6y ago
- phiresky 6y ago
  > I don't know why dd remains the standard for so many tutorials You can also do something like dd if=file.iso of=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB Where the last part is basically the brand and name of your USB stick, not much guess work required. Though I've never seen that in any tutorials.
  - smcameron 6y ago
    What I do to find out which is which is use lsscsi: $ lsscsi [0:0:0:0] disk ATA ST2000DX001-1NS1 CC41 /dev/sda [3:0:0:0] cd/dvd ASUS SH-224FB 1.00 /dev/sr0 [4:0:0:0] disk Generic- Multiple Reader 1.11 /dev/sdb
    - dirkt 6y ago
      Unfortunately you have to do this again after each reboot, or whenever you plug in an USB stick, so the dev/by-* option does help when you do that repeatedly for the same device.
  - jeroenhd 6y ago
    It's really unfortunate how tutorials still spell out all the basics to keep the beginners with them, but then rely on the old-fashioned device names when it comes to writing images. Using sda is fine if you're experienced and know what drive has what name, but technical references should stop relying on them already. It's not like you're getting some useful cross-platform compatibility out of old-fashioned commands or anything, every modern Unix has its own way to specify disk names.
    - csydas 6y ago
      I think the simplest explanation is that the authors themselves never considered the other options or were even aware of them -- they simply are sharing some historical knowledge they had. Very kind and good behavior! But I would imagine it's something rote and probably just a shortcoming in collective knowledge. In short, I think ignorance is the simpler answer ;)
- jwr 6y ago
  > I don't know why dd remains the standard for so many tutorials While dd isn't the best tool for writing images to devices (if only because of its arcane and bizarre command-line syntax), it is a valuable tool when you want to recover data from media that has errors. The 'conv=noerror' option is a lifesaver for when you want to recover something from your media. But overall I agree: plenty of people recommend using dd just because "it's always been done that way" and often also because it makes them look smart :-)
  - rzzzt 6y ago
    ddrescue can also help with that use case: https://www.gnu.org/software/ddrescue/ https://www.gnu.org/software/ddrescue/
    - lizknope 6y ago
      I always used ddrescue like this. Even on badly scratched discs it was able to recover most of the data. ddrescue -b 2048 -d -r 3 -R -v /dev/sr0 image.iso image.log
      - compsciphd 6y ago
        I used to use 2048, that's probably the wrong value to use. want to use something that corresponds to the ECC block size of the underlying media. but yes, I do that as well (came to say mostly the same comtent, but I'd do r -1)
        colejohnson66 6y ago
        Is the ECC available to the OS? I thought the drive handled the ECC and just reported errors?
        wtallis 6y ago
        If you're trying to read with a smaller block size than the media's native block size, then you'll make two or more attempts to read data from each corrupted block that's unrecoverable—making your recovery process much longer. If you try to read with a larger block size than the media's native block size, you'll get errors for chunks of that size even when part of the data may have been recoverable. The above is true whether or not the OS has access to the media's raw ECC data.
- loloquwowndueo 6y ago
  You assume one has access to a GUI. Not always the case.
- sildur 6y ago
  That's another advantage of using Qubes. Worst case scenario I botch the image-burning Qube, which is temporary anyways.
- megous 6y ago
  One mitigation is to recommend users using symlinks in ls /dev/disk/by-id instead of /dev/sd* Other than that, someone can replace the whole of Etcher with a simple bash script that will list available block devices with nice descriptive names taken from sysfs, allow the user to select the one he wants to flash to, and handle flashing of potentially compressed image and verifying the result automatically. It would probably not be much longer than 100 LoC, at least on Linux.
- TacticalCoder 6y ago
  That said I remember a roommate erasing it's Windows "C:" drive using Ghost (that is "Ghost" before Norton bought it). No later than yesterday I copied my wife's entire (Windows 8.1) HDD to an SSD using the free (GUI) version of "Macrium Reflect". I'm pretty sure you can make the exact same mistake: copying destination unto source instead of the contratry. I used a Windows software and not DD for it was a Windows computer and I didn't feel like booting a live Linux CD to do the dump but under Linux, I always use dd. Is it really that hard to learn that in dd the 'i' in "if" means "input" and that the 'o' in "of" means "output"? The problem with "making things simple using a GUI" is that you typically totally lose the ability to do not just advanced things but even "average" things. For example for read-only media I always write the checksum on the media itself, using a sharpie. That way I can easily verify that my disk (or the copy) is ok doing a dd and piping into sha256sum (there are some gotchas to keep in mind but it works fine when you know how to do it). Like, say, a Debian install ISO. I like to have these on read-only medium and make sure the checksum matches the official one (so I prefer a write-once / read-only DVD than a read/write memory stick). How do you that with the GUI? Piping into a cryptographic hash? I can understand that people prefer GUI over command line for many things but I think that people imaging entire disks are at least power users and can learn the difference between "input" and "output". Heck, maybe it's even doing them a service to teach them to use the CLI. Maybe it's the opportunity to teach them about piping, about cryptographic hashes, etc. And once again: you can totally screw up with a GUI too. I don't mean it in a bad way at all but... Linux on the desktop (which I use since 20 years) ain't exactly enjoying a huge market share compared to Windows or OS X and I think that's fine. If a Linux user cannot or isn't willing to learn dd, maybe that user is better served by Windows or OS X. And really: I don't mean it in a bad way. I don't think Linux has to "win" the Desktop war. I don't think it's wrong not to use Linux. But I do think it's wrong to believe users willing to learn Linux cannot learn the difference between input and ouptut. Also a strong case can be made that someone for whom it's a "disaster" to overwrite it's main drive is one hard disk failure away from disaster anyway. I'm for educating users, not baby-feeding them with tools that are going to keep them in the dark and reinforce their bad practices (like not doing proper backups and hence being "one hard drive failure" away from disaster).
- 1337shadow 6y ago
  Just check dmesg when you plug in your USB drive, it will tell you what device it created for it.
- JasonFruit 6y ago
  It's really not that hard to use correctly. Maybe it shouldn't be the standard for beginners' tutorials, sure, but it's not terrifying once you know how it works, as long as you exercise appropriate caution. And that's the whole benefit of command-line tools; they're like a chainsaw in that they get a whole lot of work done in a hurry, but you have to point them the right direction and keep them under control.
- tzs 6y ago
  Better permissions could help with this. Make it so that there is a user other than root that can write to the block device the flash drive is on, and then you sudo to that user to write images rather than sudoing to root. Then accidentally trying to write to the main drive will get a permission error.
- alerighi 6y ago
  dd has the advantage of specifying a larger buffer. I've seen for example that in MacOS writing an image to a SD card with the internal reader of my macbook is a lot faster with a block size of 1M in respect to the classical 4k. Also the advantage of dd is that you can run it with sudo easily, while with cat it will not work, since the redirection is done by the shell and not the cat binary itself, and you either have to open a root shell, or pipe the output of cat in `sudo tee filename >/dev/null` that is less than ideal. Also I think the problem is of Linux that lets you write on disks that are mounted. In MacOS is forbidden and you must umount the drive (so overwriting your root partition by error is impossible).
- andrewzah 6y ago
  Alternatively, I have never written to the wrong device when using dd. I always double and triple-check with lsblk and blkid before committing to it. dd is always available, unlike other tools. If a user blindly copy-pastes and destroys their main drive, that's a very good lesson to thoroughly double check before running destructive operations.
  - colejohnson66 6y ago
    Just because you should do something doesn’t mean we should make it easy to screw up. And just because you and I double check doesn’t mean everyone else does. It is their fault, sure, but it’s also not when the internet guides don’t give you a massive warning about double checking.
    - kirse 6y ago
      The adventure doesn't start until things go wrong, just think of it as Linux's way of providing a lifetime of adventures and great stories, like trying to fix your parent's computer at 3am before they wake up and realize you thought you had the dual-boot Windows/SuSE setup working but now the master boot record is corrupted and you're still a Linux noob without StackOverflow or Google for answers. Battle scars my friend.
  - alex_smart 6y ago
    To add to what the other user said, just because you can do something with one tool, doesn't mean we shouldn't promote other tools that are easier to use and that make it easier for you to do the job without accidentally shooting yourself in the foot.
- m463 6y ago
  I accidentally repartitioned my main linux drive once. I just ran gparted: $ gparted and instead of complaining that I didn't specify an argument, it chose the first one: / When I realized what I had done, I couldn't recover the partition table, but I managed to rsync everything elsewhere. I'm sure there was a better/safer way but everything was recovered in the end.
- lvass 6y ago
  I once mistakenly overwrote an entire disk (or wrong partition, I don't remember) when trying to install a GNU/Linux distribution to one partition using a graphical installer. Never happened using dd. Sure some graphical applications can be helpful in this regard, but there's really no reason to assume the one you end up using will have you avoid this type of mistake. I'd rather use something that is stable and be vigilant as steadily required.
dspillett 6y ago
pv's progress display is very useful in many contexts, it has been one if the first things I make sure is installed on a new system for years, though I have recently come across https://github.com/Xfennec/progress https://github.com/Xfennec/progress which is brilliant for some times when pv is not an option or when you want to watch the status of several things which aren't convenient to arrange together via tmux/byobu. Or just for when you forget to use PV and don't want to cancel to change the command.
- ktpsns 6y ago
  Thanks for the tip for progress! I didn't know this utility and it sounds really helpful.
  - dspillett 6y ago
    I found it while putting together something similar myself (I might restart mine later, it was going to be a web & TUI based graph output, but progress+watch does the basic job more than well enough for now).
deleted 6y ago
[deleted]
yyyk 6y ago
I don't think it's a good idea to use a program that isn't aware of the device buffer to copy a CD/DVD. Such failures are rare nowadays, but installing some CLI program like cdrecord isn't difficult either.
reidrac 6y ago
I used to work on a hosting company and proper use of dd was important when copying data from LVM in a Xen host; unfortunately I seem to have forgotten most of it. Some pointers: dd has oflags; may be oflags=direct is faster? You can also use oflags=sparse and sometimes save space by creating a sparse file.
- natmaka 6y ago
  > may be oflags=direct is faster? oflag=direct does direct I/O => copied data won't go into the buffercache. On Linux search for 'O_DIRECT' in the open(2) manpage. oflag (for output) and iflag (for input) are indeed useful. During/after a massive non-'direct' copy a system running other processes which benefit from data in the buffercache may crawl if the system, while copying, replaces some of it by the copied data, then has to re-read it. In other terms this seems adequate when copying data which will not be, after the copy, soon read by any process. A raw filesystem image is a good candidate. As usual YMMV. If most of the data to be copied already is in the buffercache or if it will occupy some unused part of the core memory... such optimization is useless. However in most cases (on most adequately-dimensioned non-idle systems) 'O_DIRECT' induces less systemwide load than cp, cat, pv(...) when copying a large set of data if most of it will not be, then, immediately read by anything. Other tools (cp, cat, pv...) just cannot easily work in 'O_DIRECT' mode. Using some trick to enable it thank to a local version of openat() and LD_PRELOAD (which calls openat in O_DIRECT mode), albeit possible, isn't realistic in most contexts. $ cd ~/tmp $ strace -e openat dd if=/etc/hosts of=useless.tmp count=1 >& nodirect $ strace -e openat dd if=/etc/hosts of=useless.tmp iflag=direct oflag=direct count=1 >& direct $ diff direct nodirect 5,6c5,6 < openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_DIRECT) = 3 < openat(AT_FDCWD, "useless.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0666) = 3 --- > openat(AT_FDCWD, "/etc/hosts", O_RDONLY) = 3 > openat(AT_FDCWD, "useless.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 Moreover 'dd' has many options without equivalent in most other readily available tools.
  - natmaka 6y ago
    Also check https://github.com/Feh/nocache https://github.com/Feh/nocache
- 6y ago
dmurray 6y ago
I always thought dd stood for "duplicate disk" or "duplicate device" and as such it was the "right" command to use here. Apparently it's nothing to do with that and originally comes from "data definition" though. https://en.m.wikipedia.org/wiki/Dd_(Unix) https://en.m.wikipedia.org/wiki/Dd_(Unix)
- GTP 6y ago
  dd is also known as "Data Destroyer", because it's easy to make a mistake and overwrite the wrong device. This actually happened to me once, but I still use it when I need to write an ISO to a USB drive anyway.
  - RMPR 6y ago
    An advice that was mentioned many times in this thread to avoid writing to the wrong disk. Use the symlinks in /dev/disks/* instead of /dev/sd* directly. Since they are more intuitive, using them will likely reduce the odds of you making a mistake.
  - MaKey 6y ago
    Just avoid using /dev/sdX, use /dev/disk/by-id/ instead.
- _wldu 6y ago
  It's original name was cc (convert and copy). Because that is what it does. They could not use that name because that was the name of the compiler. So, they went to the next letter of the alphabet and named it dd.
xorcist 6y ago
That answer does not seem very useful to me. The use cases are different. Reasons to use dd include being able to set the block size, seek and skip to resume copies or extract parts of the data, and a whole bunch of special options for the very specific use case of large binary files or block devices. Reasons to use cp include it being file based, so metadata and recursive copying is natural, efficient, and easy to use. This probably includes most copying on a daily basis. See also rsync.
- jeroenhd 6y ago
  One advantage dd has, specifically when it comes to copying physical media, is that you can tell it to continue in case of a read error, instead of crapping out halfway through. The lack of such functionality can make copying a scratched CD quite challenging with tools like cat or cp. You also have the choice to ignore the data that could not be read (dangerous) or to write a padded block of the correct size in its place and continue. I agree that the use cases for all tools are different so comparing them fairly is impossible. dd is still my go-to for making disk images, but it's unnecessary and has high overhead without a lot of tweaking for simple tasks like file operations. cat is often a good tool for things like piped commands where you need a flag cat supports (so you can't use shell redirection) and cp/rsync are obviously superior for copying files rather than just data. Sure, uu could hack my commandsin such a way that one tool can do another's job, but why would I?
- IgorPartola 6y ago
  Notice that they way they use cp isn’t to copy the files contained on the disc but to copy the image of the disc itself.
IgorPartola 6y ago
The buried lede here is that cat can be faster than dd: https://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/9492#9492 https://unix.stackexchange.com/questions/9432/is-there-a-way... The other important part is that cat will not have issues with binary data, which I am embarrassed to admit I assumed was not the case for the reasons stated in TFA. pv is the tool I want to use for things like copying images to SD cards but I can never remember the exact syntax needed to get it to show me the percentage since I don’t use it often. dd has a decent syntax for its command parameters and has some fun options like ability to skip, so I can quickly create large empty files.
- Aachen 6y ago
  > pv is the tool I want to use for things like copying images to SD cards but I can never remember the exact syntax needed to get it to show me the percentage You're in luck: the "syntax needed" is to literally not use any options: pv your.img > /dev/sdb
  - IgorPartola 6y ago
    Haha good point. The first time I encountered pv it was in the form of like cat your.img | pv ... > /dev/sdb and then you had to tell it what size to expect and enable it showing the progress bar, etc.
    - syncsynchalt 6y ago
      Good ol' "Useless Use of Cat": http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
    - Aachen 6y ago
      True, if you pipe a data stream to it, it can't tell what the size is going to be and you'd need the --size option. The progress bar is still default, though.
dfox 6y ago
Many years ago I wrote https://github.com/adh/random-tools/blob/master/byc.c https://github.com/adh/random-tools/blob/master/byc.c which tends to be the tool that I use for copying images from/to block device (it was originally meant for something else which is why you need -p for it to do the right thing).
pontifier 6y ago
I've been dealing with a lot of CD copying problems lately. I'm working on a setup to automatically detect and correct errors reading audio CDs which I've found don't use checksums or other mechanisms to prevent small defects from mangling the bitstream. I'd previously used cdparanoia, but found it not paranoid enough.(for some reason it always outputs aiff regardless of options as well) I'm looking at defeating the drive cache by actually reading the disc in a second and 3rd drive and doing a NASA style "all 3 must agree". It's annoying, but I've got 800,000 CDs to rip right now, and I have to make sure each one is perfect.(Without proprietary solutions) I also need to be able to positively reduplicate heavily scratched discs. It's a tough problem.
- unnouinceput 6y ago
  800k? are you insane? even if everything else is automated and you have 800K devices at your fingertip ready to go, the simple amount of time to lift CD, open player tray / insert CD / close tray is in decades!. Let's say you do this 8 hours per day for 5 days. The above series of movements can take around 30 seconds. That's 400k minutes which results in 400k / 40 hours == 10k weeks. At 50 weeks per year (only 2 weeks vacation each year) that still results in 200 years!!. Even if you do one CD every second this result in 200 years / 30 ==~7 years.
  - pontifier 6y ago
    I very well might be. I took over Murfie when they shut down, and am in the process of re-building the company. People don't understand the scale, and why I might want to make sure I do it right the first time rather than get halfway through and have to start over because I missed something.
    - gruez 6y ago
      I'm assuming you're the guy in this article? https://www.theverge.com/2020/2/5/21121594/crossies-murfie-madison-wisconsin-arkansas-1-million-abandoned-cds https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...
cjdell 6y ago
I'm so glad that DD now has the status=progress option. Still not widely known.
- TacticalCoder 6y ago
  I used to cat dd into "pv" : )
  - hyperdimension 6y ago
    Before I learned of the flag, I used to do a 'while true; do killall -SIGUSR1 dd; sleep 5; done' loop in another terminal.
    - saagarjha 6y ago
      SIGINFO on macOS.
andrewmackrodt 6y ago
Do any of the tools mentioned in the title support audio and/or multi-session disc formats? AFAIK dd is not suitable for these purposes (although the original author mentions they wish to create ISO images so presumably they only care about single session data). Windows had a lot of great disc software in the 90s/00s which would handle just about any disc format one would encounter, e.g. Alcohol 120%, CloneCD, Disc Juggler. Some of these were capable of backing up the various protections of the time such as SecureROM and Starforce. dd is a powerful tool but I often find it the "wrong" tool, e.g. is it desired to clone a 2GB partition layout to a 128GB usb stick for a live CD and have to manually edit my partition tables myself?
abaines 6y ago
I've always used cdrdao read-cd --read-raw to backup CD/DVDs, along with toc2cue and bchunk to convert to .iso. I have no idea if the resulting iso is "more accurate" than using dd - but they've always worked, even from CDs with copy protection and obscure file systems (old mac CDs).
pontifier 6y ago
I use tail all the time with "-f" to see log files in realtime, but seeing it as an option for reading an entire CD with "-c +1" was surprising.
maerF0x0 6y ago
Performance. This is an I/O-bound process; the main influence in performance is the buffer size: the tool reads a chunk from the source, writes the chunk to the destination, repeats. If the chunk is too small, the computer spends its time switching between tasks. If the chunk is too large, the read and write operations can't be parallelized. The optimal chunk size on a PC is typically around a few megabytes but this is obviously very dependent on the OS, on the hardware, and on what else the computer is doing. I made benchmarks for hard disk to hard disk copies a while ago, on Linux, which showed that for copies within the same disk, dd with a large buffer size has the advantage, but for cross-disk copies, cat won over any dd buffer size. As a meta comment, it has been a shocking revelation in my career that the difference between "get it work" engineers and "make it right, make it fast"[3] has been this kind of low level knowledge. At least in my context, the majority of the bugs that have come up in the past while haven't been because someone didn't know how to concatenate 2 strings or even that many off by 1 errors. They have been things like not knowing the difference between theoretical TCP models and how their actually implemented on the OS in a high throughput environment (see TIME_WAIT[1] if you're curious), or understanding misorder of operations in distributed computing[2]. [1]: https://9oelm.github.io/2018-05-06--Listening,-Established,-Close_wait-and-Time_wait-in-netstat/ https://9oelm.github.io/2018-05-06--Listening,-Established,-... [2]: http://book.mixu.net/distsys/ http://book.mixu.net/distsys/ [3]: https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast
- tyingq 6y ago
  Protocols that don't have a length header, but instead rely on delimiters is a good example. And, they are everywhere.
tyingq 6y ago
One non-obvious reason dd is often used in examples is because of device permissions. You can't (easily/legibly) present a one-liner that combines cat and sudo because sudo doesn't give the shell redirection (sudo cat >> /dev/whatever) permissions. But "sudo dd of=/dev/whatever" works fine. Edit: Yes, you can make a one liner via sh -c, tee, etc, but depending on cut/paste with quotes is tricky, or the one-liner gets long.
- colejohnson66 6y ago
  You can also just use `su`, run your command, then `exit`, but you’re right: `sudo dd` is much easier. (This obviously ignores that you should almost never use `su`)
  - na85 6y ago
    > you should almost never use `su` Why?
    - colejohnson66 6y ago
      The same reason you should be logged in as root. It has its uses, but you need to be careful as it can be easy to forget you’re running as root and screw something up. Explicitly putting `sudo` before every command makes it clear what you’re doing.
blincoln 6y ago
Any time CDs/CD-ROMs (as opposed to DVDs) are discussed, I get really nervous unless someone explicitly confirms that the tool being used handles XA/2352-bytes-per-sector data correctly. Supposedly cdrdao will.[1] DVDs are for the most part pretty easy to duplicate identically. CDs have a lot of quirks, and it's very easy to end up with a useless "backup". Especially if the file format is ISO instead of bin/cue or similar. If one is trying to duplicate CDs/CD-ROMs, it's really vital to cross all your Ts and dot all your Is. [1] https://consolecopyworld.com/psx/psx_copy_patch_linux.shtml https://consolecopyworld.com/psx/psx_copy_patch_linux.shtml
- teej 6y ago
  Is there a reason that the copying process needs to be aware of how bytes are organized? Why can’t you just read the bytes from disk A and write those exact bytes to disk B?
  - snarfy 6y ago
    Which bytes? A raw 2352 byte copy that includes the P and Q channels (timing) can behave differently than a 2048 byte copy depending on the software.
    - harry8 6y ago
      Ok but why should we care about that in copying? This is my mental model for what's going on: There's a bunch of bytes stored on a CD-ROM in a defined order. Zeroes and ones. Copy them in order. You should now have a file on magnetic disk or flash that is precisely those byes in precisely that order. Anything that can make sense of one should be able to make sense of the other. What am I missing here?
      - snarfy 6y ago
        There is more than one set of bytes. You may not care about copying the inode structure when you are copying your files from point a to point b, but you should when you are cloning the disk. Many times the software on the CD is looking at the physical layout of the disk, not just the logical data, to function correctly.
codethief 6y ago
What's the best way to rip DVDs through the CLI these days? Are the error correction techniques in the DVD standard enough to ensure that a simple `cat /dev/dvd0` or `dd /dev/dvd0` is going to work (and, in particular, fail if an error can't be corrected)?
saagarjha 6y ago
For those using dd (or cp) on macOS, ⌃T will send the process a SIGINFO and print some additional information about progress. On Linux, SIGUSR1 will do something similar, but it's not as convenient.

134 points by polm23 6y ago

taneliv 6y ago

Interesting that dvdisaster is not mentioned in the answers.

redis_mlc 6y ago
Thanks for mentioning dvdisaster. Looks like it was originally written for optical disks, but I suppose it would work for other media: https://en.wikipedia.org/wiki/Dvdisaster https://en.wikipedia.org/wiki/Dvdisaster

AdmiralAsshat 6y ago

The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb. I don't know why dd remains the standard for so many tutorials when there are competent GUI tools that can take all the guesswork out of it. It's like a perpetual hazing ritual for new Linux users.

extraduder_ire 6y ago
dd can be faster than alternatives, if you set the block size right, and is always available, even without a GUI. I've often used whatever disk utility came with my distro. Or, rufus (rufus.ie) if I'm on windows.
- sjburt 6y ago
  You can also just use cp, and then you don't need to worry about the block size. Really only need dd if you have to limit how much data is read.
sitzkrieg 6y ago
its simple, already installed and has huge existing tow. at least with newer classes of device storage drivers like nvme you end up with completely different names for system drives!
inetknght 6y ago
> I don't know why dd remains the standard for so many tutorials when there are competent GUI tools You miss the point entirely then. The point is to use a command line because what you think is competent for a GUI tool isn't nearly as intuitive as you think especially when a GUI isn't available.
tinus_hn 6y ago
Why do you think dd is special in that it can overwrite devices? Do you think ‘cat’ can’t? The root user is special in that it can overwrite devices. Look at what you’re typing before using its powers.
nemosaltat 6y ago
The “simple GUI” tool I keep seeing pushed Maker forums is Etcher. I needed to flash something from my work PC so I gave it a shot- 90MB of Electron bloat, while offering no extra functionality over dd . Plus, dd doesn’t advertise to me during the flash, and certainly doesn’t phone home to balena. For folks just getting started or more comfortable with a GUI, I’d recommend giving USBimager[0] a look. It does exactly what’d you’d expect based on the name, it performant, and they have native apps. No affiliation, just a fan of a KISS app done right. [0] https://gitlab.com/bztsrc/usbimager https://gitlab.com/bztsrc/usbimager
- trinix912 6y ago
  There's also Rufus, sadly it's Windows-only.
  - ccmcarey 6y ago
    Woeusb is pretty decent on linux, and does the same uefi-ntfs magic that Rufus does, and that you'll need if you're burning a large ISO like recent Windows ones. Though if it's just for ISOs, ventoy is fantastic, just drag and drop the file and no burning at all :)
    - jorvi 6y ago
      > that you'll need if you're burning a large ISO like recent Windows ones. I ran into this very issue when trying to make a bootable Windows 10 USB on macOS. No amount of fiddling with dd, unetbootin or Etcher resulted in a bootable USB. Despite being principled about it, I had to admit defeat and just pulled an old <4GB Windows 10 iso and flashed that to the stick. I know I could have installed Linux through a virtual machine and got it done that way, but that seemed horrible overkill. Oh well.
      - moondev 6y ago
        Double click iso to mount it, format usb as fat32 or exfat, copy and paste contents of mounted iso to usb drive This has always worked for me to boot UEFI installers
zenlot 6y ago
> and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb One trick to be sure that you've typed correctly, start with 'echo': # echo dd if=xxx of=yyy [enter] This way you'll be able to check that command will be executed with the parameters you want. Especially helpful if you're running loops, e.g: # for i in xxx; do echo yyy; done I would never use GUI for that, as you can't be sure what it will execute, no matter what it shows.
- Kwpolska 6y ago
  How would `echo dd if=linux.iso of=/dev/sda` protect you from nuking the wrong drive? To be sure you aren’t overwriting the wrong drive, you need to read the command before executing it — and you don’t need echo for that, just pause for a second before hitting Enter — and you need to know which drive is which, and that requires running a different command, or using eg. /dev/disk/* instead of /dev/sd??.
  - deleted 6y ago
    [deleted]
skissane 6y ago
> The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb. Wouldn't one (at least partial) solution here be, that the kernel should refuse to let you write directly to a block device in use by a mounted filesystem? (Maybe combined with some special ioctl/whatever to bypass that restriction if you ever really need to.) Then, if you are running dd from an OS running on your main drive, the kernel will refuse to let dd overwrite the main drive, but will let it overwrite the flash drive (which presumably is not mounted, and anyway shouldn't be if you are about to overwrite it)
- Hello71 6y ago
  There are two major problems with this: partitions/device-mapper and backwards compatibility. First, partitions inherently overlap the parent block device. You would need to carefully track exactly which portions of the device are in use, or your solution is useless (it doesn't protect /dev/sda when /dev/sda1 is in use) or blocks the vast majority of use cases (/dev/sda2 cannot be used when /dev/sda1 is in use). The same applies for device-mapper, but worse. Secondly, the Linux kernel has a very strong backwards compatibility guarantee. Any change that would break valid (or even invalid) uses will be loudly rejected by Linus. With your idea, many disk management programs will be broken.
  - TacticalCoder 6y ago
    You could though probably make a wrapper that only kicks in when dd is called from the command line (as opposed to, say, from a script or some GUI program) and asking a confirmation "Do you really want to overwrite your main drive?", without breaking backward compatibility? I have written a few wrappers like that on my own system preventing me from making a few common mistakes of mine (like scp'ing a file locally to a filename ressembling an IP address instead of on a remote server ;)
    - Hello71 6y ago
phiresky 6y ago
> I don't know why dd remains the standard for so many tutorials You can also do something like dd if=file.iso of=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB Where the last part is basically the brand and name of your USB stick, not much guess work required. Though I've never seen that in any tutorials.
- smcameron 6y ago
  What I do to find out which is which is use lsscsi: $ lsscsi [0:0:0:0] disk ATA ST2000DX001-1NS1 CC41 /dev/sda [3:0:0:0] cd/dvd ASUS SH-224FB 1.00 /dev/sr0 [4:0:0:0] disk Generic- Multiple Reader 1.11 /dev/sdb
  - dirkt 6y ago
    Unfortunately you have to do this again after each reboot, or whenever you plug in an USB stick, so the dev/by-* option does help when you do that repeatedly for the same device.
- jeroenhd 6y ago
  It's really unfortunate how tutorials still spell out all the basics to keep the beginners with them, but then rely on the old-fashioned device names when it comes to writing images. Using sda is fine if you're experienced and know what drive has what name, but technical references should stop relying on them already. It's not like you're getting some useful cross-platform compatibility out of old-fashioned commands or anything, every modern Unix has its own way to specify disk names.
  - csydas 6y ago
    I think the simplest explanation is that the authors themselves never considered the other options or were even aware of them -- they simply are sharing some historical knowledge they had. Very kind and good behavior! But I would imagine it's something rote and probably just a shortcoming in collective knowledge. In short, I think ignorance is the simpler answer ;)
jwr 6y ago
> I don't know why dd remains the standard for so many tutorials While dd isn't the best tool for writing images to devices (if only because of its arcane and bizarre command-line syntax), it is a valuable tool when you want to recover data from media that has errors. The 'conv=noerror' option is a lifesaver for when you want to recover something from your media. But overall I agree: plenty of people recommend using dd just because "it's always been done that way" and often also because it makes them look smart :-)
- rzzzt 6y ago
  ddrescue can also help with that use case: https://www.gnu.org/software/ddrescue/ https://www.gnu.org/software/ddrescue/
  - lizknope 6y ago
    I always used ddrescue like this. Even on badly scratched discs it was able to recover most of the data. ddrescue -b 2048 -d -r 3 -R -v /dev/sr0 image.iso image.log
    - compsciphd 6y ago
      I used to use 2048, that's probably the wrong value to use. want to use something that corresponds to the ECC block size of the underlying media. but yes, I do that as well (came to say mostly the same comtent, but I'd do r -1)
      - colejohnson66 6y ago
        Is the ECC available to the OS? I thought the drive handled the ECC and just reported errors?
        wtallis 6y ago
        If you're trying to read with a smaller block size than the media's native block size, then you'll make two or more attempts to read data from each corrupted block that's unrecoverable—making your recovery process much longer. If you try to read with a larger block size than the media's native block size, you'll get errors for chunks of that size even when part of the data may have been recoverable. The above is true whether or not the OS has access to the media's raw ECC data.
loloquwowndueo 6y ago
You assume one has access to a GUI. Not always the case.
sildur 6y ago
That's another advantage of using Qubes. Worst case scenario I botch the image-burning Qube, which is temporary anyways.
megous 6y ago
One mitigation is to recommend users using symlinks in ls /dev/disk/by-id instead of /dev/sd* Other than that, someone can replace the whole of Etcher with a simple bash script that will list available block devices with nice descriptive names taken from sysfs, allow the user to select the one he wants to flash to, and handle flashing of potentially compressed image and verifying the result automatically. It would probably not be much longer than 100 LoC, at least on Linux.
TacticalCoder 6y ago
That said I remember a roommate erasing it's Windows "C:" drive using Ghost (that is "Ghost" before Norton bought it). No later than yesterday I copied my wife's entire (Windows 8.1) HDD to an SSD using the free (GUI) version of "Macrium Reflect". I'm pretty sure you can make the exact same mistake: copying destination unto source instead of the contratry. I used a Windows software and not DD for it was a Windows computer and I didn't feel like booting a live Linux CD to do the dump but under Linux, I always use dd. Is it really that hard to learn that in dd the 'i' in "if" means "input" and that the 'o' in "of" means "output"? The problem with "making things simple using a GUI" is that you typically totally lose the ability to do not just advanced things but even "average" things. For example for read-only media I always write the checksum on the media itself, using a sharpie. That way I can easily verify that my disk (or the copy) is ok doing a dd and piping into sha256sum (there are some gotchas to keep in mind but it works fine when you know how to do it). Like, say, a Debian install ISO. I like to have these on read-only medium and make sure the checksum matches the official one (so I prefer a write-once / read-only DVD than a read/write memory stick). How do you that with the GUI? Piping into a cryptographic hash? I can understand that people prefer GUI over command line for many things but I think that people imaging entire disks are at least power users and can learn the difference between "input" and "output". Heck, maybe it's even doing them a service to teach them to use the CLI. Maybe it's the opportunity to teach them about piping, about cryptographic hashes, etc. And once again: you can totally screw up with a GUI too. I don't mean it in a bad way at all but... Linux on the desktop (which I use since 20 years) ain't exactly enjoying a huge market share compared to Windows or OS X and I think that's fine. If a Linux user cannot or isn't willing to learn dd, maybe that user is better served by Windows or OS X. And really: I don't mean it in a bad way. I don't think Linux has to "win" the Desktop war. I don't think it's wrong not to use Linux. But I do think it's wrong to believe users willing to learn Linux cannot learn the difference between input and ouptut. Also a strong case can be made that someone for whom it's a "disaster" to overwrite it's main drive is one hard disk failure away from disaster anyway. I'm for educating users, not baby-feeding them with tools that are going to keep them in the dark and reinforce their bad practices (like not doing proper backups and hence being "one hard drive failure" away from disaster).
1337shadow 6y ago
Just check dmesg when you plug in your USB drive, it will tell you what device it created for it.
JasonFruit 6y ago
It's really not that hard to use correctly. Maybe it shouldn't be the standard for beginners' tutorials, sure, but it's not terrifying once you know how it works, as long as you exercise appropriate caution. And that's the whole benefit of command-line tools; they're like a chainsaw in that they get a whole lot of work done in a hurry, but you have to point them the right direction and keep them under control.
tzs 6y ago
Better permissions could help with this. Make it so that there is a user other than root that can write to the block device the flash drive is on, and then you sudo to that user to write images rather than sudoing to root. Then accidentally trying to write to the main drive will get a permission error.
alerighi 6y ago
dd has the advantage of specifying a larger buffer. I've seen for example that in MacOS writing an image to a SD card with the internal reader of my macbook is a lot faster with a block size of 1M in respect to the classical 4k. Also the advantage of dd is that you can run it with sudo easily, while with cat it will not work, since the redirection is done by the shell and not the cat binary itself, and you either have to open a root shell, or pipe the output of cat in `sudo tee filename >/dev/null` that is less than ideal. Also I think the problem is of Linux that lets you write on disks that are mounted. In MacOS is forbidden and you must umount the drive (so overwriting your root partition by error is impossible).
andrewzah 6y ago
Alternatively, I have never written to the wrong device when using dd. I always double and triple-check with lsblk and blkid before committing to it. dd is always available, unlike other tools. If a user blindly copy-pastes and destroys their main drive, that's a very good lesson to thoroughly double check before running destructive operations.
- colejohnson66 6y ago
  Just because you should do something doesn’t mean we should make it easy to screw up. And just because you and I double check doesn’t mean everyone else does. It is their fault, sure, but it’s also not when the internet guides don’t give you a massive warning about double checking.
  - kirse 6y ago
    The adventure doesn't start until things go wrong, just think of it as Linux's way of providing a lifetime of adventures and great stories, like trying to fix your parent's computer at 3am before they wake up and realize you thought you had the dual-boot Windows/SuSE setup working but now the master boot record is corrupted and you're still a Linux noob without StackOverflow or Google for answers. Battle scars my friend.
- alex_smart 6y ago
  To add to what the other user said, just because you can do something with one tool, doesn't mean we shouldn't promote other tools that are easier to use and that make it easier for you to do the job without accidentally shooting yourself in the foot.
m463 6y ago
I accidentally repartitioned my main linux drive once. I just ran gparted: $ gparted and instead of complaining that I didn't specify an argument, it chose the first one: / When I realized what I had done, I couldn't recover the partition table, but I managed to rsync everything elsewhere. I'm sure there was a better/safer way but everything was recovered in the end.
lvass 6y ago
I once mistakenly overwrote an entire disk (or wrong partition, I don't remember) when trying to install a GNU/Linux distribution to one partition using a graphical installer. Never happened using dd. Sure some graphical applications can be helpful in this regard, but there's really no reason to assume the one you end up using will have you avoid this type of mistake. I'd rather use something that is stable and be vigilant as steadily required.

dspillett 6y ago

pv's progress display is very useful in many contexts, it has been one if the first things I make sure is installed on a new system for years, though I have recently come across https://github.com/Xfennec/progress https://github.com/Xfennec/progress which is brilliant for some times when pv is not an option or when you want to watch the status of several things which aren't convenient to arrange together via tmux/byobu. Or just for when you forget to use PV and don't want to cancel to change the command.

ktpsns 6y ago
Thanks for the tip for progress! I didn't know this utility and it sounds really helpful.
- dspillett 6y ago
  I found it while putting together something similar myself (I might restart mine later, it was going to be a web & TUI based graph output, but progress+watch does the basic job more than well enough for now).

deleted 6y ago

[deleted]

yyyk 6y ago

I don't think it's a good idea to use a program that isn't aware of the device buffer to copy a CD/DVD. Such failures are rare nowadays, but installing some CLI program like cdrecord isn't difficult either.

reidrac 6y ago

I used to work on a hosting company and proper use of dd was important when copying data from LVM in a Xen host; unfortunately I seem to have forgotten most of it. Some pointers: dd has oflags; may be oflags=direct is faster? You can also use oflags=sparse and sometimes save space by creating a sparse file.

natmaka 6y ago
> may be oflags=direct is faster? oflag=direct does direct I/O => copied data won't go into the buffercache. On Linux search for 'O_DIRECT' in the open(2) manpage. oflag (for output) and iflag (for input) are indeed useful. During/after a massive non-'direct' copy a system running other processes which benefit from data in the buffercache may crawl if the system, while copying, replaces some of it by the copied data, then has to re-read it. In other terms this seems adequate when copying data which will not be, after the copy, soon read by any process. A raw filesystem image is a good candidate. As usual YMMV. If most of the data to be copied already is in the buffercache or if it will occupy some unused part of the core memory... such optimization is useless. However in most cases (on most adequately-dimensioned non-idle systems) 'O_DIRECT' induces less systemwide load than cp, cat, pv(...) when copying a large set of data if most of it will not be, then, immediately read by anything. Other tools (cp, cat, pv...) just cannot easily work in 'O_DIRECT' mode. Using some trick to enable it thank to a local version of openat() and LD_PRELOAD (which calls openat in O_DIRECT mode), albeit possible, isn't realistic in most contexts. $ cd ~/tmp $ strace -e openat dd if=/etc/hosts of=useless.tmp count=1 >& nodirect $ strace -e openat dd if=/etc/hosts of=useless.tmp iflag=direct oflag=direct count=1 >& direct $ diff direct nodirect 5,6c5,6 < openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_DIRECT) = 3 < openat(AT_FDCWD, "useless.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_DIRECT, 0666) = 3 --- > openat(AT_FDCWD, "/etc/hosts", O_RDONLY) = 3 > openat(AT_FDCWD, "useless.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 Moreover 'dd' has many options without equivalent in most other readily available tools.
- natmaka 6y ago
  Also check https://github.com/Feh/nocache https://github.com/Feh/nocache
6y ago

dmurray 6y ago

I always thought dd stood for "duplicate disk" or "duplicate device" and as such it was the "right" command to use here. Apparently it's nothing to do with that and originally comes from "data definition" though. https://en.m.wikipedia.org/wiki/Dd_(Unix) https://en.m.wikipedia.org/wiki/Dd_(Unix)

GTP 6y ago
dd is also known as "Data Destroyer", because it's easy to make a mistake and overwrite the wrong device. This actually happened to me once, but I still use it when I need to write an ISO to a USB drive anyway.
- RMPR 6y ago
  An advice that was mentioned many times in this thread to avoid writing to the wrong disk. Use the symlinks in /dev/disks/* instead of /dev/sd* directly. Since they are more intuitive, using them will likely reduce the odds of you making a mistake.
- MaKey 6y ago
  Just avoid using /dev/sdX, use /dev/disk/by-id/ instead.
_wldu 6y ago
It's original name was cc (convert and copy). Because that is what it does. They could not use that name because that was the name of the compiler. So, they went to the next letter of the alphabet and named it dd.

xorcist 6y ago

That answer does not seem very useful to me. The use cases are different. Reasons to use dd include being able to set the block size, seek and skip to resume copies or extract parts of the data, and a whole bunch of special options for the very specific use case of large binary files or block devices. Reasons to use cp include it being file based, so metadata and recursive copying is natural, efficient, and easy to use. This probably includes most copying on a daily basis. See also rsync.

jeroenhd 6y ago
One advantage dd has, specifically when it comes to copying physical media, is that you can tell it to continue in case of a read error, instead of crapping out halfway through. The lack of such functionality can make copying a scratched CD quite challenging with tools like cat or cp. You also have the choice to ignore the data that could not be read (dangerous) or to write a padded block of the correct size in its place and continue. I agree that the use cases for all tools are different so comparing them fairly is impossible. dd is still my go-to for making disk images, but it's unnecessary and has high overhead without a lot of tweaking for simple tasks like file operations. cat is often a good tool for things like piped commands where you need a flag cat supports (so you can't use shell redirection) and cp/rsync are obviously superior for copying files rather than just data. Sure, uu could hack my commandsin such a way that one tool can do another's job, but why would I?
IgorPartola 6y ago
Notice that they way they use cp isn’t to copy the files contained on the disc but to copy the image of the disc itself.

IgorPartola 6y ago

The buried lede here is that cat can be faster than dd: https://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/9492#9492 https://unix.stackexchange.com/questions/9432/is-there-a-way... The other important part is that cat will not have issues with binary data, which I am embarrassed to admit I assumed was not the case for the reasons stated in TFA. pv is the tool I want to use for things like copying images to SD cards but I can never remember the exact syntax needed to get it to show me the percentage since I don’t use it often. dd has a decent syntax for its command parameters and has some fun options like ability to skip, so I can quickly create large empty files.

Aachen 6y ago
> pv is the tool I want to use for things like copying images to SD cards but I can never remember the exact syntax needed to get it to show me the percentage You're in luck: the "syntax needed" is to literally not use any options: pv your.img > /dev/sdb
- IgorPartola 6y ago
  Haha good point. The first time I encountered pv it was in the form of like cat your.img | pv ... > /dev/sdb and then you had to tell it what size to expect and enable it showing the progress bar, etc.
  - syncsynchalt 6y ago
    Good ol' "Useless Use of Cat": http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
  - Aachen 6y ago
    True, if you pipe a data stream to it, it can't tell what the size is going to be and you'd need the --size option. The progress bar is still default, though.

dfox 6y ago

Many years ago I wrote https://github.com/adh/random-tools/blob/master/byc.c https://github.com/adh/random-tools/blob/master/byc.c which tends to be the tool that I use for copying images from/to block device (it was originally meant for something else which is why you need -p for it to do the right thing).

pontifier 6y ago

I've been dealing with a lot of CD copying problems lately. I'm working on a setup to automatically detect and correct errors reading audio CDs which I've found don't use checksums or other mechanisms to prevent small defects from mangling the bitstream. I'd previously used cdparanoia, but found it not paranoid enough.(for some reason it always outputs aiff regardless of options as well) I'm looking at defeating the drive cache by actually reading the disc in a second and 3rd drive and doing a NASA style "all 3 must agree". It's annoying, but I've got 800,000 CDs to rip right now, and I have to make sure each one is perfect.(Without proprietary solutions) I also need to be able to positively reduplicate heavily scratched discs. It's a tough problem.

unnouinceput 6y ago
800k? are you insane? even if everything else is automated and you have 800K devices at your fingertip ready to go, the simple amount of time to lift CD, open player tray / insert CD / close tray is in decades!. Let's say you do this 8 hours per day for 5 days. The above series of movements can take around 30 seconds. That's 400k minutes which results in 400k / 40 hours == 10k weeks. At 50 weeks per year (only 2 weeks vacation each year) that still results in 200 years!!. Even if you do one CD every second this result in 200 years / 30 ==~7 years.
- pontifier 6y ago
  I very well might be. I took over Murfie when they shut down, and am in the process of re-building the company. People don't understand the scale, and why I might want to make sure I do it right the first time rather than get halfway through and have to start over because I missed something.
  - gruez 6y ago
    I'm assuming you're the guy in this article? https://www.theverge.com/2020/2/5/21121594/crossies-murfie-madison-wisconsin-arkansas-1-million-abandoned-cds https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...

cjdell 6y ago

I'm so glad that DD now has the status=progress option. Still not widely known.

TacticalCoder 6y ago
I used to cat dd into "pv" : )
- hyperdimension 6y ago
  Before I learned of the flag, I used to do a 'while true; do killall -SIGUSR1 dd; sleep 5; done' loop in another terminal.
  - saagarjha 6y ago
    SIGINFO on macOS.

andrewmackrodt 6y ago

Do any of the tools mentioned in the title support audio and/or multi-session disc formats? AFAIK dd is not suitable for these purposes (although the original author mentions they wish to create ISO images so presumably they only care about single session data). Windows had a lot of great disc software in the 90s/00s which would handle just about any disc format one would encounter, e.g. Alcohol 120%, CloneCD, Disc Juggler. Some of these were capable of backing up the various protections of the time such as SecureROM and Starforce. dd is a powerful tool but I often find it the "wrong" tool, e.g. is it desired to clone a 2GB partition layout to a 128GB usb stick for a live CD and have to manually edit my partition tables myself?

abaines 6y ago

I've always used cdrdao read-cd --read-raw to backup CD/DVDs, along with toc2cue and bchunk to convert to .iso. I have no idea if the resulting iso is "more accurate" than using dd - but they've always worked, even from CDs with copy protection and obscure file systems (old mac CDs).

pontifier 6y ago

I use tail all the time with "-f" to see log files in realtime, but seeing it as an option for reading an entire CD with "-c +1" was surprising.

maerF0x0 6y ago

Performance. This is an I/O-bound process; the main influence in performance is the buffer size: the tool reads a chunk from the source, writes the chunk to the destination, repeats. If the chunk is too small, the computer spends its time switching between tasks. If the chunk is too large, the read and write operations can't be parallelized. The optimal chunk size on a PC is typically around a few megabytes but this is obviously very dependent on the OS, on the hardware, and on what else the computer is doing. I made benchmarks for hard disk to hard disk copies a while ago, on Linux, which showed that for copies within the same disk, dd with a large buffer size has the advantage, but for cross-disk copies, cat won over any dd buffer size. As a meta comment, it has been a shocking revelation in my career that the difference between "get it work" engineers and "make it right, make it fast"[3] has been this kind of low level knowledge. At least in my context, the majority of the bugs that have come up in the past while haven't been because someone didn't know how to concatenate 2 strings or even that many off by 1 errors. They have been things like not knowing the difference between theoretical TCP models and how their actually implemented on the OS in a high throughput environment (see TIME_WAIT[1] if you're curious), or understanding misorder of operations in distributed computing[2]. [1]: https://9oelm.github.io/2018-05-06--Listening,-Established,-Close_wait-and-Time_wait-in-netstat/ https://9oelm.github.io/2018-05-06--Listening,-Established,-... [2]: http://book.mixu.net/distsys/ http://book.mixu.net/distsys/ [3]: https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast

tyingq 6y ago
Protocols that don't have a length header, but instead rely on delimiters is a good example. And, they are everywhere.

tyingq 6y ago

One non-obvious reason dd is often used in examples is because of device permissions. You can't (easily/legibly) present a one-liner that combines cat and sudo because sudo doesn't give the shell redirection (sudo cat >> /dev/whatever) permissions. But "sudo dd of=/dev/whatever" works fine. Edit: Yes, you can make a one liner via sh -c, tee, etc, but depending on cut/paste with quotes is tricky, or the one-liner gets long.

colejohnson66 6y ago
You can also just use `su`, run your command, then `exit`, but you’re right: `sudo dd` is much easier. (This obviously ignores that you should almost never use `su`)
- na85 6y ago
  > you should almost never use `su` Why?
  - colejohnson66 6y ago
    The same reason you should be logged in as root. It has its uses, but you need to be careful as it can be easy to forget you’re running as root and screw something up. Explicitly putting `sudo` before every command makes it clear what you’re doing.

blincoln 6y ago

Any time CDs/CD-ROMs (as opposed to DVDs) are discussed, I get really nervous unless someone explicitly confirms that the tool being used handles XA/2352-bytes-per-sector data correctly. Supposedly cdrdao will.[1] DVDs are for the most part pretty easy to duplicate identically. CDs have a lot of quirks, and it's very easy to end up with a useless "backup". Especially if the file format is ISO instead of bin/cue or similar. If one is trying to duplicate CDs/CD-ROMs, it's really vital to cross all your Ts and dot all your Is. [1] https://consolecopyworld.com/psx/psx_copy_patch_linux.shtml https://consolecopyworld.com/psx/psx_copy_patch_linux.shtml

teej 6y ago
Is there a reason that the copying process needs to be aware of how bytes are organized? Why can’t you just read the bytes from disk A and write those exact bytes to disk B?
- snarfy 6y ago
  Which bytes? A raw 2352 byte copy that includes the P and Q channels (timing) can behave differently than a 2048 byte copy depending on the software.
  - harry8 6y ago
    Ok but why should we care about that in copying? This is my mental model for what's going on: There's a bunch of bytes stored on a CD-ROM in a defined order. Zeroes and ones. Copy them in order. You should now have a file on magnetic disk or flash that is precisely those byes in precisely that order. Anything that can make sense of one should be able to make sense of the other. What am I missing here?
    - snarfy 6y ago
      There is more than one set of bytes. You may not care about copying the inode structure when you are copying your files from point a to point b, but you should when you are cloning the disk. Many times the software on the CD is looking at the physical layout of the disk, not just the logical data, to function correctly.

codethief 6y ago

What's the best way to rip DVDs through the CLI these days? Are the error correction techniques in the DVD standard enough to ensure that a simple `cat /dev/dvd0` or `dd /dev/dvd0` is going to work (and, in particular, fail if an error can't be corrected)?

saagarjha 6y ago

For those using dd (or cp) on macOS, ⌃T will send the process a SIGINFO and print some additional information about progress. On Linux, SIGUSR1 will do something similar, but it's not as convenient.