10 ms·
The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to th
by AdmiralAsshat 6y ago
The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb.
I don't know why dd remains the standard for so many tutorials when there are competent GUI tools that can take all the guesswork out of it. It's like a perpetual hazing ritual for new Linux users.
- extraduder_ire 6y agodd can be faster than alternatives, if you set the block size right, and is always available, even without a GUI. I've often used whatever disk utility came with my distro. Or, rufus (rufus.ie) if I'm on windows.
- sjburt 6y agoYou can also just use cp, and then you don't need to worry about the block size. Really only need dd if you have to limit how much data is read.
- sitzkrieg 6y agoits simple, already installed and has huge existing tow. at least with newer classes of device storage drivers like nvme you end up with completely different names for system drives!
- inetknght 6y ago> I don't know why dd remains the standard for so many tutorials when there are competent GUI tools You miss the point entirely then. The point is to use a command line because what you think is competent for a GUI tool isn't nearly as intuitive as you think especially when a GUI isn't available.
- tinus_hn 6y agoWhy do you think dd is special in that it can overwrite devices? Do you think ‘cat’ can’t? The root user is special in that it can overwrite devices. Look at what you’re typing before using its powers.
- nemosaltat 6y agoThe “simple GUI” tool I keep seeing pushed Maker forums is Etcher. I needed to flash something from my work PC so I gave it a shot- 90MB of Electron bloat, while offering no extra functionality over dd . Plus, dd doesn’t advertise to me during the flash, and certainly doesn’t phone home to balena. For folks just getting started or more comfortable with a GUI, I’d recommend giving USBimager[0] a look. It does exactly what’d you’d expect based on the name, it performant, and they have native apps. No affiliation, just a fan of a KISS app done right. [0] https://gitlab.com/bztsrc/usbimager https://gitlab.com/bztsrc/usbimager
- trinix912 6y agoThere's also Rufus, sadly it's Windows-only.
- ccmcarey 6y agoWoeusb is pretty decent on linux, and does the same uefi-ntfs magic that Rufus does, and that you'll need if you're burning a large ISO like recent Windows ones. Though if it's just for ISOs, ventoy is fantastic, just drag and drop the file and no burning at all :)
- jorvi 6y ago> that you'll need if you're burning a large ISO like recent Windows ones. I ran into this very issue when trying to make a bootable Windows 10 USB on macOS. No amount of fiddling with dd, unetbootin or Etcher resulted in a bootable USB. Despite being principled about it, I had to admit defeat and just pulled an old <4GB Windows 10 iso and flashed that to the stick. I know I could have installed Linux through a virtual machine and got it done that way, but that seemed horrible overkill. Oh well.
- moondev 6y agoDouble click iso to mount it, format usb as fat32 or exfat, copy and paste contents of mounted iso to usb drive This has always worked for me to boot UEFI installers
- zenlot 6y ago> and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb One trick to be sure that you've typed correctly, start with 'echo': # echo dd if=xxx of=yyy [enter] This way you'll be able to check that command will be executed with the parameters you want. Especially helpful if you're running loops, e.g: # for i in xxx; do echo yyy; done I would never use GUI for that, as you can't be sure what it will execute, no matter what it shows.
- Kwpolska 6y agoHow would `echo dd if=linux.iso of=/dev/sda` protect you from nuking the wrong drive? To be sure you aren’t overwriting the wrong drive, you need to read the command before executing it — and you don’t need echo for that, just pause for a second before hitting Enter — and you need to know which drive is which, and that requires running a different command, or using eg. /dev/disk/* instead of /dev/sd??.
- deleted 6y ago[deleted]
- skissane 6y ago> The prospect of using dd for imaging terrifies me to this day, primarily because you still hear horror stories of people trying to write a Linux installer to their flash drive to try out, and accidentally overwriting their main drive instead because they mangled /dev/sda instead of /dev/sdb. Wouldn't one (at least partial) solution here be, that the kernel should refuse to let you write directly to a block device in use by a mounted filesystem? (Maybe combined with some special ioctl/whatever to bypass that restriction if you ever really need to.) Then, if you are running dd from an OS running on your main drive, the kernel will refuse to let dd overwrite the main drive, but will let it overwrite the flash drive (which presumably is not mounted, and anyway shouldn't be if you are about to overwrite it)
- Hello71 6y agoThere are two major problems with this: partitions/device-mapper and backwards compatibility. First, partitions inherently overlap the parent block device. You would need to carefully track exactly which portions of the device are in use, or your solution is useless (it doesn't protect /dev/sda when /dev/sda1 is in use) or blocks the vast majority of use cases (/dev/sda2 cannot be used when /dev/sda1 is in use). The same applies for device-mapper, but worse. Secondly, the Linux kernel has a very strong backwards compatibility guarantee. Any change that would break valid (or even invalid) uses will be loudly rejected by Linus. With your idea, many disk management programs will be broken.
- TacticalCoder 6y agoYou could though probably make a wrapper that only kicks in when dd is called from the command line (as opposed to, say, from a script or some GUI program) and asking a confirmation "Do you really want to overwrite your main drive?", without breaking backward compatibility? I have written a few wrappers like that on my own system preventing me from making a few common mistakes of mine (like scp'ing a file locally to a filename ressembling an IP address instead of on a remote server ;)
- Hello71 6y ago
- phiresky 6y ago> I don't know why dd remains the standard for so many tutorials You can also do something like dd if=file.iso of=/dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB Where the last part is basically the brand and name of your USB stick, not much guess work required. Though I've never seen that in any tutorials.
- smcameron 6y agoWhat I do to find out which is which is use lsscsi: $ lsscsi [0:0:0:0] disk ATA ST2000DX001-1NS1 CC41 /dev/sda [3:0:0:0] cd/dvd ASUS SH-224FB 1.00 /dev/sr0 [4:0:0:0] disk Generic- Multiple Reader 1.11 /dev/sdb
- dirkt 6y agoUnfortunately you have to do this again after each reboot, or whenever you plug in an USB stick, so the dev/by-* option does help when you do that repeatedly for the same device.
- jeroenhd 6y agoIt's really unfortunate how tutorials still spell out all the basics to keep the beginners with them, but then rely on the old-fashioned device names when it comes to writing images. Using sda is fine if you're experienced and know what drive has what name, but technical references should stop relying on them already. It's not like you're getting some useful cross-platform compatibility out of old-fashioned commands or anything, every modern Unix has its own way to specify disk names.
- csydas 6y agoI think the simplest explanation is that the authors themselves never considered the other options or were even aware of them -- they simply are sharing some historical knowledge they had. Very kind and good behavior! But I would imagine it's something rote and probably just a shortcoming in collective knowledge. In short, I think ignorance is the simpler answer ;)
- jwr 6y ago> I don't know why dd remains the standard for so many tutorials While dd isn't the best tool for writing images to devices (if only because of its arcane and bizarre command-line syntax), it is a valuable tool when you want to recover data from media that has errors. The 'conv=noerror' option is a lifesaver for when you want to recover something from your media. But overall I agree: plenty of people recommend using dd just because "it's always been done that way" and often also because it makes them look smart :-)
- rzzzt 6y agoddrescue can also help with that use case: https://www.gnu.org/software/ddrescue/ https://www.gnu.org/software/ddrescue/
- lizknope 6y agoI always used ddrescue like this. Even on badly scratched discs it was able to recover most of the data. ddrescue -b 2048 -d -r 3 -R -v /dev/sr0 image.iso image.log
- compsciphd 6y agoI used to use 2048, that's probably the wrong value to use. want to use something that corresponds to the ECC block size of the underlying media. but yes, I do that as well (came to say mostly the same comtent, but I'd do r -1)
- colejohnson66 6y agoIs the ECC available to the OS? I thought the drive handled the ECC and just reported errors?
- wtallis 6y agoIf you're trying to read with a smaller block size than the media's native block size, then you'll make two or more attempts to read data from each corrupted block that's unrecoverable—making your recovery process much longer. If you try to read with a larger block size than the media's native block size, you'll get errors for chunks of that size even when part of the data may have been recoverable. The above is true whether or not the OS has access to the media's raw ECC data.
- loloquwowndueo 6y agoYou assume one has access to a GUI. Not always the case.
- sildur 6y agoThat's another advantage of using Qubes. Worst case scenario I botch the image-burning Qube, which is temporary anyways.
- megous 6y agoOne mitigation is to recommend users using symlinks in ls /dev/disk/by-id instead of /dev/sd* Other than that, someone can replace the whole of Etcher with a simple bash script that will list available block devices with nice descriptive names taken from sysfs, allow the user to select the one he wants to flash to, and handle flashing of potentially compressed image and verifying the result automatically. It would probably not be much longer than 100 LoC, at least on Linux.
- TacticalCoder 6y agoThat said I remember a roommate erasing it's Windows "C:" drive using Ghost (that is "Ghost" before Norton bought it). No later than yesterday I copied my wife's entire (Windows 8.1) HDD to an SSD using the free (GUI) version of "Macrium Reflect". I'm pretty sure you can make the exact same mistake: copying destination unto source instead of the contratry. I used a Windows software and not DD for it was a Windows computer and I didn't feel like booting a live Linux CD to do the dump but under Linux, I always use dd. Is it really that hard to learn that in dd the 'i' in "if" means "input" and that the 'o' in "of" means "output"? The problem with "making things simple using a GUI" is that you typically totally lose the ability to do not just advanced things but even "average" things. For example for read-only media I always write the checksum on the media itself, using a sharpie. That way I can easily verify that my disk (or the copy) is ok doing a dd and piping into sha256sum (there are some gotchas to keep in mind but it works fine when you know how to do it). Like, say, a Debian install ISO. I like to have these on read-only medium and make sure the checksum matches the official one (so I prefer a write-once / read-only DVD than a read/write memory stick). How do you that with the GUI? Piping into a cryptographic hash? I can understand that people prefer GUI over command line for many things but I think that people imaging entire disks are at least power users and can learn the difference between "input" and "output". Heck, maybe it's even doing them a service to teach them to use the CLI. Maybe it's the opportunity to teach them about piping, about cryptographic hashes, etc. And once again: you can totally screw up with a GUI too. I don't mean it in a bad way at all but... Linux on the desktop (which I use since 20 years) ain't exactly enjoying a huge market share compared to Windows or OS X and I think that's fine. If a Linux user cannot or isn't willing to learn dd, maybe that user is better served by Windows or OS X. And really: I don't mean it in a bad way. I don't think Linux has to "win" the Desktop war. I don't think it's wrong not to use Linux. But I do think it's wrong to believe users willing to learn Linux cannot learn the difference between input and ouptut. Also a strong case can be made that someone for whom it's a "disaster" to overwrite it's main drive is one hard disk failure away from disaster anyway. I'm for educating users, not baby-feeding them with tools that are going to keep them in the dark and reinforce their bad practices (like not doing proper backups and hence being "one hard drive failure" away from disaster).
- 1337shadow 6y agoJust check dmesg when you plug in your USB drive, it will tell you what device it created for it.
- JasonFruit 6y agoIt's really not that hard to use correctly. Maybe it shouldn't be the standard for beginners' tutorials, sure, but it's not terrifying once you know how it works, as long as you exercise appropriate caution. And that's the whole benefit of command-line tools; they're like a chainsaw in that they get a whole lot of work done in a hurry, but you have to point them the right direction and keep them under control.
- tzs 6y agoBetter permissions could help with this. Make it so that there is a user other than root that can write to the block device the flash drive is on, and then you sudo to that user to write images rather than sudoing to root. Then accidentally trying to write to the main drive will get a permission error.
- alerighi 6y agodd has the advantage of specifying a larger buffer. I've seen for example that in MacOS writing an image to a SD card with the internal reader of my macbook is a lot faster with a block size of 1M in respect to the classical 4k. Also the advantage of dd is that you can run it with sudo easily, while with cat it will not work, since the redirection is done by the shell and not the cat binary itself, and you either have to open a root shell, or pipe the output of cat in `sudo tee filename >/dev/null` that is less than ideal. Also I think the problem is of Linux that lets you write on disks that are mounted. In MacOS is forbidden and you must umount the drive (so overwriting your root partition by error is impossible).
- andrewzah 6y agoAlternatively, I have never written to the wrong device when using dd. I always double and triple-check with lsblk and blkid before committing to it. dd is always available, unlike other tools. If a user blindly copy-pastes and destroys their main drive, that's a very good lesson to thoroughly double check before running destructive operations.
- colejohnson66 6y agoJust because you should do something doesn’t mean we should make it easy to screw up. And just because you and I double check doesn’t mean everyone else does. It is their fault, sure, but it’s also not when the internet guides don’t give you a massive warning about double checking.
- kirse 6y agoThe adventure doesn't start until things go wrong, just think of it as Linux's way of providing a lifetime of adventures and great stories, like trying to fix your parent's computer at 3am before they wake up and realize you thought you had the dual-boot Windows/SuSE setup working but now the master boot record is corrupted and you're still a Linux noob without StackOverflow or Google for answers. Battle scars my friend.
- alex_smart 6y agoTo add to what the other user said, just because you can do something with one tool, doesn't mean we shouldn't promote other tools that are easier to use and that make it easier for you to do the job without accidentally shooting yourself in the foot.
- m463 6y agoI accidentally repartitioned my main linux drive once. I just ran gparted: $ gparted and instead of complaining that I didn't specify an argument, it chose the first one: / When I realized what I had done, I couldn't recover the partition table, but I managed to rsync everything elsewhere. I'm sure there was a better/safer way but everything was recovered in the end.
- lvass 6y agoI once mistakenly overwrote an entire disk (or wrong partition, I don't remember) when trying to install a GNU/Linux distribution to one partition using a graphical installer. Never happened using dd. Sure some graphical applications can be helpful in this regard, but there's really no reason to assume the one you end up using will have you avoid this type of mistake. I'd rather use something that is stable and be vigilant as steadily required.