5 ms·
Useful uses of dd(1). There remain some useful applications of dd. These may of course be achieved by other mechanisms, but typically less conveniently. 1. R
by dredmorbius 4y ago
Useful uses of dd(1).
There remain some useful applications of dd. These may of course be achieved by other mechanisms, but typically less conveniently.
1. Read a specific number of blocks or bytes from a source:
dd if=/dev/hda of=/root/mbr bs=512 count=1
This will make a copy of, say, your master boot record (first 512 bytes of your first disk drive) and stash it in your /root directory.
2. Read from specific bytes of a file
dd if=mydata skip=1k bs=32 count=1
Reads 32 bytes after the first 1024 (1k) bytes of "mydata".
3. Write to specific bytes of a file
dd if=source of=target seek=10k bs=512 count=1 conv=notrunc
That should write 512 bytes from "source" beginning 10k into "target". (I've not tested this, you should verify.)
4. Create a sparse file. Sparse files appear to have a nonzero size, but take up no space on disk, until data is actually written to them. These are often used as "inflating" dynamic filesystem images for virtual machines.
dd if=/dev/zero of=sparsefile bs=1 count=0 seek=20000M # Create 20 GB sparse file
5. Case conversions. Sure, you could use tr(1), but where's the sport?
dd if=MixEdCaSE of=lcase conv=lcase # Convert to lower case
dd if=MixEdCaSE of=ucase conv=ucase # Convert to upper case
6. ASCII / EBCDIC conversions
dd if=ebcdic of=ascii conv=ascii # ebcdic -> ascii
dd if=ascii of=ebcdic conv=ebcdic # ascii -> ebcdic
When reading to or from IBM data tapes, you might find blocking / unblocking conversions useful. I've done this, but it's so long ago that I don't trust my memory on that any more. Odds are good you'll not have to worry about this.
There are other useful applications as well, though these are not typically encountered very often. Do feel free to explore and attempt these on safe media.
- cperciva 4y agocreate a sparse file Note that this can also be done using truncate(1).
- Nux 4y agoAlso fallocate (Linux only though).
- natmaka 4y agodd can do it on an existing file or stream, transforming sequences of 0 it contains into "holes".
- inopinatus 4y agocaveat operator: to ensure the conv=sparse operand achieves the desired outcome, be sure to use an output blocksize equal to st_blksize of the output filesystem.
- jhugo 4y agoor just use `cp --sparse=always` if it's from one file to another
- inopinatus 4y agoThat is not so portable, so I recommend sticking with dd.
- jhugo 4y agoYup, `dd` helps in a lot of situations when you need portability, not just this one. Commands like `cp --sparse=always` aren't worth such a blanket disrecommendation though; if you are working directly at a console as opposed to scripting you typically don't need portability.
- inopinatus 4y agoAgain, most of the consoles I work at don’t use GNU coretools.
- jhugo 4y agoThen you obviously can't use this, but many people reading my suggestion can.
- Waterluvian 4y agoApologies. Tangent: What does the (1) mean beside dd? I see this with man pages. Is it a version identifier? Edit: thank you both for taking the time to share. I appreciate the quick response.
- colejohnson66 4y agoThe “section”[0] or “subpage.” Programs are section 1, hence ‘dd(1)’ The idea is that it can be possible for a library function (section 3) to have the same name as an executable. In which case, you’d type into your shell: man 1 <name> # executable: <name>(1) man 3 <name> # function: <name>(3) Without the distinction argument, man will throw its arms up in the air and give up. However, if there’s no ambiguity (such as with ‘dd’ where only the executable exists), you can drop the section number parameter when running man. man will then search all the sections, see that ‘dd’ only exists in section 1, and go from there. [0]: https://linux.die.net/man/ https://linux.die.net/man/
- dredmorbius 4y agoCorrect. I also use the notation as a convention to indicate that I'm referring to a Unix command (or library function, etc.). E.g., "cat" might refer to a feline, but "cat(1)" should more clearly refer to the Unix / Linux command. (Where I've got proper markup, I'll typically set commands or references in monospace using backtick notation: `dd`, `cat`, etc.
- zimpenfish 4y ago> Read from specific bytes of a file Especially handy when you've fed in a huge amount of JSON (sometimes all on one line because, y'know, why not) into jq and you get the inscrutable output: parse error: Invalid numeric literal at line 1, column 236162512
- dredmorbius 4y agoUgh! I'll keep that in mind, json-parsing being among my current hobbies...
- nemetroid 4y agoTail can do this, too: tail --bytes=+236162512
- zimpenfish 4y agoHandy but `dd` is much better at giving you a limited context to look at - `dd bs=1 skip=236162504 count=20' just gives you 8 before and 12 after. (edit: I do have to concede that the `tail|head` version is much faster than the `dd` - ~11s vs ~65s in my quick test with that ^skip)
- dredmorbius 4y agoI suspect you'd get better performance increasing the blocksize, and if necessary trimming the output in a second pass. Large blocks -> efficient I/O. Within reason.
- nrclark 4y agoA note here: if you're using GNU dd, you can also use iflag=count_bytes and then set the block-size to whatever you want. That'll give you the best of both worlds.
- zimpenfish 4y agoAh, good to know. With a block size of 1M, that brings the `dd` version down to ~15s.
- photon-torpedo 4y agoCareful, your #2 and #3 are incorrect -- skip and seek operate with blocks, not bytes. So your #2 would copy 32 bytes after the first 32kB of data, and #3 would write 512 bytes at position 5120k.
- _flux 4y agoBtw, there are iflags in Gnu dd to work in bytes as well. I have ddbytes aliased to dd iflag=count_bytes,skip_bytes oflag=seek_bytes .
- dredmorbius 4y agoThanks. Most of that was from memory and not tested. The specific recipies should be vetted. The stated goals can be achieved with proper invocation.
- matja 4y ago7. Write a new MBR to a disk, keeping the partition table: dd bs=440 count=1 if=/usr/lib/syslinux/bios/mbr.bin of=/dev/sda Even in the age of EFI/GPT, that still gets used often (usually VM providers that only offer MBR boot).
- jhugo 4y agohead -b 440 /usr/lib/syslinux/bios/mbr.bin > /dev/sda
- inopinatus 4y agoI suspect you meant -c 440; I can’t find a variant of head(1) that has a -b operand on any Unix. Note that -c is not POSIX but does have widespread support. Notably missing on Solaris. Fun fact, the -c usage comes from ksh, where head is a shell builtin.
- jhugo 4y agoOops, yes, -c indeed! Thanks!
- rocqua 4y agoFor the first option a simple head -b 512 Will also copy the first 512 bytes in case you want to avoid dd for clarity. I have actually used that for moving mbrs around.
- tyingq 4y agoThough "-b" is a gnuism, and isn't there at all on many unixy OSes, or is "-c" on others.
- jimmaswell 4y agoWhat drives whoever's the second person to implement such a flag to make it different? An explicit desire to inconvenience other "tribes" of computer users and use the flag as a symbol of an in-group?
- dredmorbius 4y agoHow do you change, improve, or extend a standard if no changes may be permitted? What's keeping other implementations from adding these features?
- jimmaswell 4y agoChanges with a good reason are fine, but I don't see a good reason to make the same flag a different letter.
- tyingq 4y agoApparently a typo on the post I was replying to...there is no "-b" switch, it's "-c" or "--bytes" for gnu head. Though there are versions of head without one or both of "-c", "--bytes".
- scbrg 4y agoFollowing a convention already established in the local ecosystem, presumably. Though in this case I think previous posters are mistaken. My GNU implementation of head supports -c and not -b. Perhaps it's the fact that the longopt is --bytes that caused the confusion.
- gnubison 4y agoFun fact: 1-4 don’t work in the context of short reads — and GNU’s fullblock extension isn’t specified in POSIX.
- yepguy 4y agoMy most common use for `dd` is using it with `sudo` to direct the output of a unprivileged pipeline to a root-owned file. Instead of running `echo hello >/root/test.txt`, which will fail, I use `echo hello | sudo dd of=/root/test.txt`.
- nrclark 4y agoa note: I'd recommend using tee instead of dd for that job, or add iflag=fullblock if your dd supports it. The thing is that dd issues a read() for each block, but is doesn't actually care how many bytes it gets back in response (unless you turn on fullblock mode). This isn't really a problem when you're reading from a block device, because it's pretty uncommon to get back less data than you requested. But when you're reading from a pipe, it can/does happen sometimes. So you might ask for five 32-byte chunks, and get [32, 32, 30, 32, 32]-sized chunks instead. This has the effect of messing up the contents of file you're writing, with possibly destructive effects. To avoid it, use `tee` or something else. Or use iflag=fullblock to ensure that you get every byte you request (up to EOF or count==N).
- yepguy 4y agoI've never had any trouble, but good to know.