5 ms·

Oh thank you for submitting this to HN. I’ve been telling people not to use dd for years and everyone looks at me like I just gave birth to a full grown dinosa

Oh thank you for submitting this to HN. I’ve been telling people not to use dd for years and everyone looks at me like I just gave birth to a full grown dinosaur or something. “Well why does the entire internet say to use dd then?” Because they copy from each other just like you copied from them. Just use cat.

HomeGear 4y ago
>“Well why does the entire internet say to use dd then?” Because they copy from each other just like you copied from them. Great statement. This brings me some much needed internal clarity on my own thoughts and actions.
- chubot 4y ago
  This is basically the parable of "Grandma's Ham" https://www.executiveforum.com/cutting-off-the-ends-of-the-ham/ https://www.executiveforum.com/cutting-off-the-ends-of-the-h... tl;dr nobody in 2 generations knows why they cut the ends off the ham before cooking it, until they talked to grandma, who said her pan was too small A Unix thing that's been posted to HN for a decade, that's almost literally the same story: Understanding the bin, sbin, usr/bin , usr/sbin split https://news.ycombinator.com/item?id=3519952 https://news.ycombinator.com/item?id=3519952 tl;dr /usr/bin is separate from /bin because someone had a small hard disk once
  - ahartmetz 4y ago
    So Grandma's Ham is a variant of Chesterton's Fence... When you do or don't do something for reasons of tradition, find the real reasons for doing / not doing it.
- soneil 4y ago
  You’d be amazed how many people I’ve found that do ‘tar xzf filename’ without knowing what xzf is doing.
  - jthrowsitaway 4y ago
    Passing "z" is pretty pointless for extractions.
    - schoen 4y ago
      I think it was required until about 2010 or so for GNU tar. (I still use it from habit.) There was a problem about integrating support for different compression applications, with each one needing to get a new letter in the tar command!
      - sneusse 4y ago
        Won't this fail when the file is not gzipped but, for example, zstd compressed?
        jhugo 4y ago
        It doesn't assume gzip, it detects the compression format.
      - jhugo 4y ago
        Yup, I remember when you had to pass `z` if it was gzip, and I remember my surprise when I missed it once and it still worked (apparently about 5 years after it was no longer needed!)
  - sam_bristow 4y ago
    I'll admit to being one of those people. My brain still spells it out as eXtract Ze Files every time :-P
    - Izkata 4y ago
      You're 2/3 correct: x - extract z - gzip format f - file (must be last so it parses the filename arg correctly I also add "vv" in the middle so it lists every file as it goes, so I can see it work instead of just waiting with no output.
throwawaylinux 4y ago
What if you need to test direct IO, or test various partial-page, partial-block, or full block/page access to pagecache? Sometimes you might actually want to try using direct IO too, if buffered IO is not giving you sufficient control or reporting of errors from the device, or continuation from part-way through the copy, and in some cases possibly because you don't want to consume memory for page cache (although dd isn't good with direct IO so that's probably unlikely). That said if you don't know whether or not you need it, you don't. And those who do would know not to listen to your advice (for what they need to do) anyway, so it's not bad general advice to give.
- spudlyo 4y ago
  Direct I/O is notoriously finicky. Depending on what filesystem you're using there are apparently alignment restrictions and other corner cases to think about. I don't think it's that `dd` isn't good with Direct I/O, I just think its hard to use.
  - throwawaylinux 4y ago
    Direct IO is not finicky (on Linux, since 2.4 era kernels). It's just that at least with dd, if you don't know what it does then it probably does not do what you want (synchronous, single queue depth). If you know what you want and know what it does, then it's fine to use, and can be useful (I used it today to measure block device interrupt latency in a VM).
RaoulP 4y ago
Well, I’d like to thank the author for writing this. I always enjoy having the veil of “magic” lifted from computers. UNIX’s simplicity of “everything is a file” continues to surprise me, even though it shouldn’t any more. I think a lack of confidence in my understanding of the basics, leaves me tempted to copy from others as you mention.
- koala_man 4y ago
  Author here. You're welcome ^^
jhardy54 4y ago
> Just use cat. We’ve gone full circle. https://en.m.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat https://en.m.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_ca...
- nikau 4y ago
  The reason to use cat is so you don't risk accidentally overwriting the source file by messing up a command. Anything after the "cat file |" can't overwrite the file.
  - jhugo 4y ago
    That seems like a bad reason. You can always overwrite the file by messing up the command. cat file | cat > file (This won't, as you might hope, leave the file unchanged.) Re-read what you typed before hitting enter, and keep backups of important things.
    - nikau 4y ago
      That's an unlikely mistake. More likely is you are messing around with some new/infrequently used utility and you pass in the arguments incorrectly and specify "file" as the output instead of the input as intended.
      - jhugo 4y ago
        The one I typed is unlikely (I wrote it in response to your "Anything after the `cat file |` can't overwrite the file", which is factually incorrect as demonstrated), but there are plenty of other simple mistakes. Understand what you're running, and keep backups, otherwise you'll screw up eventually. Selecting your tools this way doesn't move the needle much.
        nikau 4y ago
        Removing cat and specifying the file directly to the utility doesn't move the needle in any tangible way either. At the end of the day do whatever pattern works for you - for me it's doing "cat file |" at the start of a pipeline and "> outfile" at the end. I also avoid globbing inside a pipeline as it can be dangerous too.
Claude_Shannon 4y ago
I've been taught at my uni to use dd...
mort96 4y ago
"Well why does the entire internet say to use dd then?" Because you usually need root to write to drives, and you can't `sudo` redirect. `sudo dd if=whatever.img of=/dev/whatever` is nicer and easier than `sudo sh -c "cat whatever.img >/dev/whatever"`.
- dinom 4y ago
  In the cases where you need root it's often even easier to use the 'tee' command instead of spawning another shell; esp. if the command has symbols which need to be escaped. Your example would then become something like... cat whatever.img | sudo tee /dev/whatever > /dev/null # https://stackoverflow.com/questions/82256 https://stackoverflow.com/questions/82256 Just like in household plumbing, the 'tee' command basically takes the input and sends it to more than one place. Naturally, running 'sudo tee' will let you send things all over but as another user. All that said, I won't speak to the comment "why does the entire internet say to use dd". I've never actually found the "whole internet" to agree on much of anything (-:
shrimp_emoji 4y ago
What initially mystified me into thinking "I guess I have to use `dd` for block devices" was learning that `rsync` doesn't work with block devices. I just tried to copy an ISO to a USB drive with `rsync`, and it didn't work. Looking it up, I read these comments[0]: "rsync operates on files which are on a filesystem. It doesn’t do comparisons between blocks. If you want to back up a block device to a file, use dd" "I completely understood the use case. rsync (still) does not operate on block devices, so it’s not the solution here." "Agreed, rsync has never operated on block devices." So I thought, "Ah. There's File Data, which is what `rsync` deals with, and RAW BLOCK data, which is what scary hardcore tools like `dd` deal with." Then the notion that `cp` and `cat` can deal with both species of data is confusing. :p 0: https://old.reddit.com/r/linuxadmin/comments/eappzm/rsync_block_device_as_a_file/ https://old.reddit.com/r/linuxadmin/comments/eappzm/rsync_bl...

HomeGear 4y ago

>“Well why does the entire internet say to use dd then?” Because they copy from each other just like you copied from them. Great statement. This brings me some much needed internal clarity on my own thoughts and actions.

chubot 4y ago
This is basically the parable of "Grandma's Ham" https://www.executiveforum.com/cutting-off-the-ends-of-the-ham/ https://www.executiveforum.com/cutting-off-the-ends-of-the-h... tl;dr nobody in 2 generations knows why they cut the ends off the ham before cooking it, until they talked to grandma, who said her pan was too small A Unix thing that's been posted to HN for a decade, that's almost literally the same story: Understanding the bin, sbin, usr/bin , usr/sbin split https://news.ycombinator.com/item?id=3519952 https://news.ycombinator.com/item?id=3519952 tl;dr /usr/bin is separate from /bin because someone had a small hard disk once
- ahartmetz 4y ago
  So Grandma's Ham is a variant of Chesterton's Fence... When you do or don't do something for reasons of tradition, find the real reasons for doing / not doing it.
soneil 4y ago
You’d be amazed how many people I’ve found that do ‘tar xzf filename’ without knowing what xzf is doing.
- jthrowsitaway 4y ago
  Passing "z" is pretty pointless for extractions.
  - schoen 4y ago
    I think it was required until about 2010 or so for GNU tar. (I still use it from habit.) There was a problem about integrating support for different compression applications, with each one needing to get a new letter in the tar command!
    - sneusse 4y ago
      Won't this fail when the file is not gzipped but, for example, zstd compressed?
      - jhugo 4y ago
        It doesn't assume gzip, it detects the compression format.
    - jhugo 4y ago
      Yup, I remember when you had to pass `z` if it was gzip, and I remember my surprise when I missed it once and it still worked (apparently about 5 years after it was no longer needed!)
- sam_bristow 4y ago
  I'll admit to being one of those people. My brain still spells it out as eXtract Ze Files every time :-P
  - Izkata 4y ago
    You're 2/3 correct: x - extract z - gzip format f - file (must be last so it parses the filename arg correctly I also add "vv" in the middle so it lists every file as it goes, so I can see it work instead of just waiting with no output.

throwawaylinux 4y ago

What if you need to test direct IO, or test various partial-page, partial-block, or full block/page access to pagecache? Sometimes you might actually want to try using direct IO too, if buffered IO is not giving you sufficient control or reporting of errors from the device, or continuation from part-way through the copy, and in some cases possibly because you don't want to consume memory for page cache (although dd isn't good with direct IO so that's probably unlikely). That said if you don't know whether or not you need it, you don't. And those who do would know not to listen to your advice (for what they need to do) anyway, so it's not bad general advice to give.

spudlyo 4y ago
Direct I/O is notoriously finicky. Depending on what filesystem you're using there are apparently alignment restrictions and other corner cases to think about. I don't think it's that `dd` isn't good with Direct I/O, I just think its hard to use.
- throwawaylinux 4y ago
  Direct IO is not finicky (on Linux, since 2.4 era kernels). It's just that at least with dd, if you don't know what it does then it probably does not do what you want (synchronous, single queue depth). If you know what you want and know what it does, then it's fine to use, and can be useful (I used it today to measure block device interrupt latency in a VM).

RaoulP 4y ago

Well, I’d like to thank the author for writing this. I always enjoy having the veil of “magic” lifted from computers. UNIX’s simplicity of “everything is a file” continues to surprise me, even though it shouldn’t any more. I think a lack of confidence in my understanding of the basics, leaves me tempted to copy from others as you mention.

koala_man 4y ago
Author here. You're welcome ^^

jhardy54 4y ago

> Just use cat. We’ve gone full circle. https://en.m.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat https://en.m.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_ca...

nikau 4y ago
The reason to use cat is so you don't risk accidentally overwriting the source file by messing up a command. Anything after the "cat file |" can't overwrite the file.
- jhugo 4y ago
  That seems like a bad reason. You can always overwrite the file by messing up the command. cat file | cat > file (This won't, as you might hope, leave the file unchanged.) Re-read what you typed before hitting enter, and keep backups of important things.
  - nikau 4y ago
    That's an unlikely mistake. More likely is you are messing around with some new/infrequently used utility and you pass in the arguments incorrectly and specify "file" as the output instead of the input as intended.
    - jhugo 4y ago
      The one I typed is unlikely (I wrote it in response to your "Anything after the `cat file |` can't overwrite the file", which is factually incorrect as demonstrated), but there are plenty of other simple mistakes. Understand what you're running, and keep backups, otherwise you'll screw up eventually. Selecting your tools this way doesn't move the needle much.
      - nikau 4y ago
        Removing cat and specifying the file directly to the utility doesn't move the needle in any tangible way either. At the end of the day do whatever pattern works for you - for me it's doing "cat file |" at the start of a pipeline and "> outfile" at the end. I also avoid globbing inside a pipeline as it can be dangerous too.

Claude_Shannon 4y ago

I've been taught at my uni to use dd...

mort96 4y ago

"Well why does the entire internet say to use dd then?" Because you usually need root to write to drives, and you can't `sudo` redirect. `sudo dd if=whatever.img of=/dev/whatever` is nicer and easier than `sudo sh -c "cat whatever.img >/dev/whatever"`.

dinom 4y ago
In the cases where you need root it's often even easier to use the 'tee' command instead of spawning another shell; esp. if the command has symbols which need to be escaped. Your example would then become something like... cat whatever.img | sudo tee /dev/whatever > /dev/null # https://stackoverflow.com/questions/82256 https://stackoverflow.com/questions/82256 Just like in household plumbing, the 'tee' command basically takes the input and sends it to more than one place. Naturally, running 'sudo tee' will let you send things all over but as another user. All that said, I won't speak to the comment "why does the entire internet say to use dd". I've never actually found the "whole internet" to agree on much of anything (-:

shrimp_emoji 4y ago

What initially mystified me into thinking "I guess I have to use `dd` for block devices" was learning that `rsync` doesn't work with block devices. I just tried to copy an ISO to a USB drive with `rsync`, and it didn't work. Looking it up, I read these comments[0]: "rsync operates on files which are on a filesystem. It doesn’t do comparisons between blocks. If you want to back up a block device to a file, use dd" "I completely understood the use case. rsync (still) does not operate on block devices, so it’s not the solution here." "Agreed, rsync has never operated on block devices." So I thought, "Ah. There's File Data, which is what `rsync` deals with, and RAW BLOCK data, which is what scary hardcore tools like `dd` deal with." Then the notion that `cp` and `cat` can deal with both species of data is confusing. :p 0: https://old.reddit.com/r/linuxadmin/comments/eappzm/rsync_block_device_as_a_file/ https://old.reddit.com/r/linuxadmin/comments/eappzm/rsync_bl...