6 ms·

As mentioned elsewhere in this post this leads to dreadful performance. You are better off using iflag/oflag options seek_bytes, skip_bytes and count_bytes to t

As mentioned elsewhere in this post this leads to dreadful performance. You are better off using iflag/oflag options seek_bytes, skip_bytes and count_bytes to treat these options as bytes and using a large block size.

shmerl 4y ago
Hm, I don't see any details on that in the man page. What's the logic there?
- dabiged 4y ago
  skip, seek and count default to counting the number of blocks. So if you want Megabytes 50-60 of a file you can do either a 1 byte block size: > dd if=/tmp/file1 bs=1 skip=50000000 count=10000000 or you can use any block size you want and treat skip and count as byte counts, not block counts. > dd if=/tmp/file1 bs=4M skip=50M count=10M iflag=skip_bytes,count_bytes Option 1 will run at Kilobytes per second as it is transferring 1 byte at a time (my test gives me 933kB/s) Option 2 will run a 100's MB per second as it is transferring 4MB at a time (my test gives me 202MB/s). Edit: What I should have said at the top of the post was "If you want to extract data from a file whose size is not an integer block multiple". In the above example you could have used a Blocksize of 10MB for the same result. You cannot do this for an oddly size extraction (say 1234567bytes).
  - shmerl 4y ago
    Interesting, thanks for the hint! Why doesn't the man page say anything about these flags? They are completely obscure becasue of that.
    - dabiged 4y ago
      I only found out because I use dd 1000's of times each day and have had to prototype some really esoteric data pipelines using it. I have even compiled a few custom versions to do various tasks. One other nuggets of wisdom which is very poorly documented in the man page: if you have a long running dd process (I often have 12+ hr dd processes reading 10+TB tapes) you can send a USR1 signal to check the progress. Very useful to check if your tape drive has had a hardware failure. Information on both this and the byte counts are in my man page (GNU coreutils 8.30 on Linux Mint 20) around lines 130.
      - shmerl 4y ago
        I knew about USR1 and it's in the man for me too, but I don't see anything about skip_bytes / count_bytes flags, which is strange. Debian testing, coreutils 9.1.
        dabiged 4y ago
        I am running Coreutils 8.3 on Linux mint 20. It is also in Coreutils 8.22 in RHEL7. https://man7.org/linux/man-pages/man1/dd.1.html https://man7.org/linux/man-pages/man1/dd.1.html Edit: I just found this in the coreutils 9.0 changelog: dd now counts bytes instead of blocks if a block count ends in "B". For example, 'dd count=100KiB' now copies 100 KiB of data, not 102,400 blocks of data. The flags count_bytes, skip_bytes and seek_bytes are therefore obsolescent and are no longer documented, though they still work.
        shmerl 4y ago
        Thanks for the pointer. So you can simply do something like: dd if=foo skip=nB count=mB bs=4M of=bar To get m bytes from offset n

shmerl 4y ago

Hm, I don't see any details on that in the man page. What's the logic there?

dabiged 4y ago
skip, seek and count default to counting the number of blocks. So if you want Megabytes 50-60 of a file you can do either a 1 byte block size: > dd if=/tmp/file1 bs=1 skip=50000000 count=10000000 or you can use any block size you want and treat skip and count as byte counts, not block counts. > dd if=/tmp/file1 bs=4M skip=50M count=10M iflag=skip_bytes,count_bytes Option 1 will run at Kilobytes per second as it is transferring 1 byte at a time (my test gives me 933kB/s) Option 2 will run a 100's MB per second as it is transferring 4MB at a time (my test gives me 202MB/s). Edit: What I should have said at the top of the post was "If you want to extract data from a file whose size is not an integer block multiple". In the above example you could have used a Blocksize of 10MB for the same result. You cannot do this for an oddly size extraction (say 1234567bytes).
- shmerl 4y ago
  Interesting, thanks for the hint! Why doesn't the man page say anything about these flags? They are completely obscure becasue of that.
  - dabiged 4y ago
    I only found out because I use dd 1000's of times each day and have had to prototype some really esoteric data pipelines using it. I have even compiled a few custom versions to do various tasks. One other nuggets of wisdom which is very poorly documented in the man page: if you have a long running dd process (I often have 12+ hr dd processes reading 10+TB tapes) you can send a USR1 signal to check the progress. Very useful to check if your tape drive has had a hardware failure. Information on both this and the byte counts are in my man page (GNU coreutils 8.30 on Linux Mint 20) around lines 130.
    - shmerl 4y ago
      I knew about USR1 and it's in the man for me too, but I don't see anything about skip_bytes / count_bytes flags, which is strange. Debian testing, coreutils 9.1.
      - dabiged 4y ago
        I am running Coreutils 8.3 on Linux mint 20. It is also in Coreutils 8.22 in RHEL7. https://man7.org/linux/man-pages/man1/dd.1.html https://man7.org/linux/man-pages/man1/dd.1.html Edit: I just found this in the coreutils 9.0 changelog: dd now counts bytes instead of blocks if a block count ends in "B". For example, 'dd count=100KiB' now copies 100 KiB of data, not 102,400 blocks of data. The flags count_bytes, skip_bytes and seek_bytes are therefore obsolescent and are no longer documented, though they still work.
        shmerl 4y ago
        Thanks for the pointer. So you can simply do something like: dd if=foo skip=nB count=mB bs=4M of=bar To get m bytes from offset n