7 ms·

In the bad old Linux days, if you read a huge amount of data off the disk (like if you were doing a backup) Linux would try to take all that data and jam it in

In the bad old Linux days, if you read a huge amount of data off the disk (like if you were doing a backup) Linux would try to take all that data and jam it in the page cache. This would push out all the useful stuff in your cache, and sometimes even cause swapping as Linux helpfully swaps out stuff to make room for more useless page cache. One of the great things about `dd` is that you have a lot of control how the input and output files are opened. You can bypass the page cache when reading data by using iflag=direct, which would stop this from happening.

guerrilla 4y ago
I think using a small bs also determines the size of the cache you use, as its the buffer.
- KMag 4y ago
  GP is talking about the Linux kernel's buffer cache. Unless you tell the kernel to operate directly on the disk, your reads come from and your writes go to pages within the kernel's buffer cache. Using a small bs probably results in a buffer of only bs bytes inside dd's address space, but the buffer cache is completely different and resides in the kernel. That is, without iflag=direct, dd will repeatedly ask the kernel to copy bs bytes from the kernel's buffer cache into dd's address space, and then ask the kernel to copy bs bytes from its address space into the kernel's buffer cache.
cosmotic 4y ago
It's great to have control over this but I suspect most users never knew this was happening, had no idea dd could bypass that behavior, nor knew which argument to pass to dd to accomplish this. It's like saying 'what makes 3d printers so great is you can make anything!' but you'd be way better off with an industrially forged object than the 3d printed object.
- birdyrooster 4y ago
  right and good luck doing injection or vacuum molding for a few copies of a niche design Edit: correcting for nitpicker
  - andi999 4y ago
    For a few copies of niche design you would do vacuum molding.
  - girvo 4y ago
    You can do vacuum forming at home pretty easily. We used to do it back in the day for one-off handheld versions of home game consoles.
    - dTal 4y ago
      That's cool, I didn't know you could do that. How do you make the mold? If (as I suspect) the answer is handcrafting, 3d printing still offers a considerable advantage in labor.
      - girvo 4y ago
        Oh gosh yes. Though for certain finishes, vacuum forming still has some upsides. Whats neat is you can use 3D printing to make the mould for vacuum forming over. This gives you the best of both worlds, in terms of the shapes/finishes that forming can achieve, but with the workflow benefits of 3D printing. Similarly, the new hotness is compression/cold forging using 3D printed moulds and carbon fibre (and other materials). Very very cool.
  - cosmotic 4y ago
    The main point wasn't the problems with 3d printers but actually looking at dd from a human centered design perspective.
a-dub 4y ago
amusingly this would also occur with writes as well. there was some hueristic in there that tried to prevent it, but it wasn't very good.
bushbaba 4y ago
I’d of thought dd always avoided page cache. what dd use case is that desired behavior?
- deleted 4y ago
  [deleted]
throwaway2048 4y ago
linux absolutely still does this FWIW, its one of the reasons that swap is a net negative.
bayindirh 4y ago
Moreover, flash drives (and all flash media) have a favorite page size, which is generally 4kB or 512kB. By defining a common denominator page size like 1024kB (with bs=1024kB), you can keep your flash drive happy with enough backlog to write, so it can perform at its peak write speed without churning, which will help with faster writes and lower write amplification, which is a win-win.
- vladvasiliu 4y ago
  Man, this brings back memories of a few years ago when I tried to dd a Linux install image on a usb drive. It would take forever and always end up with an i/o error. I figured the new PC had somehow wonky usb ports or something. But when that happened on another "known good" box, I figured it was the flash drive. But Disk Utility in MacOS worked well. Then I tried increasing the block size to 1M and everything went smoothly. It even took less time to write the whole image correctly than it took it to error out before.
- nousermane 4y ago
  (In theory at least,) kernel should take care of aggregating write blocks and those will be plenty big enough by the time it reaches target drive, all thanks to the very same page cache GP is talking about - unless you specify "oflag=direct" to dd. That being said, probably don't use too small of a block-size - this will eat up CPU in system call overhead and slow down copy regardless of target media type.
  - bayindirh 4y ago
    I don't know how internals of "cp" and the related machinery interacts with the target drive, however if you don't provide bs=1024kB, dd writes with extremely small units (1 byte at a time IIRC), which overwhelms the flash controller and creates a high CPU load at the same time. I always used dd since it provides more direct control over the transfer stream and how it's transported. I also call dd as "direct-drive" sometimes due to these capabilities of the tool.
    - deleted 4y ago
totetsu 4y ago
Still the bad old days for copying files from iOS to Linux. It seems to make a copy internally of everything you copy internally on the device before sending it, which leads to running out of free space just trying to copy things of :(
ChuckMcM 4y ago
Pretty much, and understanding what is going on "under the hood" as it were can be informative. Had the author done a 'cp myfile.foo /dev/sdb' on a UNIX system they would have found they now had a regular file named '/dev/sdb' with the contents of myfile.foo and and their sd card would have remained untouched. But you would only know that if you realized that cp would check to see if the file existed in the destination, unlink it[1], and then create a new file to copy into. The subtlety of opening the destination file first, and then writing into it, was what made dd 'special' (and it would open things in RAW mode so there wasn't any translation going on for say, terminals) but that is lost on people. Bypassing the page cache and thus not killing directory and file operations for other users of the system is a level even below that. Only the few remaining who have done things "poorly" an incurred the wrath of the other users of the system sitting in the same room really get a good feel for that :-). Fortunately for nearly everybody these days they will never have to experience that social embarrassment. :-) [1] Well unless you had noclobber set in which case it would error out.
watersb 4y ago
On macOS, you can also use the `--nocache` flag for the `ditto` command. Please keep in mind that `ditto` is a file copy and archive utility, not a block copy utility like `dd` (which is also available on macOS). An online man page for ditto: https://ss64.com/osx/ditto.html https://ss64.com/osx/ditto.html
watersb 4y ago
In my experience, Windows NT (now just Windows) is very fond of its file cache and large copies can blow up into memory paging as well. Early Windows NT was awful with this, pegging the system with a cascade of disk IO at unpredictable times, often for ten seconds or more. Can anyone suggest ways to avoid blowing the file cache on Windows with large copies? Is this even a problem anymore?

guerrilla 4y ago

I think using a small bs also determines the size of the cache you use, as its the buffer.

KMag 4y ago
GP is talking about the Linux kernel's buffer cache. Unless you tell the kernel to operate directly on the disk, your reads come from and your writes go to pages within the kernel's buffer cache. Using a small bs probably results in a buffer of only bs bytes inside dd's address space, but the buffer cache is completely different and resides in the kernel. That is, without iflag=direct, dd will repeatedly ask the kernel to copy bs bytes from the kernel's buffer cache into dd's address space, and then ask the kernel to copy bs bytes from its address space into the kernel's buffer cache.

cosmotic 4y ago

It's great to have control over this but I suspect most users never knew this was happening, had no idea dd could bypass that behavior, nor knew which argument to pass to dd to accomplish this. It's like saying 'what makes 3d printers so great is you can make anything!' but you'd be way better off with an industrially forged object than the 3d printed object.

birdyrooster 4y ago
right and good luck doing injection or vacuum molding for a few copies of a niche design Edit: correcting for nitpicker
- andi999 4y ago
  For a few copies of niche design you would do vacuum molding.
- girvo 4y ago
  You can do vacuum forming at home pretty easily. We used to do it back in the day for one-off handheld versions of home game consoles.
  - dTal 4y ago
    That's cool, I didn't know you could do that. How do you make the mold? If (as I suspect) the answer is handcrafting, 3d printing still offers a considerable advantage in labor.
    - girvo 4y ago
      Oh gosh yes. Though for certain finishes, vacuum forming still has some upsides. Whats neat is you can use 3D printing to make the mould for vacuum forming over. This gives you the best of both worlds, in terms of the shapes/finishes that forming can achieve, but with the workflow benefits of 3D printing. Similarly, the new hotness is compression/cold forging using 3D printed moulds and carbon fibre (and other materials). Very very cool.
- cosmotic 4y ago
  The main point wasn't the problems with 3d printers but actually looking at dd from a human centered design perspective.

a-dub 4y ago

amusingly this would also occur with writes as well. there was some hueristic in there that tried to prevent it, but it wasn't very good.

bushbaba 4y ago

I’d of thought dd always avoided page cache. what dd use case is that desired behavior?

deleted 4y ago
[deleted]

throwaway2048 4y ago

linux absolutely still does this FWIW, its one of the reasons that swap is a net negative.

bayindirh 4y ago

Moreover, flash drives (and all flash media) have a favorite page size, which is generally 4kB or 512kB. By defining a common denominator page size like 1024kB (with bs=1024kB), you can keep your flash drive happy with enough backlog to write, so it can perform at its peak write speed without churning, which will help with faster writes and lower write amplification, which is a win-win.

vladvasiliu 4y ago
Man, this brings back memories of a few years ago when I tried to dd a Linux install image on a usb drive. It would take forever and always end up with an i/o error. I figured the new PC had somehow wonky usb ports or something. But when that happened on another "known good" box, I figured it was the flash drive. But Disk Utility in MacOS worked well. Then I tried increasing the block size to 1M and everything went smoothly. It even took less time to write the whole image correctly than it took it to error out before.
nousermane 4y ago
(In theory at least,) kernel should take care of aggregating write blocks and those will be plenty big enough by the time it reaches target drive, all thanks to the very same page cache GP is talking about - unless you specify "oflag=direct" to dd. That being said, probably don't use too small of a block-size - this will eat up CPU in system call overhead and slow down copy regardless of target media type.
- bayindirh 4y ago
  I don't know how internals of "cp" and the related machinery interacts with the target drive, however if you don't provide bs=1024kB, dd writes with extremely small units (1 byte at a time IIRC), which overwhelms the flash controller and creates a high CPU load at the same time. I always used dd since it provides more direct control over the transfer stream and how it's transported. I also call dd as "direct-drive" sometimes due to these capabilities of the tool.
  - deleted 4y ago

totetsu 4y ago

Still the bad old days for copying files from iOS to Linux. It seems to make a copy internally of everything you copy internally on the device before sending it, which leads to running out of free space just trying to copy things of :(

ChuckMcM 4y ago

Pretty much, and understanding what is going on "under the hood" as it were can be informative. Had the author done a 'cp myfile.foo /dev/sdb' on a UNIX system they would have found they now had a regular file named '/dev/sdb' with the contents of myfile.foo and and their sd card would have remained untouched. But you would only know that if you realized that cp would check to see if the file existed in the destination, unlink it[1], and then create a new file to copy into. The subtlety of opening the destination file first, and then writing into it, was what made dd 'special' (and it would open things in RAW mode so there wasn't any translation going on for say, terminals) but that is lost on people. Bypassing the page cache and thus not killing directory and file operations for other users of the system is a level even below that. Only the few remaining who have done things "poorly" an incurred the wrath of the other users of the system sitting in the same room really get a good feel for that :-). Fortunately for nearly everybody these days they will never have to experience that social embarrassment. :-) [1] Well unless you had noclobber set in which case it would error out.

watersb 4y ago

On macOS, you can also use the `--nocache` flag for the `ditto` command. Please keep in mind that `ditto` is a file copy and archive utility, not a block copy utility like `dd` (which is also available on macOS). An online man page for ditto: https://ss64.com/osx/ditto.html https://ss64.com/osx/ditto.html

watersb 4y ago

In my experience, Windows NT (now just Windows) is very fond of its file cache and large copies can blow up into memory paging as well. Early Windows NT was awful with this, pegging the system with a cascade of disk IO at unpredictable times, often for ten seconds or more. Can anyone suggest ways to avoid blowing the file cache on Windows with large copies? Is this even a problem anymore?