3 ms·
[warning: slight offtopic) TLDR: Any recommendations on the best way to clone one harddrive to another that doesn't take forever? > Storage I/O gained an asyn
by dmix 6y ago
[warning: slight offtopic)
TLDR: Any recommendations on the best way to clone one harddrive to another that doesn't take forever?
> Storage I/O gained an asynchronous interface tailored-fit to work with the kind of applications that really needed it at the moment and nothing else.
Say you have 2x 2TB SSD harddrives and one needs to be cloned to the other.
Being the clever hacker I am who grew up using linux I simply tried unmounting the drivers and trying the usually `dd` approach (using macOS). The problem: It took >20hrs for a direct duplication of the disk. The other problem: this was legal evidence from my spouses work on a harddisk provided by police, so I assumed this was the best approach. Ultimately she had to give it in late because of my genius idea which I told her wouldn't take long.
Given a time constraint the next time this happened, we gave up `dd`, and did the old mounted disk copy/paste via Finder approach... which only took only 3hrs to get 1.2TB of files across into the other HD - via usb-c interfaces.
I've been speculating why one was 5x+ faster than the other (besides the fact `dd` doing a bit-by-bit copy of the filesystem). My initial suspicion was options provided `dd`:
> sudo dd if=/dev/rdisk2 of=/dev/rdisk3 bs=1m conv=noerror,sync
I'm not 100% familiar with the options for `dd` but I do remember a time where I changed `bs=1M` to `bs=8M` helped speed up a transfer in the past.
But I didn't do it for the sake of following the instructions on StackOverflow.
- vamega 6y agoProbably faster to copy via finder since it’s not actually copying every byte, but only the utilized bytes. It might be faster to have multiple rsync operations (via xargs or the like), but if the disk is relatively empty I can see this being faster. Finding the right level of parallelism isn’t something I can help you with, probably needs some experimentation.
- trentnelson 6y agoI wrote some Python glue that constructed a bunch of dd commands to run concurrently which helped when I last cloned a 1TB NVMe drive. Resulting commands: dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=0 seek=0 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=1302349 seek=1302349 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=2604698 seek=2604698 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=3907047 seek=3907047 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=5209396 seek=5209396 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=6511745 seek=6511745 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=7814094 seek=7814094 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=9116443 seek=9116443 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=10418792 seek=10418792 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=11721141 seek=11721141 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=13023490 seek=13023490 count=1302349 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=65536 skip=14325839 seek=14325839 count=1302358 status=progress & dd if=/dev/nvd0 of=/dev/nvd1 bs=4096 skip=15628197 seek=15628197 count=6 status=progress &