6 ms·
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent
by mrb 3y ago
In the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat.
On the destination laptop:
$ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M
On the source laptop:
$ nc x.x.x.x 1234 </dev/nvme0nX
The dd on the destination is just to buffer writes so they are faster/more efficient. Add a gzip/gunzip on the source/destination and the whole operation is a lot faster if your disk isn't full, ie. if you have many zero blocks. This is by far my favorite way to image a PC over the network. I have done this many times. Be sure to pass "--fast" to gzip as the compression is typically a bottleneck on GigE. Or better: replace gzip/gunzip with lz4/unlz4 as it's even faster. Last time I did this was to image a brand new Windows laptop with a 1TB NVMe. Took 20 min (IIRC?) over GigE and the resulting image was 20GB as the empty disk space compresses to practically nothing. I typically back up that lz4 image and years later when I donate the laptop I restore the image with unlz4 | dd. Super convenient.
That said I didn't know about that Linux kernel module nvme-tcp. We learn new things every day :) I see that its utility is more for mounting a filesystem over a remote NVMe, rather than accessing it raw with dd.
Edit: on Linux the maximum pipe buffer size is 64kB so the dd bs=X argument doesn't technically need to be larger than that. But bs=1M doesn't hurt (it buffers the 64kB reads until 1MB has been received) and it's future-proof if the pipe sizes is ever increased :) Some versions of netcat have options to control the input and output block size which would alleviate the need to use dd bs=X but on rescue discs the netcat binary is usually a version without these options.
- tripflag 3y agoThis use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hurt as well. I prefer to just nc -l -p 1234 > /dev/nvme0nX.
- mrb 3y agoPartial reads won't corrupt the data. Dd will issue other read() until 1MB of data is buffered. The iflag=fullblock is only useful when counting or skipping bytes or doing direct I/O. See line 1647: https://github.com/coreutils/coreutils/blob/master/src/dd.c#L1647 https://github.com/coreutils/coreutils/blob/master/src/dd.c#...
- adrian_b 3y agoAccording to the documentation of dd, "iflag=fullblock" is required only when dd is used with the "count=" option. Otherwise, i.e. when dd has to read the entire input file because there is no "count=" option, "iflag=fullblock" does not have any documented effect. From "info dd": "If short reads occur, as could be the case when reading from a pipe for example, ‘iflag=fullblock’ ensures that ‘count=’ counts complete input blocks rather than input read operations."
- tripflag 3y agoThank you for the correction -- it is likely that I did use count= when I ran into this some 10 years ago (and been paranoid about ever since). I thought a chunk of data was missing in the middle of the output file, causing everything after that to be shifted over, but I'm probably misremembering.
- fsckboy 3y agothank you for bringing it up! i wasn't even aware of this potential problem, and I use bs= count= and skip= seek= (sk"i"p means "input") through pipes across the net aaaallll the time for decades. it pretty much seems iflag=fullblock is a requirement if you want the counts to work, even though the failure times might be rare
- dezgeg 3y agoIsn't `nc -l -p 1234 > /dev/nvme0nX` working by accident (relying on that netcat is buffering its output in multiples of disk block size)?
- mrb 3y agoYour exact command works reliably but is inefficient. And it works by design, not accident. For starters, the default block size in most netcat implementations is tiny like 4 kB or less. So there is a higher CPU and I/O overhead. And if netcat does a partial or small read less than 4 kB, when it writes the partial block to the nvme disk, the kernel would take care of reading a full 4kB block from the nvme disk, updating it with the partial data block, and rewriting the full 4kB block to the disk, which is what makes it work, albeit inefficiently.
- jasomill 3y agoNo — the kernel buffers non-O_DIRECT writes to block devices to ensure correctness. Larger writes will be more efficient, however, if only due to reduced system call overhead. While not necessary when writing an image with the correct block size for the target device, even partial block overwrites work fine: # yes | head -c 512 > foo # losetup /dev/loop0 foo # echo 'Ham and jam and Spam a lot.' | dd bs=5 of=/dev/loop0 5+1 records in 5+1 records out 28 bytes copied, 0.000481667 s, 58.1 kB/s # hexdump -C /dev/loop0 00000000 48 61 6d 20 61 6e 64 20 6a 61 6d 20 61 6e 64 20 |Ham and jam and | 00000010 53 70 61 6d 20 61 20 6c 6f 74 2e 0a 79 0a 79 0a |Spam a lot..y.y.| 00000020 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.| * 00000200 Partial block overwrites may (= will, unless the block to be overwritten is in the kernel's buffer cache) require a read/modify/write operation, but this is transparent to the application. Finally, note that this applies to most block devices, but tape devices work differently: partial overwrites are not supported, and, in variable block mode, the size of individual write calls determines the resulting tape block sizes.
- dezgeg 3y agoSomehow I had thought even in buffered mode the kernel would only accept block-aligned and sized I/O. TIL.
- M95D 3y agoI would include bs=1M and oflag=direct for some extra speed.
- deleted 3y ago[deleted]
- bayindirh 3y agoAs a sysadmin, I'd rather use NVMe TCP or Clonezilla to do a slow write rather than trying to go 5% faster with more moving parts and chance to corrupt my drive in the process. Plus, a it'd be well deserved coffee break. Considering I'd be going at GigE speeds at best, I'd add "oflag=direct" to bypass caching on the target. A bog standard NVMe can write >300MBps unhindered, so trying to cache is moot. Lastly, parted can do partition resizing, but given the user is not a power user to begin with, it's just me nitpicking. Nice post otherwise.
- iforgotpassword 3y agoI don't see how you can consider the nvme over tcp version less moving parts. dd is installed on every system, and if you don't have nc you can still use ssh and sacrifice a bit of performance. dd if=/dev/foo | ssh dest@bar "cat > /dev/moo"
- bayindirh 3y agoNVMe over TCP encapsulates and shows me the remote device as is. Just a block device. I just copy that block device with "dd", that's all. It's just a dumb pipe encapsulated with TCP, which is already battle tested enough. Moreover, if I have fatter pipe, I can tune dd for better performance with a single command.
- darkwater 3y agonetcat encapsulates data just the same (although in a different manner), and it's even more battle-tested. NVMe over TCP use case is to actually use the remote disk over the network as it were local. If you just need to dump a whole disk like in the article, dd+netcat (or even just netcat, as someone pointed out) will work just the same.
- iforgotpassword 3y agoNvme over TCP encapsulates the entire nvme protocol in TCP, which is way more complex than just sending the raw data. It's the opposite of "a dumb pipe encapsulated in tcp", this is what the netccat approach would be. Heck if you insist on representing the drive as a block device on the remote side you could just as well use NBD, which is just about as many moving parts as nvme over tcp but still a simpler protocol.
- _flux 3y ago> there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O I guess most people don't have faster local network than an SSD can transfer. I wonder though, for those people who do, does a concurrent I/O block device replicator tool exist? Btw, you might want also use pv in the pipeline to see an ETA, although it might have a small impact on performance.
- Palomides 3y agodd has status=progress to show bytes read/written now, I just use that
- crote 3y agoI doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access. Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. If we're looking at state-of-the-art consumer tech, the fastest you're getting is a USB4 40Gbit machine-to-machine transfer - but at that point you probably have something like the Crucial T700, which has a sequential write speed of 11.800 MByte/s. The enterprise world probably doesn't look too different. You'd need a 100Gbit NIC to saturate even a single modern SSD, but any machine with such a NIC is more likely to have closer to half a dozen SSDs. At that point you're starting to be more worried about things like memory bandwidth instead. [0] [0]: http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...
- JoeAltmaier 3y agoCan't find corroboration for the assertion 'SSDs are an awful lot better at sequential writes than random writes'. Doesn't make sense at first glance. There's no head to move, as in an old-style hard drive. What else could make random write take longer on an SSD?
- 3y ago
- Multrex 3y agoSeems awesome. Can you please tell us how to use gzip or lz4 to do the imaging?
- zbentley 3y agoIf you search for “dd gzip” or “dd lz4” you can find several ways to do this. In general, interpose a gzip compression command between the sending dd and netcat, and a corresponding decompression command between the receiving netcat and dd. For example: https://unix.stackexchange.com/questions/632267 https://unix.stackexchange.com/questions/632267
- deleted 3y ago[deleted]
- ersamsa 3y agoJust cat the blockdev to a bash socket
- szundi 3y agoThis is exactly that I usually do, it works like a charm
- vient 3y ago> on Linux the maximum pipe buffer size is 64kB Note that you can increase pipe buffer, I think default maximum size is usually around 1MB. A bit tricky to do from command line, one possible implementation being https://unix.stackexchange.com/a/328364 https://unix.stackexchange.com/a/328364
- ooooo000 3y ago[flagged]
- benterix 3y ago> Hdjrnrhf Fhjrnrjg Cnn3nrmf Нос3uejr Нирш Юни до края шллш Is this some new form of Russian Ops here on HN?
- alpenbazi 3y agoSir, i dont upvote much, but your post deserves a double up, at least
- exceptione 3y ago> That said I didn't know about that Linux kernel module nvme-tcp. We learn new things every day :) I see that its utility is more for mounting a filesystem over a remote NVMe, rather than accessing it raw with dd. Aside, I guess nvme-tcp would result in less writes as you only copy files in stead of writing the whole disk over?
- NoahKAndrews 3y agoNot if you use it with dd, which will copy the blank space too
- NoahKAndrews 3y agoNot if you use it with dd, which will copy the blank space too
- khaki54 3y agoYep I've done this and it works in a pinch. 1Gb/s is also a reasonable fraction of SATA speeds too.
- fransje26 3y agoI am not often speechless, but this hit the spot. Well done Sir! Where does one learn this black art?
- deleted 3y ago[deleted]
- wang_li 3y agoWould be much better to hook this up to dump and restore. It'll only copy used data and you can do it while the source system is online. For compression the rule is that you don't do it if the CPU can't compress faster than the network.
- BuildTheRobots 3y agoIt's a little grimy, but if you use `pv` instead of `dd` on both ends you don't have to worry about specifying a sensible block size and it'll give you nice progression graphs too.
- loeg 3y agoAgree, but I'd suggest zstd instead of gzip (or lz4 is fine).
- HankB99 3y agoI came here to suggest similar. I usually go with dd if=/dev/device | mbuffer to Ethernet to mbuffer dd of=/dev/device (with some switches to select better block size and tell mbuffer to send/receive from a TCP socket) If it's on a system with a fast enough processor I can save considerable time by compressing the stream over the network connection. This is particularly true when sending a relatively fresh installation where lots of the space on the source is zeroes.
- M_bara 3y agoAbout 9 years ago, I consulted for a company that had a bad internal hack - disgruntled cofounder. Basically, a dead man’s switch was left that copied out the first 20mb of every disk to some bucket and then zeroed out. To recover the data we had to use test disk to rebuilt the partition table… but before doing that we didn’t want to touch the corrupted disks so we ended up copying out about 40tb using rescue flash disks, nectat and drive (some of the servers had a physical raid with all slots occupied so you couldn’t use some free had slots). Something along the lines of dd if=/dev/sdc bs=xxx | gzip | nc -l -p 8888 and the reverse on the other side. It actually worked surprisingly well. One thing of note,try combinations of dd bs to match with sector size - proper sizing had a large impact on dd throughput