9 ms·
Cloning a Laptop over NVMe TCP
- MeteorMarc 3y agoHow can this work if the laptops have different hardware, cq different requirements on device drivers?
- TheNewAndy 3y agoLinux ships with all the drivers installed (for a fairly high value of "all") (typically)
- AndroTux 3y agoWith modern systems this rarely is a problem. Like Andy said, Linux comes with most drivers in the kernel. Windows just installs the appropriate drivers as soon as you boot it up with new hardware. Sometimes it takes a minute or two for it to boot up after the initial swap, but then it should be fine.
- coretx 3y agoDoing this without systemd or directly from the netboot environment would be interesting.
- bayindirh 3y agoThe user didn't do it from systemd actually. Instead they booted GRML, which is not very different from a netboot environment, and hand-exported the device themselves.
- coretx 3y agoGRML is a debian live image, much different from a netboot environment. Look at https://ipxe.org/ https://ipxe.org/, boot firmware doing fiber over ethernet, iSCSI, HTTP; booting from virtually everything but NVMe over TCP.
- bayindirh 3y agoI use GRML almost daily, sometimes booting it over network and loading it to RAM. We don't use iPXE specificially, but use "vanilla" PXE regularly, too, generally to install systems with Kickstart or xCAT, depending on the cluster we're working on. I'll be using GRML again tonight to rescue some laptop drives which were retrieved from a fire, to see what can be salvaged, for example. First in forensic mode to see what's possible, then to use dd_rescue on a N95 box.
- transpute 3y agoThanks AWS/Annapurna/Nitro/Lightbits for bringing NVMe-over-TCP to Linux. https://www.techtarget.com/searchstorage/news/252459311/Lightbits-Labs-launches-TCP-based-NVMe-over-fabrics-storage https://www.techtarget.com/searchstorage/news/252459311/Ligh... > The NVM Express consortium ratified NVMe/TCP as a binding transport layer in November 2018. The standard evolved from a code base originally submitted to NVM Express by Lightbits' engineering team. https://www.lightbitslabs.com/blog/linux-distributions-nvme-support/ https://www.lightbitslabs.com/blog/linux-distributions-nvme-... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3f2304f8c6d6ed97849057bd16fee99e434ca796 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- hcfman 3y agoVery nice read! I didn't know it existed. Love it.
- hcfman 3y agoI love your explanation of your disk install with cryptsetup etc. I do a manual install as well as I always install with both mirrored and encrypted disks. The combination of the two I didn't find as easy install options (I think not at all) on the Ubuntu's I installed. Good to see a lot of this low level OS talk here.
- mrb 3y agoIn the author's scenario, there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O. All the complex commands can be replaced by a simple netcat. On the destination laptop: $ nc -l -p 1234 | dd of=/dev/nvme0nX bs=1M On the source laptop: $ nc x.x.x.x 1234 </dev/nvme0nX The dd on the destination is just to buffer writes so they are faster/more efficient. Add a gzip/gunzip on the source/destination and the whole operation is a lot faster if your disk isn't full, ie. if you have many zero blocks. This is by far my favorite way to image a PC over the network. I have done this many times. Be sure to pass "--fast" to gzip as the compression is typically a bottleneck on GigE. Or better: replace gzip/gunzip with lz4/unlz4 as it's even faster. Last time I did this was to image a brand new Windows laptop with a 1TB NVMe. Took 20 min (IIRC?) over GigE and the resulting image was 20GB as the empty disk space compresses to practically nothing. I typically back up that lz4 image and years later when I donate the laptop I restore the image with unlz4 | dd. Super convenient. That said I didn't know about that Linux kernel module nvme-tcp. We learn new things every day :) I see that its utility is more for mounting a filesystem over a remote NVMe, rather than accessing it raw with dd. Edit: on Linux the maximum pipe buffer size is 64kB so the dd bs=X argument doesn't technically need to be larger than that. But bs=1M doesn't hurt (it buffers the 64kB reads until 1MB has been received) and it's future-proof if the pipe sizes is ever increased :) Some versions of netcat have options to control the input and output block size which would alleviate the need to use dd bs=X but on rescue discs the netcat binary is usually a version without these options.
- tripflag 3y agoThis use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hurt as well. I prefer to just nc -l -p 1234 > /dev/nvme0nX.
- mrb 3y agoPartial reads won't corrupt the data. Dd will issue other read() until 1MB of data is buffered. The iflag=fullblock is only useful when counting or skipping bytes or doing direct I/O. See line 1647: https://github.com/coreutils/coreutils/blob/master/src/dd.c#L1647 https://github.com/coreutils/coreutils/blob/master/src/dd.c#...
- wjnc 3y agoSo this just bit for bit dumps a NVMe device to another location. That’s clear. So all encryption just is transferred and not touched. But doesn’t the next machine go into panic when you boot? There are probably many changes in the underlying machine? (Okay, now I read the other post. The author really knows the way. This is at least intermediate Linux.)
- jauntywundrkind 3y agoA Linux install is often remarkably hardware agnostic. Windows would panic, certainly (because so much drivers & other state is persisted & expected), but the Linux kernel when it boots kind of figures out afresh what the world is every time. That's fine. The main thing you ought to do is generate a new systemd/dbus machine-id. But past this, I fairly frequently instantiate new systems by taking a btrfs snapshot of my current machine & send that snapshot over to a new drive. Chroot onto that drive, and use bootctl to install systemd-boot, and then I have a second Linux is ready to go.
- lnxg33k1 3y agoIf you moved the wrong stuff Linux would give you a bad time too, try /proc, /dev
- fellerts 3y agoThose are pseudo-filesystems though, they aren't part of the install.
- lnxg33k1 3y agoNot sure where I said they were part of the install, but I meant that if you don’t exclude them bad things would happen to the cloning process Edit: or backup process
- account42 3y agoWith a block device copy approach like in TFA you don't need to worry about that because those filesystems do not exist on the block device. When copying files (e.g. with rsync) you need to watch out for those, yes. The best way to deal with other mounts is not to copy directly from / but instead non-recursively bind-mount / to another directory and then copy from that.
- deleted 3y ago[deleted]
- roomey 3y agoI recently had to set up a new laptop (xubuntu). Previously I cloned but I this time I wanted to refresh some of the configs. Using a usb-c cable to transfer at 10gb/s is so useful (as my only other option was WiFi). When you plug the computers together they form an ad-hoc network and you can just rsync across. As far as I could tell the link was saturated so using anything else (other protocols) would be pointless. Well not pointless, it's really good to learn new stuff, maybe just not when you are cloning your laptop (joke)!
- baq 3y agoDid it ‘just work’? Serious question since last time I tried a direct non-Ethernet connection was sometime in the 90s ;)
- adrian_b 3y agoI assume that those USB-C connectors were USB 4 or Thunderbolt, not USB 3. With Thunderbolt and operating systems that support Ethernet over Thunderbolt, a virtual network adapter is automatically configured for any Thunderbolt connector, so connecting a USB C cable between 2 such computers should just work, as if they had Ethernet 10 Gb/s connectors. With USB 3 USB C connectors, you must use USB network adapters (up to 2.5 Gb/s Ethernet).
- namibj 3y agoUSB C is perfectly capable of connecting two equals, even with USB-2. It merely requires one side to be capable of behaving as a device, with the other side behaving as a host. I.e., unlike PCIe hubs, you won't get P2P bandwidth savings on a USB hub. It just so happens that most desktop xhci controllers don't support talking "device". But where you can, you can set up a dumb bidirectional stream fairly easily, over which you can run SLIP or PPP. It's essentially just a COM port/nullmodem cable. Just as a USB endpoint instead of as a dedicated hardware wire.
- roomey 3y agoYes, both were modern enough laptops. Although the ideapad didn't advertise thunderbolt in the lspci, connecting that and the dell precision "just worked" (tm). It's very useful for sending large data using minimal equipment. No need for two cat6 cables and a router for example.
- Gabrys1 3y agoI usually set up an initial distro and then copy over my /home. Later, I just need to install the debs I'm missing, but this has the benefit of not installing stuff I don't need anymore. That said, I didn't know you could export NVMe over TCP like that, so still a nice read!
- fransje26 3y agoSame here. The only problem with that approach is that it is also copies over the .config and .cache folders, most of which are possibly not needed anymore. Or worse, they might contain configs that overrides better/newer parameters set by the new system..
- Hikikomori 3y ago~/oldhome/oldhome/oldhome ...
- jeroenhd 3y agoI'm sure there are benefits to this approach, but I've transferred laptops before by launching an installer on both and then combining dd and nc on both ends. If I recall correctly, I also added gzip to the mix to make transferring large null sections faster. With the author not having access to an ethernet port on the new laptop, I think my hacky approach might've even been faster because of the slight boost compression would've provided, given that the network speed is nowhere near the speed limit compression would add to a fast network link.
- Moneysac 3y agocould you explain how you do that exactly?
- gouggoug 3y agoThis comment explains it: https://news.ycombinator.com/item?id=39676881 https://news.ycombinator.com/item?id=39676881
- jeroenhd 3y agoBasically https://news.ycombinator.com/item?id=39676881 https://news.ycombinator.com/item?id=39676881, but also adding a pipe through gzip/gunzip in the middle for data compression. I think I did something like `nc -l -p 1234 | gunzip | dd status=progress of=/dev/nvme0n1` on the receiving end and `dd if=/dev/nvme0n1 bs=40M status=progress | gzip | nc 10.1.2.3:1234` on the sending end, after plugging an ethernet cable into both devices. In theory I could've probably also used the WiFi cards to set up a point to point network to speed up the transmission, but I couldn't be bothered with looking up how to make nc use mptcp like that.
- rincebrain 3y agoWell, if it's whole-disk encrypted, unless they told LUKS to pass TRIM through, you'd not be getting anything but essentially random data for the way the author described it.
- deleted 3y ago[deleted]
- jesprenj 3y ago> and I'm not so familiar with resizing LUKS. LUKS2 does not care about the size of the disk. If it's JSON header is present, it will by default treat the entire underlying block device as the LUKS encrypted volume/partition (sans the header), unless specified otherwise on the commandline.
- dmos62 3y agoI recently had to copy around 200gb of files over wifi. I used rsync to make sure a connection failure doesn't mean I have to start over and so that nothing is lost, but it took at least 6 hours. I wonder what could I have done better. Btw, what kinds of guarantees do you get with the dd method? Do you have to compare md5s of the resulting block level devices after?
- frankjr 3y ago> ...but it took at least 6 hours... Rsync cannot transfer more than one file at a time so if you were transferring a lot of small files that was probably the bottleneck. You could either use xargs/parallel to split the file list and run multiple instances of rsync or use something like rclone, which supports parallel transfers on its own.
- mkesper 3y agoYou could tar and zip the files into a netcat-pipe
- maccard 3y ago> I wonder what could I have done better. 6 hours is roughly 10MB/s, so you likely could have gone much much quicker. Did you compress with `-z`? If you could use ethernetyou probably could have done it at closer to 100MB/s on most deviceds, which would have been 35 minutes.
- dmos62 3y agoNo, I didn't use compression. Would it be useful over a high-bandwidth connection? I presume that it wasn't wifi bandwidth that was bottlenecking, though I've not really checked. One thing I could have done is found a way to track total progress, so that I could have noticed that this is going way too slow.
- skinner927 3y agoI’ve had compression noticeably speed things up even over a wired lan
- justsomehnguy 3y ago>> Since the new laptop didn't have an Ethernet port, I had to rely only on WiFi, and it took about 7 and a half hours to copy the entire 512GB That's because bs=40M and no status=progress.
- baq 3y agoIf you directly connect devices over WiFi without an intermediate AP you should be able to double your transfer speed. In this scenario it might have been worth it.
- rwmj 3y agoA lot of hassle compared to: nbdkit file /dev/nvme0n1 nbdcopy nbd://otherlaptop localfile
- throwawaaarrgh 3y agoThis is actually much better because nbdcopy can handle sparse files, you can set the number of connections and threads to number of cores, you can force a flush before exit, and enable a progress bar. For unencrypted drives it also supports TLS.
- verticalscaler 3y agoIf you really want a progress bar chuck in a 'pv' somewhere into the command posted at the top of the thread.
- rwmj 3y agoOr add nbdcopy -p option :-)
- brokenmachine 3y agoAnd "dd conv=sparse" to fix the other problem.
- M95D 3y agoI don't understand why he didn't pipe btrfs through network. Do a btrfs snapshot first, then btrfs send => nc => network => nc => brtfs receive. That way only blocks in use are sent.
- tarruda 3y agoThat's the first thing I thought when I saw he used btrfs. I use btrfs send/receive all the time via SSH and it works great. He could easily have setup SSH server in GRML live session. There's one caveat though: With btrfs it is not possible to send snapshots recursively, so if he had lots of recursive snapshots (which can happen in Docker/LXD/Incus), it is relatively hard to mirror the same structure in a new disk. I like btrfs, but recursive send/receive is one aspect ZFS is just better.
- streb-lo 3y agoYea, there needs to be a `snapshot -r` option or something. I like using subvolumes to manage what gets a snapshot but sometimes you want the full disk.
- deleted 3y ago[deleted]
- ptman 3y agoPipe with zstd (or similar)!
- HenryBemis 3y agoThis could also be titled: How to make one's life difficult 101 and/or I love experimenting. I've bought a few hard disks in my lifetime. Many years ago one such disk came bundled with Acronis TrueImage. Back in the day you could just buy the darn thing, after certain year they switched it to subscription model. Anyhoo.. I got the "one-off" TrueImage and I've been using it ever since. I've 'migrated' the CD (physical mini-CD) it to a USB flash disk, and have been using it ever since. I was curious for the solution as I recently bought a WinOS tablet, and would like to clone my PC to it, but this looks more like one too many hoop jumps to something that TrueImage can do in 2h hours (while watching a movie).
- greenshackle2 3y agoNext time, I'll be sure to do it the easy way by going back in time and getting a free copy of Acronis TrueImage in 1998.
- znpy 3y agoI love to see such progress, brought by nvme-over-tcp and systemd! Not so many years ago doing something similar (exporting a block device over network, and mounting it from another host) would have meant messing with iscsi which is a very cumbersome thing to do (and quite hard to master).
- pak9rabid 3y agoIf you don't need routing, you could always use the vastly simpler ATA-over-Ethernet protocol.
- everfrustrated 3y agoPretty cool. Is there any way to mount a remote volume on Mac OS using NVMe-TCP?
- throwaway81523 3y agoOh man I thought this was going to be about super fast networking by somehow connecting the PCI buses of two laptops. Instead it's the opposite, tunneling the NVMe protocol through regular Ethernet. Sort of like iScsi back in the day. Yeah there are much simpler approaches. I don't bother with container images but can install a new machine from an Ansible playbook in a few minutes. I do then have to copy user files (Borg restore) but that results in a clean new install.
- ahmetozer 3y ago[flagged]
- ahmetozer 3y ago[flagged]
- account42 3y agoI haven't actually "installed" an OS on my desktops/laptops in decades, always just copy over the files and adjust as needed. Usually just create a new filesystem though to use the opportunity to update the file system type / parameters (e.g. block size), encryption etc. and then rsync the files over. Still, if I was the planning ahead kind then using something more declarative like NixOS where you only need to copy your config and then automatically reinstall everything would probably be the better approach.
- jurgenkesker 3y agoOr just use Clonezilla? Then you also copy only the actual data blocks, and it can autoresize your partitions as well. That's how I always do it. True, I do always just take the NVME disk out of the laptop and put it in a highspeed dock.
- 1970-01-01 3y agoClonezilla is great. It's got one job and it usually succeeds the first time. My only complaint is the initial learning curve requires tinkering. It's still not at the trust level of fire and forget. Experimenting is recommended as backup is never the same thing as a backup and restore and even Clonezilla will have issues recreating partitions on disks that are very different from their source.
- stemlord 3y agoSpot on re: learning curve, it absolutely requires a test run or two before you'll get it right
- danw1979 3y agoCouldn’t they have just used netcat ?
- Levitating 3y agoI've switched laptops like 5 times in the past 2 years. Last time was yesterday. I just take my nvme drive out of the last one and I put it into the new one. I don't even lose my browser session. Edit: If you do this (and you should) you should configure predictable network adapter names (wlan0,eth0).
- jaxr 3y agoToday a new nvme is arriving and I need to do EXACTLY this. Thanks Mr poster :)
- telotortium 3y ago> copy took seven hours because the new laptop didn't have an Ethernet port Buy a USB to Ethernet adapter. It will come in handy in the future.
- tristor 3y agoThis is very clever, although somewhat unnecessary, but still useful to know. The one thing I would call out is the author used WiFi because one of the laptops didn't have Ethernet. I've encountered this situation several times myself and I've found that nearly every modern laptop supports USB in both modes, so you can simply use a USB-C cable to connect the two laptops and get a pseudo-Ethernet device this way to interconnect them, no need to use WiFi. This is hundreds of times faster than WiFi.
- timetraveller26 3y agoI was in a situation like this recently, and just want to say that if opening the pc/laptop and connecting it to the new machine is possible you should go that route, it's a little hassle compared to wait to a transfer over the network, waiting times can be deceptive. This reminds me one of my favorite articles https://blog.codinghorror.com/the-infinite-space-between-words/ https://blog.codinghorror.com/the-infinite-space-between-wor...
- slicktux 3y agoNothing like using Linux pipes and TCP to clone a rooted device for low level data recovery… ;)
- harikb 3y agoSince there is no mention of FDT (Fast Data Transfer) [1] Amazing software (transfer performance wise) written (unfortunately in) Java :) Unintuitive CLI options, but fastest transfer I have seen. So fast, if I don't artificially limit the speed, my command would sometimes hog the entire local network. -limit <rate> Restrict the transfer speed at the specified rate. K (KiloBytes/s), M (MegaBytes/s) or G (GigaBytes/s) may be used as suffixes. [1] http://monalisa.cern.ch/FDT/download.html http://monalisa.cern.ch/FDT/download.html PS: It does cause file-fragmentation on the destination side, but it shouldn't matter to anyone, practically.
- 9front 3y agoThere's nothing special about this NVMe-TCP scenario. It can be done with any storage protocol that works over TCP. Setup a target and an initiator and use dd to dump the data. iSCSI or FCoE could be used also. NVMe-TCP is just newer tech!
- INTPenis 3y agoIf setting up a new laptop takes a week then you need to learn about Ansible. I restore my working setup to a fresh install within minutes.
- pabs3 3y agoI just move my old drive to the new machine, wonder why that wasn't possible here.
- edpichler 3y agoCool! Does it work in any file system?
- edarchis 3y ago> Conclussion Is that a mix of conclusion and concussion that requires some ibuprofen ?