9 ms·

Wipe and reinstall a running Linux system via SSH (2017)

399 points by 1nvalid 8y ago

saagarjha 8y ago
The Stack Overflow answer linked is horrifyingly delightful: https://unix.stackexchange.com/questions/226872/how-to-shrink-root-filesystem-without-booting-a-livecd/227318#227318 https://unix.stackexchange.com/questions/226872/how-to-shrin...
- mehrdadn 8y ago
  I love how the first comment praises it for being "straightforward". For reference, this is how you shrink a file system on Windows: diskpart select volume C: shrink desired=4096 I guess it's nice that the Linux version lets you move the partition as well (or "shrink from behind", so to speak), but damn, "straightforward" is not a word I would have used to describe it.
  - mistaken 8y ago
    You can also resize partitions online in Linux. The point of the article was how to migrate the running system into memory and then wipe and reinstall the system. Try doing that from Windows :)
    - nolok 8y ago
      He is not talking about the article but the stack overflow question and answer of parent; and the point of that is to resize the root partition without booting into another OS, not to wipe or migrate to memory (ergo the question being asked: "How to shrink root filesystem without booting a livecd"). You can do that just fine on Windows, hell you can even do it with a nice GUI using computer management.
      - heavenlyblue 8y ago
        Yeah, but the Linux method is so much more generic; so can be used for more than just downsizing the partition.
        mehrdadn 8y ago
        It's far more generic but also practically useless for an end-user. Really, its only use seems to be for someone who is a remote sysadmin of some sort. You have to stop pretty much everything on the computer, and still go through a reboot. The only difference thing it buys you is being able to stay SSH'd at the cost of wasting so much more time and going through so much more risk and inconvenience. On Windows you'd just shrink and keep using the system as usual.
        laumars 8y ago
        I thought you'd still need to reboot on Windows? Last time I tried to do any changes to the system partition I had to reboot for it to take effect. If this is still the case then it's not really a far comparison to Windows because you could then do the same thing from a boot media in Linux (ie boot into it, resize and you're done). In fact you probably could also do that via a GUI (maybe gparted?). In any case I do agree that Linux does still leave a lot to be desired when it comes to making some of the more advanced file system operations far more complicated for end users than they need to be.
        mehrdadn 8y ago
        > I thought you'd still need to reboot on Windows? Not if all you're doing is shrinking from the end. When was the last time you tried? I'm guessing the XP days?
        laumars 8y ago
        > I'm guessing the XP days? To be fair, it might have been. I'm not a heavy Windows user but the fact I can't recall the last time I resized the system volume probably says more about how long ago it was.
      - zerr 8y ago
        Can't you do the same (in Linux) e.g. with GParted?
  - fps_doug 8y ago
    Yeah, admittedly if you don't know what's going on there it might feel pretty convoluted and "wtf am I doing here". OTOH, on Linux you could do this since forever, it's not been that long ago that Windows finally shipped tools to do so. Now actually, Windows seems even more convenient since you can easily shrink a FS that is online. On Linux I'm not even sure that is possible; maybe with btrfs but I never really needed to do that, so I have no idea.
    - nolok 8y ago
      > it's not been that long ago that Windows finally shipped tools to do so I beg to differ, Computer management > Disk management has allowed you to do that in a visual and safe way since at least windows seven in 2009. God knows since how long the underlying cli commands have been available.
      - lukeschlather 8y ago
        While setting up a new laptop I tried to use it and I ended up having to force kill Disk Management after it was "shrinking" for a couple hours. I think the underlying command had succeeded but it just froze. I've never had that kind of experience with (G)Parted, and I think I still trust the Linux tools more...
      - iforgotpassword 8y ago
        Iirc it couldn't resize mounted partitions (that was win8), and shrinking always just failed with no meaningful error message.
  - deleted 8y ago
    [deleted]
  - simias 8y ago
    This is mostly filesystem-dependent. If you have a LVM or ZFS volume for instance resizing is (usually) trivial. When you think about it resizing and in particular shrinking a filesystem is very much a non-trivial operation behind the curtains, it's not surprising that some filesystems don't support it out of the box especially since it's not a super common operation in the wild in my experience.
    - vbezhenar 8y ago
      It's not trivial and also it's likely to be very poorly tested scenario. I would recreate FS even if it supports resize.
      - filmor 8y ago
        For a lot of filesystems, growing is indeed trivial and even shrinking is explicitly supported. Why would this be less tested than any other feature?
        vbezhenar 8y ago
        Because users put their systems under various I/O workloads every day, but very few people resize their file systems and they do it very rarely.
        marcan_42 8y ago
        Case in point: I tried to shrink an NTFS filesystem offline with ntfsresize (offline!) and this happened: https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ This is a tool that is part of most Linux installers and tested by huge numbers of people, and yet things still went wrong. Shrinking filesystems is hard, and this was offline. Shrinking filesystems online is much harder.
      - simias 8y ago
        Thank you for making me double-check, you're right that ZFS can't shrink volumes but you can use resize2fs to shrink EXT2+ volumes offline relatively easily, see for instance https://blog.shadypixel.com/how-to-shrink-an-lvm-volume-safely/ https://blog.shadypixel.com/how-to-shrink-an-lvm-volume-safe... for a LVM + EXT example. What makes you say that it's poorly tested exactly? Of course if you want to shrink the rootfs without rebooting you'll have to do it while mounted and I'm not sure if that's supported by any Linux FS out there (outside of NFS I suppose). That being said I think that's understandable, implementing resizing of a live FS seems very tricky to get right and not extremely useful IMO. Can Windows really let you shrink NTFS while mounted? That's a pretty impressive feat if that's true, I wonder what motivated that.
  - nolok 8y ago
    I think by straightforward they mean you can follow it properly, that's because it's a full step by step that explains each step in enough details to be used, and even includes details for "what to do if you're not in the state expected". A lot of such guides have a fair amount of "then use tool X to do important part Y" without explaining any of it, making it horrible if you don't know X or the details of Y. Source: I once tried to fix and resize half-broken LVM volumes and if the sonology guys didn't take time to help me I would probably have just ended up wiping it all and recovering all 16 TB from backups because internet self help is still a harsh place on some linux fs matter.
    - reitanqild 8y ago
      So you are saying the Synology guys stepped you through it? That would be amazing support. (My Synology just works so I don't know what the support is like.)
      - nolok 8y ago
        They asked a permission and for me to fill a form so that they could remote connect to my NAS, and after investigating gave me on the support chat/list thingy the commands that they thought would fix it but they wouldn't run them themselves. Ended up working (issue was that a power failure happened during a shr2 reshape, and while the lvm volume had recovered mdadm hadn't and was blocked in a failed reshape loop, since it was a mix of mdadm and synology's own shr2 [which uses lvm to achieve its goal] trying to find help on the internet was next to impossible, which is why without them I would have wiped the whole thing). Was as a personnal customer and no paid support (beside having bought the product of course), within the second year of my purchase. Overall I've had a fair amount of support requests with them for my personnal NASes and the couple dozen I manage for professionnal purpose, and I'm very happy with that relationship. PS: my original support request was very detailled though, I did not just go and ask "doesn't work, fix it !"
        oger 8y ago
        Had some quite exceptional experiences with them as well. Good support for a decent product.
  - Zelmor 8y ago
    It's straightforward for me, but to each his own.
  - cormacrelf 8y ago
    I love that this line sounds like an old sea shanty: for i in dev proc sys run; do mount --move /oldroot/$i /$i; done
    - yellowapple 8y ago
      I now want to make it my goal for every line of code I write to be singable as a shanty.
    - grenoire 8y ago
      AYE, AYE, DONE!
    - perlgeek 8y ago
      https://en.wikipedia.org/wiki/Black_Perl https://en.wikipedia.org/wiki/Black_Perl comes to mind :-)
  - xorcist 8y ago
    And this is how to shrink a mounted filesystem in Linux when you use btrfs: btrfs filesystem resize 4g / It's just not supported for ext4. You can grow an ext4 file system, even when it is your currently mounted root, just not shrink it. While it's great that NTFS does (now) support it few other file systems do because it's a relatively uncommon and quite risky. I certainly wouldn't shrink mounted filesystems even on those that support it because storage systems are quite fickle and the threat of silent corruption is real. For most filesystems what you normally do is a full dump and restore. Closing all file handles and live migrating to a moved root filesystem is someone's idea of a showing off, it is absolutely not something anyone would do in production.
    - mehrdadn 8y ago
      Does that also shrink the partition? Otherwise you're not comparing equivalent things... P.S. I realize an existence proof makes us mathematically happy, but it's a little disingenuous to suggest file system experience on Linux and Windows is comparable merely because of the existence of some uncommon file system with similar capabilities. The reality is most Linux users are on ext4 rather than on btrfs or zfs, and most Windows users are on NTFS rather than ReFS or FAT32, so their experience with shrinking file systems is not going to be remotely comparable.
      - y4mi 8y ago
        > Does that also shrink the partition? Otherwise you're not comparing equivalent things... yes, it does. btfs is already the default in SUSE derivatives. It remains to be seen how the other distributions will handle it within the next ~3 yrs. its only just become stable enough to be usable in production.
        wtallis 8y ago
        The btrfs resize command only changes the size of the filesystem. A fdisk or similar command is needed to adjust the partition size as well.
  - marcosdumay 8y ago
    The "good" Linux answer would be to do the equivalent to your script on parted. I have no idea why an explanation of how to remotely install Linux on a machine while running Linux become the accepted answer. But about Windows, last year I had to resize the partitions on my work desktop, there are enough restrictions around changing things on the disk that Windows is installed that it became a week long research task for our IT support people.
    - mehrdadn 8y ago
      > But about Windows, last year I had to resize the partitions on my work desktop, there are enough restrictions around changing things on the disk that Windows is installed that it became a week long research task for our IT support people. You're suggesting the research phase for partition management is somehow easier on Linux than on Windows?
    - kilburn 8y ago
      > I have no idea why an explanation of how to remotely install Linux [mess with partitions] on a machine while running Linux The question was explicitly about shrinking the root filesystem without booting a livecd or any other OS. Gparted cannot do that directly because the root filesystem cannot be unmounted, and the appropriate answer for nearly everyone else (boot from a livecd and use gparted) doesn't apply because the question explicitly bars this option.
      - pbhjpbhj 8y ago
        The SO answer creates a root environment and uses `pivot_root`, seems you could "just" duplicate the root filesystem, pivot to that full copy, then unmount the original root and do what you wanted with it?
        kilburn 8y ago
        You can remount the root filesystem elsewhere, but not unmount it while it is still the rootfs. Your suggestion would work but it involves restarting nearly everything and having another partition with enough space to copy everything into. It can be done but is pretty hard in general, so gparted just doesn't support it. If you really want/need to do it, you can do so. Otherwise just use a livecd and make your life easier :)
  - blibble 8y ago
    I've rarely had this work successfully on Windows... normally after several hours it pops up an indecipherable error message presumably because it couldn't make the filesystem as small as I asked for, for whatever reason
- zcid 8y ago
  I can't believe I didn't know about column before. I've really been missing out.
lloeki 8y ago
Not needed anymore since DO supports custom OS images now but I found this script quite interesting, setting up a "blockplan" and applying it from a minimal root FS in RAM to entirely replace a Debian OS with an ArchLinux install unattended and without rebooting (except as a very last step to actually boot into the Arch kernel using the replaced GRUB2) https://github.com/gh2o/digitalocean-debian-to-arch#how-it-works https://github.com/gh2o/digitalocean-debian-to-arch#how-it-w... I forked it to try and convert Debian installs from ext4 to btrfs. Unfortunately while it does work, that's a very bad idea since btrfs-convert produces a fs that fails to operate properly on the long run (IIRC there is - was? - some increasing random space usage that cannot be reclaimed, ever). https://github.com/lloeki/digitalocean-ext4-to-btrfs/blob/master/ext2btrfs https://github.com/lloeki/digitalocean-ext4-to-btrfs/blob/ma... BTW, the process to convert to btrfs is excellent, only creating btrfs metadata in unallocated ext3 space, writing the btrfs header at the last minute, and using subvolumes allowing you to keep the ext3 metadata around as long as you want to roll back (obviously losing any subsequent modifications) because barring the header the whole ext filesystem and data is untouched. https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 I suppose Apple did something similar to convert from HFS+ to APFS so swiftly and so reliably.
simula67 8y ago
Previously : https://news.ycombinator.com/item?id=18741529 https://news.ycombinator.com/item?id=18741529 https://news.ycombinator.com/item?id=13618378 https://news.ycombinator.com/item?id=13618378 https://news.ycombinator.com/item?id=13622301 https://news.ycombinator.com/item?id=13622301
- exegete 8y ago
  The first two links you posted have 1 and 0 comments respectively. The third one is helpful though with lots of discussion.
thom_nic 8y ago
I actually implemented something on an embedded linux product once: a busybox initramfs that you could boot into "recovery mode," then along with dropbear, SSH into a completely in-memory system and re-flash the entire system image without having to pop out an SD card or connect a cable for DFU. The recovery mode could even be initiated remotely, so you could re-flash a device without ever touching it. Of course you have to be careful, if the re-flash failed you could be SOL :) Apparently I need to go back and improve it so we can re-flash without rebooting! These days you can use things like containers (Balena also looks very cool) to achieve a similar goal in possibly a "safer" way. But the idea of being able to re-flash the entire system while running it felt sort of like changing the engine of a car while driving it down the freeway!
- mikepurvis 8y ago
  I've implemented something very similar for upgrading headless Linux mobile robots. One nice property of this approach is that the in-memory installer environment and associated scripts can be common between USB-based install media and a remotely-triggerable kexec type installer. At first, it surprised me there wasn't more standard tooling out there for this kind of thing, but as I got more into it, I realised how specific to our particular needs my solution had become, and I could see how it would be hard to offer something generic that would be a good fit for a wide range of use-cases without being super-bloated.
stevekemp 8y ago
The closest I think I came to doing this was to migrate a running Debian system from being an i386 system to being an amd64-system, in-place: https://wiki.debian.org/CrossGrading https://wiki.debian.org/CrossGrading The first step is to update the kernel from an i386 one, so that it could run both i386 and amd64 binaries, but then you essentially overwrite every package with the version from the new architecture, and hope like hell it doesn't mess up. At the time I had a pair of servers, a mail-host, and a web-host, and I managed to successfully upgrade both, although it was a little scary. At least I had console access if things did get horribly screwed up.
- raverbashing 8y ago
  You could first chroot to a minimal stable system then update everything without having to rely on hope so much (but yeah big updates are always "exciting" and yum/apt/etc never get it completely right)
mirimir 8y ago
> This script does not have any provisions for exiting out of the new environment back into something sane. You will have to reboot when you're done. If you get anything wrong, your machine won't boot. Tough luck. So then "without rebooting" in the title is inaccurate, no? I haven't studied TFA carefully. But reinstalling running Linux systems via SSH is pretty routine, no? Using debian-installer with network-console, I mean. I occasionally reinstall remote servers with LUKS via SSH. I just login, build the installer, and reboot into it. Then I SSH to the installer, and almost complete it. Just before rebooting, I go to single-user mode, and setup dropbear in initramfs. Then I reboot. So OK, that's two reboots, not one. And if it fails, I just reboot using the control panel. If I've really fucked up, I reinstall and try again.
- zamadatix 8y ago
  Wipe and reinstall, not reinstall and load/run. I think the title is fair.
  - mirimir 8y ago
    I guess. But after reinstalling with LUKS, the box is pretty thoroughly wiped.
- cliffy 8y ago
  Maybe kexec-ing into the new kernel would be feasible? I investigated this many years ago to see if it was, but I found scant information on compatibility requirements when using kexec to hand over execution to an arbitrary kernel. The problem seems really hard though. One issue that stands out to me is that even if you properly shutdown the old kernel, will all system devices be in a 'good enough' state to be reinitialized properly by the new one? Or do some devices require a reboot for some reason?
  - mirimir 8y ago
    I have no clue. When I've built custom kernels, I'm pretty sure that the new kernel wasn't active until I rebooted. But rebooting after even dist-upgrade has just become automatic. For unattended upgrades, you can disable automatic reboot. But then, I think there's risk that some upgrades won't take effect.
Ixiaus 8y ago
You can also do this with NixOS...
vdloo 8y ago
We upgraded a whole fleet of AWS / Digitalocean instances without floating IPs from Ubuntu Precise to Xenial based on this method back in the summer of 2017. While obviously not having to do crazy stuff like this would be better, it's nice to know that it is possible if you really need it. https://www.reddit.com/r/programming/comments/6o7i8p/dont_run_this_on_any_system_you_expect_to_be_up/ https://www.reddit.com/r/programming/comments/6o7i8p/dont_ru... Recently we ran into another use-case for this in production actually, we needed to wipe a lot of servers in our datacenter remotely and we figured one of the options would be to install some OS in memory with the relevant wiping tools, pivot_root to that, unmount all disks and then perform the wipe. In the end we went a different route and opted for a custom PXE-boot image instead that the servers would boot into that scripted the whole thing.
ratling 8y ago
This whole thread just makes me glad I refuse to work on bare metal anymore for anything but the most niche of cases. AWS everything.
raintrees 8y ago
"you know you want to" Okay, I'm in.

399 points by 1nvalid 8y ago

The Stack Overflow answer linked is horrifyingly delightful: https://unix.stackexchange.com/questions/226872/how-to-shrink-root-filesystem-without-booting-a-livecd/227318#227318 https://unix.stackexchange.com/questions/226872/how-to-shrin...

mehrdadn 8y ago
I love how the first comment praises it for being "straightforward". For reference, this is how you shrink a file system on Windows: diskpart select volume C: shrink desired=4096 I guess it's nice that the Linux version lets you move the partition as well (or "shrink from behind", so to speak), but damn, "straightforward" is not a word I would have used to describe it.
- mistaken 8y ago
  You can also resize partitions online in Linux. The point of the article was how to migrate the running system into memory and then wipe and reinstall the system. Try doing that from Windows :)
  - nolok 8y ago
    He is not talking about the article but the stack overflow question and answer of parent; and the point of that is to resize the root partition without booting into another OS, not to wipe or migrate to memory (ergo the question being asked: "How to shrink root filesystem without booting a livecd"). You can do that just fine on Windows, hell you can even do it with a nice GUI using computer management.
    - heavenlyblue 8y ago
      Yeah, but the Linux method is so much more generic; so can be used for more than just downsizing the partition.
      - mehrdadn 8y ago
        It's far more generic but also practically useless for an end-user. Really, its only use seems to be for someone who is a remote sysadmin of some sort. You have to stop pretty much everything on the computer, and still go through a reboot. The only difference thing it buys you is being able to stay SSH'd at the cost of wasting so much more time and going through so much more risk and inconvenience. On Windows you'd just shrink and keep using the system as usual.
        laumars 8y ago
        I thought you'd still need to reboot on Windows? Last time I tried to do any changes to the system partition I had to reboot for it to take effect. If this is still the case then it's not really a far comparison to Windows because you could then do the same thing from a boot media in Linux (ie boot into it, resize and you're done). In fact you probably could also do that via a GUI (maybe gparted?). In any case I do agree that Linux does still leave a lot to be desired when it comes to making some of the more advanced file system operations far more complicated for end users than they need to be.
        mehrdadn 8y ago
        > I thought you'd still need to reboot on Windows? Not if all you're doing is shrinking from the end. When was the last time you tried? I'm guessing the XP days?
        laumars 8y ago
        > I'm guessing the XP days? To be fair, it might have been. I'm not a heavy Windows user but the fact I can't recall the last time I resized the system volume probably says more about how long ago it was.
    - zerr 8y ago
      Can't you do the same (in Linux) e.g. with GParted?
- fps_doug 8y ago
  Yeah, admittedly if you don't know what's going on there it might feel pretty convoluted and "wtf am I doing here". OTOH, on Linux you could do this since forever, it's not been that long ago that Windows finally shipped tools to do so. Now actually, Windows seems even more convenient since you can easily shrink a FS that is online. On Linux I'm not even sure that is possible; maybe with btrfs but I never really needed to do that, so I have no idea.
  - nolok 8y ago
    > it's not been that long ago that Windows finally shipped tools to do so I beg to differ, Computer management > Disk management has allowed you to do that in a visual and safe way since at least windows seven in 2009. God knows since how long the underlying cli commands have been available.
    - lukeschlather 8y ago
      While setting up a new laptop I tried to use it and I ended up having to force kill Disk Management after it was "shrinking" for a couple hours. I think the underlying command had succeeded but it just froze. I've never had that kind of experience with (G)Parted, and I think I still trust the Linux tools more...
    - iforgotpassword 8y ago
      Iirc it couldn't resize mounted partitions (that was win8), and shrinking always just failed with no meaningful error message.
- deleted 8y ago
  [deleted]
- simias 8y ago
  This is mostly filesystem-dependent. If you have a LVM or ZFS volume for instance resizing is (usually) trivial. When you think about it resizing and in particular shrinking a filesystem is very much a non-trivial operation behind the curtains, it's not surprising that some filesystems don't support it out of the box especially since it's not a super common operation in the wild in my experience.
  - vbezhenar 8y ago
    It's not trivial and also it's likely to be very poorly tested scenario. I would recreate FS even if it supports resize.
    - filmor 8y ago
      For a lot of filesystems, growing is indeed trivial and even shrinking is explicitly supported. Why would this be less tested than any other feature?
      - vbezhenar 8y ago
        Because users put their systems under various I/O workloads every day, but very few people resize their file systems and they do it very rarely.
        marcan_42 8y ago
        Case in point: I tried to shrink an NTFS filesystem offline with ntfsresize (offline!) and this happened: https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ https://marcan.st/2015/10/rescuing-a-broken-ntfs-filesystem/ This is a tool that is part of most Linux installers and tested by huge numbers of people, and yet things still went wrong. Shrinking filesystems is hard, and this was offline. Shrinking filesystems online is much harder.
    - simias 8y ago
      Thank you for making me double-check, you're right that ZFS can't shrink volumes but you can use resize2fs to shrink EXT2+ volumes offline relatively easily, see for instance https://blog.shadypixel.com/how-to-shrink-an-lvm-volume-safely/ https://blog.shadypixel.com/how-to-shrink-an-lvm-volume-safe... for a LVM + EXT example. What makes you say that it's poorly tested exactly? Of course if you want to shrink the rootfs without rebooting you'll have to do it while mounted and I'm not sure if that's supported by any Linux FS out there (outside of NFS I suppose). That being said I think that's understandable, implementing resizing of a live FS seems very tricky to get right and not extremely useful IMO. Can Windows really let you shrink NTFS while mounted? That's a pretty impressive feat if that's true, I wonder what motivated that.
- nolok 8y ago
  I think by straightforward they mean you can follow it properly, that's because it's a full step by step that explains each step in enough details to be used, and even includes details for "what to do if you're not in the state expected". A lot of such guides have a fair amount of "then use tool X to do important part Y" without explaining any of it, making it horrible if you don't know X or the details of Y. Source: I once tried to fix and resize half-broken LVM volumes and if the sonology guys didn't take time to help me I would probably have just ended up wiping it all and recovering all 16 TB from backups because internet self help is still a harsh place on some linux fs matter.
  - reitanqild 8y ago
    So you are saying the Synology guys stepped you through it? That would be amazing support. (My Synology just works so I don't know what the support is like.)
    - nolok 8y ago
      They asked a permission and for me to fill a form so that they could remote connect to my NAS, and after investigating gave me on the support chat/list thingy the commands that they thought would fix it but they wouldn't run them themselves. Ended up working (issue was that a power failure happened during a shr2 reshape, and while the lvm volume had recovered mdadm hadn't and was blocked in a failed reshape loop, since it was a mix of mdadm and synology's own shr2 [which uses lvm to achieve its goal] trying to find help on the internet was next to impossible, which is why without them I would have wiped the whole thing). Was as a personnal customer and no paid support (beside having bought the product of course), within the second year of my purchase. Overall I've had a fair amount of support requests with them for my personnal NASes and the couple dozen I manage for professionnal purpose, and I'm very happy with that relationship. PS: my original support request was very detailled though, I did not just go and ask "doesn't work, fix it !"
      - oger 8y ago
        Had some quite exceptional experiences with them as well. Good support for a decent product.
- Zelmor 8y ago
  It's straightforward for me, but to each his own.
- cormacrelf 8y ago
  I love that this line sounds like an old sea shanty: for i in dev proc sys run; do mount --move /oldroot/$i /$i; done
  - yellowapple 8y ago
    I now want to make it my goal for every line of code I write to be singable as a shanty.
  - grenoire 8y ago
    AYE, AYE, DONE!
  - perlgeek 8y ago
    https://en.wikipedia.org/wiki/Black_Perl https://en.wikipedia.org/wiki/Black_Perl comes to mind :-)
- xorcist 8y ago
  And this is how to shrink a mounted filesystem in Linux when you use btrfs: btrfs filesystem resize 4g / It's just not supported for ext4. You can grow an ext4 file system, even when it is your currently mounted root, just not shrink it. While it's great that NTFS does (now) support it few other file systems do because it's a relatively uncommon and quite risky. I certainly wouldn't shrink mounted filesystems even on those that support it because storage systems are quite fickle and the threat of silent corruption is real. For most filesystems what you normally do is a full dump and restore. Closing all file handles and live migrating to a moved root filesystem is someone's idea of a showing off, it is absolutely not something anyone would do in production.
  - mehrdadn 8y ago
    Does that also shrink the partition? Otherwise you're not comparing equivalent things... P.S. I realize an existence proof makes us mathematically happy, but it's a little disingenuous to suggest file system experience on Linux and Windows is comparable merely because of the existence of some uncommon file system with similar capabilities. The reality is most Linux users are on ext4 rather than on btrfs or zfs, and most Windows users are on NTFS rather than ReFS or FAT32, so their experience with shrinking file systems is not going to be remotely comparable.
    - y4mi 8y ago
      > Does that also shrink the partition? Otherwise you're not comparing equivalent things... yes, it does. btfs is already the default in SUSE derivatives. It remains to be seen how the other distributions will handle it within the next ~3 yrs. its only just become stable enough to be usable in production.
      - wtallis 8y ago
        The btrfs resize command only changes the size of the filesystem. A fdisk or similar command is needed to adjust the partition size as well.
- marcosdumay 8y ago
  The "good" Linux answer would be to do the equivalent to your script on parted. I have no idea why an explanation of how to remotely install Linux on a machine while running Linux become the accepted answer. But about Windows, last year I had to resize the partitions on my work desktop, there are enough restrictions around changing things on the disk that Windows is installed that it became a week long research task for our IT support people.
  - mehrdadn 8y ago
    > But about Windows, last year I had to resize the partitions on my work desktop, there are enough restrictions around changing things on the disk that Windows is installed that it became a week long research task for our IT support people. You're suggesting the research phase for partition management is somehow easier on Linux than on Windows?
  - kilburn 8y ago
    > I have no idea why an explanation of how to remotely install Linux [mess with partitions] on a machine while running Linux The question was explicitly about shrinking the root filesystem without booting a livecd or any other OS. Gparted cannot do that directly because the root filesystem cannot be unmounted, and the appropriate answer for nearly everyone else (boot from a livecd and use gparted) doesn't apply because the question explicitly bars this option.
    - pbhjpbhj 8y ago
      The SO answer creates a root environment and uses `pivot_root`, seems you could "just" duplicate the root filesystem, pivot to that full copy, then unmount the original root and do what you wanted with it?
      - kilburn 8y ago
        You can remount the root filesystem elsewhere, but not unmount it while it is still the rootfs. Your suggestion would work but it involves restarting nearly everything and having another partition with enough space to copy everything into. It can be done but is pretty hard in general, so gparted just doesn't support it. If you really want/need to do it, you can do so. Otherwise just use a livecd and make your life easier :)
- blibble 8y ago
  I've rarely had this work successfully on Windows... normally after several hours it pops up an indecipherable error message presumably because it couldn't make the filesystem as small as I asked for, for whatever reason
zcid 8y ago
I can't believe I didn't know about column before. I've really been missing out.

lloeki 8y ago

Not needed anymore since DO supports custom OS images now but I found this script quite interesting, setting up a "blockplan" and applying it from a minimal root FS in RAM to entirely replace a Debian OS with an ArchLinux install unattended and without rebooting (except as a very last step to actually boot into the Arch kernel using the replaced GRUB2) https://github.com/gh2o/digitalocean-debian-to-arch#how-it-works https://github.com/gh2o/digitalocean-debian-to-arch#how-it-w... I forked it to try and convert Debian installs from ext4 to btrfs. Unfortunately while it does work, that's a very bad idea since btrfs-convert produces a fs that fails to operate properly on the long run (IIRC there is - was? - some increasing random space usage that cannot be reclaimed, ever). https://github.com/lloeki/digitalocean-ext4-to-btrfs/blob/master/ext2btrfs https://github.com/lloeki/digitalocean-ext4-to-btrfs/blob/ma... BTW, the process to convert to btrfs is excellent, only creating btrfs metadata in unallocated ext3 space, writing the btrfs header at the last minute, and using subvolumes allowing you to keep the ext3 metadata around as long as you want to roll back (obviously losing any subsequent modifications) because barring the header the whole ext filesystem and data is untouched. https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 I suppose Apple did something similar to convert from HFS+ to APFS so swiftly and so reliably.

simula67 8y ago

Previously : https://news.ycombinator.com/item?id=18741529 https://news.ycombinator.com/item?id=18741529 https://news.ycombinator.com/item?id=13618378 https://news.ycombinator.com/item?id=13618378 https://news.ycombinator.com/item?id=13622301 https://news.ycombinator.com/item?id=13622301

exegete 8y ago
The first two links you posted have 1 and 0 comments respectively. The third one is helpful though with lots of discussion.

thom_nic 8y ago

I actually implemented something on an embedded linux product once: a busybox initramfs that you could boot into "recovery mode," then along with dropbear, SSH into a completely in-memory system and re-flash the entire system image without having to pop out an SD card or connect a cable for DFU. The recovery mode could even be initiated remotely, so you could re-flash a device without ever touching it. Of course you have to be careful, if the re-flash failed you could be SOL :) Apparently I need to go back and improve it so we can re-flash without rebooting! These days you can use things like containers (Balena also looks very cool) to achieve a similar goal in possibly a "safer" way. But the idea of being able to re-flash the entire system while running it felt sort of like changing the engine of a car while driving it down the freeway!

mikepurvis 8y ago
I've implemented something very similar for upgrading headless Linux mobile robots. One nice property of this approach is that the in-memory installer environment and associated scripts can be common between USB-based install media and a remotely-triggerable kexec type installer. At first, it surprised me there wasn't more standard tooling out there for this kind of thing, but as I got more into it, I realised how specific to our particular needs my solution had become, and I could see how it would be hard to offer something generic that would be a good fit for a wide range of use-cases without being super-bloated.

stevekemp 8y ago

The closest I think I came to doing this was to migrate a running Debian system from being an i386 system to being an amd64-system, in-place: https://wiki.debian.org/CrossGrading https://wiki.debian.org/CrossGrading The first step is to update the kernel from an i386 one, so that it could run both i386 and amd64 binaries, but then you essentially overwrite every package with the version from the new architecture, and hope like hell it doesn't mess up. At the time I had a pair of servers, a mail-host, and a web-host, and I managed to successfully upgrade both, although it was a little scary. At least I had console access if things did get horribly screwed up.

raverbashing 8y ago
You could first chroot to a minimal stable system then update everything without having to rely on hope so much (but yeah big updates are always "exciting" and yum/apt/etc never get it completely right)

mirimir 8y ago

> This script does not have any provisions for exiting out of the new environment back into something sane. You will have to reboot when you're done. If you get anything wrong, your machine won't boot. Tough luck. So then "without rebooting" in the title is inaccurate, no? I haven't studied TFA carefully. But reinstalling running Linux systems via SSH is pretty routine, no? Using debian-installer with network-console, I mean. I occasionally reinstall remote servers with LUKS via SSH. I just login, build the installer, and reboot into it. Then I SSH to the installer, and almost complete it. Just before rebooting, I go to single-user mode, and setup dropbear in initramfs. Then I reboot. So OK, that's two reboots, not one. And if it fails, I just reboot using the control panel. If I've really fucked up, I reinstall and try again.

zamadatix 8y ago
Wipe and reinstall, not reinstall and load/run. I think the title is fair.
- mirimir 8y ago
  I guess. But after reinstalling with LUKS, the box is pretty thoroughly wiped.
cliffy 8y ago
Maybe kexec-ing into the new kernel would be feasible? I investigated this many years ago to see if it was, but I found scant information on compatibility requirements when using kexec to hand over execution to an arbitrary kernel. The problem seems really hard though. One issue that stands out to me is that even if you properly shutdown the old kernel, will all system devices be in a 'good enough' state to be reinitialized properly by the new one? Or do some devices require a reboot for some reason?
- mirimir 8y ago
  I have no clue. When I've built custom kernels, I'm pretty sure that the new kernel wasn't active until I rebooted. But rebooting after even dist-upgrade has just become automatic. For unattended upgrades, you can disable automatic reboot. But then, I think there's risk that some upgrades won't take effect.

Ixiaus 8y ago

You can also do this with NixOS...

vdloo 8y ago

We upgraded a whole fleet of AWS / Digitalocean instances without floating IPs from Ubuntu Precise to Xenial based on this method back in the summer of 2017. While obviously not having to do crazy stuff like this would be better, it's nice to know that it is possible if you really need it. https://www.reddit.com/r/programming/comments/6o7i8p/dont_run_this_on_any_system_you_expect_to_be_up/ https://www.reddit.com/r/programming/comments/6o7i8p/dont_ru... Recently we ran into another use-case for this in production actually, we needed to wipe a lot of servers in our datacenter remotely and we figured one of the options would be to install some OS in memory with the relevant wiping tools, pivot_root to that, unmount all disks and then perform the wipe. In the end we went a different route and opted for a custom PXE-boot image instead that the servers would boot into that scripted the whole thing.

ratling 8y ago

This whole thread just makes me glad I refuse to work on bare metal anymore for anything but the most niche of cases. AWS everything.

raintrees 8y ago

"you know you want to" Okay, I'm in.