7 ms·

SSD death, tricky read-only filesystems, and systemd magic?

176 points by ingve 2y ago

db48x 2y ago
Spooky.
ciupicri 2y ago
WTF, it's 2024. systemd uses tmpfs for /tmp
- Arnavion 2y ago
  It's the default, but TFA might've masked it since they're dealing with an old laptop where they can upgrade the drive but not the RAM.
  - ciupicri 2y ago
    You're right, but this was introduced around 2012 [1] when that old laptop was manufactured. Also no one is forcing you to store GBs of data on it and indeed in practice most programs use only a couple of MBs. [1]: https://fedoraproject.org/wiki/Features/tmp-on-tmpfs https://fedoraproject.org/wiki/Features/tmp-on-tmpfs
    - NekkoDroid 2y ago
      And if programs use GBs of data, it should usually be in /var/tmp/ as that is backed usually by an actual storage device.
  - ojbyrne 2y ago
    I’m confused. The article talks about a Mac Mini, which is not a laptop. And an old enough one that the RAM is probably user-replaceable.
    - Arnavion 2y ago
      Apparently I'm illiterate. Indeed, not only is the Mac Mini a desktop, but also the author says in that same paragraph how they did in fact upgrade its RAM once in the past.
- KyleSanderson 2y ago
  same reaction, and the author not using ddrescue just makes this a tale about not following any sort of documentation when installing their distribution. There's really nothing in there that anyone should take away besides making sure they didn't hack up fstab and remove tmpfs.
  - mst 2y ago
    Having a "real" /tmp is not an uncommon preferences amongst sysadmins who've been at this for a while. Too many things that can wedge a system only leave evidence of why/how they did that in /tmp so still having it around post-reboot can be a huge aid to root cause analysis. It's a non-trivial trade-off but calling deliberately choosing that trade-off "hacking up fstab" doesn't strike me as a remotely fair description thereof.
    - ciupicri 2y ago
      Out of curiosity, what are those things that might crash the system while leaving evidence in /tmp, but not logging something useful? For what it's worth, OpenBSD which could be considered conservative says this about /tmp [1]: > Temporary files that are not preserved between system reboots. Periodically cleaned by daily(8). So no one should expect those files to be stored permanently. [1]: http://man.openbsd.org/hier http://man.openbsd.org/hier
      - mst 2y ago
        Generally, things that write tempfiles while processing data in /tmp and manage not to log enough because they wedge the system before it occurs to the code that anything's gone wrong enough to log. Yes, absolutely, "bad software, no cookie," but the usual culprit is some sort of vendor binary where the poor sod running the system has no control over that. BSD systems generally clean out an on-disk /tmp during the normal boot process, yes. There are ways around this, but when I've been responsible for babysitting craptastic vendorware it's always been on Linux or Solaris. Personally I've (after quite some grumbling about it) accepted /tmp being on tmpfs and just live with it; my current source of crankiness is "people who don't configure their systems to write to syslog" since if the box gets wedged by an I/O storm systemd will shoot systemd-journald in the head and then journald sometimes deletes all of your previous logs as it starts up.
- mhitza 2y ago
  Debian doesn't seem to use tmpfs for /tmp by default.
ajb 2y ago
journald also logs to /run/log/journal/ (a tmpfs) so until reboot you can still extract logs from it even if it didn't manage to write them to disk. See the 'storage' option in `man journald.conf`
- drycabinet 2y ago
  That part of the man page sounds confusing. Is it unconditional and enabled by default?
  - zokier 2y ago
    You have also RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles= options if you are concerned about journald overusing tmpfs. But yes, it seems unconditional that it writes the logs always somewhere, which is not unreasonable design imho.
    - ajb 2y ago
      Yes - unconditional by default, to be clear. I think if you set storage to 'none' it will switch it off completely
- iam-TJ 2y ago
  One can use: journalctl --relinquish-var to switch from /var/log/journal/ to /run/log/journal/ (which should be a tmpfs)
Arnavion 2y ago
>Even though I was using tar (and not dd, thank you very much) And from that linked article ( https://rachelbythebay.com/w/2011/12/11/cloning/ https://rachelbythebay.com/w/2011/12/11/cloning/ ): >Even if you tell it to continue after read errors ("noerror"), you've just silently corrupted something on the target disk. The idea of using `dd` / `ddrescue` is to get the contents of the drive off it ASAP before it fails. You don't want to mount it. You don't even know if the filesystem on it is sane. The priority right now is to get all the bytes off that you can, then analyze it at your leisure to recover what you can. For the same reason you would also not `dd` directly to the new disk. You'd do it to a storage area where you can inspect and manipulate it first. And: >Finally, there is the whole matter of geometry. [...] Will the partition table wind up in the right place? What about the boot sector data and secondary stuff like the locations where your boot loader might be looking? ... is not a problem for the same reason. Once you have the disk image off the failing drive, you're welcome to inspect it and copy individual files off it to the target drive at the filesystem layer (rsync, cp, etc). And so: >Second, you're dealing with individual files. If your tar or rsync (or whatever) fails to read a file due to some disk issue, you'll know about it. This is the right idea. It's just not optimal to do it from the failing drive itself.
- djbusby 2y ago
  I generally dd to an image file. (dd of=/data/bad-drive.dump) Then try tools to inspect fix the partition and FS, mount that and then rsync. That's what you're saying too, right?
  - Arnavion 2y ago
    Yes, exactly.
    - MadnessASAP 2y ago
      Nevermind trying to copy off of a failed drive that's still in use. Madness! Madness I say!
  - fencepost 2y ago
bravetraveler 2y ago
I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount/repair) it until you've made your copy. With journaling and the like (soft-RAID)... mounting at all can induce more writes than you expect; yes, even read-only. That's probably not what you want. Consistency is key. 'Just' (I know) use "dd", "ddrescue", or simply pv/cat/shell in+out redirects (some limitations apply) to evacuate it first. This gets you two things: 1) infinite tries: files/loop devices are cheap and can be copied for free 2) better odds: the new home isn't questionable, right? Stop thinking in terms of devices. You can write this drive to a file. You can write files to drives. Same for reading/checksums. It's all made up, yet it works. Magic. By skipping past the block level to the filesystem, one is limited in what they can recover. You're at the mercy of it being more sane than truly necessary. By mounting it, and potentially/indirectly writing [with bad media/connectors/whatever], you may be irrevocably breaking the filesystem. Perhaps they get into this, but wow. I know I'm not 'The One'. I hate to pretend to be; but this is day one data-preservation stuff. I'm almost certain we've all taken compliance training that covers this. Cutting corners like this is pretending to be The One; you don't know better. Nobody is perfect, I'm on my soap box because I've done the same thing - and want to save others. It hurts. I had to post when I got to "LOL it went RO, it's probably fine" (my words)... when no, the kernel interjecting to protect the data is not a reasonable first line of defense. Aside from what they chose to do... damage well-and-truly could already be done by their inaction. Having left it running/hoping for re-allocation. I wish I could have as much faith as they've shown in what little I've read so far. I close my rant with this: a little mechanical sympathy goes a long way. Now back to reading.
- viraptor 2y ago
  > mounting read only can induce more writes than you expect. Have you got a source for that? I can only find speculation on various sites, but nothing that explicitly confirms that outstanding journal entries would be written at RO mount time.
  - bravetraveler 2y ago
    Nothing readily available, can I be a source? The remaining nugget/memory on this I have is a fairly open ended... "it depends [on the filesystem]". It's been a while since I dove into this earnestly. IIRC there's no single journal mechanism - they all choose to do it (or not) in their own way. Journaling was just one example, too. Let's not ignore other quirks, like how BTRFS conflates volume/filesystem management... and may exhibit behavior impossible with XFS/ext{2,3,4} [under N failure conditions] WRT soft-RAID. By not mounting it, you're avoiding the "kernel/software filter" that adds entropy to an already-rich process. Side note: you caught me in a series of edits for phrasing :D
snvzz 2y ago
Consider rsync over tar. rsync has the (important) advantage it can generally be stopped and resumed. If the source drive's actually failing, I would instead ddrescue asap, and work with the (make it a read-only file) image. Should the filesystem need fsck, qemu-img can create a read-write image that is an overlay backed by a read-only file, and qemu-nbd can expose that through nbd, so that your kernel can use it.
- throw0101c 2y ago
  > Consider rsync over tar. I forget the exact details now, but in the past I found that cpio was the best option at times. It may have been circumstances when I didn't want the entire tree, but rather files that met specific criteria, and I was able to combine find and cpio (-o option) together.
  - NewJazz 2y ago
    Rsync has a nice glob based inclusion and exclusion rule system.
- deleted 2y ago
  [deleted]
benlivengood 2y ago
Obligatory recommendation to use ddrescue as a first pass. Personally I've been helped in at least three ways by this: * ddrescue doesn't retry immediately; it tries to get as many readable sectors/blocks as possible first and then goes back and retries. When there are many bad sectors this is way more efficient than retrying N times at each bad sector while additional readable data is sitting on the bad device. * ddrescue has configurable retries and its map file to perform rescue over any interruptions. If a device loses power or overheats and stops responding or for whatever other reason you don't have to get everything off the failing device in one pass. * If you have two failing RAID1 devices you can ddrescue from one and then from the other to get as much data as possible. If you need to recover audio CDs, use cdparanoia which takes a similar approach but uses a lot of speculative re-reads to avoid bad reconstruction efforts by some CD drives as much as possible.
userbinator 2y ago
I assume the only reason for not turning off and taking the drive out to image "offline" is because SSDs have been known to permanently brick themselves when power-cycled, but I agree with the others here that trying to interact with the filesystem on a dying storage device is a really bad idea. Incidentally, having experienced an SSD and HDD death within the past few months, the HDD definitely gave far more early warning signs (including some very audible ones) than the SSD, and failed in a "softer" fashion.
deleted 2y ago
[deleted]
blueflow 2y ago
TIL that systemd/journalctl is supposedly able to handle an read-only rootfs gracefully. I'm used to SysRq+R causing a lockup after a few seconds, on Debian & Ubuntu. Someone else dare to try?
txutxu 2y ago
Reminds me of those monitoring dashboards printing lot of unexpected output from checks, which in turn doesn't expect not being able to interact with state files on a read-only filesystem, reporting the wrong state (i.e. critical instead of unknown), etc. More fan of agents and an HA core for monitoring, than from direct push notifications, but still, used this smartd notifications like 15 years ago in Debian, and never realized of this issue (maybe because once the OS disc fails, we cared more about other issues than about the smartd notification). Those were the years were you could try to put the mechanical disc in the fridge for some minutes and try again, or with some Seagate disks, change the electronic circuit for other disk of the same model and recover the data (they used to fail in the electronic part more than the magnetic plates).
joshstrange 2y ago
> Those changes weren't managing to get flushed to the old disk with the now-RO filesystem, but they were apparently hanging out in buffers and were available for reading... or something? That makes sense, right? This is a fun issue that I've run into before. We had remote FreeBSD 11 boxes using ZFS. ZFS said the pool was fine, OS said the drive was fine, writes appear to work, but they are only going to RAM and are lost on reboot (which happens once you run out of memory, the box locks up, and someone on-site pulls the plug). Dealing with drives going bad with no indication is an interesting problem, especially when on reboot they appear to be working expect they have "time-traveled" backwards (to when the drive first went bad).
JonChesterfield 2y ago
Sort of orthogonal anecdote. I ran freebsd on a failing usb stick for about a year. It had copies=3 or similar on it and zfs just did it's best with fixing things up on the fly. Ran scrub fairly often, it told me things were bad, system kept running. That is to say, flash can stumble on with pretty high failure rate for much longer than I expected.
cf100clunk 2y ago
> smartd logging some complaint and then also barking about not having any way to send mail. What can I say - it's 2024 and I don't run SMTP stuff any more. Not hard if you use exim4, postfix, bsd-mailx, and a curses-based client like alpine to keep a low energy, simple local mail service running for just such things.

176 points by ingve 2y ago

db48x 2y ago

Spooky.

ciupicri 2y ago

WTF, it's 2024. systemd uses tmpfs for /tmp

Arnavion 2y ago
It's the default, but TFA might've masked it since they're dealing with an old laptop where they can upgrade the drive but not the RAM.
- ciupicri 2y ago
  You're right, but this was introduced around 2012 [1] when that old laptop was manufactured. Also no one is forcing you to store GBs of data on it and indeed in practice most programs use only a couple of MBs. [1]: https://fedoraproject.org/wiki/Features/tmp-on-tmpfs https://fedoraproject.org/wiki/Features/tmp-on-tmpfs
  - NekkoDroid 2y ago
    And if programs use GBs of data, it should usually be in /var/tmp/ as that is backed usually by an actual storage device.
- ojbyrne 2y ago
  I’m confused. The article talks about a Mac Mini, which is not a laptop. And an old enough one that the RAM is probably user-replaceable.
  - Arnavion 2y ago
    Apparently I'm illiterate. Indeed, not only is the Mac Mini a desktop, but also the author says in that same paragraph how they did in fact upgrade its RAM once in the past.
KyleSanderson 2y ago
same reaction, and the author not using ddrescue just makes this a tale about not following any sort of documentation when installing their distribution. There's really nothing in there that anyone should take away besides making sure they didn't hack up fstab and remove tmpfs.
- mst 2y ago
  Having a "real" /tmp is not an uncommon preferences amongst sysadmins who've been at this for a while. Too many things that can wedge a system only leave evidence of why/how they did that in /tmp so still having it around post-reboot can be a huge aid to root cause analysis. It's a non-trivial trade-off but calling deliberately choosing that trade-off "hacking up fstab" doesn't strike me as a remotely fair description thereof.
  - ciupicri 2y ago
    Out of curiosity, what are those things that might crash the system while leaving evidence in /tmp, but not logging something useful? For what it's worth, OpenBSD which could be considered conservative says this about /tmp [1]: > Temporary files that are not preserved between system reboots. Periodically cleaned by daily(8). So no one should expect those files to be stored permanently. [1]: http://man.openbsd.org/hier http://man.openbsd.org/hier
    - mst 2y ago
      Generally, things that write tempfiles while processing data in /tmp and manage not to log enough because they wedge the system before it occurs to the code that anything's gone wrong enough to log. Yes, absolutely, "bad software, no cookie," but the usual culprit is some sort of vendor binary where the poor sod running the system has no control over that. BSD systems generally clean out an on-disk /tmp during the normal boot process, yes. There are ways around this, but when I've been responsible for babysitting craptastic vendorware it's always been on Linux or Solaris. Personally I've (after quite some grumbling about it) accepted /tmp being on tmpfs and just live with it; my current source of crankiness is "people who don't configure their systems to write to syslog" since if the box gets wedged by an I/O storm systemd will shoot systemd-journald in the head and then journald sometimes deletes all of your previous logs as it starts up.
mhitza 2y ago
Debian doesn't seem to use tmpfs for /tmp by default.

ajb 2y ago

journald also logs to /run/log/journal/ (a tmpfs) so until reboot you can still extract logs from it even if it didn't manage to write them to disk. See the 'storage' option in `man journald.conf`

drycabinet 2y ago
That part of the man page sounds confusing. Is it unconditional and enabled by default?
- zokier 2y ago
  You have also RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles= options if you are concerned about journald overusing tmpfs. But yes, it seems unconditional that it writes the logs always somewhere, which is not unreasonable design imho.
  - ajb 2y ago
    Yes - unconditional by default, to be clear. I think if you set storage to 'none' it will switch it off completely
iam-TJ 2y ago
One can use: journalctl --relinquish-var to switch from /var/log/journal/ to /run/log/journal/ (which should be a tmpfs)

Arnavion 2y ago

>Even though I was using tar (and not dd, thank you very much) And from that linked article ( https://rachelbythebay.com/w/2011/12/11/cloning/ https://rachelbythebay.com/w/2011/12/11/cloning/ ): >Even if you tell it to continue after read errors ("noerror"), you've just silently corrupted something on the target disk. The idea of using `dd` / `ddrescue` is to get the contents of the drive off it ASAP before it fails. You don't want to mount it. You don't even know if the filesystem on it is sane. The priority right now is to get all the bytes off that you can, then analyze it at your leisure to recover what you can. For the same reason you would also not `dd` directly to the new disk. You'd do it to a storage area where you can inspect and manipulate it first. And: >Finally, there is the whole matter of geometry. [...] Will the partition table wind up in the right place? What about the boot sector data and secondary stuff like the locations where your boot loader might be looking? ... is not a problem for the same reason. Once you have the disk image off the failing drive, you're welcome to inspect it and copy individual files off it to the target drive at the filesystem layer (rsync, cp, etc). And so: >Second, you're dealing with individual files. If your tar or rsync (or whatever) fails to read a file due to some disk issue, you'll know about it. This is the right idea. It's just not optimal to do it from the failing drive itself.

djbusby 2y ago
I generally dd to an image file. (dd of=/data/bad-drive.dump) Then try tools to inspect fix the partition and FS, mount that and then rsync. That's what you're saying too, right?
- Arnavion 2y ago
  Yes, exactly.
  - MadnessASAP 2y ago
    Nevermind trying to copy off of a failed drive that's still in use. Madness! Madness I say!
- fencepost 2y ago

bravetraveler 2y ago

I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount/repair) it until you've made your copy. With journaling and the like (soft-RAID)... mounting at all can induce more writes than you expect; yes, even read-only. That's probably not what you want. Consistency is key. 'Just' (I know) use "dd", "ddrescue", or simply pv/cat/shell in+out redirects (some limitations apply) to evacuate it first. This gets you two things: 1) infinite tries: files/loop devices are cheap and can be copied for free 2) better odds: the new home isn't questionable, right? Stop thinking in terms of devices. You can write this drive to a file. You can write files to drives. Same for reading/checksums. It's all made up, yet it works. Magic. By skipping past the block level to the filesystem, one is limited in what they can recover. You're at the mercy of it being more sane than truly necessary. By mounting it, and potentially/indirectly writing [with bad media/connectors/whatever], you may be irrevocably breaking the filesystem. Perhaps they get into this, but wow. I know I'm not 'The One'. I hate to pretend to be; but this is day one data-preservation stuff. I'm almost certain we've all taken compliance training that covers this. Cutting corners like this is pretending to be The One; you don't know better. Nobody is perfect, I'm on my soap box because I've done the same thing - and want to save others. It hurts. I had to post when I got to "LOL it went RO, it's probably fine" (my words)... when no, the kernel interjecting to protect the data is not a reasonable first line of defense. Aside from what they chose to do... damage well-and-truly could already be done by their inaction. Having left it running/hoping for re-allocation. I wish I could have as much faith as they've shown in what little I've read so far. I close my rant with this: a little mechanical sympathy goes a long way. Now back to reading.

viraptor 2y ago
> mounting read only can induce more writes than you expect. Have you got a source for that? I can only find speculation on various sites, but nothing that explicitly confirms that outstanding journal entries would be written at RO mount time.
- bravetraveler 2y ago
  Nothing readily available, can I be a source? The remaining nugget/memory on this I have is a fairly open ended... "it depends [on the filesystem]". It's been a while since I dove into this earnestly. IIRC there's no single journal mechanism - they all choose to do it (or not) in their own way. Journaling was just one example, too. Let's not ignore other quirks, like how BTRFS conflates volume/filesystem management... and may exhibit behavior impossible with XFS/ext{2,3,4} [under N failure conditions] WRT soft-RAID. By not mounting it, you're avoiding the "kernel/software filter" that adds entropy to an already-rich process. Side note: you caught me in a series of edits for phrasing :D

snvzz 2y ago

Consider rsync over tar. rsync has the (important) advantage it can generally be stopped and resumed. If the source drive's actually failing, I would instead ddrescue asap, and work with the (make it a read-only file) image. Should the filesystem need fsck, qemu-img can create a read-write image that is an overlay backed by a read-only file, and qemu-nbd can expose that through nbd, so that your kernel can use it.

throw0101c 2y ago
> Consider rsync over tar. I forget the exact details now, but in the past I found that cpio was the best option at times. It may have been circumstances when I didn't want the entire tree, but rather files that met specific criteria, and I was able to combine find and cpio (-o option) together.
- NewJazz 2y ago
  Rsync has a nice glob based inclusion and exclusion rule system.
deleted 2y ago
[deleted]

benlivengood 2y ago

Obligatory recommendation to use ddrescue as a first pass. Personally I've been helped in at least three ways by this: * ddrescue doesn't retry immediately; it tries to get as many readable sectors/blocks as possible first and then goes back and retries. When there are many bad sectors this is way more efficient than retrying N times at each bad sector while additional readable data is sitting on the bad device. * ddrescue has configurable retries and its map file to perform rescue over any interruptions. If a device loses power or overheats and stops responding or for whatever other reason you don't have to get everything off the failing device in one pass. * If you have two failing RAID1 devices you can ddrescue from one and then from the other to get as much data as possible. If you need to recover audio CDs, use cdparanoia which takes a similar approach but uses a lot of speculative re-reads to avoid bad reconstruction efforts by some CD drives as much as possible.

userbinator 2y ago

I assume the only reason for not turning off and taking the drive out to image "offline" is because SSDs have been known to permanently brick themselves when power-cycled, but I agree with the others here that trying to interact with the filesystem on a dying storage device is a really bad idea. Incidentally, having experienced an SSD and HDD death within the past few months, the HDD definitely gave far more early warning signs (including some very audible ones) than the SSD, and failed in a "softer" fashion.

deleted 2y ago

[deleted]

blueflow 2y ago

TIL that systemd/journalctl is supposedly able to handle an read-only rootfs gracefully. I'm used to SysRq+R causing a lockup after a few seconds, on Debian & Ubuntu. Someone else dare to try?

txutxu 2y ago

Reminds me of those monitoring dashboards printing lot of unexpected output from checks, which in turn doesn't expect not being able to interact with state files on a read-only filesystem, reporting the wrong state (i.e. critical instead of unknown), etc. More fan of agents and an HA core for monitoring, than from direct push notifications, but still, used this smartd notifications like 15 years ago in Debian, and never realized of this issue (maybe because once the OS disc fails, we cared more about other issues than about the smartd notification). Those were the years were you could try to put the mechanical disc in the fridge for some minutes and try again, or with some Seagate disks, change the electronic circuit for other disk of the same model and recover the data (they used to fail in the electronic part more than the magnetic plates).

joshstrange 2y ago

> Those changes weren't managing to get flushed to the old disk with the now-RO filesystem, but they were apparently hanging out in buffers and were available for reading... or something? That makes sense, right? This is a fun issue that I've run into before. We had remote FreeBSD 11 boxes using ZFS. ZFS said the pool was fine, OS said the drive was fine, writes appear to work, but they are only going to RAM and are lost on reboot (which happens once you run out of memory, the box locks up, and someone on-site pulls the plug). Dealing with drives going bad with no indication is an interesting problem, especially when on reboot they appear to be working expect they have "time-traveled" backwards (to when the drive first went bad).

JonChesterfield 2y ago

Sort of orthogonal anecdote. I ran freebsd on a failing usb stick for about a year. It had copies=3 or similar on it and zfs just did it's best with fixing things up on the fly. Ran scrub fairly often, it told me things were bad, system kept running. That is to say, flash can stumble on with pretty high failure rate for much longer than I expected.

cf100clunk 2y ago

> smartd logging some complaint and then also barking about not having any way to send mail. What can I say - it's 2024 and I don't run SMTP stuff any more. Not hard if you use exim4, postfix, bsd-mailx, and a curses-based client like alpine to keep a low energy, simple local mail service running for just such things.