4 ms·

I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount/repair) it until you've made your c

I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount/repair) it until you've made your copy. With journaling and the like (soft-RAID)... mounting at all can induce more writes than you expect; yes, even read-only. That's probably not what you want. Consistency is key. 'Just' (I know) use "dd", "ddrescue", or simply pv/cat/shell in+out redirects (some limitations apply) to evacuate it first. This gets you two things: 1) infinite tries: files/loop devices are cheap and can be copied for free 2) better odds: the new home isn't questionable, right? Stop thinking in terms of devices. You can write this drive to a file. You can write files to drives. Same for reading/checksums. It's all made up, yet it works. Magic. By skipping past the block level to the filesystem, one is limited in what they can recover. You're at the mercy of it being more sane than truly necessary. By mounting it, and potentially/indirectly writing [with bad media/connectors/whatever], you may be irrevocably breaking the filesystem. Perhaps they get into this, but wow. I know I'm not 'The One'. I hate to pretend to be; but this is day one data-preservation stuff. I'm almost certain we've all taken compliance training that covers this. Cutting corners like this is pretending to be The One; you don't know better. Nobody is perfect, I'm on my soap box because I've done the same thing - and want to save others. It hurts. I had to post when I got to "LOL it went RO, it's probably fine" (my words)... when no, the kernel interjecting to protect the data is not a reasonable first line of defense. Aside from what they chose to do... damage well-and-truly could already be done by their inaction. Having left it running/hoping for re-allocation. I wish I could have as much faith as they've shown in what little I've read so far. I close my rant with this: a little mechanical sympathy goes a long way. Now back to reading.

viraptor 2y ago
> mounting read only can induce more writes than you expect. Have you got a source for that? I can only find speculation on various sites, but nothing that explicitly confirms that outstanding journal entries would be written at RO mount time.
- bravetraveler 2y ago
  Nothing readily available, can I be a source? The remaining nugget/memory on this I have is a fairly open ended... "it depends [on the filesystem]". It's been a while since I dove into this earnestly. IIRC there's no single journal mechanism - they all choose to do it (or not) in their own way. Journaling was just one example, too. Let's not ignore other quirks, like how BTRFS conflates volume/filesystem management... and may exhibit behavior impossible with XFS/ext{2,3,4} [under N failure conditions] WRT soft-RAID. By not mounting it, you're avoiding the "kernel/software filter" that adds entropy to an already-rich process. Side note: you caught me in a series of edits for phrasing :D
  - deleted 2y ago
    [deleted]
- mrob 2y ago
  From the util-linux "mount" man page: "Note that, depending on the filesystem type, state and kernel behavior, the system may still write to the device. For example, ext3 and ext4 will replay the journal if the filesystem is dirty. To prevent this kind of write access, you may want to mount an ext3 or ext4 filesystem with the ro,noload mount options or set the block device itself to read-only mode, see the blockdev(8) command."
  - viraptor 2y ago
    This seems to be the relevant behaviour from ext4: https://github.com/torvalds/linux/blob/8c06da67d0bd3139a97f301b4aa9c482b9d4f29e/fs/ext4/super.c#L6019 https://github.com/torvalds/linux/blob/8c06da67d0bd3139a97f3...
- tryauuum 2y ago
  ext4 read only mount can still replay journal, see the "noload" mount option
hun3 2y ago
Right, you need at least some dm-linear device mapper device in between to ensure a device is RO I think. Also, ext4 has separate "noload" option that actually instructs it not to touch the disk (ofc relying solely on "ro,noload" to not mess with your broken FS is a bad idea)
jeffbee 2y ago
There's nothing special about dd. You can, and should, simply use `cp`
- bravetraveler 2y ago
  That's the least important part about my post. I mentioned several things, one being the shell you probably already have running, that all will do a perfectly fine job - to hopefully demonstrate that. 'cp' is sparse aware. Yay. Is that a good thing, actually? If the source has zeroes, I want to write zeroes. Not sure... probably fine. I've been awake too long. 'ddrescue' is special, but again, consistency is key. I don't really care how it's copied. I just question 'should' - there are advantages elsewhere. edit: My main gist is this - living off the land applies here, too! Whatever option is available. Make the copy, check the checksums. Pull out the bigger gun if necessary. Never know when you have to rely on some binary that's surviving only in caches. Stuck with a broken system and no live environment in sight.
- o11c 2y ago
  There is, however, something special about `ddrescue` (note that there are 2 different projects named thus; I don't remember the difference) - it does all the okay sectors, then goes back and retries bad sectors. Just be sure to store the ddrescue state file somewhere persistent.
- mrcode007 2y ago
  Please show me how you cp a raw device without the concept of files. I’d love to learn. $ cp /dev/nvme0n1p1 /mnt/my-special-backup ?
  - jeffbee 2y ago
    Yes
    - mrcode007 2y ago
      Ha. Today I learned. Is this a portable approach across macOS, Linux and BSD family ?
_wire_ 2y ago
> I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount) it until you've made your copy. A further clarification of your point... If you are a wearing a forensics hat and engaged in formal recovery effort, your point is significant good advice. But when you are a regular joe trying to deal with a failing device, it's usually not clear when you've put on the forensics hat; you just find yourself struggling with a device that is not returning your data. And while you are in this moment of discovery the failure mode, the system is trying to do all its normal stuff, including maintaining the journal. So if playing the journal is a hazard to your data, it has likely already been encountered and any corruption as a result of journal processing been done. At some point the regular joe may decide to put on the forensic hat and formally rescue the device. In my experience, this is not necessarily the most productive next step. It depends on the device, its failure mode, the data and the usage scenario. Too many variables to generalize about tactics. No matter the value of your data, it's wise to begin a disciplined recovery as soon as possible. Where backups are the best place to start any recovery effort. However, once you decide to rescue the device — as opposed to nurse it along to gain access to something — that's the clear threshold to become scrupulous about read-only access: It's important to keep both the source and the destination devices in a read-only state from the beginning to end of the rescue to avoid further corruption of the copy. In this light, any journal processing that occurs for read-only mounts, presents a possibility hazardous condition with implications beyond the scope of casual system operation. This is the province of a proper forensics regime, with a rescue system configuration and protocol designed to prevent tainting the target, and also beyond the scope of a user comments on a help thread. Given all the other uncertainties, getting to the bottom of what a read-only mount actually does with the journal on your system at hand is probably not productive. But arranging for read-only mounts is given the uncertainty is certainly productive, so prevent auto-mounts while rescuing. If you want to use GNU ddescue, I've written a bash script to help with preventing auto-mount for Linux and macOS. github @c-o-pr ddrescue-helper Basically the script just writes /etc/fstab with the UUID of the filesystem and "noauto" option and unmounts the device. But there's other logic in the script that may be useful.
deleted 2y ago
[deleted]
bravetraveler 2y ago
I can't edit now... but one thing I'd like to change - it's not fair to call this their first line of defense. Rachel did the most important thing right: taking backups at the first sign of trouble, presumably beforehand too. I just feel they're off the mark foregoing block devices and having a strange process as a result... "because images are big". Speeding up doesn't normally quiet a squeaky wheel. Not to say it can't... but let's place bets :) There are options for this concern without compromising the efficacy of the whole process; for example, {de,}compression. Yes, you care about the files, but to recover them... you don't get to ignore the details of the filesystem - or the constraints you're allowing it (and yourself) to operate under. The takeaway from this isn't "substitute well-established recovery processes..." but, make sure you have backups so you can afford two perks: * ignore a failing drive for weeks * not care *at all* about your attempts to recover the source filesystem The filesystem, your storage, everything will lie. Prepare for it; make copies and accept this will take time. This chestnut still applies: "If you don't have time to do it right, you must have time to do it again." The recovery or your data?
- oceanplexian 2y ago
  The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates. They aren’t designed for a linux server, they’re designed for a disposable Windows laptop that will be chucked in the bin after a year or two. I’ve learned this lesson the hard way over, and over, and over again with these little cheap SSDs.
  - chipdart 2y ago
    > The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates. What do you mean by "endurance"? The only definition I found was the number of writes during the warranty time, and you're talking about a use case involving a very low volume of writes (i.e., save a file, keep it up for eternity). This means that warranty time (more specifically, any failure mode at all beyond excess writes) is the most likely failure mode. Not writes, just the SSD being old. Samsung advertises between 3 years and 10 years of warranty. The typical warranty of HDDs is between 3 and 5 years. Doesn't this suggest that buying you a decent SSD buys you at worst the same reliability? https://semiconductor.samsung.com/consumer-storage/support/warranty/ https://semiconductor.samsung.com/consumer-storage/support/w...
guenthert 2y ago
Dunno, man. It's an old Mac-mini, single disk (the SSD is explicitly called out) fairly obviously used for private stuff. You can wring it. In a production environment you cut you losses and restore from back-up. No-one wants to deal with half-baked recovery operations which might or might not yield a consistent state with (a little) less, but unknown, data loss.
- bravetraveler 2y ago
  It's not lost on me that this is a low stakes situation, they had backups and could afford being artful in their process. That's what makes this wheel-spinning so confounding.
  - abofh 2y ago
    I don't think this was wheel-spinning so much as yarn-pulling, finding interesting tid-bits and seeing where they go. The most interesting things (to me) are when something doesn't quite line up with expectations, and then digging until I understand what the difference was - usually I misunderstood something (or failed to recall something, like in this case, disk/filesystem caches) - and I find myself more enlightend. I totally get doing this to learn because it was an interesting process. But yea, if this were for money? I'd just restore from backups and follow the SOP - because for money, the _why_ something failed is often unimportant, the remedy is the same -- reboot, replace, correlate, correct.
    - bravetraveler 2y ago
      Playing with Schrodinger's stuff. The storage industry/equipment lies so pervasively, even when it's working as intended More importantly, perhaps, is this is a "brain worm" being spread to others. Not to armchair too much, but there's a certain responsibility by publishing your writing. I'd hope for more dutiful research before publishing/risking data with anything even remotely nearing suggestion. Loud/clear warnings. We're all playing the Telephone Game whether we want to or not.
matheusmoreira 2y ago
It's not a soap box, it's good advice!! Most people use default options which means they mount with access times enabled where every read causes a write. File systems are more complex than they seem. They also do lots of automatic house keeping in the background. Even the relatively simple ext4 has journaling. Modern file systems will even move and rebalance data in the background. My laptop fell down on the floor recently. Hard disk started reporting errors immediately afterwards. I picked it up to assess the damage and immediately noticed that Linux remounted the drive read only. I turned it off, bought a 4 TB SSD, flashed a live Linux ISO and started reviewing the GNU ddrescue documentation. When recovering data, things should be as static as possible. I'd keep the disk platters from spinning if I could.

viraptor 2y ago

> mounting read only can induce more writes than you expect. Have you got a source for that? I can only find speculation on various sites, but nothing that explicitly confirms that outstanding journal entries would be written at RO mount time.

bravetraveler 2y ago
Nothing readily available, can I be a source? The remaining nugget/memory on this I have is a fairly open ended... "it depends [on the filesystem]". It's been a while since I dove into this earnestly. IIRC there's no single journal mechanism - they all choose to do it (or not) in their own way. Journaling was just one example, too. Let's not ignore other quirks, like how BTRFS conflates volume/filesystem management... and may exhibit behavior impossible with XFS/ext{2,3,4} [under N failure conditions] WRT soft-RAID. By not mounting it, you're avoiding the "kernel/software filter" that adds entropy to an already-rich process. Side note: you caught me in a series of edits for phrasing :D
- deleted 2y ago
  [deleted]
mrob 2y ago
From the util-linux "mount" man page: "Note that, depending on the filesystem type, state and kernel behavior, the system may still write to the device. For example, ext3 and ext4 will replay the journal if the filesystem is dirty. To prevent this kind of write access, you may want to mount an ext3 or ext4 filesystem with the ro,noload mount options or set the block device itself to read-only mode, see the blockdev(8) command."
- viraptor 2y ago
  This seems to be the relevant behaviour from ext4: https://github.com/torvalds/linux/blob/8c06da67d0bd3139a97f301b4aa9c482b9d4f29e/fs/ext4/super.c#L6019 https://github.com/torvalds/linux/blob/8c06da67d0bd3139a97f3...
tryauuum 2y ago
ext4 read only mount can still replay journal, see the "noload" mount option

hun3 2y ago

Right, you need at least some dm-linear device mapper device in between to ensure a device is RO I think. Also, ext4 has separate "noload" option that actually instructs it not to touch the disk (ofc relying solely on "ro,noload" to not mess with your broken FS is a bad idea)

jeffbee 2y ago

There's nothing special about dd. You can, and should, simply use `cp`

bravetraveler 2y ago
That's the least important part about my post. I mentioned several things, one being the shell you probably already have running, that all will do a perfectly fine job - to hopefully demonstrate that. 'cp' is sparse aware. Yay. Is that a good thing, actually? If the source has zeroes, I want to write zeroes. Not sure... probably fine. I've been awake too long. 'ddrescue' is special, but again, consistency is key. I don't really care how it's copied. I just question 'should' - there are advantages elsewhere. edit: My main gist is this - living off the land applies here, too! Whatever option is available. Make the copy, check the checksums. Pull out the bigger gun if necessary. Never know when you have to rely on some binary that's surviving only in caches. Stuck with a broken system and no live environment in sight.
o11c 2y ago
There is, however, something special about `ddrescue` (note that there are 2 different projects named thus; I don't remember the difference) - it does all the okay sectors, then goes back and retries bad sectors. Just be sure to store the ddrescue state file somewhere persistent.
mrcode007 2y ago
Please show me how you cp a raw device without the concept of files. I’d love to learn. $ cp /dev/nvme0n1p1 /mnt/my-special-backup ?
- jeffbee 2y ago
  Yes
  - mrcode007 2y ago
    Ha. Today I learned. Is this a portable approach across macOS, Linux and BSD family ?

_wire_ 2y ago

> I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount) it until you've made your copy. A further clarification of your point... If you are a wearing a forensics hat and engaged in formal recovery effort, your point is significant good advice. But when you are a regular joe trying to deal with a failing device, it's usually not clear when you've put on the forensics hat; you just find yourself struggling with a device that is not returning your data. And while you are in this moment of discovery the failure mode, the system is trying to do all its normal stuff, including maintaining the journal. So if playing the journal is a hazard to your data, it has likely already been encountered and any corruption as a result of journal processing been done. At some point the regular joe may decide to put on the forensic hat and formally rescue the device. In my experience, this is not necessarily the most productive next step. It depends on the device, its failure mode, the data and the usage scenario. Too many variables to generalize about tactics. No matter the value of your data, it's wise to begin a disciplined recovery as soon as possible. Where backups are the best place to start any recovery effort. However, once you decide to rescue the device — as opposed to nurse it along to gain access to something — that's the clear threshold to become scrupulous about read-only access: It's important to keep both the source and the destination devices in a read-only state from the beginning to end of the rescue to avoid further corruption of the copy. In this light, any journal processing that occurs for read-only mounts, presents a possibility hazardous condition with implications beyond the scope of casual system operation. This is the province of a proper forensics regime, with a rescue system configuration and protocol designed to prevent tainting the target, and also beyond the scope of a user comments on a help thread. Given all the other uncertainties, getting to the bottom of what a read-only mount actually does with the journal on your system at hand is probably not productive. But arranging for read-only mounts is given the uncertainty is certainly productive, so prevent auto-mounts while rescuing. If you want to use GNU ddescue, I've written a bash script to help with preventing auto-mount for Linux and macOS. github @c-o-pr ddrescue-helper Basically the script just writes /etc/fstab with the UUID of the filesystem and "noauto" option and unmounts the device. But there's other logic in the script that may be useful.

deleted 2y ago

[deleted]

bravetraveler 2y ago

I can't edit now... but one thing I'd like to change - it's not fair to call this their first line of defense. Rachel did the most important thing right: taking backups at the first sign of trouble, presumably beforehand too. I just feel they're off the mark foregoing block devices and having a strange process as a result... "because images are big". Speeding up doesn't normally quiet a squeaky wheel. Not to say it can't... but let's place bets :) There are options for this concern without compromising the efficacy of the whole process; for example, {de,}compression. Yes, you care about the files, but to recover them... you don't get to ignore the details of the filesystem - or the constraints you're allowing it (and yourself) to operate under. The takeaway from this isn't "substitute well-established recovery processes..." but, make sure you have backups so you can afford two perks: * ignore a failing drive for weeks * not care *at all* about your attempts to recover the source filesystem The filesystem, your storage, everything will lie. Prepare for it; make copies and accept this will take time. This chestnut still applies: "If you don't have time to do it right, you must have time to do it again." The recovery or your data?

oceanplexian 2y ago
The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates. They aren’t designed for a linux server, they’re designed for a disposable Windows laptop that will be chucked in the bin after a year or two. I’ve learned this lesson the hard way over, and over, and over again with these little cheap SSDs.
- chipdart 2y ago
  > The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates. What do you mean by "endurance"? The only definition I found was the number of writes during the warranty time, and you're talking about a use case involving a very low volume of writes (i.e., save a file, keep it up for eternity). This means that warranty time (more specifically, any failure mode at all beyond excess writes) is the most likely failure mode. Not writes, just the SSD being old. Samsung advertises between 3 years and 10 years of warranty. The typical warranty of HDDs is between 3 and 5 years. Doesn't this suggest that buying you a decent SSD buys you at worst the same reliability? https://semiconductor.samsung.com/consumer-storage/support/warranty/ https://semiconductor.samsung.com/consumer-storage/support/w...

guenthert 2y ago

Dunno, man. It's an old Mac-mini, single disk (the SSD is explicitly called out) fairly obviously used for private stuff. You can wring it. In a production environment you cut you losses and restore from back-up. No-one wants to deal with half-baked recovery operations which might or might not yield a consistent state with (a little) less, but unknown, data loss.

bravetraveler 2y ago
It's not lost on me that this is a low stakes situation, they had backups and could afford being artful in their process. That's what makes this wheel-spinning so confounding.
- abofh 2y ago
  I don't think this was wheel-spinning so much as yarn-pulling, finding interesting tid-bits and seeing where they go. The most interesting things (to me) are when something doesn't quite line up with expectations, and then digging until I understand what the difference was - usually I misunderstood something (or failed to recall something, like in this case, disk/filesystem caches) - and I find myself more enlightend. I totally get doing this to learn because it was an interesting process. But yea, if this were for money? I'd just restore from backups and follow the SOP - because for money, the _why_ something failed is often unimportant, the remedy is the same -- reboot, replace, correlate, correct.
  - bravetraveler 2y ago
    Playing with Schrodinger's stuff. The storage industry/equipment lies so pervasively, even when it's working as intended More importantly, perhaps, is this is a "brain worm" being spread to others. Not to armchair too much, but there's a certain responsibility by publishing your writing. I'd hope for more dutiful research before publishing/risking data with anything even remotely nearing suggestion. Loud/clear warnings. We're all playing the Telephone Game whether we want to or not.

matheusmoreira 2y ago

It's not a soap box, it's good advice!! Most people use default options which means they mount with access times enabled where every read causes a write. File systems are more complex than they seem. They also do lots of automatic house keeping in the background. Even the relatively simple ext4 has journaling. Modern file systems will even move and rebalance data in the background. My laptop fell down on the floor recently. Hard disk started reporting errors immediately afterwards. I picked it up to assess the damage and immediately noticed that Linux remounted the drive read only. I turned it off, bought a 4 TB SSD, flashed a live Linux ISO and started reviewing the GNU ddrescue documentation. When recovering data, things should be as static as possible. I'd keep the disk platters from spinning if I could.