7 ms·

The pi supposedly can get FS corruption, but I've never seen it because every time I install an image, I run a script that puts tmpfses everywhere and turns off

The pi supposedly can get FS corruption, but I've never seen it because every time I install an image, I run a script that puts tmpfses everywhere and turns off mostly useless logging. Those things just run for ever, very reliably, as long as you don't hammer the SD card. ZFS looks so cool! Unfortunately when I eventually get a NAS I doubt I'll want to pay for anything that can run it, so I suspect I'll just be doing RAID and ext4. I always stayed away from BTRFS, because every few months I'd see a "BTRFS destroyed my data" post, followed by an argument about if it was BTRFSes fault. I see them less now, perhaps it's time to revisit?

lost_tourist 3y ago
is that a script that is maintained and up to date and available on the web? Just curious. I've done this stuff manually and have a check list, but I never wrote a script because I didn't trust myself enough just barely knowing a little bit of bash myself. Another survival technique is to get a very large sd card like 64GB because wear leveling should increase the life. I have one rpi that's been going nonstop for 4 years now and the sdcard appears to be fine. Not sure why rpi guys don't have a "choose long term reliability over kitchen sink" option for setup (or even an image!)
- eternityforest 3y ago
  It's in the kaithem-kioskify up a little earlier in this thread. Eventually I'll probably move it to a standalone thing since there seems to be a whole lot of interest in the SD protection feature, but it's missing a few of hacks for programs I don't really use anymore, like the apache logfile thing, just because I got tired of maintaining stuff that didn't have much interest.
upon_drumhead 3y ago
Ubuntu ships with ZFS support, as well as TrueNAS. I personally run my nas with Ubuntu and ZFS and love it.
- chromakode 3y ago
  Ditto!
- justinclift 3y ago
  Apparently, recent releases of Ubuntu have dropped ZFS filesystem support after the person driving that effort left. :/
  - upon_drumhead 3y ago
    Reading up on it, recent releases of Ubuntu have dropped using ZFS for your boot and root volumes, which isn't ideal, however, they still support ZFS for any other volumes, which I'd venture is it's primary usage anyway, and they don't plan on removing support for anything other then zsys/zfs root/zfs boot. But thanks for bringing this to my attention. I had missed the changes in 23.04.
    - chromakode 3y ago
      There's been a recent series of PRs which appear to be adding ZFS root support to subiquity, the new Ubuntu installer: https://github.com/canonical/subiquity/pull/1689 https://github.com/canonical/subiquity/pull/1689
      - justinclift 3y ago
        Cool, hopefully that means it's being kept as an option after all. :)
toast0 3y ago
> ZFS looks so cool! Unfortunately when I eventually get a NAS I doubt I'll want to pay for anything that can run it, so I suspect I'll just be doing RAID and ext4. FreeBSD won't break your wallet.
- boomboomsubban 3y ago
  I assumed they are talking about the commonly spread myth that you need a gig of RAM for every terabyte of storage. I think that's recommended when doing deduplication, but for a simple NAS ZFS would use a comparable amount of memory as ext4 on RAID.
  - tambourine_man 3y ago
    Is that really the case? Can you point to some references backing that up? (Hehe, unintentional pun)
    - boomboomsubban 3y ago
      TrueNAS' general requirements are 8GB, and they spell out when you want more. Most of the situations you'd want more you'd also want more on RAID. https://www.truenas.com/docs/core/gettingstarted/corehardwareguide/ https://www.truenas.com/docs/core/gettingstarted/corehardwar...
    - rincebrain 3y ago
      ZFS will happily use a large amount of RAM for caching if you have it, but it'll run fine on a recent Pi (3 or 4, or not raspberry at all). It'll run fine but more sadly on older Pis running 32-bit kernels, since it does a looooooooot of 64-bit and wider operations, so you pay a nasty tax on that on 32-bit things. (Though the virtual address space limits might actually be sadder than the 64-bit operation penalty there, really...)
      - waynesonfire 3y ago
        I'm running ZFS on the smallest AWS instance, running FreeBSD, and it does what i want it to do.
TillE 3y ago
The only filesystem I've ever completely lost to corruption was btrfs, and that was about a year ago. btrfs-restore completely failed, so if I really needed that data I guess I'd have to do some manual surgery. I got to the point in the documentation where the only recourse was "idk go ask someone on IRC". Of course if you have good backups, you can use whatever and not really worry too much about it.
- cozzyd 3y ago
  I had some ram fail in my laptop and that killed my btrfs filesystem. Though btrfs restore was able to recover almost everything eventually (which is good because my last backup was a few weeks before since I had been traveling). Decided to go back to boring old ext4.
- mattpallissard 3y ago
  I've used it extensively for many years, both professionally and personally. Historically it's been something that users needed to be paying attention to the mailing list and wikis. For the most part though, sticking to standalone or mirrored disks is pretty rock solid and has been for a long time. Ditto for subvolumes, snapshots and, send/receive. My laptop has been snapshotted and sent from one piece of hardware to the next for many years now. That said, I'm with you on the backups. Anyone who uses btrfs and doesn't have a rock solid backups is a mad man.
- eternityforest 3y ago
  Well, I guess that resets my "Has it been long enough since a BTRFS horror story that I can trust it" counter.
- VancouverMan 3y ago
  I've had the same experience the two times I've tried btrfs, about two to four years ago, in my case with Linux VMs that would occasionally be abruptly terminated. In both cases, the corruption that couldn't be automatically repaired happened within the first few of these sudden terminations. Ext4 seems to handle that scenario better. I can't think of a single instance of filesystem corruption that fsck couldn't fix, and some of those VMs have probably been abruptly terminated at least a hundred times over the years.
snailmailman 3y ago
I had multiple pi systems “bricked” after a power outage. Presumably it was file system corruption? I never looked into the issues further. I just wiped the drives and reinstalled. These were vanilla raspbian installs at the time. It happened a few times when I was first trying out my pi. These were a mix of me cutting the power and actual power outages. I only ever had these issues with SD cards. I quickly switched to running my pi off a USB external SSD, and haven’t had any problems since then. Now when the power goes out, it boots back up properly and all my services start. All of this on ext3 I think? Planning to redo things for ZFS at some point, but haven’t gotten around to it yet.
- rebelpixel 3y ago
  This happened a lot to my Pi 3 back when I used the internal SD slot. I set it to boot off USB and used an old 8gb USB2 flash drive as the boot drive and never had the problem again.
- cwillu 3y ago
  Yeah, sd cards that you haven't put through extensive crash testing simply can't be trusted. I used to have a jar of sd cards that didn't survive testing.
  - ronjouch 3y ago
    The most noticeable raspberrypi SD card life lengthener for me has been to write logs to RAM (assuming you have a stable setup and don't count on them to survive a reboot!). Our $job dashboards used to nuke an SD card every couple weeks/months, but since the move to logs-in-RAM we've been running the same SDs for years. DIY via {fs,journalctl} config , or using https://github.com/azlux/log2ram https://github.com/azlux/log2ram Also, mount the SD with the `noatime` flag of course: https://wiki.archlinux.org/title/Ext4#Disabling_access_time_update https://wiki.archlinux.org/title/Ext4#Disabling_access_time_...
    - eternityforest 3y ago
      Noatime, disable swap, logs in RAM, /tmp in RAM, .xsession errors in RAM(With logrotate or it will fill up! So many random problems can make a problem there), chromium profile folder in RAM if you do kiosk work(Browser makers must hate flash!). From what I hear Home Assistant is still not the easiest if you want to run for years on a card, not sure if that's fixed now, but it's one of the big factors blocking me from moving to HA.
akeck 3y ago
Are you willing to share your script? I'd like to compare it to mine to see if I've missed anything. Thanks!
- eternityforest 3y ago
  The version I'm using now is all tangled up with an installer and setup script(When I'm doing interactive installations I tent to try to reuse the same setup for everything, but there's a big ASCII art banner for most of the relevant stuff for the SD card. Note that this doesn't have the Apache logfile hack so Apache probably won't run if you try this and don't add something to make it's fussy logfile. https://github.com/EternityForest/KaithemAutomation/blob/develop/kaithem-kioskify.sh https://github.com/EternityForest/KaithemAutomation/blob/dev...
nine_k 3y ago
What's the point of using ext4 on a NAS, and not XFS?
- eternityforest 3y ago
  XFS is not the default on most systems and I hardly ever hear about it in general, so I really never paid much attention to it in. Seems like people say it is more CPU heavy than EXT4, so unless it's way more reliable, would it really be the best choice on a pi/router/subGHz commercial NAS chip?
  - lost_tourist 3y ago
    We used to test some installers on XFS servers for Redhat stuff. XFS need far fewer "corrupt drive" fixes that than ext4fs on those servers. (probably 1/10th as much corruption) and we were constantly just doing straight power offs on them (no soft shutdowns). I think XFS doesn't get the respect it deserves if you don't need a "fancy" file system like zfs or btrfs.
    - eternityforest 3y ago
      Huh, that does sound pretty appealing for sure.
chillfox 3y ago
I am using USB disks with ZFS connected to a standard cheap office PC as my storage server. It still provides plenty of benefits over ext4.
8fingerlouie 3y ago
> I'll just be doing RAID and ext4 Why do you need/want RAID ? RAID is for availability, but is your data really that important that you cannot wait for a restore ? Most people would be much better off using that 1..N parity drives as versioned backups instead of running RAID. I've run NAS boxes for years, but these days i'm only using single drives. My setup these days consists of laptops that synchronizes data (encrypted) to the cloud, and a small ARM machine that synchronizes cloud contents locally, and makes a versioned backup to a single drive as well as a versioned cloud backup. As for cost, it's cheaper (for me) to store my data in the cloud, than the cost of electricity required to run my NAS.
- eternityforest 3y ago
  I was going to do RAID for disk failure tolerance, either with a consumer NAS box, or a disk enclosure that does hardware RAID, and something very low power like a Zero 2 W. Isn't RAID parity slightly more space efficient than versioned backups? Or is there a better way to do redundancy that doesn't involve just replicating entire files to multiple disks? Or some kind of automated manager that puts each individual file on N different disks out of M? I mostly do embedded so reliable data storage isn't generally something I deal with, we usually leave that to the cloud or to the user, and I'm not quite familiar with what's out there.
  - 8fingerlouie 3y ago
    >Isn't RAID parity slightly more space efficient than versioned backups? It depends on your storage array. The more drives, the more space efficient RAID becomes, but RAID is still only a single copy of your data. >Or is there a better way to do redundancy that doesn't involve just replicating entire files to multiple disks Most of the industry is using erasure coding these days (https://blog.min.io/erasure-coding/ https://blog.min.io/erasure-coding/) which allows for spreading your parity and data across multiple sites. Erasure coding usually runs a layer above the filesystem, as opposed to RAID which typically runs below the filesystem (Snapraid, Mergerfs and others excluded). My personal "backup vault" is a Raspberry Pi 4 with a single 4TB external drive attached. The RPi runs Minio, and all backups are done through the S3 interface or SFTP/SMB. It is not the fastest box in the world, but it backs up (incremental) ~2TB in 30 minutes, which is "fast enough". It consumes on average 4W, which means even with worst case electricity prices of €1/kWh (which we saw last winter), it costs less than €3/month. For comparison, my NAS consumed around 50W, and at €1/kWh, that would cost €37/month in electricity alone, and then you need to add the cost of the actual hardware itself. I switched off the NAS, and purchased ~10TB of cloud storage (main storage and backup storage at two different locations) for €20/month, and keep sensitive stuff encrypted with Cryptomator.
benlivengood 3y ago
> ZFS looks so cool! Unfortunately when I eventually get a NAS I doubt I'll want to pay for anything that can run it, so I suspect I'll just be doing RAID and ext4. I ran ZFS on a Raspberry Pi 4 with 8GB of RAM just fine (under debian arm64), and I've used ZFS on a machine with 4GB of RAM for receiving snapshots.
tomatocracy 3y ago
Zfs doesn't need hugely resourced hardware. For example, it can run on a single disk and still get you cheap snapshots/rollback, on-disk checksums, potential for easy replication via send/recv, highly efficient caching via the ARC, and transparent compression amongst other things. You can also start with a single disk and then easily add a mirror later on. In terms of CPU and RAM, most NASs now are perfectly capable of running it (but do also consider speccing out a small form factor PC with a case with lots of drive bays vs prebuilt NAS - you might be surprised at what you can get for a similar cost).
forinti 3y ago
I always disable rsyslog too. Tell me about tmpfs: where do you use it?

lost_tourist 3y ago

is that a script that is maintained and up to date and available on the web? Just curious. I've done this stuff manually and have a check list, but I never wrote a script because I didn't trust myself enough just barely knowing a little bit of bash myself. Another survival technique is to get a very large sd card like 64GB because wear leveling should increase the life. I have one rpi that's been going nonstop for 4 years now and the sdcard appears to be fine. Not sure why rpi guys don't have a "choose long term reliability over kitchen sink" option for setup (or even an image!)

eternityforest 3y ago
It's in the kaithem-kioskify up a little earlier in this thread. Eventually I'll probably move it to a standalone thing since there seems to be a whole lot of interest in the SD protection feature, but it's missing a few of hacks for programs I don't really use anymore, like the apache logfile thing, just because I got tired of maintaining stuff that didn't have much interest.

upon_drumhead 3y ago

Ubuntu ships with ZFS support, as well as TrueNAS. I personally run my nas with Ubuntu and ZFS and love it.

chromakode 3y ago
Ditto!
justinclift 3y ago
Apparently, recent releases of Ubuntu have dropped ZFS filesystem support after the person driving that effort left. :/
- upon_drumhead 3y ago
  Reading up on it, recent releases of Ubuntu have dropped using ZFS for your boot and root volumes, which isn't ideal, however, they still support ZFS for any other volumes, which I'd venture is it's primary usage anyway, and they don't plan on removing support for anything other then zsys/zfs root/zfs boot. But thanks for bringing this to my attention. I had missed the changes in 23.04.
  - chromakode 3y ago
    There's been a recent series of PRs which appear to be adding ZFS root support to subiquity, the new Ubuntu installer: https://github.com/canonical/subiquity/pull/1689 https://github.com/canonical/subiquity/pull/1689
    - justinclift 3y ago
      Cool, hopefully that means it's being kept as an option after all. :)

toast0 3y ago

> ZFS looks so cool! Unfortunately when I eventually get a NAS I doubt I'll want to pay for anything that can run it, so I suspect I'll just be doing RAID and ext4. FreeBSD won't break your wallet.

boomboomsubban 3y ago
I assumed they are talking about the commonly spread myth that you need a gig of RAM for every terabyte of storage. I think that's recommended when doing deduplication, but for a simple NAS ZFS would use a comparable amount of memory as ext4 on RAID.
- tambourine_man 3y ago
  Is that really the case? Can you point to some references backing that up? (Hehe, unintentional pun)
  - boomboomsubban 3y ago
    TrueNAS' general requirements are 8GB, and they spell out when you want more. Most of the situations you'd want more you'd also want more on RAID. https://www.truenas.com/docs/core/gettingstarted/corehardwareguide/ https://www.truenas.com/docs/core/gettingstarted/corehardwar...
  - rincebrain 3y ago
    ZFS will happily use a large amount of RAM for caching if you have it, but it'll run fine on a recent Pi (3 or 4, or not raspberry at all). It'll run fine but more sadly on older Pis running 32-bit kernels, since it does a looooooooot of 64-bit and wider operations, so you pay a nasty tax on that on 32-bit things. (Though the virtual address space limits might actually be sadder than the 64-bit operation penalty there, really...)
    - waynesonfire 3y ago
      I'm running ZFS on the smallest AWS instance, running FreeBSD, and it does what i want it to do.

TillE 3y ago

The only filesystem I've ever completely lost to corruption was btrfs, and that was about a year ago. btrfs-restore completely failed, so if I really needed that data I guess I'd have to do some manual surgery. I got to the point in the documentation where the only recourse was "idk go ask someone on IRC". Of course if you have good backups, you can use whatever and not really worry too much about it.

cozzyd 3y ago
I had some ram fail in my laptop and that killed my btrfs filesystem. Though btrfs restore was able to recover almost everything eventually (which is good because my last backup was a few weeks before since I had been traveling). Decided to go back to boring old ext4.
mattpallissard 3y ago
I've used it extensively for many years, both professionally and personally. Historically it's been something that users needed to be paying attention to the mailing list and wikis. For the most part though, sticking to standalone or mirrored disks is pretty rock solid and has been for a long time. Ditto for subvolumes, snapshots and, send/receive. My laptop has been snapshotted and sent from one piece of hardware to the next for many years now. That said, I'm with you on the backups. Anyone who uses btrfs and doesn't have a rock solid backups is a mad man.
eternityforest 3y ago
Well, I guess that resets my "Has it been long enough since a BTRFS horror story that I can trust it" counter.
VancouverMan 3y ago
I've had the same experience the two times I've tried btrfs, about two to four years ago, in my case with Linux VMs that would occasionally be abruptly terminated. In both cases, the corruption that couldn't be automatically repaired happened within the first few of these sudden terminations. Ext4 seems to handle that scenario better. I can't think of a single instance of filesystem corruption that fsck couldn't fix, and some of those VMs have probably been abruptly terminated at least a hundred times over the years.

snailmailman 3y ago

I had multiple pi systems “bricked” after a power outage. Presumably it was file system corruption? I never looked into the issues further. I just wiped the drives and reinstalled. These were vanilla raspbian installs at the time. It happened a few times when I was first trying out my pi. These were a mix of me cutting the power and actual power outages. I only ever had these issues with SD cards. I quickly switched to running my pi off a USB external SSD, and haven’t had any problems since then. Now when the power goes out, it boots back up properly and all my services start. All of this on ext3 I think? Planning to redo things for ZFS at some point, but haven’t gotten around to it yet.

rebelpixel 3y ago
This happened a lot to my Pi 3 back when I used the internal SD slot. I set it to boot off USB and used an old 8gb USB2 flash drive as the boot drive and never had the problem again.
cwillu 3y ago
Yeah, sd cards that you haven't put through extensive crash testing simply can't be trusted. I used to have a jar of sd cards that didn't survive testing.
- ronjouch 3y ago
  The most noticeable raspberrypi SD card life lengthener for me has been to write logs to RAM (assuming you have a stable setup and don't count on them to survive a reboot!). Our $job dashboards used to nuke an SD card every couple weeks/months, but since the move to logs-in-RAM we've been running the same SDs for years. DIY via {fs,journalctl} config , or using https://github.com/azlux/log2ram https://github.com/azlux/log2ram Also, mount the SD with the `noatime` flag of course: https://wiki.archlinux.org/title/Ext4#Disabling_access_time_update https://wiki.archlinux.org/title/Ext4#Disabling_access_time_...
  - eternityforest 3y ago
    Noatime, disable swap, logs in RAM, /tmp in RAM, .xsession errors in RAM(With logrotate or it will fill up! So many random problems can make a problem there), chromium profile folder in RAM if you do kiosk work(Browser makers must hate flash!). From what I hear Home Assistant is still not the easiest if you want to run for years on a card, not sure if that's fixed now, but it's one of the big factors blocking me from moving to HA.

akeck 3y ago

Are you willing to share your script? I'd like to compare it to mine to see if I've missed anything. Thanks!

eternityforest 3y ago
The version I'm using now is all tangled up with an installer and setup script(When I'm doing interactive installations I tent to try to reuse the same setup for everything, but there's a big ASCII art banner for most of the relevant stuff for the SD card. Note that this doesn't have the Apache logfile hack so Apache probably won't run if you try this and don't add something to make it's fussy logfile. https://github.com/EternityForest/KaithemAutomation/blob/develop/kaithem-kioskify.sh https://github.com/EternityForest/KaithemAutomation/blob/dev...

nine_k 3y ago

What's the point of using ext4 on a NAS, and not XFS?

eternityforest 3y ago
XFS is not the default on most systems and I hardly ever hear about it in general, so I really never paid much attention to it in. Seems like people say it is more CPU heavy than EXT4, so unless it's way more reliable, would it really be the best choice on a pi/router/subGHz commercial NAS chip?
- lost_tourist 3y ago
  We used to test some installers on XFS servers for Redhat stuff. XFS need far fewer "corrupt drive" fixes that than ext4fs on those servers. (probably 1/10th as much corruption) and we were constantly just doing straight power offs on them (no soft shutdowns). I think XFS doesn't get the respect it deserves if you don't need a "fancy" file system like zfs or btrfs.
  - eternityforest 3y ago
    Huh, that does sound pretty appealing for sure.

chillfox 3y ago

I am using USB disks with ZFS connected to a standard cheap office PC as my storage server. It still provides plenty of benefits over ext4.

8fingerlouie 3y ago

> I'll just be doing RAID and ext4 Why do you need/want RAID ? RAID is for availability, but is your data really that important that you cannot wait for a restore ? Most people would be much better off using that 1..N parity drives as versioned backups instead of running RAID. I've run NAS boxes for years, but these days i'm only using single drives. My setup these days consists of laptops that synchronizes data (encrypted) to the cloud, and a small ARM machine that synchronizes cloud contents locally, and makes a versioned backup to a single drive as well as a versioned cloud backup. As for cost, it's cheaper (for me) to store my data in the cloud, than the cost of electricity required to run my NAS.

eternityforest 3y ago
I was going to do RAID for disk failure tolerance, either with a consumer NAS box, or a disk enclosure that does hardware RAID, and something very low power like a Zero 2 W. Isn't RAID parity slightly more space efficient than versioned backups? Or is there a better way to do redundancy that doesn't involve just replicating entire files to multiple disks? Or some kind of automated manager that puts each individual file on N different disks out of M? I mostly do embedded so reliable data storage isn't generally something I deal with, we usually leave that to the cloud or to the user, and I'm not quite familiar with what's out there.
- 8fingerlouie 3y ago
  >Isn't RAID parity slightly more space efficient than versioned backups? It depends on your storage array. The more drives, the more space efficient RAID becomes, but RAID is still only a single copy of your data. >Or is there a better way to do redundancy that doesn't involve just replicating entire files to multiple disks Most of the industry is using erasure coding these days (https://blog.min.io/erasure-coding/ https://blog.min.io/erasure-coding/) which allows for spreading your parity and data across multiple sites. Erasure coding usually runs a layer above the filesystem, as opposed to RAID which typically runs below the filesystem (Snapraid, Mergerfs and others excluded). My personal "backup vault" is a Raspberry Pi 4 with a single 4TB external drive attached. The RPi runs Minio, and all backups are done through the S3 interface or SFTP/SMB. It is not the fastest box in the world, but it backs up (incremental) ~2TB in 30 minutes, which is "fast enough". It consumes on average 4W, which means even with worst case electricity prices of €1/kWh (which we saw last winter), it costs less than €3/month. For comparison, my NAS consumed around 50W, and at €1/kWh, that would cost €37/month in electricity alone, and then you need to add the cost of the actual hardware itself. I switched off the NAS, and purchased ~10TB of cloud storage (main storage and backup storage at two different locations) for €20/month, and keep sensitive stuff encrypted with Cryptomator.

benlivengood 3y ago

> ZFS looks so cool! Unfortunately when I eventually get a NAS I doubt I'll want to pay for anything that can run it, so I suspect I'll just be doing RAID and ext4. I ran ZFS on a Raspberry Pi 4 with 8GB of RAM just fine (under debian arm64), and I've used ZFS on a machine with 4GB of RAM for receiving snapshots.

tomatocracy 3y ago

Zfs doesn't need hugely resourced hardware. For example, it can run on a single disk and still get you cheap snapshots/rollback, on-disk checksums, potential for easy replication via send/recv, highly efficient caching via the ARC, and transparent compression amongst other things. You can also start with a single disk and then easily add a mirror later on. In terms of CPU and RAM, most NASs now are perfectly capable of running it (but do also consider speccing out a small form factor PC with a case with lots of drive bays vs prebuilt NAS - you might be surprised at what you can get for a similar cost).

forinti 3y ago

I always disable rsyslog too. Tell me about tmpfs: where do you use it?