5 ms·

Thats a well known issue with standard distributions, and there are fairly well known ways around it... Basically, don't do continuous writing to the card, tur

Thats a well known issue with standard distributions, and there are fairly well known ways around it... Basically, don't do continuous writing to the card, turn off atime updates,possibly even work entirely from the initrd if possible. Also, use proper industrial sd cards, these have proper wear-levelling and bad-block management unlike the consumer cards. In other words, treat it as an embedded system and tailor it to your use-case.

anfractuosity 8y ago
atime refers to timestamping of files with their access time right? That sounds interesting re. turning off atime updates, how do you do that out of interest?
- Fnoord 8y ago
  > how do you do that out of interest? Mount with -o noatime. The default (-o relatime) should however be good enough for most situations.
- rad_gruchalski 8y ago
  noatime mount option
sgt 8y ago
I've tried all these "ways around it", but SD card corruption still happens on Raspberry Pi's. We use them as dashboards around the office so having them running 24/7 really tests these devices. I've found that the only working recipe for a stable Raspberry Pi is a USB hard drive (prefer SSD but mechanical also fine), and a UPS. My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself.
- TickleSteve 8y ago
  I agree, which is why in a properly designed system you would run from initrd, its a single purpose system after all. No need for any sd card access once the image has been read into RAM. The SD card is read-only in this case. Also, I wouldnt trust consumer sd cards at all, they typically lack to necessary features to store data reliably anyway.
  - sgt 8y ago
    Would TinyCore be a good option for us? We just need to run RPi's and Chromium to display a web page.
    - elcritch 8y ago
      Not familiar with TinyCore myself, but I’d recommend trying Nerves Kiosk if a webpage kiosk is your primary goal [1]. Getting all of the FS settings right and a tuned system like daviduum talks about is tricky IMHO. Though it should be possible with Yocto Linux or similar as well I’d imagine, especially with ‘fwup’ or similar in the loop. Nerves uses ‘fwup’ [2] to create a R/O squashfs filesystem for the core OS image which is also highly compressed. 1: https://github.com/LeToteTeam/kiosk_system_rpi3/blob/master/README.md https://github.com/LeToteTeam/kiosk_system_rpi3/blob/master/... 2: https://github.com/fhunleth/fwup https://github.com/fhunleth/fwup
  - bonzini 8y ago
    Or just make your root file system read-only. Running off an initrd greatly limits your filesystem size.
- kingosticks 8y ago
  Can I ask what industrial sd cards did you try? I was going to try one of those out but I'd like to know what to avoid. Thanks.
  - xeeeeeeeeeeenu 8y ago
    If you want reliability, buy SLC SD cards. 'SLC' keyword is much more important than the brand. There are a few companies that make them, for example Panasonic or Swissbit.
    - nickpsecurity 8y ago
      Good point. To elaborate, SLC means Single-Level, Cell, NAND Flash that's both simpler and longer-lasting: https://en.wikipedia.org/wiki/Multi-level_cell#Single-level_cell https://en.wikipedia.org/wiki/Multi-level_cell#Single-level_... Example suppliers: https://www.micron.com/products/nand-flash/slc-nand https://www.micron.com/products/nand-flash/slc-nand https://www.cactus-tech.com/resources/blog/details/why-slc-sd-cards-are-critical-to-industrial-systems https://www.cactus-tech.com/resources/blog/details/why-slc-s... http://www.macronix.com/en-us/products/NAND-Flash/SLC-NAND-Flash/Pages/default.aspx?_escaped_fragment_=tabs=3V1Gb http://www.macronix.com/en-us/products/NAND-Flash/SLC-NAND-F... https://business.toshiba-memory.com/en-us/product/memory/slc-nand/slc.html https://business.toshiba-memory.com/en-us/product/memory/slc...
  - sgt 8y ago
    We primarily used class 10 consumer cards including Kingston, Sandisk, Samsung etc. We also tried ATP which is sold as an industrial SD card. More info: https://www.atpinc.com/products/industrial-sd-cards https://www.atpinc.com/products/industrial-sd-cards
    - jgalt212 8y ago
      are there ssd or winchester drives that can plug into the sd card slot? Of course, the drive would need an additional power supply. Or maybe this is just over-engineered, and maybe someone makes a pi that uses an SSD by default.
- avip 8y ago
  Not to downplay the idea that pi's fs in unreliable, but there's another player here which is the SD card. Many SD cards are way less reliable than the imaginary R/W spec given by manufacturers, and there's a crazy amount of grey matter and counterfeiting.
- gsich 8y ago
  You can network boot them too, which is way more convinient. RPi3 can do it natively, the former ones need an SD card, but since it's only for the bootloader, it'll survive much longer.
  - dividuum 8y ago
    Last time I read about this [1], the firmware had some issues that prevented the boot process from working 100% reliable. Do you have any experience with network booting the RPi? [1] https://github.com/raspberrypi/firmware/issues/764 https://github.com/raspberrypi/firmware/issues/764
    - gsich 8y ago
      Yes, but I only have 2 devices that network boot. One is with PoE, one without. Both work fine, but I don't have the need to reboot them every other day and the network is not a complex structure. Filesystem is with NFS.
- rsync 8y ago
  "My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself." Is that true even if you mount the filesystem(s) read-only after boot time ? As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago when we were using either disk-on-chip or just plain old CF cards pin-compatible with the IDE connector. I had 16MB CF cards on the IDE bus that lasted 15+ years[1] in production without a hiccup, but in 2018 SATA SSDs die randomly from time to time ... [1] Yes, they were all mounted read-only ...
  - ac29 8y ago
    > As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago Those old parts used SLC (single bit per cell) flash, which has orders of magnitude higher endurance than whats used now, MLC or TLC (2 and 3 bits per cell). QLC (4 bit per cell) is starting to hit the market now.
  - mrsteveman1 8y ago
    > Is that true even if you mount the filesystem(s) read-only after boot time ? Yes, read-only filesystems are not the solution to most of the Micro SD card problems on the Pi, because the filesystem and the OS are relying on the card to work properly, and the actual consumer market Micro SD cards people are using don't always do that when used in the Pi. To manage the flash, the (dirt cheap, barely fit for purpose) controller inside a consumer Micro SD card will generally do things like move data from one area of flash to another, even when the host isn't writing to the card. One of the reasons for that, is that reading from NAND is a potentially disruptive[1] event: > If reading continually from one cell, that cell will not fail but rather one of the surrounding cells on a subsequent read. To avoid the read disturb problem the flash controller will typically count the total number of reads to a block since the last erase. When the count exceeds a target limit, the affected block is copied over to a new block, erased, then released to the block pool. There are industrial Micro SD cards that can handle most of the problems that show up on the Pi though, and they aren't absurdly expensive. The $15 ATP AF4-GUD3A and $28 ATP AF8-GUD3A are some I use in my Pi projects for this reason, and I've never seen any of them become corrupt, unlike SanDisk and other consumer cards. [1] https://en.wikipedia.org/wiki/Flash_memory#Read_disturb https://en.wikipedia.org/wiki/Flash_memory#Read_disturb
- dividuum 8y ago
  > do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale Can't confirm. I've written about this before, but I'm mainly responsible for running the https://info-beamer.com https://info-beamer.com digital signage service. It's using a custom Linux distribution that always boots into an R/O system. All user content is on a separate partition that can always be restored if there's any error. The ext4 fs is tuned in a way that the kernel defers writes quite a while to minimize write access if possible. Since customers purchase their own SD cards, we have limited control of what ends up in a device. I've seen exactly one SD card related problem thus far. There are devices that have been running >3 years 24/7, so it's doable. But you can't just use Raspbian. You have to properly design the system.
  - sgt 8y ago
    Thanks - that does give me hope.
- limaoscarjuliet 8y ago
  I use raspberry pi as a display around a mid sized company. Out of 12 setups, I have to resurrect 2-3 every year due to ssd corruption.
  - sgt 8y ago
    Are these using UPS or just directly on power? At our office, power goes off about once or twice a month when someone is working on the electrical system. We've had lots of building work done, and it's definitely not been good on the Pi's - that being until we added UPS's.

anfractuosity 8y ago

atime refers to timestamping of files with their access time right? That sounds interesting re. turning off atime updates, how do you do that out of interest?

Fnoord 8y ago
> how do you do that out of interest? Mount with -o noatime. The default (-o relatime) should however be good enough for most situations.
rad_gruchalski 8y ago
noatime mount option

sgt 8y ago

I've tried all these "ways around it", but SD card corruption still happens on Raspberry Pi's. We use them as dashboards around the office so having them running 24/7 really tests these devices. I've found that the only working recipe for a stable Raspberry Pi is a USB hard drive (prefer SSD but mechanical also fine), and a UPS. My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself.

TickleSteve 8y ago
I agree, which is why in a properly designed system you would run from initrd, its a single purpose system after all. No need for any sd card access once the image has been read into RAM. The SD card is read-only in this case. Also, I wouldnt trust consumer sd cards at all, they typically lack to necessary features to store data reliably anyway.
- sgt 8y ago
  Would TinyCore be a good option for us? We just need to run RPi's and Chromium to display a web page.
  - elcritch 8y ago
    Not familiar with TinyCore myself, but I’d recommend trying Nerves Kiosk if a webpage kiosk is your primary goal [1]. Getting all of the FS settings right and a tuned system like daviduum talks about is tricky IMHO. Though it should be possible with Yocto Linux or similar as well I’d imagine, especially with ‘fwup’ or similar in the loop. Nerves uses ‘fwup’ [2] to create a R/O squashfs filesystem for the core OS image which is also highly compressed. 1: https://github.com/LeToteTeam/kiosk_system_rpi3/blob/master/README.md https://github.com/LeToteTeam/kiosk_system_rpi3/blob/master/... 2: https://github.com/fhunleth/fwup https://github.com/fhunleth/fwup
- bonzini 8y ago
  Or just make your root file system read-only. Running off an initrd greatly limits your filesystem size.
kingosticks 8y ago
Can I ask what industrial sd cards did you try? I was going to try one of those out but I'd like to know what to avoid. Thanks.
- xeeeeeeeeeeenu 8y ago
  If you want reliability, buy SLC SD cards. 'SLC' keyword is much more important than the brand. There are a few companies that make them, for example Panasonic or Swissbit.
  - nickpsecurity 8y ago
    Good point. To elaborate, SLC means Single-Level, Cell, NAND Flash that's both simpler and longer-lasting: https://en.wikipedia.org/wiki/Multi-level_cell#Single-level_cell https://en.wikipedia.org/wiki/Multi-level_cell#Single-level_... Example suppliers: https://www.micron.com/products/nand-flash/slc-nand https://www.micron.com/products/nand-flash/slc-nand https://www.cactus-tech.com/resources/blog/details/why-slc-sd-cards-are-critical-to-industrial-systems https://www.cactus-tech.com/resources/blog/details/why-slc-s... http://www.macronix.com/en-us/products/NAND-Flash/SLC-NAND-Flash/Pages/default.aspx?_escaped_fragment_=tabs=3V1Gb http://www.macronix.com/en-us/products/NAND-Flash/SLC-NAND-F... https://business.toshiba-memory.com/en-us/product/memory/slc-nand/slc.html https://business.toshiba-memory.com/en-us/product/memory/slc...
- sgt 8y ago
  We primarily used class 10 consumer cards including Kingston, Sandisk, Samsung etc. We also tried ATP which is sold as an industrial SD card. More info: https://www.atpinc.com/products/industrial-sd-cards https://www.atpinc.com/products/industrial-sd-cards
  - jgalt212 8y ago
    are there ssd or winchester drives that can plug into the sd card slot? Of course, the drive would need an additional power supply. Or maybe this is just over-engineered, and maybe someone makes a pi that uses an SSD by default.
avip 8y ago
Not to downplay the idea that pi's fs in unreliable, but there's another player here which is the SD card. Many SD cards are way less reliable than the imaginary R/W spec given by manufacturers, and there's a crazy amount of grey matter and counterfeiting.
gsich 8y ago
You can network boot them too, which is way more convinient. RPi3 can do it natively, the former ones need an SD card, but since it's only for the bootloader, it'll survive much longer.
- dividuum 8y ago
  Last time I read about this [1], the firmware had some issues that prevented the boot process from working 100% reliable. Do you have any experience with network booting the RPi? [1] https://github.com/raspberrypi/firmware/issues/764 https://github.com/raspberrypi/firmware/issues/764
  - gsich 8y ago
    Yes, but I only have 2 devices that network boot. One is with PoE, one without. Both work fine, but I don't have the need to reboot them every other day and the network is not a complex structure. Filesystem is with NFS.
rsync 8y ago
"My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself." Is that true even if you mount the filesystem(s) read-only after boot time ? As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago when we were using either disk-on-chip or just plain old CF cards pin-compatible with the IDE connector. I had 16MB CF cards on the IDE bus that lasted 15+ years[1] in production without a hiccup, but in 2018 SATA SSDs die randomly from time to time ... [1] Yes, they were all mounted read-only ...
- ac29 8y ago
  > As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago Those old parts used SLC (single bit per cell) flash, which has orders of magnitude higher endurance than whats used now, MLC or TLC (2 and 3 bits per cell). QLC (4 bit per cell) is starting to hit the market now.
- mrsteveman1 8y ago
  > Is that true even if you mount the filesystem(s) read-only after boot time ? Yes, read-only filesystems are not the solution to most of the Micro SD card problems on the Pi, because the filesystem and the OS are relying on the card to work properly, and the actual consumer market Micro SD cards people are using don't always do that when used in the Pi. To manage the flash, the (dirt cheap, barely fit for purpose) controller inside a consumer Micro SD card will generally do things like move data from one area of flash to another, even when the host isn't writing to the card. One of the reasons for that, is that reading from NAND is a potentially disruptive[1] event: > If reading continually from one cell, that cell will not fail but rather one of the surrounding cells on a subsequent read. To avoid the read disturb problem the flash controller will typically count the total number of reads to a block since the last erase. When the count exceeds a target limit, the affected block is copied over to a new block, erased, then released to the block pool. There are industrial Micro SD cards that can handle most of the problems that show up on the Pi though, and they aren't absurdly expensive. The $15 ATP AF4-GUD3A and $28 ATP AF8-GUD3A are some I use in my Pi projects for this reason, and I've never seen any of them become corrupt, unlike SanDisk and other consumer cards. [1] https://en.wikipedia.org/wiki/Flash_memory#Read_disturb https://en.wikipedia.org/wiki/Flash_memory#Read_disturb
dividuum 8y ago
> do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale Can't confirm. I've written about this before, but I'm mainly responsible for running the https://info-beamer.com https://info-beamer.com digital signage service. It's using a custom Linux distribution that always boots into an R/O system. All user content is on a separate partition that can always be restored if there's any error. The ext4 fs is tuned in a way that the kernel defers writes quite a while to minimize write access if possible. Since customers purchase their own SD cards, we have limited control of what ends up in a device. I've seen exactly one SD card related problem thus far. There are devices that have been running >3 years 24/7, so it's doable. But you can't just use Raspbian. You have to properly design the system.
- sgt 8y ago
  Thanks - that does give me hope.
limaoscarjuliet 8y ago
I use raspberry pi as a display around a mid sized company. Out of 12 setups, I have to resurrect 2-3 every year due to ssd corruption.
- sgt 8y ago
  Are these using UPS or just directly on power? At our office, power goes off about once or twice a month when someone is working on the electrical system. We've had lots of building work done, and it's definitely not been good on the Pi's - that being until we added UPS's.