4 ms·

This would have been perfect for my last startup. Keeping those damn Pi's running when they were deployed across the country was a nightmare. The SD cards had

This would have been perfect for my last startup. Keeping those damn Pi's running when they were deployed across the country was a nightmare. The SD cards had around a 20% failure rate per year. Eventually I just started shipping each system with a backup SD card taped to the inside of the case.

Seanny123 9y ago
What did your startup do that required that many Raspberry Pis?
- learc83 9y ago
  The Pis were used in a 24 gym and tanning salon system. They ran a touchscreen for customer interaction, connected to a custom board I built that attached to the proprietary network tanning control systems use, and handled RFID access control. It worked well, but tanning regulatory changes killed it.
fest 9y ago
Have you tried SLC, pSLC or at least, industrial grade MLC cards? They are more expensive, but might be worth it in such applications. See this whitepaper on why: https://www.cactus-tech.com/files/cactus-tech.com/documents/whitepapers/An%20Overview%20of%20Pseudo-SLC%20NAND.pdf https://www.cactus-tech.com/files/cactus-tech.com/documents/...
posguy 9y ago
Were you using decent microSDs? Was each mSD getting QC'ed? We benchmark every single mSD before deploying an SBC with it, and have switched to only using Evo+ 32GB cards as they pass our QC 3/5 of the time, which is much better than the other available brands. Also, mSD corruption means your using a mSD card with a bad controller. The mSD was bound to corrupt and was a low caliber card to start with, unless you are doing a bunch of writes a mSD should last 2 years minimum, if not longer. Another tact to take would be to switch to an OrangePi PC+ (8GB eMMC included, $22) or similar, that way you know your getting good quality nand with a decent controller out of the box.
- Normal_gaussian 9y ago
  Do you mind sharing your quality control method for this? We are hitting card issues on relatively low device numbers.
  - posguy 9y ago
    Just saw this, TL;DR we transfer a large file onto and off of the mSD card. If its above 10MB/s for both (which is the base rating on a class 10 card), then it is a fine mSD card. I've also noticed that on multi-gig files speeds tend to drop to around 6MB/s, not the end of the world for us, but also not a sign of a bad mSD either.
- deleted 9y ago
  [deleted]
- learc83 9y ago
  From my experience the failures were almost always caused by a power loss. The obvious solution would have been to include a battery backup in the case.
  - posguy 9y ago
    Odd, I've only ever seen this be an issue on the Raspberry Pi platform, outside of that power loss doesn't regularly cause data loss on most Androids or Allwinner boards so long as the mSD isn't of low caliber.
ghvknvgvh 9y ago
Mount the SD card read-only and it'll be fine forever. Speaking from experience.
landswipe 9y ago
Removable storage is problematic when it comes to reliability. Simply powering the device off while it is writing will cause issues, worse is when you have vibration momentarily overcoming the sprung contacts to the card. Also mentioned by others are SLC industrial grade cards which have more stable flash memory technology, better controllers, wider operating temperatures and not to mention are much much faster. It all comes down to cost, but trying to build a reliable system with removable cards is not a good idea. The Pi design team should consider adding some reasonable quality onboard eMMC memory in the future. Regarding cards, I can vouch for http://www.atpinc.com http://www.atpinc.com which we use where I work, expensive but very reliable.
- strainer 9y ago
  "consider adding some reasonable quality onboard eMMC memory in the future." The Pi "compute modules" include onboard eMMC and so do these which are based on them.
behohippy 9y ago
I have a cluster at home, and that's about what I'm seeing. Good wall warts and good power cables reduces the chances of it happening, but doesn't bring it to zero.
widea 9y ago
At work the SD cards lasted for about 1,5 year and then gave up. I now disabled Ext4 journaling and am considering using the F2FS filesystem which is designed for flash storage.

Seanny123 9y ago

What did your startup do that required that many Raspberry Pis?

learc83 9y ago
The Pis were used in a 24 gym and tanning salon system. They ran a touchscreen for customer interaction, connected to a custom board I built that attached to the proprietary network tanning control systems use, and handled RFID access control. It worked well, but tanning regulatory changes killed it.

fest 9y ago

Have you tried SLC, pSLC or at least, industrial grade MLC cards? They are more expensive, but might be worth it in such applications. See this whitepaper on why: https://www.cactus-tech.com/files/cactus-tech.com/documents/whitepapers/An%20Overview%20of%20Pseudo-SLC%20NAND.pdf https://www.cactus-tech.com/files/cactus-tech.com/documents/...

posguy 9y ago

Were you using decent microSDs? Was each mSD getting QC'ed? We benchmark every single mSD before deploying an SBC with it, and have switched to only using Evo+ 32GB cards as they pass our QC 3/5 of the time, which is much better than the other available brands. Also, mSD corruption means your using a mSD card with a bad controller. The mSD was bound to corrupt and was a low caliber card to start with, unless you are doing a bunch of writes a mSD should last 2 years minimum, if not longer. Another tact to take would be to switch to an OrangePi PC+ (8GB eMMC included, $22) or similar, that way you know your getting good quality nand with a decent controller out of the box.

Normal_gaussian 9y ago
Do you mind sharing your quality control method for this? We are hitting card issues on relatively low device numbers.
- posguy 9y ago
  Just saw this, TL;DR we transfer a large file onto and off of the mSD card. If its above 10MB/s for both (which is the base rating on a class 10 card), then it is a fine mSD card. I've also noticed that on multi-gig files speeds tend to drop to around 6MB/s, not the end of the world for us, but also not a sign of a bad mSD either.
deleted 9y ago
[deleted]
learc83 9y ago
From my experience the failures were almost always caused by a power loss. The obvious solution would have been to include a battery backup in the case.
- posguy 9y ago
  Odd, I've only ever seen this be an issue on the Raspberry Pi platform, outside of that power loss doesn't regularly cause data loss on most Androids or Allwinner boards so long as the mSD isn't of low caliber.

ghvknvgvh 9y ago

Mount the SD card read-only and it'll be fine forever. Speaking from experience.

landswipe 9y ago

Removable storage is problematic when it comes to reliability. Simply powering the device off while it is writing will cause issues, worse is when you have vibration momentarily overcoming the sprung contacts to the card. Also mentioned by others are SLC industrial grade cards which have more stable flash memory technology, better controllers, wider operating temperatures and not to mention are much much faster. It all comes down to cost, but trying to build a reliable system with removable cards is not a good idea. The Pi design team should consider adding some reasonable quality onboard eMMC memory in the future. Regarding cards, I can vouch for http://www.atpinc.com http://www.atpinc.com which we use where I work, expensive but very reliable.

strainer 9y ago
"consider adding some reasonable quality onboard eMMC memory in the future." The Pi "compute modules" include onboard eMMC and so do these which are based on them.

behohippy 9y ago

I have a cluster at home, and that's about what I'm seeing. Good wall warts and good power cables reduces the chances of it happening, but doesn't bring it to zero.

widea 9y ago

At work the SD cards lasted for about 1,5 year and then gave up. I now disabled Ext4 journaling and am considering using the F2FS filesystem which is designed for flash storage.