3 ms·

Am I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy appli

by mdoms 6y ago

Am I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy applications.

LeoPanthera 6y ago
"Endurance" microSD cards, normally sold for use in dashcams, are much more reliable in the Pi. However I switched to netbooting my Pis, no local storage on the Pi at all. This does require you to have a netboot server already, of course.
- ikaruswill 6y ago
  Indeed, though at the point of building this cluster, "Endurance" microSD cards were not in the market yet, hence I didn't go for it.
megous 6y ago
Brand SD cards are quite reliable. If you configure your SW to avoid sources of needless constant write churn (eg. like postgresql stats collector), and use good power supply, they will last years.
- ikaruswill 6y ago
  You're absolutely right. I've running these SD cards for about 2 years now. I use log2ram to stream logs to a ramdisk and then sync that at regular intervals to disk, to extend the life of the SD cards.
- redis_mlc 6y ago
  If you use Postgres as a single master on any storage device, you will also want to disable the slave writes, which seem to be enabled by default. If your device crashes with those enabled, Postgres may not start up again without manual intervention. Also, use the most current version since Postgres has had a brutal number of CVEs in the past few years which are in metasploit.
riscy 6y ago
It’s also tricky to find an SD card with good performance. I’ve been using a spare m.2 SSD drive inside of a USB enclosure with my Pi and the IO performance has been great.
ikaruswill 6y ago
Nope, the site has its persistent storage on an SSD shared via NFS across the cluster.
numpad0 6y ago
People who complain about SD card "reliability" is just writing to it way too often. Longevity of NAND flash is measured in TBW(total bytes written, in terabytes) but they have 4MB or so minimum size for write access, so people easily end up writing effectively gigabytes over weeks or months thinking it's just kilobytes averaged and use up all its life in just few months to years. You don't need "better quality SD cards", you stop issuing writes. As for why Raspbian isn't built that way so you don't have to be Linux storage experts just to run it correctly, I don't know.
- jng 6y ago
  That's very interesting and explains a number of things. 4MB writes seems pretty high though as a minimum write block size, I've googled but can't find details among the sea of pages trying to sell me SD cards, won't you by chance have a reference? Thanks!
  - numpad0 6y ago
    Google search for "microsd erase block size"[0] From a forum post in raspberrypi.org[1]: > My 16 GB SanDisk Extreme Pro 45 MB/s UHS 1 card reports an erase block size of 4 MB. > My 8 GB Transcend SDHC Class 6 150x card reports an erase block size of 4 MB. > My 2 GB Transcend SD 150x card reports an erase block size of 8 kB. . From another post from StackExchange.com[2]: > the data cannot be overwritten without being erased first and an erase block is the smallest unit that a NAND flash storage can erase. The erase block size is typically between 128 kB and 2 MB. . According to a Micron Technical Note from 2006[3]: > The NAND Flash device discussed in this technical note is based on a 2Gb asynchronous SLC device and its parameters (unless otherwise noted). > The NAND Flash array is grouped into a series of blocks, which are the smallest erasable entities in a NAND Flash device. > A NAND Flash block is 128KB. Erasing a block sets all bits to 1 (and all bytes to FFh). Programming is necessary to change erased bits from 1 to 0. The smallest entity that can be programmed is a byte. (Note: 2 gigabits = 256 MB) . In a more modern TLC 3D NAND chip from 2016[4]: > Block Size 27,888K bytes or 27.888MB --- [0]: https://www.google.com/search?q=microsd+erase+block+size https://www.google.com/search?q=microsd+erase+block+size [1]: https://www.raspberrypi.org/forums/viewtopic.php?t=11258 https://www.raspberrypi.org/forums/viewtopic.php?t=11258 [2]: https://raspberrypi.stackexchange.com/questions/32884/ https://raspberrypi.stackexchange.com/questions/32884/ [3]: https://www.micron.com/-/media/client/global/documents/products/technical-note/nand-flash/tn2961_wear_leveling_in_nand.pdf https://www.micron.com/-/media/client/global/documents/produ... [4]: https://www.micron.com/-/media/client/global/documents/products/product-flyer/3d_nand_flyer.pdf?la=en https://www.micron.com/-/media/client/global/documents/produ...

by mdoms 6y ago

Am I understanding that he is storing web data on the SD cards of the Pis? Is that normal? SD storage is notoriously unreliable, especially in write-heavy applications.

LeoPanthera 6y ago

"Endurance" microSD cards, normally sold for use in dashcams, are much more reliable in the Pi. However I switched to netbooting my Pis, no local storage on the Pi at all. This does require you to have a netboot server already, of course.

ikaruswill 6y ago
Indeed, though at the point of building this cluster, "Endurance" microSD cards were not in the market yet, hence I didn't go for it.

megous 6y ago

Brand SD cards are quite reliable. If you configure your SW to avoid sources of needless constant write churn (eg. like postgresql stats collector), and use good power supply, they will last years.

ikaruswill 6y ago
You're absolutely right. I've running these SD cards for about 2 years now. I use log2ram to stream logs to a ramdisk and then sync that at regular intervals to disk, to extend the life of the SD cards.
redis_mlc 6y ago
If you use Postgres as a single master on any storage device, you will also want to disable the slave writes, which seem to be enabled by default. If your device crashes with those enabled, Postgres may not start up again without manual intervention. Also, use the most current version since Postgres has had a brutal number of CVEs in the past few years which are in metasploit.

riscy 6y ago

It’s also tricky to find an SD card with good performance. I’ve been using a spare m.2 SSD drive inside of a USB enclosure with my Pi and the IO performance has been great.

ikaruswill 6y ago

Nope, the site has its persistent storage on an SSD shared via NFS across the cluster.

numpad0 6y ago

People who complain about SD card "reliability" is just writing to it way too often. Longevity of NAND flash is measured in TBW(total bytes written, in terabytes) but they have 4MB or so minimum size for write access, so people easily end up writing effectively gigabytes over weeks or months thinking it's just kilobytes averaged and use up all its life in just few months to years. You don't need "better quality SD cards", you stop issuing writes. As for why Raspbian isn't built that way so you don't have to be Linux storage experts just to run it correctly, I don't know.

jng 6y ago
That's very interesting and explains a number of things. 4MB writes seems pretty high though as a minimum write block size, I've googled but can't find details among the sea of pages trying to sell me SD cards, won't you by chance have a reference? Thanks!
- numpad0 6y ago
  Google search for "microsd erase block size"[0] From a forum post in raspberrypi.org[1]: > My 16 GB SanDisk Extreme Pro 45 MB/s UHS 1 card reports an erase block size of 4 MB. > My 8 GB Transcend SDHC Class 6 150x card reports an erase block size of 4 MB. > My 2 GB Transcend SD 150x card reports an erase block size of 8 kB. . From another post from StackExchange.com[2]: > the data cannot be overwritten without being erased first and an erase block is the smallest unit that a NAND flash storage can erase. The erase block size is typically between 128 kB and 2 MB. . According to a Micron Technical Note from 2006[3]: > The NAND Flash device discussed in this technical note is based on a 2Gb asynchronous SLC device and its parameters (unless otherwise noted). > The NAND Flash array is grouped into a series of blocks, which are the smallest erasable entities in a NAND Flash device. > A NAND Flash block is 128KB. Erasing a block sets all bits to 1 (and all bytes to FFh). Programming is necessary to change erased bits from 1 to 0. The smallest entity that can be programmed is a byte. (Note: 2 gigabits = 256 MB) . In a more modern TLC 3D NAND chip from 2016[4]: > Block Size 27,888K bytes or 27.888MB --- [0]: https://www.google.com/search?q=microsd+erase+block+size https://www.google.com/search?q=microsd+erase+block+size [1]: https://www.raspberrypi.org/forums/viewtopic.php?t=11258 https://www.raspberrypi.org/forums/viewtopic.php?t=11258 [2]: https://raspberrypi.stackexchange.com/questions/32884/ https://raspberrypi.stackexchange.com/questions/32884/ [3]: https://www.micron.com/-/media/client/global/documents/products/technical-note/nand-flash/tn2961_wear_leveling_in_nand.pdf https://www.micron.com/-/media/client/global/documents/produ... [4]: https://www.micron.com/-/media/client/global/documents/products/product-flyer/3d_nand_flyer.pdf?la=en https://www.micron.com/-/media/client/global/documents/produ...