4 ms·

That one bit me really hard today. We're using hundreds of raspberry's as digital signage devices and before deploying them I checked a few of them and found o

That one bit me really hard today. We're using hundreds of raspberry's as digital signage devices and before deploying them I checked a few of them and found out that sometimes the source files of the app were empty. After a lot of headscratching and googling we boiled it down to a write delay to the SD card. If you do echo "test" > test.txt and power off the device like 5 sec. after, and power it on again, there was a file called test.txt with 0 bytes. I'm still baffled how something like this can be a common problem especially since raspberry pi usage is so widespread

gubby 8y ago
When you say you powered it off, was that a safe shutdown or did you just pull the power? Most file system drivers delay writes, I suspect some variant of your test (if it was unsafe shutdown) could be reproduced on almost any system.
- stdclass 8y ago
  It was an unsafe shutdown, meaning I just pulled the plug. But it was at least several seconds after saving it. I've never seen behaviour like this before.
  - 51lver 8y ago
    Most sd cards are formatted fat32, and do not have a virtual filesystem cache and write scheduler buffering work in ram for them. Also raspis do NOT like big fast SD cards with tight tolerances. I've had the best luck with old class 4 cards. Class 10 cards don't last nearly as long with regular power cycling.
  - 0x0 8y ago
    Did you ever run linux with ext2 (not ext3 or ext4) on a desktop computer in the early days? It used to fail spectacularly on unclean shutdowns. IIRC linux used to delay/buffer writes up to a minute or so. (Maybe it still does?)
  - cellularmitosis 8y ago
    You can force a partition to be mounted "synchronously", where writes aren't buffered in RAM and then flushed later -- every write blocks until it is written to disk. The options are the third column in your /etc/fstab file. For example, if you had "errors=remount-ro", you'd change it to "errors=remount-ro,sync". (Or is the Raspi problem actually in hardware?)
- greglindahl 8y ago
  Yes, in fact you'll find lots of source code out there that will write test.txt.tmp and then close and rename it to test.txt, exactly because this is an issue in all filesystems. From the sounds of it the time window during which RPi is vulnerable is longer than usual. A careful choice of filesystem journaling/integrity flags would probably help.
  - dividuum 8y ago
    The fact that there's a 0-byte file instead of a completely broken file system is already the result of a journal. You can reliably observe this with ext4.
pmarin 8y ago
On unix systems you can always do a sync(2) to commit the buffers to disk. https://linux.die.net/man/2/sync https://linux.die.net/man/2/sync
- cat199 8y ago
  or sync 3 times and then pull the plug https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSync https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSyn...
giobox 8y ago
I would never SD card boot a Pi in a production environment unless your software can handle a read only file system, the card corruption issues are so widely documented at this point. Thankfully USB boot volumes are supported now.
deleted 8y ago
[deleted]
johnchristopher 8y ago
In my opinion it seems the Pi was too good at fast and cheap prototyping and people started shipping products based on it, overlooking the fact that the hobbyist and unreliable nature of the board wouldn't necessarily scale as hoped. It always looked to me like the Pi was supposed to be used some steps before hardware integration and shipping. I am glad products are built on it but I was a bit worried when I had to evaluate a Pi based solution for digital signaling last year. Maintenance was a key factor in the proposals.
sevensor 8y ago
We used to type sync; sync; halt. Funny how these things come back around.
dividuum 8y ago
I operate a digital signage service (https://info-beamer.com https://info-beamer.com). It's using a custom built Linux OS that is highly optimized to deal with file system corruption and unexpected reboots: * The OS is is always read-only. While this doesn't prevent all possible corruption scenarios, it's pretty close and you can be quite sure the system always boots. * System updates are implemented as A/B. So even during updates, the OS the system booted from isn't touched. * All content data is on a third partition and together with the OS files is constantly checked for indications of corruption. * If the OS detects any bit error on the OS files, it is reinstalled (using the A/B mechanism above) * If there's a bit error on any of the content files, they are deleted and downloaded again * If the content filesystem is broken beyond repair the system reformats everything and restores everything automatically by redownloading * Various watchdogs and checks constantly monitor system health and notify you on the dashboard. The service has multi-million hours of service by now and the only case of filesystem corruption was due to a really crappy SD card. When done properly, the Pi can be reliable. That's also the reason why there is no way to properly shut down the system. Just unplugging power shouldn't be an unexpected circumstance but something you optimize for. Starting with a normal Raspbian and expecting that to work is a bad idea.
- donmcronald 8y ago
  Is your updater self rolled or based on something existing like mender.io?
  - dividuum 8y ago
    Self rolled. Built in Python.
- beamatronic 8y ago
  I have a use-case in mind where the power might disappear suddenly and often. Is there a different SBC board and OS which is optimal for this use-case?
  - jburgess777 8y ago

gubby 8y ago

When you say you powered it off, was that a safe shutdown or did you just pull the power? Most file system drivers delay writes, I suspect some variant of your test (if it was unsafe shutdown) could be reproduced on almost any system.

stdclass 8y ago
It was an unsafe shutdown, meaning I just pulled the plug. But it was at least several seconds after saving it. I've never seen behaviour like this before.
- 51lver 8y ago
  Most sd cards are formatted fat32, and do not have a virtual filesystem cache and write scheduler buffering work in ram for them. Also raspis do NOT like big fast SD cards with tight tolerances. I've had the best luck with old class 4 cards. Class 10 cards don't last nearly as long with regular power cycling.
- 0x0 8y ago
  Did you ever run linux with ext2 (not ext3 or ext4) on a desktop computer in the early days? It used to fail spectacularly on unclean shutdowns. IIRC linux used to delay/buffer writes up to a minute or so. (Maybe it still does?)
- cellularmitosis 8y ago
  You can force a partition to be mounted "synchronously", where writes aren't buffered in RAM and then flushed later -- every write blocks until it is written to disk. The options are the third column in your /etc/fstab file. For example, if you had "errors=remount-ro", you'd change it to "errors=remount-ro,sync". (Or is the Raspi problem actually in hardware?)
greglindahl 8y ago
Yes, in fact you'll find lots of source code out there that will write test.txt.tmp and then close and rename it to test.txt, exactly because this is an issue in all filesystems. From the sounds of it the time window during which RPi is vulnerable is longer than usual. A careful choice of filesystem journaling/integrity flags would probably help.
- dividuum 8y ago
  The fact that there's a 0-byte file instead of a completely broken file system is already the result of a journal. You can reliably observe this with ext4.

pmarin 8y ago

On unix systems you can always do a sync(2) to commit the buffers to disk. https://linux.die.net/man/2/sync https://linux.die.net/man/2/sync

cat199 8y ago
or sync 3 times and then pull the plug https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSync https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSyn...

giobox 8y ago

I would never SD card boot a Pi in a production environment unless your software can handle a read only file system, the card corruption issues are so widely documented at this point. Thankfully USB boot volumes are supported now.

deleted 8y ago

[deleted]

johnchristopher 8y ago

In my opinion it seems the Pi was too good at fast and cheap prototyping and people started shipping products based on it, overlooking the fact that the hobbyist and unreliable nature of the board wouldn't necessarily scale as hoped. It always looked to me like the Pi was supposed to be used some steps before hardware integration and shipping. I am glad products are built on it but I was a bit worried when I had to evaluate a Pi based solution for digital signaling last year. Maintenance was a key factor in the proposals.

sevensor 8y ago

We used to type sync; sync; halt. Funny how these things come back around.

dividuum 8y ago

I operate a digital signage service (https://info-beamer.com https://info-beamer.com). It's using a custom built Linux OS that is highly optimized to deal with file system corruption and unexpected reboots: * The OS is is always read-only. While this doesn't prevent all possible corruption scenarios, it's pretty close and you can be quite sure the system always boots. * System updates are implemented as A/B. So even during updates, the OS the system booted from isn't touched. * All content data is on a third partition and together with the OS files is constantly checked for indications of corruption. * If the OS detects any bit error on the OS files, it is reinstalled (using the A/B mechanism above) * If there's a bit error on any of the content files, they are deleted and downloaded again * If the content filesystem is broken beyond repair the system reformats everything and restores everything automatically by redownloading * Various watchdogs and checks constantly monitor system health and notify you on the dashboard. The service has multi-million hours of service by now and the only case of filesystem corruption was due to a really crappy SD card. When done properly, the Pi can be reliable. That's also the reason why there is no way to properly shut down the system. Just unplugging power shouldn't be an unexpected circumstance but something you optimize for. Starting with a normal Raspbian and expecting that to work is a bad idea.

donmcronald 8y ago
Is your updater self rolled or based on something existing like mender.io?
- dividuum 8y ago
  Self rolled. Built in Python.
beamatronic 8y ago
I have a use-case in mind where the power might disappear suddenly and often. Is there a different SBC board and OS which is optimal for this use-case?
- jburgess777 8y ago