5 ms·

Running a Raspberry Pi with a read-only root filesystem

218 points by ingve 3y ago

logicziller 3y ago
We use pi-builder that can produce reproducible images. Read-only is one of the default options.
ashwoods 3y ago
I now prefer to use overlayfs with a tmpfs overlay. Much simpler to configure and update.
- crashbunny 3y ago
  years ago I did the same, I vaguely remember following a blog post to do it. I put the scripts on github and some people found it useful. There's a deb package now that does it called overlayroot.
deleted 3y ago
[deleted]
Wowfunhappy 3y ago
> This means you can run the Pi with a read-only root filesystem, which will dramatically increase the SD card’s lifetime. I can't find an authoritative source, but some quick Googling brings up websites like Reddit and Stackexchange saying that SD cards have ~100,000 lifetime write cycles. I feel like it would be hard to exhaust that. Is this a real problem people experience?
- Proven 3y ago
  [dead]
- CliffStoll 3y ago
  My website is on a pi-4 and hasn't had a hiccup in 4 years. So far, the SD-card has worked great.
  - freedomben 3y ago
    How high of traffic is your website? And are you writing log files (like nginx or apache, plus application if applicable)?
- adobrawy 3y ago
  Yea, it's a real issue. Search for "raspberry pi dead SD card" to found report of ppl which have failing SD card every month or more often.
  - rcarmo 3y ago
    Most of those tend to be power issues.
- rcxdude 3y ago
  Yes, it's not particularly hard to wear out an SD card (especially a cheap, high-densite one), with just the wear of writing logs. It does vary a lot depending on what exactly you are going, but it's a fairly commonly reported failure among raspberry pi users.
- Prcmaker 3y ago
  I haven't had an issue with SD cards where Pis had reliable power. Where I used to live though we had frequent power cuts, of only a couple of seconds, a couple of times a day. SD card corruption was a major pain. Even in areas with more reliable power, for a 24/7 on device, occasional power outages have lead me to move away from Pis. Battery backups are, of course, an option, but increase the cost for a reliable and simple system.
- bobmcnamara 3y ago
  It was a plague for years where power failure during a write would cause an SD card to go read only or worse.
- jorvi 3y ago
  It is, but mostly due to logging to disk instead of RAM. That’s what setting a read-only root file system prevents in a roundabout way, so people have started to misidentify it as the solution.
- NegativeLatency 3y ago
  A lot of people also run them with a power supply that will undervolt, switched to ones with more amps and haven't had issues since
- monocasa 3y ago
  Yes, as someone who shipped embedded systems running off of SD cards, it's a real problem. It's not the flash endurance you hit typically, but bugs in the card firmware that cause the FTL to completely lose it's mind. Yes, even with supposedly quality brands. About the only way I found to deal with the problem is to use "industrial" SD cards that are a little slower and pretty expensive, but the firmware takes less liberties in the name of specious benchmarks.
  - stefan_ 3y ago
    It’s interesting. In my embedded experience, SD cards are absolute trash, eMMC is already much better, raw NAND is somewhere in between and then actual SSDs are much better than all of these again. Yet supposedly these are all the same basic building blocks.
jasongill 3y ago
You can skip the ntp/systemd-timesyncd/fake-hwclock stuff if you add a RTC/coin battery to the device. The Raspberry Pi 5 has built-in RTC with support for an official battery which costs about five bucks and works well for this. Older devices have aftermarket RTC's with battery backups as well.
- petee 3y ago
  Theres also a kernel option to sync PPS on a gpio from a gps clock
idatum 3y ago
These Pi* devices have served well running their Debian version for years. I had a Pi3b run ADS-B (generally RTL-SDR) for many years. Dang though, all with (seemingly) random SD storage failures. Article details seems like it can help. There are more options now though. Aarch64 ones that use eMMC. I replaced the Pi3b with a Pine64 Rock64 running FreeBSD 14 booting from eMMC. Simple and just works with all RTL-SDR needs covered. It's great to have options, both hw and os.
jauntywundrkind 3y ago
A long time ago I modified debian-live to use btrfs-send to copy the usb image into some kind of ramfs. So you could hypothetically snapshot & send back your current snapshot to the USB as you want. Having a kick ass fs at your back is great. Go for it! One of my persistently greatest annoyances about running stuff on pis has been that journald is so monolithic. There's no per-app controls. I'd love to log most system functions persistently with decent sized history (since they are very low volume), but have the main app or another app do something else. Struggling to find them ATM, but there are a number of issues opened around this way back, and all closed, saying to run multiple systemd-journal instances then configure apps to direct traffic to each, but that sure sucks to setup & configure, & is something I'd hope journald would help me with out of the box.
blackfawn 3y ago
I like using Alpine for read-only operation on the Pi right out of the box: https://wiki.alpinelinux.org/wiki/Raspberry_Pi https://wiki.alpinelinux.org/wiki/Raspberry_Pi
- GianFabien 3y ago
  Alpine has been great for me too. Use an USB Flash drive mounted as a R/W filesystem for logs, etc. fsck can usually recover those f/s when crashed. The root f/s remains intact.
- seemaze 3y ago
  This is the way. Run from ram and an excellent selection of packages to boot! I have a number of pi zero and pi 2 zero running alpine.
LeoPanthera 3y ago
It's worth noting that you don't have to do this manually. Enabling "overlay filesystem" in raspi-config will do this for you, so that all changes to the filesystem are lost after a reboot.
- HankB99 3y ago
  I believe the overlayfs configured using raspi-config is a little different in that it presents the appearance of a normal filesystem but all changes are gone following reboot. Before reboot the changes are cached in RAM which can have consequences for low RAM systems. I use the overlayfs provided by raspi-config.
- ctrlw 3y ago
  Yes, I was missing a mention of that in the blog post and maybe a brief discussion why to do the manual setup instead.
  - TMWNN 3y ago
    Indeed. The post even mentions `raspi-config`, but doesn't mention `Performance Options | Overlay File System`. Makes me think that the author isn't aware of the option at all. (The same way undoes read-only mode, but two reboots will be required instead of one.)
    - musicale 3y ago
      It is a puzzling omission.
- abdela 3y ago
  And you can create a second, small partition/filesystem if you still need to persist some things. I used to have very little data to persist and I would mount/write/unmount every time I write to deal with possible power issues.
st_goliath 3y ago
Surprisingly, the article doesn't seem to mention SquashFS[1] or EROFS[2]. Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts. For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4]. [1] https://en.wikipedia.org/wiki/SquashFS https://en.wikipedia.org/wiki/SquashFS [2] https://en.wikipedia.org/wiki/EROFS https://en.wikipedia.org/wiki/EROFS [3] https://www.sigma-star.at/blog/2022/07/squashfs-erofs/ https://www.sigma-star.at/blog/2022/07/squashfs-erofs/ [4] https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1.en.html https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1....
- m463 3y ago
  I have a pi or two running openwrt, which does this naturally. I also have a pi running pikvm. You change the filesystem by doing: # rw <-- makes filesystem read/write # (change config, or commands making filesystem changes) # ro <-- back to read-only
- geek_at 3y ago
  This is actually why I switched all my Pis to Alpine Linux since they use a RAM disk by default for RaspPis Haven't lost an SD card since, meanwhile my Octoprint RasPi has failed 4 times (mostly because of random power disconnects)
  - hnarn 3y ago
    I'm a fan of Alpine too, but I want to point out that to the best of my knowledge at least, running root from RAM is trivial and is not something you need to change distros for, you simply have to add the "toram" kernel boot parameter. For example, if your entire read-only root is contained in "filesystem.squashfs", then adding "toram=filesystem.squashfs" would read all of this into RAM on boot. I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.
eschneider 3y ago
I make a lot of embedded linux distress and there are a lot of ways to skin the r/o rootsf problem. Assuming you've got some bit of r/w storage, the easiest thing is making the whole rootfs r/o and bind mounting the bits you need writable.
- abdela 3y ago
  That is what I did as well.
floating-io 3y ago
This problem seems very overblown to me. The simple answer is "don't skimp." - Use a power supply that can deliver enough current - Use good USB cables that can actually carry said current (a lot of people miss this one) - Do not use cheap SD cards - Oversize your card so that the card can properly perform wear-levelling. I have a pair of Raspberry Pi's (2's and/or 3's IIRC) running constantly since 2015. They're running an ancient version of Redsleeve (and are on my list to rebuild but never mind that). DNS, DHCP, and a few other services run on them. They've also survived multiple abrupt powerfail events. They log to the SD card, and yet I've never had a problem. I personally believe that the fact that I'm using oversized cards with gigabytes of free space is the key. Well, perhaps that and the fact that my personal home directory is on NFS. Or maybe I'm just lucky? Who knows...
- sdenton4 3y ago
  Not sure how things are these days, but it used to be the case that power fluctuations could kill an rpi's sd card. So yeah, you could mitigate with a great power supply and a good surge protector, but in practice I found it hard to figure out which power supplies to actually trust. For my own deployments, I ended up doing read only boot from SD card, making use of a ram disk for temporary files, and hosting the actual os on a USB stick, which was much better shielded from power issues. Worked great, never had issues, but the setup was pretty non standard.
  - walteweiss 3y ago
    Chances you have any online notes on that?
    - sdenton4 3y ago
      Sadly, I don't think I do; I was mostly messing around with pi's ten years ago, and don't seem to have blog entries specifically on the usb disk. The main idea, though, was to set up the os so that the boot folder was on the SD card, with root mounted from the usb drive, and /tmp on a ramfs. Iirc, I would install normally on the SD card, and then copy data to the usb stick and edit the boot files to point where they needed to. And finally, set the SD card to read only mode once everything was settled. I think there's better options for all of this on more recent pi's, though. I think you can just boot directly from USB now, for example, instead of relying on the SD card.
anon-3988 3y ago
Thanks for this. I am using my RPI5 to learn about using remote computers, hacking around with and writing my tools to monitor stuff and do automation in Python. Just putting it here in case anyone have similar blogs/articles I can read about that goes through the same journey.
eternityforest 3y ago
I don't usually do full read only, what I'll do is run a script that turns off stuff that does not need to be writing to the disk all the time. Unfortunately, some software is database-oriented and likes to write to disk for every tiny thing, so the approach doesn't work with stuff like Home Assistant unless you carefully configure logging. The basic simple stuff doesn't really cause any user-level noticable changes: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/linux-sd-protect.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... After that, I disable and mask apt-daily (The Debian auto updater), and purge dphys-swapfile. My full set of assorted tweaks can be found here, some might not be relevant for you: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/linux-tweaks.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... Next, I often run Chromium as a kiosk, and Chromium likes to hammer the SD card, so I set the XDG folder environment variables to make it put it's stuff in RAM. My embedded chrome stuff can be found here: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/setup-kiosk-mode.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... With all these tweaks, a Pi will have excellent reliability without losing the benefits of a writable FS. If you need true industrial grade perfect reliability, you probably need an industrial card and maybe true a read-only system.
Abishek_Muthian 3y ago
I have couple of memory cards (mostly Samsung) for RPi which became readonly, I keep them separately for use cases mentioned by OP. I've since then switched to using only Toshibha memory cards in RPi & haven't had any failures.
ngcc_hk 3y ago
Wonder can you do this for those read only publishing site using even some cloud system image. Or even GitHub based web site? What is the advantage of your own read only system. It is not changed mostly after all (or all changes only from your good self).
lahvak 3y ago
Years ago I used one of the first pis as a music/podcast player in my car. The problem was that there was no proper shutdown process, you turn of the car, and the power cuts off, so I set it up to create and mount a ram disk while booting, and do all writing to the ram disk. From the comments here, it looks like the newer pis have this as a config option now.
deleted 3y ago
[deleted]
vinni2 3y ago
I realized this the hard way after about 5 of my se cards were fried.
hiAndrewQuinn 3y ago
The Pi I have powering https://hiandrewquinn.github.io/selkouutiset-archive/ https://hiandrewquinn.github.io/selkouutiset-archive/ has a read-only / . Everything happens in RAM, and resetting everything back to normal is a matter of unplugging and replugging the device back in. I could do so much more with this resource, but I'm honestly quite happy with it as it is.
cjdell 3y ago
The "buildroot" project is perfect for your embedded system. Build a custom minimal Linux system with tiny image sizes that are read-only by default. Pick exactly which services you need and no more. Less moving parts the better. I'm using it on 10 year old Pi's with USB RS485 dongles to get Ethernet access to solar equipment.
manuel_w 3y ago
There's one (not always but often enough) viable alternative to read-only root filesystems that is often overlooked. It's the possibility to copy the entire rootfs onto a ramdisk when booting. It increases the boot time and obviously disables persisting data (logs etc.), but often these are not needed anyway. It's just a few lines of code in initramfs! --- a/usr/share/initramfs-tools/scripts/local 2021-11-05 12:50:23.541088057 +0100 +++ b/usr/share/initramfs-tools/scripts/local 2021-11-05 13:02:14.483203576 +0100 @@ -180,9 +180,20 @@ # Mount root # shellcheck disable=SC2086 - if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then - panic "Failed to mount ${ROOT} as root file system." - fi + #if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then + # panic "Failed to mount ${ROOT} as root file system." + #fi + + mkdir --parents /tmp/diskroot + mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} /tmp/diskroot + + mount -t tmpfs -o size=6G none ${rootmnt?} + chmod 755 ${rootmnt} + + cp --force --archive --verbose /tmp/diskroot/* ${rootmnt} + + umount /tmp/diskroot + rm -r --force /tmp/diskroot } local_mount_fs() and `update-initramfs -u`.
- gsliepen 3y ago
  Even better, you can just put the whole root fs into the initramfs image, and never pivot to another root filesystem during boot. This also allows you to netboot a whole OS.
  - ylyn 3y ago
    One pitfall of this is that the decompressed contents of your initramfs must fit within half of your physical RAM since Linux decompresses it into a tmpfs. Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.
    - ahepp 3y ago
      Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram? Initramfs (and tmpfs) are swappable right? As you say, ramfs is the one that isn't swappable. So if your initramfs is "to big" can't it just be swapped out?
iam-TJ 3y ago
Several mentions here of using overlay file-system via the RaspberryPi specific raspi-config options. However, where-ever systemd is the init one can do it via adding to the kernel command line systemd.volatile=overlay "If this setting is set to "overlay" the root file system is set up as "overlayfs" mount combining the read-only root directory with a writable "tmpfs", so that no modifications are made to disk, but the file system may be modified nonetheless with all changes being lost at reboot." See man 7 kernel-command-line for the other volatile mode options.
rompic 3y ago
As it's not mentioned yet: log2ram already helped a lot in my tiny pi-hole setup.
pingec 3y ago
Few years ago I used this script and the r.pi is still running fine. https://gitlab.com/larsfp/rpi-readonly https://gitlab.com/larsfp/rpi-readonly
PhilipRoman 3y ago
Recently did something similar with an old weak laptop but I went all out and even avoid reads from the HDD. On startup I copy most root directories into a zram block device (maximum 4GB uncompressed, actual memory usage is 50% of disk usage), then shadow them with the copy in ram using bind mounts. Works really well, the disk is only accessed once on startup for about a minute while the system boots.
uuddlrlrbaba 3y ago
High quality large capacity SD cards are $5-$15 now, its really not worth the extra hassle to treat them specially. Just keep decent backups
ahepp 3y ago
Cool article! I would argue that /var is kinda explicitly for writing stuff to, so I would mount a tmpfs overlay on top of it as a more durable solution than mounting tmpfs on explicit directories inside it. In fact, I am interested in how the procedure in this article compares to just slapping a tmpfs overlay on top of the whole system? Could be interesting to compare.
hcfman 3y ago
If you install my sbts-aru project https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru It will shrink your partitions, add new ones and install one of these and set up a sub micro second system clock and an audio recorder suitable for sound localization with a single install command. Works on all versions of Pi in both raspbian and bookworm. If you don’t need the recorder just turn it off but you have a rootfs, a swap fs, a config fs and a disk fs.

218 points by ingve 3y ago

logicziller 3y ago

We use pi-builder that can produce reproducible images. Read-only is one of the default options.

ashwoods 3y ago

I now prefer to use overlayfs with a tmpfs overlay. Much simpler to configure and update.

crashbunny 3y ago
years ago I did the same, I vaguely remember following a blog post to do it. I put the scripts on github and some people found it useful. There's a deb package now that does it called overlayroot.

deleted 3y ago

[deleted]

Wowfunhappy 3y ago

> This means you can run the Pi with a read-only root filesystem, which will dramatically increase the SD card’s lifetime. I can't find an authoritative source, but some quick Googling brings up websites like Reddit and Stackexchange saying that SD cards have ~100,000 lifetime write cycles. I feel like it would be hard to exhaust that. Is this a real problem people experience?

Proven 3y ago
[dead]
CliffStoll 3y ago
My website is on a pi-4 and hasn't had a hiccup in 4 years. So far, the SD-card has worked great.
- freedomben 3y ago
  How high of traffic is your website? And are you writing log files (like nginx or apache, plus application if applicable)?
adobrawy 3y ago
Yea, it's a real issue. Search for "raspberry pi dead SD card" to found report of ppl which have failing SD card every month or more often.
- rcarmo 3y ago
  Most of those tend to be power issues.
rcxdude 3y ago
Yes, it's not particularly hard to wear out an SD card (especially a cheap, high-densite one), with just the wear of writing logs. It does vary a lot depending on what exactly you are going, but it's a fairly commonly reported failure among raspberry pi users.
Prcmaker 3y ago
I haven't had an issue with SD cards where Pis had reliable power. Where I used to live though we had frequent power cuts, of only a couple of seconds, a couple of times a day. SD card corruption was a major pain. Even in areas with more reliable power, for a 24/7 on device, occasional power outages have lead me to move away from Pis. Battery backups are, of course, an option, but increase the cost for a reliable and simple system.
bobmcnamara 3y ago
It was a plague for years where power failure during a write would cause an SD card to go read only or worse.
jorvi 3y ago
It is, but mostly due to logging to disk instead of RAM. That’s what setting a read-only root file system prevents in a roundabout way, so people have started to misidentify it as the solution.
NegativeLatency 3y ago
A lot of people also run them with a power supply that will undervolt, switched to ones with more amps and haven't had issues since
monocasa 3y ago
Yes, as someone who shipped embedded systems running off of SD cards, it's a real problem. It's not the flash endurance you hit typically, but bugs in the card firmware that cause the FTL to completely lose it's mind. Yes, even with supposedly quality brands. About the only way I found to deal with the problem is to use "industrial" SD cards that are a little slower and pretty expensive, but the firmware takes less liberties in the name of specious benchmarks.
- stefan_ 3y ago
  It’s interesting. In my embedded experience, SD cards are absolute trash, eMMC is already much better, raw NAND is somewhere in between and then actual SSDs are much better than all of these again. Yet supposedly these are all the same basic building blocks.

jasongill 3y ago

You can skip the ntp/systemd-timesyncd/fake-hwclock stuff if you add a RTC/coin battery to the device. The Raspberry Pi 5 has built-in RTC with support for an official battery which costs about five bucks and works well for this. Older devices have aftermarket RTC's with battery backups as well.

petee 3y ago
Theres also a kernel option to sync PPS on a gpio from a gps clock

idatum 3y ago

These Pi* devices have served well running their Debian version for years. I had a Pi3b run ADS-B (generally RTL-SDR) for many years. Dang though, all with (seemingly) random SD storage failures. Article details seems like it can help. There are more options now though. Aarch64 ones that use eMMC. I replaced the Pi3b with a Pine64 Rock64 running FreeBSD 14 booting from eMMC. Simple and just works with all RTL-SDR needs covered. It's great to have options, both hw and os.

jauntywundrkind 3y ago

A long time ago I modified debian-live to use btrfs-send to copy the usb image into some kind of ramfs. So you could hypothetically snapshot & send back your current snapshot to the USB as you want. Having a kick ass fs at your back is great. Go for it! One of my persistently greatest annoyances about running stuff on pis has been that journald is so monolithic. There's no per-app controls. I'd love to log most system functions persistently with decent sized history (since they are very low volume), but have the main app or another app do something else. Struggling to find them ATM, but there are a number of issues opened around this way back, and all closed, saying to run multiple systemd-journal instances then configure apps to direct traffic to each, but that sure sucks to setup & configure, & is something I'd hope journald would help me with out of the box.

blackfawn 3y ago

I like using Alpine for read-only operation on the Pi right out of the box: https://wiki.alpinelinux.org/wiki/Raspberry_Pi https://wiki.alpinelinux.org/wiki/Raspberry_Pi

GianFabien 3y ago
Alpine has been great for me too. Use an USB Flash drive mounted as a R/W filesystem for logs, etc. fsck can usually recover those f/s when crashed. The root f/s remains intact.
seemaze 3y ago
This is the way. Run from ram and an excellent selection of packages to boot! I have a number of pi zero and pi 2 zero running alpine.

LeoPanthera 3y ago

It's worth noting that you don't have to do this manually. Enabling "overlay filesystem" in raspi-config will do this for you, so that all changes to the filesystem are lost after a reboot.

HankB99 3y ago
I believe the overlayfs configured using raspi-config is a little different in that it presents the appearance of a normal filesystem but all changes are gone following reboot. Before reboot the changes are cached in RAM which can have consequences for low RAM systems. I use the overlayfs provided by raspi-config.
ctrlw 3y ago
Yes, I was missing a mention of that in the blog post and maybe a brief discussion why to do the manual setup instead.
- TMWNN 3y ago
  Indeed. The post even mentions `raspi-config`, but doesn't mention `Performance Options | Overlay File System`. Makes me think that the author isn't aware of the option at all. (The same way undoes read-only mode, but two reboots will be required instead of one.)
  - musicale 3y ago
    It is a puzzling omission.
abdela 3y ago
And you can create a second, small partition/filesystem if you still need to persist some things. I used to have very little data to persist and I would mount/write/unmount every time I write to deal with possible power issues.

st_goliath 3y ago

Surprisingly, the article doesn't seem to mention SquashFS[1] or EROFS[2]. Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts. For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4]. [1] https://en.wikipedia.org/wiki/SquashFS https://en.wikipedia.org/wiki/SquashFS [2] https://en.wikipedia.org/wiki/EROFS https://en.wikipedia.org/wiki/EROFS [3] https://www.sigma-star.at/blog/2022/07/squashfs-erofs/ https://www.sigma-star.at/blog/2022/07/squashfs-erofs/ [4] https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1.en.html https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1....

m463 3y ago
I have a pi or two running openwrt, which does this naturally. I also have a pi running pikvm. You change the filesystem by doing: # rw <-- makes filesystem read/write # (change config, or commands making filesystem changes) # ro <-- back to read-only
geek_at 3y ago
This is actually why I switched all my Pis to Alpine Linux since they use a RAM disk by default for RaspPis Haven't lost an SD card since, meanwhile my Octoprint RasPi has failed 4 times (mostly because of random power disconnects)
- hnarn 3y ago
  I'm a fan of Alpine too, but I want to point out that to the best of my knowledge at least, running root from RAM is trivial and is not something you need to change distros for, you simply have to add the "toram" kernel boot parameter. For example, if your entire read-only root is contained in "filesystem.squashfs", then adding "toram=filesystem.squashfs" would read all of this into RAM on boot. I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.

eschneider 3y ago

I make a lot of embedded linux distress and there are a lot of ways to skin the r/o rootsf problem. Assuming you've got some bit of r/w storage, the easiest thing is making the whole rootfs r/o and bind mounting the bits you need writable.

abdela 3y ago
That is what I did as well.

floating-io 3y ago

This problem seems very overblown to me. The simple answer is "don't skimp." - Use a power supply that can deliver enough current - Use good USB cables that can actually carry said current (a lot of people miss this one) - Do not use cheap SD cards - Oversize your card so that the card can properly perform wear-levelling. I have a pair of Raspberry Pi's (2's and/or 3's IIRC) running constantly since 2015. They're running an ancient version of Redsleeve (and are on my list to rebuild but never mind that). DNS, DHCP, and a few other services run on them. They've also survived multiple abrupt powerfail events. They log to the SD card, and yet I've never had a problem. I personally believe that the fact that I'm using oversized cards with gigabytes of free space is the key. Well, perhaps that and the fact that my personal home directory is on NFS. Or maybe I'm just lucky? Who knows...

sdenton4 3y ago
Not sure how things are these days, but it used to be the case that power fluctuations could kill an rpi's sd card. So yeah, you could mitigate with a great power supply and a good surge protector, but in practice I found it hard to figure out which power supplies to actually trust. For my own deployments, I ended up doing read only boot from SD card, making use of a ram disk for temporary files, and hosting the actual os on a USB stick, which was much better shielded from power issues. Worked great, never had issues, but the setup was pretty non standard.
- walteweiss 3y ago
  Chances you have any online notes on that?
  - sdenton4 3y ago
    Sadly, I don't think I do; I was mostly messing around with pi's ten years ago, and don't seem to have blog entries specifically on the usb disk. The main idea, though, was to set up the os so that the boot folder was on the SD card, with root mounted from the usb drive, and /tmp on a ramfs. Iirc, I would install normally on the SD card, and then copy data to the usb stick and edit the boot files to point where they needed to. And finally, set the SD card to read only mode once everything was settled. I think there's better options for all of this on more recent pi's, though. I think you can just boot directly from USB now, for example, instead of relying on the SD card.

anon-3988 3y ago

Thanks for this. I am using my RPI5 to learn about using remote computers, hacking around with and writing my tools to monitor stuff and do automation in Python. Just putting it here in case anyone have similar blogs/articles I can read about that goes through the same journey.

eternityforest 3y ago

I don't usually do full read only, what I'll do is run a script that turns off stuff that does not need to be writing to the disk all the time. Unfortunately, some software is database-oriented and likes to write to disk for every tiny thing, so the approach doesn't work with stuff like Home Assistant unless you carefully configure logging. The basic simple stuff doesn't really cause any user-level noticable changes: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/linux-sd-protect.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... After that, I disable and mask apt-daily (The Debian auto updater), and purge dphys-swapfile. My full set of assorted tweaks can be found here, some might not be relevant for you: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/linux-tweaks.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... Next, I often run Chromium as a kiosk, and Chromium likes to hammer the SD card, so I set the XDG folder environment variables to make it put it's stuff in RAM. My embedded chrome stuff can be found here: https://github.com/EternityForest/KaithemAutomation/blob/master/scripts/setup-kiosk-mode.sh https://github.com/EternityForest/KaithemAutomation/blob/mas... With all these tweaks, a Pi will have excellent reliability without losing the benefits of a writable FS. If you need true industrial grade perfect reliability, you probably need an industrial card and maybe true a read-only system.

Abishek_Muthian 3y ago

I have couple of memory cards (mostly Samsung) for RPi which became readonly, I keep them separately for use cases mentioned by OP. I've since then switched to using only Toshibha memory cards in RPi & haven't had any failures.

ngcc_hk 3y ago

Wonder can you do this for those read only publishing site using even some cloud system image. Or even GitHub based web site? What is the advantage of your own read only system. It is not changed mostly after all (or all changes only from your good self).

lahvak 3y ago

Years ago I used one of the first pis as a music/podcast player in my car. The problem was that there was no proper shutdown process, you turn of the car, and the power cuts off, so I set it up to create and mount a ram disk while booting, and do all writing to the ram disk. From the comments here, it looks like the newer pis have this as a config option now.

deleted 3y ago

[deleted]

vinni2 3y ago

I realized this the hard way after about 5 of my se cards were fried.

hiAndrewQuinn 3y ago

The Pi I have powering https://hiandrewquinn.github.io/selkouutiset-archive/ https://hiandrewquinn.github.io/selkouutiset-archive/ has a read-only / . Everything happens in RAM, and resetting everything back to normal is a matter of unplugging and replugging the device back in. I could do so much more with this resource, but I'm honestly quite happy with it as it is.

cjdell 3y ago

The "buildroot" project is perfect for your embedded system. Build a custom minimal Linux system with tiny image sizes that are read-only by default. Pick exactly which services you need and no more. Less moving parts the better. I'm using it on 10 year old Pi's with USB RS485 dongles to get Ethernet access to solar equipment.

manuel_w 3y ago

There's one (not always but often enough) viable alternative to read-only root filesystems that is often overlooked. It's the possibility to copy the entire rootfs onto a ramdisk when booting. It increases the boot time and obviously disables persisting data (logs etc.), but often these are not needed anyway. It's just a few lines of code in initramfs! --- a/usr/share/initramfs-tools/scripts/local 2021-11-05 12:50:23.541088057 +0100 +++ b/usr/share/initramfs-tools/scripts/local 2021-11-05 13:02:14.483203576 +0100 @@ -180,9 +180,20 @@ # Mount root # shellcheck disable=SC2086 - if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then - panic "Failed to mount ${ROOT} as root file system." - fi + #if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then + # panic "Failed to mount ${ROOT} as root file system." + #fi + + mkdir --parents /tmp/diskroot + mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} /tmp/diskroot + + mount -t tmpfs -o size=6G none ${rootmnt?} + chmod 755 ${rootmnt} + + cp --force --archive --verbose /tmp/diskroot/* ${rootmnt} + + umount /tmp/diskroot + rm -r --force /tmp/diskroot } local_mount_fs() and `update-initramfs -u`.

gsliepen 3y ago
Even better, you can just put the whole root fs into the initramfs image, and never pivot to another root filesystem during boot. This also allows you to netboot a whole OS.
- ylyn 3y ago
  One pitfall of this is that the decompressed contents of your initramfs must fit within half of your physical RAM since Linux decompresses it into a tmpfs. Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.
  - ahepp 3y ago
    Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram? Initramfs (and tmpfs) are swappable right? As you say, ramfs is the one that isn't swappable. So if your initramfs is "to big" can't it just be swapped out?

iam-TJ 3y ago

Several mentions here of using overlay file-system via the RaspberryPi specific raspi-config options. However, where-ever systemd is the init one can do it via adding to the kernel command line systemd.volatile=overlay "If this setting is set to "overlay" the root file system is set up as "overlayfs" mount combining the read-only root directory with a writable "tmpfs", so that no modifications are made to disk, but the file system may be modified nonetheless with all changes being lost at reboot." See man 7 kernel-command-line for the other volatile mode options.

rompic 3y ago

As it's not mentioned yet: log2ram already helped a lot in my tiny pi-hole setup.

pingec 3y ago

Few years ago I used this script and the r.pi is still running fine. https://gitlab.com/larsfp/rpi-readonly https://gitlab.com/larsfp/rpi-readonly

PhilipRoman 3y ago

Recently did something similar with an old weak laptop but I went all out and even avoid reads from the HDD. On startup I copy most root directories into a zram block device (maximum 4GB uncompressed, actual memory usage is 50% of disk usage), then shadow them with the copy in ram using bind mounts. Works really well, the disk is only accessed once on startup for about a minute while the system boots.

uuddlrlrbaba 3y ago

High quality large capacity SD cards are $5-$15 now, its really not worth the extra hassle to treat them specially. Just keep decent backups

ahepp 3y ago

Cool article! I would argue that /var is kinda explicitly for writing stuff to, so I would mount a tmpfs overlay on top of it as a more durable solution than mounting tmpfs on explicit directories inside it. In fact, I am interested in how the procedure in this article compares to just slapping a tmpfs overlay on top of the whole system? Could be interesting to compare.

hcfman 3y ago

If you install my sbts-aru project https://github.com/hcfman/sbts-aru https://github.com/hcfman/sbts-aru It will shrink your partitions, add new ones and install one of these and set up a sub micro second system clock and an audio recorder suitable for sound localization with a single install command. Works on all versions of Pi in both raspbian and bookworm. If you don’t need the recorder just turn it off but you have a rootfs, a swap fs, a config fs and a disk fs.