4 ms·

> make sure you have n+1 pi & disk machine. Why? The thing that could fail are the disks. So, one can use a RAID1 with two disks and this should, for normal ap

by jnxx 6y ago

> make sure you have n+1 pi & disk machine. Why? The thing that could fail are the disks. So, one can use a RAID1 with two disks and this should, for normal applications be enough. I had a DNS-232 before as a NAS, it had 32 MB RAM, a tiny ARM CPU, and two large disks. It was working nicely over about ten years, and one could even run a web server plus a (slow) wiki on it (I had to replace the casing once, for a botched firmware update). The main problem I had with it that it was that it was impossible to update (I was running Alt-F firmware, which was developed by the community after the original firmware had constant problems with unneeded RAID synchronization, and the vendor had to release its code due to being bound to the GPL). The Pi400 is wildly more powerful, has a much faster CPU which only draws a few Watt, and can easily be connected to several USB disks. > Raid isn't going to help you here, you need redundant nodes. Why exactly? My set-up is working well.... did I miss it needs to be somehow lighter than air (because heavier-than air flying machines are not possible, y'know)?

KaiserPro 6y ago
> Why exactly? My set-up is working well congratulations I am very happy for you! The issue is not speed, its the USB disks. firstly you are reliant on the disks presenting themselves in the same order at the same time. (historically that was challenging) ZFS can cope with this quite well, MDADM, not so much. You can use device IDs, which gives better results. Another issue I've bumped into is that mdadm has come up before the USB disk is initialised, causing all sorts of issues. in short raid over USB is just plain fragile. > Raid isn't going to help you here, you need redundant nodes. This setup will have four common failure modes: o power issue leading to corrupt "/" or any other SD card failure o usb disk breaking o OS crash borking everything o Accidental deletion Raid might protect against a usbdisk breaking, assuming you spot it in time (most people don't have monitoring.) its not even going to give you a performance boost either, its USB3. This is why n+1 is better. assuming you're not keeping each node next to each other then you are much less likely to experience a simultaneous outage.
- jnxx 6y ago
  > firstly you are reliant on the disks presenting themselves in the same order at the same time. In Linux, which is what Raspbian is, it is standard nowadays to use UUID disk labels. > Another issue I've bumped into is that mdadm has come up before the USB disk is initialised, causing all sorts of issues. in short raid over USB is just plain fragile. No issue with that. I guess systemd handles this. Only thing I found is that the hub needs to provide enough power on booting up. > power issue leading to corrupt "/" or any other SD card failure For that, I have a backup of the sd card. I am using an "industrial" SD card by the way which costs a few bucks more. Also, the card is running in read-only mode, using unionfs. Logs are written to a partition of the RAID disks. > usb disk breaking That's handled by RAID1 and backups. > OS crash borking everything I've never seen that on Linux, in 22 years using it. The closest I came to was botched EFI parameters from an Ubuntu install gone wrong, on a Desktop PC. Wait, I also had a problem in about 2003 with a broken ISA disk controller. Ah, and I am using EXT3 which is a journaling file system. > Raid might protect against a usbdisk breaking Yeah, that is what it is for. It is not a substitute for backups (but nobody said something like that, I think). In fact, it is easier to make backups from a Pi, using the standard Linux tools like tar, ssh and rsync. > its not even going to give you a performance boost either, its USB3 For my purposes, it does not need to be faster (and it is much, much faster than the old DNS-232). The Pi 400 has Gigabit Ethernet and plenty of RAM. I am running a MoinMoin wiki instance on it, and it is more than fast enough for personal information management. It can well be that a small thing like that is not fast enough for a medium-scale business with, say, a dozen clients. But this was not what my requirements were. I wanted to have a solution which is very low-power. And the Pi 400 is probably one or two Watt if idle, and the disks are spun down: https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications https://www.raspberrypi-spy.co.uk/2018/11/raspberry-pi-power-consumption-data/ https://www.raspberrypi-spy.co.uk/2018/11/raspberry-pi-power... > This is why n+1 is better So, what kind of set-up are you exactly talking about, a distributed system of some kind? A storage cluster? Ceph? DynamoDB?
  - jnxx 6y ago
    Just to complete this: As said, it is not a good idea to leave the root file system of the Raspberry, which is on an SD card, writable for heavy-duty, long-term tasks where reliability is important. It is not made for that. Among other things, this includes logging. Instead, I am using this script: https://github.com/fitu996/overlayRoot.sh https://github.com/fitu996/overlayRoot.sh which mounts an overlay-filesystem with the bottom layer being the root-fs, the upper layer a new fs on writable media, which is a RAID partition in my case, and swaps the overlayfs with the root fs. That means that from then, the sd card of the Raspberry is not modified, and all changes go to the RAID disk. If you want to modify the system configuration, you need to re-boot with a different kernel parameter, which is detailed on the linked page.
- jnxx 6y ago
  > The issue is not speed, its the USB disks. firstly you are reliant on the disks presenting themselves in the same order at the same time. (historically that was challenging) ZFS can cope with this quite well, MDADM, not so much. I have checked about that and mdadm uses uuids to identify partitions on USB disks: https://serverfault.com/questions/739580/mdadm-can-i-reconnect-a-usb-disk-as-sata-without-needing-a-rebuild https://serverfault.com/questions/739580/mdadm-can-i-reconne... https://serverfault.com/questions/460138/mdadm-disk-configuration-by-uuid https://serverfault.com/questions/460138/mdadm-disk-configur... https://www.linuxquestions.org/questions/linux-software-2/mdadm-usb-drives-and-uuid-questions-4175530170/ https://www.linuxquestions.org/questions/linux-software-2/md... Here is how it looks on my array: root@arau:~# mdadm --detail /dev/md1 /dev/md1: Version : 1.2 Creation Time : Thu Dec 10 08:56:22 2020 Raid Level : raid1 Array Size : 4821105600 (4597.76 GiB 4936.81 GB) Used Dev Size : 4821105600 (4597.76 GiB 4936.81 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Thu Apr 15 11:18:54 2021 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : arau2:1 UUID : 7c1e57ea:1f7ebff2:415c1b0a:8f959031 Events : 21463 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 18 1 active sync /dev/sdb2 root@arau:~# root@arau:~# mdadm --examine /dev/sda2 /dev/sda2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 7c1e57ea:1f7ebff2:415c1b0a:8f959031 Name : arau2:1 Creation Time : Thu Dec 10 08:56:22 2020 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 9642249512 (4597.78 GiB 4936.83 GB) Array Size : 4821105600 (4597.76 GiB 4936.81 GB) Used Dev Size : 9642211200 (4597.76 GiB 4936.81 GB) Data Offset : 264192 sectors Super Offset : 8 sectors Unused Space : before=264112 sectors, after=38312 sectors State : clean Device UUID : 52594d2c:77f9c0cc:0b4243f6:6a8e6bcb Internal Bitmap : 8 sectors from superblock Update Time : Thu Apr 15 11:18:54 2021 Bad Block Log : 512 entries available at offset 32 sectors Checksum : ca81c1ad - correct Events : 21463 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@arau:~# You see the second UUID entry? It is an USB disk. And I looked into the madm changelog and git log and I did not find any reference to problems as you describe since 2014. I am a bit confused why you refer to hypothetical problems or ones which do not exist anymore for years?

by jnxx 6y ago

KaiserPro 6y ago

> Why exactly? My set-up is working well congratulations I am very happy for you! The issue is not speed, its the USB disks. firstly you are reliant on the disks presenting themselves in the same order at the same time. (historically that was challenging) ZFS can cope with this quite well, MDADM, not so much. You can use device IDs, which gives better results. Another issue I've bumped into is that mdadm has come up before the USB disk is initialised, causing all sorts of issues. in short raid over USB is just plain fragile. > Raid isn't going to help you here, you need redundant nodes. This setup will have four common failure modes: o power issue leading to corrupt "/" or any other SD card failure o usb disk breaking o OS crash borking everything o Accidental deletion Raid might protect against a usbdisk breaking, assuming you spot it in time (most people don't have monitoring.) its not even going to give you a performance boost either, its USB3. This is why n+1 is better. assuming you're not keeping each node next to each other then you are much less likely to experience a simultaneous outage.

jnxx 6y ago
> firstly you are reliant on the disks presenting themselves in the same order at the same time. In Linux, which is what Raspbian is, it is standard nowadays to use UUID disk labels. > Another issue I've bumped into is that mdadm has come up before the USB disk is initialised, causing all sorts of issues. in short raid over USB is just plain fragile. No issue with that. I guess systemd handles this. Only thing I found is that the hub needs to provide enough power on booting up. > power issue leading to corrupt "/" or any other SD card failure For that, I have a backup of the sd card. I am using an "industrial" SD card by the way which costs a few bucks more. Also, the card is running in read-only mode, using unionfs. Logs are written to a partition of the RAID disks. > usb disk breaking That's handled by RAID1 and backups. > OS crash borking everything I've never seen that on Linux, in 22 years using it. The closest I came to was botched EFI parameters from an Ubuntu install gone wrong, on a Desktop PC. Wait, I also had a problem in about 2003 with a broken ISA disk controller. Ah, and I am using EXT3 which is a journaling file system. > Raid might protect against a usbdisk breaking Yeah, that is what it is for. It is not a substitute for backups (but nobody said something like that, I think). In fact, it is easier to make backups from a Pi, using the standard Linux tools like tar, ssh and rsync. > its not even going to give you a performance boost either, its USB3 For my purposes, it does not need to be faster (and it is much, much faster than the old DNS-232). The Pi 400 has Gigabit Ethernet and plenty of RAM. I am running a MoinMoin wiki instance on it, and it is more than fast enough for personal information management. It can well be that a small thing like that is not fast enough for a medium-scale business with, say, a dozen clients. But this was not what my requirements were. I wanted to have a solution which is very low-power. And the Pi 400 is probably one or two Watt if idle, and the disks are spun down: https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications https://en.wikipedia.org/wiki/Raspberry_Pi#Specifications https://www.raspberrypi-spy.co.uk/2018/11/raspberry-pi-power-consumption-data/ https://www.raspberrypi-spy.co.uk/2018/11/raspberry-pi-power... > This is why n+1 is better So, what kind of set-up are you exactly talking about, a distributed system of some kind? A storage cluster? Ceph? DynamoDB?
- jnxx 6y ago
  Just to complete this: As said, it is not a good idea to leave the root file system of the Raspberry, which is on an SD card, writable for heavy-duty, long-term tasks where reliability is important. It is not made for that. Among other things, this includes logging. Instead, I am using this script: https://github.com/fitu996/overlayRoot.sh https://github.com/fitu996/overlayRoot.sh which mounts an overlay-filesystem with the bottom layer being the root-fs, the upper layer a new fs on writable media, which is a RAID partition in my case, and swaps the overlayfs with the root fs. That means that from then, the sd card of the Raspberry is not modified, and all changes go to the RAID disk. If you want to modify the system configuration, you need to re-boot with a different kernel parameter, which is detailed on the linked page.
jnxx 6y ago
> The issue is not speed, its the USB disks. firstly you are reliant on the disks presenting themselves in the same order at the same time. (historically that was challenging) ZFS can cope with this quite well, MDADM, not so much. I have checked about that and mdadm uses uuids to identify partitions on USB disks: https://serverfault.com/questions/739580/mdadm-can-i-reconnect-a-usb-disk-as-sata-without-needing-a-rebuild https://serverfault.com/questions/739580/mdadm-can-i-reconne... https://serverfault.com/questions/460138/mdadm-disk-configuration-by-uuid https://serverfault.com/questions/460138/mdadm-disk-configur... https://www.linuxquestions.org/questions/linux-software-2/mdadm-usb-drives-and-uuid-questions-4175530170/ https://www.linuxquestions.org/questions/linux-software-2/md... Here is how it looks on my array: root@arau:~# mdadm --detail /dev/md1 /dev/md1: Version : 1.2 Creation Time : Thu Dec 10 08:56:22 2020 Raid Level : raid1 Array Size : 4821105600 (4597.76 GiB 4936.81 GB) Used Dev Size : 4821105600 (4597.76 GiB 4936.81 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Thu Apr 15 11:18:54 2021 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : arau2:1 UUID : 7c1e57ea:1f7ebff2:415c1b0a:8f959031 Events : 21463 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 18 1 active sync /dev/sdb2 root@arau:~# root@arau:~# mdadm --examine /dev/sda2 /dev/sda2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 7c1e57ea:1f7ebff2:415c1b0a:8f959031 Name : arau2:1 Creation Time : Thu Dec 10 08:56:22 2020 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 9642249512 (4597.78 GiB 4936.83 GB) Array Size : 4821105600 (4597.76 GiB 4936.81 GB) Used Dev Size : 9642211200 (4597.76 GiB 4936.81 GB) Data Offset : 264192 sectors Super Offset : 8 sectors Unused Space : before=264112 sectors, after=38312 sectors State : clean Device UUID : 52594d2c:77f9c0cc:0b4243f6:6a8e6bcb Internal Bitmap : 8 sectors from superblock Update Time : Thu Apr 15 11:18:54 2021 Bad Block Log : 512 entries available at offset 32 sectors Checksum : ca81c1ad - correct Events : 21463 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@arau:~# You see the second UUID entry? It is an USB disk. And I looked into the madm changelog and git log and I did not find any reference to problems as you describe since 2014. I am a bit confused why you refer to hypothetical problems or ones which do not exist anymore for years?