3 ms·

In addition, had the OP used disk partitions, a valid partition table (either MBR or GPT) would have been written to disk. Every Linux installer that I've used

In addition, had the OP used disk partitions, a valid partition table (either MBR or GPT) would have been written to disk. Every Linux installer that I've used sets up disk partitions for software RAID volumes. I don't think an installer for a major distro will let you use a bare device, even if you wanted to, and all of the instructions/tutorials I've seen on setting up mdadm include the configuration of partitions. Seems like the OP went out of their way to configure software RAID in a non-standard manner, without understanding the consequences. I hate to kick the OP while they're down, but it doesn't look like this is an issue that someone would encounter with a default/recommended configuration.

nh2 8y ago
No worries, this is good advice, I appreciate it. (Also note I managed to fully recover from this without data loss, and I have backups.) This RAID setup is already quite old. For newer RAID setups I routinely put the RAID on partitions as you said, after more advice has appeared on the Internet that this is recommended. However, I didn't find hard arguments for this beyond "if you make the partitions a bit smaller, you can accomodate disks that aren't exactly euqually large" -- certainly I didn't encounter "if you don't do this, UEFI may trash your setup". It would be inaccurate to say I went "out of my way to set it up in a non-standard manner" though. I didn't use a distro installer, simply because the machine was already installed when I added this RAID to it. I just used straightforward `mdadm --create --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd` on bare devices, which is a supported use case by mdadm, and much documentation and tutorial material still mention that you can choose bare devices vs partition (probably hard to blame them for that, given that this is the first and only mainboard I have encountered that does this).
- bradfa 8y ago
  Additionally, unless you do an mdadm software RAID using full disks, you cannot ensure that the EFI system partition will be RAIDed. But if you want to avoid having your mdadm metadata at the start of the drives, you can have it placed in various locations which may help, see: https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#Sub-versions_of_the_version-1_superblock https://raid.wiki.kernel.org/index.php/RAID_superblock_forma... I use mdadm RAID1 on full disks on my desktop with the metadata at the end of the disks (superblock version 1.0). Partitioning tools will complain that my backup GPT partition table is invalid and offer to fix it (I always politely decline) but the primary one is fine. My Dell desktop doesn't mind this situation and also does not ever write any data into the EFI partition so it works well for me.
  - shawnz 8y ago
    Why would you want the EFI system partition inside of a RAID volume that's not understood by your system firmware?
    - bradfa 8y ago
      When my OS updates the bootloader it stores in the EFI system partition, I want that data to be RAIDed. I set the EFI boot order to use the first disk's EFI system partition then the second disk's. Because at the OS level they are RAIDed and because I put the mdadm metadata at the ends of the disks, the UEFI firmware sees them as normal GPT partitions (with the secondary GPT partition table corrupted at the end of the disk due to mdadm data). When the EFI firmware runs, it should only be performing reads from either of the disks and even if one entire disk is dead it can still boot the system.
knorker 8y ago
It's easier to get alignment right if you use raw disk, especially if you then run dmcrypt on top of your RAID. Unaligned partitions is a common source of bad performance, and it's not a thing that (used to) "just work". Therefore I've seen "don't use partitions" not be recommended, since it's otherwise a source of potential unalignment.
- rwmj 8y ago
  This hasn't been a problem since around 2008 when all OS vendors changed their partitioning tools to use at least 1M alignment by default. I also wrote a tool (http://libguestfs.org/virt-alignment-scan.1.html http://libguestfs.org/virt-alignment-scan.1.html) which can be used to scan and in a few cases correct for this problem.
  - sevensor 8y ago
    Incidentally, thank you for all your work on libguestfs! It's spectacularly useful.
  - justinclift 8y ago
    As a side point, I've seen mention recently that 1M alignment may not be optimal in some cases either. The recommendation was to use 4M instead, which apparently worked 100%. Just wish I could remember where I read that. I remember taking it on board as it made sense and the source seemed very credible.

nh2 8y ago

No worries, this is good advice, I appreciate it. (Also note I managed to fully recover from this without data loss, and I have backups.) This RAID setup is already quite old. For newer RAID setups I routinely put the RAID on partitions as you said, after more advice has appeared on the Internet that this is recommended. However, I didn't find hard arguments for this beyond "if you make the partitions a bit smaller, you can accomodate disks that aren't exactly euqually large" -- certainly I didn't encounter "if you don't do this, UEFI may trash your setup". It would be inaccurate to say I went "out of my way to set it up in a non-standard manner" though. I didn't use a distro installer, simply because the machine was already installed when I added this RAID to it. I just used straightforward `mdadm --create --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd` on bare devices, which is a supported use case by mdadm, and much documentation and tutorial material still mention that you can choose bare devices vs partition (probably hard to blame them for that, given that this is the first and only mainboard I have encountered that does this).

bradfa 8y ago
Additionally, unless you do an mdadm software RAID using full disks, you cannot ensure that the EFI system partition will be RAIDed. But if you want to avoid having your mdadm metadata at the start of the drives, you can have it placed in various locations which may help, see: https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#Sub-versions_of_the_version-1_superblock https://raid.wiki.kernel.org/index.php/RAID_superblock_forma... I use mdadm RAID1 on full disks on my desktop with the metadata at the end of the disks (superblock version 1.0). Partitioning tools will complain that my backup GPT partition table is invalid and offer to fix it (I always politely decline) but the primary one is fine. My Dell desktop doesn't mind this situation and also does not ever write any data into the EFI partition so it works well for me.
- shawnz 8y ago
  Why would you want the EFI system partition inside of a RAID volume that's not understood by your system firmware?
  - bradfa 8y ago
    When my OS updates the bootloader it stores in the EFI system partition, I want that data to be RAIDed. I set the EFI boot order to use the first disk's EFI system partition then the second disk's. Because at the OS level they are RAIDed and because I put the mdadm metadata at the ends of the disks, the UEFI firmware sees them as normal GPT partitions (with the secondary GPT partition table corrupted at the end of the disk due to mdadm data). When the EFI firmware runs, it should only be performing reads from either of the disks and even if one entire disk is dead it can still boot the system.

knorker 8y ago

It's easier to get alignment right if you use raw disk, especially if you then run dmcrypt on top of your RAID. Unaligned partitions is a common source of bad performance, and it's not a thing that (used to) "just work". Therefore I've seen "don't use partitions" not be recommended, since it's otherwise a source of potential unalignment.

rwmj 8y ago
This hasn't been a problem since around 2008 when all OS vendors changed their partitioning tools to use at least 1M alignment by default. I also wrote a tool (http://libguestfs.org/virt-alignment-scan.1.html http://libguestfs.org/virt-alignment-scan.1.html) which can be used to scan and in a few cases correct for this problem.
- sevensor 8y ago
  Incidentally, thank you for all your work on libguestfs! It's spectacularly useful.
- justinclift 8y ago
  As a side point, I've seen mention recently that 1M alignment may not be optimal in some cases either. The recommendation was to use 4M instead, which apparently worked 100%. Just wish I could remember where I read that. I remember taking it on board as it made sense and the source seemed very credible.