9 ms·
ASRock motherboard destroys Linux software RAID
- dalore 8y agoReading the title I first thought it was metaphorical. In that they destroyed in performance. Now I see it's literal.
- shmerl 8y agoHopefully Asrock will respond. They often don't.
- JustSomeNobody 8y agoLooks like it has been two weeks so far and no response. That's pretty bad. Someone should at least acknowledge that they are looking into it.
- creshal 8y agoIf a user buys some second-hand hardware that's 3 years obsolete and does out-of-spec things to it… yeah, I'm hardly surprised the vendor isn't looking into it.
- JustSomeNobody 8y agoReally? If that's the case, you think it's too difficult for them to just say, "Thank you for your post, but we are doing what the spec says to do, see the relevant portion here: ..."? Courtesy is a thing. At least I think it still is.
- mar77i 8y agohttps://xkcd.com/979/ https://xkcd.com/979/ See other sections in this thread: mdadm itself recommends using a partition to prevent hardware from tampering with the data. Therefore I don't see any real problem with anything around here.
- yjftsjthsd-h 8y agomdadm tells you to do that as a safeguard against buggy implementations. That does not make it okay to have buggy implementations.
- mjevans 8y agoIt also helps to safeguard against semi-buggy and forgettable humans. Literally the software equivalent of putting a label on your data. GPT with 1MB aligned partitions will probably work for the foreseeable future; if for no other reason than (popularity across OSes) inertia and it being a nice binary multiple. As a note; I refuse to use the silly 'MiB' style syntax. 1M bytes is base 10, 1MB is base 2 'near' to that, since bytes implies the native to the device base 2 alignment rather than the common among humans base 10 default.
- vectorEQ 8y agoOriginal author(s) Neil Brown Developer(s) Jes Sorensen Initial release 2001 why does this random authors of code needs to inform people of the fact their hardware has shitty bugs which are apparently never fixed because 'everyone will read best practices'. That seems problematic to me.... I would expect it sooner from the mainboard manufacturer in their information, - with perhaps a warning and a choice (uefi as gui so why not eh!) before destroying disk content.... (the OPs system clearly just does it without even prompting or giving an indication it will do it.) Before this firmwares modify the disk they could easily prompt to check if it's ok with the data's owner to modify this data. manufacturers shouldn't rely on assumptions in their code for processes which can destroy someones personal data in any case. there being a warning from some people who dive into disk drives a lot that this kind of problem exists doesn't make that situation any better.
- mar77i 8y agoWell I may be imagining a pattern here that's biased by my own mileage, but starting from what computers usually come pre-installed with (windows)...
- nottorp 8y agoI doubt whoever's paid to go through that forum (if there is anyone) is paid enough to even understand the problem...
- wyldfire 8y ago> It is important to know that a disk configured to be part of an mdadm RAID array can look like a broken EFI disk. Great detective work. When reporting a bug like this, it's this extra mile investigation that gives a lot of weight/credibility.
- richardwhiuk 8y agoIn my experience, configuring whole disks as software RAID is a world of pain. My experience is that a disk with a single partition is much more reliable when in an mdadm array. I wonder whether this was because I was also using a bugged EFI mainboard.
- tomxor 8y agoPerhaps, but either way it is not great to hear some UEFI has been designed to be "clever" to the point it's trying to hand hold users by deciding what should be in the superblock of a device without your explicit permission. I can almost forgive it for OS designed for novices but even then they tend to ask permission - however, putting these assumptions into hardware and not even asking permission - that's a broken piece hardware IMO.
- Filligree 8y agoGPT has a backup/recovery feature. If the disk was ever a valid GPT disk, then there'll be a copy of the partition table in the final block -- and if that wasn't cleared properly then a functioning UEFI is supposed to restore the primary table from it. The UEFI bios may be following the spec to the letter, here. If so, the takeaway is to always use `wipefs -a`.
- nh2 8y agoThe spec actually says that software should prompt for confirmation and report that it performed changes (see elsewhere in this thread). `wipefs -a` is a good recommendation, but perhaps it can delete too much? In my case, I used `sgdisk --zap` to delete specifically the GPT bytes without deleting other data.
- creshal 8y agoThe mdadm manpage specifically mentions to use partitions with a special partition type, not whole disks, to prevent various 3rd party software tools from messing with it.
- self_awareness 8y agoThe amount of bugs in UEFI of new motherboards shocks me. Bugs are all over the place; sometimes I change an option in Setup only to find out it's disabled after reboot, and switching the option ON for the 2nd time works. Or, on my motherboard boot menu is bugged. To boot my Windows disk, I have to choose 'Switch to English' menu entry. I'm afraid to clear EFI vars because currently at least it works, who knows what'll happen when I clear EFI vars. This really discourages me from buying new hardware.
- simias 8y agoI feel your pain but it really doesn't shock me at all. Closed source blackbox firmware made mainly by hardware vendors? What could possibly go wrong. Even AMD and Intel are very hit-or-miss with the quality of the software they produce, can you imagine the random mobo maker? Have a look at some of the vendor-maintained forks of u-boot and Linux (whose code they have to publish because the GPL) and see how many bugs you can find. Meanwhile the software embedded in these boards becomes more and more complex with basically a full blown embedded OS, graphical interface etc... Software shops have a hard enough time writing robust software these days, ASRock probably has a team of a dozen or so glorified interns copy-pasting from stack overflow (or worse, contracting third parties do to it). I'm exaggerating a bit of course but not by a massive amount from my personal experience working with hardware shops.
- rwmj 8y agoAlmost certainly the code doing the GPT partition table recovery here is stock edk2 + a bunch of proprietary drivers to initialize clocks on the motherboard. The edk2 code is open source under a BSD-ish license, although I take your point that the full blob running on the ASRock motherboard doesn't come with compilable source so it's not much help. https://github.com/tianocore/edk2 https://github.com/tianocore/edk2
- nh2 8y agoAfter this incident, I am seriously considering buying a device with all-open-source firmware (like the Raptor Talos/Blackbird on HN recently) for work, just so that I don't have to guess what the mainboard is doing, but can actually look. And perhaps even other people having had a look what the firmware actually does, or fix the behaviour myself (like adding a "Do you really want to?" popup before it wipes the superblocks).
- sova 8y agoI appreciate the courtesy of this gentleman's language. It's not only cool that he pointed out something potentially product-defeating and all the while kept a kind and informative tone. I like it when people are nice on the interneb.
- Arie 8y agoI recently encountered a bug in an Asrock UEFI (CPU power limits ignored in the X299 ITX motherboard) and contacted Asrock about it. Got a fixed version emailed to me within a couple of days. Perhaps try that instead of the forums.
- drewg123 8y agoFor every good experience like that, there is an equal and opposite bad one. I have an X399 Taichi for my TR2. I'm using 2 older U.2 NVMe drives that have a legacy boot option rom that causes the board to hang 9/10 times at POST. The only option is to disable CSM, so it will only try to use the UEFI option rom. This works fine, EXCEPT that the CSM disable is lost every time the board looses power. Eg, that configuration setting is not properly saved to non-volatile storage. When I reported this to ASrock (via email), I was told in very broken english to "re install windows". This is great advice, considering I'm running FreeBSD, not to mention that the entire issue happens at POST. Sigh.
- vetinari 8y agoIt may be a problem with your specific piece. I'm also using X399 Taichi, it had flashed probably all released non-beta firmware versions except 3.00 and 3.10, with permanently disabled CSM and it had never lost that setting.
- drewg123 8y agoFWIW, I'm currently running 3.2, and that's all I've ever run. I think there have been a few BIOSes since then, but I have not tried them. This bug is annoying, but manageable (esp. since the box is on a UPS), so I don't want to go through the hassle of an upgrade..
- washadjeffmad 8y agoThere are other benefits to using FW 3.30[1] over 3.2 wrt hardware support and features actually working (ie- IOMMU, PCI reset, ACS, etc). [1] https://www.asrock.com/MB/AMD/X399%20Taichi/index.asp#BIOS https://www.asrock.com/MB/AMD/X399%20Taichi/index.asp#BIOS
- rwmj 8y agoI suspect what's actually happening here is there was an old GPT partition table on the disk which was not wiped before mdadm -C was run. This means it has a correct primary partition table (with correct CRC etc), and MD RAID has overwritten the secondary PT at the end of the disk. Unfortunately (and I genuinely hate to be "language lawyering" when someone has lost all their data), the ASRock firmware is doing the right thing according to the letter of the UEFI spec -- see section 5.3.2 of http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_7_A%20Sept%206.pdf http://www.uefi.org/sites/default/files/resources/UEFI%20Spe... : > If the primary GPT is invalid, the backup GPT is used instead and it is located on the last logical block on the disk. If the backup GPT is valid it must be used to restore the primary GPT. If the primary GPT is valid and the backup GPT is invalid software must restore the backup GPT. Good time to promote my videos visualizing how GPT and Linux RAID work :-) I did some cool live visualizations of mdadm, mkfs.ext4, gdisk, injecting errors etc. https://rwmj.wordpress.com/2018/11/26/nbdkit-fosdem-test-presentation/#content https://rwmj.wordpress.com/2018/11/26/nbdkit-fosdem-test-pre... https://rwmj.wordpress.com/2018/11/06/nbd-graphical-viewer-raid-5-edition/#content https://rwmj.wordpress.com/2018/11/06/nbd-graphical-viewer-r...
- leni536 8y agoSo UEFI requires a GPT to be present, doesn't it? Otherwise what's the objective difference between an invalid GPT and no GPT?
- creshal 8y agoA valid GPT header either at the beginning or end of the disk. Due to how the GPT header is structured, the secondary starts with the partition table and ends with the header signature in the last 8 bytes of the disk. So unlike with MBR disks, you can't just nuke the first few sectors and call it a day, you have to wipe the whole disk (or use a GPT-aware tool like wipefs).
- vbezhenar 8y agoCan I wipe first few sectors and last few sectors?
- nik736 8y agoWhy don't you contact them directly [0] instead of writing in some forum where no staff member will ever read it? They have a contact form on their website for technical support. 0: http://event.asrock.com/tsd.asp http://event.asrock.com/tsd.asp | http://forum.asrock.com/forum_posts.asp?TID=5265&title=howto-contact-tech-support http://forum.asrock.com/forum_posts.asp?TID=5265&title=howto...
- gabrielblack 8y agoExactly !
- nh2 8y agoGooglability for people who have the same problems.
- MrStonedOne 8y agoNon public issue trackers are a waste of time because they don't allow users having the same issue to congregate around discussion of that issue. If asrock wanted people to contact them about bugs like this they should create a public bug tracker, otherwise they deserve the delayed response time from not getting this via the preferred channels.
- Uhrheber 8y agoDon't get me started with ASRock. I'm definitely NOT buying anything from them again.
- nh2 8y agoWell, I originally got this ASRock mainboard because they seem to be much better on support. For example, they have BIOS updates still in 2018, while the vendor of my previous mainboard from the same generation stopped pushing updates in 2016.
- chaoticmass 8y agoI've been using my ASRock Z97 Extreme 6 for about two years now and it's been very reliable for me. I ended up building a few systems for friends and went with ASRock for those (one a Z77 for an older 3rd gen Intel, another two was for Ryzen builds) and those have been solidly reliable as well.
- Asooka 8y agoWhat made you decide that? I've personally also written them off after I bought a mobo from them with two non-functioning RAM slots a few years ago, but I'm curious what others have seen. So far the only ones I have not had problem with were Gigabyte, but I suspect they have their occasional issues as well.. Edit: Before you ask, yes the RAM was listed in the compatibility list. Same RAM, same CPU, same chipset on a Gigabyte mobo works just fine.
- kbumsik 8y ago> I use software RAID1 on Linux using mdadm using whole disk devices (no partitions). > It is important to know that a disk configured to be part of an mdadm RAID array can look like a broken EFI disk. I hate to say this, but it is obvious that a software RAID like mdadm cannot grab raw disks as a whole. If the disk doesn't have a valid GPT and EFI partition at the beginning how it is possible to boot from GRUB, load initramfs, and then load mdadm? I think something must be misconfigured at the first place.
- nh2 8y agoIt's a fully supported use case of mdadm RAID to use bare devices. These are data disks containing no operating system, so there's no booting from them and GRUB doesn't come into play.
- kbumsik 8y agoOh I read the context wrong. I thought he/she used the disks as a bootable disk as well. But do the disks have to have a proper GPT to be safe?
- adamzochowski 8y agoNo. Just like all hardware based raids you can set mdadm to use full raw disk. The GPT is only useful when you want to use part of the disk for RAID - partitioning disk into multiple parts where only some of it will be raided. Some people prefer opposite. First raid raw disks and then partition that md storage. The other reason to put GPT (or old style partition table) is so that no system or tool will mistakenly be helpful with detrcting "uninitiated drive" and then asking to perform quick format.
- blattimwind 8y agoThese disks almost certainly don't contain an operating system, so they don't need to be bootable.
- deleted 8y ago
- exabrial 8y agoDoes software raid do anything in the era of NVMe for performance? Seems like the bottleneck would be the PCIe bus.
- adamzochowski 8y agoWhat about recoverability features? Starting with Raid 1 mirroring and then going through to less storage wastefull like Raid 5? Or more resilient Raid 6?
- exabrial 8y agoApologies, I was specifically referring to performance
- adamzochowski 8y agoYou will still want stripping (raid 0) if you truly require more performance and more unified storage (single storage pool as opposed to 6 individual drives). Usually each NVMe drive has own pcie lanes, so having two drives raided gains performance over having one drive. AMD is now really pushing into having bonus lanes, so for threadripper they published reports on 6x raid0 of nvme ssds, where they got 21gigabytes/s reads out of storage. https://community.amd.com/community/gaming/blog/2017/10/02/now-available-free-nvme-raid-upgrade-for-amd-x399-chipset?sf118245427=1 https://community.amd.com/community/gaming/blog/2017/10/02/n... Please note, it is now important to check motherboard information regarding pci lanes availability, and ensure that between GPU and M2 there are plenty pcie lanes. Some early motherboards had 4 m2 slots, but only first 3 slots guarantees pcie lanes, while utilizing 4th would force sharing of pcie lanes and be slower than first 3.
- wtallis 8y agoThe PCIe bus can only be a bottleneck if you have a fan-out PCIe switch somewhere. If all your drives are directly attached to the CPU, then the bottleneck is usually the software overhead of the RAID. Sometimes with fast SSDs, the abstraction layer can add so much latency that your investment in fast low-latency drives is squandered.
- dadrian 8y agoI had this exact problem three years ago. My software RAID was over a raw device, not a partition, and when I got new hardware for a home file server that had been running for seven years, the new motherboard wiped the superblock. I was able to recover most of the data with some data recovery tool. Unfortunately, some of the files were research data where the filename was significant, and I lost all filenames and paths during recovery.