4 ms·

Hi, sorry for my tone - it's beyond rational discussion - however I still think it's not totally coming from nowhere: [RAID] >sigh From the message that you l

by nisa 11y ago

Hi, sorry for my tone - it's beyond rational discussion - however I still think it's not totally coming from nowhere: [RAID] >sigh From the message that you linked: The part below your quote is interesting. > General note about btrfs and btrfs raid. Given that btrfs itself remains a "stabilizing, but not yet fully mature and stable filesystem", while btrfs raid will often let you recover from a bad device, sometimes that recovery is in the form of letting you mount ro, so you can access the data and copy it elsewhere, before blowing away the filesystem and starting over. I have hotplug and online-replace with ZFS and mdraid. btrfs wants a special degraded remount that might or might not be able to recover to rw someday... replacing a disk online is not really possible. Handling a failing drive is not really implemented. Don't call it stable or don't call it RAID. It's nice that I'm able to pull of the data of my ro-mounted degraded array when a disk fails.. however all of this stuff is not really documented and was very surprising to me. If you have production machine e.g. on btrfs RAID10 it sucks to not be able to replace a broken drive on the fly online. ZFS/mdraid just do it.. [RAID II] > * The read scheduling algo is currently naive, using PID evenness to determine the disk in a pair to read, rather than current IO load on the disk. > Did I miss something, or misunderstand something? If I didn't miss anything, I don't see how you can get "RAID is utterly broken" from "RAID has less-than-optimal write and read scheduling". Walking is a less than than optimal way to move from Moscow to Berlin. If I take a train I expect a train - btrfs is a train that occasionally stops and says: Well, just walk... okay, you see I'm kind of emotional invested in this shit. However it's not what's usually expected as RAID and it's not documented. We have here a lot of applications that do exactly that: a single PID reads lots of stuff very often sequentially... so RAID1 doubles throughput. Well not on btrfs. [ENOSPC] > I don't run into those, and haven't for literal years. 4x4TB RAID10 (90% free) -> ENOSPC - this was somehwere between 3.16 or 3.19 - how can this even happen? [Corruption] > I haven't run into any, ever on the handful of systems on which I use BTRFS. Good for you. Was hit by this nasty undeletable directory bug: http://www.spinics.net/lists/linux-btrfs/msg35789.html http://www.spinics.net/lists/linux-btrfs/msg35789.html And then this: http://www.spinics.net/lists/linux-btrfs/msg45108.html http://www.spinics.net/lists/linux-btrfs/msg45108.html Umounting and running btrfs --check repair is not really cool if you want some uptime... [Deadlocks] Currently btrfs deadlocks reliable in a OOM situation... With 4.2 things got better, before running some medium heavy Hadoop job or ElastichSearch indexing killed 20% to 50% of the machines due to deadlocks (with 3.19 IIRC) > Look. I'm not saying that btrfs is feature-complete, suitable for every workload, or that any given person should use it for their workload. I'm just saying that the situation is far less dire than you're making it out to be. I've been using it as the rootfs for at least one of my daily use systems for the past 5.5 years, and haven't had any trouble out of it [3] in the past 2+ years. It's probably fine for personal use. But don't think about using it for something that is demanding. You can google the presentations that say: Use it! It's stable! Enterprise! It's not. Maybe in 5 years. I just saw it breaking in too many strange ways to bother anymore with it.

simoncion 11y ago
> Hi, sorry for my tone - it's beyond rational discussion... No worries. I don't give a shit about tone, just about content. Tone is next to impossible to accurately judge through text anyway, so I don't presume to understand someone's intent unless I have a pile of evidence. :-) > ...btrfs wants a special degraded remount that might or might not be able to recover to rw someday... replacing a disk online is not really possible. But, from the email you quoted: "...while btrfs raid will often let you recover from a bad device, sometimes that recovery is in the form of letting you mount ro, so you can access the data and copy it elsewhere, before blowing away the filesystem and starting over." And [0] indicates that you can live-replace a failed drive in a btrfs RAID array. (The man page makes it seem that you might want to also pass the -r flag to the replace operation.) I've never had the money around to amass the disks required to run my btrfs array in RAID1 data mode, so I can't test the claim of that Stack Overflow post, but I do use it in single data, RAID1 metadata and system mode. I've had a drive in that array suddenly drop out due to a firewire controller hiccup, and I suffered no data loss. (The volume did get forced into ro mode, so future writes were lost, but all applications knew that those writes were lost.) Additionally, I've been able to live-add and live-remove drives from my array with no hassle (other than the expected speed degradation due to shuffling the data around) whatsoever. > ...so RAID1 doubles [read] throughput. Well not on btrfs. Maybe I'm out of the loop, but I've never heard anyone say "use RAID1 to double your read throughput". That sounds like a very nice thing to have, but far from essential. Its absence certainly doesn't make btrfs's RAID1 implementation broken. :) > Umounting and running btrfs --check repair is not really cool if you want some uptime... Agreed, very much so. One thing bothers me about that guy who was reporting the bug on the 4.1 kernel. He claimed that his mount options were: /dev/sdb /media/storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0 but the issue he was reporting smelled like it could only happen on a rw volume. Wonder what was up with that. It's also interesting that noone replied to his messages. Both linux-btrfs and #btrfs have always been quite responsive when I've sent messages. > 4x4TB RAID10 (90% free) -> ENOSPC - this was somehwere between 3.16 or 3.19 - how can this even happen? Dunno. It looks like the Global Reserve got introduced in ~3.16, so it's not likely related to that. Maybe you hit a hard-link limit or had really, really large xattrs attached to files or something? (I do know that btrfs has (had?) a surprisingly low limit on the number of hard links supported in a volume, but I have no idea how hitting that limit presents itself.) Do you remember if you let the system idle for a few minutes and tried your operation again? (Not that that's a reliable [or even reasonable] solution, mind, but does provide a useful piece of diagnostic info.) In the distant past, running a balance on empty chunks (in order to free them and free up otherwise unused metadata space) was one recommended way to work around mysterious ENOSPC issues. This isn't supposed to be required anymore, but it's a last-ditch cargo-cult thing that can't do anything worse than waste one's time. > Good for you. Was hit by this nasty undeletable directory bug... That seriously sucks. However, I don't call that data corruption. Inconvenient, bordering on unacceptable? Yes. Data corruption? Nah. Your data is still present and correct, but you can't get rid of some directories. [1] > Currently btrfs deadlocks reliable in a OOM situation... With 4.2 things got better... Is it any OOM situation, or just particular ones? I know that I've had google-chrome + Hangouts run away and nom all my memory on multiple occasions, and had my btrfs volumes function just fine after the OOM killer had its way with Chrome. Also, do you have a handle on roughly how much better it has become in 4.2, and is this problem something that the btrfs folks are aware of? > [btrfs is] probably fine for personal use. But don't think about using it for something that is demanding. That's a far softer statement than "btrfs is a shitshow ... I will refrain from using it ever again if I have a choice.". :) Anyway, I do REALLY hope I soon get the budget to set up a btrfs testbed. I'd really like to see (and help diagnose) all these failures that people keep talking about... in a carefully controlled environment, with data that I don't care about -naturally-. :) [0] http://superuser.com/questions/685364/can-a-failed-btrfs-drive-in-raid-1-be-replaced-live http://superuser.com/questions/685364/can-a-failed-btrfs-dri... [1] And -for the record- I saw an eerily similar thing happen every other month at a Windows-only programming shop I used to work for. One or more files and/or directories on the NTFS-backed Windows 2003 file server would inevitably become undeletable. [2] The sysadmin would have to take the whole thing down for a half hour to smear chicken guts in the right places to get rid of the offending files/directories. Despite these regular issues, we never decided to stop using NTFS. ;) [2] And no, noone who was using the file server had admin access. Unprivved user access was all that was needed to create these undeletable files and directories.

by nisa 11y ago

simoncion 11y ago

> Hi, sorry for my tone - it's beyond rational discussion... No worries. I don't give a shit about tone, just about content. Tone is next to impossible to accurately judge through text anyway, so I don't presume to understand someone's intent unless I have a pile of evidence. :-) > ...btrfs wants a special degraded remount that might or might not be able to recover to rw someday... replacing a disk online is not really possible. But, from the email you quoted: "...while btrfs raid will often let you recover from a bad device, sometimes that recovery is in the form of letting you mount ro, so you can access the data and copy it elsewhere, before blowing away the filesystem and starting over." And [0] indicates that you can live-replace a failed drive in a btrfs RAID array. (The man page makes it seem that you might want to also pass the -r flag to the replace operation.) I've never had the money around to amass the disks required to run my btrfs array in RAID1 data mode, so I can't test the claim of that Stack Overflow post, but I do use it in single data, RAID1 metadata and system mode. I've had a drive in that array suddenly drop out due to a firewire controller hiccup, and I suffered no data loss. (The volume did get forced into ro mode, so future writes were lost, but all applications knew that those writes were lost.) Additionally, I've been able to live-add and live-remove drives from my array with no hassle (other than the expected speed degradation due to shuffling the data around) whatsoever. > ...so RAID1 doubles [read] throughput. Well not on btrfs. Maybe I'm out of the loop, but I've never heard anyone say "use RAID1 to double your read throughput". That sounds like a very nice thing to have, but far from essential. Its absence certainly doesn't make btrfs's RAID1 implementation broken. :) > Umounting and running btrfs --check repair is not really cool if you want some uptime... Agreed, very much so. One thing bothers me about that guy who was reporting the bug on the 4.1 kernel. He claimed that his mount options were: /dev/sdb /media/storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0 but the issue he was reporting smelled like it could only happen on a rw volume. Wonder what was up with that. It's also interesting that noone replied to his messages. Both linux-btrfs and #btrfs have always been quite responsive when I've sent messages. > 4x4TB RAID10 (90% free) -> ENOSPC - this was somehwere between 3.16 or 3.19 - how can this even happen? Dunno. It looks like the Global Reserve got introduced in ~3.16, so it's not likely related to that. Maybe you hit a hard-link limit or had really, really large xattrs attached to files or something? (I do know that btrfs has (had?) a surprisingly low limit on the number of hard links supported in a volume, but I have no idea how hitting that limit presents itself.) Do you remember if you let the system idle for a few minutes and tried your operation again? (Not that that's a reliable [or even reasonable] solution, mind, but does provide a useful piece of diagnostic info.) In the distant past, running a balance on empty chunks (in order to free them and free up otherwise unused metadata space) was one recommended way to work around mysterious ENOSPC issues. This isn't supposed to be required anymore, but it's a last-ditch cargo-cult thing that can't do anything worse than waste one's time. > Good for you. Was hit by this nasty undeletable directory bug... That seriously sucks. However, I don't call that data corruption. Inconvenient, bordering on unacceptable? Yes. Data corruption? Nah. Your data is still present and correct, but you can't get rid of some directories. [1] > Currently btrfs deadlocks reliable in a OOM situation... With 4.2 things got better... Is it any OOM situation, or just particular ones? I know that I've had google-chrome + Hangouts run away and nom all my memory on multiple occasions, and had my btrfs volumes function just fine after the OOM killer had its way with Chrome. Also, do you have a handle on roughly how much better it has become in 4.2, and is this problem something that the btrfs folks are aware of? > [btrfs is] probably fine for personal use. But don't think about using it for something that is demanding. That's a far softer statement than "btrfs is a shitshow ... I will refrain from using it ever again if I have a choice.". :) Anyway, I do REALLY hope I soon get the budget to set up a btrfs testbed. I'd really like to see (and help diagnose) all these failures that people keep talking about... in a carefully controlled environment, with data that I don't care about -naturally-. :) [0] http://superuser.com/questions/685364/can-a-failed-btrfs-drive-in-raid-1-be-replaced-live http://superuser.com/questions/685364/can-a-failed-btrfs-dri... [1] And -for the record- I saw an eerily similar thing happen every other month at a Windows-only programming shop I used to work for. One or more files and/or directories on the NTFS-backed Windows 2003 file server would inevitably become undeletable. [2] The sysadmin would have to take the whole thing down for a half hour to smear chicken guts in the right places to get rid of the offending files/directories. Despite these regular issues, we never decided to stop using NTFS. ;) [2] And no, noone who was using the file server had admin access. Unprivved user access was all that was needed to create these undeletable files and directories.