> And [0] indicates that you can live-replace a failed drive in a btrfs RAID array. (The man page makes it seem that you might want to also pass the -r flag to

> And [0] indicates that you can live-replace a failed drive in a btrfs RAID array. (The man page makes it seem that you might want to also pass the -r flag to the replace operation.) Good catch! I have to test on a rented server I'm running anyway in the next days - adding a an additional drive, running replace and removing the old drive seems to work - I'm about to find out if - deleting a device, rebooting and trying to replace an non-existent device also works - I guess it's likely that I have to fumble with the provider-rescue system (so now control about the kernel version :/) I thought that the -o degraded dance was still necassary, looks like I was wrong. Let's see how the disk replacement on that servers turns out... > One thing bothers me about that guy who was reporting the bug on the 4.1 kernel. He claimed that his mount options were: /dev/sdb /media/storage2 btrfs ro,noatime,compress=lzo,space_cache 0 0 but the issue he was reporting smelled like it could only happen on a rw volume. Wonder what was up with that. That guy was me :) Basically the disk got remounted ro after the failure and I just did a grep on /proc/mounts for the fs before posting (not very wise). This has been a bug that was fixed with 4.2 (didn't saw it again at least) and btrfs check --repair fixed it. > It's also interesting that noone replied to his messages. Both linux-btrfs and #btrfs have always been quite responsive when I've sent messages. I had no time to hang around IRC at this time. Yes. Usually it's very nice there. I also want to say I'm not ranting against any of the people involved. I'm just unhappy with the current state of the technology. [ENOSPC] > Dunno. It looks like the Global Reserve got introduced in ~3.16, so it's not likely related to that. Maybe you hit a hard-link limit or had really, really large xattrs attached to files or something? (I do know that btrfs has (had?) a surprisingly low limit on the number of hard links supported in a volume, but I have no idea how hitting that limit presents itself.) Actually I don't know - the filesystem in question ran a while with the 3.13 kernel and got later upgraded to something less old. Maybe that's part of the problem. ENOSPC happened while a misconfigured Hadoop NameNode wrote tons of data 4x without checkpointing on the disk and when restarting the checkpointing server it all crashed. It's now running 4.2 and fine so far... > In the distant past, running a balance on empty chunks (in order to free them and free up otherwise unused metadata space) was one recommended way to work around mysterious ENOSPC issues. This isn't supposed to be required anymore, but it's a last-ditch cargo-cult thing that can't do anything worse than waste one's time. Yeah did that balance dance and it somewhat worked. > Is it any OOM situation, or just particular ones? I know that I've had google-chrome + Hangouts run away and nom all my memory on multiple occasions, and had my btrfs volumes function just fine after the OOM killer had its way with Chrome. Never got around to nail down this one (would love to being able to file a bug on this) - but at the time I was not really able to do much on the machines - it's a NUMA system and basically ElasticSearch and Hadoop allocated far more than 64GB memory - OOM kicked in and _after a while_ the logs where full of hanging tasks related to btrfs - at that time I've found something related on the mailing list in regards to allocations and OOM however I'm lacking the intimate knowledge to be sure if it's related. I've screamed at everyone to avoid OOM situations so I'm not sure if it's still a problem :) > Also, do you have a handle on roughly how much better it has become in 4.2, and is this problem something that the btrfs folks are aware of? So far I did not see it again. I had not enough data to file a bug or investigate further. It was something lock related IIRC. > Anyway, I do REALLY hope I soon get the budget to set up a btrfs testbed. I'd really like to see (and help diagnose) all these failures that people keep talking about... in a carefully controlled environment, with data that I don't care about -naturally-. :) Good luck! You can try running Hadoop and some heavy jobs on it. Lot's of threads that read and write lot's of data. There are some benchmarks like https://github.com/intel-hadoop/HiBench https://github.com/intel-hadoop/HiBench to stress the system. Should also work on a single system. > [1] And -for the record- I saw an eerily similar thing happen every other month at a Windows-only programming shop I used to work for. One or more files and/or directories on the NTFS-backed Windows 2003 file server would inevitably become undeletable. [2] The sysadmin would have to take the whole thing down for a half hour to smear chicken guts in the right places to get rid of the offending files/directories. Despite these regular issues, we never decided to stop using NTFS. ;) Yes. There is worse stuff out there. I was using ZFS before with lots of really fucked up disks in a kind of "don't do this, you are stupid!" setup and ZFS just did it's job and never complained, same Hadoop workload + disks with thousands of bad sectors. Maybe it's unfair to compare btrfs to this, however it was a ride full of disillusionment to think btrfs compares to ZFS. Maybe it's getting there and I was in at a bad time... I'm not so sure through.