4 ms·

Ceph and XFS actually for the longest time had an issue locking up the kernel. Ceph works fine, until it doesn't. I used gluster and Ceph for a while in product

Ceph and XFS actually for the longest time had an issue locking up the kernel. Ceph works fine, until it doesn't. I used gluster and Ceph for a while in production. I faintly remember Gluster supposedly being more performant for small writes, but it was an absolutely pain to deal with. Ceph is a bit better and ran stable until it didn't. One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes. But rook has had so many issues, many of them fixed, but somehow it seems that all kubernetes "cloudnative" released software has some quirks which you just have to accept as part of the solution(this is from someone who's oftentimes pushing for kubernetes based solutions). But I also tend to recommend clients to just spend 15k to buy a TrueNAS or something similar with iSCSI and NFS. EDIT: the XFS issue was a hung task issue that was added to the 5.6 kernel "recently". Meaning it not in Ubuntu 20.04 either. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d19f1c8e1937baf74e1962aae9f90fa3aeab463 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

ps 6y ago
Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k... “One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes.“ I believe the recommended way is using cephadm. I have been using ceph-ansible since luminous and had to deal with nasty issues or quirks as well caused probably by complexity and human errors.
- sekh60 6y ago
  You are correct about cephadm being the recommended install method. For NVMe OSDs how many daemons are using the same drive? Normally to get performance you have to colocate two or four osd daemons on a single NVMe device. Also ceph benefits the most from massive parallelism, how many clients do you have? 4 OSDd is not a lot to spread the load over, even if the drives are lightning fast. There is a new io pathway in the works called crimson which should make NVMe drives more performant. I was hoping a preview would lend in Pacific, but I guess it isn't close enough to being ready.
  - rjzzleep 6y ago
    Cephadm is very new. So yes it may be the recommended solution NOW. Precisely because all previous solutions were very fragile. I don’t have any experience with it for these reasons. Somehow Linux new filesystems take very long from initial release to reliable use. As seen also with btrfs. Maybe filesystems are just hard I don’t know but somehow the solutions seem very Hacky.
yjftsjthsd-h 6y ago
Gluster is also so much easier to set up IMO; the only reason I wouldn't use it is that it seems to have some consistency issues such that ex. you could run postgres on ceph, but not (safely) on gluster. But I don't understand it well enough to know why.
marcan_42 6y ago
XFS has a history of reclaim/memory management issues. It used to have a habit of blocking on I/O when cleaning memory even with plenty of clean page cache available. For the longest time, I had issues on my home server (XFS on top of RAID at the time) with large latency hiccups, to the tune of several seconds. This was disrupting some real-time data ingestion and causing data loss. I thought it was about committing to disk, so I added big memory buffers, but no dice. I spent years with this annoying issue. I even had a kernel patch in to increase kernel-side buffers (which were not subject to this problem) to work around it. It wasn't even consistent. Eventually one day I got sick enough of it, and sat down trying to reproduce it. I figured out that it only happened when true memory usage (including buffer cache) was ~100%; if there was truly unused memory around, things were fine, and I could evict the buffer cache and it would fix the problem until it grew to consume all free RAM again. Eventually I managed to get a stack trace of a process that was stalling even though it wasn't anywhere near writing to disk, and I found out it was stalling during a write(). To a pipe. Because the kernel had to allocate data for the pipe buffers. And it was asking XFS. And XFS decided to evict some dirty inodes, and block on that. Even with gigabytes of clean page cache available to evict. What. Swearing ensued - here I thought I had some weird kernel/hardware issue causing latency spikes, and it was XFS all along. I eventually ripped XFS out and replaced it with ext4 and that solved the issue. This eventually got fixed in 2019: https://lwn.net/Articles/795098/ https://lwn.net/Articles/795098/
- adrian_b 6y ago
  I do not doubt that you had those problems with XFS, but I believe that they must have required certain specific conditions to manifest themselves. I have been using XFS intensively for almost 20 years and I have never encountered any similar problems, so there must have been some differences in my setup, so that this behavior was not triggered. While I do not know what was different in my case, possible differences might have been caused by the fact that I have always used generous quantities of DRAM in all my computers and I have never used swap in any of them.

ps 6y ago

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k... “One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes.“ I believe the recommended way is using cephadm. I have been using ceph-ansible since luminous and had to deal with nasty issues or quirks as well caused probably by complexity and human errors.

sekh60 6y ago
You are correct about cephadm being the recommended install method. For NVMe OSDs how many daemons are using the same drive? Normally to get performance you have to colocate two or four osd daemons on a single NVMe device. Also ceph benefits the most from massive parallelism, how many clients do you have? 4 OSDd is not a lot to spread the load over, even if the drives are lightning fast. There is a new io pathway in the works called crimson which should make NVMe drives more performant. I was hoping a preview would lend in Pacific, but I guess it isn't close enough to being ready.
- rjzzleep 6y ago
  Cephadm is very new. So yes it may be the recommended solution NOW. Precisely because all previous solutions were very fragile. I don’t have any experience with it for these reasons. Somehow Linux new filesystems take very long from initial release to reliable use. As seen also with btrfs. Maybe filesystems are just hard I don’t know but somehow the solutions seem very Hacky.

yjftsjthsd-h 6y ago

Gluster is also so much easier to set up IMO; the only reason I wouldn't use it is that it seems to have some consistency issues such that ex. you could run postgres on ceph, but not (safely) on gluster. But I don't understand it well enough to know why.

marcan_42 6y ago

XFS has a history of reclaim/memory management issues. It used to have a habit of blocking on I/O when cleaning memory even with plenty of clean page cache available. For the longest time, I had issues on my home server (XFS on top of RAID at the time) with large latency hiccups, to the tune of several seconds. This was disrupting some real-time data ingestion and causing data loss. I thought it was about committing to disk, so I added big memory buffers, but no dice. I spent years with this annoying issue. I even had a kernel patch in to increase kernel-side buffers (which were not subject to this problem) to work around it. It wasn't even consistent. Eventually one day I got sick enough of it, and sat down trying to reproduce it. I figured out that it only happened when true memory usage (including buffer cache) was ~100%; if there was truly unused memory around, things were fine, and I could evict the buffer cache and it would fix the problem until it grew to consume all free RAM again. Eventually I managed to get a stack trace of a process that was stalling even though it wasn't anywhere near writing to disk, and I found out it was stalling during a write(). To a pipe. Because the kernel had to allocate data for the pipe buffers. And it was asking XFS. And XFS decided to evict some dirty inodes, and block on that. Even with gigabytes of clean page cache available to evict. What. Swearing ensued - here I thought I had some weird kernel/hardware issue causing latency spikes, and it was XFS all along. I eventually ripped XFS out and replaced it with ext4 and that solved the issue. This eventually got fixed in 2019: https://lwn.net/Articles/795098/ https://lwn.net/Articles/795098/

adrian_b 6y ago
I do not doubt that you had those problems with XFS, but I believe that they must have required certain specific conditions to manifest themselves. I have been using XFS intensively for almost 20 years and I have never encountered any similar problems, so there must have been some differences in my setup, so that this behavior was not triggered. While I do not know what was different in my case, possible differences might have been caused by the fact that I have always used generous quantities of DRAM in all my computers and I have never used swap in any of them.