3 ms·

I'm curious what your experience was with the LD_PRELOAD thing. By "problems with fork," you mean the possibility of forking without that environment variable

I'm curious what your experience was with the LD_PRELOAD thing. By "problems with fork," you mean the possibility of forking without that environment variable set via exevpe or similar? MooseFS targets exactly the same use case as GlusterFS. Yeah, actually you are quite right. I was getting MooseFS confused with a different filesystem. MooseFS looks like it has a GFS heritage. Kind of like what I am working on now... HDFS. ...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Ceph's filesystem component is reasonably stable. It's the multi-MDS use case which still had problems (at least a few years ago when I was working on the project.) The challenge was coordinating multiple metadata servers to do dynamic subtree partitioning and other distributed algorithms. Ceph has a FUSE client which you can use if you don't want to install kernel modules, and a library API. It seems that Lustre has an in-kernel server. This might have led to some of the maintenance difficulties people seem to have with it. (I never worked on Lustre myself.) I don't think this discredits the idea of in-kernel clients, which are different beasts entirely. Especially when the client is in the mainline kernel, rather than an out-of-tree patch like with Lustre in the past. It's a tough market out there for clustered filesystems. HPC is shrinking due to government sequesters and budget freezes. The growth area seems to be Hadoop, but most users prefer to just run HDFS, since it has a proven track record and is integrated well into the rest of the project. Moving into the object store world is one option, but that is also a very crowded space. It will be interesting to see how things unfold.

notacoward 12y ago
> By "problems with fork," you mean the possibility of > forking without that environment variable set via exevpe > or similar? That might be a problem, but it's not the one I was thinking of. The problem with LD_PRELOAD hacks is that they need to maintain some state about the special fds on the special file system. That state immediately becomes stale when you fork (because copy on write) and gets blown away when you exec. Therefore you always end up having to store the state in shared memory (nowadays probably an mmap'ed file) with extra grot to manage that. Even then, exec and dlopen won't work on things that aren't real files. So it's not impossible, but it gets pretty tedious (especially when you have to re-do all this for the 50+ calls you need to intercept) and there are always some awkward limitations. With regard to in-kernel clients, I'm not trying to discredit them, but they're not the only viable alternative. User-space clients have their place too, as I'm sure you know if you work on HDFS. Every criticism of FUSE clients goes double for JVM clients, but both can actually work and perform well. It seemed like some people were dissing GlusterFS because of something they'd heard from someone else, who'd heard something from someone else, based on versions of FUSE before 2.8 (when reverse invalidations and a lot of performance improvements went in). This being HN, it seemed like they were just repeating FUD uncritically, so I sought to correct it. The fact that GlusterFS uses FUSE is simply not a big issue affecting its performance.

notacoward 12y ago

> By "problems with fork," you mean the possibility of > forking without that environment variable set via exevpe > or similar? That might be a problem, but it's not the one I was thinking of. The problem with LD_PRELOAD hacks is that they need to maintain some state about the special fds on the special file system. That state immediately becomes stale when you fork (because copy on write) and gets blown away when you exec. Therefore you always end up having to store the state in shared memory (nowadays probably an mmap'ed file) with extra grot to manage that. Even then, exec and dlopen won't work on things that aren't real files. So it's not impossible, but it gets pretty tedious (especially when you have to re-do all this for the 50+ calls you need to intercept) and there are always some awkward limitations. With regard to in-kernel clients, I'm not trying to discredit them, but they're not the only viable alternative. User-space clients have their place too, as I'm sure you know if you work on HDFS. Every criticism of FUSE clients goes double for JVM clients, but both can actually work and perform well. It seemed like some people were dissing GlusterFS because of something they'd heard from someone else, who'd heard something from someone else, based on versions of FUSE before 2.8 (when reverse invalidations and a lot of performance improvements went in). This being HN, it seemed like they were just repeating FUD uncritically, so I sought to correct it. The fact that GlusterFS uses FUSE is simply not a big issue affecting its performance.