3 ms·
"It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well." We (I'm a GlusterFS dev
by notacoward 13y ago
"It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well."
We (I'm a GlusterFS developer) actually did this for a while once, and people have recently started talking about doing it again. With GFAPI it shouldn't even be that complicated, but there are still problems e.g. around fork(2).
"The Ceph distributed filesystem has an in-kernel client which is upstream."
...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Development velocity counts. The fact that it uses FUSE has rarely been the cause of performance problems in GlusterFS. More often than not, the real culprit is (relative) lack of caching, which is fixable in user space.
"XtreemFS and MooseFS are designed for a different use-case (WAN filesystem)"
Pretty true for XtreemFS - for which I have the utmost respect and which I promote at every opportunity - but MooseFS targets exactly the same use case as GlusterFS. OK, a subset, because they don't have all the features and integrations we do. ;) I'd also add PVFS/OrangeFS, which is contemporaneous with Lustre. It doesn't use FUSE, but has its own user-space "interceptor" which is equivalent.
"Lustre runs the majority of the world's supercomputers."
It runs a lot of the world's biggest supercomputers, because those people can afford to keep full-time Lustre developers on staff to baby-sit it. Not ops staff - developers to apply the latest patches, add their own, etc. I spent over two years at SiCortex trying to make Lustre usable for our customers. At that time, I believe LLNL had four Lustre developes. ORNL had slightly less. Cray, DDN, etc. each had their own as well. When it works, Lustre can be great. On the other hand, few users can afford to devote that level of staff to running a distributed file system. Those that can't will find themselves in the weeds with MDS meltdowns and "@@@ processing error" messages all the time.
Because of this, I'd say it's Lustre that's not really targeting the same market or use case as GlusterFS. We never encounter them head to head, in either the corporate or community context. The "performance at any cost" market is a hard place to make a living, and it barely overlaps at all with the "performance plus usability" market.
"It's unclear to me whether Red Hat will want to support two distributed filesystems going forward"
Why not? They've supported multiple local file systems for a long time, and there's an even bigger overlap there. When you look at both Ceph and GlusterFS in terms of distributed data systems rather than just file systems specifically, maybe things will look a bit different. Now we're talking about block and object as well as files. Maybe we're talking about integrating distributed storage with distributed computation in ways not covered by any of those interfaces. We're certainly talking about users having their own preferences in each of these areas. If there are enough APIs, and enough users who prefer one over the other for a certain kind of data or vertical market, then it makes quite a bit of sense to continue maintaining two separate systems. On the other hand, of course we want to reduce the number of components we'll have to maintain, and there will be plenty of technology sharing back and forth. Only time will tell which way the balance shifts.
- colin_mccabe 13y agoI'm curious what your experience was with the LD_PRELOAD thing. By "problems with fork," you mean the possibility of forking without that environment variable set via exevpe or similar? MooseFS targets exactly the same use case as GlusterFS. Yeah, actually you are quite right. I was getting MooseFS confused with a different filesystem. MooseFS looks like it has a GFS heritage. Kind of like what I am working on now... HDFS. ...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Ceph's filesystem component is reasonably stable. It's the multi-MDS use case which still had problems (at least a few years ago when I was working on the project.) The challenge was coordinating multiple metadata servers to do dynamic subtree partitioning and other distributed algorithms. Ceph has a FUSE client which you can use if you don't want to install kernel modules, and a library API. It seems that Lustre has an in-kernel server. This might have led to some of the maintenance difficulties people seem to have with it. (I never worked on Lustre myself.) I don't think this discredits the idea of in-kernel clients, which are different beasts entirely. Especially when the client is in the mainline kernel, rather than an out-of-tree patch like with Lustre in the past. It's a tough market out there for clustered filesystems. HPC is shrinking due to government sequesters and budget freezes. The growth area seems to be Hadoop, but most users prefer to just run HDFS, since it has a proven track record and is integrated well into the rest of the project. Moving into the object store world is one option, but that is also a very crowded space. It will be interesting to see how things unfold.
- notacoward 13y ago> By "problems with fork," you mean the possibility of > forking without that environment variable set via exevpe > or similar? That might be a problem, but it's not the one I was thinking of. The problem with LD_PRELOAD hacks is that they need to maintain some state about the special fds on the special file system. That state immediately becomes stale when you fork (because copy on write) and gets blown away when you exec. Therefore you always end up having to store the state in shared memory (nowadays probably an mmap'ed file) with extra grot to manage that. Even then, exec and dlopen won't work on things that aren't real files. So it's not impossible, but it gets pretty tedious (especially when you have to re-do all this for the 50+ calls you need to intercept) and there are always some awkward limitations. With regard to in-kernel clients, I'm not trying to discredit them, but they're not the only viable alternative. User-space clients have their place too, as I'm sure you know if you work on HDFS. Every criticism of FUSE clients goes double for JVM clients, but both can actually work and perform well. It seemed like some people were dissing GlusterFS because of something they'd heard from someone else, who'd heard something from someone else, based on versions of FUSE before 2.8 (when reverse invalidations and a lot of performance improvements went in). This being HN, it seemed like they were just repeating FUD uncritically, so I sought to correct it. The fact that GlusterFS uses FUSE is simply not a big issue affecting its performance.