5 ms·

FUSE is overridden more and more with the introduction of gfapi. Qemu and Samba can now both "talk" it, so you don't need FUSE for that any more, for example.

FUSE is overridden more and more with the introduction of gfapi. Qemu and Samba can now both "talk" it, so you don't need FUSE for that any more, for example. It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well. The performance hit is disputable, there aren't really better alternatives. Most distributed filesystems out there are using FUSE too (XtreemFS, MooseFS) and I am not touching Lustre. The Ceph distributed filesystem has an in-kernel client which is upstream. XtreemFS and MooseFS are designed for a different use-case (WAN filesystem) and aren't really directly comparable. Lustre runs the majority of the world's supercomputers. It seems a little unfair to dismiss them without at least giving a reason why. They have made some progress towards getting their kernel client upstream recently. I was a Ceph developer for a while. I read about Red Hat's recent acquisition of Inktank with interest. (Inktank was founded by Sage Weil, the creator of Ceph, to commercialize the filesystem.) Since Red Hat previously acquired the main company behind Gluster, this makes things a little-- how shall we say?-- interesting. It's unclear to me whether Red Hat will want to support two distributed filesystems going forward, or whether they will try to streamline things.

HerrMonnezza 12y ago
> Lustre runs the majority of the world's supercomputers. It seems a > little unfair to dismiss them without at least giving a reason > why. They have made some progress towards getting their kernel > client upstream recently. Lustre aims at performance and thus everything is in-kernel, which makes operations a nightmare: basically everything you have is an opaque filesystem structure on disk, and a bunch of logs which can be very difficult to read. And you have to run the kernel with Lustre patches, which only apply cleanly to RHEL and SLES; work is being done on getting Debian in, but still no luck with Ubuntu or recent mainstream kernels. Also, there is no clear upgrade path between different versions: the recommended procedure is to dump everything to another filesystem and then reload after the upgrade. There is no replication handled at the FS level. When running Lustre on top of ext4, you have to use the Lustre's own versions of the ext4 utilities, which replace the ext4 utilities in the system... In short: Lustre sacrified a lot of ops-friendliness for the sake of high-performance. This is quite the opposite choice that GlusterFS has made. (Edit: formatting)
notacoward 12y ago
"It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well." We (I'm a GlusterFS developer) actually did this for a while once, and people have recently started talking about doing it again. With GFAPI it shouldn't even be that complicated, but there are still problems e.g. around fork(2). "The Ceph distributed filesystem has an in-kernel client which is upstream." ...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Development velocity counts. The fact that it uses FUSE has rarely been the cause of performance problems in GlusterFS. More often than not, the real culprit is (relative) lack of caching, which is fixable in user space. "XtreemFS and MooseFS are designed for a different use-case (WAN filesystem)" Pretty true for XtreemFS - for which I have the utmost respect and which I promote at every opportunity - but MooseFS targets exactly the same use case as GlusterFS. OK, a subset, because they don't have all the features and integrations we do. ;) I'd also add PVFS/OrangeFS, which is contemporaneous with Lustre. It doesn't use FUSE, but has its own user-space "interceptor" which is equivalent. "Lustre runs the majority of the world's supercomputers." It runs a lot of the world's biggest supercomputers, because those people can afford to keep full-time Lustre developers on staff to baby-sit it. Not ops staff - developers to apply the latest patches, add their own, etc. I spent over two years at SiCortex trying to make Lustre usable for our customers. At that time, I believe LLNL had four Lustre developes. ORNL had slightly less. Cray, DDN, etc. each had their own as well. When it works, Lustre can be great. On the other hand, few users can afford to devote that level of staff to running a distributed file system. Those that can't will find themselves in the weeds with MDS meltdowns and "@@@ processing error" messages all the time. Because of this, I'd say it's Lustre that's not really targeting the same market or use case as GlusterFS. We never encounter them head to head, in either the corporate or community context. The "performance at any cost" market is a hard place to make a living, and it barely overlaps at all with the "performance plus usability" market. "It's unclear to me whether Red Hat will want to support two distributed filesystems going forward" Why not? They've supported multiple local file systems for a long time, and there's an even bigger overlap there. When you look at both Ceph and GlusterFS in terms of distributed data systems rather than just file systems specifically, maybe things will look a bit different. Now we're talking about block and object as well as files. Maybe we're talking about integrating distributed storage with distributed computation in ways not covered by any of those interfaces. We're certainly talking about users having their own preferences in each of these areas. If there are enough APIs, and enough users who prefer one over the other for a certain kind of data or vertical market, then it makes quite a bit of sense to continue maintaining two separate systems. On the other hand, of course we want to reduce the number of components we'll have to maintain, and there will be plenty of technology sharing back and forth. Only time will tell which way the balance shifts.

HerrMonnezza 12y ago

> Lustre runs the majority of the world's supercomputers. It seems a > little unfair to dismiss them without at least giving a reason > why. They have made some progress towards getting their kernel > client upstream recently. Lustre aims at performance and thus everything is in-kernel, which makes operations a nightmare: basically everything you have is an opaque filesystem structure on disk, and a bunch of logs which can be very difficult to read. And you have to run the kernel with Lustre patches, which only apply cleanly to RHEL and SLES; work is being done on getting Debian in, but still no luck with Ubuntu or recent mainstream kernels. Also, there is no clear upgrade path between different versions: the recommended procedure is to dump everything to another filesystem and then reload after the upgrade. There is no replication handled at the FS level. When running Lustre on top of ext4, you have to use the Lustre's own versions of the ext4 utilities, which replace the ext4 utilities in the system... In short: Lustre sacrified a lot of ops-friendliness for the sake of high-performance. This is quite the opposite choice that GlusterFS has made. (Edit: formatting)

notacoward 12y ago

"It should be possible to avoid the userspace to kernelspace transition in the client by using LD_PRELOAD with a shim library as well." We (I'm a GlusterFS developer) actually did this for a while once, and people have recently started talking about doing it again. With GFAPI it shouldn't even be that complicated, but there are still problems e.g. around fork(2). "The Ceph distributed filesystem has an in-kernel client which is upstream." ...and it's probably not a coincidence that the file system component is the one piece of Ceph that still hasn't reached production readiness. Development velocity counts. The fact that it uses FUSE has rarely been the cause of performance problems in GlusterFS. More often than not, the real culprit is (relative) lack of caching, which is fixable in user space. "XtreemFS and MooseFS are designed for a different use-case (WAN filesystem)" Pretty true for XtreemFS - for which I have the utmost respect and which I promote at every opportunity - but MooseFS targets exactly the same use case as GlusterFS. OK, a subset, because they don't have all the features and integrations we do. ;) I'd also add PVFS/OrangeFS, which is contemporaneous with Lustre. It doesn't use FUSE, but has its own user-space "interceptor" which is equivalent. "Lustre runs the majority of the world's supercomputers." It runs a lot of the world's biggest supercomputers, because those people can afford to keep full-time Lustre developers on staff to baby-sit it. Not ops staff - developers to apply the latest patches, add their own, etc. I spent over two years at SiCortex trying to make Lustre usable for our customers. At that time, I believe LLNL had four Lustre developes. ORNL had slightly less. Cray, DDN, etc. each had their own as well. When it works, Lustre can be great. On the other hand, few users can afford to devote that level of staff to running a distributed file system. Those that can't will find themselves in the weeds with MDS meltdowns and "@@@ processing error" messages all the time. Because of this, I'd say it's Lustre that's not really targeting the same market or use case as GlusterFS. We never encounter them head to head, in either the corporate or community context. The "performance at any cost" market is a hard place to make a living, and it barely overlaps at all with the "performance plus usability" market. "It's unclear to me whether Red Hat will want to support two distributed filesystems going forward" Why not? They've supported multiple local file systems for a long time, and there's an even bigger overlap there. When you look at both Ceph and GlusterFS in terms of distributed data systems rather than just file systems specifically, maybe things will look a bit different. Now we're talking about block and object as well as files. Maybe we're talking about integrating distributed storage with distributed computation in ways not covered by any of those interfaces. We're certainly talking about users having their own preferences in each of these areas. If there are enough APIs, and enough users who prefer one over the other for a certain kind of data or vertical market, then it makes quite a bit of sense to continue maintaining two separate systems. On the other hand, of course we want to reduce the number of components we'll have to maintain, and there will be plenty of technology sharing back and forth. Only time will tell which way the balance shifts.