5 ms·

I see that the GlusterFS FAQ says it is fully POSIX compliant. That's a pretty good trick. Ten years ago or so I had a suite of compliance tests I would use t

I see that the GlusterFS FAQ says it is fully POSIX compliant. That's a pretty good trick. Ten years ago or so I had a suite of compliance tests I would use to embarrass salesmen from iBrix and Panasas. The only actually POSIX-compliant distributed filesystem I could find in those days was Lustre (unrelated to Gluster, despite the naming). Lustre works well but it almost impossible to install and operate.

kbenson 11y ago
I remember at around the same time I was playing with GlusterFS, and it was interesting, but painful to configure as well. It was prone to have client/server configuration mismatches, and you had to be careful to reason about where you enabled a feature, the client or server, to make sure it failed in a safe way. It did have capabilities for Fcntl and flock file locking though, which was interesting at the time. Unfortunately ist was also somewhat unstable, and I would see segfaults every few weeks. My focus for work projects moved into different areas, and I didn't keep up with it's development, so I'm not sure what they've done in the last decade, but it was promising and refreshingly new back then. I should take another look.
batbomb 11y ago
In HPC/HTC, Lustre is very common, especially in DOE labs. I've never tried to install it, but I don't think my colleagues who have are some kind of special genius.
- jpgvm 11y ago
  Most HPC labs run distributions that go to quite a lot of pain to ensure Lustre works well. The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues. That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install. Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park. Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.
  - batbomb 11y ago
    Well most HPC installation I know are just running RHEL6/7 (except for NERSC which is using Cray linux based on SUSE still I think?) which I wouldn't peg as particularly exotic. The HA thing is know, typically Lustre is just a piece of the file system, along with local disks and GPFS (and now towards Ceph), and HPSS/tape (which can be a tier of GPFS). I'm not so sure the hope for "one filesystem to rule them all" will ever really work out, but Ceph is the best positioned.
fh973 11y ago
I invite you to check out our product, Quobyte (www.quobyte.com). Not open-source, but a parallel high-performance POSIX file system, with split-brain safe quorum replication of all components, can also do erasure coding for files, with policy-based data placement, running on standard server hardware. We designed it to yield top performance both for (parallel) file system workloads and block storage workloads. So you can run VMs and databases on it with a performance better than any other partition-tolerant software storage system. The goal is to provide customers with a scalable automated general storage platform for all workloads, a la Google, but for real world applications.

kbenson 11y ago

I remember at around the same time I was playing with GlusterFS, and it was interesting, but painful to configure as well. It was prone to have client/server configuration mismatches, and you had to be careful to reason about where you enabled a feature, the client or server, to make sure it failed in a safe way. It did have capabilities for Fcntl and flock file locking though, which was interesting at the time. Unfortunately ist was also somewhat unstable, and I would see segfaults every few weeks. My focus for work projects moved into different areas, and I didn't keep up with it's development, so I'm not sure what they've done in the last decade, but it was promising and refreshingly new back then. I should take another look.

batbomb 11y ago

In HPC/HTC, Lustre is very common, especially in DOE labs. I've never tried to install it, but I don't think my colleagues who have are some kind of special genius.

jpgvm 11y ago
Most HPC labs run distributions that go to quite a lot of pain to ensure Lustre works well. The issue with Lustre is that it usually requires kernel patches if you are not running the aforementioned distributions that are designed for HPC. Specifically the Lustre OSD used a modified version of ext2 among other issues. That could be getting better now with the new ZFS based OSD though I wouldn't put money on it being "easy" to install. Lustre also doesn't provide any means of replication. If you want to achieve HA with Lustre you need to make each OSD individually HA. This can be done with multi-pathed SAS arrays and a ton of scripting but it's still not exactly a walk in the park. Hopefully one day we will see a real high performance distributed filesystem that also bundles replication, tiering and some semblance of POSIX compatiblity. I doubt Ceph is going to be it so we are probably still 5-10 years from a solution to the problem.
- batbomb 11y ago
  Well most HPC installation I know are just running RHEL6/7 (except for NERSC which is using Cray linux based on SUSE still I think?) which I wouldn't peg as particularly exotic. The HA thing is know, typically Lustre is just a piece of the file system, along with local disks and GPFS (and now towards Ceph), and HPSS/tape (which can be a tier of GPFS). I'm not so sure the hope for "one filesystem to rule them all" will ever really work out, but Ceph is the best positioned.

fh973 11y ago

I invite you to check out our product, Quobyte (www.quobyte.com). Not open-source, but a parallel high-performance POSIX file system, with split-brain safe quorum replication of all components, can also do erasure coding for files, with policy-based data placement, running on standard server hardware. We designed it to yield top performance both for (parallel) file system workloads and block storage workloads. So you can run VMs and databases on it with a performance better than any other partition-tolerant software storage system. The goal is to provide customers with a scalable automated general storage platform for all workloads, a la Google, but for real world applications.