5 ms·

Distributed Filesystems: Fraunhofer vs. Gluster (2014)

42 points by weitzj 11y ago

dvirsky 11y ago
> "We tried Ceph about a year ago and it wasn’t nearly ready for production use" Given that it's a year-old article, that 2 year-old impression is probably very outdated. I would have loved to see Ceph benchmarked there as well.
- merb 11y ago
  The article is really outdated. There is no "FhGFS" anymore. http://www.beegfs.com/content/ http://www.beegfs.com/content/ What i don't like are metadata servers. I mean mostly I would start out small, with every project. But the "real" cluster file systems don't follow this tradition. The bare minimum of a solid production environment is probably more than 5 nodes, which is really huge for using it as a layer for content distribution in the small scale. I mean its cheaper to rent s3 / the new aws nfs solution or any other s3 like storage solution, than running a server with a distributed file system by yourself and I'm just considering the running costs, no maintance etc... Its really hard to do this on premise on a really small scale and just for availability. Most people will mostly just buy this stuff from san/nas vendors if they could afford it, if they have a system that needs to have this built in your pretty much fucked with ugly hacks / solutions.
  - true_religion 11y ago
    > I mean its cheaper to rent s3 / the new aws nfs solution or any other s3 like storage solution Sure if you never pay for bandwidth.
mbq 11y ago
There is also little known but very cool MooseFS http://www.moosefs.org/ http://www.moosefs.org/
- Nux 11y ago
  Also http://www.xtreemfs.org/ http://www.xtreemfs.org/
philchambers 11y ago
Not sure if they've worked on it, but 3 years ago Gluster performance during rebuild was about 1/100th of in normal operations making it effectively impossible to do a rebuild of a production cluster if you lost a node.
bennyturns 11y ago
> "Glfs has a severe problem with small writes, both with writing small files and small writes into large files. FhGFS does not have this bottleneck." GlusterFS has implemented many smallfile performance enhancements and with the latest release I am seeing a 20-50% improvement on smallfile performance. There are more improvements in the pipeline: http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf http://www.gluster.org/community/documentation/index.php/Fea... I would be curious to see these tests run on a properly tuned(readahead on bricks is important for spinning disks and read workloads, I didn't see any mention of brick tuning in the article) and on the latest version of glusterfs with the smallfile performance enhancements.
tetron 11y ago
Also Arvados Keep (http://arvados.org http://arvados.org) is a content-addressed distributed file system.

42 points by weitzj 11y ago

> "We tried Ceph about a year ago and it wasn’t nearly ready for production use" Given that it's a year-old article, that 2 year-old impression is probably very outdated. I would have loved to see Ceph benchmarked there as well.

merb 11y ago
The article is really outdated. There is no "FhGFS" anymore. http://www.beegfs.com/content/ http://www.beegfs.com/content/ What i don't like are metadata servers. I mean mostly I would start out small, with every project. But the "real" cluster file systems don't follow this tradition. The bare minimum of a solid production environment is probably more than 5 nodes, which is really huge for using it as a layer for content distribution in the small scale. I mean its cheaper to rent s3 / the new aws nfs solution or any other s3 like storage solution, than running a server with a distributed file system by yourself and I'm just considering the running costs, no maintance etc... Its really hard to do this on premise on a really small scale and just for availability. Most people will mostly just buy this stuff from san/nas vendors if they could afford it, if they have a system that needs to have this built in your pretty much fucked with ugly hacks / solutions.
- true_religion 11y ago
  > I mean its cheaper to rent s3 / the new aws nfs solution or any other s3 like storage solution Sure if you never pay for bandwidth.

mbq 11y ago

There is also little known but very cool MooseFS http://www.moosefs.org/ http://www.moosefs.org/

Nux 11y ago
Also http://www.xtreemfs.org/ http://www.xtreemfs.org/

philchambers 11y ago

Not sure if they've worked on it, but 3 years ago Gluster performance during rebuild was about 1/100th of in normal operations making it effectively impossible to do a rebuild of a production cluster if you lost a node.

bennyturns 11y ago

> "Glfs has a severe problem with small writes, both with writing small files and small writes into large files. FhGFS does not have this bottleneck." GlusterFS has implemented many smallfile performance enhancements and with the latest release I am seeing a 20-50% improvement on smallfile performance. There are more improvements in the pipeline: http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf http://www.gluster.org/community/documentation/index.php/Fea... I would be curious to see these tests run on a properly tuned(readahead on bricks is important for spinning disks and read workloads, I didn't see any mention of brick tuning in the article) and on the latest version of glusterfs with the smallfile performance enhancements.

tetron 11y ago

Also Arvados Keep (http://arvados.org http://arvados.org) is a content-addressed distributed file system.