5 ms·
Ceph meltdowns seem to be the worst. I've seen two other projects struggle with Ceph and eventually both gave up and moved on to something else. The main proble
by bane 9y ago
Ceph meltdowns seem to be the worst. I've seen two other projects struggle with Ceph and eventually both gave up and moved on to something else. The main problem seems to be lack of documented usage guidelines and not much written on what to do when something blows up...and things seemed to blow up with both Ceph systems frequently and in many cases without any discernable error logging.
The after actions on both systems seemed to indicate that they had simply been provisioned wrong, or had violated some other best practice, but that advice never seemed to be written down anywhere and the documentation on the Ceph site was woefully inadequate.
One team simply moved on to using HDFS and rewrote their entire approach to assume HDFS and it's been pretty solid since. I think the other group moved to using Gluster.
I really hope it's getting better, because the idea is really great, but I wouldn't recommend it to anybody right now based on what I've seen.
- api 9y agoThere are lots of other options too like LizardFS: https://lizardfs.com https://lizardfs.com Seems really nice and easy to deploy.
- linsomniac 9y agoBut Ceph seems to be supported in everything. I know it's supported in Ganeti, which I currently run (and love) and Proxmox (which I used to run).
- takeda 9y agoLizardFS/MooseFS is mounted as regular filesystem (it behaves like NFS) so I would imagine everything should work with it.
- takeda 9y agoThat one is derived from MooseFS (https://moosefs.com/ https://moosefs.com/). What turned me away from LizardFS was when they forked the project the very first thing they did was rewriting the code to be in C++, this makes much harder to merge future improvements, and feels like they make some decisions for the wrong reason (the user doesn't care what language it is written in) MooseFS is quite good and has fairly good performance, but its weakness is that is also that the central point is its master, and the open source version doesn't have HA. I know that LizardFS supposedly has HA (it didn't have it at the time I looked into it), how is that implemented?
- xyproto 9y agoMooseFS can have a second master that takes over if the first one should fail, though.
- takeda 9y agoI'm familiar with 1.x which was very basic, glad they added that. The previous approach with metadataloggers had the issue that before you can convert it to a master you had to merge changelogs which could take a while when you have many files.
- samstave 9y agowhat is the use case where one would want to implement either lizard/moose into prod? Why? What are you solving for with these?
- takeda 9y agoWhen evaluating various distributed filesystems MooseFS[1] was the fastest one. [1] I did not evaluate LizardFS but back then it was freshly forked so it was pretty much the same thing.
- acd 9y agoHave had very good experiances with MooseFS the predecessor to LizardFS, Rock steady never crashed or did something finicky. Performance was ok.