6 ms·

LizardFS – Reliable and Scalable Distributed File System

36 points by turrini 11y ago

gravypod 11y ago
I've used tahoe lafs before, but not many other distributed file systems. There doesn't seem to be much information provided in this link, and their website doesn't help much either. Does LizardFS do full end to end encryption too? That's one of the more important things to me.
- nickpsecurity 11y ago
  The MooseFS reference will help you out: http://www.moosefs.org/ http://www.moosefs.org/ You might also find this list useful: https://en.wikipedia.org/wiki/List_of_file_systems#Distributed_fault-tolerant_file_systems https://en.wikipedia.org/wiki/List_of_file_systems#Distribut...
sciurus 11y ago
This is a fork of http://www.moosefs.org/ http://www.moosefs.org/
- throwaway13337 11y ago
  I don't see what lizard does that moose doesn't. Is this just a re-branding of moose in hopes to sell consulting for it? It seems that's the goal: https://lizardfs.com/ https://lizardfs.com/
lizardlipz 11y ago
.. what are you adding to the conversation with this comment? Can you tell us your experience about using moosefs and the implications for lizardfs?
- ryanlol 11y ago
  >.. what are you adding to the conversation with this comment? It's relevant to anyone familiar with moosefs?
  - lizardlipz 11y ago
    Then I guess I'll add: It's written in C++
    - ryanlol 11y ago
      Which one do you feel adds more value to the conversation. Knowing it's written in C++, or knowing it's a fork of an existing project that many might be familiar with? Do you think people are more likely to say, "Hey, this was written in C++. I liked other C++ software so I'm going to try this" or "Hey, it's a fork of moosefs. It had some details I would've liked changed, I'll give this a shot" ?
    - takeda 11y ago
      I actually was considering to try LizardFS after trying MooseFS, but this made stop. Why would you fork a project and then rewrite it in another language? What does rewriting it in C++ add to the user? It's a giant drawback, because backporting new features from MooseFS is much much harder. If you love C++ that much you would be much better off writing a new solution in C++. This only shows poor motivations for the fork.
- deleted 11y ago
  [deleted]
- Veratyr 11y ago
  He's adding context for those people who may have heard of MooseFS but not LizardFS. It's certainly not a big contribution to the discussion but it's not entirely without value.
- dang 11y ago
  We detached this subthread from https://news.ycombinator.com/item?id=11167581 https://news.ycombinator.com/item?id=11167581 and marked it off-topic.
bryanlarsen 11y ago
Does LizardFS support atomic rename?
acd 11y ago
I´ve used MooseFS which LizardFS is based upon and its very stable in production!
sj4nz 11y ago
So, no reasoning for a fork? What was the itch? Just to use GPLv3 instead of GPLv2, can GPL be "switched" just like that?
- kiiski 11y ago
  Looking at the MooseFS source files, I don't think they can. The license header of the files says: /* ... * MooseFS is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, version 2 (only). * ... */
- takeda 11y ago
  I think the reasoning was to add missing features, mainly HA, which moosefs did not have at the time and currently provides it in its paid version. One thing I never understood though is why when forking they decided to rewrite everything in C++. Wouldn't be better to just write it from scratch?
GauntletWizard 11y ago
This is interesting, especially the native windows client, but it's based on the same tired architecture as HDFS, and has all the same limitations. 1) Single Master for Metadata One of the biggest weaknesses in this model is that you can only scale the master so far. For metadata responses to be snappy, everything in your working set has to fit in RAM. That imposes hard limits on your number of files; 100M is achievable, 1B can even be done, but more than that and you're in for hurt. 2) Chunkserver to chunkserver replication As the moosefs write diagram shows, you're waiting for chunkservers to walk a linear tree, and you get the slowest performance of each one. Which, as it turns out, is on average pretty bad. If you have a 99th percentile latency of 10x your average, you lose 20% of your speed with three replicas. Not to mention the fact that your 'single point of failure' is actually any of the three of those chunkservers, with a long retry stall if one of them errors. LizardFS supports a xor mode, so it may be avoiding this by having clients talk to more than one chunkserver directly. Combined with proper queueing and retry logic, this reduces the effect of the long tail significantly.
techdragon 11y ago
Last time I investigated about a year ago... I saw some MAJOR as in "DO NOT USE, UNSAFE, WARNING, DANGER WILL ROBINSON", issues in this project... Silent data loss/corruption type stuff, and I did not see sufficient activity on a project of this complexity to make me feel at all comfortable using it.

36 points by turrini 11y ago

I've used tahoe lafs before, but not many other distributed file systems. There doesn't seem to be much information provided in this link, and their website doesn't help much either. Does LizardFS do full end to end encryption too? That's one of the more important things to me.

nickpsecurity 11y ago
The MooseFS reference will help you out: http://www.moosefs.org/ http://www.moosefs.org/ You might also find this list useful: https://en.wikipedia.org/wiki/List_of_file_systems#Distributed_fault-tolerant_file_systems https://en.wikipedia.org/wiki/List_of_file_systems#Distribut...

sciurus 11y ago

This is a fork of http://www.moosefs.org/ http://www.moosefs.org/

throwaway13337 11y ago
I don't see what lizard does that moose doesn't. Is this just a re-branding of moose in hopes to sell consulting for it? It seems that's the goal: https://lizardfs.com/ https://lizardfs.com/

lizardlipz 11y ago

.. what are you adding to the conversation with this comment? Can you tell us your experience about using moosefs and the implications for lizardfs?

ryanlol 11y ago
>.. what are you adding to the conversation with this comment? It's relevant to anyone familiar with moosefs?
- lizardlipz 11y ago
  Then I guess I'll add: It's written in C++
  - ryanlol 11y ago
    Which one do you feel adds more value to the conversation. Knowing it's written in C++, or knowing it's a fork of an existing project that many might be familiar with? Do you think people are more likely to say, "Hey, this was written in C++. I liked other C++ software so I'm going to try this" or "Hey, it's a fork of moosefs. It had some details I would've liked changed, I'll give this a shot" ?
  - takeda 11y ago
    I actually was considering to try LizardFS after trying MooseFS, but this made stop. Why would you fork a project and then rewrite it in another language? What does rewriting it in C++ add to the user? It's a giant drawback, because backporting new features from MooseFS is much much harder. If you love C++ that much you would be much better off writing a new solution in C++. This only shows poor motivations for the fork.
deleted 11y ago
[deleted]
Veratyr 11y ago
He's adding context for those people who may have heard of MooseFS but not LizardFS. It's certainly not a big contribution to the discussion but it's not entirely without value.
dang 11y ago
We detached this subthread from https://news.ycombinator.com/item?id=11167581 https://news.ycombinator.com/item?id=11167581 and marked it off-topic.

bryanlarsen 11y ago

Does LizardFS support atomic rename?

acd 11y ago

I´ve used MooseFS which LizardFS is based upon and its very stable in production!

sj4nz 11y ago

So, no reasoning for a fork? What was the itch? Just to use GPLv3 instead of GPLv2, can GPL be "switched" just like that?

kiiski 11y ago
Looking at the MooseFS source files, I don't think they can. The license header of the files says: /* ... * MooseFS is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, version 2 (only). * ... */
takeda 11y ago
I think the reasoning was to add missing features, mainly HA, which moosefs did not have at the time and currently provides it in its paid version. One thing I never understood though is why when forking they decided to rewrite everything in C++. Wouldn't be better to just write it from scratch?

GauntletWizard 11y ago

This is interesting, especially the native windows client, but it's based on the same tired architecture as HDFS, and has all the same limitations. 1) Single Master for Metadata One of the biggest weaknesses in this model is that you can only scale the master so far. For metadata responses to be snappy, everything in your working set has to fit in RAM. That imposes hard limits on your number of files; 100M is achievable, 1B can even be done, but more than that and you're in for hurt. 2) Chunkserver to chunkserver replication As the moosefs write diagram shows, you're waiting for chunkservers to walk a linear tree, and you get the slowest performance of each one. Which, as it turns out, is on average pretty bad. If you have a 99th percentile latency of 10x your average, you lose 20% of your speed with three replicas. Not to mention the fact that your 'single point of failure' is actually any of the three of those chunkservers, with a long retry stall if one of them errors. LizardFS supports a xor mode, so it may be avoiding this by having clients talk to more than one chunkserver directly. Combined with proper queueing and retry logic, this reduces the effect of the long tail significantly.

techdragon 11y ago

Last time I investigated about a year ago... I saw some MAJOR as in "DO NOT USE, UNSAFE, WARNING, DANGER WILL ROBINSON", issues in this project... Silent data loss/corruption type stuff, and I did not see sufficient activity on a project of this complexity to make me feel at all comfortable using it.