5 ms·

How does this compare with the already popular distributed file systems IPFS?

by vog 10y ago

sabujp 10y ago
This isn't truly distributed in that sense: "Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is size 32GB, and can hold a lot of files. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is fairly small amount of data and is generally stable."
- the_duke 10y ago
  So there's no replication?
  - chrislusf 10y ago
    There is. I am not following your reasoning.
    - sabujp 10y ago
      i think he's asking if there's replication for the data stored on the master in case the master goes down, i.e. can there be multiple masters?
      - jusob 10y ago
        You can (should) have multiple masters. But each volume has a primary master. I did see other masters loose the volume state information after the primary master has gone down and been taken out of the cluster.
        chrislusf 10y ago
        Good observation. I will make the recovery much faster with gRPC calls, which can detect broken network much faster than http heartbeats.
- chrislusf 10y ago
  The master nodes only have soft states. All actual states are stored on volume servers, together with the actual data. If one master node goes down, the other master nodes will elect a new leader and control the system.
verbatim 10y ago
For one, SeaweedFS states that it is not POSIX-compliant.
- tyingq 10y ago
  It's not a filesystem either...I don't get why they keep using that terminology. It doesn't have directories, permissions, seek(), copy without the client pulling all the data and pushing it back, etc. It's a KV data store. Which is fine...nothing wrong with that. It just isn't a filesystem in any sense I'm familiar with. Edit: Ahh, I see. There's an optional piece that does provide at least directories. Still short of being a "filesystem" IMO, though: https://github.com/chrislusf/seaweedfs/wiki/Filer https://github.com/chrislusf/seaweedfs/wiki/Filer
  - chrislusf 10y ago
    There are many ways to build a filer layer. Current implementation is just one way to do it. Adding POSIX or not is more of a design choice. The blob storage layer can support both ways.
    - tyingq 10y ago
      Right. The word filesystem has a defacto meaning for many, though. Not POSIX either...FAT, NTFS, etc are filesystems. There's implied functionality that goes with the word. This isn't a filesystem anymore than etcd would be.
Royalaid 10y ago
I am not sure that SeaweedFS and IPFS fit the same use case, IPFS seems like it wants to a bedrock layer of a newer style of internet. SeaweedFS is more of a Blob store. So comparing them might be possible but also might not be the right idea.
kyledrake 10y ago
IPFS is a protocol and implementation of the distributed web. SeaweedFS is an open source version of Facebook's Haystack, a centralized filesystem (with redundancy across multiple servers). See https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf https://www.usenix.org/legacy/event/osdi10/tech/full_papers/... TLDR, Haystack implements a sort of append-only journal design where smaller files are concatenated into a single large "journal" file. After adding, you get a "filename" back that is based on the information needed to locate the data you are interested in. volumename_seekposition.jpg, for example. Essentially, you eliminate the need for FS metadata this way, which leads to less disk hits for read operations. Open volume, seek to position, read contents (including metadata) until end of file, you're done. There is then a single-master database that controls the location of the volumes (with a redundant failover), which is possible because managing volumes is less difficult than managing the position of every single file in all the volumes in the system. I think they just use MySQL, amazingly. Pretty neat idea. It's designed very specifically for photo storage at Facebook. The main drawback with this design is that it doesn't support binding filesystem paths well - to do that you need a K/V database to match the name you want to the filename needed for Haystack/SeaweedFS. SeaweedFS can do this with a separate Redis and Cassandra "filer" database, but the Redis and Cassandra backends are currently implemented as "flat namespace" stores, so filers using them may not perform directory listings at this time. Huge drawback if you need to 'ls' subdirectories of files, for example. That was a dealbreaker for me for neocities.org implementation, also maintenance was still an open question with SeaweedFS (when things fail, what do I do to fix them?) so I ended up going with CephFS.
- chrislusf 10y ago
  Thanks for the write up! You are correct on most of the points. There are postgres backed filer added already, which can list files.
  - kyledrake 10y ago
    That's great news! I'm going to take a look at it tonight. Postgres of course does needs some special work to make it failover in production gracefully to a standby server. It would be nice to see something that's designed for it like redis work, but of course, that's a limitation of how redis is designed. Regardless, I don't think I mentioned this properly, but SeaweedFS is amazing and it's very impressive, and using it at some point is not yet off the table for us. :)

How does this compare with the already popular distributed file systems IPFS?

by vog 10y ago

How does this compare with the already popular distributed file systems IPFS?

This isn't truly distributed in that sense: "Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is size 32GB, and can hold a lot of files. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is fairly small amount of data and is generally stable."

the_duke 10y ago
So there's no replication?
- chrislusf 10y ago
  There is. I am not following your reasoning.
  - sabujp 10y ago
    i think he's asking if there's replication for the data stored on the master in case the master goes down, i.e. can there be multiple masters?
    - jusob 10y ago
      You can (should) have multiple masters. But each volume has a primary master. I did see other masters loose the volume state information after the primary master has gone down and been taken out of the cluster.
      - chrislusf 10y ago
        Good observation. I will make the recovery much faster with gRPC calls, which can detect broken network much faster than http heartbeats.
chrislusf 10y ago
The master nodes only have soft states. All actual states are stored on volume servers, together with the actual data. If one master node goes down, the other master nodes will elect a new leader and control the system.

verbatim 10y ago

For one, SeaweedFS states that it is not POSIX-compliant.

tyingq 10y ago
It's not a filesystem either...I don't get why they keep using that terminology. It doesn't have directories, permissions, seek(), copy without the client pulling all the data and pushing it back, etc. It's a KV data store. Which is fine...nothing wrong with that. It just isn't a filesystem in any sense I'm familiar with. Edit: Ahh, I see. There's an optional piece that does provide at least directories. Still short of being a "filesystem" IMO, though: https://github.com/chrislusf/seaweedfs/wiki/Filer https://github.com/chrislusf/seaweedfs/wiki/Filer
- chrislusf 10y ago
  There are many ways to build a filer layer. Current implementation is just one way to do it. Adding POSIX or not is more of a design choice. The blob storage layer can support both ways.
  - tyingq 10y ago
    Right. The word filesystem has a defacto meaning for many, though. Not POSIX either...FAT, NTFS, etc are filesystems. There's implied functionality that goes with the word. This isn't a filesystem anymore than etcd would be.

Royalaid 10y ago

I am not sure that SeaweedFS and IPFS fit the same use case, IPFS seems like it wants to a bedrock layer of a newer style of internet. SeaweedFS is more of a Blob store. So comparing them might be possible but also might not be the right idea.

kyledrake 10y ago

IPFS is a protocol and implementation of the distributed web. SeaweedFS is an open source version of Facebook's Haystack, a centralized filesystem (with redundancy across multiple servers). See https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf https://www.usenix.org/legacy/event/osdi10/tech/full_papers/... TLDR, Haystack implements a sort of append-only journal design where smaller files are concatenated into a single large "journal" file. After adding, you get a "filename" back that is based on the information needed to locate the data you are interested in. volumename_seekposition.jpg, for example. Essentially, you eliminate the need for FS metadata this way, which leads to less disk hits for read operations. Open volume, seek to position, read contents (including metadata) until end of file, you're done. There is then a single-master database that controls the location of the volumes (with a redundant failover), which is possible because managing volumes is less difficult than managing the position of every single file in all the volumes in the system. I think they just use MySQL, amazingly. Pretty neat idea. It's designed very specifically for photo storage at Facebook. The main drawback with this design is that it doesn't support binding filesystem paths well - to do that you need a K/V database to match the name you want to the filename needed for Haystack/SeaweedFS. SeaweedFS can do this with a separate Redis and Cassandra "filer" database, but the Redis and Cassandra backends are currently implemented as "flat namespace" stores, so filers using them may not perform directory listings at this time. Huge drawback if you need to 'ls' subdirectories of files, for example. That was a dealbreaker for me for neocities.org implementation, also maintenance was still an open question with SeaweedFS (when things fail, what do I do to fix them?) so I ended up going with CephFS.

chrislusf 10y ago
Thanks for the write up! You are correct on most of the points. There are postgres backed filer added already, which can list files.
- kyledrake 10y ago
  That's great news! I'm going to take a look at it tonight. Postgres of course does needs some special work to make it failover in production gracefully to a standby server. It would be nice to see something that's designed for it like redis work, but of course, that's a limitation of how redis is designed. Regardless, I don't think I mentioned this properly, but SeaweedFS is amazing and it's very impressive, and using it at some point is not yet off the table for us. :)