6 ms·

Can anyone working on this project comment on how HopsFS compares to similar alternatives? The post and the paper compare it to HDFS, but that's not really fair

Can anyone working on this project comment on how HopsFS compares to similar alternatives? The post and the paper compare it to HDFS, but that's not really fair - HDFS was not designed to handle huge quantities of small files, and if you use it for that, you will of course not get good performance. Competitive offerings would be things like Qumulo, Amazon EFS, Objective FS, perhaps BeeGFS & Lustre, and so on.

jamesblonde 8y ago
HopsFS is a derivative work (a fork) of HDFS - it is still 100% wire-compatible with HDFS. It has distributed metadata (not metadata stored on the heap of a single JVM). It also supports putting small files on NVMe storage - https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... It's also open-source. So, it works in places HDFS works. HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Lustre, BeeGFS are HPC filesystems, i believe. No data locality. Amazon EFS - feels like a MapR clone, but they expose some tunings - higher latency for higher throughput and larger clusters. We don't make that trade-off. I don't know enough about Qumulo and Objective FS to comment on them.
- nathanwh 8y ago
  Does Ceph not have distrubuted metadata as well?
  - jamesblonde 8y ago
    You can have multiple metadata servers, but then you have multiple volumes - like MapR. When you move a file between two different volumes, the move operation is no longer a simple metadata operation (which is atomic). Instead the move operation involves removing the file from one metadata server, then adding it to the other server - an agreement problem. Except, they don't solve the agreement problem properly. You will notice it more when you move a large directory, as then it will go really slow - and you hope you dont have a failure while the operation is executing. reference: http://docs.ceph.com/docs/master/cephfs/add-remove-mds/ http://docs.ceph.com/docs/master/cephfs/add-remove-mds/
    - teraflop 8y ago
      That reference does not contain any information to back up your claim. The Ceph team considers configurations with multiple active metadata servers to be stable and supported; where is your evidence that this is false?
- bdeorus 8y ago
  MapR-FS also distributes metadata and NVMe support
- IvanVergiliev 8y ago
  > the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Isn't that the case for S3 as well? I haven't seen AWS mention it explicitly, but the consistency guarantees and the "unlimited scalability" claims seem to point in that direction.
  - dekhn 8y ago
    S3 is a blobject store, not a filesystem.
- notacoward 8y ago
  > HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata Then you haven't done your homework, since Gluster has had that architecture for a decade. And it supports real POSIX semantics, too - not just the HDFS subset.
  - gnufx 8y ago
    Indeed, and (as notacoward will know) the HPC filesystems OrangeFS/PVFS2, Lustre, BeeGFS, and GPFS all have -- or can have -- distributed metadata, and POSIX semantics (or POSIX modulo locking).
    - notacoward 8y ago
      Correct. I didn't mean to exclude those, but they weren't part of the original comparison. I didn't mention Ceph because somebody else already had, plus I'm not a maintainer for that as I am for Gluster. ;) It might also be useful to point out some semi-useful distinctions regarding levels of distribution. The PVFS2 lineage has had fully distributed metadata for even longer than Gluster. OTOH, both Lustre and Ceph have separate metadata servers, more than one but still fewer than there are data servers. That doesn't necessarily limit scalability, but it does increase operational complexity. I'm honestly not sure about BeeGFS or GPFS; I don't pay much attention to them since one is only fake open source and the other doesn't even try. If we want to include proprietary offerings we could also add EFS, ObjectiveFS, OneFS, PanFS, etc. BTW, for those here who know me, at $dayjob I've switched from working on Gluster to working on a more HDFS-like data store. (They chose not to call it a filesystem because it isn't one, which I consider a refreshing bit of honesty in a field full of pretenders.) It scales way beyond any of the things mentioned here, but it's proprietary so I can't compare notes as much as I was able to with Gluster. :(
      - jamesblonde 8y ago
        We have different interpretations of what we mean by "distributed system". I mean a group of servers that provide a single service. Or more precisely, metadata is a single distributed shared memory layer, where file systems semantics are same regardless on which partition/shard of the metadata a metadata operation is performed. What Ceph and many others do is support partitioned metadata. One volume == one metadata partition. But cross partition metadata operations do not have the same semantics as intra-partition metadata operations. So, moving a file/dir between volumes (partitions) is not atomic - they do not solve the hard consensus problem. We do. Gluster is also not solving the consensus problem. The end result is leaky filesystem abstractions. If you move a file between two directories, it may go fast. But between two different directories (across volumes), it can go very slow. Renaming a file in Gluster can cause it to move between hosts.
- gnufx 8y ago
  It's incorrect to imply that HPC filesystems generally can't specify data distribution or that it matters greatly on HPC network fabrics; there are publications demonstrating better performance on Hadoop benchamrks over them than HDFS. Also HPC filesystem installations routinely have millions and millions of files.

jamesblonde 8y ago

HopsFS is a derivative work (a fork) of HDFS - it is still 100% wire-compatible with HDFS. It has distributed metadata (not metadata stored on the heap of a single JVM). It also supports putting small files on NVMe storage - https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files... It's also open-source. So, it works in places HDFS works. HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Lustre, BeeGFS are HPC filesystems, i believe. No data locality. Amazon EFS - feels like a MapR clone, but they expose some tunings - higher latency for higher throughput and larger clusters. We don't make that trade-off. I don't know enough about Qumulo and Objective FS to comment on them.

nathanwh 8y ago
Does Ceph not have distrubuted metadata as well?
- jamesblonde 8y ago
  You can have multiple metadata servers, but then you have multiple volumes - like MapR. When you move a file between two different volumes, the move operation is no longer a simple metadata operation (which is atomic). Instead the move operation involves removing the file from one metadata server, then adding it to the other server - an agreement problem. Except, they don't solve the agreement problem properly. You will notice it more when you move a large directory, as then it will go really slow - and you hope you dont have a failure while the operation is executing. reference: http://docs.ceph.com/docs/master/cephfs/add-remove-mds/ http://docs.ceph.com/docs/master/cephfs/add-remove-mds/
  - teraflop 8y ago
    That reference does not contain any information to back up your claim. The Ceph team considers configurations with multiple active metadata servers to be stable and supported; where is your evidence that this is false?
bdeorus 8y ago
MapR-FS also distributes metadata and NVMe support
IvanVergiliev 8y ago
> the only distributed filesystem with distributed metadata (apart from Google Collosus - the hidden jewel in their crown). Isn't that the case for S3 as well? I haven't seen AWS mention it explicitly, but the consistency guarantees and the "unlimited scalability" claims seem to point in that direction.
- dekhn 8y ago
  S3 is a blobject store, not a filesystem.
notacoward 8y ago
> HopsFS is, to the best of our knowledge, the only distributed filesystem with distributed metadata Then you haven't done your homework, since Gluster has had that architecture for a decade. And it supports real POSIX semantics, too - not just the HDFS subset.
- gnufx 8y ago
  Indeed, and (as notacoward will know) the HPC filesystems OrangeFS/PVFS2, Lustre, BeeGFS, and GPFS all have -- or can have -- distributed metadata, and POSIX semantics (or POSIX modulo locking).
  - notacoward 8y ago
    Correct. I didn't mean to exclude those, but they weren't part of the original comparison. I didn't mention Ceph because somebody else already had, plus I'm not a maintainer for that as I am for Gluster. ;) It might also be useful to point out some semi-useful distinctions regarding levels of distribution. The PVFS2 lineage has had fully distributed metadata for even longer than Gluster. OTOH, both Lustre and Ceph have separate metadata servers, more than one but still fewer than there are data servers. That doesn't necessarily limit scalability, but it does increase operational complexity. I'm honestly not sure about BeeGFS or GPFS; I don't pay much attention to them since one is only fake open source and the other doesn't even try. If we want to include proprietary offerings we could also add EFS, ObjectiveFS, OneFS, PanFS, etc. BTW, for those here who know me, at $dayjob I've switched from working on Gluster to working on a more HDFS-like data store. (They chose not to call it a filesystem because it isn't one, which I consider a refreshing bit of honesty in a field full of pretenders.) It scales way beyond any of the things mentioned here, but it's proprietary so I can't compare notes as much as I was able to with Gluster. :(
    - jamesblonde 8y ago
      We have different interpretations of what we mean by "distributed system". I mean a group of servers that provide a single service. Or more precisely, metadata is a single distributed shared memory layer, where file systems semantics are same regardless on which partition/shard of the metadata a metadata operation is performed. What Ceph and many others do is support partitioned metadata. One volume == one metadata partition. But cross partition metadata operations do not have the same semantics as intra-partition metadata operations. So, moving a file/dir between volumes (partitions) is not atomic - they do not solve the hard consensus problem. We do. Gluster is also not solving the consensus problem. The end result is leaky filesystem abstractions. If you move a file between two directories, it may go fast. But between two different directories (across volumes), it can go very slow. Renaming a file in Gluster can cause it to move between hosts.
gnufx 8y ago
It's incorrect to imply that HPC filesystems generally can't specify data distribution or that it matters greatly on HPC network fabrics; there are publications demonstrating better performance on Hadoop benchamrks over them than HDFS. Also HPC filesystem installations routinely have millions and millions of files.