3 ms·

We have different interpretations of what we mean by "distributed system". I mean a group of servers that provide a single service. Or more precisely, metadata

We have different interpretations of what we mean by "distributed system". I mean a group of servers that provide a single service. Or more precisely, metadata is a single distributed shared memory layer, where file systems semantics are same regardless on which partition/shard of the metadata a metadata operation is performed. What Ceph and many others do is support partitioned metadata. One volume == one metadata partition. But cross partition metadata operations do not have the same semantics as intra-partition metadata operations. So, moving a file/dir between volumes (partitions) is not atomic - they do not solve the hard consensus problem. We do. Gluster is also not solving the consensus problem. The end result is leaky filesystem abstractions. If you move a file between two directories, it may go fast. But between two different directories (across volumes), it can go very slow. Renaming a file in Gluster can cause it to move between hosts.

notacoward 8y ago
> Renaming a file in Gluster can cause it to move between hosts. No. It does not. Within a volume, files will always be renamed in place on the replicas where they currently reside. There are other problems with rename in Gluster, having to do with how the locations of files are tracked when they're not where they're "supposed" to be according to the hashing scheme, but movement of data as part of rename is not one of those problems. "Across volumes" is not an interesting or relevant case, because the whole point of separate volumes in Gluster (and most other distributed filesystems) is to provide complete isolation. You seem to have this idea that users would want to have lots of volumes and still move data around between them. Is there something weird about your architecture that forces them to do that? Small volumes are operationally complex and stand resources. Generally, you'd have just one volume. Access control, quota, etc. could still be applied at the namespace or directory level, below volumes. It worked rather well at the world's largest Gluster installation, which I helped run for a year and a half, and I believe most other installations are similar. Please try to learn about other systems before you make wild claims about them, lest your audience perceive them as deliberate lies to benefit your business. I don't think your claims about Ceph are anywhere near accurate either, but I'll leave those for one of my Ceph friends to address.
- jamesblonde 8y ago
  My claim is exactly about what contributes a distributed namespace - i claim it is a namespace that spans multiple hosts and requires (distributed) consensus algorithms so that nodes that provide the metadata service (operations like rename) can execute operations only in metadata. Volumes are not just for isolation. They also partition the distributed namespace, and make it easier to scale - but at a cost. Operations that cross volumes do not solve the consensus problem, so they do not execute operations like move atomically. Whether that is a copy, then delete or something else is not the issue. They still do not provide a single distributed shared namespace abstraction with atomic operations.
  - notacoward 8y ago
    "i claim it is a namespace that spans multiple hosts and requires (distributed) consensus algorithms..." That's a very unique definition of "distributed". To many in the field, "spans multiple hosts" alone is sufficient. Some might refine it to preclude single nodes with special roles or by adding requirements such as a single view of the data etc., but it's a joke to add atomicity or implementation details such as how a consensus algorithm is used. There are plenty of distributed data stores that don't provide atomicity for all operations, and users are happy with that. There are plenty that use leader election instead of consensus to deal with consistency issues, and users are happy with that too. Inventing a novel and highly specific definition to exclude them seems more than a bit disingenuous. If we want to go that route, I'll point out that HopsFS is misnamed because it's not a filesystem according to commonly used definitions. There are many hard problems that it wimps out on, that a real filesystem must address. "Operations that cross volumes do not solve the consensus problem, so they do not execute operations like move atomically." Again, you seem to be using "volume" in a very unique way. Your architecture might misuse "volume" to mean some sort of internal convenience that's practically invisible to users, but others don't. To most, a volume is a self-contained universe, much like a virtual machine. Users are well aware that moving data between either will not have the same semantics as moving data within.
    - jamesblonde 8y ago

notacoward 8y ago

> Renaming a file in Gluster can cause it to move between hosts. No. It does not. Within a volume, files will always be renamed in place on the replicas where they currently reside. There are other problems with rename in Gluster, having to do with how the locations of files are tracked when they're not where they're "supposed" to be according to the hashing scheme, but movement of data as part of rename is not one of those problems. "Across volumes" is not an interesting or relevant case, because the whole point of separate volumes in Gluster (and most other distributed filesystems) is to provide complete isolation. You seem to have this idea that users would want to have lots of volumes and still move data around between them. Is there something weird about your architecture that forces them to do that? Small volumes are operationally complex and stand resources. Generally, you'd have just one volume. Access control, quota, etc. could still be applied at the namespace or directory level, below volumes. It worked rather well at the world's largest Gluster installation, which I helped run for a year and a half, and I believe most other installations are similar. Please try to learn about other systems before you make wild claims about them, lest your audience perceive them as deliberate lies to benefit your business. I don't think your claims about Ceph are anywhere near accurate either, but I'll leave those for one of my Ceph friends to address.

jamesblonde 8y ago
My claim is exactly about what contributes a distributed namespace - i claim it is a namespace that spans multiple hosts and requires (distributed) consensus algorithms so that nodes that provide the metadata service (operations like rename) can execute operations only in metadata. Volumes are not just for isolation. They also partition the distributed namespace, and make it easier to scale - but at a cost. Operations that cross volumes do not solve the consensus problem, so they do not execute operations like move atomically. Whether that is a copy, then delete or something else is not the issue. They still do not provide a single distributed shared namespace abstraction with atomic operations.
- notacoward 8y ago
  "i claim it is a namespace that spans multiple hosts and requires (distributed) consensus algorithms..." That's a very unique definition of "distributed". To many in the field, "spans multiple hosts" alone is sufficient. Some might refine it to preclude single nodes with special roles or by adding requirements such as a single view of the data etc., but it's a joke to add atomicity or implementation details such as how a consensus algorithm is used. There are plenty of distributed data stores that don't provide atomicity for all operations, and users are happy with that. There are plenty that use leader election instead of consensus to deal with consistency issues, and users are happy with that too. Inventing a novel and highly specific definition to exclude them seems more than a bit disingenuous. If we want to go that route, I'll point out that HopsFS is misnamed because it's not a filesystem according to commonly used definitions. There are many hard problems that it wimps out on, that a real filesystem must address. "Operations that cross volumes do not solve the consensus problem, so they do not execute operations like move atomically." Again, you seem to be using "volume" in a very unique way. Your architecture might misuse "volume" to mean some sort of internal convenience that's practically invisible to users, but others don't. To most, a volume is a self-contained universe, much like a virtual machine. Users are well aware that moving data between either will not have the same semantics as moving data within.
  - jamesblonde 8y ago