4 ms·

The hardest part of a Distributed File system (and I mean File system here) is managing the Meatadata (where a file is, where the directory is, who last did som

The hardest part of a Distributed File system (and I mean File system here) is managing the Meatadata (where a file is, where the directory is, who last did something to it.) Lustre, GFS2 and GPFS all have centralised metadatstores, which is both a boon and a drawback. What I can't figure out is what they've done here. It appears like metadata is stored in a special partition ("journal") which is shared? But there is a control process as well.

jamesblonde 8y ago
It doesn't look like PolarFS has distributed metadata. HopsFS (HDFS derivative work) stores distributed metadata, enabling 16X throughput improvements over HDFS - https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... http://www.hops.io http://www.hops.io Technically, the hard part is correct concurrent and consistent operations across shards (partitions). Disclaimer: one of the designers of HopsFS.
- ddorian43 8y ago
  What do you think are some gotchas/limitations when using mysql-ndb-cluster ?
  - jamesblonde 8y ago
    It's become very stable with the years. It's somehow never penetrated the silicon valley echo chamber, probably due to instability in its earlier years. Operationally, it's got most things you need - on-line add node, rolling upgrades, monitoring, etc. We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.
    - ryanworl 8y ago
      Have you explored using FoundationDB? It doesn't have a SQL interface (anymore), which might mean it isn't suitable if SQL is required.
      - jamesblonde 8y ago
        Not yet, it's only recently been open-sourced. We have a startup now commercializing Hops, called Logical Clocks (of course), so we're busy with that.
        ryanworl 8y ago
        The community is very helpful over on the forums (https://forums.foundationdb.org https://forums.foundationdb.org), so if you do end up having time to check it out, come on over! I'm sure the team at Apple (I'm not affiliated with them) would love to talk about it.
        ddorian43 8y ago
        They're against coprocessors (having functions in the db) so you'll always be somewhat limited. Doing with shared-memory-etc will be hard. Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.
gnufx 8y ago
With respect to metadata, the premise of PVFS^WOrangeFS is that providing full POSIX semantics (specifically locking?) makes the issues even worse, so it doesn't try, to decent effect. That may rule it out, for some applications like databases, though. Lustre has DNE for distributed metadata now. Presumably multi-tenancy would be important in this sort of application.

jamesblonde 8y ago

It doesn't look like PolarFS has distributed metadata. HopsFS (HDFS derivative work) stores distributed metadata, enabling 16X throughput improvements over HDFS - https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... http://www.hops.io http://www.hops.io Technically, the hard part is correct concurrent and consistent operations across shards (partitions). Disclaimer: one of the designers of HopsFS.

ddorian43 8y ago
What do you think are some gotchas/limitations when using mysql-ndb-cluster ?
- jamesblonde 8y ago
  It's become very stable with the years. It's somehow never penetrated the silicon valley echo chamber, probably due to instability in its earlier years. Operationally, it's got most things you need - on-line add node, rolling upgrades, monitoring, etc. We use NDB to store both the metadata in-memory, but also to store small files on NVMe disks. We had talks with lots of other DB vendors, but, frankly, none of them have high performance support for cross-partition transactions, which is needed. DBs like VoltDB, MemSQL, NuoDB all have promise, but serialize cross-shard operations.
  - ryanworl 8y ago
    Have you explored using FoundationDB? It doesn't have a SQL interface (anymore), which might mean it isn't suitable if SQL is required.
    - jamesblonde 8y ago
      Not yet, it's only recently been open-sourced. We have a startup now commercializing Hops, called Logical Clocks (of course), so we're busy with that.
      - ryanworl 8y ago
        The community is very helpful over on the forums (https://forums.foundationdb.org https://forums.foundationdb.org), so if you do end up having time to check it out, come on over! I'm sure the team at Apple (I'm not affiliated with them) would love to talk about it.
        ddorian43 8y ago
        They're against coprocessors (having functions in the db) so you'll always be somewhat limited. Doing with shared-memory-etc will be hard. Also you can shard many tables as hash, so most hot-path transactions be inside a server, which you can't guarantee with range-sharding.

gnufx 8y ago

With respect to metadata, the premise of PVFS^WOrangeFS is that providing full POSIX semantics (specifically locking?) makes the issues even worse, so it doesn't try, to decent effect. That may rule it out, for some applications like databases, though. Lustre has DNE for distributed metadata now. Presumably multi-tenancy would be important in this sort of application.