3 ms·

its not a "natural" choice for a developer, unless you've worked in a large scale unix network that's been setup properly. We've been told for years that file

its not a "natural" choice for a developer, unless you've worked in a large scale unix network that's been setup properly. We've been told for years that file system primitives are bad, unscalable and nasty. part of this is because NFS can be a dick, but a lot is because large scale file systems generally are fast, scalable, coherent, secure and reliable (pick two). A lot of things can be managed with S3, but it has a lot of flaws. Unless you've worked at a place where you've had access to something like GPFS(or whatever IBM calls it now), where you can put hooks into everything, and programme how you want to mirror/cache/copy namespaces, it all seems impossible. Take FB for instance. They've gone balls deep with an S3 clone, which is both slow and difficult to use. Every operation takes about 3 seconds. (pulling a 10k file? pulling a 100meg file? list operation? mkdir? all 3 seconds.) Its an interface on top of the underlying blob store, so its looks like it stores it's metadata in a SQL Db (ie like lustre with metadata node and object storage nodes) That sucks for machine learning or any kind of "normal" file operations this means there is lots of heavy engineering to get something like performance that would be solved by an ephemeral clustered file system. The other big thing to note is that clustered filesystems are a massive massive pain in the arse to manage, especially if you choose the wrong one. (looks knowingly in the direction of ceph and gluster) you need to know what tradeoffs you are choosing before you install. Lustre is/was effectively a raid-0 over the network, Ceph is just plain inefficient, gluster is just silly, GPFS is held back by IBM being IBM. for most people clustered filesystems are wrong choice.

verisimilitudes 4y ago
> Unless you've worked at a place where you've had access to something like GPFS(or whatever IBM calls it now), where you can put hooks into everything, and programme how you want to mirror/cache/copy namespaces, it all seems impossible. Do I understand correctly that this would be one possible interpretation of this sentence: If you've only used UNIX, and not something IBM made decades ago, then it seems impossible. That's hilarious.
- KaiserPro 4y ago
  IF you want to take it in isolation, and assume that no-one uses GPFS, then yeah? The general gist is that unless you've worked in a place that has setup a "proper" network, that is roaming home directories, mapped posix storage, kerberised user account, and some sort of multi-machine graph based job dispatch system (ie airflow, grid engine, large scale k8s with a batch plugin, aws's batch pixar's tractor), then you will have not seen a need for a large clustered filesystem with a posix-like interface. moreover, given the "filesystems are hard and bad, use object storage instead" noise that we've had since we joined the cloud era, why would you _try_ to use a clustered filesystem? Especially as most of the "free" ones are shite.

verisimilitudes 4y ago

> Unless you've worked at a place where you've had access to something like GPFS(or whatever IBM calls it now), where you can put hooks into everything, and programme how you want to mirror/cache/copy namespaces, it all seems impossible. Do I understand correctly that this would be one possible interpretation of this sentence: If you've only used UNIX, and not something IBM made decades ago, then it seems impossible. That's hilarious.

KaiserPro 4y ago
IF you want to take it in isolation, and assume that no-one uses GPFS, then yeah? The general gist is that unless you've worked in a place that has setup a "proper" network, that is roaming home directories, mapped posix storage, kerberised user account, and some sort of multi-machine graph based job dispatch system (ie airflow, grid engine, large scale k8s with a batch plugin, aws's batch pixar's tractor), then you will have not seen a need for a large clustered filesystem with a posix-like interface. moreover, given the "filesystems are hard and bad, use object storage instead" noise that we've had since we joined the cloud era, why would you _try_ to use a clustered filesystem? Especially as most of the "free" ones are shite.