3 ms·
Is there really a need for a filesystem? Just pull from a bucket and it's done. Push updates to the bucket and...it's done. I see the need for "sharing" in giv
by pstuart 1y ago
Is there really a need for a filesystem? Just pull from a bucket and it's done. Push updates to the bucket and...it's done.
I see the need for "sharing" in giving access to the data, but not to have it represented on the filesystem (other than giving the illusion of local dev)
- nickaggarwal 1y agoIf you want de couple model loading and distribution, If you do it in application when you need application pulling from bucket can be slow
- jacobsenscott 1y agoWhat if all your code is already written to use a filesystem, and you want to change the backing store from nfs to object store? Or what if you want to abstract away the specific blob store?
- nickaggarwal 1y agoThis was our exact use case backing store from nfs to object store. You can cloud cloud-specific mount providers and use a thin client in the middle
- bayindirh 1y agoWhen you want to feed your GPUs with what they need (model, weights, data), that kind of hoops slow you down exponentially. You need to be able to stream data to pinned memory, or to the GPU directly (hence GPUDirect) to keep your cards saturated. This is why systems like Weka exist, and why Lustre is still being developed and polished. These systems reach tremendous speeds. This is not an exaggeration when you connect terabits of bandwidth to these storage systems.
- topspin 1y agoI haven't had to feed a building full of GPUs, but I do wonder: does Ceph ever appear on your radar for this sort of work? And if not, why? Seems like it would fit rather well. Yet it never seems to get mentioned.