3 ms·

I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well

by stonogo 10y ago

I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well for you, but "googleable" is not the bar that is generally set for the HPC problem space.

KaiserPro 10y ago
Let me expand: What I was angling at is unpartitioned clustered filesystems are fragile. Dirty nodes cause lots of problems. Storage area networks which then interface via another means is much easier to look after, for little/no performance hit. (it can be a lot faster because there is less coordination/chatter) Would I use Lustre for long term storage, with 100% uptime requirement? no. WOuld I use it as a linearly scalable scratch space? yes. Would I use GPFS (or whatever its been rebranded to) instead? yes, probably. If I was hosting lost of small(<10tb dataset) disparate apps, would I put them all on the same namespace? fuckno. Would I wire all my clients directly into the same clustered namespace? not if I could avoid it. The issue is this, in HPC, or any other multi-node scheduled based system(basically mainframes but without the documentation, or the error checking) nodes die in new and interesting ways, If you have shared memory then its surprising how well interesting problems propagate. In most HPC senarios, you can, if you are desperate stop and restart from a known good point. If you are serving public things, you don't get that option. So its in your best interest to partition. Now, as for the googable bar, Unless everyone running the cluster is intimately familiar with the filesystem, including the interesting ways its fucked with VFS, How the metadata server handles stale locks, sudden bursts in lost clients or what ever, you need google. Even if you are a master, you might forget.

3 ms·

I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well

by stonogo 10y ago

KaiserPro 10y ago
Let me expand: What I was angling at is unpartitioned clustered filesystems are fragile. Dirty nodes cause lots of problems. Storage area networks which then interface via another means is much easier to look after, for little/no performance hit. (it can be a lot faster because there is less coordination/chatter) Would I use Lustre for long term storage, with 100% uptime requirement? no. WOuld I use it as a linearly scalable scratch space? yes. Would I use GPFS (or whatever its been rebranded to) instead? yes, probably. If I was hosting lost of small(<10tb dataset) disparate apps, would I put them all on the same namespace? fuckno. Would I wire all my clients directly into the same clustered namespace? not if I could avoid it. The issue is this, in HPC, or any other multi-node scheduled based system(basically mainframes but without the documentation, or the error checking) nodes die in new and interesting ways, If you have shared memory then its surprising how well interesting problems propagate. In most HPC senarios, you can, if you are desperate stop and restart from a known good point. If you are serving public things, you don't get that option. So its in your best interest to partition. Now, as for the googable bar, Unless everyone running the cluster is intimately familiar with the filesystem, including the interesting ways its fucked with VFS, How the metadata server handles stale locks, sudden bursts in lost clients or what ever, you need google. Even if you are a master, you might forget.