3 ms·

Common programming model of parallel supercomputers expect something like one shared filesystem across all nodes (used for input and output and checkpointing an

by dfox 14y ago

Common programming model of parallel supercomputers expect something like one shared filesystem across all nodes (used for input and output and checkpointing and such, almost never as temporary per-node storage). And most readily available and scalable enough implementations of that expect exactly this architecture as it is easier to implement in software, reason about it's performance and also significantly cheaper and easier to maintain.

paulsutter 14y ago
Yes, and there's no reason that filesystem can't be distributed across local drives on all nodes. Read up on GFS or any modern distributed filesystem.
- TallGuyShort 14y ago
  Distributed filesystems are designed with commodity hardware and traditional data-centers in mind. Why SHOULD it be distributed across local drives on all nodes? These are computationally intensive workloads (like supercomputers), not storage I/O intensive workloads (like mainframes or Hadoop clusters). If you're managing hardware on this scale and you don't need local drives, why on earth wouldn't you put your storage in a separate, central location?
- dfox 14y ago
  Distributed filesystem in the GFS (assuming you mean RedHat's GFS2) sense, like GFS itself, SGI's CXFS, Apple's Xsan or even clustering support in almost all Enterprise DBMS's, does not mean that the data itself is distributed, only that the filesystem (or DB) logic is distributed and only shared component is single central (eg. FC or iSCSI attached) dumb block device without any special understanding of the filesystem, which is exact opposite of using local hard drives of the nodes. Things like Lustre or GoogleFS use distributed storage nodes that are normally separate from client nodes. While in both cases nothing prevents you from running both client applications and storage server process on same node there is little reason to do so, for both operational (disks are in one place) and economic reasons (attaching 20 disks to one node is cheaper than attaching 20 disks to 20 nodes). In fact few years ago I've actively searched for filesystem with built-in redundancy and storage capacity distributed across client nodes and found almost nothing that was designed for such use case. Only thing I'm aware of that was close to what you propose is/was essentially an wrapper for NFS in (open)MOSIX, that was able to redirect I/O from process to local disk when it found out that it was migrated to same node as NFS server runs on. With the important note that (open)MOSIX is not exactly meaningful HPC solution, but interesting nevertheless.

by dfox 14y ago

paulsutter 14y ago

Yes, and there's no reason that filesystem can't be distributed across local drives on all nodes. Read up on GFS or any modern distributed filesystem.

TallGuyShort 14y ago
Distributed filesystems are designed with commodity hardware and traditional data-centers in mind. Why SHOULD it be distributed across local drives on all nodes? These are computationally intensive workloads (like supercomputers), not storage I/O intensive workloads (like mainframes or Hadoop clusters). If you're managing hardware on this scale and you don't need local drives, why on earth wouldn't you put your storage in a separate, central location?
dfox 14y ago
Distributed filesystem in the GFS (assuming you mean RedHat's GFS2) sense, like GFS itself, SGI's CXFS, Apple's Xsan or even clustering support in almost all Enterprise DBMS's, does not mean that the data itself is distributed, only that the filesystem (or DB) logic is distributed and only shared component is single central (eg. FC or iSCSI attached) dumb block device without any special understanding of the filesystem, which is exact opposite of using local hard drives of the nodes. Things like Lustre or GoogleFS use distributed storage nodes that are normally separate from client nodes. While in both cases nothing prevents you from running both client applications and storage server process on same node there is little reason to do so, for both operational (disks are in one place) and economic reasons (attaching 20 disks to one node is cheaper than attaching 20 disks to 20 nodes). In fact few years ago I've actively searched for filesystem with built-in redundancy and storage capacity distributed across client nodes and found almost nothing that was designed for such use case. Only thing I'm aware of that was close to what you propose is/was essentially an wrapper for NFS in (open)MOSIX, that was able to redirect I/O from process to local disk when it found out that it was migrated to same node as NFS server runs on. With the important note that (open)MOSIX is not exactly meaningful HPC solution, but interesting nevertheless.