3 ms·

The ideas of unifying independent memory, disk and cpu stretch back to at least 1988 with LOCUS, eventually becoming OpenSSI. Single system image clusters were

The ideas of unifying independent memory, disk and cpu stretch back to at least 1988 with LOCUS, eventually becoming OpenSSI. Single system image clusters were designed to manage tasks running on multiple computers as if they were running inside a single system. There were several SSI cluster solutions as of 1994. PVM, beginning in 1989, was a popular choice for parallel processing and distributed computing. It extended a virtual machine concept to allow processes to interact between disparate, loosely networked machines, even with different platforms and architectures. With increased parallel processing came a need for more distributed file systems (around in one form or another since the 1960s). They allowed processes on different nodes to interoperate without care of where the data was actually stored, and also more efficiently than if stored in a single node or array. PVFS, a parallel filesystem designed by NASA in 1993 to work with PVM, distributed storage and access of an object's parts over multiple nodes. This would later be ported to Linux for use in Beowulf clusters. GPFS, an IBM filesystem also developed in 1993, worked similarly. Google File System was eventually designed by the same researchers who created Coda in 1987, an object data store turned into a distributed [parallel] filesystem. So distributed filesystems evolved in parallel by different groups that based their work on earlier research projects such as AFS2 and Vesta. Google and Inktomi definitely had success with their implementations, but basically everyone was starting down that path as community computers became more available and parallel processing became more important.

chubot 9y ago
Hm, I've heard of PVM but don't know much about it. I don't doubt that there were projects to to unify memory/disk/CPU across distinct machines before Inktomi/Google. I don't think they used Ethernet and Unix sockets, but I don't know for sure. It's debatable how important that distinction is. Although, to me, commodity clusters also means commodity networking, not just commodity PCs, because otherwise the clusters wouldn't have been as cost effective overall. Probably the more important distinction is how many nodes are working together, which of course correlates with how many faults there are, and the sophistication of the software that handles the faults. I imagine earlier systems were in the 10-100 node range. Inktomi and early Google were more in the 1000 node range as far as I understand, and Google's clusters are up to at least 10K nodes and 100K CPUs/disks (probably more by now). At every order of magnitude there are different engineering challenges. There are always precursors, and the NOW paper cites a couple in section 2.3 (MiLi91, Zho92). A 10 or 100 node cluster is definitely a precursor, but you could still call it qualitatively different than 1K to 10K nodes. Also one funny thing is that Google/Facebook no longer use "commodity" clusters, if you take that word literally. You can no longer buy what they use in small quantities! e.g. a design from http://www.opencompute.org/ http://www.opencompute.org/ They're back to the model of highly engineered, relatively homogeneous, relatively reliable hardware. Not a bunch of PCs cobbled together with clever software!
- peterwwillis 9y ago
  Unix sockets? You mean Unix domain sockets? That only works on a single host. Perhaps you meant Berkeley sockets, the API for both Internet and Unix sockets. This became a POSIX standard and many systems emulate it, even Windows. PVM used them, and built abstractions and new methods on top of the socket APIs to make building distributed parallel-processing applications easier. And Ethernet was, for a time, much more expensive and slower than other solutions, not to mention buggy. It is still slow in some cases (top supercomputers use InfiniBand). Adopting Ethernet was just something that happened due to a confluence of factors (I believe mostly IBM supporting it, and then everyone else, and it becoming cheaper, faster and more reliable over time). So this was just an industry thing, not really a distributed processing thing. The number of nodes is less important than the number of systems - that is, if LLNL wants to do grid computing with CalTech, you're talking about parallel processing over two distinct systems. Even at Google, they may have multiple teams, each with a large distributed system on a different network, and they all have to interoperate. For this, you have to use PVM/MPI, or something similar. This was a significant step in distributed systems engineering. Google didn't need to interoperate with random systems, though, so they didn't need to use PVM/MPI, so their systems were simpler. I think the commodity computing helped them do massive processing cheaper, but at a point there's a lot of waste there, so they engineered out the waste in their new custom systems... which happen to be more expensive and complicated. Everything old is new again.

chubot 9y ago

Hm, I've heard of PVM but don't know much about it. I don't doubt that there were projects to to unify memory/disk/CPU across distinct machines before Inktomi/Google. I don't think they used Ethernet and Unix sockets, but I don't know for sure. It's debatable how important that distinction is. Although, to me, commodity clusters also means commodity networking, not just commodity PCs, because otherwise the clusters wouldn't have been as cost effective overall. Probably the more important distinction is how many nodes are working together, which of course correlates with how many faults there are, and the sophistication of the software that handles the faults. I imagine earlier systems were in the 10-100 node range. Inktomi and early Google were more in the 1000 node range as far as I understand, and Google's clusters are up to at least 10K nodes and 100K CPUs/disks (probably more by now). At every order of magnitude there are different engineering challenges. There are always precursors, and the NOW paper cites a couple in section 2.3 (MiLi91, Zho92). A 10 or 100 node cluster is definitely a precursor, but you could still call it qualitatively different than 1K to 10K nodes. Also one funny thing is that Google/Facebook no longer use "commodity" clusters, if you take that word literally. You can no longer buy what they use in small quantities! e.g. a design from http://www.opencompute.org/ http://www.opencompute.org/ They're back to the model of highly engineered, relatively homogeneous, relatively reliable hardware. Not a bunch of PCs cobbled together with clever software!

peterwwillis 9y ago
Unix sockets? You mean Unix domain sockets? That only works on a single host. Perhaps you meant Berkeley sockets, the API for both Internet and Unix sockets. This became a POSIX standard and many systems emulate it, even Windows. PVM used them, and built abstractions and new methods on top of the socket APIs to make building distributed parallel-processing applications easier. And Ethernet was, for a time, much more expensive and slower than other solutions, not to mention buggy. It is still slow in some cases (top supercomputers use InfiniBand). Adopting Ethernet was just something that happened due to a confluence of factors (I believe mostly IBM supporting it, and then everyone else, and it becoming cheaper, faster and more reliable over time). So this was just an industry thing, not really a distributed processing thing. The number of nodes is less important than the number of systems - that is, if LLNL wants to do grid computing with CalTech, you're talking about parallel processing over two distinct systems. Even at Google, they may have multiple teams, each with a large distributed system on a different network, and they all have to interoperate. For this, you have to use PVM/MPI, or something similar. This was a significant step in distributed systems engineering. Google didn't need to interoperate with random systems, though, so they didn't need to use PVM/MPI, so their systems were simpler. I think the commodity computing helped them do massive processing cheaper, but at a point there's a lot of waste there, so they engineered out the waste in their new custom systems... which happen to be more expensive and complicated. Everything old is new again.