5 ms·

Hosting providers still don't really use distributed computing. Cloud hosting providers do, but mainly because they are great at selling the idea that the custo

Hosting providers still don't really use distributed computing. Cloud hosting providers do, but mainly because they are great at selling the idea that the customer should help the hosting provider save money by accepting increased complexity. What you're describing, clusters based on commodity PCs, have existed in some form since shortly after commodity PCs were introduced. Clusters in general have existed since 1977. The thing is that most computers were big and expensive, so there was practically no hardware available to make clusters on the cheap until the 90's. The NOW workstation you reference was the progenitor of Beowulf clusters, developed at NASA in 1994. [https://www.researchgate.net/publication/24321665_Clustered_Workstations_and_their_Potential_Role_as_High_Speed_Compute_Processors https://www.researchgate.net/publication/24321665_Clustered_...] But vendors were making similar clusters at the same time, because they all had to solve the same problem: handle workloads too big for a single machine. The Tandem Himalayan and IBM S/390 Parallel Sysplex were also both introduced in 1994, but those were big commercial products, not commodity clusters. Many other commercial cluster solutions had been produced in the previous 10 years. So it's not exactly some mind-blowing paradigm shift, and more an increased focus on reducing the cost of computing, which has always been the pink elephant in the room.

chubot 9y ago
Well, I'd argue that Inktomi and Google are qualitatively different than earlier clusters for 2 of the 3 reasons mentioned in the NOW paper: - dramatically improving virtual memory and file system performance by using the aggregate DRAM of a NOW as a giant cache for disk; - achieving cheap, highly available, and scalable file storage by using redundant arrays of workstation disks, using the LAN as the I/O backplane; and Also: - Compute clusters with shared memory are completely different than NOW clusters. - Compute clusters existed before, but most were not fault tolerant as far as I understand. Commodity disks have a (much?) higher failure rate than commodity CPUs. As far as I know, NASA was mainly using compute clusters. As far as I understand, "Beowulf" clusters were mostly about compute. That is, doing a lot of computation on a relatively small amount of data. In contrast, Google and Inktomi were about doing a small amount of computation on huge data, and also operating live services. Though, I'm interested in any specific examples of earlier systems that unified separate memories and disks from commodity PCs. For example, BigTable definitely unified distributed memory and disk into a single resource, but it came much later. Also, to be truly called "commodity", the memory/disk should be unified over Ethernet, e.g. using a Unix socket interface and not something custom.
- peterwwillis 9y ago
  The ideas of unifying independent memory, disk and cpu stretch back to at least 1988 with LOCUS, eventually becoming OpenSSI. Single system image clusters were designed to manage tasks running on multiple computers as if they were running inside a single system. There were several SSI cluster solutions as of 1994. PVM, beginning in 1989, was a popular choice for parallel processing and distributed computing. It extended a virtual machine concept to allow processes to interact between disparate, loosely networked machines, even with different platforms and architectures. With increased parallel processing came a need for more distributed file systems (around in one form or another since the 1960s). They allowed processes on different nodes to interoperate without care of where the data was actually stored, and also more efficiently than if stored in a single node or array. PVFS, a parallel filesystem designed by NASA in 1993 to work with PVM, distributed storage and access of an object's parts over multiple nodes. This would later be ported to Linux for use in Beowulf clusters. GPFS, an IBM filesystem also developed in 1993, worked similarly. Google File System was eventually designed by the same researchers who created Coda in 1987, an object data store turned into a distributed [parallel] filesystem. So distributed filesystems evolved in parallel by different groups that based their work on earlier research projects such as AFS2 and Vesta. Google and Inktomi definitely had success with their implementations, but basically everyone was starting down that path as community computers became more available and parallel processing became more important.
  - chubot 9y ago
    Hm, I've heard of PVM but don't know much about it. I don't doubt that there were projects to to unify memory/disk/CPU across distinct machines before Inktomi/Google. I don't think they used Ethernet and Unix sockets, but I don't know for sure. It's debatable how important that distinction is. Although, to me, commodity clusters also means commodity networking, not just commodity PCs, because otherwise the clusters wouldn't have been as cost effective overall. Probably the more important distinction is how many nodes are working together, which of course correlates with how many faults there are, and the sophistication of the software that handles the faults. I imagine earlier systems were in the 10-100 node range. Inktomi and early Google were more in the 1000 node range as far as I understand, and Google's clusters are up to at least 10K nodes and 100K CPUs/disks (probably more by now). At every order of magnitude there are different engineering challenges. There are always precursors, and the NOW paper cites a couple in section 2.3 (MiLi91, Zho92). A 10 or 100 node cluster is definitely a precursor, but you could still call it qualitatively different than 1K to 10K nodes. Also one funny thing is that Google/Facebook no longer use "commodity" clusters, if you take that word literally. You can no longer buy what they use in small quantities! e.g. a design from http://www.opencompute.org/ http://www.opencompute.org/ They're back to the model of highly engineered, relatively homogeneous, relatively reliable hardware. Not a bunch of PCs cobbled together with clever software!

chubot 9y ago

Well, I'd argue that Inktomi and Google are qualitatively different than earlier clusters for 2 of the 3 reasons mentioned in the NOW paper: - dramatically improving virtual memory and file system performance by using the aggregate DRAM of a NOW as a giant cache for disk; - achieving cheap, highly available, and scalable file storage by using redundant arrays of workstation disks, using the LAN as the I/O backplane; and Also: - Compute clusters with shared memory are completely different than NOW clusters. - Compute clusters existed before, but most were not fault tolerant as far as I understand. Commodity disks have a (much?) higher failure rate than commodity CPUs. As far as I know, NASA was mainly using compute clusters. As far as I understand, "Beowulf" clusters were mostly about compute. That is, doing a lot of computation on a relatively small amount of data. In contrast, Google and Inktomi were about doing a small amount of computation on huge data, and also operating live services. Though, I'm interested in any specific examples of earlier systems that unified separate memories and disks from commodity PCs. For example, BigTable definitely unified distributed memory and disk into a single resource, but it came much later. Also, to be truly called "commodity", the memory/disk should be unified over Ethernet, e.g. using a Unix socket interface and not something custom.

peterwwillis 9y ago
The ideas of unifying independent memory, disk and cpu stretch back to at least 1988 with LOCUS, eventually becoming OpenSSI. Single system image clusters were designed to manage tasks running on multiple computers as if they were running inside a single system. There were several SSI cluster solutions as of 1994. PVM, beginning in 1989, was a popular choice for parallel processing and distributed computing. It extended a virtual machine concept to allow processes to interact between disparate, loosely networked machines, even with different platforms and architectures. With increased parallel processing came a need for more distributed file systems (around in one form or another since the 1960s). They allowed processes on different nodes to interoperate without care of where the data was actually stored, and also more efficiently than if stored in a single node or array. PVFS, a parallel filesystem designed by NASA in 1993 to work with PVM, distributed storage and access of an object's parts over multiple nodes. This would later be ported to Linux for use in Beowulf clusters. GPFS, an IBM filesystem also developed in 1993, worked similarly. Google File System was eventually designed by the same researchers who created Coda in 1987, an object data store turned into a distributed [parallel] filesystem. So distributed filesystems evolved in parallel by different groups that based their work on earlier research projects such as AFS2 and Vesta. Google and Inktomi definitely had success with their implementations, but basically everyone was starting down that path as community computers became more available and parallel processing became more important.
- chubot 9y ago
  Hm, I've heard of PVM but don't know much about it. I don't doubt that there were projects to to unify memory/disk/CPU across distinct machines before Inktomi/Google. I don't think they used Ethernet and Unix sockets, but I don't know for sure. It's debatable how important that distinction is. Although, to me, commodity clusters also means commodity networking, not just commodity PCs, because otherwise the clusters wouldn't have been as cost effective overall. Probably the more important distinction is how many nodes are working together, which of course correlates with how many faults there are, and the sophistication of the software that handles the faults. I imagine earlier systems were in the 10-100 node range. Inktomi and early Google were more in the 1000 node range as far as I understand, and Google's clusters are up to at least 10K nodes and 100K CPUs/disks (probably more by now). At every order of magnitude there are different engineering challenges. There are always precursors, and the NOW paper cites a couple in section 2.3 (MiLi91, Zho92). A 10 or 100 node cluster is definitely a precursor, but you could still call it qualitatively different than 1K to 10K nodes. Also one funny thing is that Google/Facebook no longer use "commodity" clusters, if you take that word literally. You can no longer buy what they use in small quantities! e.g. a design from http://www.opencompute.org/ http://www.opencompute.org/ They're back to the model of highly engineered, relatively homogeneous, relatively reliable hardware. Not a bunch of PCs cobbled together with clever software!