4 ms·
In distributed computing, which is the kind of computing that tends to go on in data centres, there are two types of problem - large amounts of CPU on relativel
by ajessup 15y ago
In distributed computing, which is the kind of computing that tends to go on in data centres, there are two types of problem - large amounts of CPU on relatively small amounts of data, or small amounts of CPU on a relatively large amount of data.
The former is generally found in scientific computing (say, protein folding studies, or SETI@Home). In these kinds of problems, it's acceptable to have a widely distributed system with significant latency between nodes, because even though it takes you a long time to move the data across to another node, it's offset by the time you save in having another CPU perform work on it.
The latter is generally what you find in the business and web applications that are generating most of the heat in data centres today. These are things like stats aggregation, building search indicies, or plain old DB seeks. In these cases, it often doesn't make sense to distribute the work out over a connection with high latency, since by the time you've done that, performed a relatively cheap CPU operation on it, and sent it back down the wire, it would have been quicker just to queue it up locally and let the same node deal with when it had some CPU slots spare.
In other words the higher the latency between nodes, the less efficient the entire system becomes, and the less economical horizontal scaling becomes. In addition, there's a ton of extra overhead to maintain concurrency when latency is high.
It could be great for some niche problems though - like CDN, P2P relays, or local backup for nearby machines.
- marshallp 15y agoActually, compute intensive jobs are becoming more prevalant over time, and request-response systems such as computer vision, speech recognition, and search (which are quickly gaining ground) are embarrassingly parallel tasks that don't require low latency dense clusters.
- itmag 15y agoCan you contact me? I want to pick your brain.
- marshallp 15y agosure, added my email to profile
- ajessup 15y agoNo, most of these do. To build a search engine for example, you need to collect a massive amount of data (say, every web page on the internet), and perform a relatively trivial amount of computation on it (building a TF/IDF term index of each term, computing PageRank, etc.) to build your indexes. While building a search index is computationally intensive, the number of CPU cycles exercised per GB of data is relatively low. So if you wanted to 'distribute' this problem by farming out the data across a high latency connection to have it processed on another node, and then have it returned to you would actually be slower in many cases than just processing locally on a decent machine.
- marshallp 15y agoit depends on the what is meant by a computer. i'd refer to it as a mini-cluster that can fit into a basement for heating in the context of this article (and soon enough , moore's law, that will be a desktop size). On that scale, all these tasks are embarassingly parallel. Exceptions to this are multiuser systems where users interactions matter, such as facebook, in which case a huge cluster is required. My argument is that more computing will devoted to the former (basically machine learning, and scientific/engineering optimization) than the latter (multi-user databases) over time, simply because humans can only enter so much info (all 7 billion of us) compared to what machines can gather and compute.