4 ms·
The author argues that the number of meaningful operations a device can do per second will not keep growing exponentially. The reason he gives is that while we
by no_gravity 9y ago
The author argues that the number of meaningful operations a device can do per second will not keep growing exponentially. The reason he gives is that while we can add more parts that work in parallel, this has limited benefit:
The speed up starts to disappear as silicon is left idle
because there just aren’t enough different things to do.
I think there will be enough 'things to do'. Looking at real world applications of more processing power like AI and simulations, I expect them to be perfectly parallelizable.
- ianhowson 9y ago> I expect them to be perfectly parallelizable Unfortunately, the real world does not meet your expectation. You're talking about a class of problems called 'embarrassingly parallel': https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel Only a very tiny proportion of problems fit into that category. AI and simulations tend to be more easily parallelizable, but CPU-CPU communication is slow at any scale and imposes a limit to how much parallelism can be exploited. The original statement is accurate: > The speed up starts to disappear as silicon is left idle because there just aren’t enough different things to do Efficient parallel algorithms require a problem that can be divided into smaller independent components and solved separately. Not every problem can be divided in this way. Crypto algorithms are the classic example; there's no way to perform round N+1 without first performing round N, and this is very much by design.
- threatofrain 9y agoI wonder if those stubbornly serial problems may be arbitrarily approximated by parallel solutions.
- hyperpallium 9y agoOr a less efficient (per computation) parallel version, that scales.
- ianhowson 9y agoA parallel algorithm is always less efficient (per unit of computational resources) due to the extra communication overhead. That's the tradeoff for being able to utilise more total computational resources.
- hyperpallium 9y agoSure; I meant a different algorithm. e.g. inefficient by recomputing the same result in some cases; perhaps like parallel searches that may overlap. Such strategies would not have been researched much because they seem not just inefficient but stupid. But when you have cores to burn, they may turn out to be clever... I know of a case where similar happened when silicon became much cheaper several decades ago... the inventor said it just wouldn't have occurred to anyone before then, because it required an infeasible amount of silicon. People couldn't (or wouldn't) think in those terms.
- eleitl 9y ago> Only a very tiny proportions fit into that category Actually, it is exactly the other way round: most problems in the real world are about local communications of cells in a 3 dimensional lattice. Including, relativistic limits to communication. So an ideal architecture for that is a 3D cellular automaton. With bigger cells, you've got nodes on a 3d mesh (torus). Incidentally, the topology of most supercomputers.
- empath75 9y ago> Only a very tiny proportion of problems fit into that category I think more correctly, only a tiny proportion of solutions to problems fit into that category, and I think that's largely because of the street light effect. The entire history of math and science has been written by people working out problems in their head, or on paper, which is more or less a serial process. People just don't think well in parallel, and until recently, neither did computers. We have a lot of serial solutions because that's where the light was, basically. I think we're going to start finding a large variety of problems that turn out to have parallel solutions -- including a lot of problems that we didn't even know had solutions.
- tzahola 9y agoAmdahl’s law
- zaarn 9y agoSadly, in the real world most silicon for users will be busy rendering a cat trying to fit into a box and then falling over or alternatively attempting and failing to properly layout a word document while also accommodating macros inside said word document. Word documents and tumbling cats don't really benefit from more than a few cores.
- librvf 9y ago> Sadly, in the real world most silicon for users will be busy rendering a cat trying to fit into a box and then falling over Better than ideological proselytizing and political propaganda.
- imtringued 9y agoOn the other hand they don't need more than one good core.
- doomlaser 9y agohttps://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law
- grogers 9y agohttps://en.m.wikipedia.org/wiki/Gustafson%27s_law https://en.m.wikipedia.org/wiki/Gustafson%27s_law