5 ms·

I think one of the particularly interesting aspects of Machine learning is that the research into it has been focused on parallelism and partitioning from day 1

I think one of the particularly interesting aspects of Machine learning is that the research into it has been focused on parallelism and partitioning from day 1. The result of this is that those work loads are particularly well suited to just scaling to more chips. It may be that we can continue our computational climb for a while longer simply by scaling out in a way that consumer driven products really weren't able to do.

segfaultbuserr 7y ago
> It may be that we can continue our computational climb for a while longer I have a random question. Imagine this hypothetical scenario: semiconductor manufacturing comes to a halt completely, and new physics and technology is still decades ahead, can we further improve the practical performance of general computation by innovation alone? I imagine, instead of riding on the train of Moore's Law, more resources would be invested to design optimization and R&D of new architectures, e.g. faster FPU, faster pipeline, etc. Also, previously discarded and ignored ideas may be implemented again, and delivers some real advancement. While exponential growth is not possible, at least linear scaling should be the case. For example, computers without clocks. Based on this comment on HN, https://news.ycombinator.com/item?id=19554248 https://news.ycombinator.com/item?id=19554248, one major obstacle of clockless chip is the entire VLSI toolchain is designed and optimized for synchronous logic, and this can be changed if the industry invest some serious resources. Another example comes to my mind is high-level programming based on hardware, e.g. Lisp machine. Does my imagination make any sense?
- gameswithgo 7y ago
  >can we further improve the practical performance of general computation by innovation alone? we can improve the performance of average programs by a factor of ~100x simply by more careful software engineering. Don't use python or javascript, optimize code for cpu cache hits, use more cores effectively, etc. However games, maybe only 2x if anything.
  - segfaultbuserr 7y ago
    I was referring to the computational capabilities that hardware can provide, not the performance of an average program. The parent article talks about semiconductor manufacturing, you can save your rants on inefficient and useless programs for another HN article.
    - BubRoss 7y ago
      It's not a 'rant'. This is exactly why speeding up CPUs is more difficult, even with billions of transistors. People expect to write whatever they want and have it run faster. 1% of the transistors in a CPU are actually running instructions, the other 99% are trying to keep it busy.
      - segfaultbuserr 7y ago
        > gameswithgo: Don't use python or javascript It is clearly a rant to me! Don't get me wrong, I'm not a huge fan of JavaScript, but discussing about useless programs is simply not what I'm asking here. I'm not interested in how Google Chrome can be 1000% faster. > People expect to write whatever they want and have it run faster. 1% of the transistors in a CPU are actually running instructions, the other 99% are trying to keep it busy. I agree, now, your perspective makes the talk on software interesting. Recently I've seen an article C Is Not a Low-level Language (https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479), in the article the author argued that the in-order, synchronous, sequential execution model from the PDP-11 heyday is outdated. However vast majority of programs (i.e. C programs) are still written based on this model, so a CPU must use a lot of resources to dispatch these sequential code and introduce countless transparent optimizations (e.g. ILP), to make existing sequential code faster, on the other hand, the compilers are becoming monsters because they must be as intelligent as possible to understand the algorithms in a program and rewrite them automatically for optimum performance on a modern CPU. As a result of this disparity, the capabilities of what the hardware can actually provide is often underutilized. The author purposes that we should try discarding the PDP-11's classical "in-order, synchronous" view of a program, and try developing new programming languages and models that designed with the capabilities of modern hardware in mind, such as low-level parallelism to eliminates this disparity. So the CPUs can focus on what they are good at with less overhead of dispatching the instructions.
- scott_s 7y ago
  I've thought this way for a while. Once (if) we start feeling the diminishing returns in transistor scaling, there will be more incentive in the computer architecture side to experiment with the relatively fixed transistor budget that we have.
- Symmetry 7y ago
  Yes almost certainly we can keep speeding things up, just a lot more slowly than we are speeding things up right now. There's a lot of room for improvements in more specialized single-purpose hardware, more different sorts of cores tuned for different workloads, and academia has a lot of ideas for algorithmic improvements too. But don't expect these to provide the same rapid increase in capabilities that we've seen with transistors shrinking. Eventually we'll have nanotube transistor or photonic or spintronic or nano-electro-mechanical or quantum computers. There are firm physical laws saying what the limits of efficient computation are and we haven't reached them yet.
0xcde4c3db 7y ago
ML's suitability for parallelism mostly comes from the fact that bigger models/state spaces tend to be more powerful. In cases where there no benefit in scaling the problem horizontally, you probably won't have much luck scaling the implementation horizontally either.
kudokatz 7y ago
For those that are interested, I was able to attend a talk by the guy that started Graphcore. His experienced approach to customized chip design for ML/AI is a really great thing to tap into. Some of the themes were: maximize power usage as its a limiting compute factor, remove off-chip memory, and scale horizontally at the individual processor level rather than "sharing resources across cores". https://www.graphcore.ai/ https://www.graphcore.ai/

segfaultbuserr 7y ago

> It may be that we can continue our computational climb for a while longer I have a random question. Imagine this hypothetical scenario: semiconductor manufacturing comes to a halt completely, and new physics and technology is still decades ahead, can we further improve the practical performance of general computation by innovation alone? I imagine, instead of riding on the train of Moore's Law, more resources would be invested to design optimization and R&D of new architectures, e.g. faster FPU, faster pipeline, etc. Also, previously discarded and ignored ideas may be implemented again, and delivers some real advancement. While exponential growth is not possible, at least linear scaling should be the case. For example, computers without clocks. Based on this comment on HN, https://news.ycombinator.com/item?id=19554248 https://news.ycombinator.com/item?id=19554248, one major obstacle of clockless chip is the entire VLSI toolchain is designed and optimized for synchronous logic, and this can be changed if the industry invest some serious resources. Another example comes to my mind is high-level programming based on hardware, e.g. Lisp machine. Does my imagination make any sense?

gameswithgo 7y ago
>can we further improve the practical performance of general computation by innovation alone? we can improve the performance of average programs by a factor of ~100x simply by more careful software engineering. Don't use python or javascript, optimize code for cpu cache hits, use more cores effectively, etc. However games, maybe only 2x if anything.
- segfaultbuserr 7y ago
  I was referring to the computational capabilities that hardware can provide, not the performance of an average program. The parent article talks about semiconductor manufacturing, you can save your rants on inefficient and useless programs for another HN article.
  - BubRoss 7y ago
    It's not a 'rant'. This is exactly why speeding up CPUs is more difficult, even with billions of transistors. People expect to write whatever they want and have it run faster. 1% of the transistors in a CPU are actually running instructions, the other 99% are trying to keep it busy.
    - segfaultbuserr 7y ago
      > gameswithgo: Don't use python or javascript It is clearly a rant to me! Don't get me wrong, I'm not a huge fan of JavaScript, but discussing about useless programs is simply not what I'm asking here. I'm not interested in how Google Chrome can be 1000% faster. > People expect to write whatever they want and have it run faster. 1% of the transistors in a CPU are actually running instructions, the other 99% are trying to keep it busy. I agree, now, your perspective makes the talk on software interesting. Recently I've seen an article C Is Not a Low-level Language (https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479), in the article the author argued that the in-order, synchronous, sequential execution model from the PDP-11 heyday is outdated. However vast majority of programs (i.e. C programs) are still written based on this model, so a CPU must use a lot of resources to dispatch these sequential code and introduce countless transparent optimizations (e.g. ILP), to make existing sequential code faster, on the other hand, the compilers are becoming monsters because they must be as intelligent as possible to understand the algorithms in a program and rewrite them automatically for optimum performance on a modern CPU. As a result of this disparity, the capabilities of what the hardware can actually provide is often underutilized. The author purposes that we should try discarding the PDP-11's classical "in-order, synchronous" view of a program, and try developing new programming languages and models that designed with the capabilities of modern hardware in mind, such as low-level parallelism to eliminates this disparity. So the CPUs can focus on what they are good at with less overhead of dispatching the instructions.
scott_s 7y ago
I've thought this way for a while. Once (if) we start feeling the diminishing returns in transistor scaling, there will be more incentive in the computer architecture side to experiment with the relatively fixed transistor budget that we have.
Symmetry 7y ago
Yes almost certainly we can keep speeding things up, just a lot more slowly than we are speeding things up right now. There's a lot of room for improvements in more specialized single-purpose hardware, more different sorts of cores tuned for different workloads, and academia has a lot of ideas for algorithmic improvements too. But don't expect these to provide the same rapid increase in capabilities that we've seen with transistors shrinking. Eventually we'll have nanotube transistor or photonic or spintronic or nano-electro-mechanical or quantum computers. There are firm physical laws saying what the limits of efficient computation are and we haven't reached them yet.

0xcde4c3db 7y ago

ML's suitability for parallelism mostly comes from the fact that bigger models/state spaces tend to be more powerful. In cases where there no benefit in scaling the problem horizontally, you probably won't have much luck scaling the implementation horizontally either.

kudokatz 7y ago

For those that are interested, I was able to attend a talk by the guy that started Graphcore. His experienced approach to customized chip design for ML/AI is a really great thing to tap into. Some of the themes were: maximize power usage as its a limiting compute factor, remove off-chip memory, and scale horizontally at the individual processor level rather than "sharing resources across cores". https://www.graphcore.ai/ https://www.graphcore.ai/