3 ms·

I lean strongly toward your second paragraph. If we can make many core programming easier I think we can dispense with a ton of complexity and just scale out co

by api 8y ago

I lean strongly toward your second paragraph. If we can make many core programming easier I think we can dispense with a ton of complexity and just scale out cores with transistor counts. I think an under researched area is the application of deep learning to compilers. What could be done there for parallelism? We are still kind of stuck in programming models developed back when core frequencies were scaling geometrically. We hit that wall in the early 2000's and still haven't quite assimilated it. We are tackling it at the macro scale of multiple discrete systems via modern devops but the systems (e.g. Kubernetes) are baroque, clunky, and hard to manage.

deepnotderp 8y ago
Regarding deep learning for compilers, at Vathys we're using deep learning in compilers for deep learning code and it works well, achieving pretty good results even compared to human implementations. But it's not that much of a gain and it can often get stuck on some code, especially if it's sufficiently new code. Just as surprisingly, genetic algorithms are almost as good but faster and more robust. Finally, I think people point to examples of much faster hardware accelerated workloads and optimized assembly code that share these characteristics: 1. Are relatively embarrassingly parallel 2. Are relatively control flow light 3. Generally tend to exhibit high compute/bandwith ratios. Hence, you see deep learning hardware, bitcoin mining hardware, optimized compression codec assembly, etc. pointed to as examples of acceleration, but this doesn't apply to all workloads, e.g. a web app.
jcranmer 8y ago
> I think an under researched area is the application of deep learning to compilers. What could be done there for parallelism? I don't think it's under-researched, because we already know that it's only ever going to do a lousy job. The problem of autoparallelization is mostly stymied by difficulty in computing the legality of transformations, and secondarily by the problem of tuning parameters [1]. We've realized that it's easier to just get the users to tell us about these things, sidestepping the problem for the most part. > We are still kind of stuck in programming models developed back when core frequencies were scaling geometrically. The programming model for parallelism is dataflow. We already know that: the tricky thing of parallelism is communication, so annotating all of the edges of communication makes the scheduling problem much, much easier. [1] Theoretically, machine learning is good at tuning parameters. In practice, you get good results with the dumbest algorithms already, and the available headroom for performance is generally swamped by the fact that small changes such as function size can dramatically alter the performance characteristics of code elsewhere in the application.

by api 8y ago

deepnotderp 8y ago

Regarding deep learning for compilers, at Vathys we're using deep learning in compilers for deep learning code and it works well, achieving pretty good results even compared to human implementations. But it's not that much of a gain and it can often get stuck on some code, especially if it's sufficiently new code. Just as surprisingly, genetic algorithms are almost as good but faster and more robust. Finally, I think people point to examples of much faster hardware accelerated workloads and optimized assembly code that share these characteristics: 1. Are relatively embarrassingly parallel 2. Are relatively control flow light 3. Generally tend to exhibit high compute/bandwith ratios. Hence, you see deep learning hardware, bitcoin mining hardware, optimized compression codec assembly, etc. pointed to as examples of acceleration, but this doesn't apply to all workloads, e.g. a web app.

jcranmer 8y ago

> I think an under researched area is the application of deep learning to compilers. What could be done there for parallelism? I don't think it's under-researched, because we already know that it's only ever going to do a lousy job. The problem of autoparallelization is mostly stymied by difficulty in computing the legality of transformations, and secondarily by the problem of tuning parameters [1]. We've realized that it's easier to just get the users to tell us about these things, sidestepping the problem for the most part. > We are still kind of stuck in programming models developed back when core frequencies were scaling geometrically. The programming model for parallelism is dataflow. We already know that: the tricky thing of parallelism is communication, so annotating all of the edges of communication makes the scheduling problem much, much easier. [1] Theoretically, machine learning is good at tuning parameters. In practice, you get good results with the dumbest algorithms already, and the available headroom for performance is generally swamped by the fact that small changes such as function size can dramatically alter the performance characteristics of code elsewhere in the application.