3 ms·

Without GPU support I don't see how this is a good idea, if we are talking about HPC applications. Based on my understanding, a large number of dense matrix ope

by n00b101 13y ago

Without GPU support I don't see how this is a good idea, if we are talking about HPC applications. Based on my understanding, a large number of dense matrix operations can be significantly accelerated in GPUs.

poulson 13y ago
Accelerator support is something that happens within the node and is in some sense orthogonal to the high-level design. I recently received funding to add such support to the library (and I hope to add it within the next year). Also, not all supercomputers have accelerators (consider Blue Gene/Q), and often simply having access to more memory is more of a concern than solving the problem at the absolute fastest rate.
dmlorenzetti 13y ago
This targets distributed-memory applications, so it's not clear why failing to support GPUs with the first iteration makes it a bad idea. I imagine the authors could make a single node of the cluster use a GPU, if they wanted.
- n00b101 13y ago
  Maybe I should have just said that I think GPU support would be a great addition. Distributed memory and GPUs are not mutually exclusive. Multi-GPU clusters are extremely common. In fact the latest devices (e.g. Tesla K10) have multiple GPU processors packaged in a single card, so it is necessary for applications to target multiple GPUs. There is explicit support for distributed-memory applications in GPUs through the "GPUDirect" technology that allows peer-to-peer DMA and RDMA transfers between GPUs. Given that reports of 30-50x GPU performance gains (versus CPUs) are common, the issue is important because it means solving a problem with (say) $10,000 of kit instead of $500,000.
  - jedbrown 13y ago
    30x to 50x claims are the result of a terrible CPU implementation. When normalized by power consumption (e.g., TDP) or by acquisition cost (assume high-end rather than consumer versions, as with all supercomputing centers), you'll find about a 2x advantage for operations like DGEMM and much less for other operations. Strong scaling is important for many super-computing applications and accelerators like GPUS and Intel MIC are much slower for that purpose. For examples, look at the MKL benchmarks for Xeon Phi (http://software.intel.com/en-us/intel-mkl/ http://software.intel.com/en-us/intel-mkl/) and normalize as 3 Sandy Bridge sockets per Xeon Phi (common configurations pair one SB socket with one Xeon Phi, the Phi has more than twice the TDP and costs more than twice as much). Don't forget to look at QR and Cholesky, for which the Phi at best breaks even, but only for enormous matrix sizes.
    - poulson 13y ago
      Agreed. Accelerators can certainly lead to speedups, but it's hard to overemphasize how important it is to also have an optimized vanilla pure-MPI implementation (if nothing else, to compare the 'accelerated' version to).
    - m_mueller 13y ago
      It may sound strange, but one of the advantages of GPUs is actually the programming model, hence why many applications suddenly perform 20-50x faster instead of only the theoretical 5-7x. Let me explain: One of the most common issues with x86 HPC applications coming from the scientific crowd is a lack of vector optimization such as loop unrolling. Even having the right compiler flags is rather difficult for this kind of thing. Another reason is a lack of understanding on how to program for memory bandwidth optimization. GPU programming on the other hand, especially with CUDA, is hard to get into at first, but once you have the right formula you can apply it pretty easily to most common tasks. Getting to, say, 70% of model performance on GPU is much easier than on x86. One reason is the implicitly bandwidth optimized idiomatic way of writing CUDA/OpenCL programs as a set of scalar kernels applied over a whole data region - this allows the programmer to think of block dimensions in an abstract way - no need to fiddle around manually with loops to achieve this. There is also no need to use any intrinsics, just plain C in idiomatic CUDA is enough. So, to wrap it up, there is more to GPU programming than just the hardware itself, the software model actually makes a lot more sense than traditional OOP/procedural programming for HPC - often resulting in higher than expected speedups when going from idiomatic x86 to idiomatic GPGPU (since there is no such thing as easy to learn idioms for HPC x86 programming). And btw. Xeon Phi is the result of Intel not understanding exactly this interrelation, since it doesn't even have OpenCL support as of now.

by n00b101 13y ago

poulson 13y ago

Accelerator support is something that happens within the node and is in some sense orthogonal to the high-level design. I recently received funding to add such support to the library (and I hope to add it within the next year). Also, not all supercomputers have accelerators (consider Blue Gene/Q), and often simply having access to more memory is more of a concern than solving the problem at the absolute fastest rate.

dmlorenzetti 13y ago

This targets distributed-memory applications, so it's not clear why failing to support GPUs with the first iteration makes it a bad idea. I imagine the authors could make a single node of the cluster use a GPU, if they wanted.

n00b101 13y ago
Maybe I should have just said that I think GPU support would be a great addition. Distributed memory and GPUs are not mutually exclusive. Multi-GPU clusters are extremely common. In fact the latest devices (e.g. Tesla K10) have multiple GPU processors packaged in a single card, so it is necessary for applications to target multiple GPUs. There is explicit support for distributed-memory applications in GPUs through the "GPUDirect" technology that allows peer-to-peer DMA and RDMA transfers between GPUs. Given that reports of 30-50x GPU performance gains (versus CPUs) are common, the issue is important because it means solving a problem with (say) $10,000 of kit instead of $500,000.
- jedbrown 13y ago
  30x to 50x claims are the result of a terrible CPU implementation. When normalized by power consumption (e.g., TDP) or by acquisition cost (assume high-end rather than consumer versions, as with all supercomputing centers), you'll find about a 2x advantage for operations like DGEMM and much less for other operations. Strong scaling is important for many super-computing applications and accelerators like GPUS and Intel MIC are much slower for that purpose. For examples, look at the MKL benchmarks for Xeon Phi (http://software.intel.com/en-us/intel-mkl/ http://software.intel.com/en-us/intel-mkl/) and normalize as 3 Sandy Bridge sockets per Xeon Phi (common configurations pair one SB socket with one Xeon Phi, the Phi has more than twice the TDP and costs more than twice as much). Don't forget to look at QR and Cholesky, for which the Phi at best breaks even, but only for enormous matrix sizes.
  - poulson 13y ago
    Agreed. Accelerators can certainly lead to speedups, but it's hard to overemphasize how important it is to also have an optimized vanilla pure-MPI implementation (if nothing else, to compare the 'accelerated' version to).
  - m_mueller 13y ago
    It may sound strange, but one of the advantages of GPUs is actually the programming model, hence why many applications suddenly perform 20-50x faster instead of only the theoretical 5-7x. Let me explain: One of the most common issues with x86 HPC applications coming from the scientific crowd is a lack of vector optimization such as loop unrolling. Even having the right compiler flags is rather difficult for this kind of thing. Another reason is a lack of understanding on how to program for memory bandwidth optimization. GPU programming on the other hand, especially with CUDA, is hard to get into at first, but once you have the right formula you can apply it pretty easily to most common tasks. Getting to, say, 70% of model performance on GPU is much easier than on x86. One reason is the implicitly bandwidth optimized idiomatic way of writing CUDA/OpenCL programs as a set of scalar kernels applied over a whole data region - this allows the programmer to think of block dimensions in an abstract way - no need to fiddle around manually with loops to achieve this. There is also no need to use any intrinsics, just plain C in idiomatic CUDA is enough. So, to wrap it up, there is more to GPU programming than just the hardware itself, the software model actually makes a lot more sense than traditional OOP/procedural programming for HPC - often resulting in higher than expected speedups when going from idiomatic x86 to idiomatic GPGPU (since there is no such thing as easy to learn idioms for HPC x86 programming). And btw. Xeon Phi is the result of Intel not understanding exactly this interrelation, since it doesn't even have OpenCL support as of now.