5 ms·

I met Naveen Rao after Intel bought Nervana. He seemed pretty adamant about getting stuff shipped fast. In contrast, the Xeon folks own all the politics and s

I met Naveen Rao after Intel bought Nervana. He seemed pretty adamant about getting stuff shipped fast. In contrast, the Xeon folks own all the politics and seem to want the transition to be very gradual. Plus the Phi folks get phased out. They had done a Nervana trial at Facebook but then flaked on other trials. Clearly Intel is trying to desperately manage their books. Having Nervana and friends on a Xeon chip could be a huge positive change for software. Not only could we toss out the issue of GPU memory transfer, but Nvidia GPUs aren’t so great with concurrency, and here with the linux kernel we might have a chance to beat Nvidia. Naveen sure would like that... Nervana once had a Maxwell compiler that was better than Nvidia’s.

namibj 7y ago
maxas still exists, and works for Pascal too. A Volta/Turing assembler like maxas would be nice, but there is no Nervana doing the microbenchmarking to identify the instruction encoding well enough.
shaklee3 7y ago
They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general. Nowadays it's really a moot point with Nvidia's Cutlass being open source.
- choppaface 7y ago
  True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware / OS instead of Tensorflow (or the crazy guy behind Tensorpack) using a threadpool. A lot of training is still I/O bound at some level, and processors only support so many PCI-e lanes...
  - jlebar 7y ago
    A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
    - tempguy9999 7y ago
      That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper_lake https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.
      - drewg123 7y ago
        Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference. I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
        tempguy9999 7y ago
        I haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say. To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")
      - shaklee3 7y ago
        Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.
- jlebar 7y ago
  Cutlass doesn't match the performance of cublass, even on nvidia's benchmarks. Hand assembly is still alive and well!
  - shaklee3 7y ago
    That's somewhat nitpicking. Cutlass is 90-95% of cublas in many cases, and there are things Cutlass can do that cublas can't. Cublas will always be marginally faster, but it won't matter in most cases, especially if that means saving a kernel call.

namibj 7y ago

maxas still exists, and works for Pascal too. A Volta/Turing assembler like maxas would be nice, but there is no Nervana doing the microbenchmarking to identify the instruction encoding well enough.

shaklee3 7y ago

They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general. Nowadays it's really a moot point with Nvidia's Cutlass being open source.

choppaface 7y ago
True story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware / OS instead of Tensorflow (or the crazy guy behind Tensorpack) using a threadpool. A lot of training is still I/O bound at some level, and processors only support so many PCI-e lanes...
- jlebar 7y ago
  A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
  - tempguy9999 7y ago
    That sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper_lake https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.
    - drewg123 7y ago
      Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference. I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
      - tempguy9999 7y ago
        I haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say. To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")
    - shaklee3 7y ago
      Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.
jlebar 7y ago
Cutlass doesn't match the performance of cublass, even on nvidia's benchmarks. Hand assembly is still alive and well!
- shaklee3 7y ago
  That's somewhat nitpicking. Cutlass is 90-95% of cublas in many cases, and there are things Cutlass can do that cublas can't. Cublas will always be marginally faster, but it won't matter in most cases, especially if that means saving a kernel call.