3 ms·
They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembl
by shaklee3 7y ago
They had an assembler where one person wrote kernels that were faster than cublas in a lot of cases. Afaik, nobody ever released anything else with that assembler, and Nvidia caught up to that performance quickly. In talking with the cublas devs, it seemed more that maxas kernels were highly tuned for specific sizes, whereas cublas/cudnn had to be more general.
Nowadays it's really a moot point with Nvidia's Cutlass being open source.
- choppaface 7y agoTrue story-- the Nervana Maxwell stuff didn't go very far-- but it was noteworthy because they had both that small win as well as their own hardware platform. One other thought about the Nervana-Xeon convergence is that the support for more memory (thru DDR, Optane, or even just mmap'ed NVME) will be a big win for modeling and large-minibatch SGD. For example, the minibatch fetching could be pushed to the hardware / OS instead of Tensorflow (or the crazy guy behind Tensorpack) using a threadpool. A lot of training is still I/O bound at some level, and processors only support so many PCI-e lanes...
- jlebar 7y agoA V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units. bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
- tempguy9999 7y agoThat sounded a bit low. This https://en.wikichip.org/wiki/intel/microarchitectures/cooper_lake https://en.wikichip.org/wiki/intel/microarchitectures/cooper... says "Higher bandwidth (174.84 GiB/s, up from 119.209 GiB/s)" I don't know if memory bandwidth matters for this type of job, though.
- drewg123 7y agoIs that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference. I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
- tempguy9999 7y agoI haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say. To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")
- shaklee3 7y agoIntel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.
- choppaface 7y agoCorrect, GPUs can do SGD iterations much much faster once the data is in GPU RAM. But if you have a ton of data relative to compute, you might be better off with a shallower model and/or fewer iterations. Here’s one start-up serving these sorts of projects: https://www.memverge.com/ https://www.memverge.com/ When GPUs kickstarted deep learning research in 2012, people had already studied shallow models on mapreduce for a decade or so. Once NVME / Optane and modern CPUs get 1-10TB of useful “memory” in the hands of grad students, there should be another wave of new research. To date, my experience has been that 1TB of “memory” is only commonly available in industry.
- amelius 7y agoCan't deep learning be done using 16 bit fixed point instead?
- cbhl 7y agoIf you're using fixed point, wouldn't you just as easily substitute 16-bit integers instead?
- jlebar 7y agoCutlass doesn't match the performance of cublass, even on nvidia's benchmarks. Hand assembly is still alive and well!
- shaklee3 7y agoThat's somewhat nitpicking. Cutlass is 90-95% of cublas in many cases, and there are things Cutlass can do that cublas can't. Cublas will always be marginally faster, but it won't matter in most cases, especially if that means saving a kernel call.