3 ms·

Whether it's substantially better for deep learning depends on whether they chose to neuter fp16 like they did in all other consumer GPUs. The main bottleneck i

Whether it's substantially better for deep learning depends on whether they chose to neuter fp16 like they did in all other consumer GPUs. The main bottleneck is not actually the size of the model per se (sizes of the models have been mostly going _down_ as of late), it's the size of the batch during training. Both batch size and model size determine the size of the main memory hog: activations that you need to keep to do training. You don't need to keep activations to do inference: you can discard them after they're fed into the next layer. So it's not entirely accurate to say that this is useless for DL. What could make it totally kickass for AI would be a proper, non-neutered implementation of fp16, similar to what you'd find in P100, V100 and (surprisingly) Jetson TX2, which would effectively double the possible batch size, which in turn leads to quicker convergence. It could also halve the model size as an added bonus. This is NVIDIA, however, so I would bet good money fp16 is still neutered in "consumer" SKUs.

bitL 8y ago
Of course, for inferencing assuming >100 TFlops on float 16 and >400 TOPS in INT4 as demoed by Jenson would be awesome for already trained/pruned models. Though for training I am not so sure, even V100 have rough edges in doing that with TensorFlow. And for model size - I guess it all depends on area in which are you are focusing; I run out of memory all the time these days even with batches of size 1 :(
saltcured 8y ago
If it's data size in GPU RAM you are concerned about, couldn't you store fp16 and cast into fp32 just in the kernel? In OpenCL, you would do this with vload_halfN and vstore_halfN to convert during load and store. You won't get double throughput compared to fp32, but you shouldn't fall back to some terribly slow path either.
- h4b4n3r0 8y ago
  I haven't looked into it myself, but it could be that due to all this massaging you'd lose more on throughput than you gain on memory use. It's similar to doing 8 bit quantized stuff on general purpose CPUs. It's very hard to make it any faster than float32 due to all the futzing that has to be done before and after the (rather small) computation. In comparison, no futzing at all is needed for float32: load 4/8/16 things at a time (depending on the ISA), do your stuff, store 4/8/16 things at a time. Simple.
  - saltcured 8y ago
    Right, you won't exceed the fp32 calculation rate, unless perhaps it was bandwidth-starved accessing the slowest memory tier. You are doing all fp32 operations after all. What you can do is fit twice the number of array elements into which ever GPU memory layer you decide to store as fp16, so potentially work on half as many blocks in a decomposed problem or twice the practical problem size on a non-decomposed problem. You can separately decide which layer of cache to target for your loads and stores which convert, and then use fp32 with normalized 0-1 range math in that inner tier. You only have to introduce some saturation or rounding if your math heavily depends on the fp16 representation. The load and store routines are vectorized. You load one fp32 SIMD vector from one fp16 SIMD vector of the same size (edit: I mean same number of elements) and vice versa.
    - h4b4n3r0 8y ago
      The point is, you might end up with compute throughput _substantially_ worse than raw fp32. No one will take that tradeoff.
      - saltcured 8y ago
        I have used fp16 buffers frequently on NVIDIA GPUs with OpenGL in generations ranging from GTX 760 (GK104) to Titan X (GM200 and GP102) as well as mobile GPUs like GT 730M (GK208M). I do this for things like ray-casting volume rendering, where the dynamic range, precision, and space tradeoffs are very important. My custom shaders performing texture-mapping and blending are implicitly performing the same underlying half-load and half-store operations to work with these stored formats. The OpenGL shader model is that you are working in normalized fp math and the storage format is hidden, controlled independently with format flags during buffer allocation, so the format conversions are implicit during the loads and stores on the buffers. The shaders on fp16 data perform very well and this is non-sequential access patterns where individual 3 or 4-wide vectors are being loaded and stored for individual multi-channel voxels and pixels. If I remember correctly, I only found one bad case where the OpenGL stack seemed to fail to do this well, and it was something like a 2-channel fp16 buffer where performance would fall off a cliff. Using 1, 3, or 4-channel buffers (even with padding) would perform pretty consistently with either uint8, uint16, fp16, or fp32 storage formats. It's possible they just don't have a properly tuned 2-channel texture sampling routine in their driver, and I've never had a need to explore 2-wide vector access in OpenCL.
voxadam 8y ago
I wasn't aware that the Jetson TX1 and TX2 supported fp16; that's interesting and good to know. Thanks for the heads up.
- h4b4n3r0 8y ago
  IIRC all recent NVIDIA GPUs support fp16, it's just that the perf of fp16 is severely hampered on "consumer" grade hardware, so the memory/perf tradeoff is not viable. I mean, on the one hand I can see why that is: fp16 is not terribly useful in games. But on the other, it's probably nearly the exact same die with a few things disabled here and there to differentiate it from $7K Tesla GPUs, which as an engineer I find super tacky, much like MS deliberately disabling features in Windows Home that don't really need to be disabled.

bitL 8y ago

Of course, for inferencing assuming >100 TFlops on float 16 and >400 TOPS in INT4 as demoed by Jenson would be awesome for already trained/pruned models. Though for training I am not so sure, even V100 have rough edges in doing that with TensorFlow. And for model size - I guess it all depends on area in which are you are focusing; I run out of memory all the time these days even with batches of size 1 :(

saltcured 8y ago

If it's data size in GPU RAM you are concerned about, couldn't you store fp16 and cast into fp32 just in the kernel? In OpenCL, you would do this with vload_halfN and vstore_halfN to convert during load and store. You won't get double throughput compared to fp32, but you shouldn't fall back to some terribly slow path either.

h4b4n3r0 8y ago
I haven't looked into it myself, but it could be that due to all this massaging you'd lose more on throughput than you gain on memory use. It's similar to doing 8 bit quantized stuff on general purpose CPUs. It's very hard to make it any faster than float32 due to all the futzing that has to be done before and after the (rather small) computation. In comparison, no futzing at all is needed for float32: load 4/8/16 things at a time (depending on the ISA), do your stuff, store 4/8/16 things at a time. Simple.
- saltcured 8y ago
  Right, you won't exceed the fp32 calculation rate, unless perhaps it was bandwidth-starved accessing the slowest memory tier. You are doing all fp32 operations after all. What you can do is fit twice the number of array elements into which ever GPU memory layer you decide to store as fp16, so potentially work on half as many blocks in a decomposed problem or twice the practical problem size on a non-decomposed problem. You can separately decide which layer of cache to target for your loads and stores which convert, and then use fp32 with normalized 0-1 range math in that inner tier. You only have to introduce some saturation or rounding if your math heavily depends on the fp16 representation. The load and store routines are vectorized. You load one fp32 SIMD vector from one fp16 SIMD vector of the same size (edit: I mean same number of elements) and vice versa.
  - h4b4n3r0 8y ago
    The point is, you might end up with compute throughput _substantially_ worse than raw fp32. No one will take that tradeoff.
    - saltcured 8y ago
      I have used fp16 buffers frequently on NVIDIA GPUs with OpenGL in generations ranging from GTX 760 (GK104) to Titan X (GM200 and GP102) as well as mobile GPUs like GT 730M (GK208M). I do this for things like ray-casting volume rendering, where the dynamic range, precision, and space tradeoffs are very important. My custom shaders performing texture-mapping and blending are implicitly performing the same underlying half-load and half-store operations to work with these stored formats. The OpenGL shader model is that you are working in normalized fp math and the storage format is hidden, controlled independently with format flags during buffer allocation, so the format conversions are implicit during the loads and stores on the buffers. The shaders on fp16 data perform very well and this is non-sequential access patterns where individual 3 or 4-wide vectors are being loaded and stored for individual multi-channel voxels and pixels. If I remember correctly, I only found one bad case where the OpenGL stack seemed to fail to do this well, and it was something like a 2-channel fp16 buffer where performance would fall off a cliff. Using 1, 3, or 4-channel buffers (even with padding) would perform pretty consistently with either uint8, uint16, fp16, or fp32 storage formats. It's possible they just don't have a properly tuned 2-channel texture sampling routine in their driver, and I've never had a need to explore 2-wide vector access in OpenCL.

voxadam 8y ago

I wasn't aware that the Jetson TX1 and TX2 supported fp16; that's interesting and good to know. Thanks for the heads up.

h4b4n3r0 8y ago
IIRC all recent NVIDIA GPUs support fp16, it's just that the perf of fp16 is severely hampered on "consumer" grade hardware, so the memory/perf tradeoff is not viable. I mean, on the one hand I can see why that is: fp16 is not terribly useful in games. But on the other, it's probably nearly the exact same die with a few things disabled here and there to differentiate it from $7K Tesla GPUs, which as an engineer I find super tacky, much like MS deliberately disabling features in Windows Home that don't really need to be disabled.