4 ms·

A modern Xeon has 10s of MBs of cache, which has drastically higher memory bandwidth. What matters is how big the matrices (the "model") is not the input data.

by boulos 10y ago

A modern Xeon has 10s of MBs of cache, which has drastically higher memory bandwidth. What matters is how big the matrices (the "model") is not the input data. With native 8-bit or 16-bit support, you can fit an awful lot of model in 30MB. I'd recommend reading this paper about matrix math on GPUs from friends of mine way back in the day: https://graphics.stanford.edu/papers/gpumatrixmult/gpumatrixmult.pdf https://graphics.stanford.edu/papers/gpumatrixmult/gpumatrix... . While NVIDIA has built much larger register files and L2 caches since then, a modern Xeon still is unbeatable when something fits in L2 or even L3 cache. I believe streaming the input data in to update the weights is usually done with non Cache polluting instructions (mm_stream equivalents with the NT hint), so it's not hard to keep it fed.

feelix 10y ago
In general neural nets today are so slow (due being computational resource hungry, in clock cycles, memory, and memory bandwidth) that even 4 of the fastest GPU's money can buy stacked together still take days or weeks to train a production quality model. Trying to peg this onto one of their CPU's will have a result much like they have had in pegging the graphics processing stuff onto their CPU's. That is, very meagre results that is fitting for only the lowest resource intensive things that a consumer may want to do. 30MB to hold a model in is almost nothing. Even 12GB is insufficient to train something like imagenet. But in the same vein as their iGPU implementation, it may be just enough to satisfy the requirements of the average consumer, and still be able to make a large dent in Nvidia's market share.
- spangry 10y ago
  Unless you've got extremely low I/O latency, allowing you to work in smaller batches in parallel without loosing all of your parallelisation speedup to the 'tax' you pay when loading data & instructions from RAM to <insert processing unit here>. Although I have no idea what that implies for the hardware discussed above, as it's way over my head.
modeless 10y ago
SOTA models are much, much larger than 30 MB. You can't use 8 bits for training (yet?), and Intel doesn't support 16-bit float (they need to catch up with Nvidia on that, it's really a travesty, we could have used it years ago for audio and imaging). Besides, the top end Xeons with the most cache are fully 10 times more expensive than a GTX 1080.
- olavgg 10y ago
  Correct, we may have a few MB of text that is training data. But the trained models are several GB's
- varelse 10y ago
  IMO the travesty is crippling FP16 on consumer GPUs for exactly the reasons you cite. And given that AMD's Vega GPUs apparently will possess full speed FP16, it appears to be yet another in a long line of arbitrary performance cripplings for the consumer space that is IMO the most annoying feature of CUDA development. But while FP16 is useful for audio and imaging, FP16 nearly killed NVIDIA over a decade ago* when it lacked the dynamic range for DirectX 9 HDR effects in contemporary games without banding. FP32 was more than enough for the task, but power hungry, and thus NV30 could be used to figuratively fry eggs while AMD GPUs had FP24, which was just enough for these effects. These days, it's all going in the opposite direction w/r to deep learning, but I find it ironic that INT8/INT16 is missing from the Tesla flagship P100, but present on GP102 and GP104, the consumer GPUs (and yes, I know about Tesla P40, but that lacks fast FP16). I agree with the top poster that Intel could make quite a comeback here given how far it's currently behind. I also agree that it would be hard to dethrone NVIDIA without higher bandwidth memory, but I don't think that's necessary, a bloody nose is more than enough to turn heads IMO. https://en.wikipedia.org/wiki/GeForce_FX_series https://en.wikipedia.org/wiki/GeForce_FX_series
raverbashing 10y ago
Models usually don't fit the cache (the MNIST one might fit, but that's a toy model today) Also backpropagation makes you iterate through the whole model at each step

by boulos 10y ago

feelix 10y ago

In general neural nets today are so slow (due being computational resource hungry, in clock cycles, memory, and memory bandwidth) that even 4 of the fastest GPU's money can buy stacked together still take days or weeks to train a production quality model. Trying to peg this onto one of their CPU's will have a result much like they have had in pegging the graphics processing stuff onto their CPU's. That is, very meagre results that is fitting for only the lowest resource intensive things that a consumer may want to do. 30MB to hold a model in is almost nothing. Even 12GB is insufficient to train something like imagenet. But in the same vein as their iGPU implementation, it may be just enough to satisfy the requirements of the average consumer, and still be able to make a large dent in Nvidia's market share.

spangry 10y ago
Unless you've got extremely low I/O latency, allowing you to work in smaller batches in parallel without loosing all of your parallelisation speedup to the 'tax' you pay when loading data & instructions from RAM to <insert processing unit here>. Although I have no idea what that implies for the hardware discussed above, as it's way over my head.

modeless 10y ago

SOTA models are much, much larger than 30 MB. You can't use 8 bits for training (yet?), and Intel doesn't support 16-bit float (they need to catch up with Nvidia on that, it's really a travesty, we could have used it years ago for audio and imaging). Besides, the top end Xeons with the most cache are fully 10 times more expensive than a GTX 1080.

olavgg 10y ago
Correct, we may have a few MB of text that is training data. But the trained models are several GB's
varelse 10y ago
IMO the travesty is crippling FP16 on consumer GPUs for exactly the reasons you cite. And given that AMD's Vega GPUs apparently will possess full speed FP16, it appears to be yet another in a long line of arbitrary performance cripplings for the consumer space that is IMO the most annoying feature of CUDA development. But while FP16 is useful for audio and imaging, FP16 nearly killed NVIDIA over a decade ago* when it lacked the dynamic range for DirectX 9 HDR effects in contemporary games without banding. FP32 was more than enough for the task, but power hungry, and thus NV30 could be used to figuratively fry eggs while AMD GPUs had FP24, which was just enough for these effects. These days, it's all going in the opposite direction w/r to deep learning, but I find it ironic that INT8/INT16 is missing from the Tesla flagship P100, but present on GP102 and GP104, the consumer GPUs (and yes, I know about Tesla P40, but that lacks fast FP16). I agree with the top poster that Intel could make quite a comeback here given how far it's currently behind. I also agree that it would be hard to dethrone NVIDIA without higher bandwidth memory, but I don't think that's necessary, a bloody nose is more than enough to turn heads IMO. https://en.wikipedia.org/wiki/GeForce_FX_series https://en.wikipedia.org/wiki/GeForce_FX_series

raverbashing 10y ago

Models usually don't fit the cache (the MNIST one might fit, but that's a toy model today) Also backpropagation makes you iterate through the whole model at each step