5 ms·
18GB is huge! An NVIDIA V100 has 6MB of L2 memory. HBM is off-chip, and vastly (~100x) slower.
by Veedrac 7y ago
18GB is huge! An NVIDIA V100 has 6MB of L2 memory. HBM is off-chip, and vastly (~100x) slower.
- baybal2 7y ago18GB of very fast memory will still be just as hard to keep fed with data as that 6MB cache
- Veedrac 7y agoThe idea is that the whole model resides in the fast memory, so you don't need to ‘keep it fed’.
- dmitrygr 7y ago44K/core is very little memory
- Veedrac 7y agoIndeed, but cores are only responsible for small fragments of the network, so don't need huge amounts of memory.
- dmitrygr 7y agoUnless you need to multiply large matrices, where you need access to very large rows and columns...like in...ML applications
- Veedrac 7y agoThat's what the absurdly fast interconnect is for. You send the data to where the weights are.
- dmitrygr 7y agoAbsurdly fast != Single cycle It will be physically impossible to access that much memory single cycle at anything approaching reasonable speeds. I suppose you could do it at 5Hz :)
- Veedrac 7y agoA core receives data over the interconnect. It uses its fast memory and local compute to do its part of the matrix multiplication. It streams the results back out when it's done. The interconnect doesn't give you single-cycle access to the whole memory pool, but it doesn't need to.
- dmitrygr 7y agoI think it is telling that in one sentence there is a claim that it is faster than nvidia, and in another, a claim it does tensor flow. I do not think this architecture could do both of these at once. It could not do tensor flow fast enough (not enough local fast mem) to compete even with a moderate array of GPUs
- deepnotderp 7y agoHey! You seem knowledgeable, mind emailing me at tapabrata_ghosh [at] vathys (dot) ai ?
- bmh 7y agoThat's true, but it doesn't match their claim of keeping all of the model on the chip. An 18 GB cache is huge for sure, but that's not what they claim.