5 ms·
The trade-off can be that you're not as free in designing the architecture of your model, and you have to train it in a special way. This is actually a huge obs
by WithinReason 4y ago
The trade-off can be that you're not as free in designing the architecture of your model, and you have to train it in a special way. This is actually a huge obstacle to making neural networks more efficient. You could design HW that would run networks 10x-100x more efficiently, but the networks would have to be specially trained for the HW, and the SW investment to create the SW infrastructure to go with your HW is 10x the effort of designing the HW. Nvidia knows this in the area of GPUs, and they have 10x as many SW engineers than HW engineers. Neural network HW startups have yet to learn it.
- m3kw9 4y agoCan anyone explain further as a simple example?
- ip26 4y agoNeural networks are commonly evaluated with matrix multiplication. A matrix multiply is orders of magnitude faster if all three matrices fit in your caches. So you want to tune the matrix size to the hardware.
- WithinReason 4y agoIn addition, once you did that (tuned your hardware to a matrix size) it can be inefficient to use smaller matrices. So even if you can make a network that uses smaller components the HW will not gain the corresponding speedup.
- zasdffaa 4y agoMatMult = the dot product, effectively. I can't believe that allowing smaller vectors is difficult or even non-trivial in hardware.
- svnt 4y agoYou’re assuming caches know anything about dot products.
- zasdffaa 4y agoThat seems a non-sequiteur to me. Can you elaborate?
- WithinReason 4y agoGPUs are a good example: SIMD means all 128 threads do the same work. If only 1 thread has work to do the other 127 threads still take up resources.
- zasdffaa 4y agoAh, right, gotcha
- deleted 4y ago[deleted]
- carschno 4y ago"The hardware lottery" describes roughly the same phenomenon: https://hardwarelottery.github.io/ https://hardwarelottery.github.io/
- ComplexSystems 4y agoThis is one reason why I wish there were more interest in FPGAs.
- rbanffy 4y agoThe idea of DRAM with lots and lots of tiny FPGAs is an intriguing idea. I wish I were a chip designer to run with it and see where it got me.
- mlazos 4y agoIMO FPGAs sound like they’d be good with extra flexibility but in practice ASICs are just better. Higher frequency, lower power and there just isn’t a way to make up for the frequency gap. You can quantize and do fancier stuff, but then you need to fine tune or train on your actual inference HW to ensure you’re actually correct. You can emulate your quantization scheme and train on GPUs, but then your GPU training will be slow af. Building a true model specific bitstream for an FPGA that is actually utilizing all of the resources of the FPGA is not simple, and even if you do, overcoming the frequency gap is hard.