4 ms·

Hardware for Deep Learning. Part 4: ASIC

90 points by lelf 6y ago

ipsum2 6y ago
> Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons. Instead, the company prefers to let customers try out the CS-1 using their own neural networks and data. I suspect that despite Cerebras being a massive technical achievement (wafer-scale computing!) it performs worse than standard GPUs, which is why they won't release benchmarks on standard models (e.g. resnet/transformers/etc)
2sk21 6y ago
Looking at the TPU architecture was fascinating for an old-timer like me. I looked into implementing backpropagation on a Connection Machine CM2 back in the late 1980s. I got to the point of implementing the training algorithm in C* (an SIMD variant of C). However, I ultimately decided in favor of using a shared memory multi-processor. The main problem was that time on my unversity's CM2 was very hard to come by.
syntaxing 6y ago
Is there any particular options for a hobbyists?
deleted 6y ago
[deleted]
37ef_ced3 6y ago
In 2018 Pete Bannon showed me the first prototype of Tesla's deep learning ASIC (for training). It was sitting out on a desk, attached to a big heat-sink, near Elon Musk's empty chair in Tesla's Palo Alto office They were having trouble writing a compiler for their custom hardware. So I asked, why not just use a GPU, and use Nvidia's libraries and compilers? Bannon became angry and defensive, and insisted that Tesla couldn't become dependent on Nvidia So that's another reason to build your own hardware
- syntaxing 6y ago
  Do you think it was the right choice? Like did the custom hardware save money and make them more competitive? They announced that their board can do 144 TFlops but TFlops is such a bad gauge on performance.
- ttul 6y ago
  The math for ASICs works out if you’re going to mint hundreds of thousands of chips. Tesla is building 500K cars per year, so they definitely cross the threshold for economic viability. The NRE costs for the best TSMC node is easily >$10M, but that is peanuts for Tesla against 500,000 units. Source: I once spent a year analyzing custom ASICs for Bitcoin mining.
  - darksaints 6y ago
    But the parent was talking about training. I don't think tesla is training machine learning models in their cars.
    - nine_k 6y ago
      Still it's $20 per car for the cost of training hardware, no matter how many units of this hardware is required, within reason. This is a rather acceptable cost. And they may potentially have an upper hand in development of such ASICs, in-house tools for them, etc. Of course there is a risk that the results won't be impressive. Likely it's a calculated risk.

90 points by lelf 6y ago

> Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons. Instead, the company prefers to let customers try out the CS-1 using their own neural networks and data. I suspect that despite Cerebras being a massive technical achievement (wafer-scale computing!) it performs worse than standard GPUs, which is why they won't release benchmarks on standard models (e.g. resnet/transformers/etc)

2sk21 6y ago

Looking at the TPU architecture was fascinating for an old-timer like me. I looked into implementing backpropagation on a Connection Machine CM2 back in the late 1980s. I got to the point of implementing the training algorithm in C* (an SIMD variant of C). However, I ultimately decided in favor of using a shared memory multi-processor. The main problem was that time on my unversity's CM2 was very hard to come by.

syntaxing 6y ago

Is there any particular options for a hobbyists?

deleted 6y ago

[deleted]

37ef_ced3 6y ago

In 2018 Pete Bannon showed me the first prototype of Tesla's deep learning ASIC (for training). It was sitting out on a desk, attached to a big heat-sink, near Elon Musk's empty chair in Tesla's Palo Alto office They were having trouble writing a compiler for their custom hardware. So I asked, why not just use a GPU, and use Nvidia's libraries and compilers? Bannon became angry and defensive, and insisted that Tesla couldn't become dependent on Nvidia So that's another reason to build your own hardware

syntaxing 6y ago
Do you think it was the right choice? Like did the custom hardware save money and make them more competitive? They announced that their board can do 144 TFlops but TFlops is such a bad gauge on performance.
ttul 6y ago
The math for ASICs works out if you’re going to mint hundreds of thousands of chips. Tesla is building 500K cars per year, so they definitely cross the threshold for economic viability. The NRE costs for the best TSMC node is easily >$10M, but that is peanuts for Tesla against 500,000 units. Source: I once spent a year analyzing custom ASICs for Bitcoin mining.
- darksaints 6y ago
  But the parent was talking about training. I don't think tesla is training machine learning models in their cars.
  - nine_k 6y ago
    Still it's $20 per car for the cost of training hardware, no matter how many units of this hardware is required, within reason. This is a rather acceptable cost. And they may potentially have an upper hand in development of such ASICs, in-house tools for them, etc. Of course there is a risk that the results won't be impressive. Likely it's a calculated risk.