4 ms·

I wonder how Tesla's Dojo supercomputer is doing. I don't recall hearing any details about its progress lately. Maybe we'll learn something new from the Hot Ch

I wonder how Tesla's Dojo supercomputer is doing. I don't recall hearing any details about its progress lately. Maybe we'll learn something new from the Hot Chips event mentioned in TFA.

giuliomagnifico 4y ago
Interested me too. At the moment is the number 7 of the world Supercomputer by number of GPUs.
4eleven7 4y ago
If not, then we'll likely learn more at Tesla AI day #2, on September 30th.
sigmoid10 4y ago
Tesla will be at the Machine Learning and Systems conference in 2 weeks and if not then we'll probably hear about the Dojo D1 chip at Tesla AI day which is one month later.
dragontamer 4y ago
It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia, or whoever else became #1 at neural net hardware. New chips are always fun to see and get discussed, but there are simple economic / mass design issues at play here. The Tesla custom chip just won't be made at the volume of NVidia or AMD. And Intel's attempt to join the market seems like the better approach (even if it fails). You need to get the video game crowd to fund the research and development these days. Video gamers want high speed SIMD for faster graphics. Without this source of money (and millions of chips sold per year), it looks difficult to sustainably upgrade an architecture. -------- If not the PC market, then at least try to get the console market or phone market or some other consumer electronics fad wrapped up in things.
- Geee 4y ago
  Making a chip for your own use should be more optimal than a chip made for a wide range of customers. Also, you don't need to pay the hefty markup for NVidia. If you need a lot of chips, there's a point on the curve where it becomes more economical to design your own chip.
  - titoCA321 4y ago
    You have to invest into the people and the hardware and software to make the chip. And then you have to support that chip throughout the years. There's reason the U.S Government got out of making their own chips and laying their own fiber-cables.
  - jacquesm 4y ago
    Tesla isn't even close to those kind of volumes.
- sanxiyn 4y ago
  Google seems to be doing well with TPU without being funded by video game crowd.
  - sidibe 4y ago
    Google has a ton of internal and external customers that can amortize the cost of this kind of project. They've also got lots of similar programs of developing server hardware and some organizational experience in how to hook new stuff up and keep it utilized. The whole dojo program always seemed wildly ambitious reach to me for a company like Tesla. More power to them if they pull it off but it's not a surprise to me that it's not what they're using a year after they thought it was almost ready
    - vardump 4y ago
      If Tesla's chip is any good, perhaps they can also sell access to Dojo supercomputer.
- sailingparrot 4y ago
  > It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia > The Tesla custom chip just won't be made at the volume of NVidia or AMD. I don't think Tesla ever aimed to produce a chip for the public, it's purely to be used in their own products, so they don't have to be competitive with NVIDIA in flops/$ or worry about keeping up with NVIDIA production rate. The chip only needs to be better, for the specific task they care about, for a specific metric they care about. Considering that NVIDIA has to make a card that works for everyone, they have to make a huge number of tradeoffs, such as supporting many different number formats going from FP64 to INT4, their cores have to support a ton of different instructions, they need to keep a large amount of space on the chip for large memory buffers to support training of models with billions of parameters, they need to support GPUs interconnect for clustering etc. etc. Given the importance of real estate on a chip, those tradeoff have large implications. Tesla has 10x less constraints: * For the training chip they only support their custom 8 and 16 bits format iic, so they can redistribute cores in consequence. They can ditch many instructions that they'll never use. They can size the memories to be exactly what they need, which is probably less than 80GB needed for large transformers, and use the freed space to add more cores etc. * For the inference chip, they can additionally ditch all the interconnects as they just need one per car and reduce memory drastically freeing up even more space for cores. If NVIDIA didn't have to balance so many different use cases on such a small area, then yes it would probably make no sense to try to do your own thing. But since they have to be general purpose, if you target a very specific use case you can be far less knowledgeable than them in GPU architecture and still come out ahead for your use case. Specificity is the same reason why it still possible to write CUDA kernels for some basic operation that will run 3x faster than the official NVIDIA implementation that was done by some amazing engineers: they have to make their operations efficient for every input size, you don't.
  - dragontamer 4y ago
    Except this is an article about how Tesla has bought 7000+ Nvidia A100 GPUs in their latest supercomputer, rather than their D1 Dojo custom chip. ----- EDIT: And ~7000 chips seems too small for a custom-run of chips. Any "supercomputer" Tesla builds has to be much much larger than this before a custom-chip project likely becomes cost-efficient.

I wonder how Tesla's Dojo supercomputer is doing. I don't recall hearing any details about its progress lately. Maybe we'll learn something new from the Hot Chips event mentioned in TFA.

giuliomagnifico 4y ago

Interested me too. At the moment is the number 7 of the world Supercomputer by number of GPUs.

4eleven7 4y ago

If not, then we'll likely learn more at Tesla AI day #2, on September 30th.

sigmoid10 4y ago

Tesla will be at the Machine Learning and Systems conference in 2 weeks and if not then we'll probably hear about the Dojo D1 chip at Tesla AI day which is one month later.

dragontamer 4y ago

It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia, or whoever else became #1 at neural net hardware. New chips are always fun to see and get discussed, but there are simple economic / mass design issues at play here. The Tesla custom chip just won't be made at the volume of NVidia or AMD. And Intel's attempt to join the market seems like the better approach (even if it fails). You need to get the video game crowd to fund the research and development these days. Video gamers want high speed SIMD for faster graphics. Without this source of money (and millions of chips sold per year), it looks difficult to sustainably upgrade an architecture. -------- If not the PC market, then at least try to get the console market or phone market or some other consumer electronics fad wrapped up in things.

Geee 4y ago
Making a chip for your own use should be more optimal than a chip made for a wide range of customers. Also, you don't need to pay the hefty markup for NVidia. If you need a lot of chips, there's a point on the curve where it becomes more economical to design your own chip.
- titoCA321 4y ago
  You have to invest into the people and the hardware and software to make the chip. And then you have to support that chip throughout the years. There's reason the U.S Government got out of making their own chips and laying their own fiber-cables.
- jacquesm 4y ago
  Tesla isn't even close to those kind of volumes.
sanxiyn 4y ago
Google seems to be doing well with TPU without being funded by video game crowd.
- sidibe 4y ago
  Google has a ton of internal and external customers that can amortize the cost of this kind of project. They've also got lots of similar programs of developing server hardware and some organizational experience in how to hook new stuff up and keep it utilized. The whole dojo program always seemed wildly ambitious reach to me for a company like Tesla. More power to them if they pull it off but it's not a surprise to me that it's not what they're using a year after they thought it was almost ready
  - vardump 4y ago
    If Tesla's chip is any good, perhaps they can also sell access to Dojo supercomputer.
sailingparrot 4y ago
> It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia > The Tesla custom chip just won't be made at the volume of NVidia or AMD. I don't think Tesla ever aimed to produce a chip for the public, it's purely to be used in their own products, so they don't have to be competitive with NVIDIA in flops/$ or worry about keeping up with NVIDIA production rate. The chip only needs to be better, for the specific task they care about, for a specific metric they care about. Considering that NVIDIA has to make a card that works for everyone, they have to make a huge number of tradeoffs, such as supporting many different number formats going from FP64 to INT4, their cores have to support a ton of different instructions, they need to keep a large amount of space on the chip for large memory buffers to support training of models with billions of parameters, they need to support GPUs interconnect for clustering etc. etc. Given the importance of real estate on a chip, those tradeoff have large implications. Tesla has 10x less constraints: * For the training chip they only support their custom 8 and 16 bits format iic, so they can redistribute cores in consequence. They can ditch many instructions that they'll never use. They can size the memories to be exactly what they need, which is probably less than 80GB needed for large transformers, and use the freed space to add more cores etc. * For the inference chip, they can additionally ditch all the interconnects as they just need one per car and reduce memory drastically freeing up even more space for cores. If NVIDIA didn't have to balance so many different use cases on such a small area, then yes it would probably make no sense to try to do your own thing. But since they have to be general purpose, if you target a very specific use case you can be far less knowledgeable than them in GPU architecture and still come out ahead for your use case. Specificity is the same reason why it still possible to write CUDA kernels for some basic operation that will run 3x faster than the official NVIDIA implementation that was done by some amazing engineers: they have to make their operations efficient for every input size, you don't.
- dragontamer 4y ago
  Except this is an article about how Tesla has bought 7000+ Nvidia A100 GPUs in their latest supercomputer, rather than their D1 Dojo custom chip. ----- EDIT: And ~7000 chips seems too small for a custom-run of chips. Any "supercomputer" Tesla builds has to be much much larger than this before a custom-chip project likely becomes cost-efficient.