9 ms·

Tesla Dojo Custom AI Supercomputer at HC34

77 points by belltaco 4y ago

extheat 4y ago
Cool. Can anyone chime in on how this compares to other ML SoC/ASICs? I know many places are going hard on general purpose GPUs, but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward.
- dragontamer 4y ago
  They'll be at a process disadvantage. D1 is allegedly TSMC 7nm, as per last year's information (https://www.tomshardware.com/news/tesla-d1-ai-chip https://www.tomshardware.com/news/tesla-d1-ai-chip). An ASIC can strip out features they don't need and save some space. But a good chunk of modern GPUs are memory-controllers, registers, and SIMD-cores. And modern GPUs (both AMD's MI250x and NVidia's A100) have 16-bit matrix multiplication units (aka: Tensor cores). Once we factor in the process disadvantage, I'm not sure if the D1 will be as competitive as they hope. Tesla's hope is that their D1 chip has more 16-bit matrix multiplication cores than the NVidia / AMD designs. But A100 is quite solid, and NVidia Hopper has been announced at HC34 (aka: NVidia's next generation). https://www.nvidia.com/en-us/technologies/hopper-architecture/ https://www.nvidia.com/en-us/technologies/hopper-architectur... ------- Most of this presentation on the Tesla Dojo is about the interconnect system. Alas, NVidia's on like the 4th (or was it 5th?) generation of NVlink, available from their DGX servers (and I'm sure a Hopper version will come out soon). AMD's not far behind, also with a lot of good presentations this year from HC34 that points out how AMD's "Frontier" Supercomputer has huge bandwidth. In particular, each MI250X GPU is a twin-chiplet design (two GPUs per... GPU), with 5 high-speed links to connect to other GPUs in a high-speed fashion. There's a reason why Frontier is the #1 supercomputer of the world right now, in both absolute Double-precision FLOPs, and in Green Double-precision FLOPS-per-watt. NVidia's Hopper will be hitting 4nm. AMD's MI250x is 5nm. That means the D1 chip has less than 1/2 the transistors at the same area compared to NVidia. > but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward. Only if you keep up with the process shrinks. 7nm is getting long in the tooth now. All eyes are looking forward to 5nm, 4nm, and even 3nm designs (now that Apple is the customer of TSMC's 3nm node). ----------- That being said, if the 7nm node is cheaper, maybe this exercise was still cost-effective for Tesla. As the newer nodes obsolete the older nodes, the older nodes become more cost-effective. Cost-efficiency is less popular / less cool, but still an effective business plan. > but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward. The issue is that it probably costs hundreds-of-millions of dollars to design something like the D1. Sure, the mass production of the chip afterwards will be incredible, but chips have stupidly high startup costs (masking, engineering, research, etc. etc.) GPUs on the other hand, are more general purpose and are applicable to more situations. So you can sell the GPU to more customers and spread out the R&D costs. In particular, GPUs capture the attention of the video game crowd, who will fund high-end GPU research just to play video games. Much like how Intel's laptops allow servers to share the R&D effort, so too does NVidia's consumer GPUs share research/development costs with their high end A100 cards.
  - Robotbeat 4y ago
    Nvidia’s stuff is good but it’s pretty high margin. They don’t give access to it for cheap,& in the last five years, the cost per unit performance has been nearly flat. They’re also more generalized than Tesla needs. Performance advantages from process shrinks have also stagnated. A good time for a custom approach.
    - dragontamer 4y ago
      NVidia is claiming 1000 16-bit Tensor-FLOPS on Hopper: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-hopper-architecture... While Tesla is claiming less than 400 Tensor-FLOPS on D1. So yeah, the claims of NVidia's GH100 / Hopper GPU are an order of magnitude faster than the D1. Which is no surprise, because when your transistors are less than 1/2 the size of the competition, you can easily have 2x the performance in a embarrassingly parallel problem. -------- Note that the A100, released in 2020, offers 312 TFlops of 16-bit Tensor matrix-multiplication operations per second. Meaning D1's chip is barely competitive against the 2-year-old NVidia A100, let alone the next-generation Hopper. And note that NVidia's server-GPUs (like A100 or GH100) already come in prepackaged supercomputer formats with extremely-high speed data-links between them. See the DGX line of NVidia supercomputers. https://www.nvidia.com/en-us/data-center/dgx-station-a100/ https://www.nvidia.com/en-us/data-center/dgx-station-a100/ -------- You can't beat physics. Smaller transistors use less power, while more transistors offer more parallelism. A process-node advantage is huge.
      - Robotbeat 4y ago
        The transistors aren’t always literally smaller with each processor node, and there are negative quantum effects that also occur as you shrink. But anyway, you dodged my entire point about cost-to-performance ratio by looking just at performance. If NVidia is insisting on pocketing all the performance advantages of the process shrink as profit, then it still make sense for Tesla to do this.
        dragontamer 4y ago
- sinenomine 4y ago
  It's a very focused and impressive effort. Training from SRAM can be one-two orders of magnitude faster than training from DRAM. Modern GPUs, even with some process advantage (and no shipments her, H100) strive to be general purpose processors too much, and that caps their peak deep learning performance. Every serious player will have to make or buy their own TPU.
- Patrick-STH 4y ago
  This is a crazy oversimplification, but let me take a shot. The difference to others is easily a large "novel size" discussion. Most training SoCs are focused on building something the size of an NVIDIA GPU, but designed for ML versus general purpose GPU HPC compute (FP64) plus ML. Often those accelerators today have a few types of models they are optimized for. NVIDIA is the baseline and so the competitors are looking for areas where they can get a large boost at a lower cost with something about the size of a A100/H100. Cerebras is perhaps the biggest exception with its WSE-2, a wafer size chip. Having the wafer size chip means that Cerebras does not need to go into higher-latency and higher-power off-package interconnect as frequently because its chip is 50x larger. In turn, Cerebras drives performance and cost savings by not needing NVLink4 NVSwitches / InfiniBand. Tesla's Dojo Tile is 25 chips roughly equivalent in size to a NVIDIA GPU in a single package with die-to-die communication facilitated by the base tile and then built for scale up units. Tesla also has focused on the interconnect and pipeline feeding the D1s and Tiles. Ultimately, I think that it takes something beyond a "solution X saves 30% over NVIDIA in these workloads in performance/ $" to survive. NVIDIA has a massive software ecosystem and can handle more types of tasks versus some of the other AI accelerators. That goes beyond just the training and also to other parts of the data prep and movement pipeline. NVIDIA extracts high margins from this work so that is why some effectively are competing with "it costs less and on some problems can be faster" architectures but what Tesla, Cerebras, Google, and a few others have another level of differentiation. Nothing is perfect, nor was that explanation, but just a high-level view of why the technology featured is impactful.
jeffbee 4y ago
They recycled a lot of material from last year's presentation. Tesla is the only company you really have to say this about: there is a non-negligible probability that this thing doesn't exist. There are no published results and we aren't seeing the mind-blowing pace of FSD improvements that Musk promised us in 2020 when Dojo 1.0 was only a year away.
- threatripper 4y ago
  Musk promising FSD by the end of the year - year after year - is one major thing that tarnished the Tesla image in my mind. I now assume that he has to believe that in order to avoid lawsuits. Maybe the same is with the computing infrastructure. They need to build it to show that they honestly believed it could work - even though they failed to deliver on their promises many times. That said, they do make progress and the real reason why it's slow is probably because the problem is actually really hard and nobody has found the fast way to the solution yet.
  - killyourcar 4y ago
    Musk is absolutely the reason I would never consider buying another Tesla, having just sold mine. Dude just straight up lies. He may not realize he's lying, but he lies constantly. Edit, because I'm getting downvoted. Here's some examples: self driving, battery swaps, robotic snake chargers, cybertruck windows, Bitcoin won't be converted to fiat, starlink speeds will improve, he will sell his home, first mars mission 2024, Tesla solar roofs, power packs at every supercharger, brake pads on Tesla cars will never need to be replaced, the gigafactory will be 100% renewable powered by 2020, fixing the Flint water crisis, making bricks from boring company waste, founding a media credibility organization, and probably a TON more. Oh man, I forgot he took money to fly tourists to the moon. There are so many lies he's told. So many.
    - savagej 4y ago
      My autopilot drives me on the highway just like he said it would. That was what he predicted about 5 years ago. FSD is taking longer - so what. It's progressing for the public far more than another other attempt.
chell830 4y ago
- LightG 4y ago
  - chell830 4y ago
- MengerSponge 4y ago
  Nice
threeseed 4y ago
I wonder how this compares to the other revolutionary supercomputer: Power Mac G4
hello390 4y ago
- iseanstevens 4y ago
  Nice
londons_explore 4y ago
If this stuff is any good, Tesla should make it available for everyone to use colab-style. A hosted Jupyter notebook in a sandboxed VM able to send jobs to this new silicon is something that might be possible to set up by a small team in a few months, and could turn into a billion dollar business. As a bonus, Tesla can use revenue from that to grow this supercomputer, while using any spare/unsold capacity for themselves.
- kjksf 4y ago
  It is already Tesla's plan to build AWS-style paid access to Dojo. I think they said that during the first AI Day. Here's a 19 minute supercut of AI Day: https://www.youtube.com/watch?v=keWEE9FwS9o https://www.youtube.com/watch?v=keWEE9FwS9o
  - londons_explore 4y ago
    I suspect they've just deprioritized it for other work. But I think that was the wrong strategic move - they should have opened it up, together with some 'Tesla AI' demo models, in a colab environment. They can hire new employees to do that - it is separate work from that involved in making the self driving car, and will not block or interfere. The only reason I think they might not is that they don't want to step on Googles toes - there are very close links between Musk companies and Google, and a direct competitor to Googles TPU product might hurt that relationship more than it generates in revenue.
    - schiffern 4y ago
      > They can hire new employees The bottleneck isn't employees, it's chips.
      - londons_explore 4y ago
        If that were the case, they'd have already launched a 'demo' version with very low usage limits. Users can start building/porting their models, and then run them in a few months when the next batch of chips arrives.
loser777 4y ago
The most striking thing about the architecture is that it appears so heterogeneous and complex. Considering the vast amount of software/machine learning engineering behind model/data/pipeline parallelism schemes like Megatron-LM and ZeRO (which target hardware topologies that seem almost simple by comparison) I'm curious what abstractions are in place to make this beast of an architecture friendly to programmers. Can you program a small tile in the same way you would a large tile and like you would in CUDA for a large/small GPU? Are there dedicated kernel teams that implement common blocks like multiheaded attention with the topology in mind so researchers/engineers doing modeling don't have to worry about scaling the model architecture in a hardware-friendly way? Do they have a monstrous fork of PyTorch with "Dojo" supported natively?
- pclmulqdq 4y ago
  I'm not sure I would call the architecture very complex. It's about as simple as you can make a scale-out supercomputer. I assume they essentially do static positioning of the cluster for training jobs, and have a translation layer from the TensorFlow middle-end to their thing. Google did a similar thing with their TPUs, so it makes sense that they would have architected TF to accept exotic supercomputers as backends.
  - dekhn 4y ago
    Tensorflow (and pytorch) convert your computation graph (constructed in python) to XLA, which is then specialized to a specific hardware architecture. XLA is a good intermediate language and in fact, you can convert some memory movement in the graph to network calls, allowing you to run on parallel systems (like a cluster of GPUs or TPUs with their own non-host-based networking). It still requires many experts, both to write the XLA to hardware translation, and ML engineers who know how to write TF python that executes quickly. (note: Google has transitioned many projects to Jax, which also writes to XLA, as TF ended up being a bit of a pig with wings)
    - mistrial9 4y ago
      > TF ended up being a bit of a pig with wings Can you say more about this?
simonjgreen 4y ago
I wonder what the current owners of Tilera and their series of Tile processors think about Teslas naming here
- Robotbeat 4y ago
  Proper noun?
mg 4y ago
It's interesting to see to what extent matrix multiplication and backpropagation are dominating the AI space these days. I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. If there are any papers out there arguing for/against neural networks to stay in the king's seat forever, I would love to see them.
- nielsinho 4y ago
  You can take a look here: https://youtu.be/27PYlj-qNb0 https://youtu.be/27PYlj-qNb0
  - mg 4y ago
    That is a genetic algorithm at work. With "genetic programming" I was referring to applying genetic algorithms to code. So that the result of the evolution is a program that performs well doing a task like driving a car. Nice research on this has been done by John R. Koza. But nothing new came out in this area for quite a while.
- JustFinishedBSG 4y ago
  > I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. Why would they? If you assume that your objective is kinda smooth genetic algorithms and related are guaranteed to suck. And most "real" world things of interest can be assumed to be pretty smooth
  - mg 4y ago
    There are two parts to this. One is the representation: Neural Net vs Program. The other is the search approach: Back propagation vs crossover and mutation. Your smoothness argument applies to the search approach. I would say "smooth" is not enough to describe the search space of real world solutions. I would rather call it fractal like. There are many smooth areas in a visualization of the Mandelbrot set. But the most interesting things happen in areas that are not smooth but follow some delicate logic. You find tiny local minima there that are hard to reach via gradient descent. Genetic algorithms have a lot going for them in this type of search space. To pick one factor: GA's distribute the search power so that areas of the search space that are more promising get more resources. Similar to the optimal approach to deal with so called "multi-armed bandit" problems, GA's divide the search space in an optimal way, computation wise.
dadof4 4y ago
So they have one of the world’s most powerful super computers and they still can’t figure out how to make the car handle an incoming merge lane or how to properly drive down a two lane road without a lane lines…. I grew so frustrated with Tesla constantly trying to solve hard problems without mastering the easy stuff I sold the car.
- mciancia 4y ago
  Maybe it not an easy problem?
  - te_chris 4y ago
    It must be, he’s been selling Full Self Driving for years.
    - solardev 4y ago
      To be fair, it has been Full Self Driving for years. Here's the car. Drive it yourself.
      - rvz 4y ago
        Yes, you have to still pay attention to the road and drive the car, since the car is still unable to drive itself reliably without a human, which that was supposedly the Level 5 Full Self Driving (FSD) promise of 2020 with the 1 Million robo-taxis still missing. The product has been 'Fools Self Driving' demoware for years.
        thunky 4y ago
        You missed the joke.
        rvz 4y ago
        ? My comment already agreed with it, and it plays right in to the entire point of you having to still pay attention and drive the car yourself. Hence why I said it is 'Fools Self Driving' Is this you as well? [0] [0] https://news.ycombinator.com/item?id=32559771 https://news.ycombinator.com/item?id=32559771
sdfg413t 4y ago
I don't understand why they need this. Tesla has been selling fully-autonomous cars with L5 automation for years. They totally cracked self driving cars back in 2016, to the extent that "the person in the driver’s seat is only there for legal reasons", and have had 1,000,000 fully autonomous robotaxis on the road since 2020. What work remains to be done? Still, it is good to see powerful hardware being put to good use for once. I understand it is silly to feel sorry for inanimate objects, but it saddens me to see silicon squandered by some manchild at a national lab working on a dead-end vanity project when it could be crunching numbers for Tesla or—better yet—mining Bitcoin.

77 points by belltaco 4y ago

Cool. Can anyone chime in on how this compares to other ML SoC/ASICs? I know many places are going hard on general purpose GPUs, but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward.

dragontamer 4y ago
They'll be at a process disadvantage. D1 is allegedly TSMC 7nm, as per last year's information (https://www.tomshardware.com/news/tesla-d1-ai-chip https://www.tomshardware.com/news/tesla-d1-ai-chip). An ASIC can strip out features they don't need and save some space. But a good chunk of modern GPUs are memory-controllers, registers, and SIMD-cores. And modern GPUs (both AMD's MI250x and NVidia's A100) have 16-bit matrix multiplication units (aka: Tensor cores). Once we factor in the process disadvantage, I'm not sure if the D1 will be as competitive as they hope. Tesla's hope is that their D1 chip has more 16-bit matrix multiplication cores than the NVidia / AMD designs. But A100 is quite solid, and NVidia Hopper has been announced at HC34 (aka: NVidia's next generation). https://www.nvidia.com/en-us/technologies/hopper-architecture/ https://www.nvidia.com/en-us/technologies/hopper-architectur... ------- Most of this presentation on the Tesla Dojo is about the interconnect system. Alas, NVidia's on like the 4th (or was it 5th?) generation of NVlink, available from their DGX servers (and I'm sure a Hopper version will come out soon). AMD's not far behind, also with a lot of good presentations this year from HC34 that points out how AMD's "Frontier" Supercomputer has huge bandwidth. In particular, each MI250X GPU is a twin-chiplet design (two GPUs per... GPU), with 5 high-speed links to connect to other GPUs in a high-speed fashion. There's a reason why Frontier is the #1 supercomputer of the world right now, in both absolute Double-precision FLOPs, and in Green Double-precision FLOPS-per-watt. NVidia's Hopper will be hitting 4nm. AMD's MI250x is 5nm. That means the D1 chip has less than 1/2 the transistors at the same area compared to NVidia. > but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward. Only if you keep up with the process shrinks. 7nm is getting long in the tooth now. All eyes are looking forward to 5nm, 4nm, and even 3nm designs (now that Apple is the customer of TSMC's 3nm node). ----------- That being said, if the 7nm node is cheaper, maybe this exercise was still cost-effective for Tesla. As the newer nodes obsolete the older nodes, the older nodes become more cost-effective. Cost-efficiency is less popular / less cool, but still an effective business plan. > but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward. The issue is that it probably costs hundreds-of-millions of dollars to design something like the D1. Sure, the mass production of the chip afterwards will be incredible, but chips have stupidly high startup costs (masking, engineering, research, etc. etc.) GPUs on the other hand, are more general purpose and are applicable to more situations. So you can sell the GPU to more customers and spread out the R&D costs. In particular, GPUs capture the attention of the video game crowd, who will fund high-end GPU research just to play video games. Much like how Intel's laptops allow servers to share the R&D effort, so too does NVidia's consumer GPUs share research/development costs with their high end A100 cards.
- Robotbeat 4y ago
  Nvidia’s stuff is good but it’s pretty high margin. They don’t give access to it for cheap,& in the last five years, the cost per unit performance has been nearly flat. They’re also more generalized than Tesla needs. Performance advantages from process shrinks have also stagnated. A good time for a custom approach.
  - dragontamer 4y ago
    NVidia is claiming 1000 16-bit Tensor-FLOPS on Hopper: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-hopper-architecture... While Tesla is claiming less than 400 Tensor-FLOPS on D1. So yeah, the claims of NVidia's GH100 / Hopper GPU are an order of magnitude faster than the D1. Which is no surprise, because when your transistors are less than 1/2 the size of the competition, you can easily have 2x the performance in a embarrassingly parallel problem. -------- Note that the A100, released in 2020, offers 312 TFlops of 16-bit Tensor matrix-multiplication operations per second. Meaning D1's chip is barely competitive against the 2-year-old NVidia A100, let alone the next-generation Hopper. And note that NVidia's server-GPUs (like A100 or GH100) already come in prepackaged supercomputer formats with extremely-high speed data-links between them. See the DGX line of NVidia supercomputers. https://www.nvidia.com/en-us/data-center/dgx-station-a100/ https://www.nvidia.com/en-us/data-center/dgx-station-a100/ -------- You can't beat physics. Smaller transistors use less power, while more transistors offer more parallelism. A process-node advantage is huge.
    - Robotbeat 4y ago
      The transistors aren’t always literally smaller with each processor node, and there are negative quantum effects that also occur as you shrink. But anyway, you dodged my entire point about cost-to-performance ratio by looking just at performance. If NVidia is insisting on pocketing all the performance advantages of the process shrink as profit, then it still make sense for Tesla to do this.
      - dragontamer 4y ago
sinenomine 4y ago
It's a very focused and impressive effort. Training from SRAM can be one-two orders of magnitude faster than training from DRAM. Modern GPUs, even with some process advantage (and no shipments her, H100) strive to be general purpose processors too much, and that caps their peak deep learning performance. Every serious player will have to make or buy their own TPU.
Patrick-STH 4y ago
This is a crazy oversimplification, but let me take a shot. The difference to others is easily a large "novel size" discussion. Most training SoCs are focused on building something the size of an NVIDIA GPU, but designed for ML versus general purpose GPU HPC compute (FP64) plus ML. Often those accelerators today have a few types of models they are optimized for. NVIDIA is the baseline and so the competitors are looking for areas where they can get a large boost at a lower cost with something about the size of a A100/H100. Cerebras is perhaps the biggest exception with its WSE-2, a wafer size chip. Having the wafer size chip means that Cerebras does not need to go into higher-latency and higher-power off-package interconnect as frequently because its chip is 50x larger. In turn, Cerebras drives performance and cost savings by not needing NVLink4 NVSwitches / InfiniBand. Tesla's Dojo Tile is 25 chips roughly equivalent in size to a NVIDIA GPU in a single package with die-to-die communication facilitated by the base tile and then built for scale up units. Tesla also has focused on the interconnect and pipeline feeding the D1s and Tiles. Ultimately, I think that it takes something beyond a "solution X saves 30% over NVIDIA in these workloads in performance/ $" to survive. NVIDIA has a massive software ecosystem and can handle more types of tasks versus some of the other AI accelerators. That goes beyond just the training and also to other parts of the data prep and movement pipeline. NVIDIA extracts high margins from this work so that is why some effectively are competing with "it costs less and on some problems can be faster" architectures but what Tesla, Cerebras, Google, and a few others have another level of differentiation. Nothing is perfect, nor was that explanation, but just a high-level view of why the technology featured is impactful.

jeffbee 4y ago

They recycled a lot of material from last year's presentation. Tesla is the only company you really have to say this about: there is a non-negligible probability that this thing doesn't exist. There are no published results and we aren't seeing the mind-blowing pace of FSD improvements that Musk promised us in 2020 when Dojo 1.0 was only a year away.

threatripper 4y ago
Musk promising FSD by the end of the year - year after year - is one major thing that tarnished the Tesla image in my mind. I now assume that he has to believe that in order to avoid lawsuits. Maybe the same is with the computing infrastructure. They need to build it to show that they honestly believed it could work - even though they failed to deliver on their promises many times. That said, they do make progress and the real reason why it's slow is probably because the problem is actually really hard and nobody has found the fast way to the solution yet.
- killyourcar 4y ago
  Musk is absolutely the reason I would never consider buying another Tesla, having just sold mine. Dude just straight up lies. He may not realize he's lying, but he lies constantly. Edit, because I'm getting downvoted. Here's some examples: self driving, battery swaps, robotic snake chargers, cybertruck windows, Bitcoin won't be converted to fiat, starlink speeds will improve, he will sell his home, first mars mission 2024, Tesla solar roofs, power packs at every supercharger, brake pads on Tesla cars will never need to be replaced, the gigafactory will be 100% renewable powered by 2020, fixing the Flint water crisis, making bricks from boring company waste, founding a media credibility organization, and probably a TON more. Oh man, I forgot he took money to fly tourists to the moon. There are so many lies he's told. So many.
  - savagej 4y ago
    My autopilot drives me on the highway just like he said it would. That was what he predicted about 5 years ago. FSD is taking longer - so what. It's progressing for the public far more than another other attempt.

chell830 4y ago

LightG 4y ago
- chell830 4y ago
MengerSponge 4y ago
Nice

threeseed 4y ago

I wonder how this compares to the other revolutionary supercomputer: Power Mac G4

hello390 4y ago

iseanstevens 4y ago
Nice

londons_explore 4y ago

If this stuff is any good, Tesla should make it available for everyone to use colab-style. A hosted Jupyter notebook in a sandboxed VM able to send jobs to this new silicon is something that might be possible to set up by a small team in a few months, and could turn into a billion dollar business. As a bonus, Tesla can use revenue from that to grow this supercomputer, while using any spare/unsold capacity for themselves.

kjksf 4y ago
It is already Tesla's plan to build AWS-style paid access to Dojo. I think they said that during the first AI Day. Here's a 19 minute supercut of AI Day: https://www.youtube.com/watch?v=keWEE9FwS9o https://www.youtube.com/watch?v=keWEE9FwS9o
- londons_explore 4y ago
  I suspect they've just deprioritized it for other work. But I think that was the wrong strategic move - they should have opened it up, together with some 'Tesla AI' demo models, in a colab environment. They can hire new employees to do that - it is separate work from that involved in making the self driving car, and will not block or interfere. The only reason I think they might not is that they don't want to step on Googles toes - there are very close links between Musk companies and Google, and a direct competitor to Googles TPU product might hurt that relationship more than it generates in revenue.
  - schiffern 4y ago
    > They can hire new employees The bottleneck isn't employees, it's chips.
    - londons_explore 4y ago
      If that were the case, they'd have already launched a 'demo' version with very low usage limits. Users can start building/porting their models, and then run them in a few months when the next batch of chips arrives.

loser777 4y ago

The most striking thing about the architecture is that it appears so heterogeneous and complex. Considering the vast amount of software/machine learning engineering behind model/data/pipeline parallelism schemes like Megatron-LM and ZeRO (which target hardware topologies that seem almost simple by comparison) I'm curious what abstractions are in place to make this beast of an architecture friendly to programmers. Can you program a small tile in the same way you would a large tile and like you would in CUDA for a large/small GPU? Are there dedicated kernel teams that implement common blocks like multiheaded attention with the topology in mind so researchers/engineers doing modeling don't have to worry about scaling the model architecture in a hardware-friendly way? Do they have a monstrous fork of PyTorch with "Dojo" supported natively?

pclmulqdq 4y ago
I'm not sure I would call the architecture very complex. It's about as simple as you can make a scale-out supercomputer. I assume they essentially do static positioning of the cluster for training jobs, and have a translation layer from the TensorFlow middle-end to their thing. Google did a similar thing with their TPUs, so it makes sense that they would have architected TF to accept exotic supercomputers as backends.
- dekhn 4y ago
  Tensorflow (and pytorch) convert your computation graph (constructed in python) to XLA, which is then specialized to a specific hardware architecture. XLA is a good intermediate language and in fact, you can convert some memory movement in the graph to network calls, allowing you to run on parallel systems (like a cluster of GPUs or TPUs with their own non-host-based networking). It still requires many experts, both to write the XLA to hardware translation, and ML engineers who know how to write TF python that executes quickly. (note: Google has transitioned many projects to Jax, which also writes to XLA, as TF ended up being a bit of a pig with wings)
  - mistrial9 4y ago
    > TF ended up being a bit of a pig with wings Can you say more about this?

simonjgreen 4y ago

I wonder what the current owners of Tilera and their series of Tile processors think about Teslas naming here

Robotbeat 4y ago
Proper noun?

mg 4y ago

It's interesting to see to what extent matrix multiplication and backpropagation are dominating the AI space these days. I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. If there are any papers out there arguing for/against neural networks to stay in the king's seat forever, I would love to see them.

nielsinho 4y ago
You can take a look here: https://youtu.be/27PYlj-qNb0 https://youtu.be/27PYlj-qNb0
- mg 4y ago
  That is a genetic algorithm at work. With "genetic programming" I was referring to applying genetic algorithms to code. So that the result of the evolution is a program that performs well doing a task like driving a car. Nice research on this has been done by John R. Koza. But nothing new came out in this area for quite a while.
JustFinishedBSG 4y ago
> I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. Why would they? If you assume that your objective is kinda smooth genetic algorithms and related are guaranteed to suck. And most "real" world things of interest can be assumed to be pretty smooth
- mg 4y ago
  There are two parts to this. One is the representation: Neural Net vs Program. The other is the search approach: Back propagation vs crossover and mutation. Your smoothness argument applies to the search approach. I would say "smooth" is not enough to describe the search space of real world solutions. I would rather call it fractal like. There are many smooth areas in a visualization of the Mandelbrot set. But the most interesting things happen in areas that are not smooth but follow some delicate logic. You find tiny local minima there that are hard to reach via gradient descent. Genetic algorithms have a lot going for them in this type of search space. To pick one factor: GA's distribute the search power so that areas of the search space that are more promising get more resources. Similar to the optimal approach to deal with so called "multi-armed bandit" problems, GA's divide the search space in an optimal way, computation wise.

dadof4 4y ago

So they have one of the world’s most powerful super computers and they still can’t figure out how to make the car handle an incoming merge lane or how to properly drive down a two lane road without a lane lines…. I grew so frustrated with Tesla constantly trying to solve hard problems without mastering the easy stuff I sold the car.

mciancia 4y ago
Maybe it not an easy problem?
- te_chris 4y ago
  It must be, he’s been selling Full Self Driving for years.
  - solardev 4y ago
    To be fair, it has been Full Self Driving for years. Here's the car. Drive it yourself.
    - rvz 4y ago
      Yes, you have to still pay attention to the road and drive the car, since the car is still unable to drive itself reliably without a human, which that was supposedly the Level 5 Full Self Driving (FSD) promise of 2020 with the 1 Million robo-taxis still missing. The product has been 'Fools Self Driving' demoware for years.
      - thunky 4y ago
        You missed the joke.
        rvz 4y ago
        ? My comment already agreed with it, and it plays right in to the entire point of you having to still pay attention and drive the car yourself. Hence why I said it is 'Fools Self Driving' Is this you as well? [0] [0] https://news.ycombinator.com/item?id=32559771 https://news.ycombinator.com/item?id=32559771

sdfg413t 4y ago

I don't understand why they need this. Tesla has been selling fully-autonomous cars with L5 automation for years. They totally cracked self driving cars back in 2016, to the extent that "the person in the driver’s seat is only there for legal reasons", and have had 1,000,000 fully autonomous robotaxis on the road since 2020. What work remains to be done? Still, it is good to see powerful hardware being put to good use for once. I understand it is silly to feel sorry for inanimate objects, but it saddens me to see silicon squandered by some manchild at a national lab working on a dead-end vanity project when it could be crunching numbers for Tesla or—better yet—mining Bitcoin.