11 ms·

Tesla’s Dojo Microarchitecture

231 points by zdw 4y ago

buildbot 4y ago
It is incredibly odd tesla decided to invest in this level of engineering, even perfect hardware is like, 1/5th the battle. Now you get to stand up the entire software stack, make it performant, and hope you can compete with whatever Nvidia has in the pipeline.
- etaioinshrdlu 4y ago
  As far as I can tell Tesla doesn't even use Dojo seriously yet. The real work is still done on NVIDIA hardware.
  - deleted 4y ago
    [deleted]
  - cjbgkagh 4y ago
    It seems like to me they took a big bet and it didn’t pay off. They may not have felt like they could rely on Nvidia to deliver the promised Tensor Core performance.
    - kjksf 4y ago
      It's a bit premature to call this a failure given they are barely at a stage of making first chips and validating them. You don't seem to understand the timelines of designing chips like that from scratch. Dojo has been in works for almost 7 years. Tesla will continue to work on this for the next 20 years. There will be Dojo v2, Dojo v3 etc. just like at SpaceX there was Falcon 1, Falcon 9, Falcon Heavy. This still might end up a failure but they clearly feel confident enough to talk about Dojo publicly, which wasn't the case for the first 5 years of its development.
      - cjbgkagh 4y ago
        Thanks for explaining to me what I don’t know. Clearly you’re a Tesla fanboy. There is no doubt that it is an amazing piece of tech but I’m not confident Tesla will be able to pull off beating NVidia. Especially when compared to NVidias Tensor Cores and economics of scale. I don’t like their whole approach to self driving ML. I know a lot of people disagree with me so I would rather not get into it. I think a lot of the talk of future performance is posturing in order to get NVidia chips at cheaper prices.
        minhazm 4y ago
        I'm sure that it's at least partially for them to have a better negotiating position with NVIDIA. But the reality is that Tesla actually has some very good expertise in chip design. Jim Keller worked there for several years and along with Pete Bannon designed their Autopilot HW3 computer which is in 1+ million cars right now. At the time that HW3 was released it outperformed what NVIDIA had to offer. That said it's not likely they'll be able to beat NVIDIA, but they may be able to beat them for their hyper domain specific use cases. Additionally NVIDIA chips are difficult to get and they're extremely expensive. Even if Tesla can get something that performs 80-90% as good as NVIDIA but at significantly lower cost then it may still be worth it.
        cjbgkagh 4y ago
        I know these things. I think some of the dojo architecture was a reaction to their FSD chips being over optimized for resnet style models. They’re targeting extremely large models which is a new frontier for ML and in my view not the panacea they hoped it would be. I think with better ML they wouldn’t need so many chips, be it NVidia or Dojo.
      - mupuff1234 4y ago
        It's even more premature to predict what will happen 20 years in the future.
  - kjksf 4y ago
    Yes, per the talk Dojo is at the stage of having made first chips. But your comment is neither here not there. What do you think is a timeline for designing a chip like Dojo from scratch? Tesla has been working on this for almost 7 years (https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/ https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/). They might not work fast enough for your satisfaction but can you point any other car company designing chips like Dojo? Hell, can you point to any company at all that can come out of the gate with a chip design competitive with the best design of NVIDIA, a company that makes nothing but chips? After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences.
    - panick21_ 4y ago
      Also, even if the first version didn't make financial sense, building the team and the infrastructure required to make this work will likely translate into a next generation chip. Part of Tesla vertical integration mantra is that you are building internal capacity.
    - FireBeyond 4y ago
      > After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences. They are clearly confident that FSD will work, to the point of promising it "this year"... coincidentally, they've been promising "this year" for the last 7 years. And FSD still fails in spectacular/hilarious/horrifying ways, and isn't _remotely_ close.
    - aunty_helen 4y ago
      >After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences. Being real for a second, Tesla don’t have an amazing track record of delivery on their ai software side of things.
    - p1esk 4y ago
      Hell, can you point to any company at all that can come out of the gate with a chip design competitive with the best design of NVIDIA, a company that makes nothing but chips? Google.
- SuperscalarMeme 4y ago
  Not to mention that in-house silicon is all about economies of scale, this is even more of a puzzling move
  - kjksf 4y ago
    Tesla has a path to economies of scale: they already announced that if Dojo works as expected they'll make it available to others as an AWS-style service. Which is brilliant: they might end up making money on this. AI is clearly here to stay. The demand for AI training will clearly explode in the future. Running training in-house is not easy or cheap. You don't just plug in 1000 NVIDIA GPUs. You need massive up-front payment for GPUs and you're basically running your own extremely energy hungry datacenter . Tesla might built and operate massive datacenters. They'll use as much as they need for internal needs and sell the remaining capacity to others. This might take 5 years but the path to do it is clear.
    - scoopertrooper 4y ago
      I don’t see how they’re going to commercialise this as a cloud compute service. For one, they’ve built a chip that operates in a fundamentally different way to other chips. So any other company that wanted to use it would have to invest a considerable amount of resources in building up the institutional knowledge to use it effectively. Additionally, the lack of virtual memory and multi-tasking support renders it pretty much impossible to divide up compute between multiple customers. So, commercialising this would require customers renting out the whole unit, which is contrary to how cloud computing usually works. Are there companies out there that have the capital and use cases necessary to fit into Dojo Cloud? Maybe, though not one I’ve worked for. Would they trust the stable genius currently heading up Tesla enough to make such an investment? Perhaps, but I wouldn’t, but what do I know?
      - kjksf 4y ago
        Part of the plan is to have PyTorch compatibility. Dojo has it's own IR but they also have a PyTorch to Dojo compiler. People's opinion of Musk wont matter: either Dojo will be a capable service at a good price or it won't. People will use it based on merits.
- dragontamer 4y ago
  Dojo is rumored 7nm to boot. They'll be competing with a process node disadvantage. TSMC is down to 3nm and 5nm already. AMD chips like MI250x are 5nm with tensor matrix multiplication units. NVidia Hopper will be like 4nm IIRC?
  - kjksf 4y ago
    During first AI day Tesla already said that they have plans and ideas for 10x improvements for Dojo v2. And I'm sure one of the improvements is to use 5nm or 3nm process. They've already been working on this for almost 7 years. This is not some side projects but a serious operation. They have to ship something sometime and they decided to ship this now. I'm confident that they know about 5nm process and chose 7nm for good reasons. There will be next version and next process.
    - sbierwagen 4y ago
      >And I'm sure one of the improvements is to use 5nm or 3nm process. Implementing a process shrink is not just scaling the masks by the appropriate percentage. It often is a completely different optical train, at a different wavelength, different pellicle, changing the refractive index of the immersion fluid, different multiple patterning. It takes months (years?) of work. For many applications it's worth it, but it's not at all a slam dunk. The vast majority of ASICs are designed for a particular node and never upgraded. It's the kind of crazy long-term speculative capital investment TSLA might have indulged in when the stock was at $414 a share, but it's nowhere remotely near that today. They're already engaging in layoffs, today. Why spend the money?
      - rbanffy 4y ago
        > vast majority of ASICs are designed for a particular node and never upgraded They are, if the new node is cheap enough to justify the investment in shrinking the design. In most ASICs, being faster won’t make people rip out their embedded electronics for new models.
  - panick21_ 4y ago
- pookeh 4y ago
  Perhaps they do this so they can carry over this tech into their cars. If you are making millions of cars per year, with tens or hundreds of chips per car, it is easy to reason doing this on your own instead of outsourcing it.
- jillesvangurp 4y ago
  I think they developed this during the years where all the nvidia hardware was gobbled up by crypto miners. The resulting pricing probably did not help either. Tesla is big enough that they can afford to experiment a little. This sounds like an experiment that actually worked. Once they had it working, doing more of it and iterating on it is just business as usual for them.
  - cyber_kinetist 4y ago
    > This sounds like an experiment that actually worked. Are you sure? We haven’t even seen the chip in any physical form yet. We haven’t seen it running any code.
    - seunosewa 4y ago
      The self-driving AI they use this chip for appears to be less than impressive from my perspective. Its mistakes seem to come up in the news quite frequently.
      - johnsimer 4y ago
        Dojo’s D1 chip for training AFAIK is not being used yet. Tesla still trains with their ~7000 cluster of A100s And the D1 is for their training machine anyways. Not their inference chip in the cars
      - jillesvangurp 4y ago
        Are you aware of a better implementation though? It might not be perfect; it's still miles ahead of competing systems. Yes, it makes mistakes. But that's a rather poor argument to dismiss the whole strategy and technology. From my point of view, Tesla is converging on good enough more rapidly than anyone else in the industry. Courtesy of their AI strategy and technology; including the in house developed chips.
        friendzis 4y ago
        > Tesla is converging on good enough This is a big problem, though. Any ADAS is a safety critical system and while "good enough" can help push volumes, it is extremely bad from safety perspective. If you really want to read on this u/adamjosephcook/ has some awesome writing going into far deeper detail than I could explaining this particular issue.
- torginus 4y ago
  Let me play the devil's advocate - when you have an $1T company, investing a couple million into something like this signals to investors that you have big plans for the future and have a leg up on the competition in the self-driving space. Even if that increases the stock price by 1%, this investment pays for itself a hundred times over.
  - lclarkmichalek 4y ago
    Setting up a hardware platform of this kind that is useable in production costs significantly more than a couple of million
    - ricardobeat 4y ago
      Still a blip for a company that was toying with $1B in Bitcoin.
  - baq 4y ago
    this is more like couple hundred million TBH
    - dragontamer 4y ago
      Hundred million for the chip alone, a few hundred million for software (compiler, assembler, router, protocols, communications, load balancing, server, supercomputer utilization, power management). The $100-million chip might very well be the easy part of this project...
      - sinenomine 4y ago
        As a software person it baffles me how we tend to overestimate our value compared to hardware. The chip design field is really demanding, especially with deep submicron processes and with rigorous testing & verification culture that blows out of the water almost everything software organizations do. I doubt developing a compiler for this chip will be as costly as your estimate, especially given mature opensource compiler frameworks like LLVM (already adapted to similar TPU-like architectures) and upcoming pytorch glow, apache TVM ML-specific compiler frameworks. And especially given in our industry there are people with advanced degrees who are willing to jump through additional hoops and even accept pay cuts, just to work on a cool compiler. I respect the hardware for what it is and what it could enable.
hello639 4y ago
- leobg 4y ago
  You made a new account to post a yawn?
happytiger 4y ago
They’re onshoring chip production and the Tesla stack needs a fab for contracts and competitiveness. Makes sense.
astrange 4y ago
This is meant to run in a datacenter, not a car, right? There’s some value to custom ASICS if you can get it - Google seems happy with TPUs and I assume those aren’t on the latest Nvidia-competitive process.
- dragontamer 4y ago
  If they are spending this much money on the data center chip, they probably will port the assembly language over to an ASIC inference chip for their cars. I honestly can't see how they make (or save) enough money with just the data center chip here. Maybe if the inference chip is on a cheaper 12nm process or something... Maybe?
  - akelly 4y ago
    Tesla has been shipping custom 14nm inference chips in cars since 2019. https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip
  - kjksf 4y ago
    It's not about saving money. It's about accelerating nn training and winning robotaxi market, which could make them the most profitable company ever.
    - dragontamer 4y ago
      NVidia chips can do this job. (EDIT: The job of training a neural network I should say. I don't think that's enough to get self driving, but Dojo ain't anything but a faster NN training chip anyway) The question is if they are saving money compared to buying an off the shelf DGX system. I really doubt it.
      - jml78 4y ago
        Tesla is very vertically integrated. This is just how they operate. You can make the argument that they shouldnt be so vertically integrated but it has worked for them thus far.
1958325146 4y ago
Can anyone tell if this architecture makes sense to ship in their actual cars to replace whatever is doing the computation there now? That is the only way I can see this making any sense to develop.
- WatchDog 4y ago
  They have already developed and deployed custom silicon for their cars. There is probably some overlap in the IP between these chips, but Dojo is optimized for operating in a large data center cluster for training purposes, whereas their car chips are optimized for running pre-trained models.
- kjksf 4y ago
  Tesla has already designed and manufactures an FSD chip that runs nn inference in the car. This is a chip designed to accelerate nn training in their datacenter. The effort and expense makes perfect sense if you consider: - the costs are spread over millions of car (3 millions, today, tens of millions in the future) - it helps them win robotaxi market that is potentially so lucrative that the cost of developing custom chips will be peanuts
- deleted 4y ago
  [deleted]
- epgui 4y ago
  This is not for putting in cars. This is for building giant supercomputers for research and development, inside a building (no wheels).
slt2021 4y ago
they want to get federal funding for chip manufacturing on US soil? That would be typical of Elon Musk to get some federal funding $$$$$ but Elon can make it work on par if not better than nVidia, just because Tesla can hardcode and overoptimize for one single task (FSD AI), while nVidia will always be limited in keeping their chips generic enough for all purpose NN training/inference and even gaming + backwards compatibility for all historical stuff. Tesla can work with blank slate and can reap performance benefits over there
- RC_ITR 4y ago
  Neither Tesla nor Nvidia do (or want to) manufacture chips. That's an extremely complicated thing that requires armies of PhDs (chem, EE, physics, etc.) and billions of dollars. You can think of what this is as 'maps' that manufacturers instantiate as chips. IDK what tools Tesla is actually using, but for context, the work that Tesla employees would do looks like this: https://www.cadence.com/en_US/home/tools/custom-ic-analog-rf-design/layout-design/virtuoso-layout-suite.html https://www.cadence.com/en_US/home/tools/custom-ic-analog-rf...
- kjksf 4y ago
  Tesla started this effort 6 years ago (https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/ https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/). So no, your conspiracy theory does not align with reality.
TomVDB 4y ago
With no virtual memory, no caches, and interface processors instead of direct access to external DRAM, this thing must be a programming nightmare? Having tons of small CPUs with fast local SRAM is of course not a new idea. Back in 1998, I talked to a startup that believed it could replace standard cell ASIC design with tiny CPUs that had custom instruction sets. (I didn't believe it could: it's extremely area inefficient and way to power hungry for that kind of application. The startup went nowhere.) And the IBM Cell is indeed an obvious inspiration. But AFAIK, the IBM Cell was hard to program. I've seen PS3 presentations where it was primarily used as a software defined GPU, because it was just too difficult to use as a general purpose processor. Now NOT being a general purpose process is the whole point of Dojo, so maybe they can make it work. But from my limited experience with CUDA, virtual memory and direct access to DRAM is a major plus, even if the high performance compute routines make intensive use of shared memory. The fact that an interface processor is involved (how?) in managing your local SRAM must make synchronization much more complex than with CUDA, where everything is handled by the same SM that manages the calculations: your warp issues a load, it waits on a barrier, the calculations happens, sometimes in a side unit in which case you again wait on a barrier, you offload the data and wait on a barrier. And while one warp waits on a barrier, another warp can take over. It's pretty straightforward. The Dojo model suggests that "wait on a barrier" becomes "wait on the interface processor".
- rowanG077 4y ago
  You are right that this won't work well with any language that assumes a "normal" processor. But a small language that is written for it could be fine.
- ThrownAllTheWay 4y ago
  Also investing in a branch predictor when the intended workload doesn't seem at all scalar is a confusing choice to me. Also the 362 F16 TFLOPs sounds super impressive, except the memory bandwidth is I think 800 GB/s (or is it 5 times that? Or effectively less than that if data has to be passed along multiple hops? I'm a bit confused), which means having to do 1000 ops (or 200? or more?) on each 16 bit value loaded in. Maybe you could do that, but it feels like you'd probably end up bandwidth bound most of the time.
keepquestioning 4y ago
There was no reason to waste money for this. You don't want to focus on two incredibly resource-intense and error-prone processes in the same company (Automotive, Semiconductor). This is a hype project simply created to pump up the stock price.
- AlotOfReading 4y ago
  While I agree that most custom silicon is a bit silly (and the costs dramatically underestimated), it's worth noting that most of the serious companies in the AV space have some some amount of custom silicon for ML either in the works or already in production. Waymo is using TPUs for example and Cruise has its own chips as well. It's the cool thing to do in SV lately. Doesn't help that Nvidia is such a toxic partner that the prospect of not working with them makes custom silicon seem almost reasonable.
- kjksf 4y ago
  There's a legion of people convinced of their superior intellect with "helpful" advice about what Musk / Tesla / SpaceX should or should no do. After first AI day Tesla stock dropped significantly. Wall Street doesn't understand Tesla as anything else but a car company and won't understand AI until it start producing billions in revenues. It's exceedingly poor stock pump. They've been working on Dojo for almost 7 years. They kept it secret for the first 5. Kind of contradicts the "stock pump" memes. They start talking about it now because those efforts are very advanced and they want to hire more people.
  - stephencanon 4y ago
    It has not really been a secret, they just haven’t been talking about details much. It was pretty clear to anyone who works in related areas what they were up to.
- panick21_ 4y ago
  Its so hilarious when people think everything Tesla does is some 4D chess about stock price, when in reality Musk really doesn't care about stock price at all. He literally thinks multiple years ahead, and doesn't change that strategy because some short term head winds. > two incredibly resource-intense and error-prone processes in the same company (Automotive, Semiconductor). If you think making these chips is excessive, you don't know Tesla. Tesla is the most vertically integrated car company in the world (maybe BYD compares). Compared to literally designing not only their own batteries cells, but their own battery chemistries plus their own battery materials process AND their own fully up battery manufacturing factories. They started doing this a decade ago and only this year the first cars rolled of the line and being sold to costumers. That is years and years of investment, planning and internal capacity building. In 2017 people were laughing at them for attempting to do all their own manufactriong. Now they have industry dominating automotive margin. They literally have their own materials team that makes their own aluminum for casting and have co-developed production machines that are the size of a whole house. This is now slowly being copied by other manufactures. They also make their own glass, seats and a whole host of other things that most car companies don't make. Doing these things in-house cost billions and billions in capx. Also, Dojo is not the only chip they make, they have other chips that are already being sold in their cars by the millions every year. Dojo is a pittance of investment compared to that and a fraction of the total investment into the overall self-driving stack. Just because you don't agree with the overall strategy (assuming from your comment) doesn't mean that they many, many billions Tesla spends isn't real, or its all some sort of stock price play. So, if you think Dojo is about stock price, I don't know what to tell you, its just wrong.
Animats 4y ago
OK, but why? Is this somehow supposed to make their flaky "self-driving" work? Or what? Certainly you can build custom chips for deep learning. It's mostly a simple inner loop, replicated many times, after all. Ideal case for ASICs. But will this actually benefit Tesla as a car company?
Sirened 4y ago
neat :) Love seeing other people building new and weird architectures. I wonder what their actual use case for SMT was, I don't really buy the "one thread for vector, one thread for pulling data into SRAM". I'm also a bit surprised to see they didn't go for a VLIW ISA, it always seemed like these tightly integrated data processing chips were the ideal candidates since binary compatibility isn't an issue when you're building your own HW.
TekMol 4y ago
The main concern mentioned in the comments seems to be that programmers will find it hard to work with this hardware. But is that really the intention? Wouldn't it be enough if there is one program written for it, one that takes a bunch of inputs and a bunch of outputs and then creates a set of NN weights that perform well? So the AWS style offer would not be "rent our hardware and program on it" but "rent our hardware/software stack, throw your input/output pairs at it and get back a set (billions) of NN weights that perform well".
- WithinReason 4y ago
  But then why have e.g. a branch predictor instead of making it more specialised?
  - amelius 4y ago
    Perhaps because they will still run multiple programs, just not at the same time. Or because the program behaves differently in different places, and it is too big to optimize.
mrb 4y ago
In terms of absolute compute performance per chip, perf per watt, and perf per die area, it looks like Dojo matches or surpasses the best GPUs of today: «Tesla claims a die with 354 Dojo cores can hit 362 BF16 TFLOPS at 2 GHz» For comparison, the fastest single-chip GPU today is the AMD MI250X which has 220 "cores" (compute units) totaling 383 BF16 TFLOPS at 1.7 GHz, and that's a monster 560 watt chip. The Dojo chip is likely under this 560 was TDP, so more efficient. And Tesla provides roughly the same compute performance, but the chip is arrayed in 61% more cores, meaning it is far more suitable to handle branchy code. Also Tesla claims the die measures only 645 mm², compared to 1540 mm² for AMD. So the wafer fabrication cost is roughly half(!) of AMD's! If Tesla has truly managed to build that, I'm impressed. Edit: I missed that Tesla claims "less than 600 watt" per chip. So we know its comparable or less than AMD (560 watt). Edit 2: 25 dies are packed on a single system-on-wafer. That's 15 kW on a disc of 30 cm (12 in) of diameter. Sheesh! That must require an ungodly liquid cooling system! Edit 3: there is more info, including rendering of the host interface card at: https://www.servethehome.com/tesla-dojo-custom-ai-supercomputer-at-hc34/ https://www.servethehome.com/tesla-dojo-custom-ai-supercompu... and https://www.servethehome.com/tesla-dojo-ai-system-microarchitecture/ https://www.servethehome.com/tesla-dojo-ai-system-microarchi... Edit 4: found a pic of the liquid cooling system - as expected ;) https://media.datacenterdynamics.com/media/images/training_tiles_III.original.png https://media.datacenterdynamics.com/media/images/training_t... source: https://www.datacenterdynamics.com/en/news/tesla-details-dojo-supercomputer-reveals-dojo-d1-chip-and-training-tile-module/ https://www.datacenterdynamics.com/en/news/tesla-details-doj... And they say the first tile was "tested last week" as of August 20th... This confirms my suspicion that the system is barely (?) functional. Also see "Venkataramanan appeared to even surprised Andrej Karpathy, Tesla’s head of AI, on stage by revealing for the first time that Dojo training tile ran one of his neural networks" from https://electrek.co/2021/08/20/tesla-dojo-supercomputer-worlds-new-most-powerful-ai-training-machine/ https://electrek.co/2021/08/20/tesla-dojo-supercomputer-worl...
- deleted 4y ago
  [deleted]
- WithinReason 4y ago
  A 2x efficiency improvement from a GPU to a specialised ASIC not particularly impressive. How much would you gain by removing the graphics related stuff from a GPU (texture pipelines, vertex processing, etc.?). In addition, they lose existing programming models like OpenCL and the compiler progress that happened in the last 10 years and have to roll their own. The amount of SW work needed to support this must be a lot to get the same ease of use out of it as GPUs. Maybe they made it more CPU like to make it easier to program?
frozenport 4y ago
AMD pays for this flexibility by spending about 44% of “Zeppelin” die area on logic other than cores and cache. Dojo only spends 28.9% of die area on things other than SRAM and cores. Doesn't sound worth it? You're basically spending a ton of money to fab a chip for what amounts to like a 30% training performance boost? Just buy more chips.
rbanffy 4y ago
The same way we like to talk about odd OSs, like Lisp Machines, Plan 9, Oberon, Smalltalk, I'd love to see one of these things used as a desktop CPU (since I never got to play with one of the desktoppable Xeon Phi's) just to see the weird and interesting OSs and applications that would emerge from it. I know it must be horrendously difficult to program for general purpose use, but that's the point - figuring out what CAN be done and what an OS for it would look like.
mannyv 4y ago
Discussions about how this will fail to be productized as a cloud service are premature. There are many, many ways to do SAAS. If the benefit is there people will find a way to use it. In any case the problem is moving data, then the compute side. If Tesla is really going to do it they have someone working on that already. SAAS is not rocket science, and the issues are well-known.

231 points by zdw 4y ago

It is incredibly odd tesla decided to invest in this level of engineering, even perfect hardware is like, 1/5th the battle. Now you get to stand up the entire software stack, make it performant, and hope you can compete with whatever Nvidia has in the pipeline.

etaioinshrdlu 4y ago
As far as I can tell Tesla doesn't even use Dojo seriously yet. The real work is still done on NVIDIA hardware.
- deleted 4y ago
  [deleted]
- cjbgkagh 4y ago
  It seems like to me they took a big bet and it didn’t pay off. They may not have felt like they could rely on Nvidia to deliver the promised Tensor Core performance.
  - kjksf 4y ago
    It's a bit premature to call this a failure given they are barely at a stage of making first chips and validating them. You don't seem to understand the timelines of designing chips like that from scratch. Dojo has been in works for almost 7 years. Tesla will continue to work on this for the next 20 years. There will be Dojo v2, Dojo v3 etc. just like at SpaceX there was Falcon 1, Falcon 9, Falcon Heavy. This still might end up a failure but they clearly feel confident enough to talk about Dojo publicly, which wasn't the case for the first 5 years of its development.
    - cjbgkagh 4y ago
      Thanks for explaining to me what I don’t know. Clearly you’re a Tesla fanboy. There is no doubt that it is an amazing piece of tech but I’m not confident Tesla will be able to pull off beating NVidia. Especially when compared to NVidias Tensor Cores and economics of scale. I don’t like their whole approach to self driving ML. I know a lot of people disagree with me so I would rather not get into it. I think a lot of the talk of future performance is posturing in order to get NVidia chips at cheaper prices.
      - minhazm 4y ago
        I'm sure that it's at least partially for them to have a better negotiating position with NVIDIA. But the reality is that Tesla actually has some very good expertise in chip design. Jim Keller worked there for several years and along with Pete Bannon designed their Autopilot HW3 computer which is in 1+ million cars right now. At the time that HW3 was released it outperformed what NVIDIA had to offer. That said it's not likely they'll be able to beat NVIDIA, but they may be able to beat them for their hyper domain specific use cases. Additionally NVIDIA chips are difficult to get and they're extremely expensive. Even if Tesla can get something that performs 80-90% as good as NVIDIA but at significantly lower cost then it may still be worth it.
        cjbgkagh 4y ago
        I know these things. I think some of the dojo architecture was a reaction to their FSD chips being over optimized for resnet style models. They’re targeting extremely large models which is a new frontier for ML and in my view not the panacea they hoped it would be. I think with better ML they wouldn’t need so many chips, be it NVidia or Dojo.
    - mupuff1234 4y ago
      It's even more premature to predict what will happen 20 years in the future.
- kjksf 4y ago
  Yes, per the talk Dojo is at the stage of having made first chips. But your comment is neither here not there. What do you think is a timeline for designing a chip like Dojo from scratch? Tesla has been working on this for almost 7 years (https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/ https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/). They might not work fast enough for your satisfaction but can you point any other car company designing chips like Dojo? Hell, can you point to any company at all that can come out of the gate with a chip design competitive with the best design of NVIDIA, a company that makes nothing but chips? After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences.
  - panick21_ 4y ago
    Also, even if the first version didn't make financial sense, building the team and the infrastructure required to make this work will likely translate into a next generation chip. Part of Tesla vertical integration mantra is that you are building internal capacity.
  - FireBeyond 4y ago
    > After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences. They are clearly confident that FSD will work, to the point of promising it "this year"... coincidentally, they've been promising "this year" for the last 7 years. And FSD still fails in spectacular/hilarious/horrifying ways, and isn't _remotely_ close.
  - aunty_helen 4y ago
    >After 7 years of work they are clearly confident that this will work or else they would scarp the project instead of giving talks at conferences. Being real for a second, Tesla don’t have an amazing track record of delivery on their ai software side of things.
  - p1esk 4y ago
    Hell, can you point to any company at all that can come out of the gate with a chip design competitive with the best design of NVIDIA, a company that makes nothing but chips? Google.
SuperscalarMeme 4y ago
Not to mention that in-house silicon is all about economies of scale, this is even more of a puzzling move
- kjksf 4y ago
  Tesla has a path to economies of scale: they already announced that if Dojo works as expected they'll make it available to others as an AWS-style service. Which is brilliant: they might end up making money on this. AI is clearly here to stay. The demand for AI training will clearly explode in the future. Running training in-house is not easy or cheap. You don't just plug in 1000 NVIDIA GPUs. You need massive up-front payment for GPUs and you're basically running your own extremely energy hungry datacenter . Tesla might built and operate massive datacenters. They'll use as much as they need for internal needs and sell the remaining capacity to others. This might take 5 years but the path to do it is clear.
  - scoopertrooper 4y ago
    I don’t see how they’re going to commercialise this as a cloud compute service. For one, they’ve built a chip that operates in a fundamentally different way to other chips. So any other company that wanted to use it would have to invest a considerable amount of resources in building up the institutional knowledge to use it effectively. Additionally, the lack of virtual memory and multi-tasking support renders it pretty much impossible to divide up compute between multiple customers. So, commercialising this would require customers renting out the whole unit, which is contrary to how cloud computing usually works. Are there companies out there that have the capital and use cases necessary to fit into Dojo Cloud? Maybe, though not one I’ve worked for. Would they trust the stable genius currently heading up Tesla enough to make such an investment? Perhaps, but I wouldn’t, but what do I know?
    - kjksf 4y ago
      Part of the plan is to have PyTorch compatibility. Dojo has it's own IR but they also have a PyTorch to Dojo compiler. People's opinion of Musk wont matter: either Dojo will be a capable service at a good price or it won't. People will use it based on merits.
dragontamer 4y ago
Dojo is rumored 7nm to boot. They'll be competing with a process node disadvantage. TSMC is down to 3nm and 5nm already. AMD chips like MI250x are 5nm with tensor matrix multiplication units. NVidia Hopper will be like 4nm IIRC?
- kjksf 4y ago
  During first AI day Tesla already said that they have plans and ideas for 10x improvements for Dojo v2. And I'm sure one of the improvements is to use 5nm or 3nm process. They've already been working on this for almost 7 years. This is not some side projects but a serious operation. They have to ship something sometime and they decided to ship this now. I'm confident that they know about 5nm process and chose 7nm for good reasons. There will be next version and next process.
  - sbierwagen 4y ago
    >And I'm sure one of the improvements is to use 5nm or 3nm process. Implementing a process shrink is not just scaling the masks by the appropriate percentage. It often is a completely different optical train, at a different wavelength, different pellicle, changing the refractive index of the immersion fluid, different multiple patterning. It takes months (years?) of work. For many applications it's worth it, but it's not at all a slam dunk. The vast majority of ASICs are designed for a particular node and never upgraded. It's the kind of crazy long-term speculative capital investment TSLA might have indulged in when the stock was at $414 a share, but it's nowhere remotely near that today. They're already engaging in layoffs, today. Why spend the money?
    - rbanffy 4y ago
      > vast majority of ASICs are designed for a particular node and never upgraded They are, if the new node is cheap enough to justify the investment in shrinking the design. In most ASICs, being faster won’t make people rip out their embedded electronics for new models.
- panick21_ 4y ago
pookeh 4y ago
Perhaps they do this so they can carry over this tech into their cars. If you are making millions of cars per year, with tens or hundreds of chips per car, it is easy to reason doing this on your own instead of outsourcing it.
jillesvangurp 4y ago
I think they developed this during the years where all the nvidia hardware was gobbled up by crypto miners. The resulting pricing probably did not help either. Tesla is big enough that they can afford to experiment a little. This sounds like an experiment that actually worked. Once they had it working, doing more of it and iterating on it is just business as usual for them.
- cyber_kinetist 4y ago
  > This sounds like an experiment that actually worked. Are you sure? We haven’t even seen the chip in any physical form yet. We haven’t seen it running any code.
  - seunosewa 4y ago
    The self-driving AI they use this chip for appears to be less than impressive from my perspective. Its mistakes seem to come up in the news quite frequently.
    - johnsimer 4y ago
      Dojo’s D1 chip for training AFAIK is not being used yet. Tesla still trains with their ~7000 cluster of A100s And the D1 is for their training machine anyways. Not their inference chip in the cars
    - jillesvangurp 4y ago
      Are you aware of a better implementation though? It might not be perfect; it's still miles ahead of competing systems. Yes, it makes mistakes. But that's a rather poor argument to dismiss the whole strategy and technology. From my point of view, Tesla is converging on good enough more rapidly than anyone else in the industry. Courtesy of their AI strategy and technology; including the in house developed chips.
      - friendzis 4y ago
        > Tesla is converging on good enough This is a big problem, though. Any ADAS is a safety critical system and while "good enough" can help push volumes, it is extremely bad from safety perspective. If you really want to read on this u/adamjosephcook/ has some awesome writing going into far deeper detail than I could explaining this particular issue.
torginus 4y ago
Let me play the devil's advocate - when you have an $1T company, investing a couple million into something like this signals to investors that you have big plans for the future and have a leg up on the competition in the self-driving space. Even if that increases the stock price by 1%, this investment pays for itself a hundred times over.
- lclarkmichalek 4y ago
  Setting up a hardware platform of this kind that is useable in production costs significantly more than a couple of million
  - ricardobeat 4y ago
    Still a blip for a company that was toying with $1B in Bitcoin.
- baq 4y ago
  this is more like couple hundred million TBH
  - dragontamer 4y ago
    Hundred million for the chip alone, a few hundred million for software (compiler, assembler, router, protocols, communications, load balancing, server, supercomputer utilization, power management). The $100-million chip might very well be the easy part of this project...
    - sinenomine 4y ago
      As a software person it baffles me how we tend to overestimate our value compared to hardware. The chip design field is really demanding, especially with deep submicron processes and with rigorous testing & verification culture that blows out of the water almost everything software organizations do. I doubt developing a compiler for this chip will be as costly as your estimate, especially given mature opensource compiler frameworks like LLVM (already adapted to similar TPU-like architectures) and upcoming pytorch glow, apache TVM ML-specific compiler frameworks. And especially given in our industry there are people with advanced degrees who are willing to jump through additional hoops and even accept pay cuts, just to work on a cool compiler. I respect the hardware for what it is and what it could enable.

hello639 4y ago

leobg 4y ago
You made a new account to post a yawn?

happytiger 4y ago

They’re onshoring chip production and the Tesla stack needs a fab for contracts and competitiveness. Makes sense.

astrange 4y ago

This is meant to run in a datacenter, not a car, right? There’s some value to custom ASICS if you can get it - Google seems happy with TPUs and I assume those aren’t on the latest Nvidia-competitive process.

dragontamer 4y ago
If they are spending this much money on the data center chip, they probably will port the assembly language over to an ASIC inference chip for their cars. I honestly can't see how they make (or save) enough money with just the data center chip here. Maybe if the inference chip is on a cheaper 12nm process or something... Maybe?
- akelly 4y ago
  Tesla has been shipping custom 14nm inference chips in cars since 2019. https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip
- kjksf 4y ago
  It's not about saving money. It's about accelerating nn training and winning robotaxi market, which could make them the most profitable company ever.
  - dragontamer 4y ago
    NVidia chips can do this job. (EDIT: The job of training a neural network I should say. I don't think that's enough to get self driving, but Dojo ain't anything but a faster NN training chip anyway) The question is if they are saving money compared to buying an off the shelf DGX system. I really doubt it.
    - jml78 4y ago
      Tesla is very vertically integrated. This is just how they operate. You can make the argument that they shouldnt be so vertically integrated but it has worked for them thus far.

1958325146 4y ago

Can anyone tell if this architecture makes sense to ship in their actual cars to replace whatever is doing the computation there now? That is the only way I can see this making any sense to develop.

WatchDog 4y ago
They have already developed and deployed custom silicon for their cars. There is probably some overlap in the IP between these chips, but Dojo is optimized for operating in a large data center cluster for training purposes, whereas their car chips are optimized for running pre-trained models.
kjksf 4y ago
Tesla has already designed and manufactures an FSD chip that runs nn inference in the car. This is a chip designed to accelerate nn training in their datacenter. The effort and expense makes perfect sense if you consider: - the costs are spread over millions of car (3 millions, today, tens of millions in the future) - it helps them win robotaxi market that is potentially so lucrative that the cost of developing custom chips will be peanuts
deleted 4y ago
[deleted]
epgui 4y ago
This is not for putting in cars. This is for building giant supercomputers for research and development, inside a building (no wheels).

slt2021 4y ago

they want to get federal funding for chip manufacturing on US soil? That would be typical of Elon Musk to get some federal funding $$$$$ but Elon can make it work on par if not better than nVidia, just because Tesla can hardcode and overoptimize for one single task (FSD AI), while nVidia will always be limited in keeping their chips generic enough for all purpose NN training/inference and even gaming + backwards compatibility for all historical stuff. Tesla can work with blank slate and can reap performance benefits over there

RC_ITR 4y ago
Neither Tesla nor Nvidia do (or want to) manufacture chips. That's an extremely complicated thing that requires armies of PhDs (chem, EE, physics, etc.) and billions of dollars. You can think of what this is as 'maps' that manufacturers instantiate as chips. IDK what tools Tesla is actually using, but for context, the work that Tesla employees would do looks like this: https://www.cadence.com/en_US/home/tools/custom-ic-analog-rf-design/layout-design/virtuoso-layout-suite.html https://www.cadence.com/en_US/home/tools/custom-ic-analog-rf...
kjksf 4y ago
Tesla started this effort 6 years ago (https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/ https://www.linkedin.com/in/ganesh-venkataramanan-99272a3/). So no, your conspiracy theory does not align with reality.

TomVDB 4y ago

With no virtual memory, no caches, and interface processors instead of direct access to external DRAM, this thing must be a programming nightmare? Having tons of small CPUs with fast local SRAM is of course not a new idea. Back in 1998, I talked to a startup that believed it could replace standard cell ASIC design with tiny CPUs that had custom instruction sets. (I didn't believe it could: it's extremely area inefficient and way to power hungry for that kind of application. The startup went nowhere.) And the IBM Cell is indeed an obvious inspiration. But AFAIK, the IBM Cell was hard to program. I've seen PS3 presentations where it was primarily used as a software defined GPU, because it was just too difficult to use as a general purpose processor. Now NOT being a general purpose process is the whole point of Dojo, so maybe they can make it work. But from my limited experience with CUDA, virtual memory and direct access to DRAM is a major plus, even if the high performance compute routines make intensive use of shared memory. The fact that an interface processor is involved (how?) in managing your local SRAM must make synchronization much more complex than with CUDA, where everything is handled by the same SM that manages the calculations: your warp issues a load, it waits on a barrier, the calculations happens, sometimes in a side unit in which case you again wait on a barrier, you offload the data and wait on a barrier. And while one warp waits on a barrier, another warp can take over. It's pretty straightforward. The Dojo model suggests that "wait on a barrier" becomes "wait on the interface processor".

rowanG077 4y ago
You are right that this won't work well with any language that assumes a "normal" processor. But a small language that is written for it could be fine.
ThrownAllTheWay 4y ago
Also investing in a branch predictor when the intended workload doesn't seem at all scalar is a confusing choice to me. Also the 362 F16 TFLOPs sounds super impressive, except the memory bandwidth is I think 800 GB/s (or is it 5 times that? Or effectively less than that if data has to be passed along multiple hops? I'm a bit confused), which means having to do 1000 ops (or 200? or more?) on each 16 bit value loaded in. Maybe you could do that, but it feels like you'd probably end up bandwidth bound most of the time.

keepquestioning 4y ago

There was no reason to waste money for this. You don't want to focus on two incredibly resource-intense and error-prone processes in the same company (Automotive, Semiconductor). This is a hype project simply created to pump up the stock price.

AlotOfReading 4y ago
While I agree that most custom silicon is a bit silly (and the costs dramatically underestimated), it's worth noting that most of the serious companies in the AV space have some some amount of custom silicon for ML either in the works or already in production. Waymo is using TPUs for example and Cruise has its own chips as well. It's the cool thing to do in SV lately. Doesn't help that Nvidia is such a toxic partner that the prospect of not working with them makes custom silicon seem almost reasonable.
kjksf 4y ago
There's a legion of people convinced of their superior intellect with "helpful" advice about what Musk / Tesla / SpaceX should or should no do. After first AI day Tesla stock dropped significantly. Wall Street doesn't understand Tesla as anything else but a car company and won't understand AI until it start producing billions in revenues. It's exceedingly poor stock pump. They've been working on Dojo for almost 7 years. They kept it secret for the first 5. Kind of contradicts the "stock pump" memes. They start talking about it now because those efforts are very advanced and they want to hire more people.
- stephencanon 4y ago
  It has not really been a secret, they just haven’t been talking about details much. It was pretty clear to anyone who works in related areas what they were up to.
panick21_ 4y ago
Its so hilarious when people think everything Tesla does is some 4D chess about stock price, when in reality Musk really doesn't care about stock price at all. He literally thinks multiple years ahead, and doesn't change that strategy because some short term head winds. > two incredibly resource-intense and error-prone processes in the same company (Automotive, Semiconductor). If you think making these chips is excessive, you don't know Tesla. Tesla is the most vertically integrated car company in the world (maybe BYD compares). Compared to literally designing not only their own batteries cells, but their own battery chemistries plus their own battery materials process AND their own fully up battery manufacturing factories. They started doing this a decade ago and only this year the first cars rolled of the line and being sold to costumers. That is years and years of investment, planning and internal capacity building. In 2017 people were laughing at them for attempting to do all their own manufactriong. Now they have industry dominating automotive margin. They literally have their own materials team that makes their own aluminum for casting and have co-developed production machines that are the size of a whole house. This is now slowly being copied by other manufactures. They also make their own glass, seats and a whole host of other things that most car companies don't make. Doing these things in-house cost billions and billions in capx. Also, Dojo is not the only chip they make, they have other chips that are already being sold in their cars by the millions every year. Dojo is a pittance of investment compared to that and a fraction of the total investment into the overall self-driving stack. Just because you don't agree with the overall strategy (assuming from your comment) doesn't mean that they many, many billions Tesla spends isn't real, or its all some sort of stock price play. So, if you think Dojo is about stock price, I don't know what to tell you, its just wrong.

Animats 4y ago

OK, but why? Is this somehow supposed to make their flaky "self-driving" work? Or what? Certainly you can build custom chips for deep learning. It's mostly a simple inner loop, replicated many times, after all. Ideal case for ASICs. But will this actually benefit Tesla as a car company?

Sirened 4y ago

neat :) Love seeing other people building new and weird architectures. I wonder what their actual use case for SMT was, I don't really buy the "one thread for vector, one thread for pulling data into SRAM". I'm also a bit surprised to see they didn't go for a VLIW ISA, it always seemed like these tightly integrated data processing chips were the ideal candidates since binary compatibility isn't an issue when you're building your own HW.

TekMol 4y ago

The main concern mentioned in the comments seems to be that programmers will find it hard to work with this hardware. But is that really the intention? Wouldn't it be enough if there is one program written for it, one that takes a bunch of inputs and a bunch of outputs and then creates a set of NN weights that perform well? So the AWS style offer would not be "rent our hardware and program on it" but "rent our hardware/software stack, throw your input/output pairs at it and get back a set (billions) of NN weights that perform well".

WithinReason 4y ago
But then why have e.g. a branch predictor instead of making it more specialised?
- amelius 4y ago
  Perhaps because they will still run multiple programs, just not at the same time. Or because the program behaves differently in different places, and it is too big to optimize.

mrb 4y ago

In terms of absolute compute performance per chip, perf per watt, and perf per die area, it looks like Dojo matches or surpasses the best GPUs of today: «Tesla claims a die with 354 Dojo cores can hit 362 BF16 TFLOPS at 2 GHz» For comparison, the fastest single-chip GPU today is the AMD MI250X which has 220 "cores" (compute units) totaling 383 BF16 TFLOPS at 1.7 GHz, and that's a monster 560 watt chip. The Dojo chip is likely under this 560 was TDP, so more efficient. And Tesla provides roughly the same compute performance, but the chip is arrayed in 61% more cores, meaning it is far more suitable to handle branchy code. Also Tesla claims the die measures only 645 mm², compared to 1540 mm² for AMD. So the wafer fabrication cost is roughly half(!) of AMD's! If Tesla has truly managed to build that, I'm impressed. Edit: I missed that Tesla claims "less than 600 watt" per chip. So we know its comparable or less than AMD (560 watt). Edit 2: 25 dies are packed on a single system-on-wafer. That's 15 kW on a disc of 30 cm (12 in) of diameter. Sheesh! That must require an ungodly liquid cooling system! Edit 3: there is more info, including rendering of the host interface card at: https://www.servethehome.com/tesla-dojo-custom-ai-supercomputer-at-hc34/ https://www.servethehome.com/tesla-dojo-custom-ai-supercompu... and https://www.servethehome.com/tesla-dojo-ai-system-microarchitecture/ https://www.servethehome.com/tesla-dojo-ai-system-microarchi... Edit 4: found a pic of the liquid cooling system - as expected ;) https://media.datacenterdynamics.com/media/images/training_tiles_III.original.png https://media.datacenterdynamics.com/media/images/training_t... source: https://www.datacenterdynamics.com/en/news/tesla-details-dojo-supercomputer-reveals-dojo-d1-chip-and-training-tile-module/ https://www.datacenterdynamics.com/en/news/tesla-details-doj... And they say the first tile was "tested last week" as of August 20th... This confirms my suspicion that the system is barely (?) functional. Also see "Venkataramanan appeared to even surprised Andrej Karpathy, Tesla’s head of AI, on stage by revealing for the first time that Dojo training tile ran one of his neural networks" from https://electrek.co/2021/08/20/tesla-dojo-supercomputer-worlds-new-most-powerful-ai-training-machine/ https://electrek.co/2021/08/20/tesla-dojo-supercomputer-worl...

deleted 4y ago
[deleted]
WithinReason 4y ago
A 2x efficiency improvement from a GPU to a specialised ASIC not particularly impressive. How much would you gain by removing the graphics related stuff from a GPU (texture pipelines, vertex processing, etc.?). In addition, they lose existing programming models like OpenCL and the compiler progress that happened in the last 10 years and have to roll their own. The amount of SW work needed to support this must be a lot to get the same ease of use out of it as GPUs. Maybe they made it more CPU like to make it easier to program?

frozenport 4y ago

AMD pays for this flexibility by spending about 44% of “Zeppelin” die area on logic other than cores and cache. Dojo only spends 28.9% of die area on things other than SRAM and cores. Doesn't sound worth it? You're basically spending a ton of money to fab a chip for what amounts to like a 30% training performance boost? Just buy more chips.

rbanffy 4y ago

The same way we like to talk about odd OSs, like Lisp Machines, Plan 9, Oberon, Smalltalk, I'd love to see one of these things used as a desktop CPU (since I never got to play with one of the desktoppable Xeon Phi's) just to see the weird and interesting OSs and applications that would emerge from it. I know it must be horrendously difficult to program for general purpose use, but that's the point - figuring out what CAN be done and what an OS for it would look like.

mannyv 4y ago

Discussions about how this will fail to be productized as a cloud service are premature. There are many, many ways to do SAAS. If the benefit is there people will find a way to use it. In any case the problem is moving data, then the compute side. If Tesla is really going to do it they have someone working on that already. SAAS is not rocket science, and the issues are well-known.