9 ms·
Tesla Project Dojo Overview
- deleted 5y ago[deleted]
- judge2020 5y agoarchive.org: https://web.archive.org/web/20210822201245/https://perspectives.mvdirona.com/2021/08/tesla-project-dojo-overview/ https://web.archive.org/web/20210822201245/https://perspecti... Google cache: https://webcache.googleusercontent.com/search?q=cache:ceA_7THV35wJ:https://perspectives.mvdirona.com/2021/08/tesla-project-dojo-overview/+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:ceA_7T...
- rpmisms 5y agoFitting an ExaFLOP in 10 cabinets is so powerful as to be absurd.
- volta83 5y agoExaFLOP != ExaFLOP
- volta83 5y agoNot sure what the downvotes are for, but ExaFLOP != ExaFLOP for a long time. An A100 is an ExaFLOP GPU, if you count INT8 as FLOP. Then you have the 10 different flavors of FP8, FP16, etc. and even the multiple flavors of FP32. What people usually count as "ExaFLOP" is FP64, but this hardware can't even do FP32, much less FP64. So yeah, an ExaFLOP != an ExaFLOP anymore, because these types of announcements never say "An ExaFLOP of what". Everybody's brain has an infinite througput of FP0 (0 bit FP format). That doesn't mean that your brain can compute faster than Dojo, even though infinite ExaFLOPs >>>> 1 ExaFLOP. The reason is that we are counting different things. So what's the throughput of this hardware in actual FP64 ExaFLOPs ? Zero, it doesn't support them. Not a very impressive marketing statement though. But what this means is that this hardware would be extremely bad at, e.g., solving linear systems of equations.
- CrazyStat 5y ago>So yeah, an ExaFLOP != an ExaFLOP anymore, because these types of announcements never say "An ExaFLOP of what". I thought Tesla was pretty clear about which floating point formats they were talking about. When they first introduced the D1 they put up a slide with FLOPS measurements for FP16 and FP32. The impressive numbers (exaFLOP in ten cabinets for example) are based on FP16, of course.
- londons_explore 5y agoTesla at their scale are not short of space. They can afford a few more square feet of tent or wherever they want to keep their computer. I just don't see why volumetric efficiency matters at all for this computer. All that matters is usable flop per dollar. And if you put it in a cold place with very low electricity costs (eg. iceland), the electrical efficiency doesn't matter much either. All you care about is how cheap can you make high speed transistors. The Dojo design looks like it really wasn't optimising for what matters at all.
- dantle 5y agoLocality has everything to do with performance. It's why your L1 cache is faster than your L2 cache.
- gpm 5y agoBased on their presentation and design choices (and a tiny bit of experience) it looks to me like the limiting factor is actually bandwidth and latency not flops or electricity or heating or space or even the cost of the transistors themselves. A ton of flops doesn't help if you can't feed them data to process. The reason they gave for doing things like "knocking out the walls of cabinets" was bandwidth, not space. It turns out that the highest bandwidth solutions are also quite compact. One giant piece of silicon instead of many smaller pieces of silicon is great because the different parts of the giant piece of silicon can talk to eachother very quickly with very high bandwidth. Short wires are better than long wires, because you have less noise so you can fit in more signal (also less latency). It's also just not clear to me how you imagine making things bigger would make them cheaper.
- rpmisms 5y agoI'm not a hardware person at all, but my understanding is that distributed computing systems are hard to scale due to latency, and a huge factor in latency at this scale is physical distance. This was also mentioned in the AI Day presentation, and listed as a major factor in the design of Dojo.
- TekMol 5y agoI am surprised that massive computation in one place is currently the best performing approach to this. We are searching for an algorithm which turns camera data into a vector model, right? This sounds very parallelizable. Have a lot of computers travel the search space individually. When one finds a algorithm better than the current best solution, have it call the others "Hey guys, this algo works better then what we got so far. Everyone iterate on this one now!".
- bdamm 5y agoTo farm out to cloud computing resources? Very expensive. It isn’t surprising to me that a company with a massive and known compute load would choose to pack that compute resource as efficiently as possible. Supercomputers are still a business.
- dannyw 5y agoCloud computing is also like 100-200% surcharged, even after volume discounting. If you are not rapidly growing with uncertainty over resource requirements, you're going to get far better financial outcomes by taking your infra in-house. Example: Uber.
- tetsuto 5y agoI'd slightly disagree with this - the real benefits from cloud are in the things you don't need to do. If you're just using the cloud as a normal datacentre (but in another location) you're not going to have a good time. EG using a managed Database as opposed to rolling your own saves on patching, management etc...
- dannyw 5y agoIf you're not rapidly growing and are paying millions in cloud compute, you should hire your own infra team and start managing your own database, so you can save millions of dollars.
- krasin 5y ago
- finolex1 5y agoNit: The person who presented the system is Ganesh Venkataramanan, not Genesh Venugopal, as stated in the article.
- clubdorothe 5y agoWhich manufacturer is Tesla using for their dojo chips?
- robin_reala 5y agoTSMC.
- bob1029 5y agoAnyone ever wonder about the possibility of Tesla/Elon entering into semiconductor manufacturing? I can't imagine this thought hasn't crossed some minds in their boardroom.
- stetrain 5y agoEven Apple doesn't do the actual manufacturing. I'm not sure adding another product line with crazy difficult production processes would be a good idea for Tesla.
- josephpmay 5y agoBut Tesla likes to bring production in-house, whereas Apple outsources everything possible. It would not surprise me in the slightest if Tesla is looking to build a fab at some point in the future
- stetrain 5y agoI don't think it's profitable to build a leading edge chip fab with only yourself as the customer. At least, nobody else is doing it. Even with Intel's huge sales volumes they are still trying to bring in other customers to use their fabs to share the cost. So I think getting in the fab business for Tesla would mean trying to be a direct competitor with TSMC, Intel, etc. by making chips for a large number of customers. That seems way down the list of Tesla's priorities and best uses of dropping hundreds of billions of dollars.
- 5y ago
- 01100011 5y agoPrevious discussion: https://news.ycombinator.com/item?id=28251396 https://news.ycombinator.com/item?id=28251396
- deleted 5y ago[deleted]
- Animats 5y agoIt's a nice piece of hardware, reminiscent of old IBM water-cooled mainframes. But as a solution to self-driving, it's doubling down on a failed strategy. If they haven't been able to avoid hitting solid obstacles with what they've been doing, more data won't help.
- delaaxe 5y agoDid you watch their presentation? They're definitely not trying to solve self-driving with just "more data"
- kjksf 5y agoYes, more data will help. More data did help with all other Deep Learning problems, from Go to speech synthesis. GPT-3 is orders of magnitude better than GPT-2 because it has more data. It's really Deep Learning 101. Furthermore, it's a non-sequitur of an argument. Dojo is generic neural network training chip. It can be used to speed up any and all Deep Learning problems. Self driving is one such problem. During AI Day Tesla also presented, in depth, the architecture of the Deep Learning network they use. If you have a critique of their approach or can show how a competitor's approach is better, then do share with the class. If you don't it's just lazy "Tesla bad because they solved a problem that no-one else solved"
- Weryj 5y agoIt's not just more data, it's more unique scenarios with more samples of each. More edge cases to train on, that's got to help when training on more complex dimensions on the existing model. They're not just going to bloat the dataset with straight driving.
- delaaxe 5y agoI've worked in data science and most of time, "more data" is just more noise. The guy from comma.ai says the same thing.
- berdario 5y ago1 - you should prove why you think that this problem is just like the other deep learning problems 2 - it's not true that "no-one else solved" this problem. If we define this problem as: have a self-driving vehicle which safely navigates its environment and does not endanger its passengers or the people outside. See the Waymo whitepapers: waymo.com/safety/performance-data Ultimately, even if we agree that what Tesla is working on is orders of magnitude better than what's currently available from them, that might not cut it: Let's assume that the current obstacle detection is 99% accurate... if the newer versions/improved models are 99.9%, 99.99%, etc. accurate... we'll still have a 0.1, 0.01, 0.001%, etc of chance of an obstacle not being recognized. Tesla cars are a small fraction of those on the road, and there've been several deaths already. If million of Teslas will be sold, those small percentages will still mean a significant number of accidents and deaths caused by Tesla. If instead a Lidar reliably detects all obstacles, well in advance of the vehicle approaching it (and the car will stop/disable self driving if the Lidar becomes unoperable, e.g. due to bad weather)... it would be irresponsible to persuade huge swaths of the population to let a machine drive the vehicle, without providing the extra safety that a Lidar enables.
- chronic2703 5y ago“Wide FSD beta in two weeks”, my ass. Tesla is at least a decade away from the shit Elon is tweeting about. Plan your next car purchases accordingly. Sincerely, a Model Y owner
- tenpies 5y agoAlso: Tesla can make a nice car, but doesn't like doing so very often. Please look up the pre-delivery checklists. Tesla is not like a regular manufacturer where you can walk around a vehicle and determine if you want to accept it. You have to actually test things - even the things that we've been doing right since the Model T. These things are built in tents by staff that is treated quite poorly, and you should be skeptical in proportion to that information. Here is a good example: https://github.com/polymorphic/tesla-model-y-checklist https://github.com/polymorphic/tesla-model-y-checklist Although I noticed it's missing "check that the roof and trunk are in fact water proof". Getting any of these things fixed after you take delivery is a huge PITA.
- jsight 5y agoChecklists like that make things sound much worse than they are. Common issues are really minor compared to the things listed there. And, tbh, hang around some forums for the company that has been "doing right since the Model T" and you'll see some of the same things. Roof alignment in particular has been a little bit of a problem for the Mach E. With both companies, there are some horror stories after delivery too. With both companies, the vast majority of cars are great and end up with really happy customers.
- leesec 5y agoYou're blowing things out if proportio lol. Teslas are extremely loved cars by their owners on average. I doubt they would be so highly rated if they had regular issues are you're suggesting.
- panick21 5y agoModel S Plaid just had amazing reviews by pretty much ever reviewer. Check list make sense for all manufactures. > These things are built in tents by staff that is treated quite poorly, and you should be skeptical in proportion to that information. People are obsessed with this tent. Its just a simple stable structure that you can build quickly. It changes nothing about the manufacturing line being in a building. The tend idea was set up when a guy with 30 years of manufacturing experience from BMW and other German car makers. They knew what they were doing, and its actually the cars from the tend that improved the quality problems they had early on. > Getting any of these things fixed after you take delivery is a huge PITA. Most of these things can be fixed by the mobile service, they can do it while your are not even there. Often the fix it while you work.
- bertil 5y agoAny document on what they plan to do with all that heat afterwards? Warm water for the lavatories?
- iamgopal 5y agoTesla can generate whole lot of static 3D world data location wise that can be precompute so that the car computer only have to augment it with dynamic data and hence much less compute requirement.
- ricardobeat 5y agoThat's not the approach they are taking though. There are no maps, only the trained NN is deployed to the vehicles. Making the system handle any unpredictable scenario without ever relying on precomputed maps is likely far safer.
- iamgopal 5y agoOk, I hope the black box is working, is there any research for how two trained Neural Network can be merged ?
- mensetmanusman 5y agoIf we could do that, could we merge brains?
- krona 5y agoMaking the system handle any unpredictable scenario without ever relying on precomputed maps is likely far safer. Sounds rather counterintuitive to me. I feel far safer driving on my regular commute because I know where the potholes are, where the blind junctions are despite the missing signs. I know exactly how fast I can go around the roundabout built with a ridiculous camber in wet weather that sends many people in to the ditch every year. etc. Knowing all this, I can bias my attention to the cars/people around me rather than the environment.
- moogly 5y agoThere are 0-day no maps for potholes. There are no 0-day maps for temporarily disabled traffic lights, road construction, accidents etc. It takes at least decade to build up something like Google Maps, but a lot of the data is already out-of-date.
- deleted 5y ago[deleted]
- 0xdeadb00f 5y agoThis website is not loading.
- thspimpolds 5y agoHow ironic... James is VP and DE at AWS...
- all_usernames 5y agoCan someone explain like I'm five what the massive bandwidth is necessary for?
- IHLayman 5y ago[My first attempt at an ELI5…] This is needed for massive ML models. GPT-3 for example has 175 billion parameters. The model training job simply won’t fit on a single processor’s memory by itself. To calculate these models, the bigger job has to be broken up into many smaller jobs, across different processors. But, to complete the big job, all of those smaller jobs need to talk to one another from time to time. Massive bandwidth speeds up that crosstalk so that the training can happen faster.
- IHLayman 5y ago15 kW per tile, 12 tiles per rack… 180kW per rack at 480V three-phase is 375 amps per rack. Jesus. Good luck finding a rack that can support that. Either you build your own rack (and this could be Tesla’s plan, but I couldn’t find details on that), or you put maybe 3 tiles per rack. Hell, most GPU installations won’t put more that 4 GPU servers per rack for this very reason. I can’t imagine how heavy the wiring would be.
- vinay_ys 5y ago1.25MB/node is a lot of cache.