13 ms·
An inside look at the custom CPUs in Tesla's Dojo Supercomputer
- hirundo 5y agoNow assume that Moore's law stays healthy and in a few decades we're carrying around dojo-equivalent smart phones. Or implants. What do our apps do with all that compute? Detailed convincing AR, accurate real-time translation, identify almost any object in sight, turing-test passing answering machine. We'll be carrying around neural-net training hardware that is different than the stuff between our ears, often worse but sometimes much better. A cyber brain. Scary af.
- jameskilton 5y agoI wish I had your optimism. All I expect are better ways to deliver targeted advertising.
- deleted 5y ago[deleted]
- pplante 5y agoWe will have 7 more layers of UI abstraction to solve the problem of not adequately warming the users hands as they use it during extreme winter weather caused by climate change.
- roca 5y agoPerformance per watt is not improving that much, nor is feature size. You'll probably never be carrying around a supercomputer with this many FLOPS.
- cma 5y agoEven with reversible computing?
- nefitty 5y agoWhat about cloud services. A 5G connection to a supercomputer on demand.
- roca 5y agoYou can have that now. That's now that the parent was hoping for.
- roca 5y agoIf we ever use reversible computing it won't look anything like this Tesla stuff, that's for sure.
- freemint 5y agoLinear algebra is reversible if you nudge the matrixes so that the determinant is non-zero.
- vlovich123 5y agoMoore’s law is quite dead. Transistor density isn’t doubling. We’re still getting speed increases due to architectural improvements, but transistor density is capping out because thermals are a problem. That’s why you see these massive wafer designs and they’re not shrinking.
- WoodenChair 5y ago> Transistor density isn’t doubling. It pretty much continues to double: https://en.m.wikipedia.org/wiki/Transistor_count#/media/File%3AMoore's_Law_Transistor_Count_1970-2020.png https://en.m.wikipedia.org/wiki/Transistor_count#/media/File...
- Baeocystin 5y agoRaw transistor count isn't the same as density.
- WoodenChair 5y agoSure, but if you look at the raw transistor count of a series of similarly sized dies (take Apple's A series for example), it holds up. It only doesn't hold up if you look at Intel the last 5 years. If you look at series' made on TSMC or Samsung, it holds up.
- smolder 5y agoIf you look at CPU performance relative to density as it has progressed over the last few decades, there's a clear decline in speed improvement. Denser only means faster to a point, regardless of Intel's process update failures.
- WoodenChair 5y agoNot sure what you’re replying to. The statement I’m replying to didn’t say anything about performance. It said “ Transistor density isn’t doubling.”
- icytrumpet 5y agoThat's not a world I want to live in. Not even a little bit.
- pbalau 5y ago> What do our apps do with all that compute? javascript
- sillysaurusx 5y agoThis isn’t far from the truth. But embrace it rather than feel upset by it. The world of programmers can be unlocked by a good API. And the underlying components can be C{,++}, as V8 is.
- mysterydip 5y agoBased on current computing resource usage: ads.
- flipbrad 5y agoIt's going to be really good at scanning your texts and photos to check they don't contain unacceptable content.
- rPlayer6554 5y ago> What do our apps do with all that compute? Detailed convincing AR, accurate real-time translation, identify almost any object in sight, turing-test passing answering machine. We'll be carrying around neural-net training hardware Or more likely they'll spend the compute power on the latest JavaScript framework....
- Keyframe 5y agoPredict weather and issue an AR warning where to seek shelter from nearby tornado and a passing heat dome, all sponsored by your telecom and facilitated by facebook, of course.
- _ph_ 5y agoIf have a recent iPhone, you are carrying some special purpose neural net hardware around in your pocket already. I think this is a trend which is going to continue. First obvious application is, that speech recognition moves onto the phones themselves, no network required any more.
- rowanG077 5y agoInsane, truly insane. Meanwhile laptop CPUs, with the notable exception of M1, are impossible to cool.
- kevingadd 5y agoNot sure that's relevant here - the reason they can cool this is because they're custom-designing housing, server racks, etc for these chips based on the amount of power they draw. You could cool pretty much anything if you were able to give it this much love and care.
- zrm 5y agoAlso, the reason most laptops run hot is that most modern high performance processors are thermally limited. It means the cooling is never "sufficient" because if you make the cooling better then you get a faster processor instead of a cooler laptop.
- panick21 5y agoIs this power from the top method something other people do?
- antattack 5y agoAwesome. I like where Tesla R&D money is going.
- etaioinshrdlu 5y agoI agree that this seems quite similar to Cerebras. Does anyone think Tesla might sell raw compute hardware?
- AtlasBarfed 5y agoThat's a business far easier to develop than a personal robot.
- mastax 5y agoWow. All that packaging and power delivery must be incredibly expensive. Yet they claim massive TCO advantage over Nvidia. Amazing what you can do when not paying for Nvidia's 60% margins ;) If this is as good as they say, they should spin this off and sell these. Probably won't.
- 01100011 5y agoDoes that include an accurate NRE estimate?
- minitoar 5y agoI'm assuming this is a reference to the dojo scene in The Matrix.
- ra7 5y ago> If their claims are true, Tesla has 1 upped everyone in the AI hardware and software field. I’m skeptical, but this is also a hardware geek’s wet dream. How is the author declaring Tesla has one upped everyone in AI hardware and software while the article has exactly zero references to TPUs?
- new_realist 5y agoAlso, these chips are as yet only in Tesla’s labs, not in production. What’s cooking in secret in NVIDIA’s labs right now?NVIDIA doesn’t have to hype their stuff, so they don’t tip their hand to competitors until it’s ready for sale. Creating a lab beast is one thing, making it useful and economical in production is another.
- shaklee3 5y agoIt's also a fictional product at this point. Even if the hardware existed, if you can't program it, it won't succeed.
- TaylorAlexander 5y agoIn their presentation yesterday they made it clear that they have a full software stack for programming it. I don't know what is marketing hype and what is real, but they suggested it is very easy for them to program for it.
- creato 5y agoEvery hardware vendor ever says that. It's probably true for the people that built the software stack. Whether it is true for third parties is TBD.
- TaylorAlexander 5y agoThere’s no third party here. As far as I can tell Tesla designed the chips and wrote the software stack to work together. They don’t want to rely on third parties for their critical infrastructure and AI chip design is vital to their success. At least this is what I can grok as an outsider.
- danceparty 5y agocustom ISA, 400W TDP. Sounds like a real pain in the ass =)
- freemint 5y agoAs long as the ISA can get compiled for from Tensorflow, Torch or whatever DeepLearning AST is used geht's comppiled down to it the user won't notice unless he is optimizing his model for the hardware in which cases he would also have to care about the equivalent implementation details
- ipsum2 5y agoTaking everything at face value: Dojo is overall a very impressive project! - Communication speed is one of the biggest bottlenecks to large models, so their bandwidth of 4TBps is very smart. - They claim a 1.3x perf/watt improvement, which is not really that great for ASICs compared to GPUs. Perf/watt is probably the most important number in datacenters. - They only use SRAM, no DRAM. This is a huge mistake, which limits their model size. You can only fit a ~10GB model inside a single tile, versus 80GB models for a single A100 GPU. - Software / compiler stack is as or more important than the hardware itself, because it dictates how much real performance you can squeeze out of the chips. I think Tesla will need to heavily focus on this area before getting anywhere close to real-world GPU performance. Overall, I imagine the project will have similar pitfalls to Cerebras.
- 01100011 5y agoYeah, a lot of that probably doesn't matter for Tesla's current internal use case but all of it matters when you're talking about commercializing it.
- deleted 5y ago[deleted]
- modeless 5y agoThey said they have DRAM on the sides where it also connects to PCI-E. I guess the chips in the middle won't be able to access it easily though.
- deleted 5y ago[deleted]
- formerly_proven 5y agoThe thermal solution and packaging looks a lot like mainframe systems from the 80s (IBM, Siemens, Japanese). Siemens H100: https://pbs.twimg.com/media/Ee56Q2bWkAA8vBE?format=jpg&name=large https://pbs.twimg.com/media/Ee56Q2bWkAA8vBE?format=jpg&name=... https://pbs.twimg.com/media/Ee56Q2ZX0AEqAWo?format=jpg&name=large https://pbs.twimg.com/media/Ee56Q2ZX0AEqAWo?format=jpg&name=... IBM: https://www.youtube.com/watch?v=xQ3oJlt4GrI https://www.youtube.com/watch?v=xQ3oJlt4GrI
- numpad0 5y agoLooks a bit like PEZY SC[1] massive multicore MIMD architecture. That one was basically shot down for grant frauds involving CEO but I wonder if this is going to be a spiritual successor. 1: https://news.mynavi.jp/article/hetero_manycore-4/ https://news.mynavi.jp/article/hetero_manycore-4/
- mola 5y agoBenchmarks?
- modeless 5y agoI thought it was strange that throughout is the same for CFP8 and BF16. Why support a fancy new 8 bit format if it's not faster than 16 bits? And there weren't any details about CFP8 so who knows what it really is.
- bagels 5y agoI think they said custom floating point?
- hencoappel 5y agoCould be more efficient than BF16?
- ericd 5y agoMemory usage?
- skee_0x4459 5y agoi cant stop thinking about the tesla bot. was he he outright lying? is it just a ploy to recruit robotics people? is it even plausible? i think the most challenging aspect of the idea is interacting with the world, picking up and handling various objects. its obvious from the presentation that fsd is very good at placing itself in space and mapping out its environment as well as devising routes even when accounting for things like other moving objects. boston dynamics has built machines that take the input of a joystick and translate that into four-legged locomotion. so if you just took FSD and bolted that on top of a spot or atlas, you would have a machine that gets you a very long way towards tesla bot. something that can walk and navigate by itself. so the question is whether or not tesla will be able to recreate what boston dynamics has done and whether or not they could do that on the time-scale that was insinuated by musk. the last time i checked, boston dynamics does not use any NN in their robots. and then theres the question of interacting with the world. i think its a fair assumption based on ai day that fsd would be able to create a very rich and accurate map of its surroundings. so that half of the question is taken care of. but how could they get it to grasp a drill and manipulate it in an intelligent way? the implication is that they would train a NN to provide that signal. but how are you going to train that? they have thousands of cars out in the world generating tons of driving data. where are they going to get data for performing these tasks? are you going to train it on every task that might be asked of it? its hypothetically possible but thats hard data to generate and label. it would take a really long time at best, and even then it would be an experiment, just as likely to fail as succeed. it seems to me that this must be a stunt where the intention is to build something that can walk around but not manipulate things or do anything useful. i would change my mind if musk came out with an explanation of how hes going to train interaction.
- pmuk 5y agoPerhaps it’ll be trained primarily through simulation?
- systemvoltage 5y agoMore here: https://news.ycombinator.com/item?id=28241884 https://news.ycombinator.com/item?id=28241884 Sorry to point out but this has nothing to do with the article which is about semiconductors.
- bsaul 5y agoI find it surprising that google, amazon and now tesla can enter the « build your own processor » market that easily. Isn’t it more likely that they’re in fact only defining the general spec, and then use the services of other companies specialized in actually designing cpus ?
- TaylorAlexander 5y agoTesla hired people who designed chips for Apple and Intel to design their chips. They are designed from the ground up for this application. It seems like they're doing all the chip design in house.
- elurg 5y agoYou can just license full CPUs and major IP blocks for stuff like IO from companies such as a ARM. The machine-learning parts are custom but the rest are probably similar to what you would find in a smartphone chip.
- _ph_ 5y agoTo design your own IC has at the same time become extremely complex and very approachable, if you have the money for it. A multi-billion transistor CPU is an insanely complex product, there are multiple layers coming together to make it possible. The base layer is the foundry, here TSMC, which does the manufacturing and provides reference design kits. They contain for example the blueprints for transistors and other electrical components. For a digital chips like processors, they are usually used as they come from the foundry. Next is a layer of software which enables the chip designs, provided by the big EDA companies. This is a rather huge layer, enabling the basic designs on the one side, but also including all kind of simulation and verification tools. And quite some engineering knowledge comes along with it. So if you want to start designing your own CPU, you still need good engineers who know what they are doing, but large parts of the whole "stack" can and have to bought from the vendors listed above. This enables the quick entries of companies into the market, who were not traditional chip design houses.
- croes 5y agoAt the end they all depend on TSMC
- cepp 5y agoRelevant cost breakdown from r/hardware: https://www.reddit.com/r/hardware/comments/p7tmuw/tesla_unveils_dojo_d1_chip_to_train_ai_models/h9mmq3w/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/hardware/comments/p7tmuw/tesla_unve...
- littlestymaar 5y ago> It appears the total tile is 15KW > For those counting at home, there are 12 total tiles per cabinet So 180kW per cabinet. I'd love to see what the cooling system looks like!
- lianqiao999 5y agoit will heat 3086kg water by 50C degree every hour, pretty easy to solve with water cooling
- fulafel 5y agoSo what's the programming model, ISA and compiler technology situation (reliance on / maturity of) like?
- fulafel 5y agoreplying to myself: according to the transcript[1] it's a custom ISA and they mentioned some things about their custom compiler, including a diagram[2]. [1]https://www.tweaktown.com/news/81229/teslas-insane-new-dojo-d1-ai-chip-full-transcript-of-its-unveiling/index.html https://www.tweaktown.com/news/81229/teslas-insane-new-dojo-... [2] https://www.tweaktown.com/image.php?image=https://static.tweaktown.com/news/8/1/81229_28_teslas-insane-new-ai-chip-has-been-revealed_full.png https://www.tweaktown.com/image.php?image=https://static.twe...
- mianos 5y agoThe rather exotic packing reminds me of the IBM ES/9000 series processors. If you like an exotic package check that one one on youtube.
- volta83 5y agoThe fact that they didn't do this: > Their current training cluster would be the 5th largest supercomputer if Tesla stopped all real workloads, ran Linpack, and submitted it to the Top500 list. which is trivial to do, and pretty much a must when bringing up the cluster to make sure its working properly, so much that most clusters do this on every maintainance, along with another bunch of benchmarks; and that they say this: > cost equivalent versus Nvidia GPU, Tesla claims they can achieve 4x the performance, 1.3x higher performance per watt, and 5x smaller footprint. but have no MLPerf results, tells you everything you need to know about it. The list of long-term hype-only AI-hardware companies with billions of dollars of VC investment and literally nothing to show is incredible and keeps growing. Every MLPerf round, the list of companies that want to submit is "huge", and 1 week before the deadline, 99.999% of them have been saying "we'll submit next round" for years. It's as-if people would spend billions on creating an F1 team, and then notice during pre-season training that the car can't even finish a lap. And then fail to even start a lap on every race of the season. And then do this again, year after year, for a decade. Burning billions and billions...
- formerly_proven 5y agoWhat's often overlooked is just because you have a shit-ton of compute nodes doesn't mean you could make it to the TOP500. You might have the compute power, but the system most likely doesn't have the connectivity. E.g. on the first slide it says this is distributed over more than three locations, which essentially guarantees that the system doesn't have supercomputer-like connectivity as a whole. Worth pointing out that the networking in a supercomputer is a very significant chunk of cost and power. And if it is tailored for AI, it might not even do 32-bit float, or only at a fraction of the "AI FLOPS".
- chrisseaton 5y agoI don't get it - they're using it for actual work, rather than burning power to run useless benchmarks for bragging rights - and you think that makes it hype?! Surely it's the opposite - running benchmarks rather than doing something useful is hype.
- 5y ago