4 ms·

Fitting an ExaFLOP in 10 cabinets is so powerful as to be absurd.

• volta83 5y ago
  ExaFLOP != ExaFLOP
  • volta83 5y ago
    Not sure what the downvotes are for, but ExaFLOP != ExaFLOP for a long time. An A100 is an ExaFLOP GPU, if you count INT8 as FLOP. Then you have the 10 different flavors of FP8, FP16, etc. and even the multiple flavors of FP32. What people usually count as "ExaFLOP" is FP64, but this hardware can't even do FP32, much less FP64. So yeah, an ExaFLOP != an ExaFLOP anymore, because these types of announcements never say "An ExaFLOP of what". Everybody's brain has an infinite througput of FP0 (0 bit FP format). That doesn't mean that your brain can compute faster than Dojo, even though infinite ExaFLOPs >>>> 1 ExaFLOP. The reason is that we are counting different things. So what's the throughput of this hardware in actual FP64 ExaFLOPs ? Zero, it doesn't support them. Not a very impressive marketing statement though. But what this means is that this hardware would be extremely bad at, e.g., solving linear systems of equations.
    • CrazyStat 5y ago
      >So yeah, an ExaFLOP != an ExaFLOP anymore, because these types of announcements never say "An ExaFLOP of what". I thought Tesla was pretty clear about which floating point formats they were talking about. When they first introduced the D1 they put up a slide with FLOPS measurements for FP16 and FP32. The impressive numbers (exaFLOP in ten cabinets for example) are based on FP16, of course.
• londons_explore 5y ago
  Tesla at their scale are not short of space. They can afford a few more square feet of tent or wherever they want to keep their computer. I just don't see why volumetric efficiency matters at all for this computer. All that matters is usable flop per dollar. And if you put it in a cold place with very low electricity costs (eg. iceland), the electrical efficiency doesn't matter much either. All you care about is how cheap can you make high speed transistors. The Dojo design looks like it really wasn't optimising for what matters at all.
  • dantle 5y ago
    Locality has everything to do with performance. It's why your L1 cache is faster than your L2 cache.
  • gpm 5y ago
    Based on their presentation and design choices (and a tiny bit of experience) it looks to me like the limiting factor is actually bandwidth and latency not flops or electricity or heating or space or even the cost of the transistors themselves. A ton of flops doesn't help if you can't feed them data to process. The reason they gave for doing things like "knocking out the walls of cabinets" was bandwidth, not space. It turns out that the highest bandwidth solutions are also quite compact. One giant piece of silicon instead of many smaller pieces of silicon is great because the different parts of the giant piece of silicon can talk to eachother very quickly with very high bandwidth. Short wires are better than long wires, because you have less noise so you can fit in more signal (also less latency). It's also just not clear to me how you imagine making things bigger would make them cheaper.
  • rpmisms 5y ago
    I'm not a hardware person at all, but my understanding is that distributed computing systems are hard to scale due to latency, and a huge factor in latency at this scale is physical distance. This was also mentioned in the AI Day presentation, and listed as a major factor in the design of Dojo.