4 ms·

Not to rain on the parade here but nvidia is not going to haul a 1 to 2 order of magnitude in a single jump. Top500 measures fp64, nvidia likes to quote fp8. Bl

Not to rain on the parade here but nvidia is not going to haul a 1 to 2 order of magnitude in a single jump. Top500 measures fp64, nvidia likes to quote fp8. Blackwell is actually a regression on fp64 performance to provide more die space for AI compute. This isn't to say the silicon isn't good just different goals being hit.

schaefer 2y ago
I’m aware. In the keynote Nvidia is quoting FP4 performance for Blackwell specifically. Frontier and every computer that has ever hit the supercomputer top 500 is measuring fp64. This is why I qualified my statement above by saying if you’re willing to compare apples and oranges. But I think this is where things open up to debate (and/or personal interpretation). My view is, if all you care about is AI workload (Specifically LLMs), then you really are seeing a full two orders of magnitude improvement. And if we are trying to get a feel for the space, and what that even means, then there really is nothing else to compare Blackwell to other than frontier, in terms of scale alone. Above I say “one or two orders of magnitude”, the “one order of magnitude” is a value I got by taking Blackwell’s FP4 values and dividing by 16, to try to conceptually convert back to a value that can be compared to fp64. — I’m perfectly happy to admit that all I’m really capable of doing here is using estimates to compare a completely new thing (the new compute architecture of Blackwell) to what came before it (the majority of the history of the supercomputing 500). But that’s okay, because that’s my only goal. —- Believe me, I get it. Blackwell will never post a linpack score. Blackwell will never run big scientific computing jobs like… global weather simulations. But as computer science nerds, how could we not be excited to see such a big move: changing the computer architecture to better suit a specific compute workload (LLMs)? I know Bitcoin has its asic processors, but to me Blackwell’s mission statement of “computing intelligence” is a lot more interesting.

schaefer 2y ago

I’m aware. In the keynote Nvidia is quoting FP4 performance for Blackwell specifically. Frontier and every computer that has ever hit the supercomputer top 500 is measuring fp64. This is why I qualified my statement above by saying if you’re willing to compare apples and oranges. But I think this is where things open up to debate (and/or personal interpretation). My view is, if all you care about is AI workload (Specifically LLMs), then you really are seeing a full two orders of magnitude improvement. And if we are trying to get a feel for the space, and what that even means, then there really is nothing else to compare Blackwell to other than frontier, in terms of scale alone. Above I say “one or two orders of magnitude”, the “one order of magnitude” is a value I got by taking Blackwell’s FP4 values and dividing by 16, to try to conceptually convert back to a value that can be compared to fp64. — I’m perfectly happy to admit that all I’m really capable of doing here is using estimates to compare a completely new thing (the new compute architecture of Blackwell) to what came before it (the majority of the history of the supercomputing 500). But that’s okay, because that’s my only goal. —- Believe me, I get it. Blackwell will never post a linpack score. Blackwell will never run big scientific computing jobs like… global weather simulations. But as computer science nerds, how could we not be excited to see such a big move: changing the computer architecture to better suit a specific compute workload (LLMs)? I know Bitcoin has its asic processors, but to me Blackwell’s mission statement of “computing intelligence” is a lot more interesting.