3 ms·

There was a video on YT (gamers nexus I think) and an excel sheet comparing jumps in performance gains between each new nvidia generation. They are becoming sma

by stanac 1y ago

There was a video on YT (gamers nexus I think) and an excel sheet comparing jumps in performance gains between each new nvidia generation. They are becoming smaller and smaller, probably now driven by AI boom where most of the silicon is used for data centers. Regardless of that, I have a feeling we are approaching the ceiling of chip performance. Just comparing PS3 and PS4 and then PS4 with PS5, performance jump is smaller and size of the hardware has become enormous and GPUs are more and more power hungry. If generational jumps were good enough we wouldn't need more power and cooling and big boxes for desktop PCs that can hold long graphic cards.

bobthepanda 1y ago
We also have hit ceilings of performance demand. As an example, 8K TVs never really got off the ground because your average consumer could give a hoot. Vision Pro is a flop because AR/VR are super niche. Crypto is more gambling than asset class. Etc. What is interesting is that it seems like the ever larger sums of money sloshing around are resulting in bigger, faster hype cycles. We are already seeing some companies face issues after blowback from adopting AI too fast.
- jay_kyburz 1y ago
  Having a local AI might kick start consumer interest in hardware again. I haven't thought about buying a beefy PC in 10 years, but I was wondering how much I would have to spend to run a cutting edge LLM here at home. (It might be too expensive to pay for LLM subscriptions when every device in your house is "thinking" all day long. A 3-5k Computer for a local llm might pay itself off after a year or two. )
  - bobthepanda 1y ago
    I don’t think this is super likely. The crypto boom showed how quickly a GPU, power-intensive computing demand mostly meant putting the chips outside the accessible price range for the consumer, and the market never really quite recovered.
- bitmasher9 1y ago
  It’s funny, but I’m actually very interested in a 8k display with the right parameters. I like to use large displays for monitors, and the pixel density of 4k at a large display is surprisingly low.
imtringued 1y ago
Most of the performance gains also came from smaller datatypes such as fp4. Going from bfloat16/fp16 to fp8 is easy. fp4 is challenging but possible. fp2 can't exist independently, because it is just the sign bit plus a single bit. The next frontier would be training directly with block floating point, where you have a shared exponent plus the two remaining bits. It's getting tight. Maybe it is possible to have mini LoRA blocks where an n times n block is approximated by the outer product of two n sized vectors. For n = 4 the savings would be 50% less FLOPs and for n=8 the savings would be 75% less FLOPs.

by stanac 1y ago

bobthepanda 1y ago

We also have hit ceilings of performance demand. As an example, 8K TVs never really got off the ground because your average consumer could give a hoot. Vision Pro is a flop because AR/VR are super niche. Crypto is more gambling than asset class. Etc. What is interesting is that it seems like the ever larger sums of money sloshing around are resulting in bigger, faster hype cycles. We are already seeing some companies face issues after blowback from adopting AI too fast.

jay_kyburz 1y ago
Having a local AI might kick start consumer interest in hardware again. I haven't thought about buying a beefy PC in 10 years, but I was wondering how much I would have to spend to run a cutting edge LLM here at home. (It might be too expensive to pay for LLM subscriptions when every device in your house is "thinking" all day long. A 3-5k Computer for a local llm might pay itself off after a year or two. )
- bobthepanda 1y ago
  I don’t think this is super likely. The crypto boom showed how quickly a GPU, power-intensive computing demand mostly meant putting the chips outside the accessible price range for the consumer, and the market never really quite recovered.
bitmasher9 1y ago
It’s funny, but I’m actually very interested in a 8k display with the right parameters. I like to use large displays for monitors, and the pixel density of 4k at a large display is surprisingly low.

imtringued 1y ago

Most of the performance gains also came from smaller datatypes such as fp4. Going from bfloat16/fp16 to fp8 is easy. fp4 is challenging but possible. fp2 can't exist independently, because it is just the sign bit plus a single bit. The next frontier would be training directly with block floating point, where you have a shared exponent plus the two remaining bits. It's getting tight. Maybe it is possible to have mini LoRA blocks where an n times n block is approximated by the outer product of two n sized vectors. For n = 4 the savings would be 50% less FLOPs and for n=8 the savings would be 75% less FLOPs.