3 ms·

Overhyped how? Overhyped like self-driving, in terms of the timelines? Or overhyped in terms of the eventual societal impact? I think people seriously underest

Overhyped how? Overhyped like self-driving, in terms of the timelines? Or overhyped in terms of the eventual societal impact? I think people seriously underestimate how much of a struggle driving down the cost of compute/$ is going to be. Every incremental gain in memory and microprocessors is getting exponentially more expensive to develop. This is also a (relatively) high interest rate environment which do not tend to favour long term speculative investments. A lot of people aren't fully aware of how OpenAI and GitHub CoPilot are bleeding red ink from the get-go. I honestly feel like people are really sleeping on the potential still. Most people I know tried it a few times around last Christmas and got bored with it and quit but I never stopped using it.

brucethemoose2 3y ago
> struggle driving down the cost of compute/$ Actually I feel that utilization of existing hardware is very poor. For instance, in LLM land, continuous batching is a thing that just starting to proliferate (at least in open source). Many are still doing training/inference in vanilla pipes in PyTorch eager mode, without even a simple `torch.compile` or aggressive quantization, much less a more advanced ML compiler. Architecture optimizations beyond llama's base architecture (like newer attention schemes used during training) are not really picked up yet. I can go on and on, and go on about Stable Diffusion as well, but what I am saying is there is tons of unpicked low hanging fruit, that's unexploited because... everyone is in a rush to just publish or deploy something, I guess? Also, Nvidia's margins are insane. The voltage/clock band they run in is insanely inefficient. I hope that is not sustainable, because sane margins/clocks with no code/hardware advancement would bring costs down a ton.
- npsomaratna 3y ago
  Can you explain more about the voltage/clock band please? Do you mean that they are very energy inefficient?
  - brucethemoose2 3y ago
    Yes. Basically, power usage increases quadradicaly with clockspeed. Running a little faster takes a lot more power. Nvidia runs GPUs smaller than an Apple M1 Ultra at 250-450W. Thats crazy for a single chip! They could sacrifice a tiny bit of performance and run with much less power... But they don't. The Cerebras CS2, for reference, has a much lower power density and runs more efficiently. Unfortunately most other AI hardware is pretty hotly clocked at the moment.
    - npsomaratna 3y ago
      Understood, thank you. I appreciate the explanation.

brucethemoose2 3y ago

> struggle driving down the cost of compute/$ Actually I feel that utilization of existing hardware is very poor. For instance, in LLM land, continuous batching is a thing that just starting to proliferate (at least in open source). Many are still doing training/inference in vanilla pipes in PyTorch eager mode, without even a simple `torch.compile` or aggressive quantization, much less a more advanced ML compiler. Architecture optimizations beyond llama's base architecture (like newer attention schemes used during training) are not really picked up yet. I can go on and on, and go on about Stable Diffusion as well, but what I am saying is there is tons of unpicked low hanging fruit, that's unexploited because... everyone is in a rush to just publish or deploy something, I guess? Also, Nvidia's margins are insane. The voltage/clock band they run in is insanely inefficient. I hope that is not sustainable, because sane margins/clocks with no code/hardware advancement would bring costs down a ton.

npsomaratna 3y ago
Can you explain more about the voltage/clock band please? Do you mean that they are very energy inefficient?
- brucethemoose2 3y ago
  Yes. Basically, power usage increases quadradicaly with clockspeed. Running a little faster takes a lot more power. Nvidia runs GPUs smaller than an Apple M1 Ultra at 250-450W. Thats crazy for a single chip! They could sacrifice a tiny bit of performance and run with much less power... But they don't. The Cerebras CS2, for reference, has a much lower power density and runs more efficiently. Unfortunately most other AI hardware is pretty hotly clocked at the moment.
  - npsomaratna 3y ago
    Understood, thank you. I appreciate the explanation.