4 ms·

It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Ap

It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.

bartwe 4y ago
That hasn't been true for optimizing compilers either.
- ChatGTP 4y ago
  Sorry what hasn’t been true ?
visarga 4y ago
I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones. The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.
- anothernewdude 4y ago
  And? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
  - revelio 4y ago
    You can specify logit biases via the API, but yes it's limited.
sgt101 4y ago
10^16 -> 10^24 flops isn't 100x, it's 100,000,000x This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT). On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...
- jimsimmons 4y ago
  A lot of it is also backlog: modern GPUs are optimised for these workloads in a way old models weren’t

bartwe 4y ago

That hasn't been true for optimizing compilers either.

ChatGTP 4y ago
Sorry what hasn’t been true ?

visarga 4y ago

I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones. The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.

anothernewdude 4y ago
And? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
- revelio 4y ago
  You can specify logit biases via the API, but yes it's limited.

sgt101 4y ago

10^16 -> 10^24 flops isn't 100x, it's 100,000,000x This happened in 10 years (training cost of largest model according to the batshit crazy god AI article in the FT). On the one hand compute power increased by about 100x and then some lunatics came and spent 10000x on it...

jimsimmons 4y ago
A lot of it is also backlog: modern GPUs are optimised for these workloads in a way old models weren’t