3 ms·
>That's... probably not sustainable for four more years. If it is, it's not sustainable much past that. You're probably not going to see a ten-trillion-dollar c
by johnthewise 2y ago
>That's... probably not sustainable for four more years. If it is, it's not sustainable much past that. You're probably not going to see a ten-trillion-dollar cluster; you're definitely not going to see a one-hundred-trillion-dollar cluster, ever.
Compute is just one part, as he claims. 3^4 is just 81X, i.e. 10B to 40B of compute, certainly within the budget of a single big tech company.
He couples compute gains with algorithmic advances, which he claims there is also a trend and with other gains(gpt to chatgpt advancements, inference time gains from searching as in alphago etc) and calls it effective compute and it might increase 5 OOM in 2027.
- AnimalMuppet 2y ago> 3^4 is just 81X Yes. > i.e. 10B to 40B of compute, certainly within the budget of a single big tech company. That's not 81X. That's 4X. 81X is 10B to 810B. That's... less within the budget of a single big tech company.
- johnthewise 2y agosorry, my bad. I assumed GPT-4 compute to be around 100M-500M, so 80X compute of gpt4 would be around 10B-40B.
- AnimalMuppet 2y agoNo, my bad. When you said "10B to 40B of compute", I read that as "growing from 10B to 40B of compute", that is, a 4X increase. I failed to understand it as "growing by 81X from current compute, which would put you at about 10B to 40B". I don't know what current GPT-4 compute is. I don't know if your assumption is correct, but I certainly do not know that it is wrong.