3 ms·
I wonder how long until training today’s ChatGPT will cost $1000 of AWS compute. 10 years? At that point, does it keep scaling or is there an S curve where 10
by epicureanideal 4y ago
I wonder how long until training today’s ChatGPT will cost $1000 of AWS compute. 10 years?
At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement?
- thfuran 4y agoI think you're underestimating the scale by a few orders of magnitude.
- the8472 4y ago> At that point, does it keep scaling or is there an S curve where 100x more data and compute only leads to a 2x improvement? Careful with the scales. 2x improvement could be interpreted from 80% human performance to 160% human performance. Or going from 10% error rate to 5% (which again crosses into superhuman territory on some tasks). Those last few bits are the critical ones.
- pphysch 4y agoWe would need to see incredible advances in energy efficiency for that to happen.
- simplyluke 4y agoWe are! Don't even have to reach for fusion potentially being commercial technology to show it. Solar is already approaching $0.03/kilowatt hour and likely to be half of that by the end of the decade. Energy getting very cheap coupled with computing capacity continuing to go way up is going to enable lots of interesting new technologies beyond LLMs