4 ms·
For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is
by jmalicki 5mo ago
For lifespan, AWS is still running a ton of T4 GPUs from 2018, that power a lot of computer vision models. A ton of these will have a long life, not all ML is about frontier LLMs.
- epolanski 5mo agoHow can it be economically viable to still run them? You can get 100x the output with the same energy use.
- deleted 5mo ago[deleted]
- jmalicki 5mo agoGood question! Maybe the capabilities of newer GPUs allow AWS to charge higher margins for them? I don't actually know.
- Marsymars 5mo agoPresumably people using AWS are paying more than they cost to run, and AWS has finite bandwidth to upgrade things due to personel, etc.
- HDBaseT 5mo agoThere has not been a "100x" in efficiency in the past 6-8 years.
- dragonwriter 5mo agoWhile the 100× is, I think, rather hyperbolic, there is a real and large efficincy difference, but its economically viable to run them because the supply of newer GPUs is insufficient to meet the demand for compute, so they can charge enough to cover costs for the old ones and a premium (relative to operating costs) for the newer ones. It would be economically unviable to run the older ones if the supply of newer ones were unconstrained, but that’s not the world we live in.
- Dylan16807 5mo agoGoing by the stats on wikipedia, T4 and B300 both do about one teraflop of half-precision math per watt? Where are the efficiency gains? Edit: It looks like they replaced INT8 and INT4 with FP8 and FP4, with the same speedups of 2x and 4x relative to FP16. That's an improvement but not that big of an improvement.
- Ekaros 5mo agoAs long as you have customers that are willing to pay more than it cost you are fine. And with AWS seemingly there is plenty of those. So question isn't is this most efficient way but will someone pay at price that is above what new hardware could attain.