7 ms·

The machine learning space probably has the easiest time to adopt a new number format. The players are big enough to define their own silicon and training/infer

The machine learning space probably has the easiest time to adopt a new number format. The players are big enough to define their own silicon and training/inference don't even need the same number format. As long as it works on their new specialized training hardware it's good to go. The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.

totoglazer 4y ago
Hardware is not cheap either at this scale. Reallly rough math follows, please anyone correct me if I err. A100 cards are >$10,000 each. A node is typically 8 and a cluster might be 64 A100s. So about a million in hardware for the medium cluster of 64 A100s. (Cross checking, a machine from lambda labs is apparently ~175k for 8 gpus, so 1.4m for the cluster above) Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.
- mirker 4y ago
  It’s still more complicated because you have to amortize costs over several years and then also keep in mind that big players will design their own hardware to avoid the A100 cost and energy. Disks and other devices will also consume power. And you datacenter cooling is now more expensive, etc.
  - totoglazer 4y ago
    Yes, it’s complicated, for sure. But if we amortize over 3 years, and triple costs for the power: it’s ~500k/yr for hardware and ~100k/yr for power. In terms of TPUs or other custom accelerators, sure, they exist. However most definitely aren’t building their own hardware. ETA: I’m not saying power is irrelevant, it clearly matters. But saying it’s the dominant financial constraint is clearly wrong, at least below Google/Amazon/Apple scale. Never mind the cost of the people running these trainings!
- threatripper 4y ago
  I see your numbers with the A100. However, consider the 3090 which is down to $1000 and uses 350W. Double that if you consider cooling and overhead in a data center. In Europe the electricity prices are through the roof and easily cost >0.5$/kWh. If you use those numbers a single card would cost you 0.35kW * 2 * 0.5$/kWh * 24h * 365 = 3066$ per year. So I guess it highly depends on your specific location and circumstances.

totoglazer 4y ago

Hardware is not cheap either at this scale. Reallly rough math follows, please anyone correct me if I err. A100 cards are >$10,000 each. A node is typically 8 and a cluster might be 64 A100s. So about a million in hardware for the medium cluster of 64 A100s. (Cross checking, a machine from lambda labs is apparently ~175k for 8 gpus, so 1.4m for the cluster above) Those A100 use 300watts each, plus let’s say 500watts for the rest. (8gpus300+500)8servers ≈ 25Kw * 8760 hours = 219k KwH / year. So if your costs are $0.15/KwH that’s only on the order of $32k/year.

mirker 4y ago
It’s still more complicated because you have to amortize costs over several years and then also keep in mind that big players will design their own hardware to avoid the A100 cost and energy. Disks and other devices will also consume power. And you datacenter cooling is now more expensive, etc.
- totoglazer 4y ago
  Yes, it’s complicated, for sure. But if we amortize over 3 years, and triple costs for the power: it’s ~500k/yr for hardware and ~100k/yr for power. In terms of TPUs or other custom accelerators, sure, they exist. However most definitely aren’t building their own hardware. ETA: I’m not saying power is irrelevant, it clearly matters. But saying it’s the dominant financial constraint is clearly wrong, at least below Google/Amazon/Apple scale. Never mind the cost of the people running these trainings!
threatripper 4y ago
I see your numbers with the A100. However, consider the 3090 which is down to $1000 and uses 350W. Double that if you consider cooling and overhead in a data center. In Europe the electricity prices are through the roof and easily cost >0.5$/kWh. If you use those numbers a single card would cost you 0.35kW * 2 * 0.5$/kWh * 24h * 365 = 3066$ per year. So I guess it highly depends on your specific location and circumstances.