3 ms·

> Not to mention that training a model is a "one time deal", where a successfully trained model can be reused by a lot of client (devices). Inference is still

> Not to mention that training a model is a "one time deal", where a successfully trained model can be reused by a lot of client (devices). Inference is still done on GPUs, and is not cheap by any means. The number of GPUs you need to run the full 175B GPT-3 model in inference is vastly inferior to what you need during training, but the numbers of replicas of the model running in the backend to serve all customer's requests is vastly superior. So it's more akin to a 747 being bought by OpenAI and kept continuously running than just one transatlantic flight.

ben_w 4y ago
Assuming the energy costs $60 per megawatt hour and that OpenAI isn’t making a loss on their “Davinci” tier, that’s 3-7 entire Bibles of output for the $ cost of the cheapest economy seats on the cheapest London to New York flights I found for the rest of this year, or (Edit: I first claimed 25 million) 1.5 billion [0] tokens per flight, using the previously estimated energy cost per 10 hour flight. Their “Ada” tier is 75 times cheaper, with corresponding upper-bounds assumptions if they are not making a loss. [0] https://www.wolframalpha.com/input?i=1500+MWh+*+%2460%2FMWh+%2F+%28%240.0600%2Fthousand+items%29 https://www.wolframalpha.com/input?i=1500+MWh+*+%2460%2FMWh+...
- wyre 4y ago
  Without a carbon tax comparing the price of two massively energy consuming entities doesn’t mean much of anything in regards to our climate crisis.
  - ben_w 4y ago
    From the point of view of climate change, the fact that data centres are often and increasingly carbon neutral (because that’s the cheapest source of electricity, no carbon tax needed) while jet fuel isn’t usually derived from biofuel means there’s a significant possibility, albeit not a certainty, of a divide by zero error.
Der_Einzige 4y ago
CPU inference, even of large transformers, is usually cheaper and is viable with enough strong CPUs.
- sailingparrot 4y ago
  CPU inference of even the smaller version of GPT-3 would be way too slow for a public API. What CPU did you benchmark on that gave you a cheaper inference price than GPU? In my experience, for GPT like transformers, they don’t come anywhere near what you can squeeze out of something like a Nvidia T4 in terms of either performance or $/token.

ben_w 4y ago

Assuming the energy costs $60 per megawatt hour and that OpenAI isn’t making a loss on their “Davinci” tier, that’s 3-7 entire Bibles of output for the $ cost of the cheapest economy seats on the cheapest London to New York flights I found for the rest of this year, or (Edit: I first claimed 25 million) 1.5 billion [0] tokens per flight, using the previously estimated energy cost per 10 hour flight. Their “Ada” tier is 75 times cheaper, with corresponding upper-bounds assumptions if they are not making a loss. [0] https://www.wolframalpha.com/input?i=1500+MWh+*+%2460%2FMWh+%2F+%28%240.0600%2Fthousand+items%29 https://www.wolframalpha.com/input?i=1500+MWh+*+%2460%2FMWh+...

wyre 4y ago
Without a carbon tax comparing the price of two massively energy consuming entities doesn’t mean much of anything in regards to our climate crisis.
- ben_w 4y ago
  From the point of view of climate change, the fact that data centres are often and increasingly carbon neutral (because that’s the cheapest source of electricity, no carbon tax needed) while jet fuel isn’t usually derived from biofuel means there’s a significant possibility, albeit not a certainty, of a divide by zero error.

Der_Einzige 4y ago

CPU inference, even of large transformers, is usually cheaper and is viable with enough strong CPUs.

sailingparrot 4y ago
CPU inference of even the smaller version of GPT-3 would be way too slow for a public API. What CPU did you benchmark on that gave you a cheaper inference price than GPU? In my experience, for GPT like transformers, they don’t come anywhere near what you can squeeze out of something like a Nvidia T4 in terms of either performance or $/token.