3 ms·

Also, I would expect openai to be taking a loss on each individual inference request as they also have a monthly fee, dalle, and loads of VC capital. No source

• tomjohnneill 3y ago
  I can definitely imagine they're not covering the amortised cost of the training with the cost per individual inference request. It seems less likely to me that they're making a significant loss on each subsequent request, but again no source from me on that either. Looking a bit more into this, I found this paper: https://arxiv.org/pdf/2311.16863.pdf https://arxiv.org/pdf/2311.16863.pdf. It references a table saying that text generation uses 0.047 kWh per 1000 inferences, which is 1-2 orders of magnitude lower than my estimate. Though that is for GPT2, so possibly tracks to something roughly in the ~0.001 kWh per inference for GPT3.5.
  • dartos 3y ago
    Well doesn’t the compute time for transformers scale roughly quadratically with model size? Would it make sense for power consumption to also scale roughly quadratically?
    • tomjohnneill 3y ago
      I'm not sure. The figures I've seen suggest that GPT3 required 10x more energy to train than GPT2 (e.g. https://www.nnlabs.org/power-requirements-of-large-language-models/#:~:text=Power%20Consumption%20of%20Different%20Large%20Language%20Models&text=In%20comparison%2C%20GPT%2D3%2C,more%20energy%20than%20GPT%2D2 https://www.nnlabs.org/power-requirements-of-large-language-....), so I think a roughly 1-2 order of magnitude increase in energy usage from GPT2 to GPT3.5 makes sense.