3 ms·
Out of curiosity, what kind of costs are we incurring per query? I don't know that much about these AI systems aside from that it's apparently too big to run o
by lucb1e 4y ago
Out of curiosity, what kind of costs are we incurring per query?
I don't know that much about these AI systems aside from that it's apparently too big to run on my own PC. Having a powerful but regular desktop system temporarily dedicated to answering my queries is quite a lot of compute power already and apparently that's not enough for something like Dall-E or chatGPT.
Perhaps to avoid being pinned down on this later ("why is this so expensive if it only costs X"), it's probably good to multiply any answer by five or ten, or factor in some of the development time/costs.
- vineyardmike 4y agoI can't find the citation, but someone related to OpenAI said that ChatGpt cost a few cents per query. Anecdotally, I looked at running an open-source GPT competitor from Meta (OPT), and the requirements for running that are vaguely on the scope of "8 GPUs with at least 400gb of GPU memory" which would be VERY expensive, even in the cloud. An 8gpu AWS EC2 instance costs $3.40/hr (p2.8xlarge), and if the model runs for 20s per request (roughly how long chatGPT takes me to get a response), that gives you $0.02 a request, not including things like storage or idle capacity. So a request could likely be $0.05-0.1, not including overhead like dev costs, training, etc.
- chrisbaker98 4y ago> I can't find the citation, but someone related to OpenAI said that ChatGpt cost a few cents per query. Here's the citation, from Sam Altman of OpenAI: https://twitter.com/sama/status/1599671496636780546 https://twitter.com/sama/status/1599671496636780546 Also, this person who doesn't appear to actually be affiliated with OpenAI estimates the running costs at $100k per day: https://twitter.com/tomgoldsteincs/status/1600196995389366274 https://twitter.com/tomgoldsteincs/status/160019699538936627...