4 ms·

The cost per question seems super high. I can't even think of an API where a single call would cost $1cent. You better have a good pricing model to follow up on

by h1fra 3y ago

The cost per question seems super high. I can't even think of an API where a single call would cost $1cent. You better have a good pricing model to follow up on this.

beebmam 3y ago
Really? I sure can. I can even imagine a single API request that triggers a job that costs $100,000. These APIs exist. They're not for individuals.
- h1fra 3y ago
  I mean yes, I have managed BigQuery jobs that cost thousand of dollars per run but the actual call is not the thing that is expensive and I'm in control of what it costs me. It's not exactly the same scenario imo
  - ainesh93 3y ago
    I think the focus here isn't necessarily on compute cost. When companies hire data scientists or analysts, they're niche-skilled and expensive. If those people spend 50-60% of their time courting ad-hoc questions from various people in the org, the cost of that employee's time (and the money spent on them doing menial tasks that are a waste of their skillset) is the biggest factor.
IanCal 3y ago
It's about 0.2 cents right? It's hard for something you interact with manually to provide positive value less than 0.2c.
- thewataccount 3y ago
  That's for the non-finetuned GPT3.5turbo model. Finetuned is 1.2cents/1k in and 1.6cents/1k out. So it'll likely be closer to 2cents depending on what you're doing. I'm not saying it's not useful, at 2c per query you have to be more "purposeful" as they could certainly add up depending on how you use it compared to 0.2c.
  - IanCal 3y ago
    Ah thanks I read that wrong.
riku_iki 3y ago
> I can't even think of an API where a single call would cost $1cent. paying engineer for the same job will be many factors more expensive.
Zetobal 3y ago
If your app is able to function better for the cost of the API calls the value might just be there.
claytonjy 3y ago
Google Speech-to-Text is one I've dealt with recently; costs over 1¢ to transcribe 1 minute of audio. Almost 8¢/minute for the medical version.
- h1fra 3y ago
  1¢ per 1minute is fair as it would take the best translator much more time than that. 1¢ per sql query seems less fair to me, but we could argue it would also cost more to ask a dev.
  - eli 3y ago
    It would obviously cost much more to pay a person to write SQL for you
Legend2440 3y ago
What? Some of the APIs I use at work cost $100+ per call. (it's not really about the call, it's about the data it provides)
jug 3y ago
This is basically what put me off when trying to use GPT-3.5 for this with SQL LangChain. And you can’t go cheaper LLM from OpenAI because then accuracy plummeted. GPT-3.5 is the sweet spot. Sure, some use cases might work but it’s not going to be a thing that Just Works™ for products even accuracy issues aside. There’s just so much data to feed into each and every prompt, schemas and all. Many of them too if you want to enable joins.

by h1fra 3y ago

The cost per question seems super high. I can't even think of an API where a single call would cost $1cent. You better have a good pricing model to follow up on this.

beebmam 3y ago

Really? I sure can. I can even imagine a single API request that triggers a job that costs $100,000. These APIs exist. They're not for individuals.

h1fra 3y ago
I mean yes, I have managed BigQuery jobs that cost thousand of dollars per run but the actual call is not the thing that is expensive and I'm in control of what it costs me. It's not exactly the same scenario imo
- ainesh93 3y ago
  I think the focus here isn't necessarily on compute cost. When companies hire data scientists or analysts, they're niche-skilled and expensive. If those people spend 50-60% of their time courting ad-hoc questions from various people in the org, the cost of that employee's time (and the money spent on them doing menial tasks that are a waste of their skillset) is the biggest factor.

IanCal 3y ago

It's about 0.2 cents right? It's hard for something you interact with manually to provide positive value less than 0.2c.

thewataccount 3y ago
That's for the non-finetuned GPT3.5turbo model. Finetuned is 1.2cents/1k in and 1.6cents/1k out. So it'll likely be closer to 2cents depending on what you're doing. I'm not saying it's not useful, at 2c per query you have to be more "purposeful" as they could certainly add up depending on how you use it compared to 0.2c.
- IanCal 3y ago
  Ah thanks I read that wrong.

riku_iki 3y ago

> I can't even think of an API where a single call would cost $1cent. paying engineer for the same job will be many factors more expensive.

Zetobal 3y ago

If your app is able to function better for the cost of the API calls the value might just be there.

claytonjy 3y ago

Google Speech-to-Text is one I've dealt with recently; costs over 1¢ to transcribe 1 minute of audio. Almost 8¢/minute for the medical version.

h1fra 3y ago
1¢ per 1minute is fair as it would take the best translator much more time than that. 1¢ per sql query seems less fair to me, but we could argue it would also cost more to ask a dev.
- eli 3y ago
  It would obviously cost much more to pay a person to write SQL for you

Legend2440 3y ago

What? Some of the APIs I use at work cost $100+ per call. (it's not really about the call, it's about the data it provides)

jug 3y ago

This is basically what put me off when trying to use GPT-3.5 for this with SQL LangChain. And you can’t go cheaper LLM from OpenAI because then accuracy plummeted. GPT-3.5 is the sweet spot. Sure, some use cases might work but it’s not going to be a thing that Just Works™ for products even accuracy issues aside. There’s just so much data to feed into each and every prompt, schemas and all. Many of them too if you want to enable joins.