4 ms·

I would be terrified of building on top of openai’s api. At any moment they can just cut off access or mark a model legacy (or change its behaviour but keep the

by nwoli 3y ago

I would be terrified of building on top of openai’s api. At any moment they can just cut off access or mark a model legacy (or change its behaviour but keep the same name) and all my work is bust.

phillipcarter 3y ago
We've been live with them since May. Their service has only gotten better for us over time - faster and no drop in accuracy. For us, the proof is in the pudding - they take being a platform seriously. We're not wed to them, and actually need to go a different route for some of our customers due to needing to sign a DPA, but the decision to go with openai has been extremely worth it.
bugglebeetle 3y ago
You should be more afraid of how often it errors and times out.
- phillipcarter 3y ago
  Ehhh, this was a problem about 3 months ago. We used to see it regularly. Now we haven't gotten a single "429 model overloaded" error in the past month. Request volume is a little higher for us too.
  - freedomben 3y ago
    May I ask what your app does or what you're using gpt for?
    - phillipcarter 3y ago
      Natural language data querying: https://docs.honeycomb.io/working-with-your-data/query-assistant/ https://docs.honeycomb.io/working-with-your-data/query-assis... User input --> bunch of contextual stuff --> API call to openai (could be any model though) --> query executes against user data and they get a result
toomuchtodo 3y ago
You should build with the mindset that you will eventually swap out ChatGPT if needed. LLM options will only improve (Llama 2 comes to mind, but lots of folks in the race). As long as you’re prepared to decouple, you won’t get caught flat footed. I mean, you are somewhat continuously (in an automated/programatic manner) running evaluations of other options against a benchmarking system for your use case, no? If not, consider if/when resourcing permits. This not only prepares you for decoupling, but also allows you to level up if OpenAI's offerings are no longer the most optimal for LLM consumption. Go build.
freedomben 3y ago
Is that any different than building on top of some other api?
- callalex 3y ago
  I guess it boils down to how much you trust the leadership that owns the API. In this case it seems like the leadership suffers from paranoid delusions, and may therefore not be trustworthy in the long term. You will have to make that judgment for yourself though. https://www.wired.com/story/what-openai-really-wants/ https://www.wired.com/story/what-openai-really-wants/
jstummbillig 3y ago
I don't see what's particularly terrifying about building on OpenAI. Yes, services can deprecate or shut down. That's true for any old service and also true for new stuff. They might swap things around but obviously OpenAI will not die of attrition any time soon. Secondly, it's not like OpenAI has a complicated API. Far from it. Any other LLM API is basically the same, just because the components are so straight forward. So, given enough capability from competitors, as far as SaaS lock-ins go, this is fairly cushion-y. What remains is a testament of how singularly potent you think the product might be and how much it would advance your stuff beyond what any other product can do. That strikes me as a rather questionable reason to not make use of it, if getting stuff done, efficiently, is what you want.
- croddin 3y ago
  You could look at it as the chat requests being the API, which is simple yes, but in a way the prompts you would write could also be considered the API, in which case it is the most complex API ever. People aren't worried about having to change the first one, it's the prompting needing to change/ model not having the right capabilities.

by nwoli 3y ago

I would be terrified of building on top of openai’s api. At any moment they can just cut off access or mark a model legacy (or change its behaviour but keep the same name) and all my work is bust.

phillipcarter 3y ago

We've been live with them since May. Their service has only gotten better for us over time - faster and no drop in accuracy. For us, the proof is in the pudding - they take being a platform seriously. We're not wed to them, and actually need to go a different route for some of our customers due to needing to sign a DPA, but the decision to go with openai has been extremely worth it.

bugglebeetle 3y ago

You should be more afraid of how often it errors and times out.

phillipcarter 3y ago
Ehhh, this was a problem about 3 months ago. We used to see it regularly. Now we haven't gotten a single "429 model overloaded" error in the past month. Request volume is a little higher for us too.
- freedomben 3y ago
  May I ask what your app does or what you're using gpt for?
  - phillipcarter 3y ago
    Natural language data querying: https://docs.honeycomb.io/working-with-your-data/query-assistant/ https://docs.honeycomb.io/working-with-your-data/query-assis... User input --> bunch of contextual stuff --> API call to openai (could be any model though) --> query executes against user data and they get a result

toomuchtodo 3y ago

You should build with the mindset that you will eventually swap out ChatGPT if needed. LLM options will only improve (Llama 2 comes to mind, but lots of folks in the race). As long as you’re prepared to decouple, you won’t get caught flat footed. I mean, you are somewhat continuously (in an automated/programatic manner) running evaluations of other options against a benchmarking system for your use case, no? If not, consider if/when resourcing permits. This not only prepares you for decoupling, but also allows you to level up if OpenAI's offerings are no longer the most optimal for LLM consumption. Go build.

freedomben 3y ago

Is that any different than building on top of some other api?

callalex 3y ago
I guess it boils down to how much you trust the leadership that owns the API. In this case it seems like the leadership suffers from paranoid delusions, and may therefore not be trustworthy in the long term. You will have to make that judgment for yourself though. https://www.wired.com/story/what-openai-really-wants/ https://www.wired.com/story/what-openai-really-wants/

jstummbillig 3y ago

I don't see what's particularly terrifying about building on OpenAI. Yes, services can deprecate or shut down. That's true for any old service and also true for new stuff. They might swap things around but obviously OpenAI will not die of attrition any time soon. Secondly, it's not like OpenAI has a complicated API. Far from it. Any other LLM API is basically the same, just because the components are so straight forward. So, given enough capability from competitors, as far as SaaS lock-ins go, this is fairly cushion-y. What remains is a testament of how singularly potent you think the product might be and how much it would advance your stuff beyond what any other product can do. That strikes me as a rather questionable reason to not make use of it, if getting stuff done, efficiently, is what you want.

croddin 3y ago
You could look at it as the chat requests being the API, which is simple yes, but in a way the prompts you would write could also be considered the API, in which case it is the most complex API ever. People aren't worried about having to change the first one, it's the prompting needing to change/ model not having the right capabilities.