5 ms·

I chose to self-host for a variety of reasons. One, I had been using gpt-3.5-turbo for a hobby project and had an abysmal accessibility experience -- I received

I chose to self-host for a variety of reasons. One, I had been using gpt-3.5-turbo for a hobby project and had an abysmal accessibility experience -- I received repeated 429s with serial requests far more sparse than the limits, at most frequent, 10 times less often than the recommended threshold. When I adjusted further the 429s kept returning. It reached a point where even llama.cpp models (on 20 cores) were definitely more performant. I received absolutely no response from customer support despite being an actual paying customer. I imagine OpenAI affiliates here will downvote this, but I found OpenAI's API accessibility to be one of the most terrible I have ever used.

pmontra 3y ago
How much are you spending to self host a LLM?
phillipcarter 3y ago
FWIW, we've been live with GPT-3.5 turbo since May and it's improved a LOT. Latency is down consistently across the board and we haven't seen a single "429 model overloaded" error in the past month: https://twitter.com/_cartermp/status/1686894576202907651/ https://twitter.com/_cartermp/status/1686894576202907651/
- waffletower 3y ago
  It is worth noting that I had the perception, incorrect or not, that larger customers were being given priority to OpenAI APIs. It is good to see that you are reporting improvements.
  - phillipcarter 3y ago
    Oh, they undoubtedly were. We have decent volume, but worked very hard in our prompting to minimize input and output tokens, so we don't have a big bill. I just think that in the past months they've added significantly more capacity and done a lot of work on their inference servers to make things run better.
    - waffletower 3y ago
      Given Meta's obvious, and most welcome, open source play with LLaMA 2, it behooves OpenAI to be performant and accessible to everyone. For my application, I am finding that 70b LLaMA 2 models are very close to the inference quality of gpt-3.5-turbo.
  - hooande 3y ago
    It's gotten better for everyone in the last few months. It used to be a nightmare, but I haven't seen a timeout or rate limit error in a long time.
justanotherunit 3y ago
Where do you host your model? I am looking around on where I can deploy one without ruining me financially.
- Mernit 3y ago
  You could easily host your model on https://beam.cloud https://beam.cloud (I'm a founder). You just add a decorator to your existing Python code: from beam import App, Runtime app = App(name="gpu-app", runtime=Runtime(gpu="T4")) @app.rest_api() def inference(): print("This is running on a GPU") Then run beam deploy {your-app}.py and boom, it's running on the cloud
  - bananapub 3y ago
    why did you decide to make such a bad pitch for your product like this?
  - jokethrowaway 3y ago
    An A10G for 1200$ per month will ruin me financially
    - coder543 3y ago
      I think the Beam website should be a lot clearer about how things work[0], but I think Beam is offering to bill you for your actual usage, in a serverless fashion. So, unless you're continuously running computations for the entire month, it won't cost $1200/mo. If it works the way I think it does, it sounds appealing, but the GPUs also feel a bit small. The A10G only has 24GB of VRAM. They say they're planning to add an A100 option, but... only the 40GB model? Nvidia has offered an 80GB A100 for several years now, which seems like it would be far more useful for pushing the limits of today's 70B+ parameter models. Quantization can get a 70B parameter model running on less VRAM, but it's definitely a trade-off, and I'm not sure how the training side of things works with regards to quantized models. Beam's focus on Python apps makes a lot of sense, but what if I want to run `llama.cpp`? Anyways, Beam is obviously a very small team, so they can't solve every problem for every person. [0]: what is the "time to idle" for serverless functions? is it instant? "Pay for what you use, down to the second" sounds good in theory, but AWS also uses per-second billing on tons of stuff... EC2 instances don't just stop billing you when they go idle, though, you have to manually shut them down and start them up. So, making the lifecycle clearer would be great. Even a quick example of how you would be billed might be helpful.
visarga 3y ago
I have had similar issues with the Azure GPT-3.5, it would respond fast sometimes, and just hang for minutes other times, not even a 429. Just blocked randomly.

pmontra 3y ago

How much are you spending to self host a LLM?

phillipcarter 3y ago

FWIW, we've been live with GPT-3.5 turbo since May and it's improved a LOT. Latency is down consistently across the board and we haven't seen a single "429 model overloaded" error in the past month: https://twitter.com/_cartermp/status/1686894576202907651/ https://twitter.com/_cartermp/status/1686894576202907651/

waffletower 3y ago
It is worth noting that I had the perception, incorrect or not, that larger customers were being given priority to OpenAI APIs. It is good to see that you are reporting improvements.
- phillipcarter 3y ago
  Oh, they undoubtedly were. We have decent volume, but worked very hard in our prompting to minimize input and output tokens, so we don't have a big bill. I just think that in the past months they've added significantly more capacity and done a lot of work on their inference servers to make things run better.
  - waffletower 3y ago
    Given Meta's obvious, and most welcome, open source play with LLaMA 2, it behooves OpenAI to be performant and accessible to everyone. For my application, I am finding that 70b LLaMA 2 models are very close to the inference quality of gpt-3.5-turbo.
- hooande 3y ago
  It's gotten better for everyone in the last few months. It used to be a nightmare, but I haven't seen a timeout or rate limit error in a long time.

justanotherunit 3y ago

Where do you host your model? I am looking around on where I can deploy one without ruining me financially.

Mernit 3y ago
You could easily host your model on https://beam.cloud https://beam.cloud (I'm a founder). You just add a decorator to your existing Python code: from beam import App, Runtime app = App(name="gpu-app", runtime=Runtime(gpu="T4")) @app.rest_api() def inference(): print("This is running on a GPU") Then run beam deploy {your-app}.py and boom, it's running on the cloud
- bananapub 3y ago
  why did you decide to make such a bad pitch for your product like this?
- jokethrowaway 3y ago
  An A10G for 1200$ per month will ruin me financially
  - coder543 3y ago
    I think the Beam website should be a lot clearer about how things work[0], but I think Beam is offering to bill you for your actual usage, in a serverless fashion. So, unless you're continuously running computations for the entire month, it won't cost $1200/mo. If it works the way I think it does, it sounds appealing, but the GPUs also feel a bit small. The A10G only has 24GB of VRAM. They say they're planning to add an A100 option, but... only the 40GB model? Nvidia has offered an 80GB A100 for several years now, which seems like it would be far more useful for pushing the limits of today's 70B+ parameter models. Quantization can get a 70B parameter model running on less VRAM, but it's definitely a trade-off, and I'm not sure how the training side of things works with regards to quantized models. Beam's focus on Python apps makes a lot of sense, but what if I want to run `llama.cpp`? Anyways, Beam is obviously a very small team, so they can't solve every problem for every person. [0]: what is the "time to idle" for serverless functions? is it instant? "Pay for what you use, down to the second" sounds good in theory, but AWS also uses per-second billing on tons of stuff... EC2 instances don't just stop billing you when they go idle, though, you have to manually shut them down and start them up. So, making the lifecycle clearer would be great. Even a quick example of how you would be billed might be helpful.

visarga 3y ago

I have had similar issues with the Azure GPT-3.5, it would respond fast sometimes, and just hang for minutes other times, not even a 429. Just blocked randomly.