5 ms·

You don't know how to run the infra for this then. For batch workloads, I can get slightly higher throughput with a 13b llama2 fine tune on my local workstation

You don't know how to run the infra for this then. For batch workloads, I can get slightly higher throughput with a 13b llama2 fine tune on my local workstation and 2x 3090s than I can get by saturating my OpenAI api key token limits for 3.5-turbo.

herbst 3y ago
That works for batch workloads but how could I scale 0 to 30 messages that need kinda fast processing per minute? I tried with one 'small' server, but allowing bigger requests to for users as well it sometimes takes over a minute to process. So I added a second smaller one that just handles smaller queries and only if the bigger one is already in use. That works for my average workload of 1-3 messages a minute to get kinda fast responses similar to OpenAI and costs me over $1000 a month on vast.ai or ~$6000 in graphic cards (2x3090 + 2x4090 without power) But that doesn't work on the weekend where I process up to 30 messages a minute. So I need another 2 X 2x4090 or more adding another $1500 on vast.ai or $6000 in hardware. Every actually owned card would add about $50 to $70 to my monthly power cost. Also vast.ai is kinda a gamble but one of the few ways to get access to cheap consumer cards. Renting servers from actual companies in data centers highly drives the price up. All while my highest monthly OpenAI bill was about $600 so far (with generous limits). Am I seeing this wrong?
- Tostino 3y ago
  Uh, not sure how I missed that last line in your original post about this being a more real-time requirement. No, that for sure complicates things if you don't have a "base load" you can use your dedicated compute for when your real-time workload is lower than capacity. You would need to design (or use) hybrid local / cloud scaling solution here if you want to keep your costs low. Something that can spin up some workers when your request queue grows, and kills them when it shrinks.
  - herbst 3y ago
    No worries. Maybe I really missed something obvious so it was worth it to type it out :) I am fine with the fact that I am either to early or my business model is flawed.

herbst 3y ago

That works for batch workloads but how could I scale 0 to 30 messages that need kinda fast processing per minute? I tried with one 'small' server, but allowing bigger requests to for users as well it sometimes takes over a minute to process. So I added a second smaller one that just handles smaller queries and only if the bigger one is already in use. That works for my average workload of 1-3 messages a minute to get kinda fast responses similar to OpenAI and costs me over $1000 a month on vast.ai or ~$6000 in graphic cards (2x3090 + 2x4090 without power) But that doesn't work on the weekend where I process up to 30 messages a minute. So I need another 2 X 2x4090 or more adding another $1500 on vast.ai or $6000 in hardware. Every actually owned card would add about $50 to $70 to my monthly power cost. Also vast.ai is kinda a gamble but one of the few ways to get access to cheap consumer cards. Renting servers from actual companies in data centers highly drives the price up. All while my highest monthly OpenAI bill was about $600 so far (with generous limits). Am I seeing this wrong?

Tostino 3y ago
Uh, not sure how I missed that last line in your original post about this being a more real-time requirement. No, that for sure complicates things if you don't have a "base load" you can use your dedicated compute for when your real-time workload is lower than capacity. You would need to design (or use) hybrid local / cloud scaling solution here if you want to keep your costs low. Something that can spin up some workers when your request queue grows, and kills them when it shrinks.
- herbst 3y ago
  No worries. Maybe I really missed something obvious so it was worth it to type it out :) I am fine with the fact that I am either to early or my business model is flawed.