5 ms·

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://re

Seems like serverless is the way to go for fast output while remaining inexpensive. e.g. https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://replicate.com/stability-ai/stablelm-tuned-alpha-7b https://github.com/runpod/serverless-workers/tree/main/workers/gpt-neox https://github.com/runpod/serverless-workers/tree/main/worke... https://modal.com/docs/guide/ex/falcon_gptq https://modal.com/docs/guide/ex/falcon_gptq

tikkun 3y ago
I think that's true if you're doing minimal usage / low utilization, otherwise a dedicated instance will be cheaper.
- menzoic 3y ago
  You are correct. The pricing model guarantees this. Pay per compute vs pay for uptime (during which you could have more compute for cheaper)
baobabKoodaa 3y ago
I think cold start times will be excessive for serverless in this use case.
- SparkyMcUnicorn 3y ago
  3 second cold start is good enough for me.