3 ms·
but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
by whimsicalism 2y ago
but inference costs arent high already and there are tons of hardware companies that can do relatively cheap LLM inference
- dagmx 2y agoInference costs per invocation aren’t high. Scale it out to billions of users and it’s a different story. Training is amortized over each inference, so the cost of inference also needs to include the cost of training to break even unless made up elsewhere