11 ms·
Came here to say this. No way you need to spend more than $1500 to run L3 8B at FP16. And you can get near-identical performance at Q8 for even less. I'm guess
by causal 2y ago
Came here to say this. No way you need to spend more than $1500 to run L3 8B at FP16. And you can get near-identical performance at Q8 for even less.
I'm guessing actual break-even time is less than half that, so maybe 2 years.
- causal 2y agoFurthermore, the AWS estimates are also really poorly done. Using EKS this way is really inefficient, and a better comparison would be AWS Bedrock Haiku which averages $0.75/M tokens: https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/ This whole post makes OpenAI look like a better deal than it actually is.
- mrinterweb 2y agoI was getting that sense too. It would not be difficult to build a desktop machine with a 4090 for around $2500. I run Llama-3 8b on my 4090, and it runs well. Plus side is I can play games with the machine too :)