3 ms·
>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT. Yes, and it doesn't even come close. Llama2-70b can run inference
by ramesh31 3y ago
>So the comparison would be the cost of renting a cloud GPU to run Llama vs querying ChatGPT.
Yes, and it doesn't even come close. Llama2-70b can run inference at 300+tokens/s on a single V100 instance at ~$0.50/hr. Anyone who can should be switching away from OpenAI right now.
- thewataccount 3y agoWhat's the best way to use LLama2-70b without existing infrastructure for orchestrating it?
- ramesh31 3y ago>What's the best way to use LLama2-70b without existing infrastructure for orchestrating it? That's an exercise left to the reader for now, and is where your value/moat lies.
- thewataccount 3y ago> That's an exercise left to the reader for now, and is where your value/moat lies. Hopefully more on-demand services enter the space. Currently where I am we don't have the resources for any type of self orchestration and our use case is so low/sporadic that we can't simply have a dedicated instance. Last I saw the current services were rather expensive but I should recheck.
- mjirv 3y agoI stumbled upon OpenRouter[0] a few days ago. Easiest I’ve seen by far (if you want SaaS, not hosting it yourself). [0] https://openrouter.ai https://openrouter.ai
- pdntspa 3y agoI bought an old server off ServerMonkey for like $700 with a stupid amount of RAM and CPUs and it runs Llama2-70b fine, if a little slowly. Good for experimenting
- cheptsov 3y agoHow do you fit Llama2-70b into V100? V100 is 16GB. Llama2-70b 4bit would require up to 40GB. Also, what do you use for inference to get 300+tokens/s?