3 ms·
Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5. You can rent a 3090 at $0.20/h on vast.ai, or
by redox99 3y ago
Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5.
You can rent a 3090 at $0.20/h on vast.ai, or $0.40/h on runpod. Using VLLM at 400t/s that's 1440000 generated tokens. Generating that amount of tokens with GPT3.5 would be $2.88.
- agnokapathetic 3y ago> "In some specific tasks, a finetuned 7B llama can work as well as GPT3.5." "some" is doing a lot of heavy lifting here. Also: don't discount the labor cost of curating a fine tuning dataset, running a FT training run, even if the hardware is cheap.
- moffkalast 3y agoUnless your use case isn't in English in which case LLama is as useful as a one-legged man in an ass-kicking contest. LLama models only really shine for things that GPTs would refuse to even consider because of corporate RLHF, and if you need to keep your data local I suppose. For the rest they're second rate at best.