4 ms·
More about my LLM tool (and Python library) here: https://llm.datasette.io/ https://llm.datasette.io/ Here's the full implementation of that llm-replicate plug
by simonw 3y ago
More about my LLM tool (and Python library) here: https://llm.datasette.io/ https://llm.datasette.io/
Here's the full implementation of that llm-replicate plugin: https://github.com/simonw/llm-replicate/blob/0.2/llm_replicate/__init__.py https://github.com/simonw/llm-replicate/blob/0.2/llm_replica...
If you want to write a plugin for some other LLM I have a detailed tutorial here: https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html https://llm.datasette.io/en/stable/plugins/tutorial-model-pl... - plus a bunch of examples linked from here: https://github.com/simonw/llm-plugins https://github.com/simonw/llm-plugins
- Anticlockwise 3y agoCan you or anyone else comment on how replicate's per-second pricing ends up comparing to OpenAI's per token pricing when using Llama2?
- simonw 3y agoMy hunch is that OpenAI is a lot cheaper. I've spent $0.26 on 115 seconds of compute with Llama 2 on Replicate so far, which is only a dozen test prompts.
- ta988 3y agoIt is insanely more expensive on replica and they don't have the 70b model yet which will make it even more prohibitive.
- richdougherty 3y agoLooks like it's here now: https://replicate.com/replicate/llama70b-v2-chat https://replicate.com/replicate/llama70b-v2-chat As for pricing, that model's pages says: "Predictions run on Nvidia A100 (80GB) GPU hardware. Predictions typically complete within 17 seconds." And the pricing page (https://replicate.com/pricing https://replicate.com/pricing) says Nvidia A100 (80GB) GPU hardware costs $0.0032 per second. So Llama 2 70B would "typically" cost under 17 x 0.0032 = $0.0544 per run.
- ta988 3y agoThank you for checking that.
- zyl1n 3y agoI am not familiar with Replicate, but based on their website, they charge per GPU type. I didn't see the GPU type set in the example. Is it baked in as part of the "a16z-infra/llama13b-v2-chat" model?
- DougBTX 3y agoThere's info about that here: https://replicate.com/a16z-infra/llama13b-v2-chat https://replicate.com/a16z-infra/llama13b-v2-chat > Run time and cost > Predictions run on Nvidia A100 (40GB) GPU hardware. Predictions typically complete within 9 seconds.