4 ms·
Pricing: $0.25/million tokens of input, $1.25/million of output GPT-3.5 Turbo is $0.50/$1.50 I've updated the Claude 3 plugin for my LLM CLI tool to support t
by simonw 3y ago
Pricing: $0.25/million tokens of input, $1.25/million of output
GPT-3.5 Turbo is $0.50/$1.50
I've updated the Claude 3 plugin for my LLM CLI tool to support the new model: https://github.com/simonw/llm-claude-3/releases/tag/0.3 https://github.com/simonw/llm-claude-3/releases/tag/0.3
pipx install llm
llm install llm-claude-3
llm keys set claude
# Paste Anthropic API key here
llm -m claude-3-haiku 'Fun facts about armadillos'
It's pretty fast! Animated GIF here: https://github.com/simonw/llm-claude-3/issues/3#issuecomment-1995909513 https://github.com/simonw/llm-claude-3/issues/3#issuecomment...
- BoorishBears 3y agoSeeing it be pretty slow in production with long prompts: - 10-15 seconds for 400 tokens out, and 4,000-10,000 tokens in. - 6-8 seconds when using Claude Instant for the same prompts Hoping it's just a rush at launch.
- porphyra 3y agoFor comparison, Groq [1] has (price per million tokens of input vs output): Llama 2 70B (4096 Context Length) ~300 tokens/s $0.70/$0.80 Llama 2 7B (2048 Context Length) ~750 tokens/s $0.10/$0.10 Mixtral, 8x7B SMoE (32K Context Length) ~480 tokens/s $0.27/$0.27 Gemma 7B (8K Context Length) ~820 tokens/s $0.10/$0.10 [1] https://wow.groq.com/ https://wow.groq.com/
- BoorishBears 3y agoAnd zero capacity. Groq is coming across a total paper tiger. No billing, unusable rate limits, and most importantly: a request queue that makes it dramatically slower than any other option. They say they're just waiting on implementing billing, but at this point it reads more like "we wouldn't be able to meet demand of all your request usages". - Groq is going through all that to offer 500tk/s theoretically, meanwhile I'm seeing Fireworks.ai come in at 300+tk/s in production use.