3 ms·
I actually just benchmarked Llama3 70B coding with aider, and it did quite well. It scored similar to GPT 3.5. You can use Llama3 70B with aider via Ollama [0]
by anotherpaulg 2y ago
I actually just benchmarked Llama3 70B coding with aider, and it did quite well. It scored similar to GPT 3.5.
You can use Llama3 70B with aider via Ollama [0]. It's also available for free via Groq [1] (with rate limits). And OpenRouter has it available [2] for low cost on their paid api.
[0] https://aider.chat/docs/llms.html#ollama https://aider.chat/docs/llms.html#ollama
[1] https://aider.chat/docs/llms.html#groq https://aider.chat/docs/llms.html#groq
[2] https://aider.chat/docs/llms.html#openrouter https://aider.chat/docs/llms.html#openrouter
- typpo 2y agoPaul's benchmarks are excellent and they're the first thing I look for to get a sense of a new model performance :) For those looking to create their own benchmarks, promptfoo[0] is one way to do this locally: prompts: - "Write this in Python 3: {{ask}}" providers: - ollama:chat:llama3:8b - ollama:chat:phi3 - ollama:chat:qwen:7b tests: - vars: ask: a function to determine if a number is prime - vars: ask: a function to split a restaurant bill given individual contributions and shared items Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases. [0] https://github.com/typpo/promptfoo https://github.com/typpo/promptfoo
- jkh1 2y agoAnother benchmark: https://www.biorxiv.org/content/10.1101/2024.04.19.590278v2.full.pdf https://www.biorxiv.org/content/10.1101/2024.04.19.590278v2....