3 ms·
Testing in Colab: Loaded into 27.7GB of VRAM, requiring an A100 (without quantization). Inferences are speedy, looks promising for a local solution compared t
by courseofaction 3y ago
Testing in Colab:
Loaded into 27.7GB of VRAM, requiring an A100 (without quantization).
Inferences are speedy, looks promising for a local solution compared to other models which have been released recently.
- MuffinFlavored 3y ago> Inferences are speedy, looks promising for a local solution compared to other models which have been released recently. Is there any kind of standardized test to gauge the quality (not the speed) of LLM answers? aka, how hard does it hallucinate?
- rgovostes 3y agoThere is the Language Model Evaluation Harness project which evaluates LLMs on over 200 tasks. HuggingFace has a leaderboard tracking performance on a subset of these tasks. https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- oidar 3y agoWould you mind sharing that notebook?
- courseofaction 3y agoSure, https://colab.research.google.com/drive/1r4FAveF9t8b8PNiqpRHACPdqSjKKV-Ai https://colab.research.google.com/drive/1r4FAveF9t8b8PNiqpRH... :)