4 ms·
How do you go about determining the aptitude of a particular LLM with a particular stack? What does your evaluation look like?
by hiatus 2y ago
How do you go about determining the aptitude of a particular LLM with a particular stack? What does your evaluation look like?
- simonw 2y ago"Vibes" Seriously. The only way that works for me is to spend a bunch of time messing around with a model trying out different things. I really want to have a more repeatable process for this. I've been tinkering with a way of setting up my own automated evals to run against different models but I'm a fair way off having something usable there. For programming libraries I find that my assumptions usually hold: if a live set is widely used and has been around for a few years it's likely to get good results. I picked the WTForm Python library recently for this reason.