4 ms·
Useless without comparing models with different settings. The same model with a different temperature, sampler, etc might as well be a different model. Nearly
by Der_Einzige 2y ago
Useless without comparing models with different settings. The same model with a different temperature, sampler, etc might as well be a different model.
Nearly all AI research does this whole “make big claims about what a model is capable of” and then they don’t do even the most basic sensitivity analysis or ablation study…
- vinckr 2y agoDo you have an example of someone who does it right? I would be interested to see how you can compare LLMs capabilities - as a layman it looks like a hard problem...