Sorry in advance for a rant against the proliferation of LLM benchmarks: I wrote a book on using LLMs in applications 14 months ago, I am sort of a fan, within

Sorry in advance for a rant against the proliferation of LLM benchmarks: I wrote a book on using LLMs in applications 14 months ago, I am sort of a fan, within reason. I don’t like all the mind-space taken up on comparisons between LLMs on standard test suites because I personally think most instruction tuned models are specifically tuned for the standard tests. I like to see which models people are choosing to use for their projects, and of course, tools like Ollama make it easy to try many models and get a least a subjective feel for what they can do. For model comparison I have my own standard little tests that are my own and private, and I can be sure models are not tuned specifically for. I do the same for commercial LLM APIs and products surfacing models. For example, I have a few short tests I run in Bard to test integration with Google Workspace data, something I am interested in, and it is interesting to track at least subjectively the slow improvement.