Ask HN: Is there a tool for testing the performance of multiple LLMs / LLM APIs?

I'm looking to build banks of system / user prompts, some with single correct answers (e.g. multiple choice or numerical answer), and others with subjective manual grading, and then administer them to Claude 3, ChatGPT-4, Gemini, Mistral, etc. I'd rather not implement my own tool unless I have to - is there an off the shelf solution (either cloud or self-hosted) for this?