6 ms·
Everyone should have their own "pelican riding a bicycle" benchmark they test new models on. And it shouldn't be shared publicly so that the models won't learn
by theshrike79 11mo ago
Everyone should have their own "pelican riding a bicycle" benchmark they test new models on.
And it shouldn't be shared publicly so that the models won't learn about it accidentally :)
- ggsp 11mo agoAny suggestions for a simple tool to set up your own local evals?
- theshrike79 11mo agoMy "tool" is just prompts saved in a text file that I feed to new models by hand. I haven't built a bespoke framework on top of it. ...yet. Crap, do I need to now? =)
- lobsterthief 11mo ago_Brain backlog project #77 created_
- ggsp 11mo agoYeah I’ve wondered about the same myself… My evals are also a pile of text snippets, as are some of my workflows. Thought I’d have a look to see what’s out there and found Promptfoo and Inspect AI. Haven’t tried either but will for my next round of evals
- kedihacker 11mo agoWell you need to stop them from getting incorporated into its training data
- dimava 11mo agoJust ask LLM to write one on top of OpenRouter, AI SDK and Bun To take your .md input file and save outputs as md files (or whatever you need) Take https://github.com/T3-Content/auto-draftify https://github.com/T3-Content/auto-draftify as example
- deleted 11mo ago[deleted]
- bluecalm 11mo agoI am asking the models to generate an image where fictional characters play chess or Texas Holdem. None of them can make a realistic chess position or poker game. Always something is off like too many pawns or too may cards, or some cards being ace-up when they shouldn't be.