3 ms·
tbh companies like anthopic, openai, create custom agents for specific benchmarks
by guluarte 1y ago
tbh companies like anthopic, openai, create custom agents for specific benchmarks
- bazmattaz 1y agoDo you have a source for this? I’m intrigued
- guluarte 1y agohttps://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5a... "we iteratively refine prompting by analyzing failure cases and developing prompts to address them."
- amelius 1y agoAren't good benchmarks supposed to be secret?
- wkat4242 1y agoThis industry is currently burning billions a month. With that much money around I don't think any secrets can exist.
- noodletheworld 1y agoHow can a benchmark be secret if you post it to an API to test a model on it? "We totally promise that when we run your benchmark against our API we won't take the data from it and use to be better at your benchmark next time" :P If you want to do it properly you have to avoid any 3rd party hosted model when you test your benchmark, which means you can't have GPT5, claude, etc. on it; and none of the benchmarks want to be 'that guy' who doesn't have all the best models on it. So no. They're not secret.
- dmos62 1y agoHow do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.
- irthomasthomas 1y agoIt just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies won't look at those requests and tune their models or prompts to beat them.
- dmos62 1y agoSounds a bit presumptious to me. Sure, they have your needle, but they also need a cost-efficient way to find it in their hay stack.
- noodletheworld 1y agoSecurity through obscurity is not security. Your api key is linked to your credit card, which is linked to your identity. …but hey, youre right. Lets just trust them not to be cheating. Cool.
- lucianbr 1y agoThey have quite large amounts of money. I don't think they need to be very cost-efficient. And they also have very smart people, so likely they can figure out a somewhat cost-efficient way. The stakes are high, for them.
- merelysounds 1y agoWould the model owners be able to identify the benchmarking session among many other similar requests?
- 1y ago
- make3 1y agoif you're able to submit multiple times, then you can learn from the hidden set