3 ms·
It just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies
by irthomasthomas 1y ago
It just means that once you send your test questions to a model API, that company now has your test. So 'private' benchmarks take it on faith that the companies won't look at those requests and tune their models or prompts to beat them.
- dmos62 1y agoSounds a bit presumptious to me. Sure, they have your needle, but they also need a cost-efficient way to find it in their hay stack.
- noodletheworld 1y agoSecurity through obscurity is not security. Your api key is linked to your credit card, which is linked to your identity. …but hey, youre right. Lets just trust them not to be cheating. Cool.
- lucianbr 1y agoThey have quite large amounts of money. I don't think they need to be very cost-efficient. And they also have very smart people, so likely they can figure out a somewhat cost-efficient way. The stakes are high, for them.
- merelysounds 1y agoWould the model owners be able to identify the benchmarking session among many other similar requests?
- irthomasthomas 1y agoDepends. Something like arc-agi might be easy as it follows a defined format. I would also guess that the usage pattern for someone running a benchmark will be quite distinct from that of a normal user, unless they take specific measures to try to blend in.