4 ms·

Anyone has an insight into how much money labs are putting into benchmarks? Just Arg-AGI-3 is quoted above 20K USD and footnote says average of 5 runs (!!). Li

by 6thbit 2mo ago

Anyone has an insight into how much money labs are putting into benchmarks? Just Arg-AGI-3 is quoted above 20K USD and footnote says average of 5 runs (!!). Likely just a drop in the bucket to the training budget but still..

stri8ted 2mo ago
20k is small potatoes for the marketing impact.
jofzar 2mo ago
It's $0/close to 0, they aren't at 100% demand so any leftover compute is "not spent". The cost they are quoting is API cost, so it's already inflated on that.