3 ms·

Why would you benchmark the LLMs for 50% success? I expect 100% success, or nearly so, to make an LLM a practical replacement for s human. 50% success is far to

Why would you benchmark the LLMs for 50% success? I expect 100% success, or nearly so, to make an LLM a practical replacement for s human. 50% success is far too unreliable. Edit: notice that I said "100%, or nearly so". I realize that 100% is an unrealistic metric for an LLM, but come on, the robots should be at least as competent as the humans they replace, and ideally much more so.

the__alchemist 1y ago
100% - that's quite the metric!
wobblyasp 1y ago
Yah I'm not really getting why that was chosen. Maybe not 100%, but something closer to 75% would be totally workable
elif 1y ago
So every time you try to fix a bug, you succeed on the first try in the expected amount of time? Of course not. This idea that AI should be correct 100% of the time is like expecting autonomous vehicles to have a 0% crash rate to be successful. It is just a coping metric which allows humans to feel superior. In reality they already outperform humans in terms of crash rates
tbalsam 1y ago
There are versions of this kind of benchmark with a higher threshold, however, it only seems to adjust the timetables by a linear amount, so you're only buying 1-2 years or so depending on what you want that % success rate to be.
tsilmanr 1y ago
50% success threshold has the lowest variance. If you'd choose 100% or even 90%, the plot would have much higher variance and difficult to discern trend. (And would require many more test samples)
gcanyon 1y ago
I'm not saying 50% is the right number, but I can imagine that a 100% requirement would produce very noisy results, while 50% might achieve a much smoother graph that produces a (hopefully) better view into the future. It might be reasonable to look at where a 95-100% result appears relative to the 50% result for past evaluations, and then project out from the projected 50% results in the future by the same amount -- but again, it might be the case that looking for 100% results relative to 50% leads to a very noisy result that can't be generalized into the future well.
stocksinsmocks 1y ago
How many humans achieve that level of success? Even with QA processes that quintuple development cost, is it even 100%? The software I regularly use is not even 95% defect free. I am pleasantly amused to see it’s the cutting edge tech bros shaking their fist at the young LLMs on their lawn. We got free intern-quality work. Take the win.

the__alchemist 1y ago

100% - that's quite the metric!

wobblyasp 1y ago

Yah I'm not really getting why that was chosen. Maybe not 100%, but something closer to 75% would be totally workable

elif 1y ago

So every time you try to fix a bug, you succeed on the first try in the expected amount of time? Of course not. This idea that AI should be correct 100% of the time is like expecting autonomous vehicles to have a 0% crash rate to be successful. It is just a coping metric which allows humans to feel superior. In reality they already outperform humans in terms of crash rates

tbalsam 1y ago

There are versions of this kind of benchmark with a higher threshold, however, it only seems to adjust the timetables by a linear amount, so you're only buying 1-2 years or so depending on what you want that % success rate to be.

tsilmanr 1y ago

50% success threshold has the lowest variance. If you'd choose 100% or even 90%, the plot would have much higher variance and difficult to discern trend. (And would require many more test samples)

gcanyon 1y ago

I'm not saying 50% is the right number, but I can imagine that a 100% requirement would produce very noisy results, while 50% might achieve a much smoother graph that produces a (hopefully) better view into the future. It might be reasonable to look at where a 95-100% result appears relative to the 50% result for past evaluations, and then project out from the projected 50% results in the future by the same amount -- but again, it might be the case that looking for 100% results relative to 50% leads to a very noisy result that can't be generalized into the future well.

stocksinsmocks 1y ago

How many humans achieve that level of success? Even with QA processes that quintuple development cost, is it even 100%? The software I regularly use is not even 95% defect free. I am pleasantly amused to see it’s the cutting edge tech bros shaking their fist at the young LLMs on their lawn. We got free intern-quality work. Take the win.