My exact thinking. It is the wrong end to measure it. As with nearly all tech benchmarks. They measure success rate, if LLM is 95% accurate it is good enough as

My exact thinking. It is the wrong end to measure it. As with nearly all tech benchmarks. They measure success rate, if LLM is 95% accurate it is good enough as human on average only get about 97%. What we dont measure is the difference between the 5% and 3% of failure rate. Some of the error in 5% error couldn't have happened to any human or are basically non-human error. Vs say 3% of human error that are different type. I still haven't found a term for this. And I have been saying it since Apple's Keyboard butterfly fiasco. When Apple supporters keep saying the key press has 99.99% success rate so the problem with the keyboard were amplified. What they dont realise was the old scissors keyboard has infinitely close to 100%, or 99.99999999999999999% success rate so to speak. A normal consumer comparing intuitively will felt the butterfly keyboard is 0.01 / 0.000000000000001 many times worst because the error doesn't happen before with previous keyboard. Since I dont see anyone online doing this I am going to call it Ksec's Law.