4 ms·

i have a high volume task i wrote an eval for and was pleasantly surprised at 2.0 flash's cost to value ratio especially compared to gpt4.1-mini/nano accuracy

jay_kyburz 1y ago
Can I ask a serious question. What task are you writing where its ok to get 7% error rate. I can't get my head around how this can be used.
- spruce_tips 1y ago
  low stakes text classification but it's something that needs to be done and couldnt be done in reasonable time frames or at reasonable price points by humans
- omneity 1y ago
  In my case, I have workloads like this where it’s possible to verify the correctness of the result after inference, so any success rate is better than 0 as it’s possible to identify the “good ones”.
  - nonethewiser 1y ago
    Aren’t you basically just saying you are able to measure the error rate? I mean that’s good, but already a given in this scenario where hes reporting the 7% error rate.
    - jsnell 1y ago
      No. If you're able to verify correctness of individual items of work, you can accept the 93% of verified items as-is and send the remaining 7% to some more expensive slow path. That's very different from just knowing the aggregate error rate.
    - yjftsjthsd-h 1y ago
      No, it's anything that's harder to write than verify. A simple example is a logic puzzle; it's hard to come up with a solution, but once you have a possible answer it's really easy to check it. In fact, it can be easier to vet multiple answers and tell the machine to try again than solve it once manually.
- dist-epoch 1y ago
  [flagged]
  - wavewrangler 1y ago
    Yeah, general propaganda and psyops are actually more effective around 12% - 15%, we find it is more accurate to the user base, thus is questioned less for standing out more /s
- 16bytes 1y ago
  There are tons of AI/ML use-cases where 7% is acceptable. Historically speaking, if you had a 15% word error rate in speech recognition, it would generally be considered useful. 7% would be performing well, and <5% would be near the top of the market. Typically, your error rate just needs to be below the usefulness threshold and in many cases the cost of errors is pretty small.
- muzani 1y ago
  I expect some manual correction after the work is done. I actually mentally counted all the times I pressed backspace while writing this paragraph, and it comes down to 45. I'm not counting the next paragraph or changing the number. Humans make a ton of errors as well. I didn't even notice how many I was making here until I started counting it. AI is super useful to just write get a first draft out, not for the final work.
- sroussey 1y ago
  You could be OCRing a page that includes a summation line, then add up all the numbers and check against the sum.

jay_kyburz 1y ago

Can I ask a serious question. What task are you writing where its ok to get 7% error rate. I can't get my head around how this can be used.

spruce_tips 1y ago
low stakes text classification but it's something that needs to be done and couldnt be done in reasonable time frames or at reasonable price points by humans
omneity 1y ago
In my case, I have workloads like this where it’s possible to verify the correctness of the result after inference, so any success rate is better than 0 as it’s possible to identify the “good ones”.
- nonethewiser 1y ago
  Aren’t you basically just saying you are able to measure the error rate? I mean that’s good, but already a given in this scenario where hes reporting the 7% error rate.
  - jsnell 1y ago
    No. If you're able to verify correctness of individual items of work, you can accept the 93% of verified items as-is and send the remaining 7% to some more expensive slow path. That's very different from just knowing the aggregate error rate.
  - yjftsjthsd-h 1y ago
    No, it's anything that's harder to write than verify. A simple example is a logic puzzle; it's hard to come up with a solution, but once you have a possible answer it's really easy to check it. In fact, it can be easier to vet multiple answers and tell the machine to try again than solve it once manually.
dist-epoch 1y ago
[flagged]
- wavewrangler 1y ago
  Yeah, general propaganda and psyops are actually more effective around 12% - 15%, we find it is more accurate to the user base, thus is questioned less for standing out more /s
16bytes 1y ago
There are tons of AI/ML use-cases where 7% is acceptable. Historically speaking, if you had a 15% word error rate in speech recognition, it would generally be considered useful. 7% would be performing well, and <5% would be near the top of the market. Typically, your error rate just needs to be below the usefulness threshold and in many cases the cost of errors is pretty small.
muzani 1y ago
I expect some manual correction after the work is done. I actually mentally counted all the times I pressed backspace while writing this paragraph, and it comes down to 45. I'm not counting the next paragraph or changing the number. Humans make a ton of errors as well. I didn't even notice how many I was making here until I started counting it. AI is super useful to just write get a first draft out, not for the final work.
sroussey 1y ago
You could be OCRing a page that includes a summation line, then add up all the numbers and check against the sum.