3 ms·
At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideati
by meowface 3mo ago
At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.
- girvo 3mo agoI use benchmarks… …that are my own private internal suite on my own code bases where I can judge the output properly I also measure wall clock time to completion which has been a surprising separator in practice.