3 ms·
What about the analysis evidences?
by gchamonlive 7mo ago
What about the analysis evidences?
- Retr0id 7mo agoYou mean the Claude output? The same claude that has "regressed to the point it cannot be trusted"?
- gchamonlive 7mo agoWhat you saying the OP fabricated/hallucinated the evidence?
- Retr0id 7mo agoI'm just saying it's epistemically unrigorous to the point of being equivalent to anecdata.
- gchamonlive 7mo agoHow should one conduct such a rigourously reproducible experiment when LLMs by nature aren't deterministic and when you don't have access to the model you are comparing to from months ago?
- Retr0id 7mo agoSomething like this: https://marginlab.ai/trackers/claude-code/ https://marginlab.ai/trackers/claude-code/ (see methodology section)
- deleted 7mo ago[deleted]
- gchamonlive 7mo agoKudos for the methodology. The only question I can come up with is that if the benchmarks are representative of daily use. Anecdotal or not, we see enough reports popping up to at least elicit some suspion as to service degradation which isn't shown in the charts. Hypothesis is that maybe the degradation experienced by users, assuming there is merit in the anecdotes, isn't picked up by the kind of tracking strategy used.
- Retr0id 7mo agoIt's not my methodology to be clear, but they have picked up actual regressions that happened in the past - e.g. https://news.ycombinator.com/item?id=46815013 https://news.ycombinator.com/item?id=46815013