3 ms·
https://marginlab.ai/trackers/claude-code/ https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve see
by conception 2mo ago
https://marginlab.ai/trackers/claude-code/ https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.
- nerdsniper 2mo agoI mean they could just be routing known benchmark questions (which all of SWEBench are) to a full-performance variant.
- svnt 2mo agoIs this using the api or using a subscription, though? The incentives are different for each, and it isn't the least bit unexpected that they would maintain API access quality while 'optimizing' the subscription experience to improve their margins (or losses) It seems to do really this you would need to crowdsource it -- users individually give the lab access to a body of subscriptions normally used by average people, and the lab occasionally runs some masked version of the task through on diverse accounts.