3 ms·
It’s a particular sort of bug that’s harder to detect because … internal Anthropic engineers don’t apply these prompts to themselves, and in fact have access to
by vessenes 6mo ago
It’s a particular sort of bug that’s harder to detect because … internal Anthropic engineers don’t apply these prompts to themselves, and in fact have access to ‘helpful only’ models that also do not have additional limitations RL’ed in. (Or perhaps they’re RL’ed out - not sure of current training mechanisms.)
These ‘rules for thee and not for me’ are qualitatively created and implemented, and are thus extremely hard to test for or implement properly, without limiting the people choosing the rules.
- QuercusMax 6mo agoThey must have some sort of smoke tests for common operations, run in a test harness with the system prompts they force on users, right? ....Right? What kind of Mickey mouse operation are they running over there?
- subscribed 6mo agoI wouldn't bet a chocolate chip cookie on that.
- vessenes 6mo agoIn the original claude degradation followup email Boris mentioned they are upping the percentage of engineers required to use the public version of claude code. I have no idea what percentage this is, or how much of a punishment it is considered to be. :) That said, I was sympathetic to the recent bug reports —- to trigger one, you’d need to have a session that waited an hour doing nothing and then very specifically tested for in-context retrieval. I don’t want to run that test, do you want to run that test?
- QuercusMax 6mo agoIDK, sounds pretty typical for my workflow - I'll start Claude on a task, go get lunch / coffee / distracted by my pets, come back in an hour, and continue my session. I would wager that this is something that happens to most users on a regular basis.
- Majromax 6mo ago> That said, I was sympathetic to the recent bug reports —- to trigger one, you’d need to have a session that waited an hour doing nothing and then very specifically tested for in-context retrieval. I don’t want to run that test, do you want to run that test? They introduced a feature/optimization that triggered after an hour's idleness, so testing that the session continued properly afterwards seems kind of important. If nothing else, even the working-as-intended feature (context cleanup) could impact model skill in a current or future model version, so it would be well worth measuring any impact as part of the test suite.