5 ms·
Generate instructions for their simulator to compute some numbers (hashes) in whatever is considered the memory of their "machine"¹. I didn't see any places whe
by measurablefunc 9mo ago
Generate instructions for their simulator to compute some numbers (hashes) in whatever is considered the memory of their "machine"¹. I didn't see any places where they actually disallow cheating b/c it says they only check the final state of the memory² so seems like if you know the final state you could just "load" the final state into memory. The cycle count is supposedly the LLM figuring out the fewest number of instructions to compute the final state but again, it's not clear what they're actually measuring b/c if you know the final state you can cheat & there is no way to tell how they're prompting the LLM to avoid the answers leaking into the prompt.
¹https://github.com/anthropics/original_performance_takehome/blob/main/problem.py#L64 https://github.com/anthropics/original_performance_takehome/...
²https://github.com/anthropics/original_performance_takehome/blob/main/problem.py#L567 https://github.com/anthropics/original_performance_takehome/...
- saagarjha 9mo agoWell, they read your code in the actual hiring loop.
- measurablefunc 9mo agoMy point still stands. I don't know what the LLM is doing so my guess is it's cheating unless there is evidence to the contrary.
- saagarjha 9mo agoWhy do you assume it’s cheating?
- measurablefunc 9mo agoBecause it's a well know failure mode of neural networks & scalar valued optimization problems in general: https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z
- red75prime 9mo agoAnd? Anthropic is not aware of this 2020 paper? The problem is not solvable?
- measurablefunc 9mo agoWhy are you asking me? Email & ask Anthropic.
- red75prime 9mo agoObviously, because you use this old paper as an argument.
- measurablefunc 9mo agoI don't have any insider information on what they know or don't know so you're welcome to keep asking nonsensical questions but eventually I'll stop answering.
- saagarjha 9mo agoAgain, you can just read the code
- measurablefunc 9mo agoYou're missing the point. There is no evidence to support their claims which means they are more than likely leaking the memory into the LLM prompt & it is cheating by simply loading constants into memory instead of computing anything. This is why formal specifications are used to constrain optimization. Without proof that the code is equivalent you might as well just load constants into memory & claim victory.
- fc417fc802 9mo ago> There is no evidence to support their claims Do you make a habit of not presuming even basic competence? You believe that Anthropic left the task running for hours, got a score back, and never bothered to examine the solution? Not even out of curiosity? Also if it was cheating you'd expect the final score to be unbelievably low. Unless you also suppose that the LLM actively attempted to deceive the human reviewers by adding extra code to burn (approximately the correct number of) cycles.
- red75prime 9mo agoI guess your answer to "Try to run Claude Code on your own 'ill-defined' problem" would be "I'm not interested." Correct? I think we can stop here then.
- KeplerBoy 9mo agoWell that's certainly a challenge when you use LLMs for this test driven style of programming.