4 ms·

Against better judgement, I ran it on my repo. Poking at the "Ask anything about the report" bot, it penalizes me heavily for not having "positive tracked outco

by a_t48 2mo ago

Against better judgement, I ran it on my repo. Poking at the "Ask anything about the report" bot, it penalizes me heavily for not having "positive tracked outcomes", as I don't report back to Claude what the outcome is. It wanted me to do this: "At the end of research-heavy sessions, add a tiny closure note: Decision: Use CacheMountStore with registry/local/GHA backends. Why: GHA cannot expose the same content.Ingester path, local import has discovery issues, registry can resolve by tag. Proof / current artifact: See files X, Y, Z. Subagent found A, B, C. Next action: Implement interface in package N. Do not add new cache-mount flags yet. Use mode=cache-mount on existing cache flags." There's no need for this, Claude is not my task tracker. It also penalized me for: - Using Claude to introspect a codebase as throwaway work and ` not close with a crisp “acceptable / risky / copy this / avoid this” decision this session.`. No action was needed, it wasn't an actionable session! - Using Fable for code review. `Several BuildKit subagent sessions produced research reports, but the transcript does not show whether those findings became implementation choices.` Yes, Fable launches subagents to code review. Valid findings get turned into fixes or WONTFIX. - `The Cloudflare CI upload failure got narrowed well, especially after you supplied the exact timestamp and challenged the /v2 routing recommendation, but the session ended without a chosen next diagnostic owner or action this session.` There's no need to pick a next diagnostic owner when I'm a solo founder with no employees. I'm choosing not to attach a report to my application. As a side note, nobody needs to see "my greatest crashout", it's embarrassing enough for me to see it for myself, much less show others.

drawnwren 2mo ago
I had the exact same experience. Claude is not my validation engine, I do not tell it when it has completed. I verify that in the code/project and end the session. I also got this, > A useful next habit is to end each correction with a concrete acceptance test, owner artifact, or stop condition: “write it into PROGRESS.md,” “make nix run .#bench fail until this is real,” “rerun this exact command,” or “do not proceed until these two choices are explicit.” > You already do this well in the biggest penance sessions. Apply it to the smaller ones too. Which I have found to be counterproductive in my personal work. Current models can generally infer acceptance tests of this level of granularity (not true for larger project-level prompts, but those don't produce good enough code for me yet -- even with specific acceptance criteria). I also got penalized for using claude in read-only mode for the same validation reason? > For read-only work, end with one of: “turn the top finding into a PR-sized plan” “mark these as accepted/rejected/deferred” “write a cleanup checklist” “give me the exact command I should run safely” “stop, no action recommended” No thanks, I'm literally just exploring the codebase. I don't want any of these. It's a little sad to be honest, I would actually enjoy a product that helped me improve prompting + ai usage.
- craigmccaskill 2mo ago
  Yeah, I've found similar issues with even basic tasks like 'draft an email response' which will then populate a md file tagged as draft. Claude will then chastise me a few days later for not sending said draft. I've had to take real steps to avoid NannAIy in the last month that haven't even been a problem before in similar workflows.
Aurornis 2mo ago
This tool might be useful for AI tool beginners who need some guidance, but using it as a scoring tool looks like a mistake. It’s just going to penalize people for having different ways of working. I don’t know how much it’s actually used to rate people. Rating anyone based on how they work instead of the results produced is an age-old mistake though.
- a_t48 2mo ago
  It's used for YC applications as an optional portion of the founders profile. I'm not sure how it's weighted.

by a_t48 2mo ago

drawnwren 2mo ago

I had the exact same experience. Claude is not my validation engine, I do not tell it when it has completed. I verify that in the code/project and end the session. I also got this, > A useful next habit is to end each correction with a concrete acceptance test, owner artifact, or stop condition: “write it into PROGRESS.md,” “make nix run .#bench fail until this is real,” “rerun this exact command,” or “do not proceed until these two choices are explicit.” > You already do this well in the biggest penance sessions. Apply it to the smaller ones too. Which I have found to be counterproductive in my personal work. Current models can generally infer acceptance tests of this level of granularity (not true for larger project-level prompts, but those don't produce good enough code for me yet -- even with specific acceptance criteria). I also got penalized for using claude in read-only mode for the same validation reason? > For read-only work, end with one of: “turn the top finding into a PR-sized plan” “mark these as accepted/rejected/deferred” “write a cleanup checklist” “give me the exact command I should run safely” “stop, no action recommended” No thanks, I'm literally just exploring the codebase. I don't want any of these. It's a little sad to be honest, I would actually enjoy a product that helped me improve prompting + ai usage.

craigmccaskill 2mo ago
Yeah, I've found similar issues with even basic tasks like 'draft an email response' which will then populate a md file tagged as draft. Claude will then chastise me a few days later for not sending said draft. I've had to take real steps to avoid NannAIy in the last month that haven't even been a problem before in similar workflows.

Aurornis 2mo ago

This tool might be useful for AI tool beginners who need some guidance, but using it as a scoring tool looks like a mistake. It’s just going to penalize people for having different ways of working. I don’t know how much it’s actually used to rate people. Rating anyone based on how they work instead of the results produced is an age-old mistake though.

a_t48 2mo ago
It's used for YC applications as an optional portion of the founders profile. I'm not sure how it's weighted.