3 ms·
Almost always, notes like these are going to be about greenfield projects. Trying to incorporate it in existing codebases (esp when the end user is a support i
by keerthiko 8mo ago
Almost always, notes like these are going to be about greenfield projects.
Trying to incorporate it in existing codebases (esp when the end user is a support interaction or more away) is still folly, except for closely reviewed and/or non-business-logic modifications.
That said, it is quite impressive to set up a simple architecture, or just list the filenames, and tell some agents to go crazy to implement what you want the application to do. But once it crosses a certain complexity, I find you need to prompt closer and closer to the weeds to see real results. I imagine a non-technical prompter cannot proceed past a certain prototype fidelity threshold, let alone make meaningful contributions to a mature codebase via LLM without a human engineer to guide and review.
- reubenmorais 8mo agoI'm using it on a large set of existing codebases full of extremely ugly legacy code, weird build systems, tons of business logic and shipping directly to prod at neckbreaking growth over the last two years, and it's delivering the same type of value that Karpathy writes about.
- 1123581321 8mo agoThese models do well changing brownfield applications that have tests because the constraints on a successful implementation are tight. Their solutions can be automatically augmented by research and documentation.
- mh2266 8mo agoI don't exactly disagree with this but I have seen models simply deleting the tests, or updating the tests to pass and declaring the failures were "unrelated to my changes", so it helpfully fixed them
- hnben 8mo agoYes. You have to treat the model like an eager yet incompetent worker, i.e. don't go full yolo mode and review everything they do.
- 1123581321 8mo agoI’ve had to deal with this a handful of times. You just have to make it restore the test, or keep trying to pass a suite of explicit red-green method tests it wrote earlier.
- jjfoooo4 8mo agoThat was true for me, but is no longer. It's been especially helpful in explaining and understanding arcane bits of legacy code behavior my users ask about. I trigger Claude to examine the code and figure out how the feature works, then tell it to update the documentation accordingly.
- chrisjj 8mo ago> I trigger Claude to examine the code and figure out how the feature works, then tell it to update the documentation accordingly. And how do you verify its output isn't total fabrication?
- _dark_matter_ 8mo agoYour docs are a contact. You can verify that contract using integration tests
- chrisjj 8mo agoContract? These docs are information answering user queries. So if you use a chatbot to generate them, I'd like to be reasonably sure they aren't laden with the fabricated misinformation for which these chatbots are famous.
- jjfoooo4 8mo agoIt's a very reasonable concern. My solution is to have the bot classify what the message is talking about as a first pass, and have a relatively strict filtering about what it responds to. For example, I have it ignore messages about code freezes, because that's a policy question that probably changes over time, and I have it ignore urgent oncall messages, because the asker there probably wants a quick response from a human. But there's a lot of questions in the vein of "How do I write a query for {results my service emits}", how does this feature work, where automation can handle a lot (and provide more complete answers than a human can off the top of their head)