4 ms·
One weird trick is to tell the LLM to ask you questions about anything that’s unclear at this point. I tell it eg to ask up to 10 questions. Often I do multiple
by manmal 1y ago
One weird trick is to tell the LLM to ask you questions about anything that’s unclear at this point. I tell it eg to ask up to 10 questions. Often I do multiple rounds of these Q&A and I‘m always surprised at the quality of the questions (w/ Opus). Getting better results that way, just because it reduces the degrees of freedom in which the agent can go off in a totally wrong direction.
- deadbabe 1y agoThis is a little anthropomorphic. The faster option is to tell it to give you the full content of an ideal context for what you’re doing and adjust or expand as necessary. Less back and forth.
- manmal 1y agoCan you give me the full content of the ideal context of what you mean here?
- rzzzt 1y agoCertainly!
- 7thpower 1y agoIt’s not though, one of the key gaps right now is that people do not provide enough direction on the tradeoffs they want to make. Generally LLMs will not ask you about them, they will just go off and build. But if you have them ask, they will often come back with important questions about things you did not specify.
- deadbabe 1y agoThey don’t know what to ask. They only assemble questions according to training data.
- fuzzzerd 1y agoWhile true, the questions are all points where the LLM would have "assumed" an answer and by asking you get to point in the right direction instead.
- 7thpower 1y agoIt seems like you are trying to steer toward a different point or topic. In the course of my work, I have found they ask valuable clarifying questions. I don’t care how they do it.
- MrDunham 1y agoThis is the correct answer. I like to go one step further than the root comment: Nearly all of my "agents" are required to ask at least three clarifying questions before they're allowed to do anything (code, write a PRD, write an email newsletter, etc) Force it to ask one at a time and it's event better, though not as step-function VS if it went off your initial ask. I think the reason is exactly what you state @7thpower: it takes a lot of thinking to really provide enough context and direction to an LLM, especially (in my opinion) because they're so cheap and require no social capital cost (vs asking a colleague / employee—where if you have them work for a week just to throw away all their work it's a very non-zero cost).
- iaw 1y agoMy routine is: Prompt 1: <define task> Do not write any code yet. Ask any questions you need for clarification now. Prompt 2: <answer questions> Do not write any code yet. What additional questions do you have? Reiterate until questions become unimportant.
- b112 1y ago[flagged]
- manmal 1y agoThe questions it asks are usually domain specific and pertaining to the problem, like modeling or „where do I get this data from ideally“.
- b112 1y agoNot blaming you, it's actually genius. You're simulating what it's seen, and therefore getting the end result -- peer discussed and reviewed SO code.
- elif 1y agoPlease keep comments in this thread non-fiction.
- th0ma5 1y agoLolll everything in this thread is unfalsifiable conjecture??
- ModernMech 1y agoAlthough you're being voted down probably for tone, this is a very interesting point.
- rpdillon 1y agoIs it? By far the majority of code the LLMs are trained on is going to be from Git repositories. So the idea that stack overflow question and answer sections with buggy code is dominating the training sets seems unlikely. Perhaps I'm misunderstanding?
- ModernMech 1y ago> Perhaps I'm misunderstanding? The post wasn't saying that StackOverflow Q&A sections with buggy code dominate the training sets. They're saying that despite whatever amount of code in there from Git repositories, the process of generating and debugging code cannot be found in the static code that exists in Github repos; that process is instead encoded in the conversations on SO, git hub issues, various forums, etc. So if you want to start from buggy code and go to correct code in the way the LLM was trained, you would do that by simulating the back and forth found in a SO question, so that when the LLM is asked for the next step, it can rely on its training.
- jackphilson 1y agoI often like to just talk out out. Stream of thought. Gives it full context of your mental model. Talk through an excalidraw diagram
- manmal 1y agoI do that as well, with Wispr Flow. But I still forget details that the questions make obvious.
- nativeit 1y agoThis is more or less what the "architect" mode is in KiloCode. It does all the planning and documentation, and then has to be switched to "Code" in order to author any of it. It allows me to ensure we're on the same page, more or less, with intentions and scope before giving it access to writing anything. It consumes ~30-40% of the tokens associated with a project, in my experience, but they seem to be used in a more productive way long-term, as it doesn't need to rehash anything later on if it got covered in planning. That said, I don't pay too close attention to my consumption, as I found that QwenCoder 30B will run on my home desktop PC (48GB RAM/12GB vRAM) in a way that's plenty functional and accomplishes my goals (albeit a little slower than Copilot on most tasks).
- CuriouslyC 1y agoWorkflow improvement: Use a repo bundler to make a single file and drop your entire codebase in gemini or chatgpt. Their whole codebase comprehension is great and you can chat for a long time without the api cost. You can even get them to comment on each other's feedback, it's great.
- manmal 1y agoUnfortunately that only works with very small projects.
- CuriouslyC 1y agoI can get useful info from gemini on 120k loc projects with repomix ignoring a few select files. If you're in the enterprise world obviously it's a different thing.