44 ms·

The article shows everything that works for this approach. But it's a bit disingenuous. At the end: > Once this is working, Xu Hao can repeat the process for

by afro88 4y ago

The article shows everything that works for this approach. But it's a bit disingenuous. At the end: > Once this is working, Xu Hao can repeat the process for the rest of the tasks in the master plan. No, he can't. After that much back and forth and getting it to fix little things where it gives responses with the full code listing again, he would have easily hit the token limit (at least with any chat LLM capable of this quality code and conversation - ChatGPT). The LLM will start hallucinating the task list, the names of functions it wrote earlier etc. and the responses would get less and less useful with more and more "this doesn't work, can you fix X". So anyone following this approach will hit a footgun after task 1. For anyone that really wants to follow this approach, the next step is to start a new chat and copy/paste the inital requirement prompt, put the task list in there, any relevant code, adjust the instruction (ie "help me with task 2") and go from there. It is of limited utility though. By step 3 (or even 2) you end up with so much code that you're at the token limit anyway and it can't write code that fits together. Where I've found ChatGPT 4 useful is getting me going on something, providing boilerplate, and unblocking me. If you don't know how to approach a problem like the "awareness layer" (like I didn't before reading the post), you can get a great breakdown and starting point from ChatGPT. Similarly, if you're not sure how to approach that view model, or write tests etc. And if you want a first draft of code or tests. All that said, I'm looking forward to much larger and affordable token limits in future.

Tostino 4y ago
You iterate on your plan after it is generated step by step. You go and edit the prompt chain you started working on step 1 on, and modify it to start working on step 2 (including any ideas or fixes you have identified while implementing step 1. Repeat until complete. You can still absolutely hit the context limit, but you are far less likely to do so if you go back and start a new prompt chain for each different thought process you are going through with it.
- afro88 4y ago
  Great idea. But does it get hard to navigate back to something in older chat histories though? I find a new separate chat with the revised initial prompt to be easier.
  - williamcotton 4y ago
    I’ve been using another call to an LLM to write or rewrite code that is separate from the main “conversation”. What I mean is that I’ve got a dialog going with an LLM and I’ve trained it to call a build() function with instructions that then returns the function, with the text of the function kept out of the dialog with the main thread.
quijoteuniv 4y ago
It's great to see that there's now a term for the type of prompting, “generated knowledge”. I've been experimenting with this technique since the beginning, and I've noticed a significant improvement in version 4. The process involves outlining the project, creating tasks, and feeding them back to chatGPT as you progress. This approach has helped me complete projects that would have otherwise taken me much longer to finish. It's also useful for creating practical tutorials. While there are plenty of tutorials available online, sometimes you need guidance on a specific set of technologies. By using generated knowledge prompts, you can get a good outline and tasks to help you understand how these technologies interact. One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness
- afro88 4y ago
  Absolutely, and same here. I've done multiple tools that would have taken 2-3 days each in 2-3 hours each. > One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness Definitely. Great advice. Another tip: don't bother asking it to fix small things. Just mention you fixed it in the next reply and move on.
- AzzieElbab 4y ago
  Something must be wrong with me. I could never get anything useful from Martin Fowler's writings, and coincidentally I cannot get any functional code out of ChatGPT. Even the boilerplate it produces for me needs to be corrected. I still use chatGPT to produce examples of abstract things but was not able to get any working code that matches concrete problems or even compiles.
  - afro88 4y ago
    Are you using the GPT4 model? There's a very significant improvement between 3.5 (the free one) and 4.
ryanjshaw 4y ago
Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way you have - it's honestly the best anti-procrastination tool I've ever used because I can tell it my intentions, what I've thought of so far... and it'll spit out a list of bite-sized chunks that get me going. I find myself looking forward to telling the AI I've completed a task. Similarly, if I'm facing a tricky design decision, I find that just writing it out for ChatGPT is extremely helpful for clarifying my thought process. I actually used to do this conversational decision making process in a text editor long before ChatGPT, but when I know there's an AI on the other end my thinking becomes clearer and more goal-oriented. And unlike talking to myself or a human friend, it's happy to just say "well if these are your concerns, let's start HERE and then see what happens".
- travisjungroth 4y ago
  Good rule of thumb with ChatGPT: you can’t exit loops. Once you’ve gone A > B > A, your best move is to start a new chat. Even then it may reproduce and you should do some similar but different task. Remember that it’s a prediction engine, weighing heavily on the existing prompt. So you say B again, or B1 and it’s like, I know what to do! A! Cause last time was A->B so let’s do it again. In your case this would be “[]Task1”, “Task1 is done”, “[]Task1”, [here is where you start a new chat or fix it yourself if possible].
  - sycren 4y ago
    Instead of starting a new chat, why not change the prompt higher up in the conversation with the relevant detail that you have gained through the responses?
    - travisjungroth 4y ago
      I think that would work and be better in some cases. The thing is you want to change things up, lose some context. I may not be able to go far enough back without losing progress to do that. But I could copy just the current state of the task (last response) to a new chat.
matchagaucho 4y ago
he would have easily hit the token limit That was my first question. Do all these tasks fit within a 4K or 8K buffer? Wouldn't be surprised, though, if it works in a 32K GPT4 token limit. Amazing things are possible.
blastro 4y ago
I completely agree with this take.
heyyyouu 4y ago
This may be a dumb question, but do you know if this something that LLM frameworks like LangChain can (or others) can help with? Aren't they designed to help with more complex prompts/logic/outputs? Or will they run into the same token limits?

by afro88 4y ago

Tostino 4y ago

You iterate on your plan after it is generated step by step. You go and edit the prompt chain you started working on step 1 on, and modify it to start working on step 2 (including any ideas or fixes you have identified while implementing step 1. Repeat until complete. You can still absolutely hit the context limit, but you are far less likely to do so if you go back and start a new prompt chain for each different thought process you are going through with it.

afro88 4y ago
Great idea. But does it get hard to navigate back to something in older chat histories though? I find a new separate chat with the revised initial prompt to be easier.
- williamcotton 4y ago
  I’ve been using another call to an LLM to write or rewrite code that is separate from the main “conversation”. What I mean is that I’ve got a dialog going with an LLM and I’ve trained it to call a build() function with instructions that then returns the function, with the text of the function kept out of the dialog with the main thread.

quijoteuniv 4y ago

It's great to see that there's now a term for the type of prompting, “generated knowledge”. I've been experimenting with this technique since the beginning, and I've noticed a significant improvement in version 4. The process involves outlining the project, creating tasks, and feeding them back to chatGPT as you progress. This approach has helped me complete projects that would have otherwise taken me much longer to finish. It's also useful for creating practical tutorials. While there are plenty of tutorials available online, sometimes you need guidance on a specific set of technologies. By using generated knowledge prompts, you can get a good outline and tasks to help you understand how these technologies interact. One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness

afro88 4y ago
Absolutely, and same here. I've done multiple tools that would have taken 2-3 days each in 2-3 hours each. > One thing to keep in mind is to avoid derailing the conversation with questions that are not relevant to the core tasks. If you get stuck on something and need to debug, it's best to use a separate conversation to avoid derailing the project's progress and the allucinations & forgettingness Definitely. Great advice. Another tip: don't bother asking it to fix small things. Just mention you fixed it in the next reply and move on.
AzzieElbab 4y ago
Something must be wrong with me. I could never get anything useful from Martin Fowler's writings, and coincidentally I cannot get any functional code out of ChatGPT. Even the boilerplate it produces for me needs to be corrected. I still use chatGPT to produce examples of abstract things but was not able to get any working code that matches concrete problems or even compiles.
- afro88 4y ago
  Are you using the GPT4 model? There's a very significant improvement between 3.5 (the free one) and 4.

ryanjshaw 4y ago

Your experience matches mine closely. I've had ChatGPT-4 do great and then it just gets confused after a while. I can literally tell it "task X is done" and it'll apologise and show me a list of tasks where X is still not done - this is clearly not just a context window issue, as I have repeated variations of my statement over and over in the same session and the issue persists. I have ended up using it the same way you have - it's honestly the best anti-procrastination tool I've ever used because I can tell it my intentions, what I've thought of so far... and it'll spit out a list of bite-sized chunks that get me going. I find myself looking forward to telling the AI I've completed a task. Similarly, if I'm facing a tricky design decision, I find that just writing it out for ChatGPT is extremely helpful for clarifying my thought process. I actually used to do this conversational decision making process in a text editor long before ChatGPT, but when I know there's an AI on the other end my thinking becomes clearer and more goal-oriented. And unlike talking to myself or a human friend, it's happy to just say "well if these are your concerns, let's start HERE and then see what happens".

travisjungroth 4y ago
Good rule of thumb with ChatGPT: you can’t exit loops. Once you’ve gone A > B > A, your best move is to start a new chat. Even then it may reproduce and you should do some similar but different task. Remember that it’s a prediction engine, weighing heavily on the existing prompt. So you say B again, or B1 and it’s like, I know what to do! A! Cause last time was A->B so let’s do it again. In your case this would be “[]Task1”, “Task1 is done”, “[]Task1”, [here is where you start a new chat or fix it yourself if possible].
- sycren 4y ago
  Instead of starting a new chat, why not change the prompt higher up in the conversation with the relevant detail that you have gained through the responses?
  - travisjungroth 4y ago
    I think that would work and be better in some cases. The thing is you want to change things up, lose some context. I may not be able to go far enough back without losing progress to do that. But I could copy just the current state of the task (last response) to a new chat.

matchagaucho 4y ago

he would have easily hit the token limit That was my first question. Do all these tasks fit within a 4K or 8K buffer? Wouldn't be surprised, though, if it works in a 32K GPT4 token limit. Amazing things are possible.

blastro 4y ago

I completely agree with this take.

heyyyouu 4y ago

This may be a dumb question, but do you know if this something that LLM frameworks like LangChain can (or others) can help with? Aren't they designed to help with more complex prompts/logic/outputs? Or will they run into the same token limits?