4 ms·

My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you wan

by bradfa 1y ago

My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you want the agent to proceed, that then you can work with the agent to craft a plan, refine the plan, and finally execute the plan. In this mode of operation, the current state of the art is quite impressive. You can't just give it a single sentence and expect it to do something complex correctly. It takes real effort and human time, just like if you were trying to get a smart and capable intern who has no real world experience to do something technical correctly. Just the AI agents work significantly faster than a human intern.

qazxcvbnmlp 1y ago
> My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you want the agent to proceed, I have a pet theory. 1. This skill requires a strong theory of mind[1]. 2. Theory of mind is more difficult in those with autism 3. The same autism that makes people really good at coding, and gives them the time to post on online forms like hn, makes it hard to understand how to work with LLMs and how others work with llms. To provide good context to the llm you need to have a good understanding of (1) what it will and will not know, (2) what you know and take for granted(ie a theory of your own mind) (3) what your expectations are. None of this you need to do when you are coding on your own, but are critical on getting a good response from the LLM. See also the black and white thinking that is common in the responses on articles like this.[2] [1]https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind [2]https://www.simplypsychology.org/black-and-white-thinking-in-autism.html https://www.simplypsychology.org/black-and-white-thinking-in...
- saulpw 1y ago
  An LLM has no mind! What is your strong theory of mind for an LLM? That it knows the whole internet and can regurgitate it like a mindless zombie?
  - kelseyfrog 1y ago
    Source? It sounds like an unprovable metaphysical statement than something that is supported by scientific evidence.
    - fao_ 1y ago
      The burden of proof is on people stating that an AI has a theory of mind, not on the reverse. Until recently it was highly debated on if dogs have theory of mind, and it took decades of evidence to come to the conclusion that yes, they do.
      - judahmeek 1y ago
        GGP didn't say that AI has a theory of mind. GGP said that using AI productively requires a theory of mind, a.k.a. being able to build a mental model of the LLM's context.
      - kelseyfrog 1y ago
        The burden of proof is on the person making the claim. It doesn't matter whether the claim is positive or negative. The default position is "We don't know if AI has a ToM."
        sirtaj 1y ago
        Am I incorrect in thinking this is as much true of the linux kernel or emacs as it is of an LLM?
      - rmwaite 1y ago
        If you read carefully you will see that they never said AI has a theory of mind.
  - xg15 1y ago
    Whether or not it has a mind is irrelevant to the problem. I think the point is, if you pretend it had a mind and write your prompt accordingly, you will get the best results.
- wild_egg 1y ago
  This actually makes a disturbing amount of sense and I think I'm going to need to chew on it for a while. Thanks for sharing!
- bigchillin 1y ago
  That simply psych article is a psyop
  - imtringued 1y ago
    That HN username is also bad news. Meanwhile yours is pretty cool. I really enjoyed the social credit memes with John Cena. How exactly do you come up with a pet theory out of nowhere, randomly diagnose people on the internet with autism based on how they use LLMs and then start linking to a most likely AI generated blog post (there was simply too much repetition) that ascribes a lot of negative attributes to them with a username that is meant to be unrecognisable. The post is basically a Kafka trap or engagement bait.
satvikpendem 1y ago
> that then you can work with the agent to craft a plan, refine the plan, and finally execute the plan And Cursor just introduced a separate plan mode themselves, so it gets even better.
theshrike79 1y ago
People don't understand that LLMs aren't humans. There's a lot of implicit context when humans are communicating. LLMs don't do that. They do have biases, like if you tell them to do something with data, they'll pretty likely grab Python as the tool. And different models have different biases and styles, you can try to guide them to your specific style with prompts, but it doesn't always work - depending on how esoteric your personal style is.
- bradfa 1y ago
  Imagining the tool is like a college intern helps me. It has no idea how the real world works. It blindly follows things it previously found online. It’s great at very common boilerplate coding tasks. But it’s super naive and will need hand holding or you to provide a huge amount of context for it so it can operate on its own. I’m still very much learning how to give it good instructions to accomplish tasks. Different tasks require different types and methods of instruction. It’s extremely interesting to me. I’m far from an expert.
  - theshrike79 1y ago
    I imagine LLMs as an endless stream of consultants, each can work only one day (context). Every day you need to bring them up to speed (prompt, accessible documentation) and give them the task of the day. If it looks like they can't finish the task (context runs out), you need to tell them to write down where they left (store context to a memory, markdown file is fine) and kick them out the door. Then GOTO 10, get the next one in.
bigstrat2003 1y ago
> My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you want the agent to proceed, that then you can work with the agent to craft a plan, refine the plan, and finally execute the plan. People say this a lot, and I'm not even saying you're wrong. But that isn't useful to me. In the time it takes me to do all that, I can just solve the problem myself. If I have to hold its hand through finding a solution, then it is a time suck, not a time saver.
- bradfa 1y ago
  There are definitely times when it’s not faster to use the tool to do the full job. But sometimes just using the tool to plan the job helps to clarify the task so a human can do it better/faster. But then there’s also tasks where using the tool is a HUGE speed up.
- andoando 1y ago
  You can literally write "Add a feature on the UI where we get a live update of new posts using a websocket connection in the backend server at /app/backend." And it will it will integrate websockets in your UI, backend and create the models, service logic, etc in under 20 seconds. Can you really do that?
  - mcv 1y ago
    That's boilerplate stuff. That's what it's best at. But the moment I want something slightly off the beaten path (and I always do), it struggles and makes mistakes.
- cindyllm 1y ago
  [dead]
- drcxd 1y ago
  Agreed. I also have to check if it has implemented the idea correctly. If my workflow is; 1. Write documentation so that the problem and even the solution to the problem is well explained. 2. Instruct coding agents to work as the document described. 3. Check its if its implementation is correct, and improve its implementation if necessary. I feel the experience is not as good as me implementing the solution myself, and it may even take more time.

by bradfa 1y ago

qazxcvbnmlp 1y ago

> My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you want the agent to proceed, I have a pet theory. 1. This skill requires a strong theory of mind[1]. 2. Theory of mind is more difficult in those with autism 3. The same autism that makes people really good at coding, and gives them the time to post on online forms like hn, makes it hard to understand how to work with LLMs and how others work with llms. To provide good context to the llm you need to have a good understanding of (1) what it will and will not know, (2) what you know and take for granted(ie a theory of your own mind) (3) what your expectations are. None of this you need to do when you are coding on your own, but are critical on getting a good response from the LLM. See also the black and white thinking that is common in the responses on articles like this.[2] [1]https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind [2]https://www.simplypsychology.org/black-and-white-thinking-in-autism.html https://www.simplypsychology.org/black-and-white-thinking-in...

saulpw 1y ago
An LLM has no mind! What is your strong theory of mind for an LLM? That it knows the whole internet and can regurgitate it like a mindless zombie?
- kelseyfrog 1y ago
  Source? It sounds like an unprovable metaphysical statement than something that is supported by scientific evidence.
  - fao_ 1y ago
    The burden of proof is on people stating that an AI has a theory of mind, not on the reverse. Until recently it was highly debated on if dogs have theory of mind, and it took decades of evidence to come to the conclusion that yes, they do.
    - judahmeek 1y ago
      GGP didn't say that AI has a theory of mind. GGP said that using AI productively requires a theory of mind, a.k.a. being able to build a mental model of the LLM's context.
    - kelseyfrog 1y ago
      The burden of proof is on the person making the claim. It doesn't matter whether the claim is positive or negative. The default position is "We don't know if AI has a ToM."
      - sirtaj 1y ago
        Am I incorrect in thinking this is as much true of the linux kernel or emacs as it is of an LLM?
    - rmwaite 1y ago
      If you read carefully you will see that they never said AI has a theory of mind.
- xg15 1y ago
  Whether or not it has a mind is irrelevant to the problem. I think the point is, if you pretend it had a mind and write your prompt accordingly, you will get the best results.
wild_egg 1y ago
This actually makes a disturbing amount of sense and I think I'm going to need to chew on it for a while. Thanks for sharing!
bigchillin 1y ago
That simply psych article is a psyop
- imtringued 1y ago
  That HN username is also bad news. Meanwhile yours is pretty cool. I really enjoyed the social credit memes with John Cena. How exactly do you come up with a pet theory out of nowhere, randomly diagnose people on the internet with autism based on how they use LLMs and then start linking to a most likely AI generated blog post (there was simply too much repetition) that ascribes a lot of negative attributes to them with a username that is meant to be unrecognisable. The post is basically a Kafka trap or engagement bait.

satvikpendem 1y ago

> that then you can work with the agent to craft a plan, refine the plan, and finally execute the plan And Cursor just introduced a separate plan mode themselves, so it gets even better.

theshrike79 1y ago

People don't understand that LLMs aren't humans. There's a lot of implicit context when humans are communicating. LLMs don't do that. They do have biases, like if you tell them to do something with data, they'll pretty likely grab Python as the tool. And different models have different biases and styles, you can try to guide them to your specific style with prompts, but it doesn't always work - depending on how esoteric your personal style is.

bradfa 1y ago
Imagining the tool is like a college intern helps me. It has no idea how the real world works. It blindly follows things it previously found online. It’s great at very common boilerplate coding tasks. But it’s super naive and will need hand holding or you to provide a huge amount of context for it so it can operate on its own. I’m still very much learning how to give it good instructions to accomplish tasks. Different tasks require different types and methods of instruction. It’s extremely interesting to me. I’m far from an expert.
- theshrike79 1y ago
  I imagine LLMs as an endless stream of consultants, each can work only one day (context). Every day you need to bring them up to speed (prompt, accessible documentation) and give them the task of the day. If it looks like they can't finish the task (context runs out), you need to tell them to write down where they left (store context to a memory, markdown file is fine) and kick them out the door. Then GOTO 10, get the next one in.

bigstrat2003 1y ago

> My experience has been that if you take the time to explain what the current state is, what your desired state should be, and to give information on how you want the agent to proceed, that then you can work with the agent to craft a plan, refine the plan, and finally execute the plan. People say this a lot, and I'm not even saying you're wrong. But that isn't useful to me. In the time it takes me to do all that, I can just solve the problem myself. If I have to hold its hand through finding a solution, then it is a time suck, not a time saver.

bradfa 1y ago
There are definitely times when it’s not faster to use the tool to do the full job. But sometimes just using the tool to plan the job helps to clarify the task so a human can do it better/faster. But then there’s also tasks where using the tool is a HUGE speed up.
andoando 1y ago
You can literally write "Add a feature on the UI where we get a live update of new posts using a websocket connection in the backend server at /app/backend." And it will it will integrate websockets in your UI, backend and create the models, service logic, etc in under 20 seconds. Can you really do that?
- mcv 1y ago
  That's boilerplate stuff. That's what it's best at. But the moment I want something slightly off the beaten path (and I always do), it struggles and makes mistakes.
cindyllm 1y ago
[dead]
drcxd 1y ago
Agreed. I also have to check if it has implemented the idea correctly. If my workflow is; 1. Write documentation so that the problem and even the solution to the problem is well explained. 2. Instruct coding agents to work as the document described. 3. Check its if its implementation is correct, and improve its implementation if necessary. I feel the experience is not as good as me implementing the solution myself, and it may even take more time.