7 ms·

>Sam A said he thinks prompt engineering won't even be a thing in 5 years. I don’t understand what this could mean. I have to do “prompt engineering” when I ta

>Sam A said he thinks prompt engineering won't even be a thing in 5 years. I don’t understand what this could mean. I have to do “prompt engineering” when I talk to other people all the time - when I need to ask them to do tasks, when I need to clarify requirements, etc. As long as we’re communicating through text and not mind reading, some level of “engineering” will be required. AI is intelligent but it’s not magic.

ekanes 3y ago
Prompt engineering now (as I understand it) is you're refining and refining the prompt itself. Perhaps what we might see is you give it a prompt then just keep adjusting the response dynamically, and then it can remember the entire concept as a saved prompt.
- newswasboring 3y ago
  I already use ChatGPT like this. After every good outcome I say something like "modify my original prompt to include all the things we learned in this chat session". It has mixed results, context window being the biggest factor I guess. Longer conversations (where I am trying to write stories specially) do not produce good prompts as it becomes super specific even if I tell it to generalize specific things.
  - daveguy 3y ago
    Edit: I may have misunderstood. You instruct it to modify the original prompt of that session to include what you learned? Do you ask it to repeat the new prompt? This is interesting, but it wouldn't help for any future sessions. --- It has zero effect. Your prompt consists of the OpenAI prompt + your conversation so far. There is no cross-prompt learning, knowledge or intelligence [except for, in the future, what OpenAI decides to include in model training of new releases]. The main reason for this is there are a limited number of tokens that can be used as the prompt. If you "include everything we learned in this chat session" it would have to include the entire chat session as part of all future prompts and you would quickly run out of tokens for the prompt. Training happens on a corpus, but not during regular use. Perceived "learning" is just context provided by the session-limited prompt.
    - travisjungroth 3y ago
      If that person is doing what I think, it’s that they get back a prompt to use next time. Write a followup email to A about B with X,Y,Z characteristics. [iterative chat] Give me back a prompt for next time. Ok: “Write a followup email to A about B with W,X,Y characteristics.”
    - newswasboring 3y ago
      > you ask it to repeat the new prompt? I thought that was understood. I basically ask it to improve my initial prompt, like asking it how could I have asked this better. Sometimes it gets good results, like when it taught me how to pose formatting to it. I did not know how to make it output in a certain user defined format, it never occured to me that I can just give it an example, the modified prompt taught me that trick. Now it's super common everywhere and I think is the basis of a lot of langchain.
- shanebellone 3y ago
  I think the intention is to eliminate the prompt component entirely. This would shift AI from "generative autofill" to a "repository of truth". The former is useless for most business applications. The latter is the holy grail of product.
ModernMech 3y ago
I see a general trend for developers to want to put syntax and semantics back into the prompting. This whole idea that we can just get rid of formal languages entirely and replace them with natural languages isn't going to pan out; we have formal languages for a reason -- you don't have to guess the magic spell that will cause the output you want, you can get the output you want because you know how the system works. It's a much less frustrating and consistent way of dealing with computers unless you fancy yourself a bureaucrat. If this actually happens, I would imagine that talking to an LLM would be a combination of formal syntax and natural language, so the task will look more like software engineering than black magic.
- pmoriarty 3y ago
  "This whole idea that we can just get rid of formal languages entirely and replace them with natural languages isn't going to pan out; we have formal languages for a reason -- you don't have to guess the magic spell that will cause the output you want, you can get the output you want because you know how the system works" This reminds me of Inform 7 (where you can use "natural language" to program text adventure games) and "visual" programming languages like Pure Data. They are fine for simple things, but when you want something complex or want to debug something they can become a nightmare. LLMs are currently better in many ways than Inform 7, and they'll likely get better still, but there will likely still be a role for formal languages. Fortunately, LLMs can take formal language as input as well, and with the addition of plugins, they'll be able to execute programs written in formal languages.
  - jstarfish 3y ago
    I thought all the time I spent on interactive fiction was wasted, but it turns out Inform 7-ish NL syntax lends itself well to building minigames in GPT. > Please function as a TTRPG engine. > All characters start with 100 tokens representing LIFE, which can range from 0 to 100. > Health scales with LIFE. At 0 LIFE, you are dead. At 100 LIFE, you are in peak physical condition. Etc.
  - professoretc 3y ago
    > This reminds me of Inform 7 (where you can use "natural language" to program text adventure games) The problem (at least, the problem I had) with Inform 7 is that it's not really natural language; it still has a strict formal grammar defining its syntax, it's just that the grammar has lots of "synonyms"; multiple syntaxes for the same thing. But it's still following strict rules, and those rules can't cover everything. This leads to situations where you write some "plain English" and it seemingly understands it, but then you write tiny variation and it throws up a syntax error.
Mezzie 3y ago
Yes. The problem isn't the AI not understanding the human's intent. The problem is the human not understanding what the human wants/thinking it knows what it wants but being wrong. Watching whole communities crowd-source/stumble their way into basic reference question principles is kind of fun, though.
- cratermoon 3y ago
  > The problem is the human not understanding what the human wants/thinking it knows what it wants but being wrong. This is really the essence of why most software development is hard. Aside from a few problems whose solutions can be mathematically or logically defined, programmers are working on software to do things humans want to do to affect the external world in some may. They're what Meir M. Lehman calls E-programs: programs to model human and social activities. The program becomes part of the world it models, e.g. air traffic control. Acknowledging the inability of humans to understand what they want shows the folly of trying to get all the requirements "up front" before starting the coding.
melvinmelih 3y ago
I think it's easiest to compare writing prompts to writing SQL. You don't write SQL on a regular basis to interact with products (and with ORMs not even directly in your code anymore), it's been abstracted in many ways.
robomartin 3y ago
> I have to do “prompt engineering” when I talk to other people all the time Pulling from my hardware background... When you design with FPGA's and write code using HDL (Hardware Description Language; Verilog, VHDL) you learn to write code the compile will use to infer the hardware structures you are after. Despite what it may seem, you are not programming, you are describing hardware and, in some cases, use use idioms or structures that will result in the desired outcome. In some ways prompt engineering can be thought of as learning how to cause the AI to deliver the desired result using the most efficient prompt text.

ekanes 3y ago

Prompt engineering now (as I understand it) is you're refining and refining the prompt itself. Perhaps what we might see is you give it a prompt then just keep adjusting the response dynamically, and then it can remember the entire concept as a saved prompt.

newswasboring 3y ago
I already use ChatGPT like this. After every good outcome I say something like "modify my original prompt to include all the things we learned in this chat session". It has mixed results, context window being the biggest factor I guess. Longer conversations (where I am trying to write stories specially) do not produce good prompts as it becomes super specific even if I tell it to generalize specific things.
- daveguy 3y ago
  Edit: I may have misunderstood. You instruct it to modify the original prompt of that session to include what you learned? Do you ask it to repeat the new prompt? This is interesting, but it wouldn't help for any future sessions. --- It has zero effect. Your prompt consists of the OpenAI prompt + your conversation so far. There is no cross-prompt learning, knowledge or intelligence [except for, in the future, what OpenAI decides to include in model training of new releases]. The main reason for this is there are a limited number of tokens that can be used as the prompt. If you "include everything we learned in this chat session" it would have to include the entire chat session as part of all future prompts and you would quickly run out of tokens for the prompt. Training happens on a corpus, but not during regular use. Perceived "learning" is just context provided by the session-limited prompt.
  - travisjungroth 3y ago
    If that person is doing what I think, it’s that they get back a prompt to use next time. Write a followup email to A about B with X,Y,Z characteristics. [iterative chat] Give me back a prompt for next time. Ok: “Write a followup email to A about B with W,X,Y characteristics.”
  - newswasboring 3y ago
    > you ask it to repeat the new prompt? I thought that was understood. I basically ask it to improve my initial prompt, like asking it how could I have asked this better. Sometimes it gets good results, like when it taught me how to pose formatting to it. I did not know how to make it output in a certain user defined format, it never occured to me that I can just give it an example, the modified prompt taught me that trick. Now it's super common everywhere and I think is the basis of a lot of langchain.
shanebellone 3y ago
I think the intention is to eliminate the prompt component entirely. This would shift AI from "generative autofill" to a "repository of truth". The former is useless for most business applications. The latter is the holy grail of product.

ModernMech 3y ago

I see a general trend for developers to want to put syntax and semantics back into the prompting. This whole idea that we can just get rid of formal languages entirely and replace them with natural languages isn't going to pan out; we have formal languages for a reason -- you don't have to guess the magic spell that will cause the output you want, you can get the output you want because you know how the system works. It's a much less frustrating and consistent way of dealing with computers unless you fancy yourself a bureaucrat. If this actually happens, I would imagine that talking to an LLM would be a combination of formal syntax and natural language, so the task will look more like software engineering than black magic.

pmoriarty 3y ago
"This whole idea that we can just get rid of formal languages entirely and replace them with natural languages isn't going to pan out; we have formal languages for a reason -- you don't have to guess the magic spell that will cause the output you want, you can get the output you want because you know how the system works" This reminds me of Inform 7 (where you can use "natural language" to program text adventure games) and "visual" programming languages like Pure Data. They are fine for simple things, but when you want something complex or want to debug something they can become a nightmare. LLMs are currently better in many ways than Inform 7, and they'll likely get better still, but there will likely still be a role for formal languages. Fortunately, LLMs can take formal language as input as well, and with the addition of plugins, they'll be able to execute programs written in formal languages.
- jstarfish 3y ago
  I thought all the time I spent on interactive fiction was wasted, but it turns out Inform 7-ish NL syntax lends itself well to building minigames in GPT. > Please function as a TTRPG engine. > All characters start with 100 tokens representing LIFE, which can range from 0 to 100. > Health scales with LIFE. At 0 LIFE, you are dead. At 100 LIFE, you are in peak physical condition. Etc.
- professoretc 3y ago
  > This reminds me of Inform 7 (where you can use "natural language" to program text adventure games) The problem (at least, the problem I had) with Inform 7 is that it's not really natural language; it still has a strict formal grammar defining its syntax, it's just that the grammar has lots of "synonyms"; multiple syntaxes for the same thing. But it's still following strict rules, and those rules can't cover everything. This leads to situations where you write some "plain English" and it seemingly understands it, but then you write tiny variation and it throws up a syntax error.

Mezzie 3y ago

Yes. The problem isn't the AI not understanding the human's intent. The problem is the human not understanding what the human wants/thinking it knows what it wants but being wrong. Watching whole communities crowd-source/stumble their way into basic reference question principles is kind of fun, though.

cratermoon 3y ago
> The problem is the human not understanding what the human wants/thinking it knows what it wants but being wrong. This is really the essence of why most software development is hard. Aside from a few problems whose solutions can be mathematically or logically defined, programmers are working on software to do things humans want to do to affect the external world in some may. They're what Meir M. Lehman calls E-programs: programs to model human and social activities. The program becomes part of the world it models, e.g. air traffic control. Acknowledging the inability of humans to understand what they want shows the folly of trying to get all the requirements "up front" before starting the coding.

melvinmelih 3y ago

I think it's easiest to compare writing prompts to writing SQL. You don't write SQL on a regular basis to interact with products (and with ORMs not even directly in your code anymore), it's been abstracted in many ways.

robomartin 3y ago

> I have to do “prompt engineering” when I talk to other people all the time Pulling from my hardware background... When you design with FPGA's and write code using HDL (Hardware Description Language; Verilog, VHDL) you learn to write code the compile will use to infer the hardware structures you are after. Despite what it may seem, you are not programming, you are describing hardware and, in some cases, use use idioms or structures that will result in the desired outcome. In some ways prompt engineering can be thought of as learning how to cause the AI to deliver the desired result using the most efficient prompt text.