6 ms·

Many think prompt engineering is just like being good at writing Google searches, where my job is to be good at knowing "how Google/GPT4 thinks"—good at writing

Many think prompt engineering is just like being good at writing Google searches, where my job is to be good at knowing "how Google/GPT4 thinks"—good at writing a single query. However, I think prompt engineering will evolve to be an actual technical role, akin to Data Engineering (the people who make the systems, pipelines, ETL jobs, etc for the data). Prompt engineers will build systems that facilitate prompt generation. Meaning that prompts will be dynamically generated or at least partially generated with modifications or additions to the raw user prompt. It's the difference to being able to write HTML vs being able to do all the backend work to dynamically generate the HTML for my Amazon homepage (including the performance benchmarks and other strategic requirements), for example.

maxbondabe 4y ago
I'm working on a prompt engineering product at a stealth mode startup. Would it be possible for me to pick your brain sometime? I'd like to understand more about your workflow, and how our product could fit into that. No pressure. My email is in my profile.
sgrove 4y ago
I’ve been doing the same thing with a number of projects, building chains of prompts from one api call to another e.g. for ConjureUI (self-creating, iterable UIs that come into existence, get used, then disappear) https://youtu.be/xgi1YX6HQBw https://youtu.be/xgi1YX6HQBw how it works to generate a full self-contained react component: 1. Take user task 2. Pass it to a prompt that requests a Product UI description of a component 3. Pass 1+2 to another that asks for which npm packages to use 4. Pass 1+2+3 to a templated prompt to write the code in a constrained manner 5. Run 4 in a sandbox to see if there are errors, if so pass it back to #4, looping It’s currently quite slow, but that’s an implementation detail I think.
- dorilama 4y ago
  > 3. Pass 1+2 to another that asks for which npm packages to use I see a fresh new generation of supply chain attack, or more prompt engineering to hopefully filter out malicious packages
  - lupire 4y ago
    Once the malicious package is added to the universe of acceptable packages, it doesn't matter much. Prompt engineering is not a solution you that.
  - sgrove 4y ago
    Yes, that wasn't a priority here, but I also don't think it's much of a concern with e.g. GPT-4's `system` vs `assistant` vs `user` roles. Would be another thing to work on, but nothing worth doom and gloom. Although, 'script(/injection) kiddie' will be an interesting phenomenon in the future...
    - dorilama 4y ago
      You can probably feed a curated list of allowed packages for this step
simonw 4y ago
Crucially important that engineers who are building systems like that that work by concatenating prompts together have a very solid understanding of prompt injection attacks.
seydor 4y ago
I don't think it will evolve at all, because models will just become better at understanding what people mean. There's no point in trying to be a better prompter as a competitive advantage. Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great OTOH i think there is space for developing GUIs for prompts. Makes them more engaging
- addisonl 4y ago
  > Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great You’re completely ignoring the system prompts that OpenAI/Bing have already set up so your “3 word query” works as you intend. These system prompts are what prompt engineering is all about.
  - seydor 4y ago
    these prompts by definition do not need engineering because they work for all cases all the time. and they can only get better over time
    - lupire 4y ago
      Huh? How would they get better without engineering? Why would it be a good idea to use them in all cases all of the time?
      - seydor 4y ago
        There are exactly 2 of those prompts in the whole world
        bee_rider 4y ago
        I bet they took a couple tries to come up with them.
    - simonw 4y ago
sorokod 4y ago
I have an idea what skills are required to dynamicly generate HTML and how to measure quality. Can you share something similar for "prompt engineering" ?
- devmunchies 4y ago
  See my other comment I just posted to someone else for examples.
  - sorokod 4y ago
    So you say "feed", "inject", etc.. but these just alternative verbs for "describing". What skills are needed? Where is the ingeneering part?
    - devmunchies 4y ago
      Well, it depends on what you need from your prompt. What engineering is required for Amazon to deliver me an HTML page? Are the backend engineers just “injecting” HTML? Of course not. It’s not the same as generating the HTML for a personal blog. As far as what skills are need, it depends on how any given team wants to modify the prompt. It’s up to the ingenuity of the engineers. Maybe they need to hit 5 other models before generating the prompt. IDK. The prompt may change for any given scenario, which is why you need engineers to build the system.
throwaway675309 4y ago
It's macro substitution on crafted templatized prompts at best. Calling it engineering dilutes the term either further than it already is. (Looking at you Salesforce "Engineer").
- bee_rider 4y ago
  The term is already maximally diluted, but it is nice that coders are getting a turn at trying to gatekeep it.
  - KyeRussell 4y ago
    Yeah. Software people getting uppity about use of the word “engineer” is…a hilarious thought. You made your bed on this one.
  - agentultra 4y ago
    Engineers Canada has recently won a case against a software developer that used the term, “engineer” in their profile.
dror 4y ago
Here are a couple of good resources around prompt engineering https://www.promptingguide.ai/ https://www.promptingguide.ai/ https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/ https://lilianweng.github.io/posts/2023-03-15-prompt-enginee... These clearly demonstrate that there's engineering skills beyond "I want you to act as a Linux terminal." Also note that there's a difference between the prompt that you use on ChatGPT and the one that you use in the GPT API or other LLMs. In the first, you're already dealing with the prompt that openai supplied to the LLM in the first place.
- IncRnd 4y ago
  I stopped reading your second link after seeing that the person teaching prompt engineering demonstrated a lack of understanding regarding proper sentence structure.
  - harlanlewis 4y ago
    According to their linkedin, the author’s job is “Head of Applied AI Research at OpenAI”.
  - lisasays 4y ago
    Right, they use slightly imperfect syntax - because they're a non-native speaker. Is that what you discount them for?
    - IncRnd 4y ago
      No. Of course that's not the reason. Please don't take your biases and make those sorts of statements about what you believe is in somebody else's thoughts. The reason was what I stated, which has nothing to do with the provenance or capabilities of that person.
arroz 4y ago
Lol
- devmunchies 4y ago
  What’s funny? Try to write a chat bot built on an LLM that adapts to your personality over time and do it without EVER dynamically changing the prompt.
divbzero 4y ago
Feature engineering [1] is probably the closest existing analogue. [1]: https://en.wikipedia.org/wiki/Feature_engineering https://en.wikipedia.org/wiki/Feature_engineering
anonzzzies 4y ago
But why call it prompt engineer and not just data or, you know, software engineering? Prompt engineer means the prompting, and that’s already quite dead as it is: you can just ask gpt4 to fix the user input for a specific goal. We do this in our pipeline and it’s obviously (much) faster and cheaper than ‘prompt engineers’, but it is usually also simply better for what we need it for. The stuff you say it will evolve in already exists and has names everyone uses.

maxbondabe 4y ago

I'm working on a prompt engineering product at a stealth mode startup. Would it be possible for me to pick your brain sometime? I'd like to understand more about your workflow, and how our product could fit into that. No pressure. My email is in my profile.

sgrove 4y ago

I’ve been doing the same thing with a number of projects, building chains of prompts from one api call to another e.g. for ConjureUI (self-creating, iterable UIs that come into existence, get used, then disappear) https://youtu.be/xgi1YX6HQBw https://youtu.be/xgi1YX6HQBw how it works to generate a full self-contained react component: 1. Take user task 2. Pass it to a prompt that requests a Product UI description of a component 3. Pass 1+2 to another that asks for which npm packages to use 4. Pass 1+2+3 to a templated prompt to write the code in a constrained manner 5. Run 4 in a sandbox to see if there are errors, if so pass it back to #4, looping It’s currently quite slow, but that’s an implementation detail I think.

dorilama 4y ago
> 3. Pass 1+2 to another that asks for which npm packages to use I see a fresh new generation of supply chain attack, or more prompt engineering to hopefully filter out malicious packages
- lupire 4y ago
  Once the malicious package is added to the universe of acceptable packages, it doesn't matter much. Prompt engineering is not a solution you that.
- sgrove 4y ago
  Yes, that wasn't a priority here, but I also don't think it's much of a concern with e.g. GPT-4's `system` vs `assistant` vs `user` roles. Would be another thing to work on, but nothing worth doom and gloom. Although, 'script(/injection) kiddie' will be an interesting phenomenon in the future...
  - dorilama 4y ago
    You can probably feed a curated list of allowed packages for this step

simonw 4y ago

Crucially important that engineers who are building systems like that that work by concatenating prompts together have a very solid understanding of prompt injection attacks.

seydor 4y ago

I don't think it will evolve at all, because models will just become better at understanding what people mean. There's no point in trying to be a better prompter as a competitive advantage. Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great OTOH i think there is space for developing GUIs for prompts. Makes them more engaging

addisonl 4y ago
> Already, chatGPT and bing can both give great on-topic answers to 3 word queries. and the fact that you can infinitely refine it is great You’re completely ignoring the system prompts that OpenAI/Bing have already set up so your “3 word query” works as you intend. These system prompts are what prompt engineering is all about.
- seydor 4y ago
  these prompts by definition do not need engineering because they work for all cases all the time. and they can only get better over time
  - lupire 4y ago
    Huh? How would they get better without engineering? Why would it be a good idea to use them in all cases all of the time?
    - seydor 4y ago
      There are exactly 2 of those prompts in the whole world
      - bee_rider 4y ago
        I bet they took a couple tries to come up with them.
  - simonw 4y ago

sorokod 4y ago

I have an idea what skills are required to dynamicly generate HTML and how to measure quality. Can you share something similar for "prompt engineering" ?

devmunchies 4y ago
See my other comment I just posted to someone else for examples.
- sorokod 4y ago
  So you say "feed", "inject", etc.. but these just alternative verbs for "describing". What skills are needed? Where is the ingeneering part?
  - devmunchies 4y ago
    Well, it depends on what you need from your prompt. What engineering is required for Amazon to deliver me an HTML page? Are the backend engineers just “injecting” HTML? Of course not. It’s not the same as generating the HTML for a personal blog. As far as what skills are need, it depends on how any given team wants to modify the prompt. It’s up to the ingenuity of the engineers. Maybe they need to hit 5 other models before generating the prompt. IDK. The prompt may change for any given scenario, which is why you need engineers to build the system.

throwaway675309 4y ago

It's macro substitution on crafted templatized prompts at best. Calling it engineering dilutes the term either further than it already is. (Looking at you Salesforce "Engineer").

bee_rider 4y ago
The term is already maximally diluted, but it is nice that coders are getting a turn at trying to gatekeep it.
- KyeRussell 4y ago
  Yeah. Software people getting uppity about use of the word “engineer” is…a hilarious thought. You made your bed on this one.
- agentultra 4y ago
  Engineers Canada has recently won a case against a software developer that used the term, “engineer” in their profile.

dror 4y ago

Here are a couple of good resources around prompt engineering https://www.promptingguide.ai/ https://www.promptingguide.ai/ https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/ https://lilianweng.github.io/posts/2023-03-15-prompt-enginee... These clearly demonstrate that there's engineering skills beyond "I want you to act as a Linux terminal." Also note that there's a difference between the prompt that you use on ChatGPT and the one that you use in the GPT API or other LLMs. In the first, you're already dealing with the prompt that openai supplied to the LLM in the first place.

IncRnd 4y ago
I stopped reading your second link after seeing that the person teaching prompt engineering demonstrated a lack of understanding regarding proper sentence structure.
- harlanlewis 4y ago
  According to their linkedin, the author’s job is “Head of Applied AI Research at OpenAI”.
- lisasays 4y ago
  Right, they use slightly imperfect syntax - because they're a non-native speaker. Is that what you discount them for?
  - IncRnd 4y ago
    No. Of course that's not the reason. Please don't take your biases and make those sorts of statements about what you believe is in somebody else's thoughts. The reason was what I stated, which has nothing to do with the provenance or capabilities of that person.

arroz 4y ago

Lol

devmunchies 4y ago
What’s funny? Try to write a chat bot built on an LLM that adapts to your personality over time and do it without EVER dynamically changing the prompt.

divbzero 4y ago

Feature engineering [1] is probably the closest existing analogue. [1]: https://en.wikipedia.org/wiki/Feature_engineering https://en.wikipedia.org/wiki/Feature_engineering

anonzzzies 4y ago

But why call it prompt engineer and not just data or, you know, software engineering? Prompt engineer means the prompting, and that’s already quite dead as it is: you can just ask gpt4 to fix the user input for a specific goal. We do this in our pipeline and it’s obviously (much) faster and cheaper than ‘prompt engineers’, but it is usually also simply better for what we need it for. The stuff you say it will evolve in already exists and has names everyone uses.