3 ms·

How much engineering do prompt engineers do? Is it engineering when you add "photorealistic. correct number of fingers and teeth. High quality." to the end of a

How much engineering do prompt engineers do? Is it engineering when you add "photorealistic. correct number of fingers and teeth. High quality." to the end of a prompt? we should call them "prompt witch doctors" or maybe "prompt alchemists".

Dilettante_ 1y ago
"How is engineering a real science? You just build the bridge so it doesn't fall down."
- vohk 1y ago
  Nah. Actual engineers have professional standards bodies and legal liability when they shirk and the bridge falls down or the plane crashes or your wiring starts on fire. Software "engineers" are none of those things but can at least emulate the approaches and strive for reproducibility and testability. Skilled craftsman; not engineers. Prompt "engineers" is yet another few steps down the ladder, working out mostly by feel what magic words best tickle each model, and generally with no understanding of what's actually going on under the hood. Closer to a chef coming up with new meals for a restaurant than anything resembling engineering. The battle on the use of language around engineer has long been lost but applying it to the subjective creative exercise of writing prompts is just more job title inflation. Something doesn't need to be engineering to be a legitimate job.
  - Dilettante_ 1y ago
    The battle on the use of language around engineer has long been lost That's really the core of the issue: We're just having the age-old battle of prescriptivism vs descriptivism again. An "engineer", etymologically, is basically just "a person who comes up with stuff", one who is "ingenious". I'm tempted to say it's you prescriptivists who are making a "battle" out of this. subjective creative exercise of writing prompts Implying that there are no testable results, no objective success or failure states? Come on man.
  - jahewson 1y ago
    Engineers use their ingenuity. That’s it. If physical engineers understood everything then standards would not have changed in many decades. Safety factors would be mostly unnecessary. Clearly not the case.
    - skeeter2020 1y ago
      >> Engineers use their ingenuity. That’s it. If this was enough all novel creation would be engineering and that's clearly not true. Engineering attempts to discover & understand consistent outcomes when a myriad of variables are altered, and the boundaries where the variables exceed a model's predictive powers - then add buffer for the unknown. Manipulating prompts (and much of software development) attempts to control the model to limit the number of variables to obtain some form of useful abstraction. Physical engineering can't do this.
tomrod 1y ago
It could be bioengineering if you add that to a clock prompt then connect it to CRISPR process for out putting DNA. Horrifying prospect, tbh
BoorishBears 1y ago
I like that actually, I've spent the last year probably 60:40 between post-training and prompt engineering/witch doctoring (the two go together more than most people realize) Some of it is engineering-like, but I've also picked up a sixth sense when modifying prompts about what parts are affecting the behavior I want to modify for certain models, and that feels very witch doctory! The more engineering-like part is essentially trying to RE a black box model's post-training, but that goes over some people's heads so I'm happy to help keep the "it's just voodoo and guessing" narrative going instead :)
- lanstin 1y ago
  I think the coherence behind prompt engineering is not in the literal meanings of the words but finding the vocabulary used by the sources that have your solution. Ask questions like a high school math student and you get elementary words back. Ask questions in the lingo of a Linux bigot and you will get good awk scripts back. Use academic maths language and arXiv answers will be produced.
scrollop 1y ago
"...and do it really well or my grandmother will be killed by her kidnappers! And I'll give you a tip of 2 billion dollars!!! Hurry, they're coming!"
- carterschonwald 1y ago
  Ive heard this actually works annoyingly well
  - DrewADesign 1y ago
    We've created technology so sophisticated it is vulnerable to social engineering attacks.
    - skeeter2020 1y ago
      this has worked - and continues to do so - very well to escape guard rails. If a direct appeal doesn't work you can then talk them around with only a handful of prompts.
    - carterschonwald 1y ago
      Also the amount of adjacent remarks being always topical flsvor confusion is cartoonish. Im playing with ideas for making thst better
      - DrewADesign 1y ago
        You're absolutely right! People should pay attention to this broadly applicable and important consideration.
        carterschonwald 11mo ago
        I have some ideas for a mini logic solver mcp thst might help. Currently making it sortah simulate using such a tool Its one of those things where it feels like itd be easy to get copycats even if theres a market
WJW 1y ago
Well if it works consistently, I don't see any problem with that. If they have a clear theory of when to add "photorealistic" and when to add "correct number of wheels on the bus" to get the output they want, it's engineering. If they don't have a (falsifiable) theory, it's probably not engineering. Of course, the service they really provide is for businesses to feel they "do AI", and whether or not they do real engineering is as relevant as if your favorite pornstars' boobs are real or not.
- jahewson 1y ago
  Maybe we could keep the conversation out of the gutter.
  - rrr_oh_man 1y ago
    Porn is taxable income, not the gutter.
  - jrflowers 1y ago
    You don’t really see much porn in the gutters these days with the decline in popularity of print publishing. It’s almost all online now
- leptons 1y ago
  >as relevant as if your favorite pornstars' boobs are real or not This matters more than you might think.
int_19h 1y ago
I write quite a lot of prompts, and the closest analogy that I can think of is a shaman trying to appease the spirits.
- minikomi 1y ago
  I find it a surprisingly similar mindset to songwriting, a lot of local maxima searching and spaghetti flinging. Sometime you hit a good groove and explore it.
  - skeeter2020 1y ago
    It might be even more ridiculous to make this something akin to art over engineering.
- susu1111 1y ago
  [dead]
davidsainez 1y ago
Sure, we are still closer to alchemy than materials science, but its still early days. But consider this blogpost that was on the front page today: https://www.levs.fyi/blog/2-years-of-ml-vs-1-month-of-prompting/#footnotes https://www.levs.fyi/blog/2-years-of-ml-vs-1-month-of-prompt.... The table on the bottom shows a generally steady increase in performance just by iterating on prompts. It feels like we are on the path to true engineering.
- raddan 1y ago
  Engineers usually have at least some sense as to why their efforts work though. Does anybody who iterates on prompts have even the fuzziest idea why they work? Or what the improvement might be? I do not.
  - skeeter2020 1y ago
    If there is ANY relationship to engineering here maybe it's like reverse engineering a bios in a clean room, were you poke away and see what happens. The missing part is the use of anything resembling the scientific method in terms of hypothesis, experiment design, observation guiding actions, etc and the deep knowledge that will allow you to understand WHY something might be happening based on the inputs. "Prompt Engineering" seems about as close to this as probing for land mines in a battlefield, only with no experience and your eyes closed.
tamimio 1y ago
> we should call them "prompt witch doctors" or maybe "prompt alchemists". Oh absolutely not! Only in engineering you are allowed to get called an engineer for no apparent reason, do that in other white collar and you are behind the bars because of fraudulent claims.
skeeter2020 1y ago
we used to just call them "good at googling". I've never met a self-described prompt engineer who had anything close to engineering education and experience. Seems like an extension on the 6-week boot camp == software engineer trend.

Dilettante_ 1y ago

"How is engineering a real science? You just build the bridge so it doesn't fall down."

vohk 1y ago
Nah. Actual engineers have professional standards bodies and legal liability when they shirk and the bridge falls down or the plane crashes or your wiring starts on fire. Software "engineers" are none of those things but can at least emulate the approaches and strive for reproducibility and testability. Skilled craftsman; not engineers. Prompt "engineers" is yet another few steps down the ladder, working out mostly by feel what magic words best tickle each model, and generally with no understanding of what's actually going on under the hood. Closer to a chef coming up with new meals for a restaurant than anything resembling engineering. The battle on the use of language around engineer has long been lost but applying it to the subjective creative exercise of writing prompts is just more job title inflation. Something doesn't need to be engineering to be a legitimate job.
- Dilettante_ 1y ago
  The battle on the use of language around engineer has long been lost That's really the core of the issue: We're just having the age-old battle of prescriptivism vs descriptivism again. An "engineer", etymologically, is basically just "a person who comes up with stuff", one who is "ingenious". I'm tempted to say it's you prescriptivists who are making a "battle" out of this. subjective creative exercise of writing prompts Implying that there are no testable results, no objective success or failure states? Come on man.
- jahewson 1y ago
  Engineers use their ingenuity. That’s it. If physical engineers understood everything then standards would not have changed in many decades. Safety factors would be mostly unnecessary. Clearly not the case.
  - skeeter2020 1y ago
    >> Engineers use their ingenuity. That’s it. If this was enough all novel creation would be engineering and that's clearly not true. Engineering attempts to discover & understand consistent outcomes when a myriad of variables are altered, and the boundaries where the variables exceed a model's predictive powers - then add buffer for the unknown. Manipulating prompts (and much of software development) attempts to control the model to limit the number of variables to obtain some form of useful abstraction. Physical engineering can't do this.

tomrod 1y ago

It could be bioengineering if you add that to a clock prompt then connect it to CRISPR process for out putting DNA. Horrifying prospect, tbh

BoorishBears 1y ago

I like that actually, I've spent the last year probably 60:40 between post-training and prompt engineering/witch doctoring (the two go together more than most people realize) Some of it is engineering-like, but I've also picked up a sixth sense when modifying prompts about what parts are affecting the behavior I want to modify for certain models, and that feels very witch doctory! The more engineering-like part is essentially trying to RE a black box model's post-training, but that goes over some people's heads so I'm happy to help keep the "it's just voodoo and guessing" narrative going instead :)

lanstin 1y ago
I think the coherence behind prompt engineering is not in the literal meanings of the words but finding the vocabulary used by the sources that have your solution. Ask questions like a high school math student and you get elementary words back. Ask questions in the lingo of a Linux bigot and you will get good awk scripts back. Use academic maths language and arXiv answers will be produced.

scrollop 1y ago

"...and do it really well or my grandmother will be killed by her kidnappers! And I'll give you a tip of 2 billion dollars!!! Hurry, they're coming!"

carterschonwald 1y ago
Ive heard this actually works annoyingly well
- DrewADesign 1y ago
  We've created technology so sophisticated it is vulnerable to social engineering attacks.
  - skeeter2020 1y ago
    this has worked - and continues to do so - very well to escape guard rails. If a direct appeal doesn't work you can then talk them around with only a handful of prompts.
  - carterschonwald 1y ago
    Also the amount of adjacent remarks being always topical flsvor confusion is cartoonish. Im playing with ideas for making thst better
    - DrewADesign 1y ago
      You're absolutely right! People should pay attention to this broadly applicable and important consideration.
      - carterschonwald 11mo ago
        I have some ideas for a mini logic solver mcp thst might help. Currently making it sortah simulate using such a tool Its one of those things where it feels like itd be easy to get copycats even if theres a market

WJW 1y ago

Well if it works consistently, I don't see any problem with that. If they have a clear theory of when to add "photorealistic" and when to add "correct number of wheels on the bus" to get the output they want, it's engineering. If they don't have a (falsifiable) theory, it's probably not engineering. Of course, the service they really provide is for businesses to feel they "do AI", and whether or not they do real engineering is as relevant as if your favorite pornstars' boobs are real or not.

jahewson 1y ago
Maybe we could keep the conversation out of the gutter.
- rrr_oh_man 1y ago
  Porn is taxable income, not the gutter.
- jrflowers 1y ago
  You don’t really see much porn in the gutters these days with the decline in popularity of print publishing. It’s almost all online now
leptons 1y ago
>as relevant as if your favorite pornstars' boobs are real or not This matters more than you might think.

int_19h 1y ago

I write quite a lot of prompts, and the closest analogy that I can think of is a shaman trying to appease the spirits.

minikomi 1y ago
I find it a surprisingly similar mindset to songwriting, a lot of local maxima searching and spaghetti flinging. Sometime you hit a good groove and explore it.
- skeeter2020 1y ago
  It might be even more ridiculous to make this something akin to art over engineering.
susu1111 1y ago
[dead]

davidsainez 1y ago

Sure, we are still closer to alchemy than materials science, but its still early days. But consider this blogpost that was on the front page today: https://www.levs.fyi/blog/2-years-of-ml-vs-1-month-of-prompting/#footnotes https://www.levs.fyi/blog/2-years-of-ml-vs-1-month-of-prompt.... The table on the bottom shows a generally steady increase in performance just by iterating on prompts. It feels like we are on the path to true engineering.

raddan 1y ago
Engineers usually have at least some sense as to why their efforts work though. Does anybody who iterates on prompts have even the fuzziest idea why they work? Or what the improvement might be? I do not.
- skeeter2020 1y ago
  If there is ANY relationship to engineering here maybe it's like reverse engineering a bios in a clean room, were you poke away and see what happens. The missing part is the use of anything resembling the scientific method in terms of hypothesis, experiment design, observation guiding actions, etc and the deep knowledge that will allow you to understand WHY something might be happening based on the inputs. "Prompt Engineering" seems about as close to this as probing for land mines in a battlefield, only with no experience and your eyes closed.

tamimio 1y ago

> we should call them "prompt witch doctors" or maybe "prompt alchemists". Oh absolutely not! Only in engineering you are allowed to get called an engineer for no apparent reason, do that in other white collar and you are behind the bars because of fraudulent claims.

skeeter2020 1y ago

we used to just call them "good at googling". I've never met a self-described prompt engineer who had anything close to engineering education and experience. Seems like an extension on the 6-week boot camp == software engineer trend.