3 ms·

I wonder if anyone yet has the job title "Query Engineer"? [EDIT: "Prompt Engineer"?] Also: if ML tech stalled now I expect we'd grow to have many Query Engin

by jefftk 4y ago

I wonder if anyone yet has the job title "Query Engineer"? [EDIT: "Prompt Engineer"?] Also: if ML tech stalled now I expect we'd grow to have many Query Engineers, but if it doesn't do we still? For how long is this sort of experimentation what getting the best performance out of an AI will look like?

ot 4y ago
"Prompt Engineering" already got some traction [1], so should be Prompt Engineer. [1] https://en.wikipedia.org/wiki/Prompt_engineering https://en.wikipedia.org/wiki/Prompt_engineering
- gorgoiler 4y ago
  I am certainly not a prompt engineer. It’s ready when it’s ready now leave me in peace to work!
  - olodus 4y ago
    ... the engineer says as he keeps scrolling through DALL-E output while adding funny words at the end.
ImprobableTruth 4y ago
I'm not sure if I believe that we'll have dedicated positions for query engineers because it's so hard to validate. Programming may seem like wizardry, but it's the rule-bound, alchemy version of wizardry, this on the other hand is more like communing with spirits. All the conjecture in this post could be right on the money, or total bunk. I don't see how anybody could reasonably hire someone for this position under this condition. How would one possibly discern between someone who can truly suss out what's going on and a techpriest who just recites learned dogma? I imagine it will definitely be an important skill, but I imagine that just like 'knowing how to google', it will be something that's just implicitly attached to a normal job.
- visarga 4y ago
  When GPT-3 came out it was very hard to coax even with prompts. Then OpenAI made Instruct GPT-3 and it suddenly became malleable and much better at taking a hint. I bet prompting is going to become even easier with time, so it won't be a skill one has to learn over years of practice anymore.
- jefftk 4y ago
  What's wrong with work-sample tests? Give them a toy problem and see how well they can solve it with a top AI and some iteration on queries.
- lmm 4y ago
  There are already plenty of jobs like that though? E.g. economists seem to have much the same issue.
- jcims 4y ago
  Just don’t use the word engineer. Some people are better at it than others if you just watch the hashtags on Twitter. There will very clearly be value derived from it before we understand it. It’s very likely, in my opinion, that people will be hired for this type of work.
vl 4y ago
They are called DBAs.
- jefftk 4y ago
  That's a very different sort of Query Engineering than what the author of this post is engaging in; I'm trying to talk about the latter (apparently also called Prompt Engineering).
soared 4y ago
My job is not incredibly far off of this. My company is based on Poland and so all the the software is written by Eastern Europeans with very different ideas of what words mean, how ux works, and how software should be structured. I have ~15 deep learning algorithms that I control some inputs/filters/etc on and my job is to get specific outputs from them every day. But I never know exactly what I’ll get and the raw inputs change so I need to respond by making changes every day. I have slowly gotten better at learning what may happen when I pull a lever, but it’s all feel and guesswork. I have no access to documentation/etc so can’t look inside the black box.
bmc7505 4y ago
"And when your surpassing creations find the answers you asked for, you can’t understand their analysis and you can’t verify their answers. You have to take their word on faith—Or you use information theory to flatten it for you, to squash the tesseract into two dimensions and the Klein bottle into three, to simplify reality and pray to whatever Gods survived the millennium that your honorable twisting of the truth hasn’t ruptured any of its load-bearing pylons. You hire people like me; the crossbred progeny of profilers and proof assistants and information theorists… In formal settings you’d call me Synthesist." —Peter Watts, Blindsight (2006)
- hoseja 4y ago
  ... and then the "hero" proceeds to not use this craft of his, at all.
  - FeepingCreature 4y ago
    As I read it, the entire book is him doing synthesis from the experts to us, the reader. He explicitly calls out that there's one character that he lets "speak for herself", implying that for everyone else, he's always translating, dumbing down for consumption. He actually says that their normal language would be unintelligible to us because it's an opaque multi-language pidgin that relies on realtime consensus access to be parsed.
    - hoseja 4y ago
      You are right, that is how it gets described at the beginning but, again, mostly inconsequential flair, no? Like adding Universal Translators and never having a Darmok and Jalad episode.
      - FeepingCreature 4y ago
        Yeah, that's a fair accusation. Part of the problem is of course that it's hard to have a novel where you only notice the primary conceit if you don't understand what the narrator is explaining. :-) Like, since perfect synthesis is undetectable; it just sounds like what you're hearing is straightforward common sense. You only notice the process when it fails. So for sure, the book could have played with that a lot more, especially since Siri's tools are meant to be broken towards the end, when he's implied to be penning this. I think there's bits where the book suggests that's what's going on ("I can't tell you what it means, only what it says"), but it never came across well to me. But then again, maybe that's just how a failure of synthesis would look. Like, you still want to understand what the book is saying, but if the mechanism is working it's invisible, and if it's not working, then you're still relying on it working to produce the thing you're reading. So it's just inherently hard to shine a light on the thing itself with itself, which of course plays into the central theme as well. Ie. because Siri is implied to not be fully conscious, it's hard for him to catch himself in the act of translating for us readers. (Of course we may say that a book designed to be hard to read is still hard to read.)

by jefftk 4y ago

ot 4y ago

"Prompt Engineering" already got some traction [1], so should be Prompt Engineer. [1] https://en.wikipedia.org/wiki/Prompt_engineering https://en.wikipedia.org/wiki/Prompt_engineering

gorgoiler 4y ago
I am certainly not a prompt engineer. It’s ready when it’s ready now leave me in peace to work!
- olodus 4y ago
  ... the engineer says as he keeps scrolling through DALL-E output while adding funny words at the end.

ImprobableTruth 4y ago

I'm not sure if I believe that we'll have dedicated positions for query engineers because it's so hard to validate. Programming may seem like wizardry, but it's the rule-bound, alchemy version of wizardry, this on the other hand is more like communing with spirits. All the conjecture in this post could be right on the money, or total bunk. I don't see how anybody could reasonably hire someone for this position under this condition. How would one possibly discern between someone who can truly suss out what's going on and a techpriest who just recites learned dogma? I imagine it will definitely be an important skill, but I imagine that just like 'knowing how to google', it will be something that's just implicitly attached to a normal job.

visarga 4y ago
When GPT-3 came out it was very hard to coax even with prompts. Then OpenAI made Instruct GPT-3 and it suddenly became malleable and much better at taking a hint. I bet prompting is going to become even easier with time, so it won't be a skill one has to learn over years of practice anymore.
jefftk 4y ago
What's wrong with work-sample tests? Give them a toy problem and see how well they can solve it with a top AI and some iteration on queries.
lmm 4y ago
There are already plenty of jobs like that though? E.g. economists seem to have much the same issue.
jcims 4y ago
Just don’t use the word engineer. Some people are better at it than others if you just watch the hashtags on Twitter. There will very clearly be value derived from it before we understand it. It’s very likely, in my opinion, that people will be hired for this type of work.

vl 4y ago

They are called DBAs.

jefftk 4y ago
That's a very different sort of Query Engineering than what the author of this post is engaging in; I'm trying to talk about the latter (apparently also called Prompt Engineering).

soared 4y ago

My job is not incredibly far off of this. My company is based on Poland and so all the the software is written by Eastern Europeans with very different ideas of what words mean, how ux works, and how software should be structured. I have ~15 deep learning algorithms that I control some inputs/filters/etc on and my job is to get specific outputs from them every day. But I never know exactly what I’ll get and the raw inputs change so I need to respond by making changes every day. I have slowly gotten better at learning what may happen when I pull a lever, but it’s all feel and guesswork. I have no access to documentation/etc so can’t look inside the black box.

bmc7505 4y ago

"And when your surpassing creations find the answers you asked for, you can’t understand their analysis and you can’t verify their answers. You have to take their word on faith—Or you use information theory to flatten it for you, to squash the tesseract into two dimensions and the Klein bottle into three, to simplify reality and pray to whatever Gods survived the millennium that your honorable twisting of the truth hasn’t ruptured any of its load-bearing pylons. You hire people like me; the crossbred progeny of profilers and proof assistants and information theorists… In formal settings you’d call me Synthesist." —Peter Watts, Blindsight (2006)

hoseja 4y ago
... and then the "hero" proceeds to not use this craft of his, at all.
- FeepingCreature 4y ago
  As I read it, the entire book is him doing synthesis from the experts to us, the reader. He explicitly calls out that there's one character that he lets "speak for herself", implying that for everyone else, he's always translating, dumbing down for consumption. He actually says that their normal language would be unintelligible to us because it's an opaque multi-language pidgin that relies on realtime consensus access to be parsed.
  - hoseja 4y ago
    You are right, that is how it gets described at the beginning but, again, mostly inconsequential flair, no? Like adding Universal Translators and never having a Darmok and Jalad episode.
    - FeepingCreature 4y ago
      Yeah, that's a fair accusation. Part of the problem is of course that it's hard to have a novel where you only notice the primary conceit if you don't understand what the narrator is explaining. :-) Like, since perfect synthesis is undetectable; it just sounds like what you're hearing is straightforward common sense. You only notice the process when it fails. So for sure, the book could have played with that a lot more, especially since Siri's tools are meant to be broken towards the end, when he's implied to be penning this. I think there's bits where the book suggests that's what's going on ("I can't tell you what it means, only what it says"), but it never came across well to me. But then again, maybe that's just how a failure of synthesis would look. Like, you still want to understand what the book is saying, but if the mechanism is working it's invisible, and if it's not working, then you're still relying on it working to produce the thing you're reading. So it's just inherently hard to shine a light on the thing itself with itself, which of course plays into the central theme as well. Ie. because Siri is implied to not be fully conscious, it's hard for him to catch himself in the act of translating for us readers. (Of course we may say that a book designed to be hard to read is still hard to read.)