5 ms·

I'm extremely skeptical that people are getting the actual prompt when they're attempting to reverse engineer it. Jasper's CEO on Twitter refuted an attempt to

I'm extremely skeptical that people are getting the actual prompt when they're attempting to reverse engineer it. Jasper's CEO on Twitter refuted an attempt to reverse engineer their prompt. The attempt used very similar language to most other approaches I've seen. https://twitter.com/DaveRogenmoser/status/1601437119603302401 https://twitter.com/DaveRogenmoser/status/160143711960330240... There's no way to verify you're getting the original prompt. It could very easily be spitting out something that sounds believable but is completely wrong. If someone from Notion is hanging around I'd love to know how close these are.

mattigames 4y ago
...are you trying to extract information from Notion's employees!? Pretty sure that qualifies as a social engineering attack! /s
- swyx 4y ago
  i assume they're ok with it given @jitl is on this thread saying "I highly recommend using prompt injection" haha
swyx 4y ago
> There's no way to verify you're getting the original prompt. (author here) I do suggest a verification method for readers to pursue https://lspace.swyx.io/i/93381455/prompt-leaks-are-harmless https://lspace.swyx.io/i/93381455/prompt-leaks-are-harmless . If the sources are correct, you should be able to come to exactly equal output given the same inputs for obviously low-temperature features. (some features, like "Poem", are probably high-temp on purpose) In fact I almost did it myself before deciding I should probably just publish first and see if people even found this interesting before sinking more time into it. The other hint of course is that the wording of the prompts i found much more closely match how I already knew (without revealing) the GPT community words their prompts in these products, including templating and goalsetting (also discussed in the article) - not present in this naive Jasper attempt.
- ZephyrBlu 4y ago
  I guess it depends what the goal of the reverse engineering is. If it's to get a prompt that produces similar output, then this seems like a reasonable result. If it's to get the original prompt, I don't think that similar output is sufficient to conclude you've succeeded. This type of reverse engineering feels more like a learning tool (What do these prompts look like?) as opposed to truly reverse engineering the original prompt.
  - eyelidlessness 4y ago
    It also depends what you hope to accomplish with what you’ve reverse engineered. For example, the spectrum of acceptably usable reverse engineered gaming consoles ranges from some baseline of targets known to work, all the way to obsessive dedication to feature and bug parity. Most (not all!) emulators opt for high compatibility, rather than exhaustive. I don’t know where that high bar is for AI prompts, but I’d bet it’s more forgiving than this exacting standard. And it’s more thorough than the learning tool characterization too.
- lelandfe 4y ago
  >> There's no way to verify you're getting the original prompt. > I do suggest a verification method for readers to pursue … you should be able to come to exactly equal output given the same inputs for obviously low-temperature inputs 90ish% of the time. This sounds like “correct, there’s no way to verify,” but with more words.
jitl 4y ago
For the action items example, some of the prompt text is produced verbatim, some is re-ordered, some new text is invented, and a bunch is missing. Keep trying! (I work at Notion)
- ZephyrBlu 4y ago
  Thanks for the context! That's better than I expected, but it's interesting a bunch of stuff is missing.
- swyx 4y ago
  action items was the hardest one!!! i referred to it as the "final boss" in the piece lol (any idea why action items is so particularly hard? it was like banging my head on a wall compared to the others. did you do some kind of hardening on it?)
  - jitl 4y ago
    ¯\_(ツ)_/¯
- anothernewdude 4y ago
  Does it matter? Not if the prompt received represents the same embedding inside the AI. The exact wording won't matter. The missing stuff may ultimately just be ignored by the AI.
  - deleted 4y ago
    [deleted]
theCrowing 4y ago
It's the same as with generative art models that use CLIP you can do a reverse search and the prompt might not be exactly the same, but the outcome is.
- ZephyrBlu 4y ago
  If that's the goal it feels a bit pointless. If you have the skill to reverse engineer a prompt that produces similar results I assume you also have the skill to just write your own prompt.
  - theCrowing 4y ago
    The reverse engineering is done by the clip model and not by hand.
    - ZephyrBlu 4y ago
      Oh, I thought you meant it was a similar situation to in this post where it's done by hand. Automatically generating prompts based on the output image is pretty cool.
      - swyx 4y ago
        yup there are a range of "CLIP interrogators" out there. behold my collection https://github.com/sw-yx/ai-notes/blob/main/stub%20notes/IMAGE2TEXT.md https://github.com/sw-yx/ai-notes/blob/main/stub%20notes/IMA...
        theCrowing 4y ago
        Oh, wow I didn't recognize it by your username at first, but I use your repository/substack a lot for research. Thanks for the work.
        swyx 4y ago
        oh what kind of research? i am always curious how people use my “open source knowledge” repos but yeah basically people like you are why i do what i do (i gave a prezo on the open source knowledge concept a few yrs ago https://youtu.be/XoATf7xGoUY https://youtu.be/XoATf7xGoUY)
IshKebab 4y ago
Why are you skeptical? You can try it yourself on ChatGPT: https://imgur.com/a/Y8DYURU https://imgur.com/a/Y8DYURU > There's no way to verify you're getting the original prompt. Of course not, but the techniques seem to work reliably when tested on known prompts. I see no reason to doubt it.
- swyx 4y ago
  thats pretty cool, its like ChatGPT is a REPL for GPT
joe_the_user 4y ago
Why would someone need the verbatim prompt? If "prompt engineering" is going to be a thing (that has value), getting a prompt close enough to produce the same results would be what most reverse-engineers would want. In fact, not-verbatim could have advantages since you might argue you aren't infringing copyright.
weregiraffe 4y ago
https://twitter.com/DaveRogenmoser/ https://twitter.com/DaveRogenmoser/ My god, I checked this account and almost puked. How do you become such a distilled piece of tech-bro?
- IshKebab 4y ago
  I mean it's not that bad but this did stand out: > Talking software is fun, but what we really got jazzed about was… buying all the houses around us so all of our friends could live on the same street. Anyone else doing this? Uhm yeah loads of people are buying all the houses on a street. What planet is this guy from?
  - alexb_ 4y ago
    The planet of cheap money borrowing...

mattigames 4y ago

...are you trying to extract information from Notion's employees!? Pretty sure that qualifies as a social engineering attack! /s

swyx 4y ago
i assume they're ok with it given @jitl is on this thread saying "I highly recommend using prompt injection" haha

swyx 4y ago

> There's no way to verify you're getting the original prompt. (author here) I do suggest a verification method for readers to pursue https://lspace.swyx.io/i/93381455/prompt-leaks-are-harmless https://lspace.swyx.io/i/93381455/prompt-leaks-are-harmless . If the sources are correct, you should be able to come to exactly equal output given the same inputs for obviously low-temperature features. (some features, like "Poem", are probably high-temp on purpose) In fact I almost did it myself before deciding I should probably just publish first and see if people even found this interesting before sinking more time into it. The other hint of course is that the wording of the prompts i found much more closely match how I already knew (without revealing) the GPT community words their prompts in these products, including templating and goalsetting (also discussed in the article) - not present in this naive Jasper attempt.

ZephyrBlu 4y ago
I guess it depends what the goal of the reverse engineering is. If it's to get a prompt that produces similar output, then this seems like a reasonable result. If it's to get the original prompt, I don't think that similar output is sufficient to conclude you've succeeded. This type of reverse engineering feels more like a learning tool (What do these prompts look like?) as opposed to truly reverse engineering the original prompt.
- eyelidlessness 4y ago
  It also depends what you hope to accomplish with what you’ve reverse engineered. For example, the spectrum of acceptably usable reverse engineered gaming consoles ranges from some baseline of targets known to work, all the way to obsessive dedication to feature and bug parity. Most (not all!) emulators opt for high compatibility, rather than exhaustive. I don’t know where that high bar is for AI prompts, but I’d bet it’s more forgiving than this exacting standard. And it’s more thorough than the learning tool characterization too.
lelandfe 4y ago
>> There's no way to verify you're getting the original prompt. > I do suggest a verification method for readers to pursue … you should be able to come to exactly equal output given the same inputs for obviously low-temperature inputs 90ish% of the time. This sounds like “correct, there’s no way to verify,” but with more words.

jitl 4y ago

For the action items example, some of the prompt text is produced verbatim, some is re-ordered, some new text is invented, and a bunch is missing. Keep trying! (I work at Notion)

ZephyrBlu 4y ago
Thanks for the context! That's better than I expected, but it's interesting a bunch of stuff is missing.
swyx 4y ago
action items was the hardest one!!! i referred to it as the "final boss" in the piece lol (any idea why action items is so particularly hard? it was like banging my head on a wall compared to the others. did you do some kind of hardening on it?)
- jitl 4y ago
  ¯\_(ツ)_/¯
anothernewdude 4y ago
Does it matter? Not if the prompt received represents the same embedding inside the AI. The exact wording won't matter. The missing stuff may ultimately just be ignored by the AI.
- deleted 4y ago
  [deleted]

theCrowing 4y ago

It's the same as with generative art models that use CLIP you can do a reverse search and the prompt might not be exactly the same, but the outcome is.

ZephyrBlu 4y ago
If that's the goal it feels a bit pointless. If you have the skill to reverse engineer a prompt that produces similar results I assume you also have the skill to just write your own prompt.
- theCrowing 4y ago
  The reverse engineering is done by the clip model and not by hand.
  - ZephyrBlu 4y ago
    Oh, I thought you meant it was a similar situation to in this post where it's done by hand. Automatically generating prompts based on the output image is pretty cool.
    - swyx 4y ago
      yup there are a range of "CLIP interrogators" out there. behold my collection https://github.com/sw-yx/ai-notes/blob/main/stub%20notes/IMAGE2TEXT.md https://github.com/sw-yx/ai-notes/blob/main/stub%20notes/IMA...
      - theCrowing 4y ago
        Oh, wow I didn't recognize it by your username at first, but I use your repository/substack a lot for research. Thanks for the work.
        swyx 4y ago
        oh what kind of research? i am always curious how people use my “open source knowledge” repos but yeah basically people like you are why i do what i do (i gave a prezo on the open source knowledge concept a few yrs ago https://youtu.be/XoATf7xGoUY https://youtu.be/XoATf7xGoUY)

IshKebab 4y ago

Why are you skeptical? You can try it yourself on ChatGPT: https://imgur.com/a/Y8DYURU https://imgur.com/a/Y8DYURU > There's no way to verify you're getting the original prompt. Of course not, but the techniques seem to work reliably when tested on known prompts. I see no reason to doubt it.

swyx 4y ago
thats pretty cool, its like ChatGPT is a REPL for GPT

joe_the_user 4y ago

Why would someone need the verbatim prompt? If "prompt engineering" is going to be a thing (that has value), getting a prompt close enough to produce the same results would be what most reverse-engineers would want. In fact, not-verbatim could have advantages since you might argue you aren't infringing copyright.

weregiraffe 4y ago

https://twitter.com/DaveRogenmoser/ https://twitter.com/DaveRogenmoser/ My god, I checked this account and almost puked. How do you become such a distilled piece of tech-bro?

IshKebab 4y ago
I mean it's not that bad but this did stand out: > Talking software is fun, but what we really got jazzed about was… buying all the houses around us so all of our friends could live on the same street. Anyone else doing this? Uhm yeah loads of people are buying all the houses on a street. What planet is this guy from?
- alexb_ 4y ago
  The planet of cheap money borrowing...