5 ms·

> 1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model. Does "Provoke better thinkin

> 1. The author is bad at prompting. There are many ways to reduce hallucinations and provoke better thinking paths for the model. Does "Provoke better thinking paths" mean re-rolling the dice until you find some hack specific to chatGPT that 'just works' or is there something more rigorous behind this?

Kareem71 3y ago
Seriously this is the part I dont understand about people parroting "prompt engineering" . Isn't it really just throwing random things at a non deterministic black box and hoping for the best?
- lsaferite 3y ago
  I find it's more like that silly experiment where you have to make a sandwich exactly as a kid (or adult) writes the instructions. You _think_ you have a good set of instructions and then you get peanut butter on the outside. So, you revisit the instructions to be clearer about what you want done. That's how I see prompt engineering. In that case, you are simply learning how the model tends to follow instructions and crafting a prompt around that. Not so much random, more purposeful.
  - FormerBandmate 3y ago
    That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want. It’s Clever Hans on steroids
    - dingosity 3y ago
      As always, it's unclear on which side of the keyboard the intelligence lies.
    - dragonwriter 3y ago
      > That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want. If its to get the model to present a fixed answer, sure. If its to get a model to do a better job at solving general classes of problems (such as when what you are optimizing is the built-in prompt in a ReAct/Reflexion implementation, not the prompt for a specific problem), that's, at a minimum, different from Clever Hans, even if its not “reasoning” (which is ill-defined).
  - Turskarama 3y ago
    Can you come up with a method that will get you a good response every single time? Because if you can't, it's not really engineering.
- BoorishBears 3y ago
  If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated. If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism. At the end of the day ML is probabilistic. You're always throwing random things at a black box and hoping for the best. There are strategies and patterns that work consistently enough (like ReACT) that they carry across many tasks, and there are some that you'll find for your specific task. And just like any piece of software you define your scope well, test for things within that scope, and monitor for poor outputs.
  - verdagon 3y ago
    ELI5 layers? Could someone like me see when I've used one layer as opposed to another, when using ChatGPT?
    - BoorishBears 3y ago
      Technically I'm taking a large liberty saying you're "activating layers", all the layers are affecting the output and you don't pick and choose them But you can imagine the model like a plinko board: just because the ball passes every peg, doesn't mean every peg changed it's trajectory. When you fine tune a model, you're trying to change how the pegs are arranged so the ball falls through the board differently. When you prompt tune you're changing how the ball will fall too. You don't get to change the board, but you can change where the ball starts or have the ball go through the board several more times than normal before the user sees it, etc. You can't see the ball falling (which layers are doing what), only where it falls, but when you spend long enough building on these models, you do get an intuition for which prompts have an outsized effect on where the ball will land.
  - notahacker 3y ago
    > If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated. > If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism. There are good reasons for that though. The first is the model-owner tuning so that given inputs yield better outputs (in theory for other users too). The second is relying on the user to diagnose and fix the error. That being the "fix" is a problem if the output is supposed to be useful to people who don't know the answers themselves, or if the model is being touted as "intelligence" with a natural language interface, which is where the scepticism comes in... I mean, a bugfix, a recommendation not to use the 3rd menu option or a "fork this" button are all valid routes to change the runtime behaviour of a program! (and yes, I get that the "tuning" might simply be creating the illusion that the model approaches wider usability, and that "fine tuning" might actually have worse side effects. So it's certainly reasonable to argue that when a company defines its models' scope as "advanced reasoning capabilities" the "tuning" might also deserve scepticism, and conversely if it defines its scope more narrowly as something like "code complete" there might be a bit more onus on the user to provide structured, valid inputs)
- bbor 3y ago
  That’s science :). We have now established many strategies for context-independent effective prompting via typical experimental research - see https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api https://help.openai.com/en/articles/6654000-best-practices-f...
  - dingosity 3y ago
    That is explicitly NOT science.
- tiborsaas 3y ago
  Almost, but it's more like throwing ideas with an expected outcome at a non deterministic black box and hoping for the best.
- dragonwriter 3y ago
  No, its not. While GPT-4 (like some but not all other LLMs) is somewhat nondeterministic (even at zero temperature), that doesn’t mean there aren’t things that have predictable effects on the distribution of behavior that can be discovered and leveraged.
- redox99 3y ago
  Obviously not. For example using Chain of Thought will increase the model's performance[1] [1] https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
- qgin 3y ago
  It’s not deterministic, but if you set temperature = 0, you’ll start of find reliable techniques for all kinds outcomes.
- throwawayadvsec 3y ago
  There are definitely patterns that transfer across different problems, it's not random at all.
- deleted 3y ago
  [deleted]
- benreesman 3y ago
  There’s even a term of art for making a plan up front and then hitting it with a low-skew latent space match: “Chain of Thought”. Yeah, it’s seen numbered lists before. And if at first you don’t succeed, anneal the temperature and re-roll until you’ve got something that looks authentic.
  - AstralStorm 3y ago
    Right, but is it actually logically sound and complete, rather than truthy? In short, if you cannot use this thing without validating every step it's worthless for logic. You might as well solve the problem yourself.
    - benreesman 3y ago
      You got me beat: IMHO these things are plenty friggin awesome already and getting cooler all the time. I don't see why there is so much ink (and money) being spilled trying to get them to do things more easily done other ways. Language models are really good at language tasks: summarization, sentiment analysis, borderline-creepy convincing chatbots, writing pretty good fiction at least in short form, the list goes on and on. At all of the traditional NLP stuff they are just super impressive. They already represent an HCI revolution with significance something like the iPhone as a lower bound: it's a super big deal. But while the details are absurdly complicated and the super modern ones represent an engineering achievement up there with anything ever done on a computer, they still fundamentally predict some probability-like metric (typically still via softmax [0]) based on some corpus of tokenized language (typically still via byte-pair [1]). And when the corpus has a bunch of conversations in it? Great at generating conversations! And when the corpus has some explanations of logical reasoning? Often passably good at looking logical. And when the corpus has short stories, novellas, and novels featuring conversations between humans and science-fiction AIs? Well they can sample from that too. But imitating William Gibson doesn't make GPT-4 any kind of sentient any more than it makes me a once-in-a-generation science fiction author. “Real motive problem, with an AI. Not human, see?” “Well, yeah, obviously.” “Nope. I mean, it’s not human. And you can’t get a handle on it. Me, I’m not human either, but I respond like one. See?” “Wait a sec,” Case said. “Are you sentient, or not?” “Well, it feels like I am, kid, but I’m really just a bunch of ROM. It’s one of them, ah, philosophical questions, I guess...” The ugly laughter sensation rattled down Case’s spine. “But I ain’t likely to write you no poem, if you follow me. Your AI, it just might. But it ain’t no way human.” [2] [0] https://github.com/facebookresearch/llama/blob/main/llama/generation.py#L144 https://github.com/facebookresearch/llama/blob/main/llama/ge... [1] https://github.com/facebookresearch/llama/blob/main/llama/tokenizer.py#L18 https://github.com/facebookresearch/llama/blob/main/llama/to... [2] https://www.litcharts.com/lit/neuromancer/characters/mccoy-dixie-flatline-pauley https://www.litcharts.com/lit/neuromancer/characters/mccoy-d...
- thumbuddy 3y ago
  Yes. Yes it is.
Verdex 3y ago
I'm kind of concerned about the concept of "bad at prompting". The hypothesis that I'm working off right now is that natural language has structure to it that happens to match some problem spaces. And this makes sense because people will naturally want to talk succinctly and with a convenient flow relative to the problems they encounter the most. Thus jargon is reborn many times over in different domains. LLMs are encoding this structure. So a good prompt is one that provides the LLM with additional information about what you expect the answer to be. And bad prompts provide neutral or disinformation. This isn't to say that being good at prompts is somehow to be disingenuous about the power of LLMs. What is better? To remember much redundant data. Or to remember simply the right sorts of ways to search for the classes of information you are after. My concern, though, is that the structure of reality doesn't have to match the way that we talk about it. The Novel and the Inexpressible* will tend to yield hallucinations. [Although, I've had this concern long before I encountered LLMs. My feeling is that there are many people who can only solve problems that match the way they talk about them.] * - technically, the difficult or unnatural to express, but I couldn't fit that into a single word.
- pixl97 3y ago
  >I'm kind of concerned about the concept of "bad at prompting". I have met many people in my life that are terrible at asking questions, so it does have some conceptual reality. But this is also why analogy is so powerful for people. It takes the way a person thinks about $A and applies parts of it to $B so they can more easily wrap their mind around it. Has anyone written a paper about testing and expressing the power of analogy in LLMs?
letitgo12345 3y ago
Some of it is randomness for sure. However the prompts make sense usually intuitively and there's a big deterministic component -- as which prompts work are likely a function of the training data which has a huge overlap between models. Edit: And even if the exact same prompts don't work on different models, similar prompts often do.
d0mine 3y ago
"Tree of Thoughts: Deliberate Problem Solving with Large Language Models" seems better than just re-rolling the dice. https://github.com/princeton-nlp/tree-of-thought-llm https://github.com/princeton-nlp/tree-of-thought-llm
jncfhnb 3y ago
Phrasing a question poorly yields poor answers from humans. Does rephrasing the question mean re rolling dice until you get a form of question they understand?
- AstralStorm 3y ago
  The questions were released multiple times and GPT was asked for clarification. If it does not understand something, it should ask about the thing it does not understand rather than go forward. Even when corrected, it tends to produce wrong results repeatedly by insisting on falsehoods or failing to ensure its logic is complete.
Nevermark 3y ago
No, it’s how you get good answers and reasoning from humans too Because human languages are not precise. Human language requests often require some back and forth, to get on the same page. It is far more efficient to discuss a problem to solve, than try to waterfall it by wasting time trying to be absolutely painfully clear, without any feedback from your problem solver. Models quickly incorporating feedback is further evidence of complex reasoning.
RugnirViking 3y ago
it's less "keep rolling dice" and more getting a sense of what will work and what won't. The thing I liken it most to is watching a colleague try to google something for 30 mins and then you find it immediately first try because you know how to get google to do what you want. Similarly, it happens extremely often that when I watch someone else using chatgpt I see what they're trying to do, and know I would have gone about it another way that would have worked.

Kareem71 3y ago

Seriously this is the part I dont understand about people parroting "prompt engineering" . Isn't it really just throwing random things at a non deterministic black box and hoping for the best?

lsaferite 3y ago
I find it's more like that silly experiment where you have to make a sandwich exactly as a kid (or adult) writes the instructions. You _think_ you have a good set of instructions and then you get peanut butter on the outside. So, you revisit the instructions to be clearer about what you want done. That's how I see prompt engineering. In that case, you are simply learning how the model tends to follow instructions and crafting a prompt around that. Not so much random, more purposeful.
- FormerBandmate 3y ago
  That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want. It’s Clever Hans on steroids
  - dingosity 3y ago
    As always, it's unclear on which side of the keyboard the intelligence lies.
  - dragonwriter 3y ago
    > That isn’t the model reasoning. That’s you figuring out exactly what parameters you need to use to make the model give the result you want. If its to get the model to present a fixed answer, sure. If its to get a model to do a better job at solving general classes of problems (such as when what you are optimizing is the built-in prompt in a ReAct/Reflexion implementation, not the prompt for a specific problem), that's, at a minimum, different from Clever Hans, even if its not “reasoning” (which is ill-defined).
- Turskarama 3y ago
  Can you come up with a method that will get you a good response every single time? Because if you can't, it's not really engineering.
BoorishBears 3y ago
If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated. If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism. At the end of the day ML is probabilistic. You're always throwing random things at a black box and hoping for the best. There are strategies and patterns that work consistently enough (like ReACT) that they carry across many tasks, and there are some that you'll find for your specific task. And just like any piece of software you define your scope well, test for things within that scope, and monitor for poor outputs.
- verdagon 3y ago
  ELI5 layers? Could someone like me see when I've used one layer as opposed to another, when using ChatGPT?
  - BoorishBears 3y ago
    Technically I'm taking a large liberty saying you're "activating layers", all the layers are affecting the output and you don't pick and choose them But you can imagine the model like a plinko board: just because the ball passes every peg, doesn't mean every peg changed it's trajectory. When you fine tune a model, you're trying to change how the pegs are arranged so the ball falls through the board differently. When you prompt tune you're changing how the ball will fall too. You don't get to change the board, but you can change where the ball starts or have the ball go through the board several more times than normal before the user sees it, etc. You can't see the ball falling (which layers are doing what), only where it falls, but when you spend long enough building on these models, you do get an intuition for which prompts have an outsized effect on where the ball will land.
- notahacker 3y ago
  > If someone says they're fine tuning a model (which is changing which layers are activated for a given input) it's generally well tolerated. > If someone says they're tuning a prompt (which is changing which layers are activated for a given input) it's met with extreme skepticism. There are good reasons for that though. The first is the model-owner tuning so that given inputs yield better outputs (in theory for other users too). The second is relying on the user to diagnose and fix the error. That being the "fix" is a problem if the output is supposed to be useful to people who don't know the answers themselves, or if the model is being touted as "intelligence" with a natural language interface, which is where the scepticism comes in... I mean, a bugfix, a recommendation not to use the 3rd menu option or a "fork this" button are all valid routes to change the runtime behaviour of a program! (and yes, I get that the "tuning" might simply be creating the illusion that the model approaches wider usability, and that "fine tuning" might actually have worse side effects. So it's certainly reasonable to argue that when a company defines its models' scope as "advanced reasoning capabilities" the "tuning" might also deserve scepticism, and conversely if it defines its scope more narrowly as something like "code complete" there might be a bit more onus on the user to provide structured, valid inputs)
bbor 3y ago
That’s science :). We have now established many strategies for context-independent effective prompting via typical experimental research - see https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api https://help.openai.com/en/articles/6654000-best-practices-f...
- dingosity 3y ago
  That is explicitly NOT science.
tiborsaas 3y ago
Almost, but it's more like throwing ideas with an expected outcome at a non deterministic black box and hoping for the best.
dragonwriter 3y ago
No, its not. While GPT-4 (like some but not all other LLMs) is somewhat nondeterministic (even at zero temperature), that doesn’t mean there aren’t things that have predictable effects on the distribution of behavior that can be discovered and leveraged.
redox99 3y ago
Obviously not. For example using Chain of Thought will increase the model's performance[1] [1] https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903
qgin 3y ago
It’s not deterministic, but if you set temperature = 0, you’ll start of find reliable techniques for all kinds outcomes.
throwawayadvsec 3y ago
There are definitely patterns that transfer across different problems, it's not random at all.
deleted 3y ago
[deleted]
benreesman 3y ago
There’s even a term of art for making a plan up front and then hitting it with a low-skew latent space match: “Chain of Thought”. Yeah, it’s seen numbered lists before. And if at first you don’t succeed, anneal the temperature and re-roll until you’ve got something that looks authentic.
- AstralStorm 3y ago
  Right, but is it actually logically sound and complete, rather than truthy? In short, if you cannot use this thing without validating every step it's worthless for logic. You might as well solve the problem yourself.
  - benreesman 3y ago
    You got me beat: IMHO these things are plenty friggin awesome already and getting cooler all the time. I don't see why there is so much ink (and money) being spilled trying to get them to do things more easily done other ways. Language models are really good at language tasks: summarization, sentiment analysis, borderline-creepy convincing chatbots, writing pretty good fiction at least in short form, the list goes on and on. At all of the traditional NLP stuff they are just super impressive. They already represent an HCI revolution with significance something like the iPhone as a lower bound: it's a super big deal. But while the details are absurdly complicated and the super modern ones represent an engineering achievement up there with anything ever done on a computer, they still fundamentally predict some probability-like metric (typically still via softmax [0]) based on some corpus of tokenized language (typically still via byte-pair [1]). And when the corpus has a bunch of conversations in it? Great at generating conversations! And when the corpus has some explanations of logical reasoning? Often passably good at looking logical. And when the corpus has short stories, novellas, and novels featuring conversations between humans and science-fiction AIs? Well they can sample from that too. But imitating William Gibson doesn't make GPT-4 any kind of sentient any more than it makes me a once-in-a-generation science fiction author. “Real motive problem, with an AI. Not human, see?” “Well, yeah, obviously.” “Nope. I mean, it’s not human. And you can’t get a handle on it. Me, I’m not human either, but I respond like one. See?” “Wait a sec,” Case said. “Are you sentient, or not?” “Well, it feels like I am, kid, but I’m really just a bunch of ROM. It’s one of them, ah, philosophical questions, I guess...” The ugly laughter sensation rattled down Case’s spine. “But I ain’t likely to write you no poem, if you follow me. Your AI, it just might. But it ain’t no way human.” [2] [0] https://github.com/facebookresearch/llama/blob/main/llama/generation.py#L144 https://github.com/facebookresearch/llama/blob/main/llama/ge... [1] https://github.com/facebookresearch/llama/blob/main/llama/tokenizer.py#L18 https://github.com/facebookresearch/llama/blob/main/llama/to... [2] https://www.litcharts.com/lit/neuromancer/characters/mccoy-dixie-flatline-pauley https://www.litcharts.com/lit/neuromancer/characters/mccoy-d...
thumbuddy 3y ago
Yes. Yes it is.

Verdex 3y ago

I'm kind of concerned about the concept of "bad at prompting". The hypothesis that I'm working off right now is that natural language has structure to it that happens to match some problem spaces. And this makes sense because people will naturally want to talk succinctly and with a convenient flow relative to the problems they encounter the most. Thus jargon is reborn many times over in different domains. LLMs are encoding this structure. So a good prompt is one that provides the LLM with additional information about what you expect the answer to be. And bad prompts provide neutral or disinformation. This isn't to say that being good at prompts is somehow to be disingenuous about the power of LLMs. What is better? To remember much redundant data. Or to remember simply the right sorts of ways to search for the classes of information you are after. My concern, though, is that the structure of reality doesn't have to match the way that we talk about it. The Novel and the Inexpressible* will tend to yield hallucinations. [Although, I've had this concern long before I encountered LLMs. My feeling is that there are many people who can only solve problems that match the way they talk about them.] * - technically, the difficult or unnatural to express, but I couldn't fit that into a single word.

pixl97 3y ago
>I'm kind of concerned about the concept of "bad at prompting". I have met many people in my life that are terrible at asking questions, so it does have some conceptual reality. But this is also why analogy is so powerful for people. It takes the way a person thinks about $A and applies parts of it to $B so they can more easily wrap their mind around it. Has anyone written a paper about testing and expressing the power of analogy in LLMs?

letitgo12345 3y ago

Some of it is randomness for sure. However the prompts make sense usually intuitively and there's a big deterministic component -- as which prompts work are likely a function of the training data which has a huge overlap between models. Edit: And even if the exact same prompts don't work on different models, similar prompts often do.

d0mine 3y ago

"Tree of Thoughts: Deliberate Problem Solving with Large Language Models" seems better than just re-rolling the dice. https://github.com/princeton-nlp/tree-of-thought-llm https://github.com/princeton-nlp/tree-of-thought-llm

jncfhnb 3y ago

Phrasing a question poorly yields poor answers from humans. Does rephrasing the question mean re rolling dice until you get a form of question they understand?

AstralStorm 3y ago
The questions were released multiple times and GPT was asked for clarification. If it does not understand something, it should ask about the thing it does not understand rather than go forward. Even when corrected, it tends to produce wrong results repeatedly by insisting on falsehoods or failing to ensure its logic is complete.

Nevermark 3y ago

No, it’s how you get good answers and reasoning from humans too Because human languages are not precise. Human language requests often require some back and forth, to get on the same page. It is far more efficient to discuss a problem to solve, than try to waterfall it by wasting time trying to be absolutely painfully clear, without any feedback from your problem solver. Models quickly incorporating feedback is further evidence of complex reasoning.

RugnirViking 3y ago

it's less "keep rolling dice" and more getting a sense of what will work and what won't. The thing I liken it most to is watching a colleague try to google something for 30 mins and then you find it immediately first try because you know how to get google to do what you want. Similarly, it happens extremely often that when I watch someone else using chatgpt I see what they're trying to do, and know I would have gone about it another way that would have worked.