6 ms·

An example of the prompt engineering phenomenon: my wife and I were recently discussing a financial decision. I'd offered my arguments in favor of one choice an

An example of the prompt engineering phenomenon: my wife and I were recently discussing a financial decision. I'd offered my arguments in favor of one choice and she was mostly persuaded but decided to check in with ChatGPT to help reassure herself that I was right. She asked the financial question in layman's terms and got the opposite answer that I had given. She showed me the result and I immediately saw the logical flaws and pointed them out to her. She pressed the model on it and it of course apologized and corrected itself. Out of curiosity I tried the prompt again, this time using financial jargon that I was familiar with and my wife was not. The intended meaning of the words was the same, the only difference is that my prompt sounded like it came from someone who knew finance. The result was that the model got it right and gave an explanation for the reasoning in exacting detail. It was an interesting result to me because it shows that experts in a field are not only more likely to recognize when a model is giving incorrect answers but they're also more likely to get correct answers because they are able to tap into a set of weights that are populated by text that knew what it was talking about. Lay people trying to use an LLM to understand an unfamiliar field are vulnerable to accidentally tapping into the "amateur" weights and ending up with an answer learned from random Reddit threads or SEO marketing blog posts, whereas experts can use jargon correctly in order to tap into answers learned from other experts.

bdangubic 1y ago
very interesting - could you share your wife's and yours prompt to provide more concrete context?
- lolinder 1y ago
  I considered it, but given the personal nature of the topic I want to keep the details off the public internet. I realize that makes my observation less specific than it could be, but I think the result should be relatively unsurprising to anyone familiar with the tools—this is just a natural consequence of the way the stats fall out.
xnorswap 1y ago
Using LLM's in any unfamiliar context is dangerous. When asking them anything, always follow up with: "Are you sure?". Far too often it'll cheerily apologise and correct their own answer.
- jon-wood 1y ago
  Often it'll even apologise and correct it's own answer despite the original answer being correct, because you just primed it's model to believe it was wrong.
- jononor 1y ago
  Not necessarily correct the answer! Sometimes it just changes it, insists that it is correct - but it might still be wrong (perhaps now in a new way)...
- klabb3 1y ago
  First, these tips and tricks that ”works for me” aren’t universal, it may or may not work, there’s literally no way to tell other than to run large scale empirical experiments, otherwise it’s just mythbuilding. Secondly, it can be even worse. I’ve been ”gaslighted” when pressing on answers I knew were incorrect (in this case, cryptography). It comes up with extremely plausibly sounding arguments, specifically addressing my counterpoints, and even chain of reasoning, yet still isn’t correct. You’d have to be a domain expert to tell it’s wrong, at which point it makes no sense to use LLMs in the first place.
  - xnorswap 1y ago
    Oh for sure, asking "Are you sure?" isn't a trick to improve accuracy, it's a trick to show people the danger of asking it in the first place. It just leaves you with two contradictory statements, much like the man with two watches who never knows the correct time.
    - klabb3 1y ago
      Ah, I think I misunderstood the sentiment. Thanks for clarifying
amelius 1y ago
Maybe try a 2-step approach: first ask the LLM to translate your question into expert-language, then ask that question :)
- aleph_minus_one 1y ago
  The expert's answer when asked in expert language is "it's complicated". :-)
thunky 1y ago
I'd like to see the before and after questions because it seems possible that the layman's version was less exact and therefore interpreted differently, even if the intention was the same. Which, can happen with humans too.
- lolinder 1y ago
  Given the topic I'm unfortunately not comfortable sharing the details in a public space like this, but the answer that it gave was not just a misinterpretation of the question, it was actually entirely wrong on the merits of its own interpretation. And even if it were a misinterpretation the result is still largely the same: if you don't know how to ask good questions you won't get good answers, which makes it dangerous to rely on the tools for things that you're not already an expert in. This is in contrast to all to people who claim to be using them for learning about important concepts (including lots of people who claim to be using them as financial advisors!).
  - loveparade 1y ago
    If you don't know how to ask a human doctor a good question you can't expect to get a good answer either. The difference is that a human doctor probably has a lot of context about you and the situation you're in, so that they probably guess what your intention behind the question is, and adjust their answer appropriately. When you talk to an LLM, it has none of that context. So the comparison isn't really fair. Has your mom ever asked you a computer question? Half of the time the question makes no sense and explaining to her why would take hours, and then she still wouldn't get it. So the best you can do is guess what she wants based on the context you have.
- deadbabe 1y ago
  Doesn’t matter. The LLM’s job should be to deliver results in the way that is intended, regardless of the skill of the prompter. If it can’t do that then it’s really no better than a Google search.
  - osigurdson 1y ago
conception 1y ago
Can you do a quick experiment and ask ChatGPT to craft a prompt to answer “laymen question “ as a financial expert? Having ChatGPT craft its own prompts is usually pretty successful for me.
- lolinder 1y ago
  That probably would work most of the time, but that's also an example of the phenomenon that TFA is talking about: you can't safely just use these tools without becoming an expert at least in the tool. The way they're currently being sold as totally accessible to everyone is dangerous.
  - diggan 1y ago
    > without becoming an expert at least in the tool Yeah, we're basically repeating the "search engine/query" problem but slightly differently. Using a search engine the right way always been a skill you needed to learn, and the ones who didn't always got poor results, and many times took those results at face value. Then Google started giving "answers" so if your query is shit, the "answer" most likely is too. Point is, I don't think this phenomenon is new, it's just way less subtle today with LLMs, at least for people who have expertise in the subjects.
    - solarwindy 1y ago
      > On two occasions I have been asked, ’Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
- osigurdson 1y ago
  Understanding the terminology can help a lot. The key is to start the LLM conversation like this, ask it what terminology is used in a particular problem domain and then frame your actual question based on this.
colinmorelli 1y ago
Related similar thing when I sent my dog's recent bloodwork to an LLM, including dates, tests, and values. The model suggested that an advancement in her kidney values (all still within normal range) were likely evidence of chronic kidney disease in its early stage. Naturally this caused some concern for my wife. But, I work in healthcare and have enough knowledge of health to know that CKD almost certainly could not advance fast enough to be the cause of the kidney value changes in the labs that were only 6 weeks apart. I asked the LLM if that's the best explanation for these values given they're only 6 weeks apart, and it adjusted its answer to say CKD is likely not the explanation as progression would happen typically over 6+ months to a year at this stage, and more likely explanations were nephrotoxins (recent NSAID use), temporary dehydration, or recent infection. We then spoke to our vet who confirmed that CKD would be unlikely to explain a shift in values like this between two tests that were just 6 weeks apart. That would almost certainly throw off someone with less knowledge about this, however. If the tests were 4-6 months apart, CKD could explain the change. It's not an implausible explanation, but it skipped over a critical piece of information (the time between tests) before originally coming to that answer.
- osigurdson 1y ago
  The internet, and now LLMs have always been bad at diagnosing medical problems. I think it comes from the data source. For instance, few articles would be linked to / popular if a given set of symptoms were just associated with not getting enough sleep. No, the articles stand out are the ones where the symptoms are associated with some rare / horrible condition. This is our LLM training data which are often missing the entire middle part of the bell curve.
  - colinmorelli 1y ago
    For what it's worth this statement is actually not entirely correct anymore. Top-end models today are on par with diagnostic capabilities of physicians on average (across many specialties), and, in some cases, can outperform them when RAG'd in with vetted clinical guidelines (like NIH data, UpToDate, etc) However, they do have particular types of failure modes that they're more prone to, and this is one of them. So they're imperfect.
rvz 1y ago
Also see the Gell-Mann amnesia effect: [0] [0] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect
cjohnson318 1y ago
I had a similar experience. I did some back of the envelope math and my wife suggested I run it through ChatGPT. After actually doing the math, I felt a lot better about my understanding of the problem and I just... don't trust an LLM to understand algebra. Yeah, they're awesome most of the time, but I don't want to trust it with something important, be wrong, and then have to explain to someone that I trusted the opinion of a couple of big matrices, over my own knowledge and experience, on a high school word problem.
- aaronbaugher 1y ago
  I've noticed Grok struggles with dates and relative time, especially when referring to things it "remembers" from earlier conversations. Even a phrase like "last night" will be wrongly interpreted sometimes. So although I've had it research numbers and create estimates for me, I wouldn't just assume the numbers are right without checking everything.
- lblume 1y ago
  Since LLMs are basically linear algebra all the way down, this is vaguely reminiscent of how human brains also have a very hard time understanding neural circuitry despite literally being made from it.
disambiguation 1y ago
This is my experience with prompting as well, but I struggle to describe it adequately. Something like the "direction" of the prompt, if it's too open ended you're likely to get mixed results, but if you give it a kind of "running start" it performs much better.
caust1c 1y ago
This anecdote corroborates my theory that it will still be critical to become an expert in your field. Everyone is treating AI like it's a zero-sum game with regards to jobs being "lost" to AI, but the reality is that the best results will come from experts in the field who have the vocabulary and knowledge to get the best answers. My fear is that people treat AI like an oracle when they should be treating it just like any other human being.
- lblume 1y ago
  What percentage of people are actually experts at their jobs though?
- xnorswap 1y ago
  People treat certain humans, or humans in certain roles, as oracles too.
- redeye100 1y ago
  This is just bad design. Or a faulty tool. Why should the job market shift to accommodate this gap in the functioning of LLMs? This is a bug that needs to be fixed. I have a personal gripe about this bringing an unfinished tool to market and then prophetizing about its usefulness. And how we all better get ready for it. This seems very hand-wavey and is looking more and more like vaporware. It's like trying to quickly build a house on an unfinished foundation. Why are we rushing to build? Can't we get the foundational things right first?
chriskanan 1y ago
What I do is to always set the context by giving the my "background" and some papers as reading material such that I've conditioned the model for whatever topic that will be discussed as the first step.
aleph_minus_one 1y ago
For the sake of discussion I want to play devil's advocate concerning your point > It was an interesting result to me because it shows that experts in a field are not only more likely to recognize when a model is giving incorrect answers but they're also more likely to get correct answers because they are able to tap into a set of weights that are populated by text that knew what it was talking about. Lay people trying to use an LLM to understand an unfamiliar field are vulnerable to accidentally tapping into the "amateur" weights and ending up with an answer learned from random Reddit threads or SEO marketing blog posts, whereas experts can use jargon correctly in order to tap into answers learned from other experts. Couldn't it be the case that people who (in this case recognizable to the AI by their choice of wording) are knowledgeable in the topic need different advise than people who know less about the topic? To give one specific examples from finance: if you know a lot about finance, getting some deep analysis and advice about what is the best way to trade some exotic options is likely sound advice. On the other hand, for people who are not deeply into finance the best advice is likely rather "don't do it!".
- lolinder 1y ago
  In some cases, sure, but not here—neither option had more risk associated with it than the other, it was just an optimization problem. The first answer that the model gave to my wife was just wrong about the math, with no room for subjectivity.
- dogleash 1y ago
  > Couldn't it be the case [...] need different advise than people who know less about the topic? > for people who are not deeply into finance the best advice is likely rather "don't do it!". Oh boy, more nanny software. This future blows.
  - aleph_minus_one 1y ago
    > Oh boy, more nanny software. This future blows. I think this topic is a little bit more complicated: this is rather a balancing of the model between 1. "giving the best possible advice to the respective person given their circumstances" vs 2. "giving the most precise answer to the query to the user" (if you ask me: the best decision would in my opinion be to give the user a choice for this, but this would be overtaxing to many users) - Freedom-loving people will hate it if they don't get 2 - On the other hand, many people would like to actually get the advice that is most helpful to them (i.e. 1), and not the one that may answer their question exactly, but is likely a bad idea for them
c22 1y ago
This same phenomenon is true for classic search engines as well. Whenever I am becoming informed on a new topic my first searches are always very naive and targeted just at discovering the relevant jargon that will let me make better searches. It turns out that many disciplines contain analogue concepts, just with different words being used to describe them. Understanding the domain specific language used is more than half the battle.
- nothercastle 1y ago
  Yeah I reminder that being the trick to get Google to provide good results was to find some key industry or area terms for what you were looking for. This doesn’t work anymore because Google search has gotten so bad
- skydhash 1y ago
  If I’m dealing with an unfamiliar domain, my next step is always wikipedia or an introductory book. Just to collect a sets of keywords and references to narrow my future searches. I don’t think I’ve ever asked google a question shaped query.
- citrin_ru 1y ago
  The difference is that in the search engine output one can somewhat judge if a site is trustworthy. E. g. if I research a legal question I would trust a guide on a government website more than an answer on Reddit which contradicts an official source. With LLM I don’t know where weight for a particular answer came from. You can ask LLM for sources but its text answer can still misrepresent these sources.
lostmsu 1y ago
A couple more alternative explanations: 1. random - sample size is too small to claim otherwise 2. phrasing in both cases had a leaning in it, and as a "yes man" LLM gave corresponding biased response.
whstl 1y ago
I very often also get better programming results than less experienced engineers, even though I'm not remotely doing any kind of "prompt engineering". Also how you ask matters a lot. Sometimes it just wants to make you happy with whatever answer, if you go along without skepticism it will definitely make garbage. Fun story: at a previous job a Product Manager made someone work a full week on a QR-Code standard that doesn't exist, except in ChatGPT's mind. It produced test cases and examples, but since nobody had a way to test When it was sent to a bank in Sweden to test, the customer was just "wait this feature doesn't exist in Sweden" and a heated discussion ensued until the PM admitted using ChatGPT to create the requirements.
jimbokun 1y ago
So LLMs need an ELI5 mode and detect when to use it. This would use the non-technical terminology but pull concepts from the more jargony model. This may or may not be easily possible by tweaking current training techniques. But it shows the many edge cases still needed to be addressed by AI models.
ivanjermakov 1y ago
LLM users are highly susceptible to the confirmation bias, by putting their expectations into the prompt.
tda 1y ago
Interesting observation. I find myself using ChatGPT to find the proper words for something. I describe my problem or algorithm in a naive way, and usually ChapGPT will present some naive python solution. But when pressed ChatGPT will tell you this problem is actually say Graph Theory. But it will still fall back to the naive python solution. So then I just start a new "conversation" and ask which graph algorithm applies to my problem and instead of a naive solution I am pointed to an optimised algorithm and a usable library that implements it.

bdangubic 1y ago

very interesting - could you share your wife's and yours prompt to provide more concrete context?

lolinder 1y ago
I considered it, but given the personal nature of the topic I want to keep the details off the public internet. I realize that makes my observation less specific than it could be, but I think the result should be relatively unsurprising to anyone familiar with the tools—this is just a natural consequence of the way the stats fall out.

xnorswap 1y ago

Using LLM's in any unfamiliar context is dangerous. When asking them anything, always follow up with: "Are you sure?". Far too often it'll cheerily apologise and correct their own answer.

jon-wood 1y ago
Often it'll even apologise and correct it's own answer despite the original answer being correct, because you just primed it's model to believe it was wrong.
jononor 1y ago
Not necessarily correct the answer! Sometimes it just changes it, insists that it is correct - but it might still be wrong (perhaps now in a new way)...
klabb3 1y ago
First, these tips and tricks that ”works for me” aren’t universal, it may or may not work, there’s literally no way to tell other than to run large scale empirical experiments, otherwise it’s just mythbuilding. Secondly, it can be even worse. I’ve been ”gaslighted” when pressing on answers I knew were incorrect (in this case, cryptography). It comes up with extremely plausibly sounding arguments, specifically addressing my counterpoints, and even chain of reasoning, yet still isn’t correct. You’d have to be a domain expert to tell it’s wrong, at which point it makes no sense to use LLMs in the first place.
- xnorswap 1y ago
  Oh for sure, asking "Are you sure?" isn't a trick to improve accuracy, it's a trick to show people the danger of asking it in the first place. It just leaves you with two contradictory statements, much like the man with two watches who never knows the correct time.
  - klabb3 1y ago
    Ah, I think I misunderstood the sentiment. Thanks for clarifying

amelius 1y ago

Maybe try a 2-step approach: first ask the LLM to translate your question into expert-language, then ask that question :)

aleph_minus_one 1y ago
The expert's answer when asked in expert language is "it's complicated". :-)

thunky 1y ago

I'd like to see the before and after questions because it seems possible that the layman's version was less exact and therefore interpreted differently, even if the intention was the same. Which, can happen with humans too.

lolinder 1y ago
Given the topic I'm unfortunately not comfortable sharing the details in a public space like this, but the answer that it gave was not just a misinterpretation of the question, it was actually entirely wrong on the merits of its own interpretation. And even if it were a misinterpretation the result is still largely the same: if you don't know how to ask good questions you won't get good answers, which makes it dangerous to rely on the tools for things that you're not already an expert in. This is in contrast to all to people who claim to be using them for learning about important concepts (including lots of people who claim to be using them as financial advisors!).
- loveparade 1y ago
  If you don't know how to ask a human doctor a good question you can't expect to get a good answer either. The difference is that a human doctor probably has a lot of context about you and the situation you're in, so that they probably guess what your intention behind the question is, and adjust their answer appropriately. When you talk to an LLM, it has none of that context. So the comparison isn't really fair. Has your mom ever asked you a computer question? Half of the time the question makes no sense and explaining to her why would take hours, and then she still wouldn't get it. So the best you can do is guess what she wants based on the context you have.
deadbabe 1y ago
Doesn’t matter. The LLM’s job should be to deliver results in the way that is intended, regardless of the skill of the prompter. If it can’t do that then it’s really no better than a Google search.
- osigurdson 1y ago

conception 1y ago

Can you do a quick experiment and ask ChatGPT to craft a prompt to answer “laymen question “ as a financial expert? Having ChatGPT craft its own prompts is usually pretty successful for me.

lolinder 1y ago
That probably would work most of the time, but that's also an example of the phenomenon that TFA is talking about: you can't safely just use these tools without becoming an expert at least in the tool. The way they're currently being sold as totally accessible to everyone is dangerous.
- diggan 1y ago
  > without becoming an expert at least in the tool Yeah, we're basically repeating the "search engine/query" problem but slightly differently. Using a search engine the right way always been a skill you needed to learn, and the ones who didn't always got poor results, and many times took those results at face value. Then Google started giving "answers" so if your query is shit, the "answer" most likely is too. Point is, I don't think this phenomenon is new, it's just way less subtle today with LLMs, at least for people who have expertise in the subjects.
  - solarwindy 1y ago
    > On two occasions I have been asked, ’Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
osigurdson 1y ago
Understanding the terminology can help a lot. The key is to start the LLM conversation like this, ask it what terminology is used in a particular problem domain and then frame your actual question based on this.

colinmorelli 1y ago

Related similar thing when I sent my dog's recent bloodwork to an LLM, including dates, tests, and values. The model suggested that an advancement in her kidney values (all still within normal range) were likely evidence of chronic kidney disease in its early stage. Naturally this caused some concern for my wife. But, I work in healthcare and have enough knowledge of health to know that CKD almost certainly could not advance fast enough to be the cause of the kidney value changes in the labs that were only 6 weeks apart. I asked the LLM if that's the best explanation for these values given they're only 6 weeks apart, and it adjusted its answer to say CKD is likely not the explanation as progression would happen typically over 6+ months to a year at this stage, and more likely explanations were nephrotoxins (recent NSAID use), temporary dehydration, or recent infection. We then spoke to our vet who confirmed that CKD would be unlikely to explain a shift in values like this between two tests that were just 6 weeks apart. That would almost certainly throw off someone with less knowledge about this, however. If the tests were 4-6 months apart, CKD could explain the change. It's not an implausible explanation, but it skipped over a critical piece of information (the time between tests) before originally coming to that answer.

osigurdson 1y ago
The internet, and now LLMs have always been bad at diagnosing medical problems. I think it comes from the data source. For instance, few articles would be linked to / popular if a given set of symptoms were just associated with not getting enough sleep. No, the articles stand out are the ones where the symptoms are associated with some rare / horrible condition. This is our LLM training data which are often missing the entire middle part of the bell curve.
- colinmorelli 1y ago
  For what it's worth this statement is actually not entirely correct anymore. Top-end models today are on par with diagnostic capabilities of physicians on average (across many specialties), and, in some cases, can outperform them when RAG'd in with vetted clinical guidelines (like NIH data, UpToDate, etc) However, they do have particular types of failure modes that they're more prone to, and this is one of them. So they're imperfect.

rvz 1y ago

Also see the Gell-Mann amnesia effect: [0] [0] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

cjohnson318 1y ago

I had a similar experience. I did some back of the envelope math and my wife suggested I run it through ChatGPT. After actually doing the math, I felt a lot better about my understanding of the problem and I just... don't trust an LLM to understand algebra. Yeah, they're awesome most of the time, but I don't want to trust it with something important, be wrong, and then have to explain to someone that I trusted the opinion of a couple of big matrices, over my own knowledge and experience, on a high school word problem.

aaronbaugher 1y ago
I've noticed Grok struggles with dates and relative time, especially when referring to things it "remembers" from earlier conversations. Even a phrase like "last night" will be wrongly interpreted sometimes. So although I've had it research numbers and create estimates for me, I wouldn't just assume the numbers are right without checking everything.
lblume 1y ago
Since LLMs are basically linear algebra all the way down, this is vaguely reminiscent of how human brains also have a very hard time understanding neural circuitry despite literally being made from it.

disambiguation 1y ago

This is my experience with prompting as well, but I struggle to describe it adequately. Something like the "direction" of the prompt, if it's too open ended you're likely to get mixed results, but if you give it a kind of "running start" it performs much better.

caust1c 1y ago

This anecdote corroborates my theory that it will still be critical to become an expert in your field. Everyone is treating AI like it's a zero-sum game with regards to jobs being "lost" to AI, but the reality is that the best results will come from experts in the field who have the vocabulary and knowledge to get the best answers. My fear is that people treat AI like an oracle when they should be treating it just like any other human being.

lblume 1y ago
What percentage of people are actually experts at their jobs though?
xnorswap 1y ago
People treat certain humans, or humans in certain roles, as oracles too.
redeye100 1y ago
This is just bad design. Or a faulty tool. Why should the job market shift to accommodate this gap in the functioning of LLMs? This is a bug that needs to be fixed. I have a personal gripe about this bringing an unfinished tool to market and then prophetizing about its usefulness. And how we all better get ready for it. This seems very hand-wavey and is looking more and more like vaporware. It's like trying to quickly build a house on an unfinished foundation. Why are we rushing to build? Can't we get the foundational things right first?

chriskanan 1y ago

What I do is to always set the context by giving the my "background" and some papers as reading material such that I've conditioned the model for whatever topic that will be discussed as the first step.

aleph_minus_one 1y ago

For the sake of discussion I want to play devil's advocate concerning your point > It was an interesting result to me because it shows that experts in a field are not only more likely to recognize when a model is giving incorrect answers but they're also more likely to get correct answers because they are able to tap into a set of weights that are populated by text that knew what it was talking about. Lay people trying to use an LLM to understand an unfamiliar field are vulnerable to accidentally tapping into the "amateur" weights and ending up with an answer learned from random Reddit threads or SEO marketing blog posts, whereas experts can use jargon correctly in order to tap into answers learned from other experts. Couldn't it be the case that people who (in this case recognizable to the AI by their choice of wording) are knowledgeable in the topic need different advise than people who know less about the topic? To give one specific examples from finance: if you know a lot about finance, getting some deep analysis and advice about what is the best way to trade some exotic options is likely sound advice. On the other hand, for people who are not deeply into finance the best advice is likely rather "don't do it!".

lolinder 1y ago
In some cases, sure, but not here—neither option had more risk associated with it than the other, it was just an optimization problem. The first answer that the model gave to my wife was just wrong about the math, with no room for subjectivity.
dogleash 1y ago
> Couldn't it be the case [...] need different advise than people who know less about the topic? > for people who are not deeply into finance the best advice is likely rather "don't do it!". Oh boy, more nanny software. This future blows.
- aleph_minus_one 1y ago
  > Oh boy, more nanny software. This future blows. I think this topic is a little bit more complicated: this is rather a balancing of the model between 1. "giving the best possible advice to the respective person given their circumstances" vs 2. "giving the most precise answer to the query to the user" (if you ask me: the best decision would in my opinion be to give the user a choice for this, but this would be overtaxing to many users) - Freedom-loving people will hate it if they don't get 2 - On the other hand, many people would like to actually get the advice that is most helpful to them (i.e. 1), and not the one that may answer their question exactly, but is likely a bad idea for them

c22 1y ago

This same phenomenon is true for classic search engines as well. Whenever I am becoming informed on a new topic my first searches are always very naive and targeted just at discovering the relevant jargon that will let me make better searches. It turns out that many disciplines contain analogue concepts, just with different words being used to describe them. Understanding the domain specific language used is more than half the battle.

nothercastle 1y ago
Yeah I reminder that being the trick to get Google to provide good results was to find some key industry or area terms for what you were looking for. This doesn’t work anymore because Google search has gotten so bad
skydhash 1y ago
If I’m dealing with an unfamiliar domain, my next step is always wikipedia or an introductory book. Just to collect a sets of keywords and references to narrow my future searches. I don’t think I’ve ever asked google a question shaped query.
citrin_ru 1y ago
The difference is that in the search engine output one can somewhat judge if a site is trustworthy. E. g. if I research a legal question I would trust a guide on a government website more than an answer on Reddit which contradicts an official source. With LLM I don’t know where weight for a particular answer came from. You can ask LLM for sources but its text answer can still misrepresent these sources.

lostmsu 1y ago

A couple more alternative explanations: 1. random - sample size is too small to claim otherwise 2. phrasing in both cases had a leaning in it, and as a "yes man" LLM gave corresponding biased response.

whstl 1y ago

I very often also get better programming results than less experienced engineers, even though I'm not remotely doing any kind of "prompt engineering". Also how you ask matters a lot. Sometimes it just wants to make you happy with whatever answer, if you go along without skepticism it will definitely make garbage. Fun story: at a previous job a Product Manager made someone work a full week on a QR-Code standard that doesn't exist, except in ChatGPT's mind. It produced test cases and examples, but since nobody had a way to test When it was sent to a bank in Sweden to test, the customer was just "wait this feature doesn't exist in Sweden" and a heated discussion ensued until the PM admitted using ChatGPT to create the requirements.

jimbokun 1y ago

So LLMs need an ELI5 mode and detect when to use it. This would use the non-technical terminology but pull concepts from the more jargony model. This may or may not be easily possible by tweaking current training techniques. But it shows the many edge cases still needed to be addressed by AI models.

ivanjermakov 1y ago

LLM users are highly susceptible to the confirmation bias, by putting their expectations into the prompt.

tda 1y ago

Interesting observation. I find myself using ChatGPT to find the proper words for something. I describe my problem or algorithm in a naive way, and usually ChapGPT will present some naive python solution. But when pressed ChatGPT will tell you this problem is actually say Graph Theory. But it will still fall back to the naive python solution. So then I just start a new "conversation" and ask which graph algorithm applies to my problem and instead of a naive solution I am pointed to an optimised algorithm and a usable library that implements it.